CVS commit: pkgsrc/net/ndpi

To: pkgsrc-changes%NetBSD.org@localhost
Subject: CVS commit: pkgsrc/net/ndpi
From: "Adam Ciarcinski" <adam%netbsd.org@localhost>
Date: Mon, 28 Mar 2022 19:24:14 +0000

Module Name:    pkgsrc
Committed By:   adam
Date:           Mon Mar 28 19:24:14 UTC 2022

Modified Files:
        pkgsrc/net/ndpi: Makefile PLIST distinfo
        pkgsrc/net/ndpi/patches: patch-src_lib_Makefile.in

Log Message:
ndpi:updated to 4.2

nDPI4.2 (Feb 2022)

New Features
- Add a "confidence" field indicating the reliability of the classification
- Add risk exceptions for services and domain names via ndpi_add_domain_risk_exceptions()
- Add ability to report whether a protocol is encrypted

New Supported Protocols and Services
- Add protocol detection for:
  - Badoo
  - Cassandra
  - EthernetIP

Improvements
- Reduce memory footprint
- Improve protocol detection for:
  - BitTorrent
  - ICloud Private Relay
  - IMAP, POP3, SMTP
  - Log4J/Log4Shell
  - Microsoft Azure
  - Pandora TV
  - RTP
  - RTSP
  - Salesforce
  - STUN
  - Whatsapp
  - QUICv2
  - Zoom
- Add flow risk:
  - NDPI_CLEAR_TEXT_CREDENTIALS
  - NDPI_POSSIBLE_EXPLOIT (Log4J)
  - NDPI_TLS_FATAL_ALERT
  - NDPI_TLS_CERTIFICATE_ABOUT_TO_EXPIRE
- Update WhatsAPP and Instagram addresses
- Update the list of default ports for QUIC
- Update WindowsUpdate URLs
- Add support for the .goog Google TLD
- Add googletagmanager.com
- Add bitmaps and API for handling compressed bitmaps
- Add JA3 in risk exceptions
- Add entropy calculation to check for suspicious (encrypted) payload
- Add extraction of hostname in SMTP
- Add RDP over UDP dissection
- Add support for TLS over IPV6 in Subject Alt Names field
- Improve JSON and CSV serialization
- Improve IPv6 support for almost all dissectors
- Improve CI and unit tests, add arm64, armhf and s390x as part of CI
- Improve WHOIS detection, reduce false positives
- Improve DGA detection for skipping potential DGAs of known/popular domain names
- Improve user agent analysis
- Reworked HTTP protocol dissection including HTTP proxy and HTTP connect

Changes
- TLS obsolete protocol is set when TLS < 1.2 (used to be 1.1)
- Numeric IPs are not considered for DGA checks
- Differentiate between standard Amazon stuff (i.e market) and AWS
- Remove Playstation VUE protocol
- Remove pandora.tv from Pandora protocol
- Remove outdated SoulSeek dissector

Fixes
- Fix race conditions
- Fix dissectors to be big-endian friendly
- Fix heap overflow in realloc wrapper
- Fix errors in Kerberos, TLS, H323, Netbios, CSGO, Bittorrent
- Fix wrong tuple comparison
- Fix ndpi_serialize_string_int64
- Fix Grease values parsing
- Fix certificate mismatch check
- Fix null-dereference read for Zattoo with IPv6
- Fix dissectors initialization for XBox, Diameter
- Fix confidence for STUN classifications
- Fix FreeBSD support
- Fix old GQUIC versions on big-endian machines
- Fix aho-corasick on big-endian machines
- Fix DGA false positive
- Fix integer overflow for QUIC
- Fix HTTP false positives
- Fix SonarCloud-CI support
- Fix clashes setting the hostname on similar protocols (FTP, SMTP)
- Fix some invalid TLS guesses
- Fix crash on ARM (Raspberry)
- Fix DNS (including fragmented DNS) dissection
- Fix parsing of IPv6 packets with extension headers
- Fix extraction of Realm attribute in STUN
- Fix support for START-TLS sessions in FTP
- Fix TCP retransmissions for multiple dissectors
- Fix DES initialisation
- Fix Git protocol dissection
- Fix certificate mismatch for TLS flows with no client hello observed
- Fix old versions of GQUIC on big-endian machines

Misc
- Add tool for generating automatically the Azure IP list

nDPI 4.0 (July 2021)

New Features
- Add API for computing RSI (Relative Strenght Index)
- Add GeoIP support
- Add fragments management
- Add API for jitter calculation
- Add single exponential smoothing API
- Add timeseries forecasting support implementing Holt-Winters with confidence interval
- Add support for MAC to radi tree and expose the full API to applications
- Add JA3+, with ALPN and elliptic curve
- Add double exponential smoothing implementation
- Extended API for managing flow risks
- Add flow risk score
- New flow risks:
  - Desktop or File Sharing Session
  - HTTP suspicious content (useful for tracking trickbot)
  - Malicious JA3
  - Malicious SHA1
  - Risky domain
  - Risky AS
  - TLS Certificate Validity Too Long
  - TLS Suspicious Extension

New Supported Protocols and Services
- New protocols:
  - AmongUs
  - AVAST SecureDNS
  - CPHA (CheckPoint High Availability Protocol)
  - DisneyPlus
  - DTLS
  - Genshin Impact
  - HP Virtual Machine Group Management (hpvirtgrp)
  - Mongodb
  - Pinterest
  - Reddit
  - Snapchat VoIP calls
  - Tumblr
  - Virtual Asssitant (Alexa, Siri)
  - Z39.50
- Add protocols to HTTP as subprotocols
- Add detection of TLS browser type
- Add connectionless DCE/RPC detection

Improvements
  - 2.5x speed bump. Example ndpiReader with a long mixed pcap
       v3.4 - nDPI throughput:       1.29 M pps / 3.35 Gb/sec
       v4.0 - nDPI throughput:       3.35 M pps / 8.68 Gb/sec
 - Improve detection/dissection of:
  - AnyDesk
  - DNS
  - Hulu
  - DCE/RPC (avoid false positives)
  - dnscrypt
  - Facebook (add new networks)
  - Fortigate
  - FTP Control
  - HTTP
    - Fix user-agent parsing
    - Fix logs when NDPI_ENABLE_DEBUG_MESSAGES is defined
  - IEC104
  - IEC60870
  - IRC
  - Netbios
  - Netflix
  - Ookla speedtest (detection over IPv6)
  - openspeedtest.com
  - Outlook / MicrosoftMail
  - QUIC
    - update to draft-33
    - improve handling of SNI
    - support for fragmented Client Hello
    - support for DNS-over-QUIC
  - RTSP
  - RTSP via HTTP
  - SNMP (reimplemented)
  - Skype
  - SSH
  - Steam (Steam Datagram Relay - SDR)
  - STUN (avoid false positives, improved Skype detection)
  - TeamViewer (add new hosts)
  - TOR (update hosts)
  - TLS
    - Certificate Subject matching
    - Check for common ALPNs
    - Reworked fingerprint calculation
    - Fix extraction for TLS signature algorithms
    - Fix ClientHello parsing
  - UPnP
  - wireguard
- Improve DGA detection
- Improve JA3
- Improve Mining detection
- Improve string matching algorithm
- Improve ndpi_pref_enable_tls_block_dissection
- Optimize speed and memory size
- Update ahocorasick library
- Improve subprotocols detection

Fixes
- Fix partial application matching
- Fix multiple segfault and leaks
- Fix uninitialized memory use
- Fix release of patterns allocated in ndpi_add_string_to_automa
- Fix return value of ndpi_match_string_subprotocol
- Fix setting of flow risks on 32 bit machines
- Fix TLS certificate threshold
- Fix a memory error in TLS JA3 code
- Fix false positives in Z39.50
- Fix off-by-one memory error for TLS-JA3
- Fix bug in ndpi_lru_find_cache
- Fix invalid xbox and playstation port guesses
- Fix CAPWAP tunnel decoding
- Fix parsing of DLT_PPP datalink type
- Fix dissection of QUIC initial packets coalesced with 0-RTT one
- Fix parsing of GTP headers
- Add bitmap boundary checks

Misc
- Update download category name
- Update category labels
- Renamed Skype in Skype_Teams (the protocol is now shared across these apps)
- Add IEC analysis wireshark plugin
- Flow risk visualization in Wireshark
- ndpiReader
  - add statistics about nDPI performance
  - fix memory leak
  - fix collecting of risks statistics
- Move installed libraries from /usr/local to /usr
- Improve NDPI_API_VERSION generation
- Update ndpi_ptree_match_addr prototype


To generate a diff of this commit:
cvs rdiff -u -r1.9 -r1.10 pkgsrc/net/ndpi/Makefile pkgsrc/net/ndpi/distinfo
cvs rdiff -u -r1.5 -r1.6 pkgsrc/net/ndpi/PLIST
cvs rdiff -u -r1.4 -r1.5 pkgsrc/net/ndpi/patches/patch-src_lib_Makefile.in

Please note that diffs are not public domain; they are subject to the
copyright notices on the relevant files.

Modified files:

Index: pkgsrc/net/ndpi/Makefile
diff -u pkgsrc/net/ndpi/Makefile:1.9 pkgsrc/net/ndpi/Makefile:1.10
--- pkgsrc/net/ndpi/Makefile:1.9        Wed Jun 23 19:31:49 2021
+++ pkgsrc/net/ndpi/Makefile    Mon Mar 28 19:24:14 2022
@@ -1,6 +1,6 @@
-# $NetBSD: Makefile,v 1.9 2021/06/23 19:31:49 adam Exp $
+# $NetBSD: Makefile,v 1.10 2022/03/28 19:24:14 adam Exp $
 
-DISTNAME=      nDPI-3.4
+DISTNAME=      nDPI-4.2
 PKGNAME=       ${DISTNAME:tl}
 CATEGORIES=    net
 MASTER_SITES=  ${MASTER_SITE_GITHUB:=ntop/}
Index: pkgsrc/net/ndpi/distinfo
diff -u pkgsrc/net/ndpi/distinfo:1.9 pkgsrc/net/ndpi/distinfo:1.10
--- pkgsrc/net/ndpi/distinfo:1.9        Tue Oct 26 11:06:06 2021
+++ pkgsrc/net/ndpi/distinfo    Mon Mar 28 19:24:14 2022
@@ -1,7 +1,7 @@
-$NetBSD: distinfo,v 1.9 2021/10/26 11:06:06 nia Exp $
+$NetBSD: distinfo,v 1.10 2022/03/28 19:24:14 adam Exp $
 
-BLAKE2s (nDPI-3.4.tar.gz) = 165355ad8959b4fd687955c8e7e9508e9393a66faae30ae276a8b690f6d8ae8f
-SHA512 (nDPI-3.4.tar.gz) = ed5a22b6ddc14ad707a18a0bd96746c1df489969faaa42016fa9aad8d414fc4ee303b96cac15c3ba86f484a80a0aaa2dd1be5f92be672912e0e0d30da4bdad4c
-Size (nDPI-3.4.tar.gz) = 37976087 bytes
+BLAKE2s (nDPI-4.2.tar.gz) = 21811cc94f7ce332fd620642025508f3fecd9f85fa9186c541f411d669d8c4e7
+SHA512 (nDPI-4.2.tar.gz) = c58b228b7de2b5fa111234bb065f9be259cd0282d0058f5985180aab6567c9a18e3453ce9062bd3c057a096ec3dfdc21b7c9680e0148e34af10f707530d853b8
+Size (nDPI-4.2.tar.gz) = 126559327 bytes
 SHA1 (patch-src_include_ndpi__includes.h) = db5da9ca24ecf32fd8c264e193af280814dc74c6
-SHA1 (patch-src_lib_Makefile.in) = 5e7abf75abf4d78af8ed9f1dfe9e755e68b0c0e4
+SHA1 (patch-src_lib_Makefile.in) = 0d6bf73f502203d8f9f77872d65cfcf8611201ee

Index: pkgsrc/net/ndpi/PLIST
diff -u pkgsrc/net/ndpi/PLIST:1.5 pkgsrc/net/ndpi/PLIST:1.6
--- pkgsrc/net/ndpi/PLIST:1.5   Wed Jun 23 19:31:49 2021
+++ pkgsrc/net/ndpi/PLIST       Mon Mar 28 19:24:14 2022
@@ -1,21 +1,24 @@
-@comment $NetBSD: PLIST,v 1.5 2021/06/23 19:31:49 adam Exp $
+@comment $NetBSD: PLIST,v 1.6 2022/03/28 19:24:14 adam Exp $
 bin/ndpiReader
 include/ndpi/ndpi_api.h
 include/ndpi/ndpi_classify.h
 include/ndpi/ndpi_config.h
 include/ndpi/ndpi_define.h
+include/ndpi/ndpi_encryption.h
 include/ndpi/ndpi_includes.h
 include/ndpi/ndpi_includes_OpenBSD.h
 include/ndpi/ndpi_main.h
+include/ndpi/ndpi_patricia_typedefs.h
 include/ndpi/ndpi_protocol_ids.h
 include/ndpi/ndpi_protocols.h
 include/ndpi/ndpi_typedefs.h
 include/ndpi/ndpi_unix.h
+include/ndpi/ndpi_utils.h
 include/ndpi/ndpi_win32.h
 lib/libndpi.a
 lib/libndpi.so
 lib/libndpi.so.${PKGVERSION}.0
-lib/libndpi.so.3
+lib/libndpi.so.4
 lib/pkgconfig/libndpi.pc
 share/ndpi/ndpiCustomCategory.txt
 share/ndpi/ndpiProtos.txt

Index: pkgsrc/net/ndpi/patches/patch-src_lib_Makefile.in
diff -u pkgsrc/net/ndpi/patches/patch-src_lib_Makefile.in:1.4 pkgsrc/net/ndpi/patches/patch-src_lib_Makefile.in:1.5
--- pkgsrc/net/ndpi/patches/patch-src_lib_Makefile.in:1.4       Wed Jun 23 19:31:49 2021
+++ pkgsrc/net/ndpi/patches/patch-src_lib_Makefile.in   Mon Mar 28 19:24:14 2022
@@ -1,8 +1,8 @@
-$NetBSD: patch-src_lib_Makefile.in,v 1.4 2021/06/23 19:31:49 adam Exp $
+$NetBSD: patch-src_lib_Makefile.in,v 1.5 2022/03/28 19:24:14 adam Exp $
 
 Fix building on Darwin.
 
---- src/lib/Makefile.in.orig   2020-10-19 14:18:25.000000000 +0000
+--- src/lib/Makefile.in.orig   2022-02-01 08:14:05.000000000 +0000
 +++ src/lib/Makefile.in
 @@ -22,8 +22,15 @@ OBJECTS   = $(patsubst protocols/%.c, pr
  HEADERS   = $(wildcard ../include/*.h)
@@ -32,13 +32,13 @@ Fix building on Darwin.
  NDPI_LIB_SHARED_BASE = libndpi
 @@ -54,7 +60,7 @@ $(NDPI_LIB_STATIC): $(OBJECTS)
  $(NDPI_LIB_SHARED): $(OBJECTS)
-       $(CC) -shared -fPIC $(SONAME_FLAG) -o $@ $(OBJECTS) $(LDFLAGS)
+       $(CC) -shared -fPIC $(SONAME_FLAG) -o $@ $(LDFLAGS) $(OBJECTS) $(LIBS)
        ln -fs $(NDPI_LIB_SHARED) $(NDPI_LIB_SHARED_BASE)
 -      ln -fs $(NDPI_LIB_SHARED) $(NDPI_LIB_SHARED_BASE).$(NDPI_VERSION_MAJOR)
 +      ln -fs $(NDPI_LIB_SHARED) $(NDPI_LIB_SHARED_SHORT)
  
  %.o: %.c $(HEADERS) Makefile
-       $(CC) $(CFLAGS) -c $< -o $@
+       $(CC) $(CPPFLAGS) $(CFLAGS) -c $< -o $@
 @@ -78,6 +84,6 @@ install: $(NDPI_LIBS)
        mkdir -p $(DESTDIR)$(libdir)
        cp $(NDPI_LIBS) $(DESTDIR)$(libdir)/

Prev by Date: CVS commit: pkgsrc/devel/ocaml-dune
Next by Date: CVS commit: pkgsrc/net/ntopng
Previous by Thread: CVS commit: pkgsrc/devel/ocaml-dune
Next by Thread: CVS commit: pkgsrc/net/ntopng
Indexes:

Home | Main Index | Thread Index | Old Index