CVS commit: pkgsrc/net/ndpi

["Adam Ciarcinski" <adam%netbsd.org@localhost>]

Module Name:    pkgsrc
Committed By:   adam
Date:           Mon Nov 20 17:57:31 UTC 2023

Modified Files:
        pkgsrc/net/ndpi: Makefile distinfo
        pkgsrc/net/ndpi/patches: patch-src_include_ndpi__includes.h
            patch-src_lib_Makefile.in

Log Message:
ndpi: updated to 4.8

4.8 Stable

Major Changes

Reworked lists implementation that decreased memory usage of orders of magnitude
Improved code robustness via extensive code fuzzing
Various improvements to overall library performance
Extended IPv6 support

New Supported Protocols and Services

Add "Heroes of the Storm" video game signature detection.
Add Apache Thrift protocol dissector.
Add Remote Management Control Protocol (RMCP).
Add Service Location Protocol dissector.
Add VK detection
Add Yandex services detection
Add a new protocol id for generic Adult Content traffic
Add a new protocol id for generic advertisement/analytics/tracking stuff
Add bitcoing protocol dissector.
Add detection of Roblox games
Add support for (un-encrypted) HTTP/2
Add support for Epic Games and GeForceNow/Nvidia
Add support for SRTP
Added BACnet dissector.
Added HAProxy protocol.
Added OICQ dissector.
Added OperaVPN detection
ProtonVPN: add basic detection
Added detection of Facebook Reels and Stories
Add an heuristic to detect fully encrypted flows
Added NDPI_MALWARE_HOST_CONTACTED flow risk
Added NDPI_TLS_ALPN_SNI_MISMATCH flow risk

Improvements

Improve protocol detection for:
FreeBSD compilation fix (C) update
Gnutella: improve detection
H323: fix false positives
HTTP: fix another memory access error
HTTP: fix extraction of filename
HTTP: fix heap-buffer-overflow
HTTP: improve extraction of metadata and of flow risks
HTTP: remove useless code about XBOX
HTTP: rework state machine
Hangout: detect Hangout/Duo/GoogleMeet/... in the STUN code
Enhance DNS risk for long hostnames (> 32)
Enhanced MS teams STUN/Azure detection
Enhanced custom port definition and improved error reporting in case of duplications
Improve detection of Alibaba flows
Improve detection of crawler/bot traffic
Improve detection of crawlers/bots
Improved MGCP detection by allowing '\r' as line feed.
Improved MS Teams detection with heuristic
Improved Steam detection by adding steamdiscover pattern.
Improved Wireguard detection
Improved checks for duplicated entries in protocols file
Improved classification further reducing memory used
Improved detection of invalid chars in DNS names
Improved domain search tet unit
Improved helper scripts.
MS Teams enhancement
MySql: improve detection
zabbix: improve detection

Tools

ndpiReader: allow to configure LRU caches TTL and size
ndpiReader: fix VXLAN de-tunneling
ndpiReader: fix export of DNS/BitTorrent attributes
ndpiReader: fix export of HTTP attributes
ndpiReader: fix flow stats
ndpiReader: fix print of flow payload
ndpiReader: improve printing of payload statistics
ndpiReader: print how many packets (per flow) were needed to perform full DPI
ndpireader: fix detection of DoH traffic based on packet distributions

Misc

ARM compilation fix
Add ndpi_domain_classify_finalize() function
Add a configuration knob to enable/disable loading of gambling list
Add a new flow risk about literal IP addresses used as SNI
Add an heuristic to detect/ignore some anomalous TCP ACK packets
Add another example of custom rules
Add support for multiline json
Add support for roaring_bitmap_xor_inplace
Add support for vxlan decapsulation
Added Source Engine dissector.
Added lists/gambling.list to extra dist.
Added slackb.com SNI.
Added ability to define an unlimited number of custom rules IP:port for the same IP (it used tobe limited to 2)
Added check to avoid skype heuristic false positives
Added comment
Added coverage targets to Makefile.am for convenience.
Added fix for better handling exceptions rollback in case of later match
Added hyperlink
Added ndpi_binary_bitmap data structure
Added ndpi_bitmap64 support
Added ndpi_bitmap_andnot API call
Added ndpi_bitmap_copy() API call
Added ndpi_bitmap_is_empty() and ndpi_bitmap_optimize() API calls
Added ndpi_domain_classify_XXX(0 API
Added ndpi_filter_add_multi() API call
Added ndpi_murmur_hash to the nDPI API
Added new API calls for implementing Bloom-filter like data structures
Added printf/fprintf replacement for some internal modules.
Added scripts to auto generate hostname/SNI *.inc files.
Added sub-domain classification fix
Added the ability to define custom protocols with arbitrary Ids in proto.txt
Added vlan_id in ndpi_flow2json() prototype
Adds new pcap for testing "funny" HTTP servers
All protocols should be excluded sooner or later
Allow init of app protocols w/o any hostnames set.
Avoid calling ndpi_reconcile_protocols() twice in ndpi_detection_giveup()
Boundary check
CI: fix Performance job
Centos7 fixes
Changed logging callback function sig.
Changes for supporting more efficient sub-string matching
Classification fixes
DNS: extract geolocation information, if available
Debian 12 fixes
Disabled query string validation in MDNS in order to avoid zapping chars that in DNS (instead) are not permitted
DisneyPlus/Hulu ip lists should be auto-generated
Extend content list of Microsoft protocols
Extend content-match list
Fix LRU/Patricia/Automa stats in ndpiReader with multiple threads
Fix MS Teams detection with heuristic
Fix access to packet/flow information
Fix an heap-buffer-overflow
Fix classification-by-ip in ndpi_detection_giveup
Fix compilation
Fix compilation in CI jobs
Fix compilation on Windows
Fix compilation with GCC-7 and latest RoaringBitmap code
Fix detection of packet direction and NDPI_UNIDIRECTIONAL_TRAFFIC risk
Fix export/serialization of flow->risk
Fix for buffer overflow in serialization
Fix insert of ip addresses into patricia tree(s)
Fix missing u_char, u_short and u_int typedefs for some platforms e.g.:
Fix packet counters
Fix some errors found by fuzzers
Fix some memory errors triggered by allocation failures
Fix some prototypes
Fix string truncation.
Fixed OpenWRT arm related build issues.
Fixed heap-buffer-overflow issue
Fixed heap-overflow if compiled with --enable-tls-sigs.
Fixed invalid use of ndpi_free(). Sorry, my fault.
Fixed missing AS_HELP_STRING in configure.ac.
Fixed two OpenWRT arm related build issues.
Fixes matches with domain name strings that start with a dot
Fixes risk mask exception handling while improving the overall performance
Implemented Count-Min Sketch [count how many times a value has been observed]
Implemented Zoom/Teams stream type detection
Implemented ndpi_XXX_reset() API calls whre XXX is ses, des, hw
Implemented ndpi_predict_linear() for predicting a timeseries value overtime
Improved debug output.
Improved invalid logging via printf().
Improved line protocol dissection with heuristic
Improved missing usage of nDPIs malloc wrapper.
Improved protocol detection exploiting IP-based guess Reworked ndpi_reconcile_protocols() that is now called only in front of a match (less overhead)
Improvement for reducing false positives
Included Gambling website data from the Polish hazard.mf.gov.pl list
Keep master protocol in ndpi_reconcile_protocols
Leak fix
Language fix
Line: fix heap-buffer-overflow error
Made VK protocol detection more strict
Make Bittorrent LRU cache IPv6 aware.
Merged new and old version of ndpi_domain_classify.c code
Mullvad VPN service added (based on entry node IP addresses)
Numeric truncation at ndpi_analyze.c at lines 101, 104, 107, 110
Numeric truncation at tls.c:1010
Ookla: rework detection
Optimizes and fixes possible out0of0boundary write in ndpi_fill_prefix_v4()
ProtonVPN: split the ip list
QUIC: add support for QUIC version 2
QUIC: export QUIC version as metadata
QUIC: fix a memory access error
QUIC: fix dissection of packets forcing VN
RDP: improve detection over UDP
RTP: remove dead-code
RTP: rework code
Refreshed ASN lists Enhanced the Line IP list with https://ipinfo.io/AS23576/125.209.252.0/24 used by line
Remove some useless checks
Remove special handling of some TCP flows without SYN
Removed overlapping port
Renamed HTTP/2 to HTTP2 as the '/' can have side effects with applications sitting on top of nDPI
Replaces free() with ndpi_free()
Rework CI jobs to try reducing CI duration
Reworked domain classification based on binary filters
Reworked initialization
Reworked ndpi_filter_xxx implementation using compressed bitmaps
Reworked teams handling
RiotGames: add detection of flows
STUN: add dissection of DTLS handshake
STUN: avoid FacebookVoip false positives
STUN: fix Skype/MsTeams detection and monitoring logic
STUN: fix detection of Google Voip apps
STUN: fix detection over TCP
STUN: improve WhatsappCall detection
STUN: keep monitoring/processing STUN flows
STUN: tell RTP from RTCP while in monitoring state
Serialization fix
Set _DEFAULT_SOURCE and _GNU_SOURCE globally.
Simplify ndpi_internal_guess_undetected_protocol()
Simplify the report of streaming multimedia info
SoftEther: fix invalid memory access
Swap from Aho-Corasick to an experimental/home-grown algorithm that uses a probabilistic approach for handling Internet domain names.
Sync unit tests results
Sync unit tests results
Sync unit tests results
Sync utests results
TLS: add basic, basic, detection of Encrypted ClientHello
TLS: fix another interger overflow in certificate processing
TLS: fix parsing of certificate elements
Test files for riit games
Test multiple ndpiReader configurations
Thrift: fix heap-buffer-overflow
Update GitHub runners versions
Update every ip lists
Update libinjection code
Update protocols documentation
Update roaring bitmap code
Updated line test result
Updated pcap detection results after Facebook Reel/Stories support
Updated results
Updated results after the latest changes
Win include change
Windows code rework
Windows compilation fixes
Windows warning checks
add 2 ns from fdn.fr to DoH section
add support for gre decapsulation
added bimap and/or with allocation
added feature to extract filename from http attachment
added new domain names
configure: add an option to enable debug build, i.e -g
fix Stack overflow caused by invalid write in ndpi_automa_match_strin…
fixed numeric truncation error
fixed numeric truncation error in diameter.c
fixed numeric truncation error in kerberos.c
fixed numeric truncation error in ndpi_main.c:6837
fixed numeric truncation error in rtcp.c
fuzz: add a new fuzzer to test TLS certificates
fuzz: add a new fuzzer triggering the payload analyzer function(s)
fuzz: add fuzzer for DGA detection code
fuzz: add fuzzer to test internal gcrypt code
fuzz: add fuzzers to test bitmap64 and domain_classify data structures
fuzz: add fuzzers to test reader_util code
fuzz: extend coverage
fuzz: extend fuzz coverage
fuzz: extend fuzzers coverage
fuzz: extend fuzzing coverage
fuzz: extend fuzzing coverage
fuzz: extend fuzzing coverage
fuzz: simplify fuzzers dependencies in CIFuzz
fuzz: some improvements and add two new fuzzers
fuzzing: extend fuzzing coverage
in case of failure, failing result files are not listed
minor fixes
oss-fuzz: sync build script with upstream
remove redefinition to vxlanhdr struct in vxlan dissector
removed useless call of ndpi_set_risk func
tests: add an option to force the overwrite of the unit tests results
tests: restore some old paths as symbolic links
tftp: check for Option Acknowledgements
tftp: check incrementation for DATA and ACK packets
tftp: rework request checking to account for options
tftp: update pcap results
version of dirent.c that is liked by both VC++ and MinGW


To generate a diff of this commit:
cvs rdiff -u -r1.12 -r1.13 pkgsrc/net/ndpi/Makefile pkgsrc/net/ndpi/distinfo
cvs rdiff -u -r1.1 -r1.2 \
    pkgsrc/net/ndpi/patches/patch-src_include_ndpi__includes.h
cvs rdiff -u -r1.6 -r1.7 pkgsrc/net/ndpi/patches/patch-src_lib_Makefile.in

Please note that diffs are not public domain; they are subject to the
copyright notices on the relevant files.

Modified files:

Index: pkgsrc/net/ndpi/Makefile
diff -u pkgsrc/net/ndpi/Makefile:1.12 pkgsrc/net/ndpi/Makefile:1.13
--- pkgsrc/net/ndpi/Makefile:1.12       Tue Nov  7 22:31:11 2023
+++ pkgsrc/net/ndpi/Makefile    Mon Nov 20 17:57:31 2023
@@ -1,8 +1,7 @@
-# $NetBSD: Makefile,v 1.12 2023/11/07 22:31:11 he Exp $
+# $NetBSD: Makefile,v 1.13 2023/11/20 17:57:31 adam Exp $
 
-DISTNAME=      nDPI-4.6
+DISTNAME=      nDPI-4.8
 PKGNAME=       ${DISTNAME:tl}
-PKGREVISION=   1
 CATEGORIES=    net
 MASTER_SITES=  ${MASTER_SITE_GITHUB:=ntop/}
 GITHUB_PROJECT=        nDPI
Index: pkgsrc/net/ndpi/distinfo
diff -u pkgsrc/net/ndpi/distinfo:1.12 pkgsrc/net/ndpi/distinfo:1.13
--- pkgsrc/net/ndpi/distinfo:1.12       Tue Nov  7 22:31:11 2023
+++ pkgsrc/net/ndpi/distinfo    Mon Nov 20 17:57:31 2023
@@ -1,8 +1,8 @@
-$NetBSD: distinfo,v 1.12 2023/11/07 22:31:11 he Exp $
+$NetBSD: distinfo,v 1.13 2023/11/20 17:57:31 adam Exp $
 
-BLAKE2s (nDPI-4.6.tar.gz) = 43db6d82a24789622f8afe1f18ba3dcd33f5eb92b76f23932732789f47e50937
-SHA512 (nDPI-4.6.tar.gz) = 12dd50eeaae0aa5c2e3c89cb53cdb9989036759bd900312d3875eaba296646c2e42be31dc2b003cbc2401ccef8fbf8d2e460b9a604d7bece360f17f937d60cad
-Size (nDPI-4.6.tar.gz) = 29705235 bytes
-SHA1 (patch-src_include_ndpi__includes.h) = db5da9ca24ecf32fd8c264e193af280814dc74c6
-SHA1 (patch-src_lib_Makefile.in) = 2f0dd8c5bacf1f1036bfb81b1139e48131122592
+BLAKE2s (nDPI-4.8.tar.gz) = 80693cd17d13f8225b52a76e504093e59c118ec3388b68ec3b8a24866aa28565
+SHA512 (nDPI-4.8.tar.gz) = f76eb99a83262a6d148509c7ce457b9af35dce3657c8191405ba15999b25c323ee0bee2907b0a7e6ba6f12a63182fde839e0098a4e8e0f63999e600526f3036b
+Size (nDPI-4.8.tar.gz) = 39479237 bytes
+SHA1 (patch-src_include_ndpi__includes.h) = f6aa1dff14da61f93e6d330be7adfe1b85772f14
+SHA1 (patch-src_lib_Makefile.in) = 4bc3a07334f689306842538590cf9509969851e4
 SHA1 (patch-src_lib_third__party_include_gcrypt_common.h) = 257a2c9b08793f8bca564dc7c6229f1d5168b5e1

Index: pkgsrc/net/ndpi/patches/patch-src_include_ndpi__includes.h
diff -u pkgsrc/net/ndpi/patches/patch-src_include_ndpi__includes.h:1.1 pkgsrc/net/ndpi/patches/patch-src_include_ndpi__includes.h:1.2
--- pkgsrc/net/ndpi/patches/patch-src_include_ndpi__includes.h:1.1      Thu Jun 15 10:06:39 2017
+++ pkgsrc/net/ndpi/patches/patch-src_include_ndpi__includes.h  Mon Nov 20 17:57:31 2023
@@ -1,12 +1,12 @@
-$NetBSD: patch-src_include_ndpi__includes.h,v 1.1 2017/06/15 10:06:39 fhajny Exp $
+$NetBSD: patch-src_include_ndpi__includes.h,v 1.2 2023/11/20 17:57:31 adam Exp $
 
 Same case with SunOS.
 
---- src/include/ndpi_includes.h.orig   2016-06-27 19:31:24.000000000 +0000
+--- src/include/ndpi_includes.h.orig   2023-10-23 09:45:30.000000000 +0000
 +++ src/include/ndpi_includes.h
-@@ -46,7 +46,7 @@
- #include <netinet/tcp.h>
+@@ -50,7 +50,7 @@
  #include <netinet/udp.h>
+ #endif
  
 -#if !defined __APPLE__ && !defined __FreeBSD__ && !defined __NetBSD__ && !defined __OpenBSD__
 +#if !defined __APPLE__ && !defined __FreeBSD__ && !defined __NetBSD__ && !defined __OpenBSD__ && !defined __sun

Index: pkgsrc/net/ndpi/patches/patch-src_lib_Makefile.in
diff -u pkgsrc/net/ndpi/patches/patch-src_lib_Makefile.in:1.6 pkgsrc/net/ndpi/patches/patch-src_lib_Makefile.in:1.7
--- pkgsrc/net/ndpi/patches/patch-src_lib_Makefile.in:1.6       Thu Jun 29 18:03:17 2023
+++ pkgsrc/net/ndpi/patches/patch-src_lib_Makefile.in   Mon Nov 20 17:57:31 2023
@@ -1,11 +1,11 @@
-$NetBSD: patch-src_lib_Makefile.in,v 1.6 2023/06/29 18:03:17 adam Exp $
+$NetBSD: patch-src_lib_Makefile.in,v 1.7 2023/11/20 17:57:31 adam Exp $
 
 Fix building on Darwin.
 
---- src/lib/Makefile.in.orig   2023-02-01 17:29:59.000000000 +0000
+--- src/lib/Makefile.in.orig   2023-10-23 09:45:30.000000000 +0000
 +++ src/lib/Makefile.in
 @@ -25,8 +25,15 @@ OBJECTS   = $(patsubst protocols/%.c, pr
- HEADERS   = $(wildcard ../include/*.h)
+ HEADERS   = $(wildcard ../include/*.h) $(wildcard ../include/*.h)
  NDPI_VERSION_MAJOR   = @NDPI_MAJOR@
  NDPI_LIB_STATIC      = libndpi.a
 +ifeq ($(OS),Darwin)
@@ -24,7 +24,7 @@ Fix building on Darwin.
  BUILD_MINGW                    = @BUILD_MINGW@
  
  ifeq ($(OS),Darwin)
--CC=clang
+-CC=clang -fno-color-diagnostics
 -SONAME_FLAG=
 +SONAME_FLAG=-install_name $(PREFIX)$(libdir)/${NDPI_LIB_SHARED}
  else