[pkgsrc/trunk]: pkgsrc/net/ndpi ndpi: updated to 3.2

To: source-changes-hg%NetBSD.org@localhost
Subject: [pkgsrc/trunk]: pkgsrc/net/ndpi ndpi: updated to 3.2
From: adam <adam%pkgsrc.org@localhost>
Date: Mon, 25 May 2020 21:27:27 +0000
details:   https://anonhg.NetBSD.org/pkgsrc/rev/7d14dbbc46d1
branches:  trunk
changeset: 432829:7d14dbbc46d1
user:      adam <adam%pkgsrc.org@localhost>
date:      Mon May 25 20:25:22 2020 +0000

description:
ndpi: updated to 3.2

nDPI 3.2:

New Features
* New API calls
  * Protocol detection: ndpi_is_protocol_detected
  * Categories: ndpi_load_categories_file / ndpi_load_category
  * JSON/TLV serialization: ndpi_serialize_string_boolean / ndpi_serialize_uint32_boolean
  * Patricia tree: ndpi_load_ipv4_ptree
  * Module initialization: ndpi_init_detection_module / ndpi_finalize_initalization
  * Base64 encoding: ndpi_base64_encode
  * JSON exprot: ndpi_flow2json
  * Print protocol: ndpi_get_l4_proto_name / ndpi_get_l4_proto_info
* Libfuzz integration
* Implemented Community ID hash (API call ndpi_flowv6_flow_hash and ndpi_flowv4_flow_hash)
* Detection of RCE in HTTP GET requests via PCRE
* Integration of the libinjection library to detect SQL injections and XSS type attacks in HTTP requests

New Supported Protocols and Services
* TLS
  * Added ALPN support
  * Added export of supported version in TLS header
* Added Telnet dissector with metadata extraction
* Added Zabbix dissector
* Added POP3/IMAP metadata extraction
* Added FTP user/password extraction
* Added NetBIOS metadata extraction
* Added Kerberos metadata extraction
* Implemented SQL Injection and XSS attack detection
* Host-based detection improvements and changes
  * Added Microsoft range
  * Added twitch.tv website
  * Added brasilbandalarga.com.br and .eaqbr.com.br as EAQ
  * Added 20.180.0.0/14, 20.184.0.0/13 range as Skype
  * Added 52.84.0.0/14 range as Amazon
  * Added ^pastebin.com
  * Changed 13.64.0.0/11 range from Skype to Microsoft
  * Refreshed Whatsapp server list, added *whatsapp-*.fbcdn.net IPs
* Added public DNSoverHTTPS servers

Improvements
* Reworked and improved the TLS dissector
* Reworked Kerberos dissector
* Improved DNS response decoding
* Support for DNS continuous flow dissection
* Improved Python bindings
* Improved Ethereum support
* Improved categories detection with streaming and HTTP
* Support for IP-based detection to compute the application protocol
* Renamed protocol 104 to IEC60870 (more meaningful)
* Added failed authentication support with FTP
* Renamed DNSoverHTTPS to handle bot DoH and DoT
* Implemented stacked DPI decoding
* Improvements for CapWAP and Bloomberg
* Improved SMB dissection
* Improved SSH dissection
* Added capwap support
* Modified API signatures for ndpi_ssl_version2str / ndpi_detection_giveup
* Removed ndpi_pref_http_dont_dissect_response / ndpi_pref_dns_dont_dissect_response (replaced by ndpi_extra_dissection_possible)

Fixes
* Fixed memory invalid access in SMTP and leaks in TLS
* Fixed a few memory leaks
* Fixrd invalid memory access in a few protocol dissectors (HTTP, memcached, Citrix, STUN, DNS, Amazon Video, TLS, Viber)
* Fixed IPv6 address format across the various platforms/distributions
* Fixed infinite loop in ndpi_workflow_process_packet
* Fixed SHA1 certificate detection
* Fixed custom protocol detection
* Fixed SMTP dissection (including email)
* Fixed Telnet dissection and invalid password report
* Fixed invalid category matching in HTTP
* Fixed Skype and STUN false positives
* Fixed SQL Injection detection
* Fixed invalid SMBv1 detection
* Fixed SSH dissection
* Fixed ndpi_ssl_version2str
* Fixed ndpi_extra_dissection_possible
* Fixed out of bounds read in ndpi_match_custom_category

Misc
* ndpiReader
  * CSV output enhancements
  * Added tunnelling decapsulation
  * Improved HTTP reporting


nDPI 3.0:

New Features
* nDPI now reports the protocol ASAP even when specific fields have not yet been dissected because such packets have not yet been observed. This is important for inline applications that can 
immediately act on traffic. Applications that need full dissection need to call the new API function ndpi_extra_dissection_possible() to check if metadata dissection has been completely performed or 
if there is more to read before declaring it completed.
* TLS (formerly identified as SSL in nDPI v2.x) is now dissected more deeply, certificate validity is extracted as well certificate SHA-1.
* nDPIreader can now export data in CSV format with option `-C`
* Implemented Sequence of Packet Length and Time (SPLT) and Byte Distribution (BD) as specified by Cisco Joy (https://github.com/cisco/joy). This allows malware activities on encrypted TLS streams. 
Read more at https://blogs.cisco.com/security/detecting-encrypted-malware-traffic-without-decryption
  * Available as library and in `ndpiReader` with option `-J`
* Promoted usage of protocol categories rather than protocol identifiers in order to classify protocols. This allows application protocols to be clustered in families and thus better managed by 
users/developers rather than using hundred of protocols unknown to most of the people.
* Added Inter-Arrival Time (IAT) calculation used to detect protocol misbehaviour (e.g. slow-DoS detection)
* Added data analysis features for computign metrics such as entropy, average, stddev, variance on a single and consistent place that will prevent when possible. This should ease traffic analysis on 
monitoring/security applications. New API calls have been implemented such as ndpi_data_XXX() to handle these calculations.
* Initial release of Python bindings available under nDPI/python.
* Implemented search of human readable strings for promoting data exfiltration detection
  * Available as library and in `ndpiReader` with option `-e`
* Fingerprints
  * JA3 (https://github.com/salesforce/ja3)
  * HASSH (https://github.com/salesforce/hassh)
  * DHCP
* Implemented a library to serialize/deserialize data in both Type-Length-Value (TLV) and JSON format
  * Used by nProbe/ntopng to exchange data via ZMQ

New Supported Protocols and Services

* DTLS (i.e. TLS over UDP)
* Hulu
* TikTok/Musical.ly
* WhatsApp Video
* DNSoverHTTPS
* Datasaver
* Line protocol
* Google Duo and Hangout merged
* WireGuard VPN
* IMO
* Zoom.us

Improvements

* TLS
  * Organizations
  * Ciphers
  * Certificate analysis
* Added PUBLISH/SUBSCRIBE methods to SIP
* Implemented STUN cache to enhance matching of STUN-based protocols
* Dissection improvements
  * Viber
  * WhatsApp
  * AmazonVideo
  * SnapChat
  * FTP
  * QUIC
  * OpenVPN support for UDP-based VPNs
  * Facebook Messenger mobile
  * Various improvements for STUN, Hangout and Duo
* Added new categories: CUSTOM_CATEGORY_ANTIMALWARE, NDPI_PROTOCOL_CATEGORY_MUSIC, NDPI_PROTOCOL_CATEGORY_VIDEO, NDPI_PROTOCOL_CATEGORY_SHOPPING, NDPI_PROTOCOL_CATEGORY_PRODUCTIVITY and 
NDPI_PROTOCOL_CATEGORY_FILE_SHARING
* Added NDPI_PROTOCOL_DANGEROUS classification

Fixes

* Fixed the dissection of certain invalid DNS responses
* Fixed Spotify dissection
* Fixed false positives with FTP and FTP_DATA
* Fix to discard STUN over TCP flows
* Fixed MySQL dissector
* Fix category detection due to missing initialization
* Fix DNS rsp_addr missing in some tiny responses
* Various hardening fixes

diffstat:

 net/ndpi/Makefile                          |   5 ++---
 net/ndpi/PLIST                             |   7 +++++--
 net/ndpi/distinfo                          |  14 ++++++--------
 net/ndpi/patches/patch-Makefile.am         |  15 ---------------
 net/ndpi/patches/patch-configure.seed      |  15 ---------------
 net/ndpi/patches/patch-src_lib_Makefile.in |  28 ++++++++++++++--------------
 6 files changed, 27 insertions(+), 57 deletions(-)

diffs (150 lines):

diff -r f3142269705b -r 7d14dbbc46d1 net/ndpi/Makefile
--- a/net/ndpi/Makefile Mon May 25 19:28:39 2020 +0000
+++ b/net/ndpi/Makefile Mon May 25 20:25:22 2020 +0000
@@ -1,8 +1,7 @@
-# $NetBSD: Makefile,v 1.7 2020/05/19 12:09:08 nia Exp $
+# $NetBSD: Makefile,v 1.8 2020/05/25 20:25:22 adam Exp $
 
-DISTNAME=      nDPI-2.8
+DISTNAME=      nDPI-3.2
 PKGNAME=       ${DISTNAME:tl}
-PKGREVISION=   1
 CATEGORIES=    net
 MASTER_SITES=  ${MASTER_SITE_GITHUB:=ntop/}
 GITHUB_PROJECT=        nDPI
diff -r f3142269705b -r 7d14dbbc46d1 net/ndpi/PLIST
--- a/net/ndpi/PLIST    Mon May 25 19:28:39 2020 +0000
+++ b/net/ndpi/PLIST    Mon May 25 20:25:22 2020 +0000
@@ -1,6 +1,7 @@
-@comment $NetBSD: PLIST,v 1.3 2019/07/12 09:30:34 adam Exp $
+@comment $NetBSD: PLIST,v 1.4 2020/05/25 20:25:22 adam Exp $
 bin/ndpiReader
 include/ndpi/ndpi_api.h
+include/ndpi/ndpi_classify.h
 include/ndpi/ndpi_config.h
 include/ndpi/ndpi_define.h
 include/ndpi/ndpi_includes.h
@@ -13,5 +14,7 @@
 lib/libndpi.a
 lib/libndpi.so
 lib/libndpi.so.${PKGVERSION}.0
-lib/libndpi.so.2
+lib/libndpi.so.3
 lib/pkgconfig/libndpi.pc
+share/ndpi/ndpiCustomCategory.txt
+share/ndpi/ndpiProtos.txt
diff -r f3142269705b -r 7d14dbbc46d1 net/ndpi/distinfo
--- a/net/ndpi/distinfo Mon May 25 19:28:39 2020 +0000
+++ b/net/ndpi/distinfo Mon May 25 20:25:22 2020 +0000
@@ -1,10 +1,8 @@
-$NetBSD: distinfo,v 1.5 2019/07/12 09:30:34 adam Exp $
+$NetBSD: distinfo,v 1.6 2020/05/25 20:25:22 adam Exp $
 
-SHA1 (nDPI-2.8.tar.gz) = 5b5f613cf55deeb040483f7c04ede7a2ed415c26
-RMD160 (nDPI-2.8.tar.gz) = c26bfadeee1db4abbf86a06fdfa338cbfaf42d9b
-SHA512 (nDPI-2.8.tar.gz) = aae757047810bbc32700ecf7ac2e5df4b93528021a50da4a0b83865a5248dfca8b1643c75aa7eeb6eac4f155638d45666db487318b2be9adb2f3412cb8ec8497
-Size (nDPI-2.8.tar.gz) = 19754226 bytes
-SHA1 (patch-Makefile.am) = 3f4649a55e519a21e1c5b645669f1febcef82d4e
-SHA1 (patch-configure.seed) = 154f2d6a690bd6e7e01862769bcf0fa4b1a86ad6
+SHA1 (nDPI-3.2.tar.gz) = 47ea3ac4a17afba1c7dd54bc8ec7a116031f16ef
+RMD160 (nDPI-3.2.tar.gz) = aaabdaf8b626925b85b880617b55143b9f6a3e8e
+SHA512 (nDPI-3.2.tar.gz) = bf3a6e66479110b8e1a51a3b66cc0a85aef98af455f2895a75fe2a8b06a9fb72a4dca3d4bbfa6f44959c5043c038927a1a8acb312e881d4e2bb544dc84b6eea8
+Size (nDPI-3.2.tar.gz) = 29586049 bytes
 SHA1 (patch-src_include_ndpi__includes.h) = db5da9ca24ecf32fd8c264e193af280814dc74c6
-SHA1 (patch-src_lib_Makefile.in) = 458e095e55a3b6f24166813a249350a32a1e0589
+SHA1 (patch-src_lib_Makefile.in) = 70431a2fb53675d221f1383afa86759a2dfd4bfc
diff -r f3142269705b -r 7d14dbbc46d1 net/ndpi/patches/patch-Makefile.am
--- a/net/ndpi/patches/patch-Makefile.am        Mon May 25 19:28:39 2020 +0000
+++ /dev/null   Thu Jan 01 00:00:00 1970 +0000
@@ -1,15 +0,0 @@
-$NetBSD: patch-Makefile.am,v 1.1 2019/07/12 09:30:34 adam Exp $
-
-Use standard pkgconfigdir.
-
---- Makefile.am.orig   2019-07-11 10:20:02.000000000 +0000
-+++ Makefile.am
-@@ -1,7 +1,7 @@
- ACLOCAL_AMFLAGS = -I m4
- SUBDIRS = src/lib example tests
- 
--pkgconfigdir = $(prefix)/libdata/pkgconfig
-+pkgconfigdir = $(prefix)/lib/pkgconfig
- pkgconfig_DATA = libndpi.pc
- 
- EXTRA_DIST = autogen.sh
diff -r f3142269705b -r 7d14dbbc46d1 net/ndpi/patches/patch-configure.seed
--- a/net/ndpi/patches/patch-configure.seed     Mon May 25 19:28:39 2020 +0000
+++ /dev/null   Thu Jan 01 00:00:00 1970 +0000
@@ -1,15 +0,0 @@
-$NetBSD: patch-configure.seed,v 1.2 2019/07/12 09:30:34 adam Exp $
-
-Portability fix.
-
---- configure.seed.orig        2019-03-01 17:04:18.000000000 +0000
-+++ configure.seed
-@@ -124,7 +124,7 @@ AC_ARG_ENABLE([json-c],
- AS_IF([test "x$enable_json_c" != "xno"], [
-        PKG_CONFIG_PATH=/usr/local/share/pkgconfig:$PKG_CONFIG_PATH
-        pkg-config --exists json-c
--       AS_IF([test "$?" == "0"],
-+       AS_IF([test "$?" = "0"],
-              [
-               CFLAGS="$CFLAGS $(pkg-config --cflags json-c)"
-               LDFLAGS="$LDFLAGS $(pkg-config --libs json-c)"
diff -r f3142269705b -r 7d14dbbc46d1 net/ndpi/patches/patch-src_lib_Makefile.in
--- a/net/ndpi/patches/patch-src_lib_Makefile.in        Mon May 25 19:28:39 2020 +0000
+++ b/net/ndpi/patches/patch-src_lib_Makefile.in        Mon May 25 20:25:22 2020 +0000
@@ -1,10 +1,10 @@
-$NetBSD: patch-src_lib_Makefile.in,v 1.2 2019/07/12 09:30:34 adam Exp $
+$NetBSD: patch-src_lib_Makefile.in,v 1.3 2020/05/25 20:25:22 adam Exp $
 
 Fix building on Darwin.
 
---- src/lib/Makefile.in.orig   2019-03-01 17:04:18.000000000 +0000
+--- src/lib/Makefile.in.orig   2020-02-20 09:15:44.000000000 +0000
 +++ src/lib/Makefile.in
-@@ -21,8 +21,15 @@ OBJECTS   = $(patsubst protocols/%.c, pr
+@@ -22,8 +22,15 @@ OBJECTS   = $(patsubst protocols/%.c, pr
  HEADERS   = $(wildcard ../include/*.h)
  NDPI_VERSION_MAJOR   = @NDPI_MAJOR@
  NDPI_LIB_STATIC      = libndpi.a
@@ -20,7 +20,7 @@
  NDPI_LIBS            = $(NDPI_LIB_STATIC) $(NDPI_LIB_SHARED)
  
  ifneq ($(OS),Windows_NT)
-@@ -30,8 +37,7 @@ OS := $(shell uname)
+@@ -31,8 +38,7 @@ OS := $(shell uname)
  endif
  
  ifeq ($(OS),Darwin)
@@ -30,20 +30,20 @@
  else
  SONAME_FLAG=-Wl,-soname,$(NDPI_LIB_SHARED_BASE).$(NDPI_VERSION_MAJOR)
  endif
-@@ -47,7 +53,7 @@ $(NDPI_LIB_STATIC): $(OBJECTS)
+@@ -48,7 +54,7 @@ $(NDPI_LIB_STATIC): $(OBJECTS)
  $(NDPI_LIB_SHARED): $(OBJECTS)
-       $(CC) -shared -fPIC $(SONAME_FLAG) -o $@ $(OBJECTS)
-       ln -Ffs $(NDPI_LIB_SHARED) $(NDPI_LIB_SHARED_BASE)
--      ln -Ffs $(NDPI_LIB_SHARED) $(NDPI_LIB_SHARED_BASE).$(NDPI_VERSION_MAJOR)
-+      ln -Ffs $(NDPI_LIB_SHARED) $(NDPI_LIB_SHARED_SHORT)
+       $(CC) -shared -fPIC $(SONAME_FLAG) -o $@ $(OBJECTS) $(LDFLAGS) @ADDITIONAL_LIBS@
+       ln -fs $(NDPI_LIB_SHARED) $(NDPI_LIB_SHARED_BASE)
+-      ln -fs $(NDPI_LIB_SHARED) $(NDPI_LIB_SHARED_BASE).$(NDPI_VERSION_MAJOR)
++      ln -fs $(NDPI_LIB_SHARED) $(NDPI_LIB_SHARED_SHORT)
  
  %.o: %.c $(HEADERS) Makefile
-       $(CC) $(CFLAGS) -c $< -o $@
-@@ -62,6 +68,6 @@ install: $(NDPI_LIBS)
+       $(CC) $(CFLAGS) -c $< -o $@ $(LDFLAGS)
+@@ -66,6 +72,6 @@ install: $(NDPI_LIBS)
        mkdir -p $(DESTDIR)$(libdir)
        cp $(NDPI_LIBS) $(DESTDIR)$(libdir)/
-       ln -Ffs $(DESTDIR)$(libdir)/$(NDPI_LIB_SHARED) $(DESTDIR)$(libdir)/$(NDPI_LIB_SHARED_BASE)
--      ln -Ffs $(DESTDIR)$(libdir)/$(NDPI_LIB_SHARED) $(DESTDIR)$(libdir)/$(NDPI_LIB_SHARED_BASE).$(NDPI_VERSION_MAJOR)
-+      ln -Ffs $(DESTDIR)$(libdir)/$(NDPI_LIB_SHARED) $(DESTDIR)$(libdir)/$(NDPI_LIB_SHARED_SHORT)
+       cp -P $(NDPI_LIB_SHARED_BASE) $(DESTDIR)$(libdir)/
+-      cp -P $(NDPI_LIB_SHARED_BASE).$(NDPI_VERSION_MAJOR) $(DESTDIR)$(libdir)/
++      cp -P $(NDPI_LIB_SHARED_SHORT) $(DESTDIR)$(libdir)/
        mkdir -p $(DESTDIR)$(includedir)
        cp ../include/*.h $(DESTDIR)$(includedir)
Prev by Date: [pkgsrc/trunk]: pkgsrc/security/courier-authlib courier-authlib: Add missing ...
Next by Date: [pkgsrc/trunk]: pkgsrc/doc Updated net/ndpi, net/ntopng
Previous by Thread: [pkgsrc/trunk]: pkgsrc/security/courier-authlib courier-authlib: Add missing ...
Next by Thread: [pkgsrc/trunk]: pkgsrc/doc Updated net/ndpi, net/ntopng
Indexes:
Home | Main Index | Thread Index | Old Index