CVS commit: pkgsrc/devel/tlsh

["Amitai Schleier" <schmonz%netbsd.org@localhost>]

Module Name:    pkgsrc
Committed By:   schmonz
Date:           Sun Jan 14 20:07:45 UTC 2024

Modified Files:
        pkgsrc/devel/tlsh: Makefile PLIST buildlink3.mk distinfo
Added Files:
        pkgsrc/devel/tlsh/patches: patch-CMakeLists.txt
Removed Files:
        pkgsrc/devel/tlsh: Makefile.common
        pkgsrc/devel/tlsh/patches: patch-utils_rand__tags.cpp

Log Message:
tlsh: update to 4.8.2. Changes:

3.5.0:
- Added the - force option
  - Allows a user to force the generation of digests for strings down to
    50 characters long

3.5.1:
- Fixed the error in the Python extension

3.5.2:
- Added the BlackHat Asia tool (presented at Arsenal)

3.7.0:
- merged in various fixes - ifdef for SPARC and RH73
- corrected TLSH_CTC_final.pdf
- added a SHA1 to the NOTICE.txt file
- improved the make.sh so that it calls the test.sh (and does
  regression tests)
- improved regression tests to confirm that the hash is calculated
  correctly in your environment
- fixed the header file C++ standard violation (reserved identifier
  violation #21)

3.7.1:
- resolved issue #29 - the force option for Python
  Step 1 - adding a regression test for strings approx of length 50
  Step 2 - add python code

3.7.2:
- added code to set the distance parameters for ROC analysis

3.7.3:
- resolving issue #44
- making static library the default

3.7.4:
- resolving issue #45
- add a timing test for TLSH

3.7.5:
- resolving issue #46
- in include/tlsh_impl.h
        #define SLIDING_WND_SIZE  5
  this can be varied between 4 to 8

3.8.0:
- Adding    // access functions - required by tools using TLSH library
  - int Lvalue();
  - int Q1ratio();
  - int Q2ratio();

3.9.0:
- resolving issue #48 - tlsh_pattern program

3.9.1:
- resolving issue #38
- putting in fix in rand_tags.cpp so that it generates identical output
  to previous version while safely working with pointers

3.9.2:
- Also merged the contents of NOTICE.txt into LICENSE. This was done
  because NOTICE.txt is sometimes accidently removed when people clone
  this repository. And the LICENSE specifically states that NOTICE.txt
  should NOT be removed.
- Also added command line option -notice which displays the
  NOTICE.txt file

3.9.3:
- currently tlsh_pattern returns all the matches
  modify tlsh_pattern to return the best match
- remove the newline from the input fields when reading in the
  tlsh_pattern file

3.9.4:
- check in order_bug program which demonstrates issue #50
- resolved issue #50 - added code to tlsh_impl.cpp to check for invalid
  call sequences to update() and final()

3.9.5:
- issue #61: added a command line option -notest - do not do any testing

3.9.6:
- Have a cmake option to build tlsh with a zero byte checksum
  (development / research option)
- Default build has 1 byte checksum - which is strongly recommended

3.9.7:
- resolving issue #50 for bin/timing_unittest

3.9.8:
- timing_unittest measures the time taken to do distance calculations
- add a command line option -size - so that you can measure the time
  taken to evaluate different sizes of string

3.9.9:
- resolve issue #62
- remove dependancy on GNUInstallDirs

3.10.0:
- Adding // access function - required by tools using TLSH library
  - int BucketValue(int bucket);
  - int Checksum(int k);

3.11.0:
- Make calculation of TLSH digests approx 7 times faster (for large
  files), done by
  - inline functions
  - unrolling loops
  - fixing the -O2 optimization option

3.11.1:
- tidy up:
  1. use fast_b_mapping() instead of b_mapping()
  2. remove declaration of unsigned r which is never used
  3. remove #include which is not required

3.12.0:
- remove floating point calculations such as log() function
  use alookup table instead

3.13.0:
- .vcproj files and instructions for builing TLSH on Windows using
  Visual Studio

3.13.1:
- fixing setup.py so that you can install Python Extension on Windows

3.14.0:
- adding sliding window size to tlsh_version
- changing test.sh to read the sliding window size

3.14.1:
- fixing error in test script for -xlen option (print statements about
  considering length were incorrect)
- improved test.sh - tests for existance of expected output files

3.15.0:
- Refactor code - so that input of directory or digest is in a struct.
  The code to process input is in library code (input_desc.cpp,
  shared_file_functions.cpp). The input routines can be used by
  myultiple programs. Also, preparing for things like csv input files.

3.15.1:
- added command line option -help to show full help information

3.15.2:
- tlsh_pattern uses refactored code introduced in 3.15.0

3.16.0:
- improved tlsh_pattern functionality
- added regression tests for tlsh_pattern

3.16.1:
- improved tlsh functionality
- add options

3.16.2:
- added regression tests for 3.16.1

3.17.0:
- Make command line option  -force        (50 char limit) the default behaviour
- Add a command line option -conservative (256 char limit)

3.17.3:
- add checking to confirm that TLSH digests are the correct length in
  - -c option
  - -d option
  - the appropriate column of -l listfile options

3.18.0:
- resolve issue #72 - remove tlsh_version

3.19.0:
- preperation for Windows build
  remove ../Testing/ from test.sh script and from regression test
  results

3.19.1:
- in test.sh and testlen.sh - make TLSH_PROG a variable

4.0.0:
- version 4: adding version identifier to each digest: 'T1'
  - adding command line option -old to generate old style digests
  - In this version - the showvers is defaulted to off - so this will
    pass the old regression tests

4.0.1:
- turing on T1 functionality by setting showvers=1 in main
- updating regression tests to have T1 at the start of digests

4.1.0:
- adding -o option for output filename (output will go to stdout if no
  output file given)
  - changed test scripts to use -o option
- adding -ojson option for json output
  - added regression test for -ojson option
- adding -onull option to output empty files / files too small as TNULL

4.2.0:
- Windows version using minGW

4.2.1:
- resolve issue #78 json objects do not validate on windows

4.2.2:
- resolve issue #81
- Pass regression tests

4.2.3:
- add regression tests that are compatible with
  https://github.com/glaslos/tlsh

4.3.0:
- issue #79 - divide by 0 if q3 == 0
  solution. if (q3 == 0) return invalid hash

4.4.0:
- Fixing Python Extension
  - updated python extension to T1 hashes (4.0.0)
  - fixed python_test.sh (which attempted to access old expected
    results files)
  - added license information to py_ext/tlshmodule.cpp

4.4.1:
- Command line options to tlsh_digest.py
  -conservative enforce 256 byte limit
  -old          generate old style hash (without "T1")
- added python functions to tlsh package (for backwards compatibility)
  tlsh.oldhash(data)
  tlsh.conservativehash(data)
  tlsh.oldconservativehash(data)

4.5.0:
- Checking in files to create pypi package

4.6.0:
- Add architecture ppc64le to travis build (Thanks ddeka2910)

4.7.0:
- Release updated package py-tlsh on Pypi.org
- Merging in pull request that adds functions to Python package
  lvalue, q1ratio, q2ratio, checksum, bucket_value and is_valid
- resolve issue #102 - correct Python version numbers

4.7.2:
- regression tests for C++ and Python functions for:
  lvalue, q1ratio, q2ratio, checksum, bucket_value
- resolve issue #95 - allow Requires-Python: >=2.7

4.8.0:
- Fix the make install target by adding the version.h in the
  installed files

4.8.1:
- Improve portability, add shared library build, install tlsh_unittest

4.8.2:
- fixed tlsh_win_version.h


To generate a diff of this commit:
cvs rdiff -u -r1.2 -r1.3 pkgsrc/devel/tlsh/Makefile pkgsrc/devel/tlsh/PLIST
cvs rdiff -u -r1.4 -r0 pkgsrc/devel/tlsh/Makefile.common
cvs rdiff -u -r1.1 -r1.2 pkgsrc/devel/tlsh/buildlink3.mk
cvs rdiff -u -r1.7 -r1.8 pkgsrc/devel/tlsh/distinfo
cvs rdiff -u -r0 -r1.1 pkgsrc/devel/tlsh/patches/patch-CMakeLists.txt
cvs rdiff -u -r1.2 -r0 pkgsrc/devel/tlsh/patches/patch-utils_rand__tags.cpp

Please note that diffs are not public domain; they are subject to the
copyright notices on the relevant files.

Modified files:

Index: pkgsrc/devel/tlsh/Makefile
diff -u pkgsrc/devel/tlsh/Makefile:1.2 pkgsrc/devel/tlsh/Makefile:1.3
--- pkgsrc/devel/tlsh/Makefile:1.2      Fri Sep  1 07:52:02 2017
+++ pkgsrc/devel/tlsh/Makefile  Sun Jan 14 20:07:44 2024
@@ -1,22 +1,28 @@
-# $NetBSD: Makefile,v 1.2 2017/09/01 07:52:02 wiz Exp $
+# $NetBSD: Makefile,v 1.3 2024/01/14 20:07:44 schmonz Exp $
 
-PKGREVISION=   1
-USE_CMAKE=     yes
-CONFIGURE_DIRS=        build/release
-CMAKE_ARG_PATH=        ../..
-USE_LANGUAGES= c c++
-# all tests fail
-#TEST_TARGET=  test
+DISTNAME=              tlsh-4.8.2
+CATEGORIES=            devel
+MASTER_SITES=          ${MASTER_SITE_GITHUB:=trendmicro/}
+GITHUB_TAG=            ${PKGVERSION_NOREV}
+
+MAINTAINER=            pkgsrc-users%NetBSD.org@localhost
+HOMEPAGE=              https://github.com/trendmicro/tlsh/
+COMMENT=               Fuzzy matching library
+LICENSE=               apache-2.0
+
+USE_CMAKE=             yes
+CONFIGURE_DIRS=                build/release
+CMAKE_ARG_PATH=                ../..
+CMAKE_ARGS+=           -DTLSH_SHARED_LIBRARY=1
+USE_LANGUAGES=         c c++
+TEST_TARGET=           test
 
 INSTALLATION_DIRS=     bin include lib
 
 post-extract:
-       mkdir ${WRKSRC}/build/release
+       ${MKDIR} ${WRKSRC}/${CONFIGURE_DIRS}
 
-do-install:
-       ${INSTALL_DATA} ${WRKSRC}/bin/tlsh_unittest ${DESTDIR}${PREFIX}/bin
+post-install:
        ${INSTALL_DATA} ${WRKSRC}/include/*.h ${DESTDIR}${PREFIX}/include
-       ${INSTALL_DATA} ${WRKSRC}/lib/libtlsh* ${DESTDIR}${PREFIX}/lib
 
-.include "Makefile.common"
 .include "../../mk/bsd.pkg.mk"
Index: pkgsrc/devel/tlsh/PLIST
diff -u pkgsrc/devel/tlsh/PLIST:1.2 pkgsrc/devel/tlsh/PLIST:1.3
--- pkgsrc/devel/tlsh/PLIST:1.2 Thu Sep  1 17:04:08 2016
+++ pkgsrc/devel/tlsh/PLIST     Sun Jan 14 20:07:45 2024
@@ -1,10 +1,16 @@
-@comment $NetBSD: PLIST,v 1.2 2016/09/01 17:04:08 wiz Exp $
+@comment $NetBSD: PLIST,v 1.3 2024/01/14 20:07:45 schmonz Exp $
+bin/tlsh
 bin/tlsh_unittest
+include/input_desc.h
+include/shared_file_functions.h
 include/tlsh.h
+include/tlsh/tlsh.h
+include/tlsh/tlsh_version.h
 include/tlsh_impl.h
 include/tlsh_util.h
-include/version.h
+include/tlsh_version.h
+include/tlsh_win_version.h
 lib/libtlsh.a
 lib/libtlsh.so
-lib/libtlsh.so.0
+lib/libtlsh.so.4
 lib/libtlsh.so.${PKGVERSION}

Index: pkgsrc/devel/tlsh/buildlink3.mk
diff -u pkgsrc/devel/tlsh/buildlink3.mk:1.1 pkgsrc/devel/tlsh/buildlink3.mk:1.2
--- pkgsrc/devel/tlsh/buildlink3.mk:1.1 Wed Dec  2 17:22:18 2015
+++ pkgsrc/devel/tlsh/buildlink3.mk     Sun Jan 14 20:07:45 2024
@@ -1,12 +1,11 @@
-# $NetBSD: buildlink3.mk,v 1.1 2015/12/02 17:22:18 wiz Exp $
+# $NetBSD: buildlink3.mk,v 1.2 2024/01/14 20:07:45 schmonz Exp $
 
 BUILDLINK_TREE+=       tlsh
 
 .if !defined(TLSH_BUILDLINK3_MK)
 TLSH_BUILDLINK3_MK:=
 
-BUILDLINK_DEPMETHOD.tlsh?=     build
-BUILDLINK_API_DEPENDS.tlsh+=   tlsh>=3.4.3
+BUILDLINK_API_DEPENDS.tlsh+=   tlsh>=4.8.2
 BUILDLINK_PKGSRCDIR.tlsh?=     ../../devel/tlsh
 .endif # TLSH_BUILDLINK3_MK
 

Index: pkgsrc/devel/tlsh/distinfo
diff -u pkgsrc/devel/tlsh/distinfo:1.7 pkgsrc/devel/tlsh/distinfo:1.8
--- pkgsrc/devel/tlsh/distinfo:1.7      Tue Oct 26 10:20:04 2021
+++ pkgsrc/devel/tlsh/distinfo  Sun Jan 14 20:07:45 2024
@@ -1,6 +1,6 @@
-$NetBSD: distinfo,v 1.7 2021/10/26 10:20:04 nia Exp $
+$NetBSD: distinfo,v 1.8 2024/01/14 20:07:45 schmonz Exp $
 
-BLAKE2s (tlsh-3.4.5.tar.gz) = 8a70191cd5be8f618d881a1927007492bd4b1384320ad39492b71ae3faeff146
-SHA512 (tlsh-3.4.5.tar.gz) = 039f9fb9fb467a9b22f85a621c46069bac1745da039657840e1ff1a53901392fcf29d3c953b0f47d831d022a5e2594925c726d4ff921614418a3bb023a4ebbdc
-Size (tlsh-3.4.5.tar.gz) = 1914815 bytes
-SHA1 (patch-utils_rand__tags.cpp) = 024d61efd590d5e16b20100eac729276fc996ee6
+BLAKE2s (tlsh-4.8.2.tar.gz) = 846377f453b4365ebcaabe09f22f5579087d50999e7edfc9dc8117277b7cb79e
+SHA512 (tlsh-4.8.2.tar.gz) = 5355c7734f5356a3a7a31ccc078254835871fc0369f1889a34131a0f810735201c68f674eb18e975b6a03a32e5c284899ca9ffdc1f583e42192188c172d2aad0
+Size (tlsh-4.8.2.tar.gz) = 3345458 bytes
+SHA1 (patch-CMakeLists.txt) = ce5dae73e7955cbf315fa97080390d6aba64f017

Added files:

Index: pkgsrc/devel/tlsh/patches/patch-CMakeLists.txt
diff -u /dev/null pkgsrc/devel/tlsh/patches/patch-CMakeLists.txt:1.1
--- /dev/null   Sun Jan 14 20:07:45 2024
+++ pkgsrc/devel/tlsh/patches/patch-CMakeLists.txt      Sun Jan 14 20:07:45 2024
@@ -0,0 +1,17 @@
+$NetBSD: patch-CMakeLists.txt,v 1.1 2024/01/14 20:07:45 schmonz Exp $
+
+Squelch configure warnings.
+
+--- CMakeLists.txt.orig        2021-09-09 05:56:28.000000000 +0000
++++ CMakeLists.txt
+@@ -52,6 +52,10 @@
+ # OF THE POSSIBILITY OF SUCH DAMAGE.
+ 
+ cmake_minimum_required(VERSION 2.6)
++# let CMAKE_MODULE_PATH override modules installed with cmake
++cmake_policy(SET CMP0017 OLD) 
++# set MACOSX_RPATH
++cmake_policy(SET CMP0042 NEW)
+ 
+ project(TLSH)
+