[pkgsrc/trunk]: pkgsrc/mail/rspamd mail/rspamd: Update to 1.7.9.

[fhajny <fhajny%pkgsrc.org@localhost>]

details:   https://anonhg.NetBSD.org/pkgsrc/rev/1c96d65e57d3
branches:  trunk
changeset: 312591:1c96d65e57d3
user:      fhajny <fhajny%pkgsrc.org@localhost>
date:      Fri Sep 07 11:19:07 2018 +0000

description:
mail/rspamd: Update to 1.7.9.

- Fix missing config files (pkg/53577).

The most important features and fixes

- Ratelimits are reworked and now work as intended (and documented)
- Clickhouse module supports data retention policies
- Reworked C modules to avoid global contexts (simplifies leaks
  detection on reload)
- Reputation plugin now supports SPF records reputation
- WebUI code is now even more conformant to the modern JS standards
- Maps are now distributed remotely with local file safety fallback to
  allow faster maps update without waiting for a new release
- Antivirus module checks attachments only (as decoded content) in
  attachments_only mode to improve AV performance by hiding the mime
  content from them

Full list of the meaningful changes

- [CritFix] Fix caseless comparison of equal length strings
- [Feature] Add HTTP basic auth support to elastic and clickhouse
  plugins
- [Feature] Add SPF selector to reputation
- [Feature] Add support of the fallback backends for HTTP maps
- [Feature] Allow to print full mime structure when extracting mime
  data
- [Feature] Allow to split symbols in reputation plugin
- [Feature] Check attachments only on AV scanners in attachments_only
  mode
- [Feature] Disable all SSL checks if ssl_no_verify flag is set
- [Feature] Implement parsing of scoped IPv6 addresses
- [Feature] Improve rspamc counters output
- [Fix] Add sanity checks when expanding SPF macros
- [Fix] Allow to parse SA rules with no spaces around =~ (dirty hack)
- [Fix] Avoid one extra byte writing
- [Fix] Deal with direct hash table
- [Fix] Detect empty text part as text, not HTML
- [Fix] Do not reduce map watch timeout for mixed http/file maps
- [Fix] Fix HTML part detection heuristic
- [Fix] Fix double free in redirectors cleanup
- [Fix] Fix legacy history handling in the controller
- [Fix] Fix messages insertion
- [Fix] Fix sending string method
- [Fix] Fix statconver command line arguments
- [Fix] Fixed argument checking for being null
- [Fix] Fixed issues reported by luacheck
- [Fix] Freeze updates queue when do actual storage update
- [Fix] HTTP map hash is per-backend and not per-map
- [Fix] Plug memory leak in fuzzy updates
- [Fix] Prefer 'MTA-Name' when producing authentication results
- [Fix] Replace bad unicode sequences instead of stopping on them
- [Fix] Set classifier version on learning
- [Project] Reworked ratelimits
- [Project] Apply topological sorting for symbols in Rspamd
- [Project] Remove global contexts from C modules
- [Project] Move performance critical hash tables to khash
- [WebUI] Avoid unused indexes
- [WebUI] Do not execute on_success callback
- [WebUI] Fix history reset for "All SERVERS" (#2346)
- [WebUI] Fix query URL for selected server
- [WebUI] Fix symbols display in legacy history,
- [WebUI] Hide symbols order selector for legacy history
- [WebUI] Refactor query functions into one
- [WebUI] Remove previously-attached event handlers
- [WebUI] Save symbols to the selected server
- [WebUI] Unify arguments of query functions
- [WebUI] Use common query functions to get graph data
- [WebUI] Use common query functions to save symbols

diffstat:

 mail/rspamd/Makefile                          |  58 ++++++++------------------
 mail/rspamd/PLIST                             |   3 +-
 mail/rspamd/cfgfiles.mk                       |   4 +
 mail/rspamd/distinfo                          |  12 +++--
 mail/rspamd/patches/patch-src_lua_lua__task.c |  15 ++++++
 mail/rspamd/patches/patch-test_CMakeLists.txt |  18 ++++++++
 6 files changed, 65 insertions(+), 45 deletions(-)

diffs (178 lines):

diff -r 2d533a9b75ae -r 1c96d65e57d3 mail/rspamd/Makefile
--- a/mail/rspamd/Makefile      Fri Sep 07 10:03:45 2018 +0000
+++ b/mail/rspamd/Makefile      Fri Sep 07 11:19:07 2018 +0000
@@ -1,7 +1,6 @@
-# $NetBSD: Makefile,v 1.45 2018/08/22 09:45:34 wiz Exp $
+# $NetBSD: Makefile,v 1.46 2018/09/07 11:19:07 fhajny Exp $
 
-DISTNAME=      rspamd-1.7.8
-PKGREVISION=   1
+DISTNAME=      rspamd-1.7.9
 CATEGORIES=    mail
 MASTER_SITES=  ${MASTER_SITE_GITHUB:=vstakhov/}
 
@@ -20,6 +19,7 @@
 USE_LANGUAGES= c99
 BUILD_DEFS+=   VARBASE
 
+# Not really true, but difficult to avoid picking up 5.3
 LUA_VERSIONS_ACCEPTED= 53
 
 CPPFLAGS.SunOS+=       -D__EXTENSIONS__
@@ -60,55 +60,35 @@
 # Torch requires LuaJIT2, but LuaJIT2 only supports Lua 5.1
 CMAKE_ARGS+=   -DENABLE_LUAJIT=OFF
 CMAKE_ARGS+=   -DENABLE_TORCH=OFF
+CMAKE_ARGS+=   -DLUA_VERSION_MAJOR=5
+CMAKE_ARGS+=   -DLUA_VERSION_MINOR=2
 
 REPLACE_PERL+= utils/rspamd_stats.pl
 
 MAKE_DIRS+=    ${PKG_SYSCONFDIR}/modules.d
+MAKE_DIRS+=    ${PKG_SYSCONFDIR}/scores.d
 
 OWN_DIRS_PERMS+=       ${DBDIR} ${RSPAMD_USER} ${RSPAMD_GROUP} 0700
 OWN_DIRS_PERMS+=       ${LOGDIR} ${RSPAMD_USER} ${RSPAMD_GROUP} 0750
 OWN_DIRS_PERMS+=       ${RUNDIR} ${RSPAMD_USER} ${RSPAMD_GROUP} 0755
 
-CFGFILES=      2tld.inc actions.conf common.conf composites.conf \
-               dmarc_whitelist.inc groups.conf logging.inc \
-               maillist.inc metrics.conf mid.inc mime_types.inc \
-               modules.conf modules.d/antivirus.conf modules.d/arc.conf \
-               modules.d/asn.conf modules.d/clickhouse.conf \
-               modules.d/dcc.conf modules.d/dkim_signing.conf \
-               modules.d/dkim.conf modules.d/dmarc.conf \
-               modules.d/elastic.conf modules.d/emails.conf \
-               modules.d/force_actions.conf \
-               modules.d/forged_recipients.conf \
-               modules.d/fuzzy_check.conf modules.d/greylist.conf \
-               modules.d/hfilter.conf modules.d/history_redis.conf \
-               modules.d/chartable.conf modules.d/ip_score.conf \
-               modules.d/maillist.conf modules.d/metadata_exporter.conf \
-               modules.d/metric_exporter.conf modules.d/mid.conf \
-               modules.d/milter_headers.conf modules.d/mime_types.conf \
-               modules.d/multimap.conf modules.d/mx_check.conf \
-               modules.d/neural.conf modules.d/once_received.conf \
-               modules.d/phishing.conf modules.d/ratelimit.conf \
-               modules.d/rbl.conf modules.d/redis.conf \
-               modules.d/regexp.conf modules.d/replies.conf \
-               modules.d/rspamd_update.conf modules.d/spamassassin.conf \
-               modules.d/spamtrap.conf modules.d/spf.conf \
-               modules.d/surbl.conf modules.d/trie.conf \
-               modules.d/url_redirector.conf modules.d/url_reputation.conf \
-               modules.d/url_tags.conf modules.d/whitelist.conf \
-               options.inc redirectors.inc rspamd.conf \
-               scores.d/fuzzy_group.conf scores.d/headers_group.conf \
-               scores.d/hfilter_group.conf scores.d/mime_types_group.conf \
-               scores.d/mua_group.conf scores.d/phishing_group.conf \
-               scores.d/policies_group.conf scores.d/rbl_group.conf \
-               scores.d/statistics_group.conf scores.d/subject_group.conf \
-               scores.d/surbl_group.conf spf_dkim_whitelist.inc \
-               statistic.conf surbl-whitelist.inc worker-controller.inc \
-               worker-fuzzy.inc worker-normal.inc worker-proxy.inc
+.include "cfgfiles.mk"
 
 .for file in ${CFGFILES}
-CONF_FILES=    ${EGDIR}/${file} ${PKG_SYSCONFDIR}/${file}
+CONF_FILES+=   ${EGDIR}/${file} ${PKG_SYSCONFDIR}/${file}
 .endfor
 
+###
+### Generate list of config files
+###
+.PHONY: cfgfiles
+cfgfiles: do-extract
+       @${ECHO} "# \$$NetBSD\$$" > ${.CURDIR}/cfgfiles.mk
+       @${ECHO} "# Do not edit, regenerate using 'make cfgfiles'" >> ${.CURDIR}/cfgfiles.mk
+       @${ECHO} "CFGFILES= \\" >> ${.CURDIR}/cfgfiles.mk
+       cd ${WRKSRC}/conf && ${FIND} * -type f | ${SORT} | ${TR} '\n' ' ' >> ${.CURDIR}/cfgfiles.mk
+       @${ECHO} >> ${.CURDIR}/cfgfiles.mk
+
 post-extract:
        ${FIND} ${WRKSRC}/conf -type d -exec ${CHMOD} ${PKGDIRMODE} {} \;
        ${FIND} ${WRKSRC}/conf -type f -exec ${CHMOD} ${DOCMODE} {} \;
diff -r 2d533a9b75ae -r 1c96d65e57d3 mail/rspamd/PLIST
--- a/mail/rspamd/PLIST Fri Sep 07 10:03:45 2018 +0000
+++ b/mail/rspamd/PLIST Fri Sep 07 11:19:07 2018 +0000
@@ -1,4 +1,4 @@
-@comment $NetBSD: PLIST,v 1.13 2018/07/31 12:27:02 fhajny Exp $
+@comment $NetBSD: PLIST,v 1.14 2018/09/07 11:19:07 fhajny Exp $
 bin/rspamadm
 bin/rspamadm-${PKGVERSION}
 bin/rspamc
@@ -57,6 +57,7 @@
 share/examples/rspamd/modules.d/redis.conf
 share/examples/rspamd/modules.d/regexp.conf
 share/examples/rspamd/modules.d/replies.conf
+share/examples/rspamd/modules.d/reputation.conf
 share/examples/rspamd/modules.d/rspamd_update.conf
 share/examples/rspamd/modules.d/spamassassin.conf
 share/examples/rspamd/modules.d/spamtrap.conf
diff -r 2d533a9b75ae -r 1c96d65e57d3 mail/rspamd/cfgfiles.mk
--- /dev/null   Thu Jan 01 00:00:00 1970 +0000
+++ b/mail/rspamd/cfgfiles.mk   Fri Sep 07 11:19:07 2018 +0000
@@ -0,0 +1,4 @@
+# $NetBSD: cfgfiles.mk,v 1.1 2018/09/07 11:19:07 fhajny Exp $
+# Do not edit, regenerate post-extract using 'make cfgfiles'
+CFGFILES= \
+2tld.inc actions.conf cgp.inc common.conf composites.conf dmarc_whitelist.inc groups.conf logging.inc maillist.inc metrics.conf mid.inc mime_types.inc modules.conf modules.d/antivirus.conf 
modules.d/arc.conf modules.d/asn.conf modules.d/chartable.conf modules.d/clickhouse.conf modules.d/dcc.conf modules.d/dkim.conf modules.d/dkim_signing.conf modules.d/dmarc.conf modules.d/elastic.conf 
modules.d/emails.conf modules.d/force_actions.conf modules.d/forged_recipients.conf modules.d/fuzzy_check.conf modules.d/greylist.conf modules.d/hfilter.conf modules.d/history_redis.conf 
modules.d/ip_score.conf modules.d/maillist.conf modules.d/metadata_exporter.conf modules.d/metric_exporter.conf modules.d/mid.conf modules.d/milter_headers.conf modules.d/mime_types.conf 
modules.d/multimap.conf modules.d/mx_check.conf modules.d/neural.conf modules.d/once_received.conf modules.d/phishing.conf modules.d/ratelimit.conf modules.d/rbl.conf modules.d/redis.conf 
modules.d/regexp.conf modules.d/replies.conf modules.d/reputation.conf modules.d/rspamd_update.conf modules.d/spamassassin.conf modules.d/spamtrap.conf modules.d/spf.conf modules.d/surbl.conf 
modules.d/trie.conf modules.d/url_redirector.conf modules.d/url_reputation.conf modules.d/url_tags.conf modules.d/whitelist.conf options.inc redirectors.inc rspamd.conf scores.d/fuzzy_group.conf 
scores.d/headers_group.conf scores.d/hfilter_group.conf scores.d/mime_types_group.conf scores.d/mua_group.conf scores.d/phishing_group.conf scores.d/policies_group.conf scores.d/rbl_group.conf 
scores.d/statistics_group.conf scores.d/subject_group.conf scores.d/surbl_group.conf spf_dkim_whitelist.inc statistic.conf surbl-whitelist.inc worker-controller.inc worker-fuzzy.inc worker-normal.inc 
worker-proxy.inc 
diff -r 2d533a9b75ae -r 1c96d65e57d3 mail/rspamd/distinfo
--- a/mail/rspamd/distinfo      Fri Sep 07 10:03:45 2018 +0000
+++ b/mail/rspamd/distinfo      Fri Sep 07 11:19:07 2018 +0000
@@ -1,8 +1,10 @@
-$NetBSD: distinfo,v 1.25 2018/07/31 12:27:02 fhajny Exp $
+$NetBSD: distinfo,v 1.26 2018/09/07 11:19:07 fhajny Exp $
 
-SHA1 (rspamd-1.7.8.tar.gz) = 8213d37209d0050d7f94eb26ad22ca85455913df
-RMD160 (rspamd-1.7.8.tar.gz) = 13de5b2fe6b8349b7731991787d5be1b111ad1d9
-SHA512 (rspamd-1.7.8.tar.gz) = 43f2350dfa4e0369cd53480bd4c8d0aa3898d7c471540281e7460a264207032aea226c4fd519a26397b9e9e8c8de3951e5a9752e010eabf1b3733a0c3e3086c2
-Size (rspamd-1.7.8.tar.gz) = 4542955 bytes
+SHA1 (rspamd-1.7.9.tar.gz) = 38b3134ac171643b59dd54c68e94e6ce492b2589
+RMD160 (rspamd-1.7.9.tar.gz) = dc519118a4d90552ea29051ba347b9ee350acd91
+SHA512 (rspamd-1.7.9.tar.gz) = d11d8c03668b85feae665ed491cd156c7430fbad30f42fe44103ea3c3a8bc988e5f8981e784e8f31aa35ba1ff9f7abd23605638e4ee3086a056b22fb3109563e
+Size (rspamd-1.7.9.tar.gz) = 4557492 bytes
 SHA1 (patch-CMakeLists.txt) = 1d3e66b52ef38638674baf4c10b68c97b2ca38ad
 SHA1 (patch-src_CMakeLists.txt) = c80015bb275ba8b9ad31491311275bf353f8f30a
+SHA1 (patch-src_lua_lua__task.c) = 19e0c8433ad08fda075f50d93bff7cedf3d4cda1
+SHA1 (patch-test_CMakeLists.txt) = 2e4aafd860f0f120928cce49b0995c739558bf0d
diff -r 2d533a9b75ae -r 1c96d65e57d3 mail/rspamd/patches/patch-src_lua_lua__task.c
--- /dev/null   Thu Jan 01 00:00:00 1970 +0000
+++ b/mail/rspamd/patches/patch-src_lua_lua__task.c     Fri Sep 07 11:19:07 2018 +0000
@@ -0,0 +1,15 @@
+$NetBSD: patch-src_lua_lua__task.c,v 1.3 2018/09/07 11:19:07 fhajny Exp $
+
+Fix deprecated Lua call.
+
+--- src/lua/lua_task.c.orig    2018-08-01 14:21:24.000000000 +0000
++++ src/lua/lua_task.c
+@@ -1307,7 +1307,7 @@ lua_task_load_from_string (lua_State * L
+ {
+       struct rspamd_task *task = NULL, **ptask;
+       const gchar *str_message = luaL_checkstring (L, 1), *err = NULL;
+-      gsize message_len = lua_strlen (L, 1);
++      gsize message_len = lua_rawlen (L, 1);
+       struct rspamd_config *cfg = NULL;
+       gboolean res = FALSE;
+ 
diff -r 2d533a9b75ae -r 1c96d65e57d3 mail/rspamd/patches/patch-test_CMakeLists.txt
--- /dev/null   Thu Jan 01 00:00:00 1970 +0000
+++ b/mail/rspamd/patches/patch-test_CMakeLists.txt     Fri Sep 07 11:19:07 2018 +0000
@@ -0,0 +1,18 @@
+$NetBSD: patch-test_CMakeLists.txt,v 1.1 2018/09/07 11:19:07 fhajny Exp $
+
+Avoid linker args that do not work on SunOS.
+
+--- test/CMakeLists.txt.orig   2018-08-01 14:21:24.000000000 +0000
++++ test/CMakeLists.txt
+@@ -19,7 +19,11 @@ SET_TARGET_PROPERTIES(rspamd-test PROPER
+ SET_TARGET_PROPERTIES(rspamd-test PROPERTIES COMPILE_FLAGS "-DRSPAMD_TEST")
+ ADD_DEPENDENCIES(rspamd-test rspamd-server)
+ IF(NOT CMAKE_SYSTEM_NAME STREQUAL "Darwin")
++IF(CMAKE_SYSTEM_NAME STREQUAL "SunOS")
++      TARGET_LINK_LIBRARIES(rspamd-test "${CMAKE_BINARY_DIR}/src/librspamd-server.a")
++ELSE(CMAKE_SYSTEM_NAME STREQUAL "SunOS")
+       TARGET_LINK_LIBRARIES(rspamd-test "-Wl,-whole-archive ${CMAKE_BINARY_DIR}/src/librspamd-server.a -Wl,-no-whole-archive")
++ENDIF(CMAKE_SYSTEM_NAME STREQUAL "SunOS")
+ ELSE(NOT CMAKE_SYSTEM_NAME STREQUAL "Darwin")
+       TARGET_LINK_LIBRARIES(rspamd-test "-Wl,-force_load ${CMAKE_BINARY_DIR}/src/librspamd-server.a")
+ ENDIF(NOT CMAKE_SYSTEM_NAME STREQUAL "Darwin")