CVS commit: pkgsrc/textproc/xapian-omega

["Amitai Schleier" <schmonz%netbsd.org@localhost>]

Module Name:    pkgsrc
Committed By:   schmonz
Date:           Mon Nov  7 13:02:45 UTC 2016

Modified Files:
        pkgsrc/textproc/xapian-omega: Makefile PLIST distinfo
        pkgsrc/textproc/xapian-omega/patches: patch-aa patch-ab

Log Message:
Update to 1.4.1. From the changelog:

omindex:

  + Also index leafname with _ and & replaced by spaces.  Literal spaces are
    often avoided in filenames, and "hello_world.txt" ought to be searchable for
    via "hello" and "world".  Partly addresses #618, reported by Julien
    Pfefferkorn.

  + Make named entity look-up (e.g. é -> 233) use the same keyword-lookup
    table approach we already use for HTML tags and built-in MIME content-types,
    rather than a std::map, which makes it faster while using less memory.

  + Avoid using the shell to run most external commands as it's unnecessary
    overhead.  For the built-in filters, the only cases which now use a shell
    are where we run two unzip commands.  For user-specified commands, a simple
    and slightly conservative test is used, which should avoid a shell in most
    common cases where it isn't needed.  Notably, environment variables set
    before the command are handled.

  + Track files which couldn't be indexed in the user metadata and skip them by
    default on subsequent runs to avoid the costs of repeatedly running a
    filter on a file it can't handle.  Run omindex with --retry-failed to retry
    such files.

  + Overhaul the "per-site" terms:
    - 'H' prefix is hostname as before, except that if the term would be > 240
      bytes (unlikely but possible) the end is hashed is the same way 'U'
      prefix terms are.
    - 'P' terms are now added for every directory level, not just the start
      URL's path.
    - A new 'J' prefix term is added with the start URL (less any trailing
      '/'), which means all files indexed from a particular "site" are now
      indexed by one term.  See #376.

  + Add 'skip' pseudo-mimetype which extensions can be mapped to, and they will
    then be reported and skipped (to complement the existing 'ignore'
    pseudo-mimetype which causes files with the specified extension to be
    quietly ignored).

  + Treat a command of 'true' specially as meaning make the text extraction a
    no-op (as actually running /bin/true effectively would).  This provides a
    way to index some file types by only meta-data.  Fixes #519, reported by
    Brian Burton.

  + Add support for wildcard mimetypes */* and *.  Combined with filter command
    ``true`` for indexing by meta-data only, you can specify a fall back case
    of indexing by meta-data only using ``--filter '*:true'``.  From a
    suggestion by Brian Burton on xapian-discuss.

  + Index message/rfc822 and message/news.  These are individually saved email
    messages and news articles.

  + Index archived web page formats MAFF and MHTML.

  + Handle .xla, yet another XL extension.

  + Handle metadata in LibreOffice HTML export (dcterms.subject,
    dcterms.description, dcterms.creator and dcterms.contributor).

  + Use zlib's gzopen() instead of invoking "gzip -dc" for compressed Abiword
    documents.

  + Add support for %f in command passed to --filter to allow specifying
    commands where the input file is not the final argument.  Fixed #570,
    reported by Charles Atkinson.

  + Allow --filter to handle commands which produce output in a temporary file
    rather than on stdout.

  + Allow --filter to specify the character set of the output the filter
    produces.

  + Handle application/vnd.ms-excel, text/x-perl and application/x-dvi via
    default --filter settings instead of hardcoded cases (now possible thanks
    to the new abilities that --filter has).

  + Add support for specifying a MIME subtype of '*' in --filter arguments.

  + Add -track-ctime option to allow omindex to pick up changes to file
    ownership and permissions.

  + Index terms from the leafname with an 'F' prefix, rather than treating them
    as more body text.  (Fixes #633, reported by Emmanuel Garette)

  + The starting URL wasn't previously URL encoded.  In 1.2.18, a minimally
    intrusive fix was implemented.  In 1.3.2, we now encode the starting URL
    as we do for the rest of the filename.

  + Don't assume .doc is application/msword but let libmagic decide, since .doc
    files may actually be RTF, and sometimes people use .doc for plain-text
    documentation.

  + Add support for indexing 'topic' and 'created date' meta-data for
    OpenDocument format and HTML.

  + Index "topic" for PDF documents.

  + Commit changes and exit, rather than skipping the current file on most
    unexpected errors reading directories or initialising libmagic - otherwise
    we can end up deleting a lot of database entries on errors like EHOSTDOWN
    when indexing network mounts.

  + Add --opendir-sleep=SECS option to allow working around problems with
    indexing files on Microsoft DFS shares.

  + If we get ENOTDIR trying to index a file, skip it quietly (unless in
    verbose mode) as we already do if we get ENOENT, since ENOTDIR is what we
    get if the file and the directory it was in got removed between us getting
    the filename and trying to open it.

  + Handle ENOENT, ENOTDIR and EACCES from readdir().

  + If we've already opened the file (as we often will have if using a modern
    libmagic with magic_descriptor() available), then use fstat() on that fd
    rather than stat()/lstat() on the pathname.

  + Pass error message string and errno value in ReadError exceptions.

  + Report strerror(errno) if we can't read a file.

  + Filtering via text/html now handles HTML documents which specify a charset.

  + Add support for indexing Microsoft Publisher files using pub2xhtml.

  + Restrict the length of what we consider to be an extension, currently to 7
    characters or whatever the longest extension in the mime_map is if it is
    longer.

  + Avoid '//' in temporary filenames (cosmetic only).

  + Extend --filter to handle commands which produce HTML on stdout.

  + Don't report an error if a file is deleted (or renamed) between us reading
    the directory entry for it and trying to read the file itself by default.
    In --verbose mode, the situation is still reported, but now with a
    specific message.

  + If omindex receives any of the signals SIGHUP, SIGINT, SIGQUIT or SIGTERM,
    then kill any active external filter child process, then handle the signal
    as we did before.  If setpgid() is available, put each external filter in
    its own process group and kill the whole process group when we get a
    signal.

  + Use magic_descriptor() if the version of libmagic we're building against
    is new enough to have it.  This eliminates an extra opening of a file
    being indexed in certain cases.

  + Use rst2html to handle .rst and .rest files.

  + Index title with an 'S' prefix rather than no prefix.

  + If the document with the highest existing docid before the run was updated,
    we were reporting it as "added", but now we correctly report it as
    "updated".

  + Catch and report std::exception explicitly, so failing to allocate memory
    is no longer reported as "Unknown exception".

omindex-list: New tool to list URLs of all the documents in a database
(or list of databases) indexed by omindex.

* The HTML parser now explicitly handles <APPLET>, <OBJECT> and <TR>.

* Use a generated compact and efficient table to convert HTML tag names
  to enum codes - this is both faster and smaller than the approach we were
  using, with the benefit that the table is auto-generated.

* Always use our built-in conversion code for the character sets it can handle
  (previously we'd use iconv if available; now we only use iconv for other
  character sets).  This gives us more consistent results, and in particular
  means we now handle BOMs better (at least when using GNU iconv).

* A lot of data labelled as "iso-8859-1" is actually "windows-1252".  The two
  only differ in characters which are control characters in iso-8859-1, so
  assume the latter when we see the former.

scriptindex:

  + Remove special error handling case noting that index=nopos was replaced
    with indexnopos - this was removed in 1.1.0 so there's been enough time to
    upgrade.

omega:

* Add support for sorting by more than one value - e.g. SORT=+1,-2

* Add $msizelower and $msizeupper which provide access to the lower and upper
  bounds on the number of matches.

* Add support for $set{weighting,coord}.

* Add weightingpurefilter option.  Normally a query consisting only of filter
  terms won't have relevance weights calculated.  This new option allows you to
  specify a weighting scheme to use for such queries, with the same values
  supported as for the existing weighting option.  For example,
  $set{weightingpurefilter,coord} will weight such queries by how many filter
  terms match each document.

* $filters now includes DATEVALUE, which means we'll force the first page when
  reloading or changing page starting from existing URLs upon upgrade to 1.4.1,
  but the exact same existing URL could be for a search without the date filter
  where we want to force the first page, so there's an inherent ambiguity
  there.  Forcing first page in this case seems the least problematic
  side-effect.

* Implement $match command for omegascript.  Patch from Richhiey Thomas.

* Add optional prefix argument to $terms.

* $snippet now uses MSet::snippet() instead of the Snipper class.

* Add $contains{STRING1,STRING2}.  Contributed by Ayush Gupta.

* Add support for negated boolean filter terms, specified by CGI parameter "N".

* Support a direction prefix on SORT: '+' for ascending, '-' for descending.
  SORTREVERSE set to non-0 now flips the direction.  Fixes #697, reported by
  Andy Chilton.

* Add options argument to $transform.

* Cache compiled regexps used in $transform.

* Add $ord OmegaScript command which returns the Unicode codepoint for the
  first character of a UTF-8 string.

* Add $chr OmegaScript command which returns the UTF-8 string for given Unicode
  codepoint.

* Add $csv OmegaScript command which escapes a string for use as a field in a
  CSV file ("always quote" mode inspired by patch from Gaurav Arora.)

* New $filters encoding which avoids collisions.  We also compare CGI parameter
  xFILTERS to what $filters would have returned in previous releases, so that
  on upgrades old format serialised filters are handled correctly.

* Fix $jsonarray not to prepend ']' to the first array element.

* Skip weighting scheme setup for a pure date range query - it won't be
  weighted anyway, so we can avoid having to parse weighting scheme parameters,
  etc.

* Use value ranges when date range filtering by value.  Should be more
  efficient than a MatchDecider, and will automatically take advantage of any
  future value range optimisations in xapian-core.

* Add default_db and default_template config options.  These allow the default
  template and default database name to be set via the config file, rather than
  being stuck with the respective defaults of "default" and "query".  Fixes
  #310, reported by Marco Hennigs.

* Add support for non-exclusive filters.  Fixes #234, reported by Thomas
  Viehmann.

* Fix handling of multiple P.<prefix> fields - previously only the first seen
  was used.  These fields are also now taken into account when deciding if the
  query has changed.  $query now returns an OmegaScript list with one entry for
  each CGI parameter passed.

* Allow setting query expansion scheme to "bo1".

* Make the $json and $jsonarray force the text to be valid UTF-8, since
  otherwise the output isn't valid JSON.

* Check parameters to $set{weighting,bm25 ...} and $set{weighting,trad ...}
  converted OK.  Based on patch from Aarsh Shah.

* Add support to $set{weighting,...} for bb2, dlh, dph, ifb2, ineb2, inl2, lm,
  pl2 when we're built against a xapian-core which is new enough to have these
  schemes.

* Add $snippet to generate a snippet of text tailored to the search.

* Add new $json and $jsonarray OmegaScript commands to support producing JSON
  output.

* Add $truncate command which truncates a string after a word.

* Add support for $set{weighting,tfidf} to allow the new TfIdfWeight weighting
  scheme to be used.

+ DEFAULTOP now defaults to AND rather than OR, since that matches what pretty
  much every search engine does these days.  Closes ticket#512.

* Allow mapping a query string prefix to more than one term prefix (which
  xapian-core has supported since 1.0.4).

* Add support for search inputs for multiple probabilistic prefixes, with
  support for per-prefix stemmers.

* Drop legacy support for handling '.' separated terms in xP - that changed in
  Omega 0.9.7, more than 5 years ago now.

* Remove support for OLDP CGI parameter which was superseded by xP
  approximately a decade ago, and isn't even documented!

* Drop special handling for R-prefixed terms in $prettyterm - we stopped
  generating these in Xapian 1.0.

templates:

* Lower case all HTML tags, attributes and values; explicitly close <option>
  tags.  Patches from Vivek Pal and Nirmal Singhania.

* Migrate Omega Templates to HTML5.  Patch from Nirmal Sighania.

* templates/query: Remove stray double quote from generated URL for spelling
  suggestion when THRESHOLD is set.  Patch from Nirmal Singhania.

* templates/opensearch: Change response feeds to support OpenSearch 1.1.
  Patch from Nirmal Singhania.

* templates/query: Fix setting setting of prefix map for P - in 1.3.2, this
  would failed to also search in the subject.  Now it also searches in the
  subject and topic.

* templates/query:

  + We now map unprefixed queries to include S-prefixed terms to match the
    change in omindex to prefixing terms from the title with S.  You may want
    to make the same update to your own templates.

  + Set up prefixes for 'author:' and 'title:'.


To generate a diff of this commit:
cvs rdiff -u -r1.30 -r1.31 pkgsrc/textproc/xapian-omega/Makefile
cvs rdiff -u -r1.5 -r1.6 pkgsrc/textproc/xapian-omega/PLIST
cvs rdiff -u -r1.18 -r1.19 pkgsrc/textproc/xapian-omega/distinfo
cvs rdiff -u -r1.7 -r1.8 pkgsrc/textproc/xapian-omega/patches/patch-aa
cvs rdiff -u -r1.3 -r1.4 pkgsrc/textproc/xapian-omega/patches/patch-ab

Please note that diffs are not public domain; they are subject to the
copyright notices on the relevant files.

Modified files:

Index: pkgsrc/textproc/xapian-omega/Makefile
diff -u pkgsrc/textproc/xapian-omega/Makefile:1.30 pkgsrc/textproc/xapian-omega/Makefile:1.31
--- pkgsrc/textproc/xapian-omega/Makefile:1.30  Sat Jul  9 06:39:08 2016
+++ pkgsrc/textproc/xapian-omega/Makefile       Mon Nov  7 13:02:45 2016
@@ -1,9 +1,8 @@
-# $NetBSD: Makefile,v 1.30 2016/07/09 06:39:08 wiz Exp $
+# $NetBSD: Makefile,v 1.31 2016/11/07 13:02:45 schmonz Exp $
 #
 
 DISTNAME=              xapian-omega-${VERSION}
-VERSION=               1.2.23
-PKGREVISION=           1
+VERSION=               1.4.1
 CATEGORIES=            textproc
 MASTER_SITES=          http://oligarchy.co.uk/xapian/${VERSION}/
 EXTRACT_SUFX=          .tar.xz

Index: pkgsrc/textproc/xapian-omega/PLIST
diff -u pkgsrc/textproc/xapian-omega/PLIST:1.5 pkgsrc/textproc/xapian-omega/PLIST:1.6
--- pkgsrc/textproc/xapian-omega/PLIST:1.5      Sat May 23 18:21:16 2015
+++ pkgsrc/textproc/xapian-omega/PLIST  Mon Nov  7 13:02:45 2016
@@ -1,11 +1,15 @@
-@comment $NetBSD: PLIST,v 1.5 2015/05/23 18:21:16 schmonz Exp $
+@comment $NetBSD: PLIST,v 1.6 2016/11/07 13:02:45 schmonz Exp $
 bin/dbi2omega
 bin/htdig2omega
 bin/mbox2omega
 bin/omindex
+bin/omindex-list
 bin/scriptindex
+libexec/cgi-bin/mhtml2html
 libexec/cgi-bin/outlookmsg2html
+libexec/cgi-bin/rfc822tohtml
 libexec/cgi-bin/xapian-omega
+man/man1/omindex-list.1
 man/man1/omindex.1
 man/man1/scriptindex.1
 share/doc/xapian-omega/cgiparams.html

Index: pkgsrc/textproc/xapian-omega/distinfo
diff -u pkgsrc/textproc/xapian-omega/distinfo:1.18 pkgsrc/textproc/xapian-omega/distinfo:1.19
--- pkgsrc/textproc/xapian-omega/distinfo:1.18  Sat Apr 30 14:14:17 2016
+++ pkgsrc/textproc/xapian-omega/distinfo       Mon Nov  7 13:02:45 2016
@@ -1,8 +1,8 @@
-$NetBSD: distinfo,v 1.18 2016/04/30 14:14:17 schmonz Exp $
+$NetBSD: distinfo,v 1.19 2016/11/07 13:02:45 schmonz Exp $
 
-SHA1 (xapian-omega-1.2.23.tar.xz) = 8102ea0f74dc2671c459831301ca649875ae55d2
-RMD160 (xapian-omega-1.2.23.tar.xz) = e5531201838f1cc8cad5784851b7b473675ce2ab
-SHA512 (xapian-omega-1.2.23.tar.xz) = 1ea2b090b75f7ab530bbb6af79eb26931684796f9b0a987530f44e7f7f9527929644880f7528198e0ea22f69feabd014783c0b01edaa6673b54f83091e9a5428
-Size (xapian-omega-1.2.23.tar.xz) = 437912 bytes
-SHA1 (patch-aa) = 3a76634c8fa525d2cc049e10c11520442d986dfa
-SHA1 (patch-ab) = f84d1d457218186dff2a31e7d621085b58c891c8
+SHA1 (xapian-omega-1.4.1.tar.xz) = 9f182e58a9355d76bc567030a72f6e3c0cce9c68
+RMD160 (xapian-omega-1.4.1.tar.xz) = e17c8848624e299fa59debcb73d03e6bdbb26f0d
+SHA512 (xapian-omega-1.4.1.tar.xz) = 19891f5d0e901027e734041fc29b223ef002029b9ffdb91797b8b382132fcf7c335114ca7bec0ff21970a7435758e24675f1577dbb615f3cacad105d8ee91415
+Size (xapian-omega-1.4.1.tar.xz) = 494868 bytes
+SHA1 (patch-aa) = 044771ca449b9d2f29695d8034c3d90aa7aa0200
+SHA1 (patch-ab) = 6de115aab051256e911d61d69255f5af8916f058

Index: pkgsrc/textproc/xapian-omega/patches/patch-aa
diff -u pkgsrc/textproc/xapian-omega/patches/patch-aa:1.7 pkgsrc/textproc/xapian-omega/patches/patch-aa:1.8
--- pkgsrc/textproc/xapian-omega/patches/patch-aa:1.7   Sat May 23 18:21:16 2015
+++ pkgsrc/textproc/xapian-omega/patches/patch-aa       Mon Nov  7 13:02:45 2016
@@ -1,28 +1,28 @@
-$NetBSD: patch-aa,v 1.7 2015/05/23 18:21:16 schmonz Exp $
+$NetBSD: patch-aa,v 1.8 2016/11/07 13:02:45 schmonz Exp $
 
 Call it "xapian-omega" and follow hier(7).
 
---- Makefile.in.orig   2015-05-21 06:11:47.000000000 +0000
+--- Makefile.in.orig   2016-10-21 04:54:57.000000000 +0000
 +++ Makefile.in
-@@ -55,7 +55,7 @@ PRE_UNINSTALL = :
+@@ -92,7 +92,7 @@ PRE_UNINSTALL = :
  POST_UNINSTALL = :
  build_triplet = @build@
  host_triplet = @host@
 -pkglibbin_PROGRAMS = omega$(EXEEXT)
 +pkglibbin_PROGRAMS = xapian-omega$(EXEEXT)
- bin_PROGRAMS = omindex$(EXEEXT) scriptindex$(EXEEXT)
- check_PROGRAMS = atomparsetest$(EXEEXT) htmlparsetest$(EXEEXT) \
-       md5test$(EXEEXT) urlenctest$(EXEEXT) utf8converttest$(EXEEXT)
-@@ -471,7 +471,7 @@ AM_CPPFLAGS = \
+ bin_PROGRAMS = omindex$(EXEEXT) scriptindex$(EXEEXT) \
+       omindex-list$(EXEEXT)
+ check_PROGRAMS = atomparsetest$(EXEEXT) csvesctest$(EXEEXT) \
+@@ -771,7 +771,7 @@ AM_CPPFLAGS = \
        -DCONFIGFILE_SYSTEM=\"$(sysconfdir)/omega.conf\" \
        -DPKGLIBBINDIR=\"$(pkglibbindir)\"
  
 -pkglibbindir = $(pkglibdir)/bin
 +pkglibbindir = $(libexecdir)/cgi-bin
- dist_pkglibbin_SCRIPTS = outlookmsg2html
+ dist_pkglibbin_SCRIPTS = mhtml2html outlookmsg2html rfc822tohtml
  dist_bin_SCRIPTS = dbi2omega htdig2omega mbox2omega
- TESTS = atomparsetest$(EXEEXT)\
-@@ -480,7 +480,7 @@ TESTS = atomparsetest$(EXEEXT)\
+ dist_check_SCRIPTS = omegatest
+@@ -786,7 +786,7 @@ TESTS = atomparsetest$(EXEEXT)\
        urlenctest$(EXEEXT)\
        utf8converttest$(EXEEXT)
  
@@ -31,14 +31,14 @@ Call it "xapian-omega" and follow hier(7
  dist_omegadata_DATA = htdig2omega.script mbox2omega.script
  dist_sysconf_DATA = omega.conf
  
-@@ -736,8 +736,8 @@ common/$(DEPDIR)/$(am__dirstamp):
-       @: > common/$(DEPDIR)/$(am__dirstamp)
+@@ -1079,8 +1079,8 @@ md5test$(EXEEXT): $(md5test_OBJECTS) $(m
  common/str.$(OBJEXT): common/$(am__dirstamp) \
        common/$(DEPDIR)/$(am__dirstamp)
+ 
 -omega$(EXEEXT): $(omega_OBJECTS) $(omega_DEPENDENCIES) $(EXTRA_omega_DEPENDENCIES) 
 -      @rm -f omega$(EXEEXT)
 +xapian-omega$(EXEEXT): $(omega_OBJECTS) $(omega_DEPENDENCIES) $(EXTRA_omega_DEPENDENCIES) 
 +      @rm -f xapian-omega$(EXEEXT)
-       $(CXXLINK) $(omega_OBJECTS) $(omega_LDADD) $(LIBS)
+       $(AM_V_CXXLD)$(CXXLINK) $(omega_OBJECTS) $(omega_LDADD) $(LIBS)
  common/getopt.$(OBJEXT): common/$(am__dirstamp) \
        common/$(DEPDIR)/$(am__dirstamp)

Index: pkgsrc/textproc/xapian-omega/patches/patch-ab
diff -u pkgsrc/textproc/xapian-omega/patches/patch-ab:1.3 pkgsrc/textproc/xapian-omega/patches/patch-ab:1.4
--- pkgsrc/textproc/xapian-omega/patches/patch-ab:1.3   Tue Jun  4 21:28:26 2013
+++ pkgsrc/textproc/xapian-omega/patches/patch-ab       Mon Nov  7 13:02:45 2016
@@ -1,10 +1,10 @@
-$NetBSD: patch-ab,v 1.3 2013/06/04 21:28:26 schmonz Exp $
+$NetBSD: patch-ab,v 1.4 2016/11/07 13:02:45 schmonz Exp $
 
 Respect ${VARBASE} and follow hier(7).
 
---- omega.conf.orig    2013-04-17 00:01:38.000000000 +0000
+--- omega.conf.orig    2016-10-21 04:54:32.000000000 +0000
 +++ omega.conf
-@@ -1,11 +1,11 @@
+@@ -1,8 +1,8 @@
  # Directory containing Xapian databases:
 -database_dir /var/lib/omega/data
 +database_dir @VARBASE@/xapian-omega/data
@@ -13,6 +13,11 @@ Respect ${VARBASE} and follow hier(7).
 -template_dir /var/lib/omega/templates
 +template_dir @VARBASE@/xapian-omega/templates
  
+ # Default template name if the CGI parameter "FMT" is not specified.
+ # (If not specified here, the default template name is "query"):
+@@ -13,7 +13,7 @@ template_dir /var/lib/omega/templates
+ #default_db default
+ 
  # Directory to write Omega logs to:
 -log_dir /var/log/omega
 +log_dir @VARBASE@/log/xapian-omega