pkgsrc-Changes-HG archive
[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index][Old Index]
[pkgsrc/trunk]: pkgsrc/textproc/xapian Update to 1.4.6. From the changelog:
details: https://anonhg.NetBSD.org/pkgsrc/rev/a0222d72312b
branches: trunk
changeset: 310151:a0222d72312b
user: schmonz <schmonz%pkgsrc.org@localhost>
date: Fri Jul 06 16:21:56 2018 +0000
description:
Update to 1.4.6. From the changelog:
API:
* API classes now support C++11 move semantics when using a compiler which
we are confident supports them (currently compilers which define
__cplusplus >= 201103 plus a special check for MSVC 2015 or later).
C++11 move semantics provide a clean and efficient way for threaded code to
hand-off Xapian objects to worker threads, but in this case it's very
unhelpful for availability of these semantics to vary by compiler as it
quietly leads to a build with non-threadsafe behaviour. To address this,
user code can #define XAPIAN_MOVE_SEMANTICS before #include <xapian.h> to
force this on, and will then get a compilation failure if the compiler lacks
suitable support.
* MSet::snippet():
+ We were only escaping output for HTML/XML in some cases, which would
potentially allow HTML to be injected into output (this has been assigned
CVE-2018-0499).
+ Include certain leading non-word characters in snippets. Previously we
started the snippet at the start of the first actual word, but there are
various cases where including non-word characters in front of the actual
word adds useful context or otherwise aids comprehension. Reported by
Robert Stepanek in https://github.com/xapian/xapian/pull/180
* Add MSetIterator::get_sort_key() method. The sort key has always been
available internally, but wasn't exposed via the public API before, which
seems like an oversight as the collapse key has long been available.
Reported by ??? on xapian-discuss.
* Database::compact():
+ Allow Compactor::resolve_duplicate_metadata() implementations to delete
entries. Previously if an implementation returned an empty string this
would result in a user meta-data entry with an empty value, which isn't
normally achievable (empty meta-data values aren't stored), and so will
cause odd behaviour. We now handle an empty returned value by interpreting
it in the natural way - it means that the merged result is to not set a
value for that key in the output database.
+ Since 1.3.5 compacting a WritableDatabase with uncommitted changes throws
Xapian::InvalidOperationError when compacting to a single-file glass
database. This release adds similar checks for chert and when compacting
to a multiple-file glass database.
+ In the unlikely event that the total number of documents or the total
length of all documents overflow when trying to compact a multi-database,
we throw an exception. This is now a DatabaseError exception instead of a
const char* exception (a hang-over from before this code was turned into a
public API in the library).
* Document::remove_term(): Handle removing term at current TermIterator
position - previously the underlying iterator was invalidated, leading to
undefined behaviour (typically a segmentation fault). Reported by Gaurav
Arora.
* TermIterator::get_termfreq() now always returns an exact answer. Previously
for multi-databases we approximated the result, which is probably either a
hang-over from when this method was used during Enquire::get_eset(), or else
due to a thinking that this method would be used in that situation (it
certainly is not now). If the user creates a TermIterator object and asks it
for term frequencies then we really should give them the correct answer - it
isn't hugely costly and the documentation doesn't warn that it might be
approximated.
* QueryParser::parse_query():
+ Now adds a colon after the prefix when prefixing a boolean term which
starts with a colon. This means the mapping is reversible, and matches
what omega actually does in this case when it tries to reverse the mapping.
Thanks to Andy Chilton for pointing out this corner case.
+ The parser now makes use of newer features in the lemon parser generator to
make parsing faster and use less memory.
* Enquire::get_mset(): Fix bug with get_mset(0, 0, X) when X > 0 which was
causing an attempt to access an element in an empty vector. Reported by
sielicki in #xapian.
* Stem:
+ Add Indonesian stemming algorithm.
+ Small optimisations to almost all stemming algorithms.
* Stopper:
+ Add Indonesian stopword list.
+ The installed version of the Finnish stopword list now has one word per
line. Previously it had several space-separated words on some lines, which
works with C++'s std::istream_iterator but may be inconvenient for use from
some other languages.
+ The installed versions of stopword lists are now sorted in byte order
rather than whatever collation order is specified by LC_COLLATE or similar
at build time. This makes the build more reproducible, and also may be
more efficient for loading into some data structures.
* WritableDatabase::replace_document(term, doc): Check for last_docid wrapping
when used on a sharded database.
* Database::locked(): Consistently throw FeatureUnavailableError on platforms
where we can't test for a database lock without trying to take it.
Previously GNU Hurd threw DatabaseLockError while platforms where we don't
use fcntl() locking at all threw UnimplementedError.
* Database and WritableDatabase constructors: Fix handling of entries for
disabled backends in stub database files to throw FeatureUnavailableError
instead of DatabaseError.
* Database::get_value_lower_bound() now works correctly for sharded databases.
Previously it returned the empty string if any shard had no values in the
specified slot.
* PostingIterator was failing to keep an internal reference to the parent
Database object for sharded databases.
* ValueIterator::skip_to() and check() had an off-by-one error in their docid
calculations in some cases with sharded databases.
* Add Database::get_total_length() method. Previously you had to calculate
this from get_avlength() and get_doccount(), taking into account rounding
issues. But even then you couldn't reliably get the exact value when total
length is large since a double's mantissa has more limited precision than an
unsigned long long.
* Add Xapian::iterator_rewound() for bidirectional iterators, to test if the
iterator is at the start (useful for testing whether we're done when
iterating backwards).
* DatabaseOpeningError exceptions now provide errno via get_error_string()
rather than turning it into a string and including it in the exception
message.
* WritableDatabase::replace_document(): when passed a Document object which
came from a database and has unmodified values, we used to always read
those values into a memory structure. Now we only do this if the document
is being replaced to the same document ID which it came from, which should
make other cases a bit more efficient.
* Enquire::get_eset(): When approximating term frequencies we now round to the
nearest integer - previously we always rounded down.
matcher:
* OP_VALUE_*: When a value slot's lower and upper bound are equal, we know
that exactly how many documents the subquery can match (either 0 or those
bounds). This also avoids a division by zero which previously happened
when trying to calculate the estimate.
* Speed up sorting by keys. Use string::compare() to avoid having to call
operator< if operator> returns false.
* Fix clamping of maxitems argument to get_mset() - it was being clamped
to db.get_doccount(), now it's clamped to db.get_doccount() - first. In
practice this doesn't actually seem to cause any issues.
* If a match time limit is in effect, when it expires we now clamp
check_at_least to first + maxitems instead of to maxitems. In practice this
also doesn't seem to actually cause any issues (at least we've failed to
construct a testcase where it actually makes an observable difference).
* Fix percentages when only some shards have positions. If the final shard
didn't have positions this would lead to under-counting the total number leaf
of subqueries which would lead to incorrect positional calculations (and a
division by zero if the top level of the query was positional. This bug was
introduced in 1.4.3.
* OP_NEAR: Fix "phantom positions", where OP_NEAR would think a term without
positional information occurred at position 1 if it had the lowest term
frequency amongst the OP_NEAR's subqueries.
* Fix termfreq used in weight calculations for a term occurring more than once
in the query. Previously the termfreq for such terms was multiplied by the
number of different query positions they appeared at.
* OP_SYNONYM: We use the doclength upper bound for the wdf upper bound of a
synonym - now we avoid fetching it twice when the doclength upper bound is
explicitly needed.
* Short-cut init() when factor is 0 in most Weight subclasses. This indicates
the object is for the term-independent weight contribution, which is always 0
for most schemes, so there's no point fetching any stats or doing any
calculations. This fixes a divide by zero for TfIdfWeight, detected by
UBSan.
* OP_OR: Fix bug which caused orcheck1 to fail once hooked up to run with the
inmemory backend.
* Iterating of positions has been sped up, which means phrase matching is now
faster (by a little over 5% in some simple tests).
* Fix use after free of QueryOptimiser hint in certain cases involving
multiple databases only some of which have positional information.
This bug was introduced by changes in xapian-core 1.4.3. Fixes #752,
reported and analysed by Robert Stepanek.
* An unweighted OP_AND_MAYBE is now optimised to just its left branch - the
other branch or branches only contribute weight, so can be completely ignored
when the operator is unweighted.
glass backend:
* Fix glass freelist bug when changes to a new database which didn't modify the
termlist table were committed. In this corner case, a block which had been
allocated to be the root block in the termlist table was leaked. This was
largely harmless, except that it was detected by Database::check() and caused
it to report an error. Reported by Antoine Beaupr? and David Bremner.
* Fix glass freelist bug with cancel_transaction(). The freelist wasn't
reset to how it was before the transaction, resulting in leaked blocks.
This was largely harmless, except that it was detected by Database::check()
and caused it to report an error.
* Improve the per-term wdf upper bound. Previously we used min(cf(term),
wdf_upper_bound(db)) which is tight for any terms which attain that
upper bound, and also for terms with termfreq == 1 (the latter are common
in the database (e.g. 66% for a database of wikipedia), but probably
much less common in searches). When termfreq > 1 we now use
max(first_wdf(term), cf(term) - first_wdf(term)), which means terms with
termfreq == 2 will also attain their bound (another 11% for the same
database) while terms with higher termfreq but below the global bound will
get a tighter bound.
* Fix Database::locked() on single-file glass db to just return false (such
databases can't be opened as a WritableDatabase so there can't be a write
lock). Previously this failed with: "DatabaseLockError: Unable to get write
lock on /flintlock: Testing lock"
* Fix compaction when both the input and output are specified as a file
descriptor. Previously this threw an exception due to an overeager check
that destination != source.
* Use O_TRUNC when compacting to single file. If the output already exists but
is larger than our output we don't want to just overwrite the start of it.
This case also used to result in confusing compaction percentages.
* Enable glass's "open_nearby_postlist" optimisation (which especially helps
large wildcard queries) for writable databases without any uncommitted
changes as well.
* Make get_unique_terms() more efficient for glass. We approximate
get_unique_terms() by the length of the termlist (which counts boolean terms
too) but clamp this to be no larger than the document length. Since we need
to open the termlist to get its length, it makes more sense to get the
document length from that termlist for no extra cost rather than looking it
up in the postlist table.
* Database::check() now checks document lengths against the stored document
length lower and upper bounds. Patch from Uppinder Chugh. Fixes
https://trac.xapian.org/ticket/617.
* Fix bogus handling of most-recently-read value slot statistics. It seems
that we get lucky and this can't actually cause a problem in practice due
to another layer of caching above, but if nothing else it's a bug waiting to
happen.
* If we fail to create the directory for a new database because the path
already exists, the exception now reports EEXIST as the errno value rather
than whatever errno value happened to be set from an earlier library call.
remote backend:
* xapian-tcpsrv --one-shot no longer forks. We need fork to handle multiple
concurrent connections, but when handling a single connection forking just
adds overhead and potentially complicates process management for our caller.
This aligns with the behaviour under __WIN32__ where we use threads instead
of forking, and service the connection from the main thread with --one-shot.
* Fix repeat call to ValueIterator::check() on the same docid to not always
set valid to true for remote backend.
inmemory backend:
* Fix repeat call to ValueIterator::check() on the same docid to not always
set valid to true for inmemory backend.
* Use binary chop instead of linear search in all places where we're searching
for a term or document - we weren't taking advantage of the sorted order
everywhere.
tools:
* xapian-delve:
+ Document values can contain binary data, so escape them by default for
output. Other options now supported are to decode as a packed integer
(like omindex uses for last modified), decode using
Xapian::sortable_unserialise(), and to show the raw form (which was the
previous behaviour).
+ Report current database revision.
* xapian-inspect:
+ Report entry count when opening table
+ Support inspecting single file DBs via a new --table option (which can also
be used with a non-single-file DB instead of specifying the path to the
table).
+ Add "first" and "last" commands which jump to the first/last entry in the
current table respectively.
+ "until" now counts and reports the number of entries advanced by.
+ Document "until" with no arguments - this advances to the end of the table,
but wasn't mentioned in the help.
+ Commands "goto" and "until" which take a key as an argument now expect the
key in the same escaped form that's used for display. This makes it much
simpler to interact with tables with binary keys.
+ Fix to expect .glass not .DB extension of glass tables.
diffstat:
textproc/xapian/Makefile.common | 4 ++--
textproc/xapian/PLIST | 3 ++-
textproc/xapian/distinfo | 11 +++++------
textproc/xapian/distinfo-bindings | 10 +++++-----
textproc/xapian/patches/patch-net_resolver.h | 14 --------------
5 files changed, 14 insertions(+), 28 deletions(-)
diffs (88 lines):
diff -r 5b9917ace349 -r a0222d72312b textproc/xapian/Makefile.common
--- a/textproc/xapian/Makefile.common Fri Jul 06 16:15:46 2018 +0000
+++ b/textproc/xapian/Makefile.common Fri Jul 06 16:21:56 2018 +0000
@@ -1,4 +1,4 @@
-# $NetBSD: Makefile.common,v 1.2 2017/07/10 17:43:43 schmonz Exp $
+# $NetBSD: Makefile.common,v 1.3 2018/07/06 16:21:56 schmonz Exp $
# used by textproc/csharp-xapian/Makefile
# used by textproc/lua-xapian/Makefile
# used by textproc/p5-Xapian/Makefile
@@ -9,7 +9,7 @@
# used by textproc/xapian/Makefile
# used by textproc/xapian-omega/Makefile
-VERSION= 1.4.4
+VERSION= 1.4.6
CATEGORIES= textproc
MASTER_SITES= http://oligarchy.co.uk/xapian/${VERSION}/
EXTRACT_SUFX= .tar.xz
diff -r 5b9917ace349 -r a0222d72312b textproc/xapian/PLIST
--- a/textproc/xapian/PLIST Fri Jul 06 16:15:46 2018 +0000
+++ b/textproc/xapian/PLIST Fri Jul 06 16:21:56 2018 +0000
@@ -1,4 +1,4 @@
-@comment $NetBSD: PLIST,v 1.12 2016/11/07 13:02:17 schmonz Exp $
+@comment $NetBSD: PLIST,v 1.13 2018/07/06 16:21:56 schmonz Exp $
bin/copydatabase
bin/quest
bin/simpleexpand
@@ -515,6 +515,7 @@
share/xapian-core/stopwords/french.list
share/xapian-core/stopwords/german.list
share/xapian-core/stopwords/hungarian.list
+share/xapian-core/stopwords/indonesian.list
share/xapian-core/stopwords/italian.list
share/xapian-core/stopwords/norwegian.list
share/xapian-core/stopwords/portuguese.list
diff -r 5b9917ace349 -r a0222d72312b textproc/xapian/distinfo
--- a/textproc/xapian/distinfo Fri Jul 06 16:15:46 2018 +0000
+++ b/textproc/xapian/distinfo Fri Jul 06 16:21:56 2018 +0000
@@ -1,8 +1,7 @@
-$NetBSD: distinfo,v 1.28 2017/07/14 12:55:45 joerg Exp $
+$NetBSD: distinfo,v 1.29 2018/07/06 16:21:56 schmonz Exp $
-SHA1 (xapian-core-1.4.4.tar.xz) = 6b8bf7eea3059dab8d5dd254c3ae0cf895bc4910
-RMD160 (xapian-core-1.4.4.tar.xz) = 19535bc7ca5c175b7ee1c4898e9e9e796e45dcb0
-SHA512 (xapian-core-1.4.4.tar.xz) = dc88bab1d82c68b29d51c2113319ddb5d16840f3544b9d5fcc7a3671f97d58f16ddff58b865ad3521ea778cbaacf73fe7346bb514a1275f1f739283a4128d001
-Size (xapian-core-1.4.4.tar.xz) = 2807952 bytes
+SHA1 (xapian-core-1.4.6.tar.xz) = e9817b6bf122453dd5c0c4166126f8bc489e7410
+RMD160 (xapian-core-1.4.6.tar.xz) = 3634fe1e7d6c41111c8cf21b10aab9584e68da0d
+SHA512 (xapian-core-1.4.6.tar.xz) = f08168eba26a24ea00c2219d2cdfac1d904b51162fdd98d437564786375245e00010986a7d6e9c8c7548ae19aab80434b5cb0051252f902b40db2976b1008fe2
+Size (xapian-core-1.4.6.tar.xz) = 2839964 bytes
SHA1 (patch-common_safesyssocket.h) = 032d441853914d510bc285bb682a98c4ee264d52
-SHA1 (patch-net_resolver.h) = 693f89bcb5d2cdc86bbf458c7f1969f2b27cf4c3
diff -r 5b9917ace349 -r a0222d72312b textproc/xapian/distinfo-bindings
--- a/textproc/xapian/distinfo-bindings Fri Jul 06 16:15:46 2018 +0000
+++ b/textproc/xapian/distinfo-bindings Fri Jul 06 16:21:56 2018 +0000
@@ -1,9 +1,9 @@
-$NetBSD: distinfo-bindings,v 1.5 2017/08/21 18:49:39 schmonz Exp $
+$NetBSD: distinfo-bindings,v 1.6 2018/07/06 16:21:56 schmonz Exp $
-SHA1 (xapian-bindings-1.4.4.tar.xz) = 1162e836f3caccee927997f0d262ef6dccd44d9f
-RMD160 (xapian-bindings-1.4.4.tar.xz) = 231d56d0989cf3a37a0b31d39680488fa1e80b39
-SHA512 (xapian-bindings-1.4.4.tar.xz) = 006e771b3ea654130b4f76f1a1caee576fd5a3fa6317ca44f1e5483671e8c83073ecbde4f3fc7f84c75590eec53acb59701af4cf9bcd72355ed9d53924492b53
-Size (xapian-bindings-1.4.4.tar.xz) = 1117140 bytes
+SHA1 (xapian-bindings-1.4.6.tar.xz) = 3818993795d0d50700bc817907e4060537473ce8
+RMD160 (xapian-bindings-1.4.6.tar.xz) = dda5d1e75cefb70399200f143ddf0c7a244a5cdc
+SHA512 (xapian-bindings-1.4.6.tar.xz) = e1d8d4c79ef8eb1b890be955b14eca7e9c7aafc6051cf9080d67948455f82ed68ab518343f532cde2a1c9e07d94b3aae8b8f5130db158c6badc9b6d118ceb733
+Size (xapian-bindings-1.4.6.tar.xz) = 1121532 bytes
SHA1 (patch-configure) = d1c3edf1efcd105aef23bf9245650971f8df6ced
SHA1 (patch-lua_Makefile.in) = 7f1c5077f0d46dfdf33c2b65f144bb08d5031330
SHA1 (patch-perl_Makefile.in) = 993b137b319d7d28c2b3a70d2e46e1a38d380578
diff -r 5b9917ace349 -r a0222d72312b textproc/xapian/patches/patch-net_resolver.h
--- a/textproc/xapian/patches/patch-net_resolver.h Fri Jul 06 16:15:46 2018 +0000
+++ /dev/null Thu Jan 01 00:00:00 1970 +0000
@@ -1,14 +0,0 @@
-$NetBSD: patch-net_resolver.h,v 1.1 2017/07/11 14:56:37 schmonz Exp $
-
-Include missing AF_INET definition to fix NetBSD build.
-
---- net/resolver.h.orig 2017-04-24 02:19:20.000000000 +0000
-+++ net/resolver.h
-@@ -23,6 +23,7 @@
-
- #include <cstring>
- #include "safenetdb.h"
-+#include "safesyssocket.h"
- #include "str.h"
- #include "xapian/error.h"
-
Home |
Main Index |
Thread Index |
Old Index