Re: full-text search

[David Holland <dholland-tech%netbsd.org@localhost>]

On Tue, Feb 15, 2011 at 11:24:58AM -0600, David Young wrote:
 > > This I agree with. We should have a real text indexing system in base,
 > > so we can do real documentation searches. There was no excuse for
 > > apropos(1) in 1998; there's even less today.
 > > 
 > > There's no reason an application for indexing your own files needs to
 > > be in base. If there isn't a good one in pkgsrc, then perhaps one
 > > should be added; if there isn't one that can be added, perhaps one
 > > should be written.
 > 
 > Dividing responsibility for searching different types or sources of
 > documents between different programs, as you suggest, will lead to
 > usability problems and duplicated design & programming effort.
 > 
 > You should not have to divert your attention from the task at
 > hand---e.g., find documentation for the is_computer_on() function---to
 > the type, ownership, or source of relevant documents.

Throwing unrelated documents together for the sake of having only one
search engine or search database, as you suggest, will lead to
usability problems and duplicated indexes and runtime work.

You should not have to divert your attention from the task at hand
(e.g., looking for docs for the curses library) to wade through a pile
of completely irrelevant documents (e.g. scholarly articles on 16th
century blasphemous expressions, or popular medieval superstitions
about witches and witchcraft, or whatever) -- by your argument we also
ought to eliminate man page sections, and I hope everyone will agree
that such a move would be foolish. Furthermore, because of permissions
issues every user will end up building a private copy of the index
material for system-level documents, which wastes nontrivial amounts
of space and time.

Also, the two tasks have substantially different requirements. A large
part of the challenge for indexing your homedir is keeping the index
up to date; this is pretty much a nonissue for man pages and system
documentation, where all needed updates can be triggered explicitly.

-- 
David A. Holland
dholland%netbsd.org@localhost