GSoC Project Progress Update: Apropos Replacement

[Abhinav Upadhyay <er.abhinav.upadhyay%gmail.com@localhost>]

Hello NetBSD

Posting update about my 3rd week's work.

I mostly worked on fixing some major issues in makemandb to get the
indexing going. Besides that I also implemented a barebones apropos(1)
which takes the user query as input and simply looks up the database.

I have started a new branch on my Github repository "search"
https://github.com/abhinav-upadhyay/apropos_replacement/branches/search
, where I am adding some experimental code for search related
features. As I get some feedback and reviews, I will commit them in
master or revert them depending on the reviews.

Currently the search branch has following two features:

1. A Stopword filter: If we are doing full text search, then we also
expect users to enter normal queries consisting of usual English
words, so we need to filter out the stopwords out of the user query in
order to get only those results which match the actual keywords in the
user query and not the stopwords.

2. A ranking function: A ranking function is very necessary, so that
Sqlite ranks and gives back the most useful results at the top. If you
try the apropos in the master branch and the one in search branch, you
will notice drastic difference in the quality of search results. But
even after this lots of effort is required to improve it.

If you would like to see the output of some sample searches:
http://pastebin.com/qhQBRNd5

I made a more detailed report on my blog where I have disucssed the
issues fixed, and also how to get the code and try it out.
http://abhinav-upadhyay.blogspot.com/2011/06/netbsd-gsoc-weekly-report-3.html

At this point of time, some community feedback will be highly useful
and valuable. For example:
- What are the most important things you would look for when
performing a search across man pages ?
- What all information should be there in the output ?
- Do you like the current results ?
- Would you like any changes in the interface of apropos(1) ?
etc.

Thanks for your interest and time :-)

Regards
Abhinav