BXR.SU, Super User's BSD Cross Reference w/ OpenGrok, publicly private beta

["Constantine A. Murenin" <cnst%NetBSD.org@localhost>]

Dear NetBSD,

It is my great pleasure to announce the immediate availability of a 
publicly private IPv6-only beta test of BXR.SU -- Super User's BSD Cross 
Reference.

BXR.SU is based on an OpenGrok fork, but it's more than just OpenGrok. 
We've fixed a number of annoyances, eliminated features that just never 
worked right from the outright, and provided integration with tools like 
CVSweb (including awesome mirrors like allbsd.org), FreeBSD's ViewVC 
(SVN), as well as Gitweb from git.freebsd.your.org, plus a tad of other 
improvements, including a complete rewrite of an mdoc parser.  Last, but 
definitely not least, is an extensive set of nginx rewrite rules that 
makes it a breeze to use BXR.SU as a deterministic URL compactor for 
referencing BSD source code.


  What's up with the publicly private beta test?

We're launching today in a publicly private beta.  Participation in the 
beta is invitation-only; everyone with IPv6 is invited.

We're cooperating with ISPs around the world, and in order to be able to 
access BXR.SU during this beta phase, you must have a special token, 
also known as a publicly routable IPv6-address, with proper 
IPv6-connectivity and upstream peering.  If you don't have IPv6 yet, but 
want to participate in this beta test ASAP, then ask your ISP for IPv6 
ASAP!  Else, if your ISP is not part of our beta rollout, you could try 
something like tunnelbroker.net from he.net.


  What's the release schedule?

BXR.SU is available through IPv6 today, 2013-04-01.  It is currently an 
IPv6-only site, with an IPv6-only glue, too.

As an IPv6-only site, we hereby declare that 2013-04-04 is an IPv4 day.

On April 4, we will temporarily enable IPv4 connectivity, for one day, 
to test the water.  (We've heard that IPv4 has some connectivity issues 
related to NAT, double-NAT, carrier-grade NAT and NAT64, and some small 
percentage of users (but significant in absolute terms) might not be 
able to access the site if an A record is published, due to the 
plentiful of misconfigurations out there; so, we want to take things 
slow, and ensure our users don't suffer from any inferior connectivity.)

If things do go well (we expect IPv6/IPv4-related improvements as time 
goes by), we will permanently publish an A record for BXR.SU on 2013-04-14.

IPv4 glue records will be published shortly thereafter, on 2013-04-24 
(we don't do this today, because we're afraid that the nameservers of 
some ISPs are not configured correctly, and our IPv6 users won't be able 
to access our site otherwise, so, we think it's a good idea to take 
things slow and in steps).


  But why another OpenGrok?

Over the years, there have been a number of OpenGrok installations that 
have made it possible to study and grok BSD code, for which we are very 
thankful to their maintainers.  However, as a general rule, none of them 
have been inclusive of all BSD flavours, all of them have had rather 
long and hard-to-remember URLs, which also didn't really look permanent 
at all, and, unfortunately, many of them no longer exist today, or some 
new uber-inclusive services like code.metager.de have recently 
flourished, with an astounding 8 second (yes, eight second) delay for 
satisfying any single search query (hot queries are returned in as 
little as just under 4 seconds by metager, yet everything is nonetheless 
buffered, so, you get no rendering at all for those whole 4 or 8 
seconds).  So, we thought this had to change.


  So, what's the deal?

It's simple.

Say, someone doesn't know who PHK is.  You can point them to:

  http://bxr.su/s?q=phk

Want to see if DragonFly keeps queue.h in sync?  Take a look at:

  http://bxr.su/d/sys/sys/queue.h

Want to look at FreeBSD's queue.h, to manually compare?  Just change the 
"d" from "/d/" (or select and replace the whole world "DragonFly" from 
"/DragonFly/") to "f", and you're in FreeBSD:

  http://bxr.su/f/sys/sys/queue.h

Too many /sys/sys/?  We've still got you covered, thanks to nginx:

  http://bxr.su/n/queue.h

Anyone uses TAILQ_SWAP?  Is that a new thing?  Check it out:

  http://bxr.su/search?q=TAILQ_SWAP

Any mentions of "OpenBSD" or "NetBSD" in FreeBSD and DragonFly?

  http://bxr.su/f,d/s?q=OpenBSD+OR+NetBSD

Who's this guy writing this email anyway?  Is he BXR'able?

  http://bxr.su/s?q=%22Constantine%20A.%20Murenin%22%20OR%20cnst

Etc.


  Just how fast is BXR.SU?

We expect that most search requests should be fulfilled (search page 
results generated) in well under 100ms.  In my tests, and according to 
OpenGrok metrics at the bottom of each search page, most search pages 
are generated in about 30 to 50ms, so, it does seem like there's some 
room to spare.  In addition, of course we use nginx, so, once generated 
at 40ms, a page should be available immediately in no time should a 
subsequent identical request come along within a couple of seconds or so.


  How does it compare with fxr.watson.org?

+ we're based on OpenGrok, instead of LXR
+ we also index userland of all BSDs, not just the kernels like the fxr
+ we do daily updates and re-index of all 4 BSDs
- we only track -CURRENT of each BSD
+ the -current that we track is actually current


  Is this a replacement to nxr.netbsd.org?

Not necessarily.  We've noted that the /gnu/ and /contrib/ directories 
of various BSDs have been constantly giving out lots of false positives 
on all kinds of search queries, and they have been excluded, for both 
the accuracy and disc space utilisation.  This only affects userland, 
and mostly excludes stuff that noone really cares about anyways (some 
exceptions were made, and this is not really in stone anyways). 
Otherwise, BXR.SU is much faster than the nxr -- nxr search is 
unbuffered, but takes several seconds on cold runs, and about half a 
second on hot runs.


  How does it compare with code.metager.de?

+ we're over 20000% faster (0.04s vs. 8s).  'nuff said.


I welcome comments, questions and suggestions.  I hope that this site 
will be useful to the BSD community, and will join other great and 
invaluable BSD-related sites that we all use and depend on.


  In summary, what's unique about bxr.su?

* Supports all 4 BSDs
* Daily updates
* Short URLs, deterministic URL shortener for BSD source code
* Kernel + non-gnu userland
* Very fast -- most search pages generated in under 50ms
* IPv6-only for now, will be dual-stacked very soon


Enjoy!

Constantine.