Summary of man-page formatting

[Kamil Rytarowski <kamil%netbsd.org@localhost>]

Hello,

I apologize for nerves and words that could be avoided.

I'm going to summarize the situation with formatters in the NetBSD base.

1. NetBSD base ships with two programs that can format manual pages from
base and most 3rd party software: BSD mandoc (newest) and GPLv2 groff
1.19.2 (old, from 2005).

2. mandoc as of today renders correctly 99.8% of manual pages from
OpenBSD ports (8776 total [1]). This means everything except [2]:

 * books/man-pages-posix
 * comms/x3270
 * devel/srecord
 * editors/xemacs21/stable
 * graphics/xfig
 * mail/fetchmail
 * mail/nmh
 * math/graphviz
 * math/mcl
 * math/lapack
 * misc/screen
 * misc/zzuf
 * news/nn
 * textproc/docbook2x
 * textproc/zoem
 * x11/motif
 * x11/qt3
 * x11/xwit

3. There is no groff on the list above of not renderable pages, as the
newest version of groff gplv3 improved the manual page in terms of
end-user content and formatting, making it fully compatible with
mandoc(1) and more valuable to users.

This means that broken formatting is at this point the fault of using
old groff in base.

4. The (possibly incomplete) list of unsupported by mandoc(1) features
in the gplv2 groff page include:

 * .do
 * .mso
 * .substring
 * \[\\$1]
 * .di
 * .chop
 * .wh
 * .URL
 * .MTO

According to mandoc(1) upstream (Ingo Schwarze) .mso, .di, .wh, .URL,
and .MTO will likely be never supported, while partial support for
others is feasible.

5. mandoc(1) is not a *roff generic purpose formatter, but only and
intentionally a replacement for "*roff -mandoc". This means that it
targets mostly manual-pages and not generic documents and avoids
features that are potentially insecure.

This is documented also in the manual page:

http://man.bsd.lv/NetBSD-8.1/man7/mandoc_roff.7#COMPATIBILITY

"tiny subset of roff requests and escapes"

http://man.bsd.lv/NetBSD-8.1/man7/mandoc_roff.7#COMPATIBILITY

"The mandoc(1) implementation of the roff language is intentionally
incomplete."

"For security reasons, mandoc(1) never reads or writes external files
except via so requests with safe relative paths.

"Diversions are not implemented, and support for traps is very incomplete."

6. Does it mean that mandoc(1) is not sufficient as a generic purpose
and universal formatter of the manual pages? What other BSDs do?

FreeBSD tries to detect in runtime whether a manual page is unsupported
and fallbacks to groff.

OpenBSD started to convert all OpenBSD ports manual pages, one by one
and preformat with groff the teeny number of leftover ones.

NetBSD ignored the issue entirely until today... until it almost
completely fixed self.

Reference https://www.openbsd.org/papers/bsdcan15-mandoc.pdf slides 27-29.

7. catman(8) was defunct by an accident between etc/man.conf r. 1.30 and
1.35, nearly two years 2012-2013, but as far as I can tell not reached a
release, as it was between 5.x and 6.x. As far as I can tell, it was
detected by an accident and nobody reported it as a bug.

8. In 2012 the cat3f/ dir was removed and marked as obsolete and
presumed to never contain any files any more, man3f/ stays untouched,
waiting for Fortran to be back.

Once Fortran will be back and populate man3f/, catman(8) will recreate
the cat3f/ on each run and ./build.sh install=/ drop cat3f/ on upgrade.

9. The upstream mandoc(1) team (as of today) is open to accept MANWIDTH
as it was repeatedly requested by users, however only for the release
after the coming one.

The closest release is planned until the end of this year and no new
features are planned.

Another release may happen between 1 and 2 years.

10. I've suggested a MANWIDTH patch, which looked fine for upstream, but
it is not guaranteed to land upstream at all or in that form.

http://netbsd.org/~kamil/patch-00287-mandoc-MANWIDTH.txt

11. For some reason we have got a rule for .me files in man.conf(5).

.me is for groff -me (groff_me(7)), but I am not aware about any manual
page in that format. This format seems to be very rare and unless I am
wrong, nothing installs it into man/ dirs.

12. I recall recurring discussions about phasing out gplv2 groff from
the base and introduction of something else.

I can see the following options:

 - do nothing, keep shipping gnu gplv2 groff
 - upgrade to gplv3 groff
 - defer general purpose formatting to software from pkgsrc
 - switch to https://github.com/aligrudi/neatroff
 - switch to https://n-t-roff.github.io/heirloom/doctools.html
 - switch to something else (anything else?)

13. Then, there is still gnu gplv2 info(1) in src/, that rots quicker
than gnu groff gplv2.

14. We still deliver vgrind(1) and some similar macros that use groff
internally.

[1] https://openports.se/statistics.php
[2] https://github.com/openbsd/ports/search?q=USE_GROFF