GSoC 2016 project proposal: Split debug symbols for pkgsrc builds

[Leonardo Taccari <leot%NetBSD.org@localhost>]

Hello to the entire tech-pkg@ list,
I have found an interesting project and if it can be considered a GSoC
project (actually it isn't) I really would like to work on it as a
student.

I am studying for a Master Degree in Computing and Automation
Engineering at Università Politecnica delle Marche in Ancona, Italy.
I am also a recent NetBSD developer and have experience in particular
as a package maintainer. I've also sporadically contributed in htdocs
and src areas too.


Rationale
---------
The ability to debug software is important not just when developing
but also when using it, e.g. post-mortem analysis.
NetBSD provides MKDEBUG and MKDEBUGLIB variables that can be set in
mk.conf in order to split debugging symbols for user-land applications
and libraries. Resulting split debugging symbols are then available
via the debug.tgz and xdebug.tgz installation sets.
NetBSD also provides Rump and to some extent also ddb(4),
ktrace(1) and DTrace that ease analysis, tracing and debugging.
All these features make NetBSD a great operating system in this regard.
However, in pkgsrc it is possible only to generate packages with
debugging symbols by providing proper CFLAGS for debugging and
setting the INSTALL_UNSTRIPPED flag. This make debugging, especially
for binary packages users, not very feasible.


About the project
-----------------
A more convenient way - like what RPM and Debian package manager do - is
to provide a <package>-{debuginfo,dbg} for <package> (where applicable)
that includes all stripped debugging symbols.
Thus, the project consists to add a mechanism in the pkgsrc
infrastructure in order to make the generation of the
<package>-{debuginfo,dbg} possible and transparent.

Providing <package>-{debuginfo,dbg} separately is also worth because
debugging symbols have a cost in term of disk space needs.
As an example, after quickly analyzing the space for NetBSD/amd64
-current installation sets it can be observed that {debug,xdebug}.tgz
sets are (in MB):

 $ ls -sk *debug.tgz | awk '{ s += $1 } END { print (s / 1024) }'
 492.578

...while all the other installation sets (not considering kern-XEN3*.tgz
kernels and {debug,xdebug}.tgz installation sets) are (in MB):

 $ ls -sk [bcegmt]* x[bcefs]* kern-GENERIC.tgz | awk '{ s += $1 } END { print (s / 1024) }'
 419.744

When extracted {debug,xdebug}.tgz need approximately 1.5GB.

Ideally all the packages that have USE_LANGUAGES "c" and/or "c99" and/or
"c++" (and maybe also other programming languages) can be compiled with
debugging symbols and it will be needed to automatically generate the
PLIST for the <package>-{debuginfo,dbg} given the PLIST of the <package>.
The <package>-{debuginfo,dbg} will be generated if a mk.conf variable
is defined (e.g. PKGSRC_MKDEBUG) and also automatically installed. In
order to handle them the <package>-{debuginfo,dbg} will DEPENDS on
<package> (of course this can be argued because - strictly speaking -
the <package>-{debuginfo,dbg} will not depends on any other packages
but in practice they are useful only if the <package> is installed).

All the above would - hopefully - not need any change in the
packages' Makefiles.
For packages that for some reasons the debugging symbols can not
be generated it will be needed to add a (per-package) Makefile
variable in order to avoid the generation and splitting of the
debug symbols (e.g. the various emulators/suse* packages).
Also any package that presents some other problems or when generating
debugging symbols isn't logical can use that variable to inhibit the
generation of debug symbols.


Related works
-------------
As stated above NetBSD already supports splitting the debugging symbols.
There are also some existing package management systems like RPM
or the Debian package manager that support that. Both NetBSD MKDEBUG*,
RPM and Debian package manager (and maybe also others) can probably
be taken as a good source for inspiration and design.

AFAIK, neither FreeBSD ports nor OpenBSD ports support splitting debug
symbols in separate packages.


Deliverables
------------
 * analyze how splitting debugging symbols works taking NetBSD MKDEBUG*
   as an example and also considering other package management systems,
   in particular for the design of the needed infrastructure.
 * design and implement the infrastructure in pkgsrc that handles the
   compilation, split of the debug symbols and generation of the
   <package>-{debuginfo,dbg} packages.
 * document the interface in ``The pkgsrc guide'' targeting pkgsrc
   users, MAINTAINERs and developers. Also provide useful documentation
   for the on-line pkgsrc documentation via the ``help'' target.
 * run a bulk build with the strip debug functionality turned on for at
   least a significant subset of packages on NetBSD in order to
   verify that the implemented infrastructure correctly works.
   This part will probably reveals problematic packages (e.g. package
   that ignore CFLAGS). Fixing these packages will also indirectly
   improve the hygiene of the pkgsrc ecosystem.


That's just a rough picture of the project and a lot of design choices
will need further discussions and pointers.
I consider this potential GSoC project a great opportunity to work
full-time on NetBSD and pkgsrc and, in particular, to become more
familiar with internals and various subsystems, not just of pkgsrc.
Last but not least, it's also a chance to cooperate with the NetBSD
and pkgsrc community in a not-so-short-term project.


Please note that currently the discussed project proposal isn't
considered as a possible GSoC project and neither has a mentor.
It was although discussed briefly here:

 http://wiki.netbsd.org/projects/project/pkgsrc-debugpkg/


Please let me know if this can be taken as a possible GSoC project
proposal (and in that case it will also need a mentor! :)). Please also
share any further ideas, issues, etc. that I've missed.
If you know any documentation/examples or any other possible kind of
resource related to this project I'm very happy to start reading and
studying it.


PS: riastradh@ kindly offered as a mentor for this project after quickly
    discussing that via IRC if another one can co-mentor and has more
    experience in pkgsrc internals as well.
    So also a co-mentor highly familiar with pkgsrc internals is
    needed.


Thank you very much for the attention!
Ciao,
L.