HEADS UP: Entropy overhaul

[Taylor R Campbell <riastradh%NetBSD.org@localhost>]

This week I committed an overhaul of the kernel entropy system.
Please let me know if you observe any snags!  For the technical
background, see the thread on tech-kern a few months ago:
<https://mail-index.NetBSD.org/tech-kern/2019/12/21/msg025876.html>.

Some user-visible changes:

- You get feedback about whether the bootloader or rc.d read the
  random seed, typically stored at /var/db/entropy-file, from messages
  in dmesg like:

	entropy: entering seed from bootloader
  or
	entropy: no seed from bootloader

  You can also query it after the fact with `rndctl -l' -- the entropy
  source called `seed' represents a seed loaded either by the
  bootloader or by rc.d.  See the DIAGNOSTICS section of the rnd(4)
  man page for more information and other related messages.

  On x86, you can specify the random seed with the `rndseed <path>'
  command, e.g. in /boot.cfg; on arm with efiboot, you can specify it
  in the `rndseed' efi environment variable, which you can set in
  /etc/efiboot.plist.  (Otherwise, rc.d will load it automatically --
  much later than the bootloader can -- in /etc/rc.d/random_seed.)

- /dev/random no longer blocks repeatedly: it will block after boot
  until the system has full entropy, and the never again.  This means
  applications that issue repeated reads from /dev/random will no
  longer repeatedly hang.

  However, for the purpose of testing applications to make sure they
  behave sensibly in the event that /dev/random does block, to
  simulate what would happen early at boot, you can re-enable the old
  behaviour with `sysctl -w kern.entropy.depletion=1'.

- You will see a lot more zeros in `rndctl -l' output than you used to
  see.  For example, on a Thinkpad T61p where I previously saw

	sd0                     946 disk estimate, collect, v, t, dt
	wd0                      44 disk estimate, collect, v, t, dt
	cd0                       0 disk estimate, collect, v, t, dt
	cpu1                     23 vm   estimate, collect, v, t, dv
	cpu0                      6 vm   estimate, collect, v, t, dv
	coretemp1-cpu1           14 env  estimate, collect, v, t, dv, dt
	coretemp0-cpu0           14 env  estimate, collect, v, t, dv, dt

  I now see

	sd0                       0 disk estimate, collect, v, t, dt
	wd0                       0 disk estimate, collect, v, t, dt
	cd0                       0 disk estimate, collect, v, t, dt
	cpu1                      0 vm   estimate, collect, v, t, dv
	cpu0                      0 vm   estimate, collect, v, t, dv
	coretemp1-cpu1            0 env  estimate, collect, v, t, dv, dt
	coretemp0-cpu0            0 env  estimate, collect, v, t, dv, dt

  This does _not_ mean the system has ceased to observe samples from
  these sources -- exactly the same data will be fed in by drivers.

  At the moment, there is nothing that records the number of bits of
  _data_ entered from each source, although you could
	dtrace -n 'fbt::rnd_add_data:entry { @[stack()] = sum(arg3)}'
  to get an approximation.  Rather, the numbers in `rndctl -l' are
  meant to be upper bounds on the number of bits of _entropy_ in the
  _processes_ that produced the data.  What does this mean?

  In the past, when the kernel took samples from devices not meant to
  be hardware RNGs, it would feed the samples into an extremely
  simple-minded model -- a model that bears no relation to the
  physical devices -- to prestidigitate a bogus fabulation of how much
  entropy is in the process that produced the data.  In other words,
  the kernel was making things up and lying to you about physical
  processes it doesn't actually know anything about.

  As part of the overhaul, I removed this dishonest fabulation; only
  drivers for devices about which the unpredictability of the
  underlying physical process is known or advertised -- generally just
  hardware RNG devices or the random seed stored on disk from the
  previous boot -- will supply nonzero entropy estimates.  (As the
  operator, you can instruct rndctl to disbelieve the entropy estimate
  of any particular device or type of device with `rndctl -E', or
  disable collecting the data altogether with `rndctl -C'.)

- You may find there are now more warnings about extracting entropy
  too early on some machines without hardware random number generator
  devices, for the same reasons `rndctl -l' shows more zeros.  The
  warning is rate-limited, and goes away as soon as the system deems
  itself to have reached full entropy once -- but if there's no
  hardware RNG and there's no random seed, it may be hard to reach
  that.

  This is generally not a new deficiency; rather it's a new warning
  about an old deficiency that NetBSD (and other systems like Linux)
  was just being silent and/or dishonest about, sometimes quietly
  leading to problems like <https://factorable.net/>.

  Here are some workarounds:

  . For virtual machines, you may be able to enable a virtual hardware
    RNG.  For instance, in qemu, you can use
	-device virtio-rng-pci
    to attach a virtual hardware RNG to pass entropy from the host to
    the guest via a virtual PCI device, and NetBSD will automatically
    take advantage of it.

    (Minor caveat: Apparently netbsd-current on aarch64 has a bug
    right now leading to an interrupt storm at boot when you do this,
    but that should be fixable!)

  . If you are working on (say) a modern x86 laptop or workstation
    with RDRAND/RDSEED, and you want to install NetBSD on an appliance
    with no hardware RNG before you deploy it, you can run
	rndctl -S seed
    on your x86 machine, copy the file `seed' over to
    `/var/db/entropy-file' on the appliance, and run
	rndctl -L /var/db/entropy-file
    on the appliance; then rc.d should take care of it from there,
    saving it on shutdown and reloading it on boot (even if the system
    crashes).

  . If you have a physical machine that has a hardware RNG but we just
    don't have a driver for it, well, let's try to write a driver for
    it!  The rnd(9) API is pretty easy and supports both synchronous
    hardware RNGs like Intel RDRAND/RDSEED where you run a CPU
    instruction and immediately get data back, and asynchronous
    hardware RNGs where you issue a request to a device and then later
    get an interrupt back with data.

  . If you _know_ the system is in an unpredictable state, but it
    doesn't think it has enough entropy according to `rndctl -l' or
    `sysctl kern.entropy.needed', you can always fool it into thinking
    there is more entropy by running
	dd if=/dev/urandom of=/dev/random bs=32 count=1
	sysctl -w kern.entropy.consolidate=1
    as root.  The kernel will assume that any data written by root to
    /dev/u?random have full entropy -- that is, that every byte was
    independently chosen by a process with uniform distribution.