mktool support for fetch

[Jonathan Perkin <jperkin%mnx.io@localhost>]

Hi!

On my vacation I've continued to work on mktool, and have now 
implemented a threaded fetch client that can significantly improve 
download speeds, especially when packages have multiple distfiles.

As an example, on my M1 MacBook Pro, I set up caddy as a localhost file 
server, and then ran in wip/grafana (over 5,000 distfiles):

  $ bmake distclean
  $ bmake MASTER_SITE_OVERRIDE=http://localhost/ fetch

The default mk/fetch/fetch, using native curl, took:

  real	47m58.062s
  user	13m35.410s
  sys	33m43.399s

With TOOLS_PLATFORM.mktool=mktool, the same operation took just:

  real	0m4.865s
  user	0m5.087s
  sys	0m7.445s

It's still in the early stages, but I'd love for those who are 
interested to test it out and let me know of any issues or feature 
requests while I have some time to work on it.  It supports direct URLs, 
failover sites, and performs both checksum and size verification for 
every file.

I've pushed the necessary change to my "dev/mktool" branch:

  https://github.com/TritonDataCenter/pkgsrc/tree/dev/mktool

i.e. you just need to apply this diff:

  https://github.com/TritonDataCenter/pkgsrc/commit/0b0e63ea7030714eebf3585f014dd1b5406fd51d

and "cargo install mktool" to install the latest client (0.1.11).

You can set the environment variable MKTOOL_JOBS to configure how many 
parallel fetches, the default is 4.

Part of this work required enhancing the pkgsrc-rs library to fully 
support distinfo files, and you may have seen that it identified some 
issues in existing distinfo files during my testing across the tree.

Enjoy!

--
Jonathan Perkin   -   mnx.io   -   pkgsrc.smartos.org
Open Source Complete Cloud   www.tritondatacenter.com