Re: mktool support for fetch

[Thomas Klausner <wiz%gatalith.at@localhost>]

On Mon, Aug 19, 2024 at 07:52:23PM +0200, Jonathan Perkin wrote:
> On my vacation I've continued to work on mktool, and have now implemented a
> threaded fetch client that can significantly improve download speeds,
> especially when packages have multiple distfiles.
> 
> As an example, on my M1 MacBook Pro, I set up caddy as a localhost file
> server, and then ran in wip/grafana (over 5,000 distfiles):
> 
>   $ bmake distclean
>   $ bmake MASTER_SITE_OVERRIDE=http://localhost/ fetch
> 
> The default mk/fetch/fetch, using native curl, took:
> 
>   real	47m58.062s
>   user	13m35.410s
>   sys	33m43.399s
> 
> With TOOLS_PLATFORM.mktool=mktool, the same operation took just:
> 
>   real	0m4.865s
>   user	0m5.087s
>   sys	0m7.445s

These are really great numbers!

> It's still in the early stages, but I'd love for those who are interested to
> test it out and let me know of any issues or feature requests while I have
> some time to work on it.  It supports direct URLs, failover sites, and
> performs both checksum and size verification for every file.
> 
> I've pushed the necessary change to my "dev/mktool" branch:
> 
>   https://github.com/TritonDataCenter/pkgsrc/tree/dev/mktool
> 
> i.e. you just need to apply this diff:
> 
>   https://github.com/TritonDataCenter/pkgsrc/commit/0b0e63ea7030714eebf3585f014dd1b5406fd51d
> 
> and "cargo install mktool" to install the latest client (0.1.11).

I'm fine with adding the TOOL ifdefs to mk/, does anyone else see
problems with that?
 Thomas