Re: mktool support for fetch

[Reinoud Zandijk <reinoud%dropje.13thmonkey.org@localhost>]

On Mon, Aug 19, 2024 at 07:52:23PM +0200, Jonathan Perkin wrote:
> On my vacation I've continued to work on mktool, and have now implemented a
> threaded fetch client that can significantly improve download speeds,
> especially when packages have multiple distfiles.
> 
> As an example, on my M1 MacBook Pro, I set up caddy as a localhost file
> server, and then ran in wip/grafana (over 5,000 distfiles):
> 
>   $ bmake distclean
>   $ bmake MASTER_SITE_OVERRIDE=http://localhost/ fetch
> 
> The default mk/fetch/fetch, using native curl, took:
> 
>   real	47m58.062s
>   user	13m35.410s
>   sys	33m43.399s
> 
> With TOOLS_PLATFORM.mktool=mktool, the same operation took just:
> 
>   real	0m4.865s
>   user	0m5.087s
>   sys	0m7.445s

This looks promising only why did you write it in Rust? What would it take to
refactor it in C using OpenMP? Shouldn't be that hard, esp. with OpenMP
working on a list of tasks to do.

If you want to do it parallel with async communications it might be slightly
trickier but still.

With regards,
Reinoud