Re: pkgin repositories

[Jonathan Perkin <jperkin%joyent.com@localhost>]

* On 2018-11-02 at 15:16 GMT, Jason Bacon wrote:

> On 11/1/18 5:44 PM, Jonathan Perkin wrote:
> > * On 2018-11-01 at 22:32 GMT, Jason Bacon wrote:
> > 
> > > My current etc/pkgin/repositories.conf is as follows:
> > > 
> > > https://mirror1.hpc.uwm.edu/pkgsrc/packages/usr/pkg/RHEL7/All
> > > https://mirror2.hpc.uwm.edu/pkgsrc/packages/usr/pkg/RHEL7/All
> > > 
> > > I've noticed a few times that if either server is down, pkgin operations
> > > like update or install will fail.
> > > 
> > > I would have thought that if one server is down, it would fail over to the
> > > next one, but it appears
> > > they all have to be available.
> > > 
> > > Is this by design, and if so, for what reason?
> > Yes, this is the current design.  This is the first I've heard of
> > someone trying to use it for round-robin or failover type access.  It
> > was introduced in order to support additional repositories that
> > contain extra packages (usually locally-built) not available from the
> > primary.
> > 
> > The multiple repository code already has some rough edges with unclear
> > semantics on exactly how packages are chosen when available from
> > multiple repositories, and I wouldn't want to commit even further to
> > trying to support that.
> > 
> Thanks - this is not terribly important.  Given the limited documentation on
> repositories.conf, I just guessed that the multiple lines represented
> redundant mirrors of the same package repo.
> 
> While we're on the subject, though, perhaps we could support this sort of
> redundancy with a separate syntax, such as multiple comma-separated URLs on
> the same line.  I think it would be a nice feature to have if it's not too
> much trouble.

It's not something we would spend time implementing, we use load
balancers at Joyent and NetBSD use a CDN, so we would have no use for
it.  Plus I don't think it's a good idea ;-)

You're welcome to take a shot at it, but it's worth considering that
there are some important factors to consider, one of the most
important being that you must ensure complete atomicity between all
the mirrors, otherwise you will get client failures when pkg_summary
and the packages diverge during a bulk build upload.

This is something you see on a single server that can be avoided by
using rsync --link-dest and switching out the repository once
everything has fully synced, but if you plan to load balance across
multiple servers then you will need to perform this atomicity across
all of them simultaneously.

-- 
Jonathan Perkin  -  Joyent, Inc.  -  www.joyent.com