Re: Split pkgdb

[Mike Pumford <mpumford%mudcovered.org.uk@localhost>]

On 30/12/2020 18:30, Greg Troxel wrote:
> Mike Pumford <mpumford%mudcovered.org.uk@localhost> writes:
>
> > Looks like I've somehow managed to split my package database in two.
> >
> > I've got files in both /usr/pkg/pkgdb and also files in
> > /var/db/pkg. Could it be that the pkg_install-20201218 package doesn't
> > have the same compatibility logic applied as the builtin version? I'm
> > using pkgin for to install binary packages built by one of my local
> > systems so it gets installed.
> >
> > Any suggestions for how to unpick the mess without uninstalling everything.
> >
> > This is all on 9.1 stable with the latest pkg_install pullups/
>
> I think 20201218 is the latest and should have the compat logic, but I
> am not 100% certain.  But that logic is "even if /usr/pkg/pkgdb is the
> compiled-in location, if /var/db/pkg exists (and /usr/pkg/pkgdb
> doesn't?) then use it".   I don't expect it to recover from split.
>
> I have written a migration script, but I haven't tried to handle your
> case.  But it will help I think.
>
> My advice follows, with  the caveat that it's not tested and you should
> pay close attention and check assumptions along the way.   This is
> easier than it's going to sound.
That's fine. I've got 1 relatively sacrificial system I can play on. 
Once it works on there I can then apply the same thing to the other 2 
tangled systems ;) I may script a few things if it seems easy and if I 
do I'll share them with the list.

> The plan is to head to one database in /usr/pkg/pkgdb.
>
> 1. Make a backup.  At the very least save a copy of /usr/pkg/pkgdb,
> pkgdb.refcount, and /var/db/pkg, pkg.refcount.

> 2. Figure out what pkgsrc branch you want to be running.  pkgsrc is
> frozen and HEAD is really close to what will become 2020Q4, so that's a
> good choice.  2020Q3 is also theoretically reasonable.  cvs up to the
> branch you are truing to do.  If your system is binary only that should
> be ok too.
This one is easy. I can freeze things at my current pkgsrc-current 
snapshot that I built successfully last weekend so I can guarantee a 
fixed pkgin database until I've got everything cleaned up. As things 
stand everything is actually operational so I can afford to take my time 
cleaning up.

> 3. Check pkg_add in /usr/sbin and /usr/pkg/sbin for version.  Make sure
> it's 20201218, or else something old like 20191008 or 20200701.  If it's
> 202012xx and not 1218, we should get that fixed.
> Everything is that version and I know that when I started all the 
pkg_install tools were pointing at the old /var/db location. This is why 
I think the compat code might not be in the pkgsrc version of pkg_install.

> 4. Get the "_NetBSD-pkgdb" script from the top level of pkgsrc HEAD.
> Alternatively get it from
>
>    http://ftp.netbsd.org/pub/pkgsrc/packages/NetBSD/
>
> but do not try to use the CDN because I am unclear on cache invalidation
> and expect this to get updated.  It is small :-)   Read it, and I think
> it will help you understand what's going on.

> 5. Try briefly to understand if the directories in /var/db/pkg and
> /usr/pkg/pkgdb collide, either literally or in package name with
> different versions.
>
> 6. In /var/db/pkg, do "mv * /usr/pkg/pkgdb".   If that errors due to
> collision and doesn't do any, try
>
>    for d in *; mv $d /usr/pkg/pkgdb
>    
> The point is to move each entry, so you only have entries in
> /usr/pkg/pkgdb.
>
> Figure out what's left.  Delete crud.  If you have two copies of
> someething, diff them and if the same, just rm the /var/db/pkg one.  If
> a conflict, make /usr/pkg/pkgdb have the new one.
>
> If you seem to have two copies of a package installed, do not use
> pkg_delete, as there is only one copy in the filesystem.  Instead just
> rm the older version's pkgdb directory.
>
> Remove /var/db/pkg.
>
> 7. In /var/db/pkg.refcount, try briefly to understand the overlap
> situation.  Then, more or less
>
>    tar cf - . | (cd /usr/pkg/pkgdb && tar xfv -)
>
> (untested, but you get the point: basically overlay the old dir onto the
> new one).  I am haziest on this part, but I don't think it's going to
> hurt you much.  For extra points be careful about not overwriting any
> files.
>
> Remove /var/db/pkg.refcount.
>
> 8. Run "_NetBSD-pkgdb check".  Expect complaints about PKG_DBDIR not
> being set, and about bad paths in refcount.  If that's all it says, run
> with the "fix" argument.   You'll get complaints but it should patch
> things up.  Run fix again, and I think it will be ok.
>
> 9. Run 'pkg_admin check' and 'pkg_admin rebuild-tree', as general pkg
> housekeeping.
>
> 10. Let us know how this went.  Thanks for being a tester of the script
> I wrote :-)
No problem. I'll give it a whirl tomorrow when I'm a bit more awake.

Mike