Re: Automated pkgsrc commit testing

[Thomas Klausner <wiz%gatalith.at@localhost>]

Hi Jonathan!

On Tue, Jul 23, 2024 at 03:53:38PM +0100, Jonathan Perkin wrote:
> In another attempt to improve the quality of what is committed to pkgsrc,
> I've spent the last few days implementing an automated testing rig on
> GitHub.

Wow, that's a really neat tool!

> The idea is that you submit a pull-request to the repository I've set up,
> and it will run a suite of automated checks against your PR.  You can then
> either modify the change in response to any failures and re-test, or
> continue and commit to pkgsrc with increased confidence that you won't cause
> regressions.
> 
> Here is what I've implemented so far.
> 
>  * The commit message is checked to conform to our current standards,
>    i.e. "pkg: description" in the first line, limited to 50 characters,
>    followed by a blank line, followed by further details with lines
>    limited to 80 characters.
> 
>  * pkglint runs against all files that you touch.  Roland kindly added
>    the -Werror flag to pkglint 23.5.0 so that this check will fail if
>    there are any warnings.

Does it expect it to be pkglint clean, or just not worse than before?
There are some cases where we cannot make a package pkglint clean, e.g.

ERROR: gmake/options.mk:13: "../../devel/gettext-lib/builtin.mk" must not be included directly. Include "../../devel/gettext-lib/buildlink3.mk" instead.

>  * The following platforms will build test your change:
> 
>      Cygwin 3.5.3 running on Windows 2022 x86_64
>      FreeBSD 14.1 x86_64
>      NetBSD 10.0 x86_64
>      OmniOS r151050 x86_64
>      OpenBSD 7.5 x86_64
>      Ubuntu 22.04 x86_64
>      macOS 13 x86_64
>      macOS 14 arm64
> 
>    Notably the NetBSD builds run with non-standard settings for
>    PREFER_PKGSRC, PKGINFODIR, and PKGMANDIR, as these are commonly
>    broken due to commits only being tested with default NetBSD settings
>    (see e.g. devel/py-gi-docgen failures highlighted by pkgsrc-bulk for
>    the last few weeks that have gone unfixed).

This is another discussion, but I think we should not assume that
committers read all pkgsrc-bulk emails, so forwarding of such issues
to pkgsrc-users or the committer(s) is probably a good idea.

>    The BSD runner that I'm using does support arm64 guests, but due to
>    virtualisation overhead I disabled them for now as they're just too
>    slow to be of practical use.  Hopefully in the future they can run on
>    native arm64 hardware.
> 
> If any of these steps result in failure you'll get an email.  For the commit
> message and pkglint checks you can view the failure via the web interface,
> and for the build tests you can download an archive containing the build
> logs and bmake output.
> 
> As an example, here's a pull request I made where I deliberately broke the
> build on SunOS:
> 
>   https://github.com/pkgsrc-ci/pkgsrc/actions/runs/10060356402
> 
> I've also implemented a workflow for pre-building packages and storing them
> on an MNX-hosted server.  The example PR above uses this cache so that
> dependent packages are installed using DEPENDS_TARGET=bin-install and
> drastically cuts down on build time.  I'll be populating the cache further
> over time (it's currently running meta-pkgs/bulk-large).
> 
> Due to the state of pkgsrc trunk right now, this is currently based on the
> 2024Q2 branch, with a couple of changes I've made to fix bootstrap on Cygwin
> and OpenBSD.

What is the 'state of pkgsrc trunk'?

> If you would like to get involved, have access to the GitHub org, have any
> questions, or have suggestions on any other tests you'd like me to implement
> (or implement yourself), please let me know.

Please add me: 0-wiz-0

Thanks,
 Thomas