Re: python 3.7 removal proposal (before upstream EOL!!)

[Greg Troxel <gdt%lexort.com@localhost>]

Joerg Sonnenberger <joerg%bec.de@localhost> writes:

First, note that I'm dealing with this because I am handling the 2022Q4
branch.  Sorry if that was too buried in my removal proposal but it's
not a surprise.  As part of PMC I feel responsible for the overall
health of pkgsrc and for the stability of quarterly branches.  I've
always tried to ask for opinions and find rough consensus, which is
different from everyone being 100% happy.

> Am Sun, Oct 30, 2022 at 08:11:13AM -0400 schrieb Greg Troxel:
>> However, many python packages fail to build with 3.7.
>
> (1) This claim is not true. The vast majority of packages that that
> don't build are either a direct result of the setuptools_scm bug or due
> to a dependency on ipython or numpy.

Our realities apparently differ.

This all started because bulk builds where showing so many failures of
py37- packages that the other failures were harder to see because of
noise.  And, it went on for a while with nobody fixing it which is
pretty good evidence of a blend between nobody cares enough and nobody
knows how.   I wasn't concerned with exactly why, or how hard it might
be to fix -- as the responsible person for the branch I care that it is
or is not broken.

The breakage was first noted in mef@'s builds on 18 October.  wiz@
posted about py37 fallout on 25 October, asking if anyone was willing to
fix.  You didn't speak up, or fix it then.  adam@ was in favor of
outright removal.  nia@ had some thoughtful comments about struggling
with upstream instability and the reasons why we have lots of versions,
but wasn't clearly outright against 3.7 removal.  And you did not say
anything; in particular you did not say that 3.7 was important to you
and therefore you would fix all the issues.

After the dicussion petered out, wiz asked me what I thought, and we
agreed that massive bulk build failures were not ok.  So I raised the
question.  I specifically asked if anyone would volunteer to do the work
for one kind of 3.7 accomodation and *nobody* said they were willing.
You have *not yet* said you are willing to generally fix things, only
things like "So far I haven't seen that many cases of difficult to fix
breakage..."  which I consider an impolite assertion that other people
should fix this for your benefit.

You also haven't explained why you need 37 in pkgsrc HEAD, vs using
2022Q3 for your porting (porting from 2.7 should have started 4 years
ago, and has been seriously overdue for a while), vs just creating a
python37 venv and pip installing what you need there.

No one else has said they are using 37 in pkgsrc HEAD.  (No one mailed
me privately either.)

We do not have a rule that packages have to be kept until formal EOL.
The general idea is benefit vs maintenance pain and when we have lots of
versions (python, pgsql), we do tend to prune at EOL, but almost always
the pain is low and by EOL we genuinely don't think anyone is using
them.  In this case, there was actual trouble, and one person who
claimed that someday they would probably use it.  And, I read rough
consensus as agreeing that the community of python module upstreams as
decided that 3.7 is functionally EOL.

I also got one private note in favor of full removal.

To try to make things easier for you, I left the python37 package there
(against the wishes of at least Adam).  The changes I did make are small
and you can locally revert them and build what you want, and fix 37
things.  That effort seems tiny compared to 2.7 porting that is so hard
that it isn't done 2+ years after 2.7 EOL.

> (2) As I mentioned, Python 3.7 is important for the migration part from
> 2.7 and anyone involved with a bigger 2.7 code base in production will
> tell you that is both an involved process and can take a while.

The question is not if it would be useful for a few, but balancing that
vs the problems imposed more broadly.

> As such I request:
>
> (1) Immediate stop any further removal of Python 3.7 support. I find it
> frankly a major violation of pksgrc protocol to do so for a change that
> has been contested before.
>
> (2) Revert the change proposed here.

We don't have a rule that one person can veto anything.

It is entirely normal to have a discussion as a group for controversial
things, and to act on mostly-consensus, especially when done by PMC and
weighing stability going into quarterly branches.  No one questioned
"It's not ok to have lots of py37-foo failures on an ongoing basis".
Nobody else is telling me that I read rough consensus wrong, publically
or privately.

And more importantly, no one has said they will sign up to fix all the
problems to prevent others from having to wade through the trouble.

As for things being fixed, I have updated and done pkg_rr a few times,
deleted all my py37 packages, and python37 from both the installed
system and binary package dir, and then did

  /usr/pkgsrc/devel/py-iniconfig $ PYTHON_VERSIONS_ACCEPTED=37 make PYTHON_VERSION_REQUIRED=37 package

and this still resulted in "ERROR: Circular dependency detected".  I'll
update and try again, but I am curious if it works for anyone else.

> (3) (Re)introduce the last 3.7-supported versions of ipython and numpy
> with the versioned Python dependency framework. For numpy, most packages
> don't even need to change beyond dropping the (then) bogus py37
> restriction.

That sounds like demanding other people maintain things so you can run
funtionally obsolete python.  I agree that the python world is too quick
to deprecate things, but I don't think pkgsrc should have a policy of
not updating to versions of packages that fail with a
technically-not-EOL python version.  We would of course be squeezed by
that and needing new enough for other packages.  nia@'s suggestion about
expanding versioned_dependencies makes sense, but so does the idea that
it is a lot of work with little benefit.  So far, what we're mostly
seeing is pretty old py3N being desupported; almost everyhing (maybe
everything?) is working with 3.10 and even 3.8 is feeling really crufty
to me.