Mailing List Archive

python, my nemesis
Hi,

I'd like to update a system that is about 1 year old. I have python3.7
and python 3.8 installed, 3.7 is the default. After updating the repo,
it was suggested to update portage first (probably useful to support
new EAPI versions).
I read about updating to python 3.9, so I created

---
~ # cat /etc/portage/package.use/py
*/* PYTHON_TARGETS: -* python3_9 python3_8
*/* PYTHON_SINGLE_TARGET: -* python3_9
---

as suggested. However, there are still collisions (and I don't really
understand them):

---
~ # emerge -p --oneshot sys-apps/portage

These are the packages that would be merged, in order:

Calculating dependencies... done!
[ebuild N ] acct-group/portage-0
[ebuild N ] acct-user/portage-0
[ebuild U ] sys-devel/automake-1.16.4 [1.16.1-r1] USE="-test%"
[ebuild NS ] dev-lang/python-3.9.6_p2 [2.7.18-r2, 3.7.8-r2, 3.8.5]
USE="sqlite* -verify-sig%"
[ebuild U ] dev-python/certifi-10001-r1 [10001]
PYTHON_TARGETS="python3_8* python3_9* (-python3_10)"
[ebuild U ] dev-python/setuptools-57.4.0-r2 [46.4.0-r3]
PYTHON_TARGETS="python3_8* python3_9* (-python3_10)"
[ebuild N ] dev-python/toml-0.10.2 USE="-test"
PYTHON_TARGETS="python3_8 python3_9 (-pypy3) (-python3_10)"
[ebuild U ] dev-python/setuptools_scm-6.0.1-r1 [4.1.2-r1]
PYTHON_TARGETS="python3_8* python3_9* (-python3_10)"
[ebuild N ] dev-python/charset_normalizer-2.0.4 USE="-test"
PYTHON_TARGETS="python3_8 python3_9 (-pypy3) (-python3_10)"
[ebuild U ] dev-python/idna-3.2 [2.10-r1] PYTHON_TARGETS="python3_8*
python3_9* (-python3_10)"
[ebuild R ] dev-python/PySocks-1.7.1-r1 PYTHON_TARGETS="python3_8*
python3_9* (-python3_10)"
[ebuild U ] dev-python/urllib3-1.26.6
[1.25.10-r1] PYTHON_TARGETS="python3_8* python3_9* (-python3_10)"
[ebuild U ] dev-python/requests-2.26.0 [2.24.0-r1]
PYTHON_TARGETS="python3_8* python3_9* (-python3_10)"
[ebuild U ] app-portage/gemato-16.2 [15.2]
PYTHON_TARGETS="python3_8* python3_9* (-python3_10)"
[ebuild U ] sys-apps/portage-3.0.20-r6 [3.0.4-r1]
PYTHON_TARGETS="python3_8* python3_9* (-python3_10)"

!!! Multiple package instances within a single package slot have been
pulled !!! into the dependency graph, resulting in a slot conflict:

sys-apps/portage:0

(sys-apps/portage-3.0.20-r6:0/0::gentoo, ebuild scheduled for merge)
USE="(ipc) native-extensions rsync-verify xattr -apidoc -build -doc
-gentoo-dev (-selinux) -test" ABI_X86="(64)"
PYTHON_TARGETS="python3_8 python3_9 (-pypy3) (-python3_10)" pulled in
by sys-apps/portage (Argument)

(sys-apps/portage-3.0.4-r1-3:0/0::gentoo, installed) USE="(ipc)
native-extensions rsync-verify xattr -apidoc -build -doc -gentoo-dev
(-selinux) -test" ABI_X86="(64)" PYTHON_TARGETS="python3_7 (-pypy3)
-python3_6 -python3_8 -python3_9" pulled in by
sys-apps/portage[python_targets_python3_7(-),-python_single_target_pypy3(-),-python_single_target_python3_6(-),-python_single_target_python3_7(-),-python_single_target_python3_8(-)]
required by (app-portage/gentoolkit-0.5.0:0/0::gentoo, installed)
USE="-test" ABI_X86="(64)" PYTHON_TARGETS="python3_7 (-pypy3)
-python3_6 -python3_8"

[...]
---



Plus many similar ones on packages like setuptools, gemato etc.
I don't see how to get out of this vicious circle, any hints?


cu
Gerrit
Re: python, my nemesis [ In reply to ]
On Mon, 2021-09-20 at 14:50 +0200, Gerrit Kuehn wrote:
> Hi,
>
> I'd like to update a system that is about 1 year old. I have python3.7
> and python 3.8 installed, 3.7 is the default. After updating the repo,
> it was suggested to update portage first (probably useful to support
> new EAPI versions).
> I read about updating to python 3.9, so I created
>
> ---
> ~ # cat /etc/portage/package.use/py
> */* PYTHON_TARGETS: -* python3_9 python3_8
> */* PYTHON_SINGLE_TARGET: -* python3_9
> ---

You should probably not mess with these variables until after your
system is 100% updated and consistent. And even then, probably not.

With our package manager written in python, you often need old python
stuff to build the new python stuff, and disabling the old python stuff
will throw a wrench into that. Even in situations where technically
some upgrade path exists, the complexity of the python dependencies
often means that the package manager will give up before it finds the
solution unless the solution is obvious. By tweaking those variables,
you make the solution less obvious to it.
Re: python, my nemesis [ In reply to ]
On Mon, 20 Sep 2021 09:18:23 -0400
Michael Orlitzky <mjo@gentoo.org> wrote:

> > ---
> > ~ # cat /etc/portage/package.use/py
> > */* PYTHON_TARGETS: -* python3_9 python3_8
> > */* PYTHON_SINGLE_TARGET: -* python3_9
> > ---

> You should probably not mess with these variables until after your
> system is 100% updated and consistent. And even then, probably not.
>
> With our package manager written in python, you often need old python
> stuff to build the new python stuff, and disabling the old python
> stuff will throw a wrench into that. Even in situations where
> technically some upgrade path exists, the complexity of the python
> dependencies often means that the package manager will give up before
> it finds the solution unless the solution is obvious. By tweaking
> those variables, you make the solution less obvious to it.

Well, this was the suggested way to go, see
https://www.gentoo.org/support/news-items/2021-05-05-python3-9.html

But also when trying "emerge -1vUD @world" (be it with or without the
package.use settings), I get stuck in conflicts (mostly on perl and
setuptools). perl issues would probably resolve once I have EAPI8
support, i.e. get new portage.


cu
Gerrit
Re: python, my nemesis [ In reply to ]
On Monday, 20 September 2021 14:56:46 BST Gerrit Kuehn wrote:

> Well, this was the suggested way to go, see
> https://www.gentoo.org/support/news-items/2021-05-05-python3-9.html
>
> But also when trying "emerge -1vUD @world" (be it with or without the
> package.use settings), I get stuck in conflicts (mostly on perl and
> setuptools). perl issues would probably resolve once I have EAPI8
> support, i.e. get new portage.
>
>
> cu
> Gerrit

With a year old system you should question if reinstalling your system after a
back up of configuration and data files would be a smarter approach. If you
*must* upgrade your current installation for learning or as an experiment,
then this is something which has been done before. You will need to walk
through all the major upgrades of the last year, by using git to gradually
catch up with the latest stable versions. This means a lot of packages will
have to be updated a number of times.

Personally, I'd back up /home /etc and world file, plus any databases or
websites if stored under /var/, then untar the latest stage 3 tarball and
update @system and @world. Overall it should be a *much* faster approach to
allow you to bring your installation up to date.
Re: python, my nemesis [ In reply to ]
On Mon, 20 Sep 2021 09:18:23 -0400
Michael Orlitzky <mjo@gentoo.org> wrote:

> With our package manager written in python, you often need old python
> stuff to build the new python stuff, and disabling the old python
> stuff will throw a wrench into that. Even in situations where
> technically some upgrade path exists, the complexity of the python
> dependencies often means that the package manager will give up before
> it finds the solution unless the solution is obvious. By tweaking
> those variables, you make the solution less obvious to it.

I took out the extra settings and solved a few conflicts manually. I'm
down to this now:

---
~ # emerge -p --oneshot sys-apps/portage --verbose-conflicts

These are the packages that would be merged, in order:

Calculating dependencies... done!
[ebuild N ] acct-group/portage-0
[ebuild N ] acct-user/portage-0
[ebuild U ] sys-apps/portage-3.0.20-r6 [3.0.4-r1]
PYTHON_TARGETS="python3_9* (-python3_10)"

!!! Multiple package instances within a single package slot have been
pulled !!! into the dependency graph, resulting in a slot conflict:

sys-apps/portage:0

(sys-apps/portage-3.0.20-r6:0/0::gentoo, ebuild scheduled for merge)
USE="(ipc) native-extensions rsync-verify xattr -apidoc -build -doc
-gentoo-dev (-selinux) -test" ABI_X86="(64)"
PYTHON_TARGETS="python3_9 (-pypy3) (-python3_10) -python3_8" pulled
in by sys-apps/portage (Argument)

(sys-apps/portage-3.0.4-r1-3:0/0::gentoo, installed) USE="(ipc)
native-extensions rsync-verify xattr -apidoc -build -doc -gentoo-dev
(-selinux) -test" ABI_X86="(64)" PYTHON_TARGETS="python3_7 (-pypy3)
-python3_6 -python3_8 -python3_9" pulled in by
sys-apps/portage[python_targets_python3_7(-),-python_single_target_pypy3(-),-python_single_target_python3_6(-),-python_single_target_python3_7(-),-python_single_target_python3_8(-)]
required by (app-portage/gentoolkit-0.5.0:0/0::gentoo, installed)
USE="-test" ABI_X86="(64)" PYTHON_TARGETS="python3_7 (-pypy3)
-python3_6 -python3_8"
---


This looks like portage is blocked by itself... so how to solve this
one?


cu
Gerrit
Re: python, my nemesis [ In reply to ]
On Mon, 2021-09-20 at 15:56 +0200, Gerrit Kuehn wrote:
>
> Well, this was the suggested way to go, see
> https://www.gentoo.org/support/news-items/2021-05-05-python3-9.html

It was only suggested for a period of about three weeks:

We are planning to switch the default Python target of Gentoo systems
on 2021-06-01, from Python 3.8 to Python 3.9. If you have not  
changed the values of PYTHON_TARGETS or PYTHON_SINGLE_TARGET, the 
change will have immediate effect on your system and the package 
manager will try to switch automatically on the next upgrade 
following the change.

>
> But also when trying "emerge -1vUD @world" (be it with or without the
> package.use settings), I get stuck in conflicts (mostly on perl and
> setuptools). perl issues would probably resolve once I have EAPI8
> support, i.e. get new portage.
>
>

You can try --backtrack=200 (or a larger value) to fix that; it was a
problem for a lot of people. Even though the end result of both
approaches is "it doesn't work," I promise you you're better off not
messing with the python stuff until the system is fully upgraded.
Upgrading a year-old system is hard enough, and the introduction of
EAPI=8 just made it worse.
Re: python, my nemesis [ In reply to ]
On Mon, 20 Sep 2021 16:31:59 +0200
Gerrit Kuehn <gerrit.kuehn@aei.mpg.de> wrote:

> This looks like portage is blocked by itself... so how to solve this
> one?

Well, I simply installed the new virtual packages manually and used
"--nodeps" on portage itself afterwards... and here we are. Looks like
new portage will update the rest of my system without any issue now (at
least it says so when checking with -p), so here we go. Still no idea
why this was blocking so hard, though.


cu
Gerrit
Re: python, my nemesis [ In reply to ]
On Mon, 20 Sept 2021 at 16:32, Gerrit Kuehn <gerrit.kuehn@aei.mpg.de> wrote:
> (sys-apps/portage-3.0.4-r1-3:0/0::gentoo, installed) USE="(ipc)
> native-extensions rsync-verify xattr -apidoc -build -doc -gentoo-dev
> (-selinux) -test" ABI_X86="(64)" PYTHON_TARGETS="python3_7 (-pypy3)
> -python3_6 -python3_8 -python3_9" pulled in by
> sys-apps/portage[python_targets_python3_7(-),-python_single_target_pypy3(-),-python_single_target_python3_6(-),-python_single_target_python3_7(-),-python_single_target_python3_8(-)]
> required by (app-portage/gentoolkit-0.5.0:0/0::gentoo, installed)
> USE="-test" ABI_X86="(64)" PYTHON_TARGETS="python3_7 (-pypy3)
> -python3_6 -python3_8"
> ---
>
>
> This looks like portage is blocked by itself... so how to solve this
> one?

This is because your gentoolkit installed version has no support for
python 3.9. You might be able to include gentoolkit in your emerge
command, or you might need to build portage with python 3.8 support as
well as 3.9. This type of problem might crop up for several other
packages as you try to get python updated.

Cheers,
Arve
Re: python, my nemesis [ In reply to ]
On Mon, 20 Sep 2021 15:29:30 +0100
Michael <confabulate@kintzios.com> wrote:

> With a year old system you should question if reinstalling your
> system after a back up of configuration and data files would be a
> smarter approach. If you *must* upgrade your current installation
> for learning or as an experiment, then this is something which has
> been done before.

I know. I'm a happy Gentoo user for more than 20 years now. However,
this is the master installation for a diskless setup providing the
baseline for more than a dozen servers. Of course, I can still reinstall
but this would be more work than just one machine + a few config files.
As one year is not too terribly old (imho) and this looked like
something that might be easily solved, I decided to ask for advice here
first.

> Personally, I'd back up /home /etc and world file, plus any databases
> or websites if stored under /var/, then untar the latest stage 3
> tarball and update @system and @world.

Just extracting stage3 over everything that is already there?

> Overall it should be a *much*
> faster approach to allow you to bring your installation up to date.

Well, yeah, your mileage may vary.


cu
Gerrit
Re: python, my nemesis [ In reply to ]
On Mon, 20 Sep 2021 16:41:17 +0200
Arve Barsnes <arve.barsnes@gmail.com> wrote:

> This is because your gentoolkit installed version has no support for
> python 3.9.

Yes, I must have been overlooking this all the time. That's why I came
here first (more/better eyes on the issue :-).

> You might be able to include gentoolkit in your emerge
> command, or you might need to build portage with python 3.8 support as
> well as 3.9. This type of problem might crop up for several other
> packages as you try to get python updated.

emerge -p1vUD @world

lists about 200 ports to rebuild now, without showing any other issues
ahead. I'll go for it (and keep my fingers crossed ;-). Thank you for
your support.


cu
Gerrit
Re: python, my nemesis [ In reply to ]
On Mon, 20 Sep 2021 10:34:16 -0400
Michael Orlitzky <mjo@gentoo.org> wrote:

> > Well, this was the suggested way to go, see
> > https://www.gentoo.org/support/news-items/2021-05-05-python3-9.html

> It was only suggested for a period of about three weeks:

Uh, yes, another thing I overlooked. I should get more sleep, I guess.
:) Thanks for pointing this out!


cu
Gerrit
Re: python, my nemesis [ In reply to ]
On Mon, 20 Sep 2021 15:56:46 +0200, Gerrit Kuehn wrote:

> > > ~ # cat /etc/portage/package.use/py
> > > */* PYTHON_TARGETS: -* python3_9 python3_8
> > > */* PYTHON_SINGLE_TARGET: -* python3_9
> > > ---
>
> > You should probably not mess with these variables until after your
> > system is 100% updated and consistent. And even then, probably not.
> >
> > With our package manager written in python, you often need old python
> > stuff to build the new python stuff, and disabling the old python
> > stuff will throw a wrench into that. Even in situations where
> > technically some upgrade path exists, the complexity of the python
> > dependencies often means that the package manager will give up before
> > it finds the solution unless the solution is obvious. By tweaking
> > those variables, you make the solution less obvious to it.
>
> Well, this was the suggested way to go, see
> https://www.gentoo.org/support/news-items/2021-05-05-python3-9.html

That news item is about going from 3.8 to 3.9, you are on 3.7. I'd try
removing the -* items are trying again.


--
Neil Bothwick

"We demand rigidly defined areas of doubt and uncertainty!"
Re: python, my nemesis [ In reply to ]
On Monday, 20 September 2021 15:52:03 BST Gerrit Kuehn wrote:
> On Mon, 20 Sep 2021 15:29:30 +0100
>
> Michael <confabulate@kintzios.com> wrote:
> > Personally, I'd back up /home /etc and world file, plus any databases
> > or websites if stored under /var/, then untar the latest stage 3
> > tarball and update @system and @world.
>
> Just extracting stage3 over everything that is already there?

No, I move out of the way the config/data files I want to keep and move them
back in after untarring the stage 3.


> > Overall it should be a *much*
> > faster approach to allow you to bring your installation up to date.
>
> Well, yeah, your mileage may vary.

Quite, if you can get your existing installation to only run a minimal number
of rebuilds to arrive at an upgraded toolchain, then the benefit of
reinstalling wouldn't be there. This could have been the case on an average
year, when python deprecations didn't accelerate and EAPI didn't change. If
however you end up tying up yourself in knots with subsequent python upgrades
and difficult to resolve conflicts, then the pain Vs gain calculus changes.
Re: python, my nemesis [ In reply to ]
On Mon, 20 Sep 2021 16:17:17 +0100
Neil Bothwick <neil@digimed.co.uk> wrote:

> > Well, this was the suggested way to go, see
> > https://www.gentoo.org/support/news-items/2021-05-05-python3-9.html

> That news item is about going from 3.8 to 3.9, you are on 3.7.

Yes and no. It started with

---
Display-If-Installed: dev-lang/python:3.7
Display-If-Installed: dev-lang/python:3.8
---

So my impression was I might find myself in a similar situation with
3.7.

> I'd try
> removing the -* items are trying again.

Same issue (actually, I tried that first and only found the news item
after this didn't work).
But I think I fixed most issues now (see my previous posts). Thank you
for taking time to look into this.


cu
Gerrit
Re: python, my nemesis [ In reply to ]
On Mon, 20 Sep 2021 16:20:58 +0100
Michael <confabulate@kintzios.com> wrote:

> > Well, yeah, your mileage may vary.

> Quite, if you can get your existing installation to only run a
> minimal number of rebuilds to arrive at an upgraded toolchain, then
> the benefit of reinstalling wouldn't be there. This could have been
> the case on an average year, when python deprecations didn't
> accelerate and EAPI didn't change. If however you end up tying up
> yourself in knots with subsequent python upgrades and difficult to
> resolve conflicts, then the pain Vs gain calculus changes.

I was already wondering why so many things happened during the
(comparatively) short time I didn't watch. Looks like people have been
in lockdown with plenty of time to come around with new things.
Fortunately, the base installation I have to update doesn't contain
many applications (e.g., no X involved). Everything sits on an NFS
server with ZFS below it, so it is quite easy to do snaphots and go
back and forth between them, and we have plenty to CPU power to rebuild
things.
OTOH, the orchestration to have the setup re-done in a fully automated
way is still under development, so I definitely wanted to try updating
before really starting over.


cu
Gerrit
RE: python, my nemesis [ In reply to ]
-----Original Message-----
From: Neil Bothwick <neil@digimed.co.uk>
Sent: Monday, September 20, 2021 8:17 AM
To: gentoo-user@lists.gentoo.org
Subject: Re: [gentoo-user] python, my nemesis

On Mon, 20 Sep 2021 15:56:46 +0200, Gerrit Kuehn wrote:

>> > > ~ # cat /etc/portage/package.use/py
>> > > */* PYTHON_TARGETS: -* python3_9 python3_8
>> > > */* PYTHON_SINGLE_TARGET: -* python3_9
>> > > ---
>>
>> > You should probably not mess with these variables until after your
>> > system is 100% updated and consistent. And even then, probably not.
>> >
>> > With our package manager written in python, you often need old
>> > python stuff to build the new python stuff, and disabling the old
>> > python stuff will throw a wrench into that. Even in situations where
>> > technically some upgrade path exists, the complexity of the python
>> > dependencies often means that the package manager will give up
>> > before it finds the solution unless the solution is obvious. By
>> > tweaking those variables, you make the solution less obvious to it.
>>
>> Well, this was the suggested way to go, see
>> https://www.gentoo.org/support/news-items/2021-05-05-python3-9.html
>
>That news item is about going from 3.8 to 3.9, you are on 3.7. I'd try removing the -* items are trying again.
>
>
When upgrading really old systems I find it useful to run:
export PYTHON_TARGETS="$(eix -I | grep -o "python[23]_[0-9]*" | sort -u)"

This will temporarily set the system (for the duration of the current session) to build for every version of python that is either available for new builds or currently in use. The dependency resolver seems to have an easier time finding a path when it doesn't think it has to throw away all the currently installed stuff first.

Note that this will end up rebuilding a lot more than is technically necessary, and then you'll need to rebuild again with the python versions you actually want and depclean the extras. Personally though I rather prefer to have the computer do more work instead of spending hours using *my* brain to find the minimum sized rebuild set to upgrade the system.

LMP
RE: python, my nemesis [ In reply to ]
-----Original Message-----
From: Michael <confabulate@kintzios.com>
Sent: Monday, September 20, 2021 8:21 AM
To: gentoo-user@lists.gentoo.org
Subject: Re: [gentoo-user] python, my nemesis

On Monday, 20 September 2021 15:52:03 BST Gerrit Kuehn wrote:
>>
>> Just extracting stage3 over everything that is already there?

>No, I move out of the way the config/data files I want to keep and move them back in after untarring the stage 3.

A less traumatic place to start I find is to unpack the stage3 into /tmp or wherever it's out of the way and then chroot into it and use quickpkg to bundle up newer versions of blocked core utilities. Not needing build deps can quite often simplify the upgrade path enough to get past certain things.

You can also (with a lot of annoyance) use the new portage that's in the tarball to get past eapi restrictions and whatnot and tell it to install the stuff it builds into your original system's folders. I've even used that to salvage systems that got moved to new hardware with an incompatible set of processor flags. Not straightforward since you have to manage both the build environment in the stage3 and the install environment in the original system, but there's very little you can't fix with this approach.

Just don't let your system build binary packages for virtual/* under any circumstances. That never ends well. I really need to write up a request to have portage blacklist those by default when buildpkg is enabled...

LMP
Re: python, my nemesis [ In reply to ]
On 2021-09-20, Michael <confabulate@kintzios.com> wrote:
> On Monday, 20 September 2021 14:56:46 BST Gerrit Kuehn wrote:
>
>> Well, this was the suggested way to go, see
>> https://www.gentoo.org/support/news-items/2021-05-05-python3-9.html
>>
>> But also when trying "emerge -1vUD @world" (be it with or without the
>> package.use settings), I get stuck in conflicts (mostly on perl and
>> setuptools). perl issues would probably resolve once I have EAPI8
>> support, i.e. get new portage.
>
> With a year old system you should question if reinstalling your
> system after a back up of configuration and data files would be a
> smarter approach.

In my experience, it will almost definitely be less work to
reinstall --- usually a _lot_ less work.

--
Grant
Re: Re: python, my nemesis [ In reply to ]
On Mon, 20 Sep 2021 17:19:55 -0000 (UTC)
Grant Edwards <grant.b.edwards@gmail.com> wrote:

> In my experience, it will almost definitely be less work to
> reinstall --- usually a _lot_ less work.

Well, after getting portage updated, everything else updated fine over
night unattendedly. In this case, this was definitely less work for me
than doing a reinstall.


cu
Gerrit