Advanced
Mailing List Archive

Mailing List Archive

  1. Home >
  2. Gentoo>
  3. User
Re: What is the best way forward? - Update 2 - SUCCESS! - CURRENT!!!
gtaylor at gentoo
Mar 8, 2021, 2:06 PM
Post #1 of 16 (966 views)
Permalink
On 2/25/21 5:31 PM, Grant Taylor wrote:
> 10 have git switch to the next day
> 20 emerge -aDUN @world
> 30 assess / deal with masked packages
> 40 goto 10
>
> It /looks/ like things are working.

*TL;DR*

DenverCoder9: DEAR PEOPLE FROM THE FUTURE ...

This method /does/ work. I have successfully brought the problem system
from ~1 year old to ~current (Gentoo portage repo < 24 hours old).

*Speed Bumps*

These were the four things that caused the biggest slow down in this
process.

1) Source packages / ebuilds no longer available.
- I found and downloaded files to DISTDIR.
- I copied some ebuilds from older versions of portage to my local
repo.
2) make.profile not using PORTDIR definition in make.conf.
- I ran into this while working on October ~ November '20 updates.
3) PYTHON_TARGETS & PYTHON_SINGLE_TARGET
- I ran into this after fixing #2.
- I had to add the following to pull Python 3.6 back in so that
things would work to add Python 3.7, before allowing the system to
remove Python 3.6 (again).
PYTHON_TARGETS="python2_7 python3_6 python3_7"
PYTHON_SINGLE_TARGET="python3_7"
4) Firefox & Thunderbird 68 disliking rust ? 1.48.
- I had to give up on retaining version 68 of Firefox and Thunderbird.
- The loss of some important extensions still really hurts.

*How*

The high level process that I used is a very close superset of what I
hypothesized.

10 have git switch to the next day
20 emerge -DUN @world
21 emerge --depclean --verbose n
22 emerge @preserved-rebuild
23 revdep-rebuild
30 assess / deal with output from steps 20-23
40 goto 10

Steps 21-23 added mid-stream to make comparison to previous message simpler.

All of these steps were in a function, `e3` (see attached file), which
relied on one variable, `d`, the count of how many days to go backwards
and set the date (`D`) to that everything should act on.

Aside: The next version of e3 would probably store `d` in a file and
subsequently re-load it from said file on each invocation. Thus
eliminating the reliance on the environment variable. I would probably
store this file in /var/tmp as /tmp and /dev/shm are cleared on boot.

After gaining enough trust in the overall process, I ended up running
the following while loop:

while e3; true; done

This allowed the system to stay busy emerging things up to the point
that something failed and needed attention.

*Setup*

I did a `git clone` of the Gentoo portage repo. Currently ~6 GB.

I then created the branches in the git repo with the following command
(from inside of the git repo directory):

for ((age=1; age<1024; age++)); do eval $(printf 'git log
--pretty=format:"%%H %%cd" --date=format:%%Y-%%m-%%d\ %%H:%%M:%%S
--after=%s --before=%s | fgrep -m1 %s' $(date +%Y-%m-%d -d "$(($age +
1)) days ago") $(date +%Y-%m-%d -d "$(($age - 1)) days ago") $(date
+%Y-%m-%d -d "$age days ago")) | read hash date time; time git checkout
-b $date $hash; done

Basically, this command starts at current; `stable`, and finds the first
(most recent) commit for a given date and creates a branch, and works
backwards for however many days configured; 1024 in the example.

*Miscellany*

I did `emerge -e @world` 3~5 times throughout the process just to make
sure that everything was consistent. I will do this once more tomorrow
after a full backup runs tonight.

I did end up removing a small list of packages that were blocking emerge
in one way or another. -- I decided that removing them to allow emerge
to complete on it's own accord was more expedient than fighting them at
the time. I will re-add them as necessary.

- net-firewall/nftables
- net-fs/ncpfs
- media-gfx/gimp
- dev-python/pycairo
- dev-python/fido2
- net-analyzer/scapy
- app-crypt/yubikey-manager

Some of the packages were subsequently pulled back in.

I did run into a bug with app-misc/pax-utils where I needed to add
"-seccomp" for the package to be able to move forward.

I also did the /usr/portage to /var/db/repos/gentoo et al. migration.

"repo" can be ambiguous when there talking about both "Gentoo portage
repo" and "git repo". Especially when the latter is managing the former.

The following packages take what seems like F O R E V E R to emerge:

- gcc
- rust
- Firefox
- Thunderbird

Link - xkcd - Wisdom of the Ancients (a.k.a. DenverCoder9)
- https://xkcd.com/979/

*Summary*

Yes, there are probably faster and / or more efficient processes to get
a Gentoo system that's ~1 year behind caught up to current. But I did
learn some things along the way. -- I tried to outline the toe
stubbers so others can avoid them.

Ultimately, I believe I have done in the last 11 days what would have
been done over the course of the last ~year. Even 11 days is longer
than necessary as I started with the while loop after getting to January
of this year. In hindsight, I believe I could have used the while loop
all along.

I hope that I have included enough details for others to be able to
reproduce this process if they need to.

Now it's time to repeat this process to bring the two companion systems
up to date.



--
Grant. . . .
unix || die

Attached Files:

  • text/plain e3 (0.75 KB)
Re: What is the best way forward? - Update 2 - SUCCESS! - CURRENT!!! [ In reply to ]
neil at digimed
Mar 8, 2021, 2:29 PM
Post #2 of 16 (966 views)
Permalink
On Mon, 8 Mar 2021 15:06:01 -0700, Grant Taylor wrote:

> The following packages take what seems like F O R E V E R to emerge:
>
> - gcc
> - rust
> - Firefox
> - Thunderbird

With hindsight, removing firefox, thunderbird and libreoffice and
replacing them with their -bin counterparts at the start of the process
would have saved much time. You could switch back to the source options
once the system is up to date.

How did you manage gcc upgrades, did you run gcc-config manually whenever
gcc was updated?

Do you feel it was worth the effort of updating for every day of the git
history? Would a larger increment have saved time, or did you think
minimising the number of issues to deal with after each emerge was more
important?

Anyway, glad it worked for you - it's more or less how I would have
approached it but never had to, so thanks for doing the legwork :)


--
Neil Bothwick

WinErr 013: Unexpected error - Huh ?
Re: What is the best way forward? - Update 2 - SUCCESS! - CURRENT!!! [ In reply to ]
gtaylor at gentoo
Mar 8, 2021, 2:44 PM
Post #3 of 16 (966 views)
Permalink
On 3/8/21 3:29 PM, Neil Bothwick wrote:
> With hindsight, removing firefox, thunderbird and libreoffice and
> replacing them with their -bin counterparts at the start of the
> process would have saved much time. You could switch back to the
> source options once the system is up to date.

You're probably correct.

However I don't think I would do that even if I had known / thought
about doing so. Partially because changing things was questionable at /
near the start and partially because this was about possibility, not
efficiency.

> How did you manage gcc upgrades, did you run gcc-config manually
> whenever gcc was updated?

Is "I ignored them and let emerge deal with it" count? I did see gcc
upgrades along the way.

I don't remember what it was at the start, probably 8.<something> or
9.<something>. I did see 9.3 somewhere along the way. gcc -v says that
10.2.0 is currently installed.

> Do you feel it was worth the effort of updating for every day of the
> git history?

I don't know if it was worth the effort or not. I initially did one day
at a time while testing the waters and going from theory to some
practical experience of the method.

Very quickly I used a different version of e (1 or 2) that took the date
as a parameter. My command line was calling e with the date derived
from the d variable and then decrementing the d variable after the e
function finished. I.e.

e $(date +%Y-%m-%d -d "$d days ago"); $((d=$d-1))

I would let that run, deal with any results, then hit the up arrow and
enter.

I just let that process continue for a while. Then at some point I
optimized it into e3 and ran that for a while. Then I optimized that
into the while e3; do true; done loop.

But I stuck with single day steps mostly from inertia. It was working.
So ... stick with it.

> Would a larger increment have saved time, or did you think minimising
> the number of issues to deal with after each emerge was more important?

Maybe. If anything, it would have saved the time for emerge to process
all of it's meta data. Much like an initial emerge vs an emerge
--resume. But again, this was about the viability of the process, not
the efficiency thereof.

I probably could have gone with a week at a time. I don't know if that
would have helped or not. I don't think I would go with more than a
week with a largely automated process.

I think one month increments probably would be pushing the envelope. I
feel like some of the Python changes were 2 or maybe 3 months apart. So
with two combined with how you landed, you might cross Python 3.6 w/o
3.7, to both 3.6 and 3.7, to w/o 3.6 and w/ 3.7 barrier. That probably
wouldn't be pretty.

> Anyway, glad it worked for you - it's more or less how I would have
> approached it but never had to, so thanks for doing the legwork :)

You're welcome.

Hence the DenverCoder9 comment, for people searching ~> reading the
mailing list archive in the future.



--
Grant. . . .
unix || die
Re: What is the best way forward? - Update 2 - SUCCESS! - CURRENT!!! [ In reply to ]
grant.b.edwards at gmail
Mar 8, 2021, 3:03 PM
Post #4 of 16 (966 views)
Permalink
On 2021-03-08, Grant Taylor <gtaylor@gentoo.tnetconsulting.net> wrote:
> On 2/25/21 5:31 PM, Grant Taylor wrote:
>> 10 have git switch to the next day
>> 20 emerge -aDUN @world
>> 30 assess / deal with masked packages
>> 40 goto 10
>>
>> It /looks/ like things are working.
>
> *TL;DR*
>
> DenverCoder9: DEAR PEOPLE FROM THE FUTURE ...
>
> This method /does/ work. I have successfully brought the problem system
> from ~1 year old to ~current (Gentoo portage repo < 24 hours old).

How do you feel it compares to just installing from scratch while
preserving whatever config and user data you care about? I've done
that quite a few times and it usually takes about 2-3 hours for the
initial install and then overnight to build a desktop environment (if
one is needed).

--
The other Grant
Re: What is the best way forward? - Update 2 - SUCCESS! - CURRENT!!! [ In reply to ]
neil at digimed
Mar 8, 2021, 3:16 PM
Post #5 of 16 (966 views)
Permalink
On Mon, 8 Mar 2021 15:44:38 -0700, Grant Taylor wrote:

> On 3/8/21 3:29 PM, Neil Bothwick wrote:
> > With hindsight, removing firefox, thunderbird and libreoffice and
> > replacing them with their -bin counterparts at the start of the
> > process would have saved much time. You could switch back to the
> > source options once the system is up to date.
>
> You're probably correct.
>
> However I don't think I would do that even if I had known / thought
> about doing so. Partially because changing things was questionable at
> / near the start and partially because this was about possibility, not
> efficiency.It would ve to be done

It would have to be done before the first update, when the repo was set
to a date just after the last update.

> > How did you manage gcc upgrades, did you run gcc-config manually
> > whenever gcc was updated?
>
> Is "I ignored them and let emerge deal with it" count? I did see gcc
> upgrades along the way.

You can rephrase that as "I left it at the default", which is an
acceptable answer :)

> I don't remember what it was at the start, probably 8.<something> or
> 9.<something>. I did see 9.3 somewhere along the way. gcc -v says
> that 10.2.0 is currently installed.

It means you probably spent a lot of time compile gcc versions only to
carry on using the old version, but as you said, this wasn't about
efficiency. You were going to emerge -e @world at the end anyway, which
would get everything built with the latest toolchain.

> > Do you feel it was worth the effort of updating for every day of the
> > git history?
>
> I don't know if it was worth the effort or not. I initially did one
> day at a time while testing the waters and going from theory to some
> practical experience of the method.
>
> Very quickly I used a different version of e (1 or 2) that took the
> date as a parameter. My command line was calling e with the date
> derived from the d variable and then decrementing the d variable after
> the e function finished. I.e.
>
> e $(date +%Y-%m-%d -d "$d days ago"); $((d=$d-1))
>
> I would let that run, deal with any results, then hit the up arrow and
> enter.
>
> I just let that process continue for a while. Then at some point I
> optimized it into e3 and ran that for a while. Then I optimized that
> into the while e3; do true; done loop.

Most of the effort for you was developing the procedure. All the real
effort was left to the computer.

> But I stuck with single day steps mostly from inertia. It was working.
> So ... stick with it.
>
> > Would a larger increment have saved time, or did you think minimising
> > the number of issues to deal with after each emerge was more
> > important?
>
> Maybe. If anything, it would have saved the time for emerge to process
> all of it's meta data. Much like an initial emerge vs an emerge
> --resume. But again, this was about the viability of the process, not
> the efficiency thereof.
>
> I probably could have gone with a week at a time. I don't know if that
> would have helped or not. I don't think I would go with more than a
> week with a largely automated process.

I was thinking of a week max.


--
Neil Bothwick

Distrust any enterprise that requires new clothes. - Henry David Thoreau
(1817-1862)
Re: Re: What is the best way forward? - Update 2 - SUCCESS! - CURRENT!!! [ In reply to ]
gtaylor at gentoo
Mar 8, 2021, 3:37 PM
Post #6 of 16 (966 views)
Permalink
On 3/8/21 4:03 PM, Grant Edwards wrote:
> How do you feel it compares to just installing from scratch while
> preserving whatever config and user data you care about? I've done
> that quite a few times and it usually takes about 2-3 hours for the
> initial install and then overnight to build a desktop environment
> (if one is needed).

I feel like installing from scratch misses a lot of things. Even
wholesale overwriting the new /etc with the old /etc is questionable.
You'd have to make sure that all the same software was installed.

Aside: I've spent too much time around other SAs that would ""recover a
down server by doing fresh installs in hours and then spending weeks to
get everything back to the way that it needed to be verses spending ~18
hours to restore from tape and have things work the way they were 24
hours prior. I also never cared for in place upgrades (installing over
top of itself) in the Windows world.

I feel *MUCH* /more/ comfortable with what I did than other solutions.
I trust that this is the same install with patches applied. I couldn't
and wouldn't say the same for an installation over the top or fresh
installation.

Don't get me wrong. I believe there are places for fresh installations.
They usually happen coordinate with new machines and / or new drives
for me.

After all, I effectively have the same thing that I would have if I had
done updates over the last year like they should have been done.



--
Grant. . . .
unix || die
Re: What is the best way forward? - Update 2 - SUCCESS! - CURRENT!!! [ In reply to ]
gtaylor at gentoo
Mar 8, 2021, 3:44 PM
Post #7 of 16 (966 views)
Permalink
On 3/8/21 4:16 PM, Neil Bothwick wrote:
> It would have to be done before the first update, when the repo was
> set to a date just after the last update.

Yes and no.

It really could have been done at any point along the way.

Also, with the git version of the portage repo, I could switch back to
the branch from any time I wanted to.

> You can rephrase that as "I left it at the default", which is an
> acceptable answer :)

*nod*

> It means you probably spent a lot of time compile gcc versions only
> to carry on using the old version, but as you said, this wasn't about
> efficiency.

Wouldn't the next execution of gcc, post Emerge & Installation use the
newly emerged binary?

If not next package in a given emerge run didn't use the new gcc, I
would fully expect that subsequent emerges would use the new gcc.

> You were going to emerge -e @world at the end anyway, which would
> get everything built with the latest toolchain.

Yes.

I have initiated a full system backup. I'll start an `emerge -e @world`
after that finishes.

I'll actually do the full suite:

1) emerge -e @world
2) emerge --depclean --verbose n
3) emerge @preserved-rebuild
4) revdep-rebuild

I expect that #3 should be a NoOp and just burn CPU cycles.

I don't know anything else that can be done to make a Gentoo box happier
(from a software standpoint).

> Most of the effort for you was developing the procedure. All the real
> effort was left to the computer.

Exactly!

Well, developing the method /and/ establishing trust therein.

> I was thinking of a week max.

I suspect that would be quite safe.



--
Grant. . . .
unix || die
Re: What is the best way forward? - Update 2 - SUCCESS! - CURRENT!!! [ In reply to ]
neil at digimed
Mar 8, 2021, 4:35 PM
Post #8 of 16 (966 views)
Permalink
On Mon, 8 Mar 2021 16:44:35 -0700, Grant Taylor wrote:

> > It means you probably spent a lot of time compile gcc versions only
> > to carry on using the old version, but as you said, this wasn't about
> > efficiency.
>
> Wouldn't the next execution of gcc, post Emerge & Installation use the
> newly emerged binary?
>
> If not next package in a given emerge run didn't use the new gcc, I
> would fully expect that subsequent emerges would use the new gcc.

Not if you went up a slot, then the old version would still continue to
be used until you ran gcc-config. However, if you were depcleaning at each
step, that would remove the previous slot and you would stay current.

I tend to keep old copies of gcc around until I'm sure things play nicely
with the new version:

% gcc-config -l
[1] x86_64-pc-linux-gnu-9.3.0
[2] x86_64-pc-linux-gnu-10.2.0 *


--
Neil Bothwick

Psychiatrists say that 1 of 4 people are mentally ill.
Check three friends. If they're OK, you're it.
Re: What is the best way forward? - Update 2 - SUCCESS! - CURRENT!!! [ In reply to ]
gtaylor at gentoo
Mar 8, 2021, 4:46 PM
Post #9 of 16 (966 views)
Permalink
On 3/8/21 5:35 PM, Neil Bothwick wrote:
> Not if you went up a slot, then the old version would still continue to
> be used until you ran gcc-config. However, if you were depcleaning at each
> step, that would remove the previous slot and you would stay current.

So my overall method, which included depclean, did allow subsequent runs
to use the new updated GCC.

Thank you for clarifying.

> I tend to keep old copies of gcc around until I'm sure things play nicely
> with the new version:
>
> % gcc-config -l
> [1] x86_64-pc-linux-gnu-9.3.0
> [2] x86_64-pc-linux-gnu-10.2.0 *

[1] aarch64-unknown-linux-gnu-9.3.0 * (cyan *)
[2] x86_64-pc-linux-gnu-10.2.0 * (green *)

The aarch64* came in as part of @openwrt-prerequisites. I should
probably remove that as I no longer need it.

Thank you for your input Neil.



--
Grant. . . .
unix || die
Re: What is the best way forward? - Update 2 - SUCCESS! - CURRENT!!! [ In reply to ]
antlists at youngman
Mar 8, 2021, 4:59 PM
Post #10 of 16 (966 views)
Permalink
On 08/03/2021 23:16, Neil Bothwick wrote:
>> I don't remember what it was at the start, probably 8.<something> or
>> 9.<something>. I did see 9.3 somewhere along the way. gcc -v says
>> that 10.2.0 is currently installed.

> It means you probably spent a lot of time compile gcc versions only to
> carry on using the old version, but as you said, this wasn't about
> efficiency. You were going to emerge -e @world at the end anyway, which
> would get everything built with the latest toolchain.
>
As I remember, you always had to use eselect to switch versions ... and
witness all the chaos with python at the moment ...

If you leave things "at the default", doesn't that screw you over when
python/kernel/gcc etc upgrade and a depclean deletes your original
default version? Or is that now fixed so you can't mess things up that way?

Cheers,
Wol
Re: Re: What is the best way forward? - Update 2 - SUCCESS! - CURRENT!!! [ In reply to ]
covici at ccs
Mar 8, 2021, 6:30 PM
Post #11 of 16 (966 views)
Permalink
On Mon, 08 Mar 2021 18:37:21 -0500,
Grant Taylor wrote:
>
> On 3/8/21 4:03 PM, Grant Edwards wrote:
> > How do you feel it compares to just installing from scratch
> > while preserving whatever config and user data you care about?
> > I've done that quite a few times and it usually takes about 2-3
> > hours for the initial install and then overnight to build a
> > desktop environment (if one is needed).
>
> I feel like installing from scratch misses a lot of things. Even
> wholesale overwriting the new /etc with the old /etc is
> questionable. You'd have to make sure that all the same software
> was installed.
>
> Aside: I've spent too much time around other SAs that would
> ""recover a down server by doing fresh installs in hours and then
> spending weeks to get everything back to the way that it needed
> to be verses spending ~18 hours to restore from tape and have
> things work the way they were 24 hours prior. I also never cared
> for in place upgrades (installing over top of itself) in the
> Windows world.
>
> I feel *MUCH* /more/ comfortable with what I did than other
> solutions. I trust that this is the same install with patches
> applied. I couldn't and wouldn't say the same for an
> installation over the top or fresh installation.
>
> Don't get me wrong. I believe there are places for fresh
> installations. They usually happen coordinate with new machines
> and / or new drives for me.
>
> After all, I effectively have the same thing that I would have if
> I had done updates over the last year like they should have been
> done.
>
>

hmmm, I had to do a sort of fresh install, I did it on my running
system and I did it because portage and other stuff was messed up in
some way I could not fix, so I did the install, copied parts of my
/etc, emerged part of my world file -- that needed cleaning up too --
and kept doing this till I was done and then copied from my chroot to
my main system. I don't necessarily recomend this, but it did get
things cleaned up and working again. At least I didn't have to change
profiles and gcc versions several times. I guess different situations
require different methods.



--
Your life is like a penny. You're going to lose it. The question is:
How do
you spend it?

John Covici wb2una
covici@ccs.covici.com
Re: Re: What is the best way forward? - Update 2 - SUCCESS! - CURRENT!!! [ In reply to ]
gtaylor at gentoo
Mar 8, 2021, 9:00 PM
Post #12 of 16 (966 views)
Permalink
On 3/8/21 7:30 PM, John Covici wrote:
> At least I didn't have to change profiles and gcc versions several
> times.

I didn't /change/ the profile. As in it was 17.0 when I started and
still is 17.0.

I did have to update the make.profile link to point to the same profile
in the alternate portage directory.

This wouldn't have been an issue if I just re-used the same portage
directory.

> I guess different situations require different methods.

Indeed.



--
Grant. . . .
unix || die
Re: What is the best way forward? - Update 2 - SUCCESS! - CURRENT!!! [ In reply to ]
neil at digimed
Mar 9, 2021, 7:13 AM
Post #13 of 16 (966 views)
Permalink
On Tue, 9 Mar 2021 00:59:29 +0000, antlists wrote:

> > It means you probably spent a lot of time compile gcc versions only to
> > carry on using the old version, but as you said, this wasn't about
> > efficiency. You were going to emerge -e @world at the end anyway,
> > which would get everything built with the latest toolchain.
> >
> As I remember, you always had to use eselect to switch versions ... and
> witness all the chaos with python at the moment ...
>
> If you leave things "at the default", doesn't that screw you over when
> python/kernel/gcc etc upgrade and a depclean deletes your original
> default version? Or is that now fixed so you can't mess things up that
> way?

You can't because eselect always shows a fallback option, which will be
used should you unmerge the selected version.


--
Neil Bothwick

IMPORTANT: The entire physical universe, including this message, may
one day collapse back into an infinitesimally small space. Should
another universe subsequently re-emerge, the existence of this message
in that universe cannot be guaranteed.
Re: What is the best way forward? - Update 2 - SUCCESS! - CURRENT!!! [ In reply to ]
smurfd at gmail
Mar 9, 2021, 8:26 AM
Post #14 of 16 (966 views)
Permalink
Massive congrats :)
Great when things work out, after doing a ton of work to get there.

br smurfd

On 2021-03-08 23:06, Grant Taylor wrote:
> On 2/25/21 5:31 PM, Grant Taylor wrote:
>> 10 have git switch to the next day
>> 20 emerge -aDUN @world
>> 30 assess / deal with masked packages
>> 40 goto 10
>>
>> It /looks/ like things are working.
>
> *TL;DR*
>
> DenverCoder9:  DEAR PEOPLE FROM THE FUTURE ...
>
> This method /does/ work.  I have successfully brought the problem
> system from ~1 year old to ~current (Gentoo portage repo < 24 hours old).
>
> *Speed Bumps*
>
> These were the four things that caused the biggest slow down in this
> process.
>
> 1)  Source packages / ebuilds no longer available.
>      - I found and downloaded files to DISTDIR.
>      - I copied some ebuilds from older versions of portage to my
> local repo.
> 2)  make.profile not using PORTDIR definition in make.conf.
>      - I ran into this while working on October ~ November '20 updates.
> 3)  PYTHON_TARGETS & PYTHON_SINGLE_TARGET
>      - I ran into this after fixing #2.
>      - I had to add the following to pull Python 3.6 back in so that
> things would work to add Python 3.7, before allowing the system to
> remove Python 3.6 (again).
>           PYTHON_TARGETS="python2_7 python3_6 python3_7"
>           PYTHON_SINGLE_TARGET="python3_7"
> 4)  Firefox & Thunderbird 68 disliking rust ? 1.48.
>      - I had to give up on retaining version 68 of Firefox and
> Thunderbird.
>         - The loss of some important extensions still really hurts.
>
> *How*
>
> The high level process that I used is a very close superset of what I
> hypothesized.
>
> 10 have git switch to the next day
> 20 emerge -DUN @world
> 21 emerge --depclean --verbose n
> 22 emerge @preserved-rebuild
> 23 revdep-rebuild
> 30 assess / deal with output from steps 20-23
> 40 goto 10
>
> Steps 21-23 added mid-stream to make comparison to previous message
> simpler.
>
> All of these steps were in a function, `e3` (see attached file), which
> relied on one variable, `d`, the count of how many days to go
> backwards and set the date (`D`) to that everything should act on.
>
> Aside:  The next version of e3 would probably store `d` in a file and
> subsequently re-load it from said file on each invocation. Thus
> eliminating the reliance on the environment variable.  I would
> probably store this file in /var/tmp as /tmp and /dev/shm are cleared
> on boot.
>
> After gaining enough trust in the overall process, I ended up running
> the following while loop:
>
>    while e3; true; done
>
> This allowed the system to stay busy emerging things up to the point
> that something failed and needed attention.
>
> *Setup*
>
> I did a `git clone` of the Gentoo portage repo.  Currently ~6 GB.
>
> I then created the branches in the git repo with the following command
> (from inside of the git repo directory):
>
>    for ((age=1; age<1024; age++)); do eval $(printf 'git log
> --pretty=format:"%%H %%cd" --date=format:%%Y-%%m-%%d\ %%H:%%M:%%S
> --after=%s --before=%s | fgrep -m1 %s' $(date +%Y-%m-%d -d "$(($age +
> 1)) days ago") $(date +%Y-%m-%d -d "$(($age - 1)) days ago") $(date
> +%Y-%m-%d -d "$age days ago")) | read hash date time; time git
> checkout -b $date $hash; done
>
> Basically, this command starts at current; `stable`, and finds the
> first (most recent) commit for a given date and creates a branch, and
> works backwards for however many days configured; 1024 in the example.
>
> *Miscellany*
>
> I did `emerge -e @world` 3~5 times throughout the process just to make
> sure that everything was consistent.  I will do this once more
> tomorrow after a full backup runs tonight.
>
> I did end up removing a small list of packages that were blocking
> emerge in one way or another.  --  I decided that removing them to
> allow emerge to complete on it's own accord was more expedient than
> fighting them at the time.  I will re-add them as necessary.
>
>  - net-firewall/nftables
>  - net-fs/ncpfs
>  - media-gfx/gimp
>  - dev-python/pycairo
>  - dev-python/fido2
>  - net-analyzer/scapy
>  - app-crypt/yubikey-manager
>
> Some of the packages were subsequently pulled back in.
>
> I did run into a bug with app-misc/pax-utils where I needed to add
> "-seccomp" for the package to be able to move forward.
>
> I also did the /usr/portage to /var/db/repos/gentoo et al. migration.
>
> "repo" can be ambiguous when there talking about both "Gentoo portage
> repo" and "git repo".  Especially when the latter is managing the former.
>
> The following packages take what seems like  F O R E V E R  to emerge:
>
>  - gcc
>  - rust
>  - Firefox
>  - Thunderbird
>
> Link - xkcd - Wisdom of the Ancients (a.k.a. DenverCoder9)
>  - https://xkcd.com/979/
>
> *Summary*
>
> Yes, there are probably faster and / or more efficient processes to
> get a Gentoo system that's ~1 year behind caught up to current.  But I
> did learn some things along the way.  --  I tried to outline the toe
> stubbers so others can avoid them.
>
> Ultimately, I believe I have done in the last 11 days what would have
> been done over the course of the last ~year.  Even 11 days is longer
> than necessary as I started with the while loop after getting to
> January of this year.  In hindsight, I believe I could have used the
> while loop all along.
>
> I hope that I have included enough details for others to be able to
> reproduce this process if they need to.
>
> Now it's time to repeat this process to bring the two companion
> systems up to date.
>
>
>
Re: What is the best way forward? - Update 2 - SUCCESS! - CURRENT!!! [ In reply to ]
neil at digimed
Mar 9, 2021, 9:34 AM
Post #15 of 16 (966 views)
Permalink
On Tue, 9 Mar 2021 17:26:41 +0100, smurfd wrote:

> Massive congrats :)
> Great when things work out, after doing a ton of work to get there.

As the great Douglas Adams once said

"I am rarely happier than when spending an entire day programming my
computer to perform automatically a task that would otherwise take me a
good ten seconds to do by hand.”"

> br smurfd
>
> On 2021-03-08 23:06, Grant Taylor wrote:
> > On 2/25/21 5:31 PM, Grant Taylor wrote:
> >> 10 have git switch to the next day
> >> 20 emerge -aDUN @world
> >> 30 assess / deal with masked packages
> >> 40 goto 10
> >>
> >> It /looks/ like things are working.
> >
> > *TL;DR*
> >
> > DenverCoder9:  DEAR PEOPLE FROM THE FUTURE ...
> >
> > This method /does/ work.  I have successfully brought the problem
> > system from ~1 year old to ~current (Gentoo portage repo < 24 hours
> > old).
> >
> > *Speed Bumps*
> >
> > These were the four things that caused the biggest slow down in this
> > process.
> >
> > 1)  Source packages / ebuilds no longer available.
> >      - I found and downloaded files to DISTDIR.
> >      - I copied some ebuilds from older versions of portage to my
> > local repo.
> > 2)  make.profile not using PORTDIR definition in make.conf.
> >      - I ran into this while working on October ~ November '20
> > updates. 3)  PYTHON_TARGETS & PYTHON_SINGLE_TARGET
> >      - I ran into this after fixing #2.
> >      - I had to add the following to pull Python 3.6 back in so that
> > things would work to add Python 3.7, before allowing the system to
> > remove Python 3.6 (again).
> >           PYTHON_TARGETS="python2_7 python3_6 python3_7"
> >           PYTHON_SINGLE_TARGET="python3_7"
> > 4)  Firefox & Thunderbird 68 disliking rust ? 1.48.
> >      - I had to give up on retaining version 68 of Firefox and
> > Thunderbird.
> >         - The loss of some important extensions still really hurts.
> >
> > *How*
> >
> > The high level process that I used is a very close superset of what I
> > hypothesized.
> >
> > 10 have git switch to the next day
> > 20 emerge -DUN @world
> > 21 emerge --depclean --verbose n
> > 22 emerge @preserved-rebuild
> > 23 revdep-rebuild
> > 30 assess / deal with output from steps 20-23
> > 40 goto 10
> >
> > Steps 21-23 added mid-stream to make comparison to previous message
> > simpler.
> >
> > All of these steps were in a function, `e3` (see attached file),
> > which relied on one variable, `d`, the count of how many days to go
> > backwards and set the date (`D`) to that everything should act on.
> >
> > Aside:  The next version of e3 would probably store `d` in a file and
> > subsequently re-load it from said file on each invocation. Thus
> > eliminating the reliance on the environment variable.  I would
> > probably store this file in /var/tmp as /tmp and /dev/shm are cleared
> > on boot.
> >
> > After gaining enough trust in the overall process, I ended up running
> > the following while loop:
> >
> >    while e3; true; done
> >
> > This allowed the system to stay busy emerging things up to the point
> > that something failed and needed attention.
> >
> > *Setup*
> >
> > I did a `git clone` of the Gentoo portage repo.  Currently ~6 GB.
> >
> > I then created the branches in the git repo with the following
> > command (from inside of the git repo directory):
> >
> >    for ((age=1; age<1024; age++)); do eval $(printf 'git log
> > --pretty=format:"%%H %%cd" --date=format:%%Y-%%m-%%d\ %%H:%%M:%%S
> > --after=%s --before=%s | fgrep -m1 %s' $(date +%Y-%m-%d -d "$(($age +
> > 1)) days ago") $(date +%Y-%m-%d -d "$(($age - 1)) days ago") $(date
> > +%Y-%m-%d -d "$age days ago")) | read hash date time; time git
> > checkout -b $date $hash; done
> >
> > Basically, this command starts at current; `stable`, and finds the
> > first (most recent) commit for a given date and creates a branch, and
> > works backwards for however many days configured; 1024 in the example.
> >
> > *Miscellany*
> >
> > I did `emerge -e @world` 3~5 times throughout the process just to
> > make sure that everything was consistent.  I will do this once more
> > tomorrow after a full backup runs tonight.
> >
> > I did end up removing a small list of packages that were blocking
> > emerge in one way or another.  --  I decided that removing them to
> > allow emerge to complete on it's own accord was more expedient than
> > fighting them at the time.  I will re-add them as necessary.
> >
> >  - net-firewall/nftables
> >  - net-fs/ncpfs
> >  - media-gfx/gimp
> >  - dev-python/pycairo
> >  - dev-python/fido2
> >  - net-analyzer/scapy
> >  - app-crypt/yubikey-manager
> >
> > Some of the packages were subsequently pulled back in.
> >
> > I did run into a bug with app-misc/pax-utils where I needed to add
> > "-seccomp" for the package to be able to move forward.
> >
> > I also did the /usr/portage to /var/db/repos/gentoo et al. migration.
> >
> > "repo" can be ambiguous when there talking about both "Gentoo portage
> > repo" and "git repo".  Especially when the latter is managing the
> > former.
> >
> > The following packages take what seems like  F O R E V E R  to emerge:
> >
> >  - gcc
> >  - rust
> >  - Firefox
> >  - Thunderbird
> >
> > Link - xkcd - Wisdom of the Ancients (a.k.a. DenverCoder9)
> >  - https://xkcd.com/979/
> >
> > *Summary*
> >
> > Yes, there are probably faster and / or more efficient processes to
> > get a Gentoo system that's ~1 year behind caught up to current.  But
> > I did learn some things along the way.  --  I tried to outline the
> > toe stubbers so others can avoid them.
> >
> > Ultimately, I believe I have done in the last 11 days what would have
> > been done over the course of the last ~year.  Even 11 days is longer
> > than necessary as I started with the while loop after getting to
> > January of this year.  In hindsight, I believe I could have used the
> > while loop all along.
> >
> > I hope that I have included enough details for others to be able to
> > reproduce this process if they need to.
> >
> > Now it's time to repeat this process to bring the two companion
> > systems up to date.
> >
> >
> >
>
>




--
Neil Bothwick

Teamwork is essential; it gives the enemy other people to shoot at.
Re: What is the best way forward? - Update 2 - SUCCESS! - CURRENT!!! [ In reply to ]
gtaylor at gentoo
Mar 10, 2021, 8:09 PM
Post #16 of 16 (966 views)
Permalink
On 3/8/21 5:59 PM, antlists wrote:
> As I remember, you always had to use eselect to switch versions ... and
> witness all the chaos with python at the moment ...

I don't know.

> If you leave things "at the default", doesn't that screw you over when
> python/kernel/gcc etc upgrade and a depclean deletes your original
> default version? Or is that now fixed so you can't mess things up that way?

I've not changed the kernel yet. I didn't knowingly have any problems
with Python changes /related/ /to/ gcc.

My Python problem was that my make.profile was pointing to the old
portage directory that was still back at last March while emerge was
using a newer incrementally updating version of portage. -- I consider
this to be my fault.



--
Grant. . . .
unix || die

©2024 GT.net. A Gossamer Threads company.
5th Floor, 455 Granville St., Vancouver, BC V6C 1T1, Canada | Legal