Mailing List Archive

Hi.

Nothing compares to Chromium (browser) in terms of compilation times. On
my system with 12 core threads it takes about 8 hours to compile - which
is 4 times longer than 10 years ago with 2 core threads ;)

Libreoffice takes a few hours, but less than half of chromium. Nothing
gets close to Chromium. But otherwise webkitgtk and qtwebengine are to
big ones - but still about a quarter of Chromium.

Kristian Poul Herkild

Den 11.09.2023 kl. 21.19 skrev Alan McKinnon:
> After my long time away from Gentoo, I thought perhaps some packages
> that always took ages to compile would have improved. I needed to change
> to ~amd64 anyway (dumb n00b mistake leaving it at amd64). So that's what
> I did and let emerge do it's thing.
>
> chromium has been building since 10:14, it's now 21:16 and still going
> so 9 hours at least on this machine to build a browser - almost as bad
> as openoffice at it's worst (regularly took 12 hours). Nodejs also took
> a while, but I didn't record time.
>
>
> What other packages have huge build times?
>
> --
> Alan McKinnon
> alan dot mckinnon at gmail dot com

On 2023-09-13, Kristian Poul Herkild <kristian@herkild.dk> wrote:

> Nothing compares to Chromium (browser) in terms of compilation times. On
> my system with 12 core threads it takes about 8 hours to compile - which
> is 4 times longer than 10 years ago with 2 core threads ;)

About a year ago I finally gave up building Chromium and switched to
www-client/google-chrome. It got to the point where it sometimes took
longer to build Chromium than it did for the next version to come out.

--
Grant

On Wed, 13 Sep 2023 21:20:20 -0000 (UTC), Grant Edwards wrote:

> About a year ago I finally gave up building Chromium and switched to
> www-client/google-chrome. It got to the point where it sometimes took
> longer to build Chromium than it did for the next version to come out.

That's why I run stable Chromium on an otherwise testing system.


--
Neil Bothwick

We all know what comes after 'X', said Tom, wisely.

On Wednesday, 13 September 2023 16:14:09 BST Michael wrote:

> I recall this being discussed in a previous thread, but if your CPU has 24
> threads and you've set:
>
> EMERGE_DEFAULT_OPTS="--jobs=4 --load-average=32
> MAKEOPTS="-j14"
>
> You will be asking emerge to run up to 4 x 14 = 56 threads, which could
> potentially eat up to 2G of RAM each, i.e. 112G of RAM. This will exhaust
> your 64G of RAM, not taking into account whatever else the OS will be trying
> to run at the time.

Yes, I understand that, but I've spent a lot of time watching, and
instrumenting, and in practice the load doesn't exceed about 33.

> The --load-average=32 is normally expected to be a floating point number

That stipulation has only appeared recently. I have tried adding a '.0' to the
number, and it's made no noticeaible difference. Besides, I could attempt
pedantry and declare that the set of all integers is a proper subset of all
real numbers. ;-)

> Of course, not all emerges use make and you may never or rarely emerge 4 x
> monster packages in parallel to need 2G of RAM for each thread at the same
> time.

I have actually had three or four big packages running together, but my
observation is that they don't pump the load up too far.

> If only we had at our disposal some AI algorithm to calculate dynamically
> each time we run emerge the optimal combo of parallel emerge jobs and
> number of make tasks, so as to achieve the highest total time saving Vs
> energy spent! Or just the highest total time saving. ;-)

Yes, that's what we need, all right.

> I haven't performed any meaningful comparisons to determine where the
> greatest gains are to be achieved. Parallel emerges of many small
> packages, or a large number of make jobs for big packages. The combination
> would change each time according to the individual packages waiting for an
> update. In my use case, instinctively feels more beneficial reducing the
> time I have to wait for huge packages like qtwebengine to finish, than
> accelerating the updates of half a dozen smaller packages.

That is the difficulty. I do often rebuild a new system, not trusting the
existing one any longer, and a lot of time is spent fiddling with four tiny
packages at a time in the early and middle stages, then benefitting from the
limit of 4 once the desktop packages begin.

> Therefore, as a rule I leave EMERGE_DEFAULT_OPTS unset. I set MAKEOPTS jobs
> to the number of CPU threads +1 and the load average at 95%, so I can
> continue using the PC without any noticeable latency. On PCs where RAM is
> constrained I reduce the MAKEOPTS in /etc/portage/ package.env for any large
> packages which are bound to start swapping and thrashing the disk.

Interesting. Do you mean 95% of the jobs figure? I'll do some more
experimenting.

Meanwhile, perhaps I'll run new builds in two stages: the first without any
desktop packages - I do have sets defined to enable that - and the second with
them. I can't do that with emerge -e though.

--
Regards,
Peter.

On Thursday, 14 September 2023 10:49:35 BST Peter Humphrey wrote:
> On Wednesday, 13 September 2023 16:14:09 BST Michael wrote:
> > I recall this being discussed in a previous thread, but if your CPU has 24
> > threads and you've set:
> >
> > EMERGE_DEFAULT_OPTS="--jobs=4 --load-average=32
> > MAKEOPTS="-j14"
> >
> > You will be asking emerge to run up to 4 x 14 = 56 threads, which could
> > potentially eat up to 2G of RAM each, i.e. 112G of RAM. This will exhaust
> > your 64G of RAM, not taking into account whatever else the OS will be
> > trying to run at the time.
>
> Yes, I understand that, but I've spent a lot of time watching, and
> instrumenting, and in practice the load doesn't exceed about 33.
>
> > The --load-average=32 is normally expected to be a floating point number
>
> That stipulation has only appeared recently. I have tried adding a '.0' to
> the number, and it's made no noticeaible difference. Besides, I could
> attempt pedantry and declare that the set of all integers is a proper
> subset of all real numbers. ;-)
>
> > Of course, not all emerges use make and you may never or rarely emerge 4 x
> > monster packages in parallel to need 2G of RAM for each thread at the same
> > time.
>
> I have actually had three or four big packages running together, but my
> observation is that they don't pump the load up too far.
>
> > If only we had at our disposal some AI algorithm to calculate dynamically
> > each time we run emerge the optimal combo of parallel emerge jobs and
> > number of make tasks, so as to achieve the highest total time saving Vs
> > energy spent! Or just the highest total time saving. ;-)
>
> Yes, that's what we need, all right.
>
> > I haven't performed any meaningful comparisons to determine where the
> > greatest gains are to be achieved. Parallel emerges of many small
> > packages, or a large number of make jobs for big packages. The
> > combination
> > would change each time according to the individual packages waiting for an
> > update. In my use case, instinctively feels more beneficial reducing the
> > time I have to wait for huge packages like qtwebengine to finish, than
> > accelerating the updates of half a dozen smaller packages.
>
> That is the difficulty. I do often rebuild a new system, not trusting the
> existing one any longer, and a lot of time is spent fiddling with four tiny
> packages at a time in the early and middle stages, then benefitting from the
> limit of 4 once the desktop packages begin.

I very rarely reinstall. So rarely, I can't remember the last time I did it.


> > Therefore, as a rule I leave EMERGE_DEFAULT_OPTS unset. I set MAKEOPTS
> > jobs to the number of CPU threads +1 and the load average at 95%, so I
> > can continue using the PC without any noticeable latency. On PCs where
> > RAM is constrained I reduce the MAKEOPTS in /etc/portage/ package.env for
> > any large packages which are bound to start swapping and thrashing the
> > disk.
> Interesting. Do you mean 95% of the jobs figure? I'll do some more
> experimenting.

Yes. Example:

A CPU with 4 cores and 8 threads is set like this:

MAKEOPTS="-j8 -l7.2"

This allows up to 8 make jobs at a time, with an average load being kept at
90% to leave some breathing space for day to day desktop work and minimise too
much/frequent swapping:

8 x 0.9 = 7.2

Looking at top shows 7 to 8 compiling jobs at a time. The PC in question has
16G of RAM. I've observed small packages rarely if ever reach 2G of RAM per
job. So, I set the above MAKEOPTS in make.conf, but bypass it for large
packages in /etc/portage/env/ by setting, e.g.:

/etc/portage/env/j4.conf

with MAKEOPTS="-j4" in it and specify this for each large package in /etc/
portage/package.env, e.g.:

www-client/qtwebengine -j4.conf
www-client/firefox -j4.conf
dev-lang/rust j4.conf
etc.


> Meanwhile, perhaps I'll run new builds in two stages: the first without any
> desktop packages - I do have sets defined to enable that - and the second
> with them. I can't do that with emerge -e though.

Unlike MAKEOPTS, the EMERGE_DEFAULT_OPTS variable cannot be set in /etc/
portage/package.env. Therefore, I leave the latter unset in make.conf and
specify it on the CLI when I'm about to update a lot of smaller packages and
remember to do it. ;-)