Mailing List Archive: how to prevent ebuild from checking for available space

how to prevent ebuild from checking for available space

Mar 22, 2024, 9:57 AM

Post #1 of 7 (142 views)

In this case, the offending package is dev-lang/rust, but this has
happened to me previously with other packages that require a lot of
space and time to build.

The build fails for any reason. Since it has already progressed over
half way through the build, I would like to continue the build instead
of starting from scratch. I fix whatever caused the issue. In this
particular case, it failed for out of space, so I added another G to the
ramdisk mounted at /var/tmp/portageg. I issue "ebuild
path/to/package.ebuild compile" but it fails immediately because there
isn't enough free space. The problem is that it's checking for enough
free space to do the entire build, not just to continue from where it
left off.

I've done some searching, and found one old forum post which suggests to
just temporarily remove the space check from the ebuild.

In this case, that seems to be line 236 of
/usr/portage/dev-lang/rust/rust-1.75.0-r1.ebuild, which is
"CHECKREQS_DISK_BUILD=${M}M check-reqs_pkg_${EBUILD_PHASE}" the last
line withing pre_build_checks(). However, if I comment out that line,
and run "ebuild path/to/ebuild manifest" another attempt to compile
still gives the same error about not enough space.

Am I commenting the wrong line? Have I missed something about where
this check is atually done? Is it actually possible to do what I'm
trying to do?

Thanks for any pointers or suggestions.

Jack

Re: how to prevent ebuild from checking for available space [ In reply to ]

neil at digimed

Mar 22, 2024, 1:22 PM

Post #2 of 7 (140 views)

Permalink

On Fri, 22 Mar 2024 12:57:34 -0400, Jack wrote:

> In this case, the offending package is dev-lang/rust, but this has
> happened to me previously with other packages that require a lot of
> space and time to build.
>
> The build fails for any reason. Since it has already progressed over
> half way through the build, I would like to continue the build instead
> of starting from scratch. I fix whatever caused the issue. In this
> particular case, it failed for out of space, so I added another G to
> the ramdisk mounted at /var/tmp/portageg. I issue "ebuild
> path/to/package.ebuild compile" but it fails immediately because there
> isn't enough free space. The problem is that it's checking for enough
> free space to do the entire build, not just to continue from where it
> left off.

Why not add more to the ramdisk, assuming it is a tmpfs. If it needs more
than your physical memory, it will use swap, but that won't happen
because you only need the extra space.

I generally avoid this problem by having large packages use a
PORTAGE_TMPDIR on a spinning disk. It may be slightly slower, but that
doesn't really matter on a long compile, not nearly as much as having to
restart it!

>
> I've done some searching, and found one old forum post which suggests
> to just temporarily remove the space check from the ebuild.
>
> In this case, that seems to be line 236 of
> /usr/portage/dev-lang/rust/rust-1.75.0-r1.ebuild, which is
> "CHECKREQS_DISK_BUILD=${M}M check-reqs_pkg_${EBUILD_PHASE}" the last
> line withing pre_build_checks(). However, if I comment out that line,
> and run "ebuild path/to/ebuild manifest" another attempt to compile
> still gives the same error about not enough space.

Have you tried setting M to a smaller value immediately before that line?

--
Neil Bothwick

Religious error: (A)tone, (R)epent, (I)mmolate?

Re: how to prevent ebuild from checking for available space [ In reply to ]

ostroffjh at users

Mar 22, 2024, 1:32 PM

Post #3 of 7 (140 views)

Permalink

On 2024.03.22 16:22, Neil Bothwick wrote:
> On Fri, 22 Mar 2024 12:57:34 -0400, Jack wrote:
>
> > In this case, the offending package is dev-lang/rust, but this has
> > happened to me previously with other packages that require a lot of
> > space and time to build.
> >
> > The build fails for any reason.? Since it has already progressed
> over
> > half way through the build, I would like to continue the build
> instead
> > of starting from scratch.? I fix whatever caused the issue.? In this
> > particular case, it failed for out of space, so I added another G to
> > the ramdisk mounted at /var/tmp/portageg.? I issue "ebuild
> > path/to/package.ebuild compile" but it fails immediately because
> there
> > isn't enough free space.? The problem is that it's checking for
> enough
> > free space to do the entire build, not just to continue from where
> it
> > left off.
>
> Why not add more to the ramdisk, assuming it is a tmpfs. If it needs
> more
> than your physical memory, it will use swap, but that won't happen
> because you only need the extra space.
That's actually what I did. The problem is not how to get enough
space, it's how to resume the emerge instead of starting over, once I
have added the space.

It was initially set to 14G (out of 32G RAM) and I added 2G. I suppose
I can add another 14G, but that would only leave 4G for the system
itself. I'm not sure how well that would work, but I suppose it's
worth a try. The worst that happens is I reboot and start the emerge
from scratch.
>
> I generally avoid this problem by having large packages use a
> PORTAGE_TMPDIR on a spinning disk. It may be slightly slower, but that
> doesn't really matter on a long compile, not nearly as much as having
> to
> restart it!
>
> >
> > I've done some searching, and found one old forum post which
> suggests
> > to just temporarily remove the space check from the ebuild.
> >
> > In this case, that seems to be line 236 of
> > /usr/portage/dev-lang/rust/rust-1.75.0-r1.ebuild, which is
> > "CHECKREQS_DISK_BUILD=${M}M check-reqs_pkg_${EBUILD_PHASE}" the last
> > line withing pre_build_checks().? However, if I comment out that
> line,
> > and run "ebuild path/to/ebuild manifest" another attempt to compile
> > still gives the same error about not enough space.
>
> Have you tried setting M to a smaller value immediately before that
> line?
I tried that first, and when that didn't work, I tried commenting out
the line I thought was calling the check. I'll try again, to be sure
it was a reasonable value for M.
>
>
> --
> Neil Bothwick
>
> Religious error: (A)tone, (R)epent, (I)mmolate?
>

Re: how to prevent ebuild from checking for available space [ In reply to ]

ostroffjh at users

Mar 22, 2024, 5:11 PM

Post #4 of 7 (139 views)

Permalink

On 2024.03.22 16:22, Neil Bothwick wrote:
> On Fri, 22 Mar 2024 12:57:34 -0400, Jack wrote:
>
> > In this case, the offending package is dev-lang/rust, but this has
> > happened to me previously with other packages that require a lot of
> > space and time to build.
> >
> > The build fails for any reason.? Since it has already progressed
> over
> > half way through the build, I would like to continue the build
> instead
> > of starting from scratch.? I fix whatever caused the issue.? In this
> > particular case, it failed for out of space, so I added another G to
> > the ramdisk mounted at /var/tmp/portageg.? I issue "ebuild
> > path/to/package.ebuild compile" but it fails immediately because
> there
> > isn't enough free space.? The problem is that it's checking for
> enough
> > free space to do the entire build, not just to continue from where
> it
> > left off.
>
> Why not add more to the ramdisk, assuming it is a tmpfs. If it needs
> more
> than your physical memory, it will use swap, but that won't happen
> because you only need the extra space.
>
> I generally avoid this problem by having large packages use a
> PORTAGE_TMPDIR on a spinning disk. It may be slightly slower, but that
> doesn't really matter on a long compile, not nearly as much as having
> to
> restart it!
>
> >
> > I've done some searching, and found one old forum post which
> suggests
> > to just temporarily remove the space check from the ebuild.
> >
> > In this case, that seems to be line 236 of
> > /usr/portage/dev-lang/rust/rust-1.75.0-r1.ebuild, which is
> > "CHECKREQS_DISK_BUILD=${M}M check-reqs_pkg_${EBUILD_PHASE}" the last
> > line withing pre_build_checks().? However, if I comment out that
> line,
> > and run "ebuild path/to/ebuild manifest" another attempt to compile
> > still gives the same error about not enough space.
>
> Have you tried setting M to a smaller value immediately before that
> line?
It seems the problem is that the enviroment file in the temp dir of the
build area is sourced when you run ebuild/emerge. (It's among the
first output when you run ebuild.) Since that file was created based
on the state of the ebuild when the first emerge/ebuild was run,
editing the ebuild file doesn't affect it. I ended up finding a place
to unset CHECKREQS_FAILED, and the compile is now continuing.

Re: how to prevent ebuild from checking for available space [ In reply to ]

neil at digimed

Mar 23, 2024, 2:20 AM

Post #5 of 7 (132 views)

Permalink

On Fri, 22 Mar 2024 16:32:28 -0400, Jack wrote:

> > Why not add more to the ramdisk, assuming it is a tmpfs. If it needs
> > more
> > than your physical memory, it will use swap, but that won't happen
> > because you only need the extra space.
> That's actually what I did. The problem is not how to get enough
> space, it's how to resume the emerge instead of starting over, once I
> have added the space.
>
> It was initially set to 14G (out of 32G RAM) and I added 2G. I suppose
> I can add another 14G, but that would only leave 4G for the system
> itself. I'm not sure how well that would work, but I suppose it's
> worth a try.

tmpfs only uses the space it needs, so it would appear to the ebuild that
there is plenty of space, but it would only use and extra gig or two of
your RAM.

For me, avoiding tmpfs for big ebuilds is the least hassle, using
package.env.

% cat /etc/portage/package.env/chromium
www-client/chromium disk-tmpdir.conf ...

% cat /etc/portage/env/disk-tmpdir.conf
PORTAGE_TMPDIR="/mnt/scratch"

--
Neil Bothwick

Linux like wigwam. No windows, no gates, Apache inside.

Re: how to prevent ebuild from checking for available space [ In reply to ]

bog at khumba

Mar 23, 2024, 6:04 PM

Post #6 of 7 (122 views)

Permalink

On Fri, 22 Mar 2024 20:11:57 -0400
Jack <ostroffjh@users.sourceforge.net> wrote:

> It seems the problem is that the enviroment file in the temp dir of
> the build area is sourced when you run ebuild/emerge. (It's among
> the first output when you run ebuild.) Since that file was created
> based on the state of the ebuild when the first emerge/ebuild was
> run, editing the ebuild file doesn't affect it. I ended up finding a
> place to unset CHECKREQS_FAILED, and the compile is now continuing.

According to check-reqs.eclass, you can also bypass the check by
exporting CHECKREQS_DONOTHING=1. I assume this would work for a
resumed build too.

Cheers,
Bryan

Re: how to prevent ebuild from checking for available space [ In reply to ]

antlists at youngman

Mar 25, 2024, 4:32 AM

Post #7 of 7 (96 views)

Permalink

On 23/03/2024 09:20, Neil Bothwick wrote:
> tmpfs only uses the space it needs, so it would appear to the ebuild that
> there is plenty of space, but it would only use and extra gig or two of
> your RAM.
>
> For me, avoiding tmpfs for big ebuilds is the least hassle, using
> package.env.

I've yet to set my current system up with a tmpfs, but that's my
preferred setup.

Iirc, my current mobo takes 4x16GB chips, so "double ram as swap" makes
128GB - I put that on every disk (I have 3 or 4), so I've probably got
not far short of 1/2TB swap!

So nearly all of my build spaces, I would typically have a ramdisk of
maybe 128GB.

As previously mentioned, it only uses what space it needs, so I guess
with 32GB of real RAM, with most stuff my system wouldn't touch swap
even with huge ramdisks being declared.

Cheers,
Wol