Mailing List Archive

On Sat, 2024-04-06 at 17:06 +0200, Andreas K. Huettel wrote:
> Hi all,
>
> so here's a small update on the state of the 23.0 profiles:
>

Why was this silently added to make.defaults for all 23.0 profiles?

> # This just makes sense nowadays, if only for distfiles...
> USE="lzma zstd"

It doesn't help with distfiles in any way, wasn't discussed here,
doesn't belong there, and creates a mess on systems where they should
be disabled. Use per-package defaults if they're important for certain
packages.

On 06/04/2024 17.06, Andreas K. Huettel wrote:
> Hi all,
>
> so here's a small update on the state of the 23.0 profiles:

Thanks for the update and the work on the 23.0 profiles. :)


> Most 17.x profiles have been downgraded to "exp".

I could imagine there is a reason to downgrade those back to 'exp',
could you elaborate a bit on that?

Isn't it bit strange that a 'stable' profiles gets downgraded back to
'exp'? Then again, I am not sure about the implications of this nor
about the rationale behind it.

However, I also notice that there is a outstanding PR that reverts that
[1]. Maybe we should introduce a new state 'oldstable' or so?

- Flow


1: https://github.com/gentoo/gentoo/pull/35871

> Thanks for the update and the work on the 23.0 profiles. :)
>> Most 17.x profiles have been downgraded to "exp".
> I could imagine there is a reason to downgrade those back to 'exp',
> could you elaborate a bit on that?
>
> Isn't it bit strange that a 'stable' profiles gets downgraded back to
> 'exp'? Then again, I am not sure about the implications of this nor
> about the rationale behind it.
>
> However, I also notice that there is a outstanding PR that reverts
> that [1]. Maybe we should introduce a new state 'oldstable' or so?

I see no way of migrating to 23.0 profile because of not-recompilable
packages that are installed (over 4 years) which block --emptytree,
and do not wish to be forced to migrate to merged-usr on an openrc box
without a compelling need (on principle).

Will patching back the 17.0 profile files into the portage tree if and
when they are removed work?

Are there any options at all for this situation (like freezing the the
last supported tree protecting it from emerge-syncs, and using an
overlay for further updates?)

> > Most 17.x profiles have been downgraded to "exp".
>
> I could imagine there is a reason to downgrade those back to 'exp',
> could you elaborate a bit on that?
>
> Isn't it bit strange that a 'stable' profiles gets downgraded back to
> 'exp'? Then again, I am not sure about the implications of this nor
> about the rationale behind it.

Mostly so the load on the CI does not suddenly double.

There's no real reason why we can't keep a few of the profiles stable.
Also not much reason to do that though...

>
> However, I also notice that there is a outstanding PR that reverts that
> [1]. Maybe we should introduce a new state 'oldstable' or so?
>
> - Flow
>
>
> 1: https://github.com/gentoo/gentoo/pull/35871
>


--
Andreas K. H?ttel
dilfridge@gentoo.org
Gentoo Linux developer
(council, toolchain, base-system, perl, libreoffice)

Am Sonntag, 7. April 2024, 04:03:01 CEST schrieb Michael Orlitzky:
> On Sat, 2024-04-06 at 17:06 +0200, Andreas K. Huettel wrote:
> > Hi all,
> >
> > so here's a small update on the state of the 23.0 profiles:
> >
>
> Why was this silently added to make.defaults for all 23.0 profiles?
>
> > # This just makes sense nowadays, if only for distfiles...
> > USE="lzma zstd"

Uhh, I dont really remember, I think some Chinese-sounding guy asked
me for it... (j/k)

Jokes aside, we did have bz2 in the default useflags for ages, and
at the time this made a lot of sense since xz/lzma and zstd were
steadily becoming the most prevalent compression algorithms.

And for anyone interested in the timeline, this was one of the first
additions.

commit 99a7cb9e0b1728ca75242ddfee6357dc008bd1cd
Author: Andreas K. H?ttel <dilfridge@gentoo.org>
AuthorDate: Sun Nov 13 19:26:40 2022 +0100
Commit: Andreas K. H?ttel <dilfridge@gentoo.org>
CommitDate: Sun Nov 13 19:27:36 2022 +0100

profiles: Set USE="xz zstd" in 23.0


--
Andreas K. H?ttel
dilfridge@gentoo.org
Gentoo Linux developer
(council, toolchain, base-system, perl, libreoffice)

On Sun, 2024-04-07 at 14:35 +0200, Andreas K. Huettel wrote:
>
> Uhh, I dont really remember, I think some Chinese-sounding guy asked
> me for it... (j/k)

It is remarkably bad timing. How it looks: Gentoo's response to the xz
incident is to have me rebuild my entire system with everything that
could possibly be linked to liblzma, linked to liblzma. Even on the
hardened profiles, and with no easy way to prevent it.



> Jokes aside, we did have bz2 in the default useflags for ages, and
> at the time this made a lot of sense since xz/lzma and zstd were
> steadily becoming the most prevalent compression algorithms.

The flags that predate IUSE defaults can of course be forgiven.

What is improved by having these flags in every profile, vs only (say)
the desktop profiles, or in IUSE defaults?

Here's what is made worse: I can't undo it. To maintain the status quo
(I was quite happy to not have 100 packages pointlessly linked to
liblzma last week), what I want to do is set USE="-lzma -zstd". But if
I do that, then it overrides the IUSE defaults for the few packages
whose maintainers are doing the right thing, and setting IUSE="+lzma
+zstd" where it is important. For example, sys-apps/kmod, where the
defaults ensure that your system isn't made unbootable by accident.

The other option is for me to perpetually maintain a list of everything
that uses lzma/zstd inside my package.use. And to avoid the problem
above, I now have to read the ebuilds to see which ones have defaults
and which ones don't. This is also not a great way to spend my time.

tl;dr can we turn them back off in the profile? In any scenario where
they are beneficial, there's a better place to put them.

Am Sonntag, 7. April 2024, 14:51:55 CEST schrieb Michael Orlitzky:
> On Sun, 2024-04-07 at 14:35 +0200, Andreas K. Huettel wrote:
> >
> > Uhh, I dont really remember, I think some Chinese-sounding guy asked
> > me for it... (j/k)
>
> It is remarkably bad timing. How it looks: Gentoo's response to the xz
> incident is to have me rebuild my entire system with everything that
> could possibly be linked to liblzma, linked to liblzma. Even on the
> hardened profiles, and with no easy way to prevent it.

Well, we're now working with the best-audited compression library ever,
I guess.

> tl;dr can we turn them back off in the profile? In any scenario where
> they are beneficial, there's a better place to put them.

Easily doable with lzma, if there is consensus for it.

Slightly more complex for zstd since this affects gcc and binutils.
Still doable though.

--
Andreas K. H?ttel
dilfridge@gentoo.org
Gentoo Linux developer
(council, toolchain, base-system, perl, libreoffice)

> I see no way of migrating to 23.0 profile because of not-recompilable
> packages that are installed (over 4 years) which block --emptytree,
> and do not wish to be forced to migrate to merged-usr on an openrc box
> without a compelling need (on principle).

That sounds a bit like self-inflicted pain.

> Will patching back the 17.0 profile files into the portage tree if and
> when they are removed work?

Unknown.

> Are there any options at all for this situation (like freezing the the
> last supported tree protecting it from emerge-syncs, and using an
> overlay for further updates?)

You can try to just skip these packages (with --exclude) during the
"emerge --emptytree ..." step. It should work, but no guarantees given.

--
Andreas K. H?ttel
dilfridge@gentoo.org
Gentoo Linux developer
(council, toolchain, base-system, perl, libreoffice)

On Sun, 2024-04-07 at 08:51 -0400, Michael Orlitzky wrote:
> On Sun, 2024-04-07 at 14:35 +0200, Andreas K. Huettel wrote:
> >
> > Uhh, I dont really remember, I think some Chinese-sounding guy asked
> > me for it... (j/k)
>
> It is remarkably bad timing. How it looks: Gentoo's response to the xz
> incident is to have me rebuild my entire system with everything that
> could possibly be linked to liblzma, linked to liblzma. Even on the
> hardened profiles, and with no easy way to prevent it.

So, what you're basically saying, is that the best Gentoo response right
now would be to frantically remove LZMA support everywhere? I'm sure
that would be so much better than our response of masking vulnerable
versions and issuing a statement.

--
Best regards,
Micha? Górny

On Sun, 2024-04-07 at 16:48 +0200, Micha? Górny wrote:
>
> So, what you're basically saying, is that the best Gentoo response right
> now would be to frantically remove LZMA support everywhere? I'm sure
> that would be so much better than our response of masking vulnerable
> versions and issuing a statement.
>

Only in the sense that people who park their cars in the bike lane are
basically Hitler. This really has nothing to do with the xz thing. The
timing was funny, that's all.

What I am saying is that I want the freedom to not have things
pointlessly enabled on my systems, because similar problems (and worse)
happen all day every day. The less exposure I have, the better. The
liblzma backdoor was timely because it will prevent most people from
telling me I'm being paranoid, but it could have been USE=anything on
any other day. Moving the defaults out of the high-level profiles will
give control back to the user, hence my complaint about it.

On Sun, 7 Apr 2024 at 22:09, Michael Orlitzky <mjo@gentoo.org> wrote:
<snip>
> What I am saying is that I want the freedom to not have things
> pointlessly enabled on my systems, because similar problems (and worse)
> happen all day every day. The less exposure I have, the better. The
> liblzma backdoor was timely because it will prevent most people from
> telling me I'm being paranoid, but it could have been USE=anything on
> any other day. Moving the defaults out of the high-level profiles will
> give control back to the user, hence my complaint about it.
>

I agree, to be honest. The spirit of profiles has always felt like it
switches on safe/sane defaults that you'd expect for the name (a
desktop plasma profile switches on all the useful desktop USE flags, a
basic profile enables the bare minimum for a bootable system, etc),
giving an expected functionality in the resulting outcome of a
re-merge of world.

Outside of this, preferred compression tools, preferred editors
etc...should be up to the user, or implied in the profile name if it's
going to be switched on in the profile defaults. I don't use zstd
myself, I prefer xz or lz4 depending on my purpose. It's on my system
because some things I chose to have required it. It feels un-Gentoo
for me to have zstd around _just because_, which the profile default
would bring into play.

--
Ninpo

On Mon, 2024-04-08 at 01:22 +0100, Alex Boag-Munroe wrote:
> On Sun, 7 Apr 2024 at 22:09, Michael Orlitzky <mjo@gentoo.org> wrote:
> <snip>
> > What I am saying is that I want the freedom to not have things
> > pointlessly enabled on my systems, because similar problems (and worse)
> > happen all day every day. The less exposure I have, the better. The
> > liblzma backdoor was timely because it will prevent most people from
> > telling me I'm being paranoid, but it could have been USE=anything on
> > any other day. Moving the defaults out of the high-level profiles will
> > give control back to the user, hence my complaint about it.
> >
>
> I agree, to be honest. The spirit of profiles has always felt like it
> switches on safe/sane defaults that you'd expect for the name (a
> desktop plasma profile switches on all the useful desktop USE flags, a
> basic profile enables the bare minimum for a bootable system, etc),
> giving an expected functionality in the resulting outcome of a
> re-merge of world.

Precisely.

> Outside of this, preferred compression tools, preferred editors
> etc...should be up to the user, or implied in the profile name if it's
> going to be switched on in the profile defaults. I don't use zstd
> myself, I prefer xz or lz4 depending on my purpose. It's on my system
> because some things I chose to have required it. It feels un-Gentoo
> for me to have zstd around _just because_, which the profile default
> would bring into play.
>

It's not a "preferred compression tool". "Preferred compression tool"
is selected via adding the package to your @world set. The flag is used
for enable specific functionality on packages. This function may be
limited to being able to optionally compress something. But it could
e.g. also be responsible for being able to, say, open a specific file
format (and I'm not talking of explicitly .xz compressed files)
or a database, or receive proper interoperability elsewhere.

The cost of enabling support for a compression library that's already
installed by default (because you need it to unpack distfiles) is very
little compared to the cost of suddenly discovering that things don't
work.

--
Best regards,
Micha? Górny

Andreas K. Huettel posted on Sun, 07 Apr 2024 15:07:01 +0200 as excerpted:

> Am Sonntag, 7. April 2024, 14:51:55 CEST schrieb Michael Orlitzky:

[USE="lzma zstd" in 23.0 profiles]

>> [R]emarkably bad timing. How it looks: Gentoo's response to the xz
>> incident is to have me rebuild my entire system with everything that
>> could possibly be linked to liblzma, linked to liblzma. Even on the
>> hardened profiles, and with no easy way to prevent it.

Agreed. Timing is ... unfortunate, making for absolutely terrible
appearances. Tho for better or worse Gentoo will likely avoid the bad
press Arch or the the big guys would get for such a play as we're simply
not mainline enough any longer (Arch having eclipsed us as "the techie
distro" in the press years ago now).

> Well, we're now working with the best-audited compression library ever,
> I guess.

Also agreed...

>> tl;dr can we turn them back off in the profile? In any scenario where
>> they are beneficial, there's a better place to put them.

That's the core operational debate. Perhaps better to debate zstd and
lzma separately due to timing and relative ease (see below).

> Easily doable with lzma, if there is consensus for it.

Given lzma's easy, I'd vote for the revert, if only due to the unfortunate
timing. It can always be reconsidered later, even if for pragmatic
reasons "later" ends up being the /next/ profile upgrade, presumably some
years away.

But with the 17 downgrade to exp (if not deprecated yet), if we're
changing it (and not temporarily reverting the 17 exp) it should be ASAP!

> Slightly more complex for zstd since this affects gcc and binutils.
> Still doable though.

For zstad I'd keep as-is because it's both more difficult and lacks the
direct timing issues.

TLDR stop!

FWIW, no effect either way here/personally, because I configured portage
to ignore profile USE flags (as well as IUSE-defaults) years ago, in large
part precisely /because/ of the undesired USE-flag churn. And in general
it has made me a /much/ happier gentooer, because USE-flag no longer
blindly toggle without my having to go unduly out of my way to find out
why. (I already review the git log when a USE flag suddenly (dis)?
appears, unless it's blindingly obvious why without checking the log.)

The one thing I wish I had was an indication of IUSE-defaults for changes
on upgrades and for new packages. Sure I can (and do) grep for IUSE if I
have reason to wonder (more frequently for new packages if I don't know
whether I want something enabled or not), but I imagine I miss most of the
on-upgrade ISUE-default-changes as unlike the flag entirely (dis)?
appearing or (un)masking (which is still active from the profile) there's
nothing alerting me to IUSE-default changes.

--
Duncan - List replies preferred. No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master." Richard Stallman

On Sun, 2024-04-07 at 15:07 +0200, Andreas K. Huettel wrote:
> > tl;dr can we turn them back off in the profile? In any scenario where
> > they are beneficial, there's a better place to put them.
>
> Easily doable with lzma, if there is consensus for it.
>
> Slightly more complex for zstd since this affects gcc and binutils.
> Still doable though.
>

Thanks:

* https://bugs.gentoo.org/928932
* https://bugs.gentoo.org/928933

Michael Orlitzky wrote:
> On Sun, 2024-04-07 at 15:07 +0200, Andreas K. Huettel wrote:
>
>>> tl;dr can we turn them back off in the profile? In any scenario where
>>> they are beneficial, there's a better place to put them.
>>
>> Easily doable with lzma, if there is consensus for it.
>>
>> Slightly more complex for zstd since this affects gcc and binutils.
>> Still doable though.
>
> Thanks:
>
> * https://bugs.gentoo.org/928932
> * https://bugs.gentoo.org/928933

I know this thread is only for people actually involved in Gentoo
decision making, but I'll add my 2c anyway.

I'm sure nobody is surprised that I support Michael Orlitzky here 100%.

My personal "dream" is to have a Gentoo in the future where *all*
compression is optional, only enabled by those who want it, not forced
on anybody.

In my opinion the importance of compression in general diminishes every
year that goes by as naturally the trend in storage space has to be that
it increases. So compression will increasingly become a) an extra
undesirable security risk (it's quite complex to write and maintain
which only increases rather than decreases the likelihood of security
issues) and b) a cpu cycle waster (cpu resources will likely remain more
precious than storage).

I'd love to eventually see a Gentoo where most upstream source is pulled
in untouched and uncompressed by default and if people want compression
they can enable it. So I would hope that as each new profile release
comes, Gentoo becomes less chained to any particular compression
libraries than it was before, not more. But I'm aware this is
unrealistic today from the pov of Gentoo infra. Still, I'm allowed to
dream right?

P.S. This is not a demand, just 2c, and yes, I do know ultimately the
ones who role their sleeves up and submit patches will decide these things.

Eddie

* "Andreas K. Huettel" <2862978.mvXUDI8C0e @pinacolada> :
Wrote on Sun, 07 Apr 2024 15:27:42 +0200:

>> I see no way of migrating to 23.0 profile because of not-recompilable
>> packages that are installed (over 4 years) which block --emptytree,
>> and do not wish to be forced to migrate to merged-usr on an openrc box
>> without a compelling need (on principle).
> That sounds a bit like self-inflicted pain.
>> Will patching back the 17.0 profile files into the portage tree if and
>> when they are removed work?
> Unknown.
>
>> Are there any options at all for this situation (like freezing the the
>> last supported tree protecting it from emerge-syncs, and using an
>> overlay for further updates?)
>
> You can try to just skip these packages (with --exclude) during the
> "emerge --emptytree ..." step. It should work, but no guarantees given.

I switched the make.conf symlink (from
portage/profiles/default/linux/amd64/17.1 to
portage/profiles/default/linux/amd64/23.0/split-usr ) and the only
difference in the emerge --info output is that LDFLAGS now
additionally has "-Wl,-z,pack-relative-relocs"

My use pattern is I'm only emerging packages by hand and setting
useflags on a case by case basis. Also I have binpkgs going back to 5
years that I don't want to lose (by going to merged-usr, right now I
can unpack and test these in a pinch). I also have various other
packages outside the gentoo tree which depend on stuff which is not in
gentoo portage anymore, which are really not rebuildable, but because
of past portage flexibility multiple installed work fine and can be
tested at the same time.

The question is: Now if I don't attempt to do a rebuild and just
update libtool and do further upgrades and installs on a case by case
basis, it will will eventually pick up the new profile defaults. Is
there any foreseeable downside to just doing this?