Mailing List Archive

On Sat, 30 Mar 2024 03:07:13 -0000
"Eddie Chapman" <eddie@ehuk.net> wrote:

> Given what we've learnt in the last 24hrs about xz utilities, you
> could forgive a paranoid person for seriously considering getting rid
> entirely of them from their systems, especially since there are
> suitable alternatives available. Some might say that's a bit
> extreme, xz-utils will get a thorough audit and it will all be fine.
> But when a malicious actor has been a key maintainer of something as
> complex as a decompression utility for years, I'm not sure I could
> ever trust that codebase again. Maybe a complete rewrite will emerge,
> but I'm personally unwilling to continue using xz utils in the
> meantime for uncompressing anything on my systems, even if it is done
> by an unprivileged process.
>
> I see that many system package ebuilds unconditionally expect
> app-arch/xz-utils to be installed simply to be able to decompress the
> source archive in SRC_URI. So simply specifying -lzma on your system
> isn't going to get rid of it.
>
> No one could have been expected to foresee what's happened with
> xz-utils, but now that it's here, perhaps Gentoo (and other projects
> that do) should consider not relying on a single decompression
> algorithm for source archives, even just as an insurance against some
> other yet unknown disaster with one algorithm or another in future?
>
> And yes I'm sure there will be individual packages that currently
> absolutely need xz-utils installed during the build process, and one
> or two that absolutely have to have it available at runtime, but those
> bridges can be crossed as and when.
>
> Eddie
>
>

I think this is an overreaction and we should wait for the dust to
settle before making drastic disruptive changes.

orbea wrote:
> On Sat, 30 Mar 2024 03:07:13 -0000
> "Eddie Chapman" <eddie@ehuk.net> wrote:
>
>> Given what we've learnt in the last 24hrs about xz utilities, you
>> could forgive a paranoid person for seriously considering getting rid
>> entirely of them from their systems, especially since there are
>> suitable alternatives available. Some might say that's a bit
>> extreme, xz-utils will get a thorough audit and it will all be fine.
>> But when a malicious actor has been a key maintainer of something as
>> complex as a decompression utility for years, I'm not sure I could
>> ever trust that codebase again. Maybe a complete rewrite will emerge,
>> but I'm personally unwilling to continue using xz utils in the
>> meantime for uncompressing anything on my systems, even if it is done
>> by an unprivileged process.
>>
>> I see that many system package ebuilds unconditionally expect
>> app-arch/xz-utils to be installed simply to be able to decompress the
>> source archive in SRC_URI. So simply specifying -lzma on your system
>> isn't going to get rid of it.
>>
>> No one could have been expected to foresee what's happened with
>> xz-utils, but now that it's here, perhaps Gentoo (and other projects
>> that do) should consider not relying on a single decompression
>> algorithm for source archives, even just as an insurance against some
>> other yet unknown disaster with one algorithm or another in future?
>>
>> And yes I'm sure there will be individual packages that currently
>> absolutely need xz-utils installed during the build process, and one
>> or two that absolutely have to have it available at runtime, but those
>> bridges can be crossed as and when.
>>
>> Eddie
>>
>>
> I think this is an overreaction and we should wait for the dust to
> settle before making drastic disruptive changes.
>
>


From the news item email: 


"Impact
======

Our current understanding of the backdoor is that is does not affect
Gentoo systems, because

1. the backdoor only appears to be included on specific systems and
Gentoo does not qualify;
2. the backdoor as it is currently understood targets OpenSSH patched to
work with systemd-notify support. Gentoo does not support or include
these patches;

Analysis is still ongoing, however, and additional vectors may still be
identified. For this reason we are still issuing this advisory as if
that will be the case."


When I started reading it, I was concerned as well as I know it is used on my system. However, when I got to the part about it not likely to affect Gentoo, my level of concern dropped significantly. If this is still true, there's no need to be concerned. If things has changed and it does affect Gentoo, I'm sure there will be changes made that will either fix the issue for good or at least provide a workaround until a solution is found. Gentoo has some awesome devs. Someone will find a solution. I notice that it has already been changed in the tree to a version that does not have the malicious code. That alone should be a solution until a new plan is made.

While I'm a little concerned and hope for a proper solution, I'm not to worried. I certainly don't think we should overreact this early. Give the devs and upstream time to work this out.

Just a users opinion.

Dale

:-) :-)

Dale posted on Sat, 30 Mar 2024 02:06:26 -0500 as excerpted:

> Gentoo has some awesome devs.

Agreed with the whole thing and the above is a bit of an aside from the
thread, but it's worth repeating!

Thanks devs! (And security contributors, infra providers, testers,
tinder-box runners, bug reporters and wranglers, docs/wiki/lists/forums/
chat contributors, and all of the above for our upstreams too... it takes
a big community to make a full distro and we all have our part! =:^)

--
Duncan - List replies preferred. No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master." Richard Stallman

On Sat, Mar 30, 2024 at 3:06?AM Dale <rdalek1967@gmail.com> wrote:
>
> when I got to the part about it not likely to affect Gentoo, my level of concern dropped significantly. If this is still true, there's no need to be concerned.

"not likely" is the best way to characterize this. The exploit has
not been fully analyzed, and it could have additional malicious
behavior, either designed by its author, or perhaps even unintended by
its author.

I just wanted to toss in that caveat, but agree that the defaults
deployed in Gentoo seem the most sensible for general use. There is
nothing magical about xz - ANY widely-used library could have
something like this embedded in it, and the attacker exploited what
they had access to in order to go after a configuration that was going
to be widely deployed and reachable (xz+deb/rpm+systemd+openssh). If
the attacker had an intended target that used gentoo+openrc and access
to something in our supply chain, this could have been a vulnerability
that only impacted Gentoo.

I think the big lesson here is that FOSS continues to suffer from core
dependencies that are challenged for resources, and that efforts to
fix this have to constantly keep up with the changing landscape. xz
is going on 15 years old, but I don't think it was nearly as commonly
used until fairly recently.

libz has been a pretty well-known source of security flaws for ages
(granted, usually not intentional like this). It isn't too surprising
that in both cases compression libraries were targeted, as these are
so widely depended on.

This is getting tangential, but part of me wonders if there is a
better way to do authentication. Programs like ssh tend to run as
root so that they can authenticate users and then fork and suid to the
appropriate user. Could some OS-level facility be created to allow
unprivileged processes to run the daemons and then as part of the
authentication process they can have the OS accept a controlled and
minimal set of data to create the process as the new user and hand
over the connection? PAM already has a large amount of control over
the authentication process, so it seems like we just need to change
the security context that this runs in. That's just
brainstorming-level thinking though - there could be obvious issues
with this that just haven't occurred to me.

--
Rich

Rich, Duncan, Dale, orbea, you have to admit the situation with xz-utils
is nothing like the typical scenario people usually worry about, where a
bad actor manages to compromise a project and slip something into a widely
used piece of software. No, this is the the bad actor *themselves* being a
principal author of the software, working stealthily and in very
sophisticated ways for years, to manoeuvrer themselves and their software
into a position of trust in the ecosystem whereby they were almost able to
pull off the mother of all security nightmares for the world. And many
very smart people reviewed what they did and were fooled by them (which is
no reflection on those people, it was just because the bad actor did a
very, very good job of fooling them). I have to ask, if you still trust a
codebase to be right at the heart of your system after that, what on earth
would it take for you to start to feel uncomfortable??!!

Sometimes, it's good when you're inside the house that is on fire, to
*not* stand there and say to yourself "well the engineers who built this
place must have built it to withstand a fire, I'm sure it will stop
burning soon. And anyway, the fire brigade will be here soon, I'm sure it
will all be fine". I'm not saying the world of OSS & Linux is on fire, of
course not. This is a very isolated and rare situation with just 1 piece
of software. No, I'm just using probably a probably bad analogy to make
the following point: while almost all of the time a reasoned, "lets just
calm down and think about this" approach is right, in some rare situations
it is important to see a situation as serious as it and act accordingly.

In this case, if I weigh up the benefits of using this piece of software
versus another (relatively small gains in file size reduction, some gains
in resource usage) against the risks of continuing to use it (and lets be
realistic about those risks please rather than "I'm sure it will all be
fine"), the risks are far greater.

Note, I'm not advocating ripping xz-utils out of tree, all I'm saying is
wouldn't it be nice if there were at least 2 alternatives to choose from?
That doesn't have to be disruptive in any way, people who wish to continue
using and trusting xz-utils should be able to continue to do so without
any friction whatsoever.

Rich Freeman wrote:
> On Sat, Mar 30, 2024 at 3:06?AM Dale <rdalek1967@gmail.com> wrote:
>
>>
>> when I got to the part about it not likely to affect Gentoo, my level
>> of concern dropped significantly. If this is still true, there's no
>> need to be concerned.
>
> "not likely" is the best way to characterize this. The exploit has
> not been fully analyzed, and it could have additional malicious behavior,
> either designed by its author, or perhaps even unintended by its author.
>
> I just wanted to toss in that caveat, but agree that the defaults
> deployed in Gentoo seem the most sensible for general use. There is
> nothing magical about xz - ANY widely-used library could have something
> like this embedded in it, and the attacker exploited what they had access
> to in order to go after a configuration that was going to be widely
> deployed and reachable (xz+deb/rpm+systemd+openssh). If the attacker had
> an intended target that used gentoo+openrc and access to something in our
> supply chain, this could have been a vulnerability that only impacted
> Gentoo.
>
>
> I think the big lesson here is that FOSS continues to suffer from core
> dependencies that are challenged for resources, and that efforts to fix
> this have to constantly keep up with the changing landscape. xz is going
> on 15 years old, but I don't think it was nearly as commonly used until
> fairly recently.
>
> libz has been a pretty well-known source of security flaws for ages
> (granted, usually not intentional like this). It isn't too surprising
> that in both cases compression libraries were targeted, as these are so
> widely depended on.
>
> This is getting tangential, but part of me wonders if there is a
> better way to do authentication. Programs like ssh tend to run as root so
> that they can authenticate users and then fork and suid to the appropriate
> user. Could some OS-level facility be created to allow unprivileged
> processes to run the daemons and then as part of the authentication
> process they can have the OS accept a controlled and minimal set of data
> to create the process as the new user and hand over the connection? PAM
> already has a large amount of control over the authentication process, so
> it seems like we just need to change the security context that this runs
> in. That's just brainstorming-level thinking though - there could be
> obvious issues with this that just haven't occurred to me.
>
> --
> Rich
>
>
>
>

On Sat, 2024-03-30 at 14:57 +0000, Eddie Chapman wrote:
> Note, I'm not advocating ripping xz-utils out of tree, all I'm saying is
> wouldn't it be nice if there were at least 2 alternatives to choose from?
> That doesn't have to be disruptive in any way, people who wish to continue
> using and trusting xz-utils should be able to continue to do so without
> any friction whatsoever.

So, you're basically saying we should go out of our way, recompress all
distfiles using two alternative compression formats, increase mirror
load four times and add a lot of complexity to ebuilds, right?

--
Best regards,
Micha? Górny

On Sat, Mar 30, 2024 at 10:57?AM Eddie Chapman <eddie@ehuk.net> wrote:
>
> No, this is the the bad actor *themselves* being a
> principal author of the software, working stealthily and in very
> sophisticated ways for years, to manoeuvrer themselves and their software
> into a position of trust in the ecosystem whereby they were almost able to
> pull off the mother of all security nightmares for the world.

This is entirely speculative at this point. It isn't certain that the
author is the one behind the exploit, and if they were, it is not
known for how long their intentions were malicious, or even what their
motivations were. It is also unclear what pseudonymous accounts with
what projects are associated with the attacker.

You could end up being right, but it probably makes sense to at least
give things a few days for more facts to become available, before
making decisions to retool the entire distro.

I think the bigger challenge is what could have been done to prevent
this sort of problem in the first place. There are so many projects
that end up with code running as root that have one or two people
taking care of them, and if somebody does the work to become one of
those maintainers, there aren't many people looking out for problems.

I think one thing that would help here is for distros to have better
ways to ensure that the code in the scm matches the code in the
tarball. It is pretty common for releases to be manipulated in some
way (even if only to gpg sign them, but often to switch from commit
IDs to version numbers and so on), and that can be a place where stuff
gets added. That still says nothing about obfuscated code, which this
also involved.

--
Rich

Micha? Górny wrote:
> On Sat, 2024-03-30 at 14:57 +0000, Eddie Chapman wrote:
>
>> Note, I'm not advocating ripping xz-utils out of tree, all I'm saying
>> is wouldn't it be nice if there were at least 2 alternatives to choose
>> from? That doesn't have to be disruptive in any way, people who wish to
>> continue using and trusting xz-utils should be able to continue to do so
>> without any friction whatsoever.
>
> So, you're basically saying we should go out of our way, recompress all
> distfiles using two alternative compression formats, increase mirror load
> four times and add a lot of complexity to ebuilds, right?
>
> --
> Best regards,
> Micha? Górny
>

Yes that's a very good point, that was something I was wondering in
weighing up both sides, what the costs would be practically, as I don't
know the realities of running Gentoo infrastructure. And maybe the costs
is just too high of a price to pay.

I wonder if increased use of git repos rather than distributed tarballs
could be part of a solution to those issues, although that could put quite
a storage burden on every user. Unless they were all shallow git pulls and
the user could optionally choose to tar up the git directory after clone
with compression. But yes granted then there is even more ebuild
complexity.

On Sat, 30 Mar 2024 16:02:25 +0100
Micha? Górny <mgorny@gentoo.org> wrote:

> On Sat, 2024-03-30 at 14:57 +0000, Eddie Chapman wrote:
> > Note, I'm not advocating ripping xz-utils out of tree, all I'm
> > saying is wouldn't it be nice if there were at least 2 alternatives
> > to choose from? That doesn't have to be disruptive in any way,
> > people who wish to continue using and trusting xz-utils should be
> > able to continue to do so without any friction whatsoever.
>
> So, you're basically saying we should go out of our way, recompress
> all distfiles using two alternative compression formats, increase
> mirror load four times and add a lot of complexity to ebuilds, right?
>

How would Gentoo even recompress all .xz distfiles if xz-utils is not
even in the repo? And this will certainly be a reoccurring theme...

On Sat, 2024-03-30 at 15:17 +0000, Eddie Chapman wrote:
> Micha? Górny wrote:
> > On Sat, 2024-03-30 at 14:57 +0000, Eddie Chapman wrote:
> >
> > > Note, I'm not advocating ripping xz-utils out of tree, all I'm saying
> > > is wouldn't it be nice if there were at least 2 alternatives to choose
> > > from? That doesn't have to be disruptive in any way, people who wish to
> > > continue using and trusting xz-utils should be able to continue to do so
> > > without any friction whatsoever.
> >
> > So, you're basically saying we should go out of our way, recompress all
> > distfiles using two alternative compression formats, increase mirror load
> > four times and add a lot of complexity to ebuilds, right?
> >
> > --
> > Best regards,
> > Micha? Górny
> >
>
> Yes that's a very good point, that was something I was wondering in
> weighing up both sides, what the costs would be practically, as I don't
> know the realities of running Gentoo infrastructure. And maybe the costs
> is just too high of a price to pay.
>
> I wonder if increased use of git repos rather than distributed tarballs
> could be part of a solution to those issues, although that could put quite
> a storage burden on every user. Unless they were all shallow git pulls and
> the user could optionally choose to tar up the git directory after clone
> with compression. But yes granted then there is even more ebuild
> complexity.
>

Should we convert git repositories to Mercurial and Bazaar too, to avoid
relying too much on a single tool?

--
Best regards,
Micha? Górny

Micha? Górny wrote:
> On Sat, 2024-03-30 at 15:17 +0000, Eddie Chapman wrote:
>
>> Micha? Górny wrote:
>>
>>> On Sat, 2024-03-30 at 14:57 +0000, Eddie Chapman wrote:
>>>
>>>
>>>> Note, I'm not advocating ripping xz-utils out of tree, all I'm
>>>> saying is wouldn't it be nice if there were at least 2 alternatives
>>>> to choose from? That doesn't have to be disruptive in any way,
>>>> people who wish to continue using and trusting xz-utils should be
>>>> able to continue to do so without any friction whatsoever.
>>>
>>> So, you're basically saying we should go out of our way, recompress
>>> all distfiles using two alternative compression formats, increase
>>> mirror load four times and add a lot of complexity to ebuilds, right?
>>>
>>> --
>>> Best regards,
>>> Micha? Górny
>>>
>>>
>>
>> Yes that's a very good point, that was something I was wondering in
>> weighing up both sides, what the costs would be practically, as I don't
>> know the realities of running Gentoo infrastructure. And maybe the
>> costs is just too high of a price to pay.
>>
>> I wonder if increased use of git repos rather than distributed tarballs
>> could be part of a solution to those issues, although that could put
>> quite a storage burden on every user. Unless they were all shallow git
>> pulls and the user could optionally choose to tar up the git directory
>> after clone with compression. But yes granted then there is even more
>> ebuild complexity.
>>
>
> Should we convert git repositories to Mercurial and Bazaar too, to avoid
> relying too much on a single tool?
>
> --
> Best regards,
> Micha? Górny
>

I sense that question may have been slightly in jest :-) At least I hope
so as it could also be interpreted as an attempt at ridicule. I'll take it
as the former. In case you are seriously asking; of course not, that's
totally unnecessary. The objective is simply to obtain the upstream source
code intact. We don't need whatever version control of their source they
are using, which of course is the whole point of fetching distributed
tarballs. My suggestion of git pulls is just to address your point of
resource usage on gentoo infra, it reduces the need to store binary dist
files. I've also heard some argue that relying on distributed tarballs is
part of the overall problem and what the bad actor was taking advantage
of. They may have a point.

Eddie Chapman wrote:
> Micha? Górny wrote:
>> On Sat, 2024-03-30 at 14:57 +0000, Eddie Chapman wrote:
>>
>>> Note, I'm not advocating ripping xz-utils out of tree, all I'm saying
>>> is wouldn't it be nice if there were at least 2 alternatives to choose
>>> from? That doesn't have to be disruptive in any way, people who wish to
>>> continue using and trusting xz-utils should be able to continue to do so
>>> without any friction whatsoever.
>> So, you're basically saying we should go out of our way, recompress all
>> distfiles using two alternative compression formats, increase mirror load
>> four times and add a lot of complexity to ebuilds, right?
>>
>> --
>> Best regards,
>> Micha? Górny
>>
> Yes that's a very good point, that was something I was wondering in
> weighing up both sides, what the costs would be practically, as I don't
> know the realities of running Gentoo infrastructure. And maybe the costs
> is just too high of a price to pay.
>
> I wonder if increased use of git repos rather than distributed tarballs
> could be part of a solution to those issues, although that could put quite
> a storage burden on every user. Unless they were all shallow git pulls and
> the user could optionally choose to tar up the git directory after clone
> with compression. But yes granted then there is even more ebuild
> complexity.
>
>
> .
>

There is a lot of unknowns out there.  From what I've read, the person
responsible for writing the code inserted this hack.  There may be no
way to prevent this.  Basically, the person that should have been
trusted with this code violated that trust.  Why is unknown but I'm as
curious about that as anything.  It's like when someone goes to a
grocery store to buy a tomato.  They want organic and there is a organic
sticker on the tomato.  You either trust that sticker, and the
person/company who put it on there, or you don't trust that sticker at
all and avoid buying all tomatoes.  The trust starts with the
person/company that puts that sticker on the tomato.  The person who was
trusted with that code, broke that trust.  There is likely hundreds of
packages out there in the exact same position.  Any package that has few
or only one person writing the code can do the same thing. 

While this should be analyzed as more info comes in, right now, we
should let the devs get us back to as safe a place as possible.  Since
it appears to affect systemd users who don't use Gentoo, which is a huge
target, they certainly need to react as quickly as they can to the devs
actions.  Let's just not overreact just yet.  The devs has rolled back
to a safe, safer, version.  Let time and more info sort this out. If it
is needed, xz will go away, which shouldn't come as a surprise.  I'm
sure the person who did this will never get that trust back. 

Long term, this is going to be interesting to see what all gets
revealed.  The why is one thing.  Another is how to prevent if it can be
at all. 

I'm going back to my hole now. 

Dale

:-)  :-) 

P. S.  Links that some may want to follow, instead of a -dev thread. 

https://bugs.gentoo.org/928134

https://forums.gentoo.org/viewtopic.php?p=8821925

------ Original Message ------
From "Eddie Chapman" <eddie@ehuk.net>
To gentoo-dev@lists.gentoo.org
Date 30.03.2024 16:17:19
Subject Re: [gentoo-dev] Current unavoidable use of xz utils in Gentoo

>Micha? Górny wrote:
>> On Sat, 2024-03-30 at 14:57 +0000, Eddie Chapman wrote:
>>
>>> Note, I'm not advocating ripping xz-utils out of tree, all I'm saying
>>> is wouldn't it be nice if there were at least 2 alternatives to choose
>>> from? That doesn't have to be disruptive in any way, people who wish to
>>> continue using and trusting xz-utils should be able to continue to do so
>>> without any friction whatsoever.
>>
>> So, you're basically saying we should go out of our way, recompress all
>> distfiles using two alternative compression formats, increase mirror load
>> four times and add a lot of complexity to ebuilds, right?
>>
>> --
>> Best regards,
>> Micha? Górny
>>
>
>Yes that's a very good point, that was something I was wondering in
>weighing up both sides, what the costs would be practically, as I don't
>know the realities of running Gentoo infrastructure. And maybe the costs
>is just too high of a price to pay.
>
>I wonder if increased use of git repos rather than distributed tarballs
>could be part of a solution to those issues, although that could put quite
>a storage burden on every user. Unless they were all shallow git pulls and
>the user could optionally choose to tar up the git directory after clone
>with compression. But yes granted then there is even more ebuild
>complexity.
>
Huh ... I read your original message as

"wouldn't it be nice to have at least 2 alternative [implementations of
xz-utils]
to choose from"

As long as the file format itself is not inherently messed up, the
archives
could stay as .xz, only a "minimal" unxz (similar to unrar) would be
required
to access the contents.

Regards,
s.

Rich Freeman wrote:
> On Sat, Mar 30, 2024 at 10:57?AM Eddie Chapman <eddie@ehuk.net> wrote:
>
>> No, this is the the bad actor *themselves* being a
>> principal author of the software, working stealthily and in very
>> sophisticated ways for years, to manoeuvrer themselves and their
>> software into a position of trust in the ecosystem whereby they were
>> almost able to pull off the mother of all security nightmares for the
>> world.
>
> This is entirely speculative at this point. It isn't certain that the
> author is the one behind the exploit, and if they were, it is not known for
> how long their intentions were malicious, or even what their motivations
> were. It is also unclear what pseudonymous accounts with what projects
> are associated with the attacker.

For the purposes of this discussion I'm not speculating nor interested in
*who* is behind this, or whether or whoever committed commits was a victim
of account takeover. Certain key actions that have been taken over time by
whoever is/was behind this do not require any speculation, they speak for
themselves, and are clearly malicious. There is no need to wait for
anything more to be revealed to be able to plainly see how bad it is.

While we wait and see, huge numbers of people might be suffering real and
serious consequences of continued use of xz-utils. The consequences may be
completely hidden, if you go by how well the bad actor here has managed to
hide what they have done. If you are following developments you can see
right now with your own eyes how incredibly difficult it is for our
smartest people to unravel and pick through what this actor has done. To
have faith that everything malicious that the perpetrator has done will be
unravelled over time by our collective smart minds by going over the
codebase with a fine tooth-comb puts far too much faith in human beings
and takes unnecessary risks for something that is not worth that risk when
there are alternatives. If you were looking for a compression tool for a
new project, why would anyone sane take such risks for such little gain?
You just wouldn't. Of course the reason there is hesitancy is because xz
has become so deeply entrenched in our world, it's become almost too hard
to extrapolate ourselves from it. I dare say the attacker realised this
and probably sought to take advantage of that fact.

However, I do acknowledge and realise the significant practical
difficulties that would be involved in making xz-utils something optional
within Gentoo.

Stefan Schmiedl wrote:
> ------ Original Message ------
>
>> From "Eddie Chapman" <eddie@ehuk.net>
>>
> To gentoo-dev@lists.gentoo.org
> Date 30.03.2024 16:17:19
> Subject Re: [gentoo-dev] Current unavoidable use of xz utils in Gentoo
>
>> Micha? Górny wrote:
>>
>>> On Sat, 2024-03-30 at 14:57 +0000, Eddie Chapman wrote:
>>>
>>>
>>>> Note, I'm not advocating ripping xz-utils out of tree, all I'm
>>>> saying is wouldn't it be nice if there were at least 2 alternatives
>>>> to choose from? That doesn't have to be disruptive in any way,
>>>> people who wish to continue using and trusting xz-utils should be
>>>> able to continue to do so without any friction whatsoever.
>>>
>>> So, you're basically saying we should go out of our way, recompress
>>> all distfiles using two alternative compression formats, increase
>>> mirror load four times and add a lot of complexity to ebuilds, right?
>>>
>>> --
>>> Best regards,
>>> Micha? Górny
>>
>> Yes that's a very good point, that was something I was wondering in
>> weighing up both sides, what the costs would be practically, as I don't
>> know the realities of running Gentoo infrastructure. And maybe the
>> costs is just too high of a price to pay.
>>
>> I wonder if increased use of git repos rather than distributed tarballs
>> could be part of a solution to those issues, although that could put
>> quite a storage burden on every user. Unless they were all shallow git
>> pulls and the user could optionally choose to tar up the git directory
>> after clone with compression. But yes granted then there is even more
>> ebuild complexity.
>>
> Huh ... I read your original message as
>
> "wouldn't it be nice to have at least 2 alternative [implementations of
> xz-utils] to choose from"
>
> As long as the file format itself is not inherently messed up, the
> archives could stay as .xz, only a "minimal" unxz (similar to unrar) would
> be required to access the contents.
>
> Regards,
> s.

I see, no, I originally meant to have two compression/decompression
formats; LZMA (xz) and another. But yes the way you understood it is
interesting. Initially I dismissed this idea as would take too long for a
new tool to reach stability. But yes what you suggest is it could be a
very simple implementation that only does decompression so perhaps more
realistically achievable. Still a tall order. I wonder if any general
purpose languages (python, perl, etc) have developed their own built in
LZMA decompression implementation? I doubt it, would have thought they'd
just link against liblzma.so and not re-invent the wheel.

Eddie Chapman wrote:
> Micha? Górny wrote:
>
>> On Sat, 2024-03-30 at 14:57 +0000, Eddie Chapman wrote:
>>
>>
>>> Note, I'm not advocating ripping xz-utils out of tree, all I'm saying
>>> is wouldn't it be nice if there were at least 2 alternatives to
>>> choose from? That doesn't have to be disruptive in any way, people who
>>> wish to continue using and trusting xz-utils should be able to
>>> continue to do so without any friction whatsoever.
>>
>> So, you're basically saying we should go out of our way, recompress all
>> distfiles using two alternative compression formats, increase mirror
>> load four times and add a lot of complexity to ebuilds, right?
>>
>> --
>> Best regards,
>> Micha? Górny
>>
> Yes that's a very good point, that was something I was wondering in
> weighing up both sides, what the costs would be practically, as I don't
> know the realities of running Gentoo infrastructure. And maybe the costs
> is just too high of a price to pay.
>
> I wonder if increased use of git repos rather than distributed tarballs
> could be part of a solution to those issues, although that could put quite
> a storage burden on every user. Unless they were all shallow git pulls
> and the user could optionally choose to tar up the git directory after
> clone with compression. But yes granted then there is even more ebuild
> complexity.
>

I've been thinking a little about how Gentoo without
compression/decompression of distfiles could work, as a feature, without
any impact on the existing world order, and no increased stress on Gentoo
infra. I was wondering how palatable the following idea might be to others
...

The basis of the idea is to add a feature to Portage which would let a
person optionally indicate in make.conf that whenever a path in SRC_URI
resolves to a file with a compression extension (.gz, .bz2, .xz, etc),
that Portage should attempt to fetch it without the compression extension.

So as an example, lets take sys-apps/pciutils, which currently has:
SRC_URI="https://mj.ucw.cz/download/linux/pci/${P}.tar.gz"

the feature would tell portage to simply translate this to:
SRC_URI="https://mj.ucw.cz/download/linux/pci/${P}.tar"

So perhaps it could be a flag that goes in FEATURES= called something like
"strip_dist_comp" or something similar, or maybe someone has a better idea
about that.

Now, of course, I'm not proposing that Gentoo infra keeps uncompressed
versions of distfiles. So by default Portage would encounter a 404 error
when it tries to fetch the uncompressed file from Gentoo mirrors.

However, this feature would then pave the way for a person to then
configure Portage to fetch distfiles from their own server as well as
Gentoo mirrors, and that person could then keep their own uncompressed
versions of distfiles on their server, for however many and whichever
distfiles they might wish to keep there, as the compressed version would
get fetched from a Gentoo mirror if the uncompressed version is not there.
Such a person would then have to obtain or create their own uncompressed
distfile independently.

A caveat of this solution would be that one would have to disable checksum
verification (and gpg checks?) for this to work, as of course there would
be no checksum for the uncompressed version in the Manifest, and Gentoo
infra certainly should not be expected to especially uncompress each
distfile once in order to generate an extra checksum for the Manifest. In
fact I'd consider than undesirable, as anyone paranoid enough to want to
do this would not trust such a checksum anyway, since it would be a
checksum of a file that has been compressed at source and then
decompressed on Gentoo infra, potentially introducing vulnerabilities.
However, the lack of checksum is not a problem for someone who wants to
keep distfiles on their own server, as such a person can also be
responsible themselves for first verifying whatever they put on there, and
for keeping said server secured from tampering.

This seems to me to be something that would probably be relatively
straightforward to implement within Portage, maybe with just a few lines
around the python code that fetches the SRC_URI, and zero extra work or
resources required from Gentoo infra.

I'd consider it a feature for anyone who wants to eliminate a whole
potential class of vulnerabilities that may or may not be present either
now or in future in compression algorithm tools. Surely that would be a
nice feature to have for some folk?

"Eddie Chapman" <eddie@ehuk.net> writes:

> Given what we've learnt in the last 24hrs about xz utilities, you could
> forgive a paranoid person for seriously considering getting rid entirely
> of them from their systems, especially since there are suitable
> alternatives available. Some might say that's a bit extreme, xz-utils
> will get a thorough audit and it will all be fine. But when a malicious
> actor has been a key maintainer of something as complex as a decompression
> utility for years, I'm not sure I could ever trust that codebase again.
> Maybe a complete rewrite will emerge, but I'm personally unwilling to
> continue using xz utils in the meantime for uncompressing anything on my
> systems, even if it is done by an unprivileged process.

My own view is that there'll be a time for introspection, reflection,
and discussion of changes once the crisis is over. We're not there yet.

But I don't think us fetching several variants of compression formats
and testing & verifying all of them is feasible.

I also think it's (and I don't mean this derogatorily towards you) naive
for people in general to suggest that this is really specific to xz at
all. Unfortunately, there's many. many projects this could've happened to.

>
> I see that many system package ebuilds unconditionally expect
> app-arch/xz-utils to be installed simply to be able to decompress the
> source archive in SRC_URI. So simply specifying -lzma on your system isn't
> going to get rid of it.
>
> No one could have been expected to foresee what's happened with xz-utils,
> but now that it's here, perhaps Gentoo (and other projects that do) should
> consider not relying on a single decompression algorithm for source
> archives, even just as an insurance against some other yet unknown
> disaster with one algorithm or another in future?

I think there's real discussions to be had about relying on dist
tarballs and such but I don't really see how we could try to avoid
compression algorithms.

>
> And yes I'm sure there will be individual packages that currently
> absolutely need xz-utils installed during the build process, and one or
> two that absolutely have to have it available at runtime, but those
> bridges can be crossed as and when.
>
> Eddie

thanks,
sam

On 3/29/24 11:07 PM, Eddie Chapman wrote:
> Given what we've learnt in the last 24hrs about xz utilities, you could
> forgive a paranoid person for seriously considering getting rid entirely
> of them from their systems, especially since there are suitable
> alternatives available. Some might say that's a bit extreme, xz-utils
> will get a thorough audit and it will all be fine. But when a malicious
> actor has been a key maintainer of something as complex as a decompression
> utility for years, I'm not sure I could ever trust that codebase again.
> Maybe a complete rewrite will emerge, but I'm personally unwilling to
> continue using xz utils in the meantime for uncompressing anything on my
> systems, even if it is done by an unprivileged process.


It suffices to downgrade to the version of xz before a social
engineering attack by a malicious actor to gain maintainership of the xz
project.

Have you been linked to this yet?
https://www.mail-archive.com/xz-devel@tukaani.org/msg00571.html


--
Eli Schwartz

On 3/30/24 11:17 AM, Eddie Chapman wrote:
> Yes that's a very good point, that was something I was wondering in
> weighing up both sides, what the costs would be practically, as I don't
> know the realities of running Gentoo infrastructure. And maybe the costs
> is just too high of a price to pay.
>
> I wonder if increased use of git repos rather than distributed tarballs
> could be part of a solution to those issues, although that could put quite
> a storage burden on every user. Unless they were all shallow git pulls and
> the user could optionally choose to tar up the git directory after clone
> with compression. But yes granted then there is even more ebuild
> complexity.


Live ebuilds cannot have keywords, so using git repos is not a valid
option. There's not really much to discuss here.

Recompressing all distfiles in gentoo-specific ways is... definitely a
decision. It's a decision that Debian has made, mind you, so it's not
like Gentoo would be breaking new ground here, but frankly I don't
really regard that as fundamentally palatable.


--
Eli Schwartz

A decompression implementation all in rust it would seem.

https://github.com/gendx/lzma-rs



On Sat, Mar 30, 2024 at 12:36?PM Eddie Chapman <eddie@ehuk.net> wrote:

> Stefan Schmiedl wrote:
> > ------ Original Message ------
> >
> >> From "Eddie Chapman" <eddie@ehuk.net>
> >>
> > To gentoo-dev@lists.gentoo.org
> > Date 30.03.2024 16:17:19
> > Subject Re: [gentoo-dev] Current unavoidable use of xz utils in Gentoo
> >
> >> Micha? Górny wrote:
> >>
> >>> On Sat, 2024-03-30 at 14:57 +0000, Eddie Chapman wrote:
> >>>
> >>>
> >>>> Note, I'm not advocating ripping xz-utils out of tree, all I'm
> >>>> saying is wouldn't it be nice if there were at least 2 alternatives
> >>>> to choose from? That doesn't have to be disruptive in any way,
> >>>> people who wish to continue using and trusting xz-utils should be
> >>>> able to continue to do so without any friction whatsoever.
> >>>
> >>> So, you're basically saying we should go out of our way, recompress
> >>> all distfiles using two alternative compression formats, increase
> >>> mirror load four times and add a lot of complexity to ebuilds, right?
> >>>
> >>> --
> >>> Best regards,
> >>> Micha? Górny
> >>
> >> Yes that's a very good point, that was something I was wondering in
> >> weighing up both sides, what the costs would be practically, as I don't
> >> know the realities of running Gentoo infrastructure. And maybe the
> >> costs is just too high of a price to pay.
> >>
> >> I wonder if increased use of git repos rather than distributed tarballs
> >> could be part of a solution to those issues, although that could put
> >> quite a storage burden on every user. Unless they were all shallow git
> >> pulls and the user could optionally choose to tar up the git directory
> >> after clone with compression. But yes granted then there is even more
> >> ebuild complexity.
> >>
> > Huh ... I read your original message as
> >
> > "wouldn't it be nice to have at least 2 alternative [implementations of
> > xz-utils] to choose from"
> >
> > As long as the file format itself is not inherently messed up, the
> > archives could stay as .xz, only a "minimal" unxz (similar to unrar)
> would
> > be required to access the contents.
> >
> > Regards,
> > s.
>
> I see, no, I originally meant to have two compression/decompression
> formats; LZMA (xz) and another. But yes the way you understood it is
> interesting. Initially I dismissed this idea as would take too long for a
> new tool to reach stability. But yes what you suggest is it could be a
> very simple implementation that only does decompression so perhaps more
> realistically achievable. Still a tall order. I wonder if any general
> purpose languages (python, perl, etc) have developed their own built in
> LZMA decompression implementation? I doubt it, would have thought they'd
> just link against liblzma.so and not re-invent the wheel.
>
>
>

Eli Schwartz wrote:
> On 3/29/24 11:07 PM, Eddie Chapman wrote:
>
>> Given what we've learnt in the last 24hrs about xz utilities, you could
>> forgive a paranoid person for seriously considering getting rid
>> entirely of them from their systems, especially since there are suitable
>> alternatives available. Some might say that's a bit extreme, xz-utils
>> will get a thorough audit and it will all be fine. But when a
>> malicious actor has been a key maintainer of something as complex as a
>> decompression utility for years, I'm not sure I could ever trust that
>> codebase again. Maybe a complete rewrite will emerge, but I'm personally
>> unwilling to continue using xz utils in the meantime for uncompressing
>> anything on my systems, even if it is done by an unprivileged process.
>
>
> It suffices to downgrade to the version of xz before a social
> engineering attack by a malicious actor to gain maintainership of the xz
> project.
>
> Have you been linked to this yet?
> https://www.mail-archive.com/xz-devel@tukaani.org/msg00571.html
>
> --
> Eli Schwartz
>

Yes I saw that yesterday. It only increased my level of concern about the
project ten-fold rather than decreased it, particularly because of "he has
been helping a lot off-list and is practically a co-maintainer already".
It's not possible to just downgrade to before the bad actor's commits and
then feel fine about things because they have been heavily involved
offline even before commit access. We'll never know how much and when
because I also cannot trust what the apparently innocent maintainer (who
is most likely a victim here as well) might say about that now. Not
because of anything about them (I don't know them or anything about them),
just because of what has happened, there is too much of an incentive for
that person to now downplay the involvement of the bad actor. I'm sorry
if that may seem harsh but, in my view, this situation is so severe it
warrants it. The world is facing threats from very sophisticated and
capable bad actors, mostly criminal organisations. If people here want to
run systems that are actually secure and also have other people trust
their stewardship, then things need to be taken seriously and high
standards need to be maintained. Especially where it is a tool that is not
super essential (it has just become heavily entrenched) and where there
are great alternatives, there should be no hesitancy to jettison a project
that has been infiltrated to such an extent as we have seen here (this is
far beyond just some devs workstation got compromised and there was a few
bad commits made it into the repo). At the moment there is far too much of
a cavalier attitude about the whole thing being shown by too many,
including here I'm sad to see.

On 2024-03-31 01:33, Eli Schwartz wrote:
> On 3/29/24 11:07 PM, Eddie Chapman wrote:
>> Given what we've learnt in the last 24hrs about xz utilities, you
>> could
>> forgive a paranoid person for seriously considering getting rid
>> entirely
>> of them from their systems, especially since there are suitable
>> alternatives available. Some might say that's a bit extreme, xz-utils
>> will get a thorough audit and it will all be fine. But when a
>> malicious
>> actor has been a key maintainer of something as complex as a
>> decompression
>> utility for years, I'm not sure I could ever trust that codebase
>> again.
>> Maybe a complete rewrite will emerge, but I'm personally unwilling to
>> continue using xz utils in the meantime for uncompressing anything on
>> my
>> systems, even if it is done by an unprivileged process.
>
>
> It suffices to downgrade to the version of xz before a social
> engineering attack by a malicious actor to gain maintainership of the
> xz
> project.
>
> Have you been linked to this yet?
> https://www.mail-archive.com/xz-devel@tukaani.org/msg00571.html

> Wed, 29 Jun 2022 13:07:07 -0700

This is 2 years ago.

Had I seen someone say that a bad actor would spend years gaining the
trust of FOSS
project maintainers in order to gain commit access and introduce such
sophisticated
back doors, I would have told them to take their meds.
This is insane.

Not even this seems impossible anymore:
https://01.me/en/2014/11/insert-backdoor-into-compiler/

If this happened to something like firefox, I don't think anyone would
have found out.
No one bats an eye if a website loads 0.5s longer.

--
Linux-gentoo-x86_64-Intel-R-_Core-TM-_i5-7400_CPU_@_3.00GHz

COMMON_FLAGS="-O3 -pipe -march=native -fno-stack-protector
-ftree-vectorize -ffast-math -funswitch-loops -fuse-linker-plugin -flto
-fdevirtualize-at-ltrans -fno-plt -fno-semantic-interposition
-falign-functions=64 -fgraphite-identity -floop-nest-optimize"

USE="-* git verify-sig rsync-verify man alsa X grub ssl ipv6 lto
libressl olde-gentoo asm native-symlinks threads jit jumbo-build minimal
strip system-man"

INSTALL_MASK="/etc/systemd /lib/systemd /usr/lib/systemd
/usr/lib/modules-load.d /usr/lib/tmpfiles.d *tmpfiles* /var/lib/dbus
/lib/udev /usr/share/icons /usr/share/applications
/usr/share/gtk-3.0/emoji"

Hi Eddie,

On 31/3/24 21:13, Eddie Chapman wrote:
> At the moment there is far too much of
> a cavalier attitude about the whole thing being shown by too many,
> including here I'm sad to see.

It's obvious that this is something that you are very worried about, but
I think that you need to take a deep breath and relax a little. I have
not seen a cavalier attitude towards this issue.

What I see instead are developers who have made an initial assessment
and disclosure, have taken some actions to mitigate the severity of the
issue, and are carefully continuing their investigations (over a major
holiday in large parts of the world) so that they can issue sane
responses to actual threats, and not a knee-jerk response like 'Gentoo
should re-encode all xzs in other formats' which, as was discussed
above, adds significant complexity for no real benefit.

I've seen from your previous emails to the list that you know what
paragraphs are and how to use them to break up your content into
digestible chunks. Please continue using them - it makes it
significantly easier to respond to your ideas and gives off an aura of
professionalism that you will need if you want your concerns to be taken
seriously and addressed directly.

Cheers,

Matt

Matt Jolly wrote:
> Hi Eddie,
>
> On 31/3/24 21:13, Eddie Chapman wrote:
>
>> At the moment there is far too much of
>> a cavalier attitude about the whole thing being shown by too many,
>> including here I'm sad to see.
>
> It's obvious that this is something that you are very worried about, but
> I think that you need to take a deep breath and relax a little. I have
> not seen a cavalier attitude towards this issue.

No, I don't need to do that. I don't appreciate suggestions to "just calm
down", especially when I'm not being hysterical. Your comment to me just
reinforces what I mean when I say there is far too much of a cavalier
attitude.

> What I see instead are developers who have made an initial assessment
> and disclosure, have taken some actions to mitigate the severity of the
> issue, and are carefully continuing their investigations (over a major
> holiday in large parts of the world) so that they can issue sane responses
> to actual threats, and not a knee-jerk response like 'Gentoo should
> re-encode all xzs in other formats' which, as was discussed above, adds
> significant complexity for no real benefit.

I stand by and reiterate my view that there is far too much of a cavalier
attitude towards the matter in general out there including here in Gentoo.
But not in particular here, it is everywhere where this is being discussed
at the moment.

But please think a little about what I mean when I say a "cavalier
attitude", and what it does NOT mean. It does not mean that a lot of
people are not working very hard to analyse and learn about this issue and
taking steps to try to mitigate it. It does not mean people are not well
intentioned, everyone wants to fix this. I have great appreciation and
admiration for a lot of fantastic work I see going on including by people
involved in Gentoo. But I believe it will only really be beneficial in the
far future, not right now.

How are people in general being cavalier? By trying desperately to salvage
the situation with xz-utils above all else, by focussing too much on how
the original author of xz-utils and rallying round them (absolutely a
great thing to do but has absolutely nothing to do with what is good or
not good for users as a whole right now), there is too much clouded
judgment. There is more I could argue about why I use that word, but I
know by now that I am going against the grain of what the majority want
and it's not what people want to hear so I'm done, this discussion is now
a waste of everyone's time here including mine.

> I've seen from your previous emails to the list that you know what
> paragraphs are and how to use them to break up your content into digestible
> chunks. Please continue using them - it makes it significantly easier to
> respond to your ideas and gives off an aura of professionalism that you
> will need if you want your concerns to be taken seriously and addressed
> directly.

Yes you are right, I do apologise for not using paragraphs in my last
message, I slipped there, thanks for pointing it out. I've tried to do so
in this message.

> Cheers,
>
> Matt

And I should make more of an effort to sign off, it's a little more
friendly :-)

Regards,
Eddie