Mailing List Archive

If that’s working, it could at least be on an user personnal page on the
wiki as well.

Le 04/04/2024 à 10:32, Sam James a écrit :
> Eli Schwartz <eschwartz93@gmail.com> writes:
>
>> On 4/3/24 11:30 AM, Eddie Chapman wrote:
>>> Just to report I've been able to remove app-arch/xz-utils from my own
>>> workstation, with 2412 packages installed and running kde. I'm going to
>>> roll it out to my other gentoo systems which have a lot less stuff on them
>>> so am confident will be fine. It's not completely trivial but not as
>>> difficult as I imagined it to be, certainly something an advance Gentoo
>>> user could do if they wanted, with instructions. It does involve a
>>> relatively small hack and functionality previously provided by xz-utils is
>>> replaced by app-arch/p7zip.
>>
>> I'd just like to clarify my previous posts: what you're describing here
>> is neat and productive and valid to my eyes. Actually, I wish this had
>> been the topic of the *first* post in this thread. :)
> Completely agreed. We just prefer shorter text and focusing on technical
> changes.
>
> This sounds fun!
>
>> [...]
> thanks,
> sam
>

Eli Schwartz wrote:
> On 4/3/24 11:30 AM, Eddie Chapman wrote:
>
>> Just to report I've been able to remove app-arch/xz-utils from my own
>> workstation, with 2412 packages installed and running kde. I'm going to
>> roll it out to my other gentoo systems which have a lot less stuff on
>> them so am confident will be fine. It's not completely trivial but not
>> as difficult as I imagined it to be, certainly something an advance
>> Gentoo
>> user could do if they wanted, with instructions. It does involve a
>> relatively small hack and functionality previously provided by xz-utils
>> is replaced by app-arch/p7zip.
>
>
> I'd just like to clarify my previous posts: what you're describing here
> is neat and productive and valid to my eyes. Actually, I wish this had been
> the topic of the *first* post in this thread. :)
>
> Replacing implementations has several great uses. There's some prior art
> in make.conf, but it doesn't go far enough:
>
> PORTAGE_BZIP2_COMMAND
> BINPKG_COMPRESS
> BINPKG_COMPRESS_FLAGS
>
>
> Disregarding the security component entirely, one might wish to use pixz
> or pigz instead of the default programs. Why not 7zip as well?

One of my emails elsewhere in this thread (easy to miss in a long thread,
I know) I discussed pixz, pigz and 7zip. The former two were not suitable
for me as both rely on xz utils. However I will probably switch from p7zip
to the latest upstream 7zip in the near future, for reasons discussed in
that email.

> In terms of security, this suggests an easy and simple way both to allow
> users to depclean xz-utils without sacrificing the ability to install
> packages using *.tar.xz sources, and for Gentoo to roll out an update that
> would do this distribution-wide if necessary via a trivial configuration
> change.
>
> https://dev.gentoo.org/~ulm/pms/head/pms.html#section-12.3.15 may need
> updating to allow this. But it seems very valid to propose doing exactly
> that. I am not sure why it specifies e.g. "must ensure that GNU gzip" with
> heavy ties to implementations, when it doesn't specify such for
> compression.

That would certainly be a nice improvement for all users if it were ever
to come to pass.

> I'm guessing what you did was override/hook the unpack phase helper
> function and divert it to 7zip instead. ;) It would be interesting to have
> actual hooks for that instead.

Yes it is in the unpack phase where emerge calls /usr/bin/xz mostly. In
fact I didn't have to touch emerge/portage, it was more crude, I
uninstalled app-arch/xz-utils (and put it in
/etc/portage/profile/package.provided) and replaced /usr/bin/xz with a
bash script to behave like what the unpack phase was expecting, but using
/usr/lib64/p7zip/7za to do the decompression.

However, I need to do some more work on this "wrapper" (though it's more
than a wrapper) as I found one package where xz is called from the install
phase and my script doesn't handle that yet it just throws an error for
anything other than the unpack phase case (which is 99.9 percent of
packages).

But ultimately doing something along the lines of what you suggest instead
would of course be much better than this dirty hack (though it works just
fine for me for now).

Since there appears to be some interest I'll put together a single email
to the list later today detailing everything, as I needed to do more
things overall in addition to replacing /usr/bin/xz.

Sam James wrote:
> Eli Schwartz <eschwartz93@gmail.com> writes:
>
>> On 4/3/24 11:30 AM, Eddie Chapman wrote:
>>
>>> Just to report I've been able to remove app-arch/xz-utils from my own
>>> workstation, with 2412 packages installed and running kde. I'm going
>>> to roll it out to my other gentoo systems which have a lot less stuff
>>> on them so am confident will be fine. It's not completely trivial but
>>> not as difficult as I imagined it to be, certainly something an
>>> advance Gentoo user could do if they wanted, with instructions. It
>>> does involve a relatively small hack and functionality previously
>>> provided by xz-utils is replaced by app-arch/p7zip.
>>
>> I'd just like to clarify my previous posts: what you're describing here
>> is neat and productive and valid to my eyes. Actually, I wish this had
>> been the topic of the *first* post in this thread. :)
>
> Completely agreed. We just prefer shorter text and focusing on technical
> changes.
>
> This sounds fun!
>
>
>> [...]
>>
>
> thanks, sam

Well, I didn't think my first post was so bad, but OK, I'll take that
criticism onboard and in future will think more about how I bring
something to this list, and will take into account what you and Eli have
said. Thanks for sharing your thoughts above, both of you, in a
constructive way.

regards,
Eddie

On 04/04/2024 15:24, Eddie Chapman wrote:
> Since there appears to be some interest I'll put together a single email
> to the list later today detailing everything, as I needed to do more
> things overall in addition to replacing /usr/bin/xz.

Below is a guide I've written to removing app-arch/xz-utils in case
anyone else wants to do so. Attached is the current version of the Bash
wrapper script I now use in place of /usr/bin/xz

Comments, corrections on anything technical in the guide or script are
welcome, apart from flames about how this is ridiculous and unnecessary :-).

Best wishes,
Eddie


==== Guide to removing xz utils on a Gentoo system ====

=== Introduction ===

This guide is for people who wish to remove xz utils (app-arch/xz-utils)
from a Gentoo system.

I've been able to remove xz utils from two Gentoo workstations with 2412
packages and KDE 5.x as the desktop, and it has not been painful at all.
I've gone on to remove it from several Gentoo server systems without any
pain. These are all SElinux systems.

In this guide we replace app-arch/xz-utils with app-arch/p7zip which
will do all the work of uncompressing xz distfiles for Portage going
forward. It works perfectly fine for that right now.

I've written a bash wrapper script which is designed to be installed as
/usr/bin/xz, which is referred to in the instructions below. It is
attached to this email as xz.txt. It tries to takes care of
decompressing .xz files transparently whenever Portage runs /usr/bin/xz,
by behaving like it but using app-arch/p7zip in the background. You will
need it if you want to get rid of app-arch/xz-utils. But don't blindly
use it, check it yourself first of course. If you don't like it you will
either need to write your own script, or hack emerge/Portage in various
places to use something else to decompress xz files.

You're mileage may vary with any of this, proceed at your own risk,
don't blame me if you break your system or lose data.


=== Warnings / Caveats / Breakages ===

Before you do this, you should identify whether you have applications or
scripts which use the Tukaani xz utils, or that link against
liblzma.so.5. This could include non-Gentoo apps or scripts you run
which call any of the xz utils (xz, unxz, xzgrep|xzegrep|xzfgrep, xzcat,
xzcmp, xzdec, xzdiff, lzma, unlzma, lzgrep|lzegrep|lzfgrep, lzmainfo,
lzmadec, lzcmp, lzdiff, lzcat). Those programs will all be gone, so you
should not do this if you want or need them and cannot use alternatives.

99% of packages in Gentoo work fine without xz utils, it's just that
some might optionally link against liblzma.so.5 in order to provide
support for xz (de)compression along with other algorithms. We will
rebuild those packages so they don't link against liblzma.so.5 anymore.

xx utils is a relative newcomer to the Linux/OSS/GNU world so you will
find there aren't any low level system packages that absolutely need it
to do their main job. You are highly unlikely to render your system
completely unbootable doing this.

But removing it does carry some risk. You might discover along the way
there is some application you have installed that cannot function
without xz utils. You might just have to uninstall it and find an
alternative, if the situation cannot be resolved by creating your own
custom ebuild and tweaking configure/meson options. But worst case if
you have to uninstall a package and other packages depend on it, you
might have to remove them too, and I'm sure you know how that remove
list can potentially turn into a long one once all deps are worked out.

You will lose some things. I've had to uninstall the following two
packages for now:

media-gfx/gimp
kde-apps/ark (and kde-apps/kdeutils-meta which depends on it)

(I'll probably figure out later how to coax them into working without
xz. There might even be upstream updates soon that make xz optional, who
knows. I'll also need to add to my world file at some point everything
that was in kde-apps/kdeutils-meta.)

If you run another desktop (e.g. Gnome) I've no idea what might or might
not need xz utils. The situation with your desktop environment may be
worse, more painful, or impossible.

You will lose lzma support in the core Python language (dev-lang/python)
in 3.x versions and higher (not sure when exactly support was introduced
but 2.7 does not have it, 3.11 & 3.12 do), so if you have python scripts
that happen to need that, well, they will definitely throw a big error
after this :-) But I was able to rebuild the 179 dev-python packages on
my workstations and everything in app-portage and none of them
complained. I've been able to go on and do plenty of rebuilding with
Portage after this without any problem, so core Python functionality in
Gentoo is fine (although see next paragraph about Gemato).

There is one significant thing that breaks, which is Gemato
(app-portage/gemato). Gemato requires lzma support in core python in
order to do GPG signature verification. This means you will have to say
goodbye (for now) to verifying upstream GPG signatures on distfiles, and
verification of Portage metadata after doing an emerge --sync. These
features have been added to Portage relatively recently (2022?) so are
"nice to have", without them your system is just less hardened, but
still with the very high level of security that Gentoo systems have has
always had prior to these features, in my opinion. Personally I can live
without them for now. Verifying hashes in Manifest files still works
fine and that's the main thing. You may disagree in which case, well,
don't do this then. I'm going to figure out an alternative way I can
verify Portage metadata soon, as there are other ways if you are creative.

In practise this means you have to use USE="-verify-sig" for every
emerge with a package that has a corresponding sec-keys package, and you
have to set:

sync-rsync-verify-metamanifest = no

in files in /etc/portage/repos.conf/

But after doing that all works fine.

Here's some other very minor things you might lose if you are currently
using them:

- KDE users will lose xz compression support from KArchive
(kde-frameworks/karchive). AFAICT this has NOT had any impact on my own
KDE experience, I've not seen any errors and everything I use works fine
in my KDE sessions. KArchive will still support GZip, BZip2 and Zstd,
just not xz. I suspect nothing that uses KArchive is using xz by
default, but I'm not completely sure. All I know is my KDE sessions are
running fine without it, and I can do everything in KDE I did before
(apart from use Ark of course, see above). I don't know anything about
KArchive. Full details of compression support in KArchive are at
https://api.kde.org/frameworks/karchive/html/classKCompressionDevice.html

- Portage binary packages: You cannot use xz compression if you create
Portage binary packages. You will need to use one of bzip2, gzip, lz4,
lzip, lzop, or zstd in BINPKG_COMPRESS in make.conf instead of xz (if
that is what you were using, or is it the default?). I have always used
gzip so no probs for me, creating binary packages works fine, I've
already updated several Gentoo systems from many binary packages I've
created using gzip without xz utils installed.

- Grub bootloader: If you happened to have been using the optional, not
used by default, --compress argument for grub-install, and you happen to
have chosen xz, well you can't anymore. You will have to use gz or lzo
instead, or stop using --compress if you don't like either of those two.
Grub still builds, installs, works fine without xz utils for almost
everyone. But if you did happen to previously use --compress=xz with
grub-install before, make sure you check out fully what you might or
might not have to do before next rebooting (I have no idea, I have never
used this feature, Grub has continued working fine for me after
rebuilding it without xz-utils and running grub-install again on my boot
drives).

- Dovecot: net-mail/dovecot links liblzma.so.5 in order to support it's
optional Zlib plugin (
https://doc.dovecot.org/configuration_manual/zlib_plugin/ ) for
reading/writing compressed mail files. Despite the plugin being called
"Zlib" it supports several different compression algorithms. At one time
they supported xz, but in recent Dovecot releases they decided to
deprecate it. They still support reading (not writing) xz compressed
files, so when net-mail/dovecot is built, if it finds liblzma.so.5 it
will use it, if it doesn't find it, it wont, and then you just have no
support for xz in the Zlib plugin (again, only *if* you are using that
plugin, which is not default). From what I can gather, if you use this
plugin you should migrate away from using xz compressed mail files (to
another supported compression). So you should do that before you do
this, if that applies to you. I use Dovecot but never enabled mail file
compression so this did not affect me, Dovecot has continued working
fine with the mail stores I look after.

- Mariadb: If you happen to make use of the optional InnoDB Page
Compression feature in Mariadb (
https://mariadb.com/kb/en/innodb-page-compression/ ), and if you happen
to have chosen lzma compression for that feature (not the default)
rather than one of the other 5 algorithms, then that is very unlucky,
you will need to change that in your MariaDB installation in order to
use one of the other 5 compression algorithms instead. dev-db/mariadb
during build will automatically pick up support for the compression
algorithms you have installed on the system, you don't currently specify
anything in the ebuild that affects that. So if you have dev-db/mariadb
installed you will have to rebuild it after removing xz utils as it
links against liblzma.so.5 for this feature, and on rebuilding it you
will lose support for lzma in InnoDB Page Compression. If you don't know
if you are using it, this sql query will tell you:

SHOW GLOBAL VARIABLES LIKE 'innodb_compression_algorithm';

On my MariaDB 10.6 server it returned:

+------------------------------+-------+
| Variable_name | Value |
+------------------------------+-------+
| innodb_compression_algorithm | zlib |
+------------------------------+-------+

So I was not using lzma and was not affected. Tested: my MariaDB 10.6
server is now using rebuilt dev-db/mariadb without liblzma.so.5 and is
running with no problems.

- sys-apps/fwupd might stop working properly (though it will still build
fine) due to what you have to change with dev-libs/libxmlb below. I'm
not sure as I haven't checked yet, I just suspect it will. So bear that
in mind if you need to rely on sys-apps/fwupd at the moment. But this
"might" is temporary, upstream has now decided to make lzma optional, so
this will trickle down to Gentoo soon.

- app-arch/rpm will probably not be able to extract some rpm archives if
they are compressed with xz, but I haven't checked that yet. Though it
will still build fine. This does not affect building Gentoo packages
which come with .rpm distfiles (e.g. libreoffice), Portage uses rpm2tgz
for that and my script takes care of the rest.


=== The instructions ===

Follow them in order.

1. Do an emerge --sync and @world update first to make sure any
upgrades/updates have been applied. Makes it easier for the
things you need to do after you remove xz utils.

2. Install p7zip: emerge app-arch/p7zip

3. Add -lzma to USE flags in make.conf

4. Rebuild @world. This will rebuild only a few packages which
respect -lzma

5. Copy the bash wrapper script to somewhere on the machine you
are doing this on (but NOT to /usr/bin yet)

6. Prepare the script to be installed. Rename it to "xz" (with
no extension), set permissions to 0755, owned by root:root.

7. On an SElinux installation, set the SElinux context of the
script to whatever the current /usr/bin/xz binary is set to.

8. Remove xz utils, ignoring the warning about it being part
of system: emerge --unmerge app-arch/xz-utils
Once it is removed Portage will tell you that it preserved
liblzma.so.5. More on that below.

9. Install the bash wrapper script to /usr/bin/xz

10. Add the following line:
app-arch/xz-utils-5.4.2
to /etc/portage/profile/package.provided

11. Remove kde-apps/kdeutils-meta and kde-apps/ark if you use
KDE, and media-gfx/gimp if you use it:
emerge --unmerge kde-apps/kdeutils-meta kde-apps/ark media-gfx/gimp

12. (optional) Add -verify-sig to USE flags in make.conf. If you
do you will soon have to rebuild all packages that rely on it.
If you don't, you can just add USE="-verify-sig" in front of
every emerge command you have to do from now on, or add to
individual packages in your package.use file.

13. Now you will need to rebuild all packages with files that rely
on the preserved liblzma.so.5 library. See below for further
notes about that.

14. set:
sync-rsync-verify-metamanifest = no
in applicable files in /etc/portage/repos.conf/ before you do
your next emerge --sync

15. Eventually, you will have to rebuild all packages that have
corresponding signatures in sec-keys.

That's all, enjoy life without app-arch/xz-utils! But read on for more
info about step 13.


=== Notes about Step 13 ===

These are the packages that I needed to rebuild on my systems before the
preserved liblzma.so.5 library was finally removed by Portage:

app-arch/libarchive
app-arch/rpm
sys-boot/grub
dev-db/mariadb
dev-lang/python:2.7
kde-frameworks/karchive
dev-lang/python:3.11 (needs custom ebuild, see below)
dev-lang/python:3.12 (needs custom ebuild, see below)
net-mail/dovecot (needs custom ebuild, see below)
dev-libs/libxmlb (needs custom ebuild, see last note at the bottom of
this guide)

There might be others on your system. In most cases just rebuilding them
will be enough. Some you might be able to clone the ebuild to your local
repo and tweak configure/meson options so that the package does not link
against liblzma.so.5. There may be packages with issues too difficult to
resolve so you might have to just uninstall them if you can't live
without them :-( (or resign yourself to rolling back and having to live
with xz utils)

Remember you will need to specify USE="-verify-sig" for any packages
that rely on that, in whichever is your preferred way.

From my list I had to clone the following 3 packages to my local
ebuilds directory with small modification to each in order to get them
to build without linking against liblzma.so.5:
net-mail/dovecot
dev-lang/python:3.11
dev-lang/python:3.12

Here are 3 diffs showing what I had to change:

--- /usr/portage/net-mail/dovecot/dovecot-2.3.21-r1.ebuild
+++ /usr/local/portage/net-mail/dovecot/dovecot-2.3.21-r1.ebuild
@@ -43,7 +43,6 @@

DEPEND="
app-arch/bzip2
- app-arch/xz-utils
dev-libs/icu:=
dev-libs/openssl:0=
sys-libs/zlib:=
@@ -126,7 +125,7 @@
--disable-rpath \
--with-bzlib \
--without-libbsd \
- --with-lzma \
+ --without-lzma \
--with-icu \
--with-ssl \
--with-zlib \

--- /usr/portage/dev-lang/python/python-3.11.8_p1.ebuild
+++ /usr/local/portage/dev-lang/python/python-3.11.8_p1.ebuild
@@ -179,6 +179,7 @@
# Avoid as many dependencies as possible for the cross build.
cat >> Makefile <<-EOF || die
MODULE_NIS_STATE=disabled
+ MODULE__LZMA_STATE=disabled
MODULE__DBM_STATE=disabled
MODULE__GDBM_STATE=disabled
MODULE__DBM_STATE=disabled
@@ -328,7 +329,7 @@
fi

# force-disable modules we don't want built
- local disable_modules=( NIS )
+ local disable_modules=( NIS _LZMA )
use gdbm || disable_modules+=( _GDBM _DBM )
use sqlite || disable_modules+=( _SQLITE3 )
use ssl || disable_modules+=( _HASHLIB _SSL )


--- /usr/portage/dev-lang/python/python-3.12.2_p1.ebuild
+++ /usr/local/portage/dev-lang/python/python-3.12.2_p1.ebuild
@@ -177,6 +177,7 @@
cat > Modules/Setup.local <<-EOF || die
*disabled*
nis
+ _lzma
_dbm _gdbm
_sqlite3
_hashlib _ssl
@@ -299,6 +300,7 @@
cat > Modules/Setup.local <<-EOF || die
*disabled*
nis
+ _lzma
$(usev !gdbm '_gdbm _dbm')
$(usev !sqlite '_sqlite3')
$(usev !ssl '_hashlib _ssl')


Lastly, I needed to create a custom dev-libs/libxmlb ebuild in order to
upgrade it from 0.3.14 (latest in Gentoo at time of writing) to 0.3.15.

I also needed to apply a very recent patch from upstream, from this
commit, which makes LZMA support optional:
https://github.com/hughsie/libxmlb/commit/bdf845510fbed40b88465b2272ccad9e93656639

and I needed to make some small changes to the ebuild.

So this is what you need to do at the time of writing (6th April 2024):

1. Copy the in-tree /usr/portage/dev-libs/libxmlb ebuild directory into
your local ebuilds directory.

2. Rename the ebuild file from libxmlb-0.3.14.ebuild to
libxmlb-0.3.15.ebuild

3. Download the raw patch, you can use this link:

https://github.com/hughsie/libxmlb/commit/bdf845510fbed40b88465b2272ccad9e93656639.patch
rename it to:
libxmlb-0.3.15-make_lzma_optional.patch
and place it in the local "files" directory.

4. Modify the new ebuild according to the diff below. Then just rebuild it.

--- /usr/portage/dev-libs/libxmlb/libxmlb-0.3.14.ebuild
+++ /usr/local/portage/dev-libs/libxmlb/libxmlb-0.3.15.ebuild
@@ -14,15 +14,15 @@
SLOT="0/2" # libxmlb.so version

KEYWORDS="amd64 ~arm arm64 ~loong ppc ppc64 ~riscv x86"
-IUSE="doc introspection stemmer test +zstd"
+IUSE="doc introspection -lzma stemmer test +zstd"

RESTRICT="!test? ( test )"

RDEPEND="
- app-arch/xz-utils
dev-libs/glib:2
sys-apps/util-linux
stemmer? ( dev-libs/snowball-stemmer:= )
+ lzma? ( app-arch/xz-utils:= )
zstd? ( app-arch/zstd:= )
"

@@ -43,6 +43,7 @@

PATCHES=(
"${FILESDIR}"/${PN}-0.3.12-no_installed_tests.patch
+ "${FILESDIR}"/${PN}-0.3.15-make_lzma_optional.patch
)

python_check_deps() {
@@ -60,6 +61,7 @@
$(meson_use stemmer)
$(meson_use test tests)
$(meson_use zstd)
+ $(meson_feature lzma)
)
meson_src_configure
}

>>>>> On Sat, 06 Apr 2024, Eddie Chapman wrote:

> [...] this is ridiculous and unnecessary :-).

Indeed.

SCNR,
Ulrich

On 2024.04.06 12:57, Eddie Chapman wrote:
> On 04/04/2024 15:24, Eddie Chapman wrote:
> > Since there appears to be some interest I'll put together a single
> email
> > to the list later today detailing everything, as I needed to do more
> > things overall in addition to replacing /usr/bin/xz.
>
> Below is a guide I've written to removing app-arch/xz-utils in case
> anyone else wants to do so. Attached is the current version of the
> Bash
> wrapper script I now use in place of /usr/bin/xz
>
> Comments, corrections on anything technical in the guide or script are
>
> welcome, apart from flames about how this is ridiculous and
> unnecessary :-).
>
> Best wishes,
> Eddie
>
>[snip method]

"Because I can" is a good enough reason to do anything with Gentoo.

--
Regards,

Roy Bamford
(Neddyseagoon) a member of
elections
gentoo-ops
forum-mods
arm64

On 06-04-2024 12:57:23 +0100, Eddie Chapman wrote:
> There is one significant thing that breaks, which is Gemato
> (app-portage/gemato). Gemato requires lzma support in core python in
> order to do GPG signature verification. This means you will have to say
> goodbye (for now) to verifying upstream GPG signatures on distfiles, and
> verification of Portage metadata after doing an emerge --sync. These
> features have been added to Portage relatively recently (2022?) so are
> "nice to have", without them your system is just less hardened, but
> still with the very high level of security that Gentoo systems have has
> always had prior to these features, in my opinion. Personally I can live
> without them for now. Verifying hashes in Manifest files still works
> fine and that's the main thing. You may disagree in which case, well,
> don't do this then. I'm going to figure out an alternative way I can
> verify Portage metadata soon, as there are other ways if you are creative.

If you just want to verify signatures and manifests after sync,
qmanifest from portage-utils can help you do this.

Thanks,
Fabian


--
Fabian Groffen
Gentoo on a different level

Eddie Chapman <eddie@ehuk.net> writes:

> On 04/04/2024 15:24, Eddie Chapman wrote:
>> Since there appears to be some interest I'll put together a single email
>> to the list later today detailing everything, as I needed to do more
>> things overall in addition to replacing /usr/bin/xz.
>
> Below is a guide I've written to removing app-arch/xz-utils in case
> anyone else wants to do so. Attached is the current version of the
> Bash wrapper script I now use in place of /usr/bin/xz
>
> Comments, corrections on anything technical in the guide or script are
> welcome, apart from flames about how this is ridiculous and
> unnecessary :-).

For an experiment I'm doing (distinct from trying to purge xz-utils,
just verification work), I've packaged the following:
* app-arch/gxz (pure Go impl.)
* app-arch/7zip (7zip upstream are supporting Linux now, app-arch/p7zip
was an unofficial port)

You might find those useful too.

At a glance, it appears https://github.com/fpgaminer/rust-lzma and
https://github.com/gendx/lzma-rs don't provide executables - just a
library - so I didn't bother looking further.

>
> Best wishes,
> Eddie
>
>
> ==== Guide to removing xz utils on a Gentoo system ====
>
> [...]
> There is one significant thing that breaks, which is Gemato
> (app-portage/gemato). Gemato requires lzma support in core python in
> order to do GPG signature verification. This means you will have to
> say goodbye (for now) to verifying upstream GPG signatures on
> distfiles, and verification of Portage metadata after doing an emerge
> --sync. These features have been added to Portage relatively recently
> (2022?) so are "nice to have", without them your system is just less

No.. much older. It was introduced around the time of the github mirror
being hacked. It's not just theatre!

Like, this is very much NOT hypothetical.

It's not just about metadata, it's about the ebuilds if using rsync, or
the whole git checkout if using git.

> hardened, but still with the very high level of security that Gentoo
> systems have has always had prior to these features, in my
> opinion. Personally I can live without them for now. Verifying hashes
> in Manifest files still works fine and that's the main thing. You may
> disagree in which case, well, don't do this then. I'm going to figure
> out an alternative way I can verify Portage metadata soon, as there
> are other ways if you are creative.

See grobian's reply which should help.

> [...]
> - Portage binary packages: You cannot use xz compression if you create
> Portage binary packages. You will need to use one of bzip2, gzip,
> lz4, lzip, lzop, or zstd in BINPKG_COMPRESS in make.conf instead of
> xz (if that is what you were using, or is it the default?). I have

zstd is the default for "new" installs (since a few years ago), yeah.

> [...]
> - sys-apps/fwupd might stop working properly (though it will still
> build fine) due to what you have to change with dev-libs/libxmlb
> below. I'm not sure as I haven't checked yet, I just suspect it
> will. So bear that in mind if you need to rely on sys-apps/fwupd at
> the moment. But this "might" is temporary, upstream has now decided
> to make lzma optional, so this will trickle down to Gentoo soon.

Just for completeness, this is
https://blogs.gnome.org/hughsie/2024/04/03/fwupd-and-xz-metadata/.

> [...]

Fabian Groffen wrote:
> If you just want to verify signatures and manifests after sync,
> qmanifest from portage-utils can help you do this.
>
> Thanks,
> Fabian

Thanks for the pointer, and I see you are one of the authors, thanks for
writing a very useful tool!

Sam James wrote:
> Eddie Chapman <eddie@ehuk.net> writes:
>> Below is a guide I've written to removing app-arch/xz-utils in case
>> anyone else wants to do so. Attached is the current version of the Bash
>> wrapper script I now use in place of /usr/bin/xz
>>
>> Comments, corrections on anything technical in the guide or script are
>> welcome, apart from flames about how this is ridiculous and unnecessary
>> :-).
>
> For an experiment I'm doing (distinct from trying to purge xz-utils,
> just verification work), I've packaged the following: * app-arch/gxz (pure
> Go impl.)
> * app-arch/7zip (7zip upstream are supporting Linux now, app-arch/p7zip
> was an unofficial port)
>
> You might find those useful too.

That's fantastic. I wrote about p7zip vs. upstream 7zip in another mail in
this thread and was intending to create a local ebuild for 7zip soon but
won't have to know it's in tree :-)

> At a glance, it appears https://github.com/fpgaminer/rust-lzma and
> https://github.com/gendx/lzma-rs don't provide executables - just a
> library - so I didn't bother looking further.
>>
>> ==== Guide to removing xz utils on a Gentoo system ====
>>
>> [...]
>> There is one significant thing that breaks, which is Gemato
>> (app-portage/gemato). Gemato requires lzma support in core python in
>> order to do GPG signature verification. This means you will have to say
>> goodbye (for now) to verifying upstream GPG signatures on distfiles, and
>> verification of Portage metadata after doing an emerge --sync. These
>> features have been added to Portage relatively recently (2022?) so are
>> "nice to have", without them your system is just less
>
> No.. much older. It was introduced around the time of the github mirror
> being hacked. It's not just theatre!
>
> Like, this is very much NOT hypothetical.

Thanks, couldn't remember when it was.

> It's not just about metadata, it's about the ebuilds if using rsync, or
> the whole git checkout if using git.

Completely agree with you that this was a great feature to be added from a
security point of view. Without it there was still a level of trust,
however small, that could be placed in the choice of mirror. But there's
no doubt gpg sigs of repo data are order of magnitudes better, so yes it
was a little unjust to describe it as only "nice to have".

But in the current situation I personally consider it so critically
important to get rid of xz utils from my systems that a short, temporary
period of not having this while switching to another method of
verification I consider an acceptable tradeoff (side note to anyone
reading: yes I know how at odds I am with the rest of the world on this,
it has now been argued to death in this thread so for anyone thinking
about replying about that, maybe lets do everyone a favour, agree to
disagree, and move on :-) )

>> hardened, but still with the very high level of security that Gentoo
>> systems have has always had prior to these features, in my opinion.
>> Personally I can live without them for now. Verifying hashes
>> in Manifest files still works fine and that's the main thing. You may
>> disagree in which case, well, don't do this then. I'm going to figure
>> out an alternative way I can verify Portage metadata soon, as there are
>> other ways if you are creative.
>
> See grobian's reply which should help.
>
>
>> [...]
>> - Portage binary packages: You cannot use xz compression if you create
>> Portage binary packages. You will need to use one of bzip2, gzip,
>> lz4, lzip, lzop, or zstd in BINPKG_COMPRESS in make.conf instead of xz
>> (if that is what you were using, or is it the default?). I have
>>
>
> zstd is the default for "new" installs (since a few years ago), yeah.
>
>> [...]
>> - sys-apps/fwupd might stop working properly (though it will still
>> build fine) due to what you have to change with dev-libs/libxmlb below.
>> I'm not sure as I haven't checked yet, I just suspect it
>> will. So bear that in mind if you need to rely on sys-apps/fwupd at the
>> moment. But this "might" is temporary, upstream has now decided to make
>> lzma optional, so this will trickle down to Gentoo soon.
>
> Just for completeness, this is
> https://blogs.gnome.org/hughsie/2024/04/03/fwupd-and-xz-metadata/.
>

Thanks for all the useful additions of info :-)

cheers,
Eddie

On 6.4.2024 14.57, Eddie Chapman wrote:
>
> --- /usr/portage/net-mail/dovecot/dovecot-2.3.21-r1.ebuild
> +++ /usr/local/portage/net-mail/dovecot/dovecot-2.3.21-r1.ebuild
> @@ -43,7 +43,6 @@
>
>  DEPEND="
>         app-arch/bzip2
> -       app-arch/xz-utils
>         dev-libs/icu:=
>         dev-libs/openssl:0=
>         sys-libs/zlib:=
> @@ -126,7 +125,7 @@
>                 --disable-rpath \
>                 --with-bzlib \
>                 --without-libbsd \
> -               --with-lzma \
> +               --without-lzma \
>                 --with-icu \
>                 --with-ssl \
>                 --with-zlib \
>
> --- /usr/portage/dev-lang/python/python-3.11.8_p1.ebuild
> +++ /usr/local/portage/dev-lang/python/python-3.11.8_p1.ebuild
> @@ -179,6 +179,7 @@
>         # Avoid as many dependencies as possible for the cross build.
>         cat >> Makefile <<-EOF || die
>                 MODULE_NIS_STATE=disabled
> +               MODULE__LZMA_STATE=disabled
>                 MODULE__DBM_STATE=disabled
>                 MODULE__GDBM_STATE=disabled
>                 MODULE__DBM_STATE=disabled
> @@ -328,7 +329,7 @@
>         fi
>
>         # force-disable modules we don't want built
> -       local disable_modules=( NIS )
> +       local disable_modules=( NIS _LZMA )
>         use gdbm || disable_modules+=( _GDBM _DBM )
>         use sqlite || disable_modules+=( _SQLITE3 )
>         use ssl || disable_modules+=( _HASHLIB _SSL )
>
>
> --- /usr/portage/dev-lang/python/python-3.12.2_p1.ebuild
> +++ /usr/local/portage/dev-lang/python/python-3.12.2_p1.ebuild
> @@ -177,6 +177,7 @@
>         cat > Modules/Setup.local <<-EOF || die
>                 *disabled*
>                 nis
> +               _lzma
>                 _dbm _gdbm
>                 _sqlite3
>                 _hashlib _ssl
> @@ -299,6 +300,7 @@
>         cat > Modules/Setup.local <<-EOF || die
>                 *disabled*
>                 nis
> +               _lzma
>                 $(usev !gdbm '_gdbm _dbm')
>                 $(usev !sqlite '_sqlite3')
>                 $(usev !ssl '_hashlib _ssl')
>
>
> Lastly, I needed to create a custom dev-libs/libxmlb ebuild in order to
> upgrade it from 0.3.14 (latest in Gentoo at time of writing) to 0.3.15.
>
> I also needed to apply a very recent patch from upstream, from this
> commit, which makes LZMA support optional:
> https://github.com/hughsie/libxmlb/commit/bdf845510fbed40b88465b2272ccad9e93656639
>
> and I needed to make some small changes to the ebuild.
>
> So this is what you need to do at the time of writing (6th April 2024):
>
> 1. Copy the in-tree /usr/portage/dev-libs/libxmlb ebuild directory into
> your local ebuilds directory.
>
> 2. Rename the ebuild file from libxmlb-0.3.14.ebuild to
> libxmlb-0.3.15.ebuild
>
> 3. Download the raw patch, you can use this link:
>
> https://github.com/hughsie/libxmlb/commit/bdf845510fbed40b88465b2272ccad9e93656639.patch
>    rename it to:
>    libxmlb-0.3.15-make_lzma_optional.patch
>    and place it in the local "files" directory.
>
> 4. Modify the new ebuild according to the diff below. Then just rebuild it.
>
> --- /usr/portage/dev-libs/libxmlb/libxmlb-0.3.14.ebuild
> +++ /usr/local/portage/dev-libs/libxmlb/libxmlb-0.3.15.ebuild
> @@ -14,15 +14,15 @@
>  SLOT="0/2" # libxmlb.so version
>
>  KEYWORDS="amd64 ~arm arm64 ~loong ppc ppc64 ~riscv x86"
> -IUSE="doc introspection stemmer test +zstd"
> +IUSE="doc introspection -lzma stemmer test +zstd"
>
>  RESTRICT="!test? ( test )"
>
>  RDEPEND="
> -       app-arch/xz-utils
>         dev-libs/glib:2
>         sys-apps/util-linux
>         stemmer? ( dev-libs/snowball-stemmer:= )
> +       lzma? ( app-arch/xz-utils:= )
>         zstd? ( app-arch/zstd:= )
>  "
>
> @@ -43,6 +43,7 @@
>
>  PATCHES=(
>         "${FILESDIR}"/${PN}-0.3.12-no_installed_tests.patch
> +       "${FILESDIR}"/${PN}-0.3.15-make_lzma_optional.patch
>  )
>
>  python_check_deps() {
> @@ -60,6 +61,7 @@
>                 $(meson_use stemmer)
>                 $(meson_use test tests)
>                 $(meson_use zstd)
> +               $(meson_feature lzma)
>         )
>         meson_src_configure
>  }

Hey,

I'll admit I didn't read everything, but I just want to point out you
may not have to edit ebuilds at all. If xz-utils is package.provided
portage should ignore the dependency without you removing the dep from
an ebuild. Then you can utilize /etc/portage/patches to apply any
patches and finally try using EXTRA_ECONF and MYMESONARGS to override
configure options via package.env.

-- juippis

Joonas Niilola posted on Thu, 11 Apr 2024 08:21:39 +0300 as excerpted:

> I just want to point out you
> may not have to edit ebuilds at all. If xz-utils is package.provided
> portage should ignore the dependency without you removing the dep from
> an ebuild.

package.provided:

YMMV, but here rather than doing package.provided I create a "null-ebuild"
for the package in my overlay. Much like virtual/* packages from which I
took the idea except of course that they're named as the category/package
they're replacing instead of virtual/*, null-ebuilds install no files but
allow detailed control of IUSE, slot, etc, in case some of the revdeps
need that.

For versioning, my convention is -999 or -n.999 to imply a virtual (tho I
do have a real perl bigint package v 1.999.842 installed...), much like
the -9999/-n.9999 variants imply a live-package, with similar effect in
terms of preferring it to any reasonable real version number as well. So
for xz-utils it'd be app-arch/xz-utils-999 as it's not slotted, or app-
arch/xz-utils-5.999 if it were or if something wants 5.x specifically. Or
use five-nines or six-nines or ten nines...

A null-pkg I actually use here? sys-fs/udisks-2.999:2 (slot 2 dep actually
required by some of its rev-deps). udisks itself is a script so doesn't
provide headers necessary to build other things and should be a runtime-
only dep. As a script the installation itself would be too trivial to
bother with, were it not for its absolutely wicked pulled-in deps for
functionality I'm not going to use and don't have turned on for my kernel
in any case. Luckily kde/solid/kio/etc degrade functionality gracefully
if their attempted udisks calls return command-not-found, making it an
ideal candidate for null-pkging.

--
Duncan - List replies preferred. No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master." Richard Stallman

Joonas Niilola wrote:
> Hey,
>
> I'll admit I didn't read everything, but I just want to point out you
> may not have to edit ebuilds at all. If xz-utils is package.provided
> portage should ignore the dependency without you removing the dep from an
> ebuild. Then you can utilize /etc/portage/patches to apply any patches and
> finally try using EXTRA_ECONF and MYMESONARGS to override configure
> options via package.env.
>
> -- juippis

Hi Joonas,

The local ebuilds in the guide were not created because of the xz-utils
dep. If you search through ebuilds in the tree there are hundreds of
packages that specify xz-utils as a hard dep, so yes, as you say,
package.provided takes care of all of then.

No, the ebuilds were needed for various customisations to build
arguments. However, the dev-libs/libxmlb ebuild is no longer needed as,
since I wrote the guide, libxmlb 0.3.17, which makes liblzma.so.5 dep
optional, is now in Gentoo, thanks whoever added that :-)

You might be able to dispense with the need for the separate
net-mail/dovecot ebuild by using EXTRA_ECONF, as you say. However,
AFAICS local dev-lang/python ebuilds are unavoidable, unfortunately,
you'll see why if you look at the diffs for them in my guide. It would
be wonderful if dev-lang/python made its liblzma dep optional. It would
be a simple change to the ebuild. However, I suspect the developers
might feel that *not* depending on liblzma.so.5 is unsupported because
it results in Gemato failing due to lack of support in core python for
liblzma. The only way around that issue I can see is for Gemato to
instead use /usr/bin/xz like the rest of Portage does. If that were to
happen then dev-lang/python could be modified to respect -lzma and I
can't see that anything significant in Gentoo would miss it. Then if
there are any dev-python packages that need liblzma in core python
either presently or in future (I've not encountered any yet) then of
course they would just need a hard dependency on dev-lang/python with
lzma USE flag set.

I now have many Gentoo systems running without xz-utils installed (using
my wrapper script from the guide) and I've not had a single issue
anywhere, everything working perfectly, so I'm delighted that it has
been possible :-)

Eddie