Mailing List Archive

[RFC] Removing obsolete thick Manifest compatibility from MetaManifests
Hello,

TL;DR: I'd like to disable thick Manifest support in Portage, in order
to disable some of the compatibility quirks from MetaManifest format.
All files would still be verified by gemato.


We're using GLEP 74 MetaManifests for 2 years now. The specification
was originally written to account for compatibility with existing
(thick) Manifest files. I believe we can start considering removing
at least some of that compatibility today.

What I'd like to propose is disabling thick Manifests in the rsync
variant of Gentoo repository (in layout.conf). This would cause Portage
to stop verifying file entries directly (on-sync verification via gemato
would still happen). Notably, this would limit the needed compatibility
to DIST entries.

Why?

1. Thick Manifest verification happening in Portage is mostly redundant
these days, and when it's not its advantages are weak.

1a. Majority of Portage users are using on-sync verification via gemato.
In this case, repeated partial checks from Portage are at most
redundant.

1b. While not using gemato, Portage verifies only leaf Manifests without
checking the OpenPGP signature. There's no real security gain in this.

1c. With transmission-level checksumming (and filesystem-level checksums
becoming more common), there is no reason to assume we need to verify
integrity of rsync result.

2. Thick Manifest support in Portage is still relying on legacy entries.
While technically we could either make Portage use gemato fully, or
reimplement the new features, I don't think it's worth the effort given
the above.

2a. Removing legacy entries from ::gentoo will make it possible to
remove the backwards compatibility code from gemato. We may also remove
some of the redundant code from Portage.

2b. We will gain the ability to use the new format more efficiently.
In particular, I'm considering moving non-DIST entries to category-level
Manifests and taking advantage of compression (but I don't know if it's
going to provide real gain at the moment).

3. Thick Manifests are generally PITA to power users and developers.

3a. You need to regenerate them every time you edit an ebuild. It's
like having to type 'yes, I really wanted to edit that file' every time.

3b. You need to regenerate Manifests when moving ebuilds between git
and rsync checkouts.

3c. Proxied maintainers keep forgetting about that and submitting thick
Manifests.


WDYT?

--
Best regards,
Micha? Górny
Re: [RFC] Removing obsolete thick Manifest compatibility from MetaManifests [ In reply to ]
Hi,

On 10/24/19 2:37 PM, Micha? Górny wrote:
> Hello,
>
> TL;DR: I'd like to disable thick Manifest support in Portage, in order
> to disable some of the compatibility quirks from MetaManifest format.
> All files would still be verified by gemato.

>
> WDYT?
>

I'm using Gentoo Prefix as a Meta Distribution, where I can not use the
Gentoo infrastructure, but have to provide the master services on my own.

While I've been able to set up rsync & distfiles master ~10 years ago,
where I haven't discovered any docs for so far, I feel lost when I think
of setting up additional master services for gemato without some docs.

Facing the removal of thick Manifest support (which I don't want to block
in any way), I've started to search for some docs how to set up these
additional master services to support gemato, but failed.

What I've found is the Infrastructure wiki project, providing a list of
servers maintained by infra, but nothing about how to set them up.

Did I miss something?

Thanks!
/haubi/
Re: Re: [RFC] Removing obsolete thick Manifest compatibility from MetaManifests [ In reply to ]
On Tue, 2019-11-05 at 14:01 +0100, Michael Haubenwallner wrote:
> Hi,
>
> On 10/24/19 2:37 PM, Micha? Górny wrote:
> > Hello,
> >
> > TL;DR: I'd like to disable thick Manifest support in Portage, in order
> > to disable some of the compatibility quirks from MetaManifest format.
> > All files would still be verified by gemato.
> > WDYT?
> >
>
> I'm using Gentoo Prefix as a Meta Distribution, where I can not use the
> Gentoo infrastructure, but have to provide the master services on my own.
>
> While I've been able to set up rsync & distfiles master ~10 years ago,
> where I haven't discovered any docs for so far, I feel lost when I think
> of setting up additional master services for gemato without some docs.
>
> Facing the removal of thick Manifest support (which I don't want to block
> in any way), I've started to search for some docs how to set up these
> additional master services to support gemato, but failed.
>
> What I've found is the Infrastructure wiki project, providing a list of
> servers maintained by infra, but nothing about how to set them up.
>
> Did I miss something?
>

gemato has commands to create and update MetaManifests. Also there's
(much faster) trivial script in utils/ directory of gemato.

The relevant infra script is:
https://gitweb.gentoo.org/infra/mastermirror-scripts.git/tree/rsync-gen.sh#n243

--
Best regards,
Micha? Górny