Mailing List Archive

[EAPI 8 RFC] Selective fetch/mirror (un-)restriction
Hello, everyone.

I'd like to start a series of mails dedicated to features proposed for
including in EAPI 8. For a start, I'd like to discuss the topic of
selective fetch restriction [1]. It has been discussed at least in 2013
[2], and since it's finally got chance to be included, I think it's
worthwhile to rehash it.


The problem
===========
Fetch/mirror restriction as of now can only be applied to all distfiles
at once. This causes problems in the (rather rare) case when we'd like
to add some additional files to SRC_URI that do not require the big
restriction. As a result, the user is forced to manually fetch all
the files even if only one truly requires it.


Proposed solution
=================
The current proposal is based on extending the current URI syntax to
permit excluding individual files from the restriction. The idea is to
prepend 'fetch+' to protocol to undo fetch restriction, or to prepend
'mirror+' to undo fetch & mirror restrictions.

Example 1: removing mirror restriction from files

RESTRICT="mirror"
SRC_URI="https://example.com/you-cant-mirror-this.tar.bz2
mirror+https://example.com/but-you-can-mirror-this.tar.gz"

Example 2: removing fetch & mirror restriction from files

RESTRICT="fetch"
SRC_URI="https://example.com/you-cant-fetch-this.zip
mirror+https://example.com/but-you-can-mirror-this.tar.gz"

Example 3: removing fetch restriction while leaving mirror restriction

RESTRICT="fetch"
SRC_URI="https://example.com/you-cant-fetch-this.zip
fetch+https://example.com/you-cant-mirror-this.tar.bz2"


Comments
========
WDYT?


References
==========
[1] https://bugs.gentoo.org/371413
[2] https://archives.gentoo.org/gentoo-dev/message/b0823618d5d3cc61bbed1e88dc2f144d

--
Best regards,
Micha? Górny
Re: [EAPI 8 RFC] Selective fetch/mirror (un-)restriction [ In reply to ]
Il giorno lun 16 dic 2019 alle ore 13:39 Micha? Górny <mgorny@gentoo.org>
ha scritto:

>
>
> Comments
> ========
> WDYT?
>
> what about getting rid of RESTRICT="fetch" and manage everything inside
SRC_URI? Would that be technically feasible?
Ideally marking only the not re-distributable download and leaving
untouched the others
Re: [EAPI 8 RFC] Selective fetch/mirror (un-)restriction [ In reply to ]
>>>>> On Mon, 16 Dec 2019, Micha? Górny wrote:

> Proposed solution
> =================
> The current proposal is based on extending the current URI syntax to
> permit excluding individual files from the restriction. The idea is to
> prepend 'fetch+' to protocol to undo fetch restriction, or to prepend
> 'mirror+' to undo fetch & mirror restrictions.

> Example 1: removing mirror restriction from files

> RESTRICT="mirror"
> SRC_URI="https://example.com/you-cant-mirror-this.tar.bz2
> mirror+https://example.com/but-you-can-mirror-this.tar.gz"

> Example 2: removing fetch & mirror restriction from files

> RESTRICT="fetch"
> SRC_URI="https://example.com/you-cant-fetch-this.zip
> mirror+https://example.com/but-you-can-mirror-this.tar.gz"

> Example 3: removing fetch restriction while leaving mirror restriction

> RESTRICT="fetch"
> SRC_URI="https://example.com/you-cant-fetch-this.zip
> fetch+https://example.com/you-cant-mirror-this.tar.bz2"

Looks good, but what is slightly confusing is that it doesn't map
one-to-one to the RESTRICT tokens:

- RESTRICT="mirror" enables mirror restriction, and it is undone by
"mirror+", as expected.

- RESTRICT="fetch" enables both fetch and mirror restriction, but it is
undone by "mirror+" as well, not by "fetch+" (which disables only
fetch restriction).

I had already asked this in bug 371413 [1], but is there an actual usage
case for example 3? Because if there isn't, we might get away with only
supporting "mirror+", which should be less error prone.

Ulrich

> [1] https://bugs.gentoo.org/371413
Re: [EAPI 8 RFC] Selective fetch/mirror (un-)restriction [ In reply to ]
>>>>> On Mon, 16 Dec 2019, Francesco Riosa wrote:

> what about getting rid of RESTRICT="fetch" and manage everything
> inside SRC_URI? Would that be technically feasible? Ideally marking
> only the not re-distributable download and leaving untouched the
> others

That would have the disadvantage that mirror and bindist restrictions
(which are strongly correlated) would be listed in different places.

Ulrich
Re: [EAPI 8 RFC] Selective fetch/mirror (un-)restriction [ In reply to ]
On Mon, Dec 16, 2019 at 8:33 AM Ulrich Mueller <ulm@gentoo.org> wrote:
>
> >>>>> On Mon, 16 Dec 2019, Francesco Riosa wrote:
>
> > what about getting rid of RESTRICT="fetch" and manage everything
> > inside SRC_URI? Would that be technically feasible? Ideally marking
> > only the not re-distributable download and leaving untouched the
> > others
>
> That would have the disadvantage that mirror and bindist restrictions
> (which are strongly correlated) would be listed in different places.

An obvious way to address that is to also move the mirror restriction
into SRC_URI, so that they are both exclusively in this location as
well. I believe those are the only two redistribution-oriented
options for RESTRICT currently.

--
Rich
Re: [EAPI 8 RFC] Selective fetch/mirror (un-)restriction [ In reply to ]
On Mon, 2019-12-16 at 14:16 +0100, Ulrich Mueller wrote:
> > > > > > On Mon, 16 Dec 2019, Micha? Górny wrote:
> > Proposed solution
> > =================
> > The current proposal is based on extending the current URI syntax to
> > permit excluding individual files from the restriction. The idea is to
> > prepend 'fetch+' to protocol to undo fetch restriction, or to prepend
> > 'mirror+' to undo fetch & mirror restrictions.
> > Example 1: removing mirror restriction from files
> > RESTRICT="mirror"
> > SRC_URI="https://example.com/you-cant-mirror-this.tar.bz2
> > mirror+https://example.com/but-you-can-mirror-this.tar.gz"
> > Example 2: removing fetch & mirror restriction from files
> > RESTRICT="fetch"
> > SRC_URI="https://example.com/you-cant-fetch-this.zip
> > mirror+https://example.com/but-you-can-mirror-this.tar.gz"
> > Example 3: removing fetch restriction while leaving mirror restriction
> > RESTRICT="fetch"
> > SRC_URI="https://example.com/you-cant-fetch-this.zip
> > fetch+https://example.com/you-cant-mirror-this.tar.bz2"
>
> Looks good, but what is slightly confusing is that it doesn't map
> one-to-one to the RESTRICT tokens:
>
> - RESTRICT="mirror" enables mirror restriction, and it is undone by
> "mirror+", as expected.
>
> - RESTRICT="fetch" enables both fetch and mirror restriction, but it is
> undone by "mirror+" as well, not by "fetch+" (which disables only
> fetch restriction).
>
> I had already asked this in bug 371413 [1], but is there an actual usage
> case for example 3? Because if there isn't, we might get away with only
> supporting "mirror+", which should be less error prone.
>

Does this really solve the problem? The labels are still the other way
around, except that you throw 'fetch+' away as invalid.

While at it, I'm wondering if 'mirror+mirror://foo' can be confusing.

--
Best regards,
Micha? Górny
Re: [EAPI 8 RFC] Selective fetch/mirror (un-)restriction [ In reply to ]
Dnia December 16, 2019 1:16:12 PM UTC, Ulrich Mueller <ulm@gentoo.org> napisa?(a):
>>>>>> On Mon, 16 Dec 2019, Micha? Górny wrote:
>
>> Proposed solution
>> =================
>> The current proposal is based on extending the current URI syntax to
>> permit excluding individual files from the restriction. The idea is
>to
>> prepend 'fetch+' to protocol to undo fetch restriction, or to prepend
>> 'mirror+' to undo fetch & mirror restrictions.
>
>> Example 1: removing mirror restriction from files
>
>> RESTRICT="mirror"
>> SRC_URI="https://example.com/you-cant-mirror-this.tar.bz2
>> mirror+https://example.com/but-you-can-mirror-this.tar.gz"
>
>> Example 2: removing fetch & mirror restriction from files
>
>> RESTRICT="fetch"
>> SRC_URI="https://example.com/you-cant-fetch-this.zip
>> mirror+https://example.com/but-you-can-mirror-this.tar.gz"
>
>> Example 3: removing fetch restriction while leaving mirror
>restriction
>
>> RESTRICT="fetch"
>> SRC_URI="https://example.com/you-cant-fetch-this.zip
>> fetch+https://example.com/you-cant-mirror-this.tar.bz2"
>
>Looks good, but what is slightly confusing is that it doesn't map
>one-to-one to the RESTRICT tokens:
>
>- RESTRICT="mirror" enables mirror restriction, and it is undone by
> "mirror+", as expected.
>
>- RESTRICT="fetch" enables both fetch and mirror restriction, but it is
> undone by "mirror+" as well, not by "fetch+" (which disables only
> fetch restriction).
>
>I had already asked this in bug 371413 [1], but is there an actual
>usage
>case for example 3? Because if there isn't, we might get away with only
>supporting "mirror+", which should be less error prone.

Actually, what about the original example provided by Vadim? It's a game + translations, all rights reserved. We can't mirror them but we can fetch them.

>
>Ulrich
>
>> [1] https://bugs.gentoo.org/371413


--
Best regards,
Micha? Górny
Re: [EAPI 8 RFC] Selective fetch/mirror (un-)restriction [ In reply to ]
>>>>> On Fri, 20 Dec 2019, Micha? Górny wrote:

>>> Example 3: removing fetch restriction while leaving mirror
>>> restriction

>>> RESTRICT="fetch"
>>> SRC_URI="https://example.com/you-cant-fetch-this.zip
>>> fetch+https://example.com/you-cant-mirror-this.tar.bz2"

>> I had already asked this in bug 371413 [1], but is there an actual
>> usage case for example 3? Because if there isn't, we might get away
>> with only supporting "mirror+", which should be less error prone.

> Actually, what about the original example provided by Vadim? It's a
> game + translations, all rights reserved. We can't mirror them but we
> can fetch them.

Thank you. That answers my question.