Mailing List Archive

On Monday, 4 September 2023 11:12:51 BST Walter Dnes wrote:
> I may be misunderstanding, but it seems to me that local mirrors are
> all-or-nothing. In the interests of saving bandwidth, I'd like to have
> a client first check the host's /var/cache/distfiles directory for a
> source tarball file. If not found, then fail over to another mirror as
> per GENTOO_MIRRORS in /etc/portage/make.conf. This would require emerge
> doing the lookup and potential failover for each file. Is this possible?

Unless I misunderstand what you're asking, isn't this what takes place anyway?

Emerge will look at /var/cache/distfiles and if the source tarball is missing,
will try to fetch it from the list of GENTOO_MIRRORS you have configured. For
example, if you have:

GENTOO_MIRRORS="http://192.168.2.35/my_gentoo_distfiles/ https://
www.mirrorservice.org/sites/distfiles.gentoo.org/"

any host configured as above will first look at its own /var/cache/distfiles,
then at the first configured mirror which happens to be in the same LAN, then
whatever other external mirrors you have added. You could of course export
your local mirror's LAN ${DISTDIR} via NFS for your hosts to use.

PS. Is http_replicator still available/maintained? I see the wiki mentions
apt-cacher-ng for local distfiles cache.

On Mon, 04 Sep 2023 14:04:53 +0100, Michael wrote:

> On Monday, 4 September 2023 11:12:51 BST Walter Dnes wrote:
> > I may be misunderstanding, but it seems to me that local mirrors are
> > all-or-nothing. In the interests of saving bandwidth, I'd like to
> > have a client first check the host's /var/cache/distfiles directory
> > for a source tarball file. If not found, then fail over to another
> > mirror as per GENTOO_MIRRORS in /etc/portage/make.conf. This would
> > require emerge doing the lookup and potential failover for each file.
> > Is this possible?
>
> Unless I misunderstand what you're asking, isn't this what takes place
> anyway?

The reference to a host makes me think Walter wants one machne to hold
the distfiles for all on the network.

> PS. Is http_replicator still available/maintained? I see the wiki
> mentions apt-cacher-ng for local distfiles cache.

I use apt-cacher-ng for this and it does what you are looking for.


--
Neil Bothwick

The facts, although interesting, are usually irrelevant.

Quick n dirty solution:

put all distfiles on a central server
FS mount that remote dir to /var/cache/distfiles on all hosts

Alan

On Mon, Sep 4, 2023 at 7:21?PM Neil Bothwick <neil@digimed.co.uk> wrote:

> On Mon, 04 Sep 2023 14:04:53 +0100, Michael wrote:
>
> > On Monday, 4 September 2023 11:12:51 BST Walter Dnes wrote:
> > > I may be misunderstanding, but it seems to me that local mirrors are
> > > all-or-nothing. In the interests of saving bandwidth, I'd like to
> > > have a client first check the host's /var/cache/distfiles directory
> > > for a source tarball file. If not found, then fail over to another
> > > mirror as per GENTOO_MIRRORS in /etc/portage/make.conf. This would
> > > require emerge doing the lookup and potential failover for each file.
> > > Is this possible?
> >
> > Unless I misunderstand what you're asking, isn't this what takes place
> > anyway?
>
> The reference to a host makes me think Walter wants one machne to hold
> the distfiles for all on the network.
>
> > PS. Is http_replicator still available/maintained? I see the wiki
> > mentions apt-cacher-ng for local distfiles cache.
>
> I use apt-cacher-ng for this and it does what you are looking for.
>
>
> --
> Neil Bothwick
>
> The facts, although interesting, are usually irrelevant.
>


--
Alan McKinnon
alan dot mckinnon at gmail dot com

On Mon, 4 Sep 2023 19:49:56 +0200, Alan McKinnon wrote:

> Quick n dirty solution:
>
> put all distfiles on a central server
> FS mount that remote dir to /var/cache/distfiles on all hosts

That's what I used to do, but had problems when downloading the same file
from to clients at once.

BTW Welcome back Alan, but leave your dirty top-posting in Archland :P

>
> Alan
>
> On Mon, Sep 4, 2023 at 7:21?PM Neil Bothwick <neil@digimed.co.uk> wrote:
>
> > On Mon, 04 Sep 2023 14:04:53 +0100, Michael wrote:
> >
> > > On Monday, 4 September 2023 11:12:51 BST Walter Dnes wrote:
> > > > I may be misunderstanding, but it seems to me that local
> > > > mirrors are all-or-nothing. In the interests of saving
> > > > bandwidth, I'd like to have a client first check the host's
> > > > /var/cache/distfiles directory for a source tarball file. If not
> > > > found, then fail over to another mirror as per GENTOO_MIRRORS in
> > > > /etc/portage/make.conf. This would require emerge doing the
> > > > lookup and potential failover for each file. Is this possible?
> > >
> > > Unless I misunderstand what you're asking, isn't this what takes
> > > place anyway?
> >
> > The reference to a host makes me think Walter wants one machne to hold
> > the distfiles for all on the network.
> >
> > > PS. Is http_replicator still available/maintained? I see the wiki
> > > mentions apt-cacher-ng for local distfiles cache.
> >
> > I use apt-cacher-ng for this and it does what you are looking for.
> >
> >
> > --
> > Neil Bothwick
> >
> > The facts, although interesting, are usually irrelevant.
> >
>
>




--
Neil Bothwick

Not one shred of evidence supports the notion that life is serious.

On Mon, Sep 4, 2023 at 8:26?PM Neil Bothwick <neil@digimed.co.uk> wrote:

> On Mon, 4 Sep 2023 19:49:56 +0200, Alan McKinnon wrote:
>
> > Quick n dirty solution:
> >
> > put all distfiles on a central server
> > FS mount that remote dir to /var/cache/distfiles on all hosts
>
> That's what I used to do, but had problems when downloading the same file
> from to clients at once.
>
> BTW Welcome back Alan, but leave your dirty top-posting in Archland :P
>
>
Eh, I use gmail in the browser ..... the blerry thing is built to top post,
like Outlook


Alan

On Mon, Sep 4, 2023 at 12:38?PM Alan McKinnon <alan.mckinnon@gmail.com>
wrote:
>
>
>
> On Mon, Sep 4, 2023 at 8:26?PM Neil Bothwick <neil@digimed.co.uk> wrote:
>>
>> On Mon, 4 Sep 2023 19:49:56 +0200, Alan McKinnon wrote:
>>
>> > Quick n dirty solution:
>> >
>> > put all distfiles on a central server
>> > FS mount that remote dir to /var/cache/distfiles on all hosts
>>
>> That's what I used to do, but had problems when downloading the same file
>> from to clients at once.
>>
>> BTW Welcome back Alan, but leave your dirty top-posting in Archland :P
>>
>
> Eh, I use gmail in the browser ..... the blerry thing is built to top
post, like Outlook
>
>
> Alan

>
I'm a GMail user also. Sadly you'll want to not only bottom post, but also
select all text you're responding to, remove formatting (Ctrl-V) and then
type your response or you'll be down voted for responding in HTML.

I hate it also, but this list is easily one of my favorites and I'm no
longer
a Gentoo user.

Best wishes,
Mark

On Mon, 4 Sept 2023 at 22:02, Mark Knecht <markknecht@gmail.com> wrote:
> I'm a GMail user also. Sadly you'll want to not only bottom post, but also
> select all text you're responding to, remove formatting (Ctrl-V) and then
> type your response or you'll be down voted for responding in HTML.
>
> I hate it also, but this list is easily one of my favorites and I'm no longer
> a Gentoo user.

I also use both GMail and Outlook, and you can set up GMail to be
plain text by default. It's not too difficult to click the three dots
and type your reply where it belongs. Not so easy in Outlook which is
actively hating its users.

Regards,
Arve

On Mon, Sep 04, 2023 at 06:21:27PM +0100, Neil Bothwick wrote
> On Mon, 04 Sep 2023 14:04:53 +0100, Michael wrote:
>
> The reference to a host makes me think Walter wants one machne to hold
> the distfiles for all on the network.

Correct.

> I use apt-cacher-ng for this and it does what you are looking for.

https://wiki.gentoo.org/wiki/Local_distfiles_cache#Open_issues says...

> Open issues
>
> * Apt-cacher-ng installs a cron job to delete unreferenced files
> from the cache. Since the Gentoo cache contains no index files,
> this probably deletes either everything or nothing from it.

Have you run into ptoblems with this?

It looks like remote-mounting /var/cache/distfiles might be the
quick-n-dirty solution like Alan suggested. And I never have a need to
have emerge run simultaneously on more than one machine.

--
I've seen things, you people wouldn't believe; Gopher, Netscape with
frames, the first Browser Wars. Searching for pages with AltaVista,
pop-up windows self-replicating, trying to uninstall RealPlayer. All
those moments, will be lost in time like tears in rain... time to die.

On Mon, 4 Sep 2023 22:54:38 -0400, Walter Dnes wrote:

> On Mon, Sep 04, 2023 at 06:21:27PM +0100, Neil Bothwick wrote

> > I use apt-cacher-ng for this and it does what you are looking for.
>
> https://wiki.gentoo.org/wiki/Local_distfiles_cache#Open_issues says...
>
> > Open issues
> >
> > * Apt-cacher-ng installs a cron job to delete unreferenced files
> > from the cache. Since the Gentoo cache contains no index files,
> > this probably deletes either everything or nothing from it.
>
> Have you run into ptoblems with this?

Not that I've noticed.

> It looks like remote-mounting /var/cache/distfiles might be the
> quick-n-dirty solution like Alan suggested. And I never have a need to
> have emerge run simultaneously on more than one machine.

It seems the simplest suitable solution for your needs.


--
Neil Bothwick

A pessimist complains about the noise when opportunity knocks.

Am 04.09.23 um 15:04 schrieb Michael:
> PS. Is http_replicator still available/maintained? I see the wiki mentions
> apt-cacher-ng for local distfiles cache.

IIRC it died when python or perl 2.x was no longer available.

On Tue, Sep 05, 2023 at 08:04:19AM +0100, Neil Bothwick wrote
> On Mon, 4 Sep 2023 22:54:38 -0400, Walter Dnes wrote:
>
> > It looks like remote-mounting /var/cache/distfiles might be the
> > quick-n-dirty solution like Alan suggested. And I never have a need to
> > have emerge run simultaneously on more than one machine.
>
> It seems the simplest suitable solution for your needs.

And while we're at it, howsabout "emerge --sync" on one host, and then
remote-mounting /var/db/repos/gentoo ?

--
I've seen things, you people wouldn't believe; Gopher, Netscape with
frames, the first Browser Wars. Searching for pages with AltaVista,
pop-up windows self-replicating, trying to uninstall RealPlayer. All
those moments, will be lost in time like tears in rain... time to die.

On Tuesday, 5 September 2023 10:06:42 BST hitachi303 wrote:
> Am 04.09.23 um 15:04 schrieb Michael:
> > PS. Is http_replicator still available/maintained? I see the wiki
> > mentions
> > apt-cacher-ng for local distfiles cache.
>
> IIRC it died when python or perl 2.x was no longer available.

It's worth mentioning another option, not involving a full caching proxy as
such. It may suit Walter's workflow so I mention it here for completeness. A
lightweight browser can serve files directly from the local mirror's own /var/
cache/distfiles.

Unlike a caching proxy, files not already downloaded/needed by the local
mirror won't be available locally, so the clients will download these from
external mirrors. The local server will need to be updated first, to fetch
all the latest source file versions. If there is a large difference in
package requirements between different clients and the server, then only a
partial bandwidth efficiency will be achieved this way. Also if all the
clients require the same packages not available on the server, then these will
be downloaded by each client separately from external mirrors, further eroding
any savings in overall bandwidth.

Setting up a small footprint browser like gatling, or just a couple of lines
of python http.server, will allow any light on resources gentoo machine to
become a useful local mirror with minimal configuration.

On Tuesday, 5 September 2023 15:58:05 BST Walter Dnes wrote:
> On Tue, Sep 05, 2023 at 08:04:19AM +0100, Neil Bothwick wrote
>
> > On Mon, 4 Sep 2023 22:54:38 -0400, Walter Dnes wrote:
> > > It looks like remote-mounting /var/cache/distfiles might be the
> > >
> > > quick-n-dirty solution like Alan suggested. And I never have a need to
> > > have emerge run simultaneously on more than one machine.
> >
> > It seems the simplest suitable solution for your needs.
>
> And while we're at it, howsabout "emerge --sync" on one host, and then
> remote-mounting /var/db/repos/gentoo ?

Take a look here:

https://wiki.gentoo.org/wiki/Local_Mirror

On Tue, 05 Sep 2023 16:07:16 +0100, Michael wrote:

> > And while we're at it, howsabout "emerge --sync" on one host, and
> > then remote-mounting /var/db/repos/gentoo ?
>
> Take a look here:
>
> https://wiki.gentoo.org/wiki/Local_Mirror

I used to do that, the server syncing with Gentoo then everything else
syncing with that. However, when I switched to git syncing, it was so
fast that there was no point in running a local mirror - everything syncs
directly from Github now.


--
Neil Bothwick

BBS: (n.) a system for connecting computers and exchanging gossip,
facts, and uninformed speculation under false names.

On Tue, Sep 05, 2023 at 04:07:16PM +0100, Michael wrote
> On Tuesday, 5 September 2023 15:58:05 BST Walter Dnes wrote:
> >
> > And while we're at it, howsabout "emerge --sync" on one host, and then
> > remote-mounting /var/db/repos/gentoo ?
>
> Take a look here:
>
> https://wiki.gentoo.org/wiki/Local_Mirror

That's what I do right now. I figure that remote-mounting might make
things easier.

--
I've seen things, you people wouldn't believe; Gopher, Netscape with
frames, the first Browser Wars. Searching for pages with AltaVista,
pop-up windows self-replicating, trying to uninstall RealPlayer. All
those moments, will be lost in time like tears in rain... time to die.

>
>
> From: Alan McKinnon alan.mckinnon@gmail.com<mailto:alan.mckinnon@gmail.com>
> Sent: Monday, September 4, 2023 12:38 PM
> To: gentoo-user@lists.gentoo.org<mailto:gentoo-user@lists.gentoo.org>
> Subject: Re: [gentoo-user] Is distfile partial mirror with failover possible?
>
> BTW Welcome back Alan, but leave your dirty top-posting in Archland :P
>
> Eh, I use gmail in the browser ..... the blerry thing is built to top post, like Outlook
>
>
> Alan
>

I'm in the same boat, but Notepad++ on Windows and Vim on Linux both support column-mode editing, so a couple quick cut-n-pastes and I can preserve the illusion that I'm using a sensible client.

LMP

On Tuesday, 5 September 2023 16:33:43 BST Walter Dnes wrote:
> On Tue, Sep 05, 2023 at 04:07:16PM +0100, Michael wrote
>
> > On Tuesday, 5 September 2023 15:58:05 BST Walter Dnes wrote:
> > > And while we're at it, howsabout "emerge --sync" on one host, and then
> > >
> > > remote-mounting /var/db/repos/gentoo ?
> >
> > Take a look here:
> >
> > https://wiki.gentoo.org/wiki/Local_Mirror
>
> That's what I do right now. I figure that remote-mounting might make
> things easier.

There's more work required to set it up than just rsyncd and NFS requires you
run the local mirror all the time if the clients expect it to be up & running
when they boot.

On 5/9/23 22:58, Walter Dnes wrote:
> On Tue, Sep 05, 2023 at 08:04:19AM +0100, Neil Bothwick wrote
>> On Mon, 4 Sep 2023 22:54:38 -0400, Walter Dnes wrote:
>>
>>> It looks like remote-mounting /var/cache/distfiles might be the
>>> quick-n-dirty solution like Alan suggested. And I never have a need to
>>> have emerge run simultaneously on more than one machine.
>> It seems the simplest suitable solution for your needs.
> And while we're at it, howsabout "emerge --sync" on one host, and then
> remote-mounting /var/db/repos/gentoo ?
>
Similarly - I was mounting a single location on moosefs (network
filesystem) for ~15 hosts.  One host updates using --sync.  One
buildhost for each architecture building packages.  Distfiles are
handled the same way. Then "emerge -k" on the remainder checking for
packages that needed an actual build.  If none, parallel merge (via
ansible) or else single builds.  Resulting packages are also stored
remotely per arch (various amd64, arm32, arm64 etc.)

Working well for a few years, but I am in the process of converting to
nfs with a similar structure as I have decommissioned the moosefs system
and many machines that are no longer needed.  A single machine will
handle built packages, distfiles and repos which are mounted on the
other systems via nfs. I will look into apt-cacher-ng as I was a fan of
http-replicator until it was deprecated.

BillK