Mailing List Archive

On 7/31/22 15:51, n952162 wrote:
> I've been running gentoo for years now, and every time I go to --sync,
> it's really a painful process.
>
> The process can take *very* before you find out if it succeeded or not.
>
> I try different repos.conf servers - one works for a while, then
> doesn't, then later, the new one doesn't work anymore and the old one
> works again.
>
> I've asked here about it multiple times and get the answers
>
> - "I don't have a problem"
>
> - "just change the server"
>
> - "keep trying"
>
> It can take several hours before it finally works
>
> It seems like a time-out problem.  Or maybe a memory problem ... In any
> case, it doesn't seem like it ought to be difficult to at least know
> what the problem is.
>
> Or?

Well, since "it works for me" I can only suggest some things to think of
and to try.  While a memory problem is possible, if it is timing out,
I'd look at a possible network problem.

First, as you talk about repos.conf, is your issue with the main portage
repo or some overlay?  Second, for the problem (or just pick one if it
happens with several) server, can you ping that server without
problems?  Third (maybe should have been first) what is your sync method
and what is GENTOO_MIRRORS in your make.conf?  Related to that last one,
have you run mirrorselect recently?

No solution, I know, but maybe something to trigger a useful thought.

n952162 wrote:
> I've been running gentoo for years now, and every time I go to --sync,
> it's really a painful process.
>
> The process can take *very* before you find out if it succeeded or not.
>
> I try different repos.conf servers - one works for a while, then
> doesn't, then later, the new one doesn't work anymore and the old one
> works again.
>
> I've asked here about it multiple times and get the answers
>
> - "I don't have a problem"
>
> - "just change the server"
>
> - "keep trying"
>
> It can take several hours before it finally works
>
> It seems like a time-out problem.  Or maybe a memory problem ... In any
> case, it doesn't seem like it ought to be difficult to at least know
> what the problem is.
>
> Or?
>
>
>

This wiki page and mirrorselect may help.

https://wiki.gentoo.org/wiki/Mirrorselect

I think this is the command I use to get the best server to sync with. 
You may want to make a backup of make.conf first, just in case.

mirrorselect -D -s4

One of those should help some unless you have a internet connection
problem.  If you have problems with multiple servers, it could be a
connection problem and you may figure that out.  It tests by downloading
a file.  If it fails on most or all servers, connection problem.  If it
works on most, just a bad server(s) most likely. 

Hope that helps.

Dale

:-)  :-) 

On Sun, 31 Jul 2022 21:51:05 +0200, n952162 wrote:

> I've been running gentoo for years now, and every time I go to --sync,
> it's really a painful process.
>
> The process can take *very* before you find out if it succeeded or not.
>
> I try different repos.conf servers - one works for a while, then
> doesn't, then later, the new one doesn't work anymore and the old one
> works again.

I switched to using git for syncing, from github, and haven't looked
back. It is *much* faster, several times faster syncing from github than
using rsync to sync from a local mirror, and github is always there.

% cat /etc/portage/repos.conf/gentoo.conf
[DEFAULT]
main-repo = gentoo

[gentoo]
priority = 20
location = /var/portage
sync-type = git
sync-uri = https://github.com/gentoo-mirror/gentoo
auto-sync = yes


--
Neil Bothwick

Cereal Killer Strikes Again! Cap'n Crunch found dead...

Sonntag, 31. Juli 2022 21:51:
 
> I've been running gentoo for years now, and every time I go to --sync,
> it's really a painful process.

> The process can take *very* [long] before you find out if it succeeded or not.
...

> It can take several hours before it finally works
 
Use a tool like atop to get some info about the throughput
of your network connection. Something like "ss -ntp" might
show interesting info about the state of the connection 
between your machine and the server, too.
 
You might have problems with DNS name resolution or your
box patiently trying to establish an IPv6 connection when
your internet connection is IPv4 only.
 
But switching to git, like Neil said, does make things faster.
 
FWIW, my config file reads like this:
 
# cat /etc/portage/repos.conf/gentoo.conf
[DEFAULT]
main-repo = gentoo

[gentoo]
location = /var/db/repos/gentoo
sync-type = git
sync-uri = https://github.com/gentoo-mirror/gentoo.git
auto-sync = yes
sync-git-verify-commit-signature = yes
sync-openpgp-key-path = /usr/share/openpgp-keys/gentoo-release.asc
 
good luck,
s.

> It seems like a time-out problem.  Or maybe a memory problem ... In any
> case, it doesn't seem like it ought to be difficult to at least know
> what the problem is.

> Or?

Had a similar issue to the OP, and it turned out to be a flaky internet
connection. I switched ISP's and, while sync's still take longer than I'd
like, I don't have the failures, time outs I did before.

On Sun, Jul 31, 2022 at 1:54 PM Stefan Schmiedl <s@xss.de> wrote:

> Sonntag, 31. Juli 2022 21:51:
>
>
>
> I've been running gentoo for years now, and every time I go to --sync,
> it's really a painful process.
>
>
> The process can take *very* [long] before you find out if it succeeded or
> not.
>
> ...
>
>
> It can take several hours before it finally works
>
>
>
> Use a tool like atop to get some info about the throughput
>
> of your network connection. Something like "ss -ntp" might
>
> show interesting info about the state of the connection
>
> between your machine and the server, too.
>
>
>
> You might have problems with DNS name resolution or your
>
> box patiently trying to establish an IPv6 connection when
>
> your internet connection is IPv4 only.
>
>
>
> But switching to git, like Neil said, does make things faster.
>
>
>
> FWIW, my config file reads like this:
>
>
>
> # cat /etc/portage/repos.conf/gentoo.conf
>
> [DEFAULT]
>
> main-repo = gentoo
>
>
> [gentoo]
>
> location = /var/db/repos/gentoo
>
> sync-type = git
>
> sync-uri = https://github.com/gentoo-mirror/gentoo.git
>
> auto-sync = yes
>
> sync-git-verify-commit-signature = yes
>
> sync-openpgp-key-path = /usr/share/openpgp-keys/gentoo-release.asc
>
>
>
> good luck,
>
> s.
>
>
> It seems like a time-out problem. Or maybe a memory problem ... In any
> case, it doesn't seem like it ought to be difficult to at least know
> what the problem is.
>
>
> Or?
>
>
>
>
>
>

--
Lee ????
<ny6p01@gmail.com>

On Sun, 31 Jul 2022 15:55:51 -0700, Lee wrote:

> Had a similar issue to the OP, and it turned out to be a flaky internet
> connection. I switched ISP's and, while sync's still take longer than
> I'd like, I don't have the failures, time outs I did before.

I have no complaints about the time taken to sync

% time emerge --sync
>>> Syncing repository 'gentoo' into '/var/portage'...
/var/portage
/usr/bin/git fetch origin
[snip]
=== Sync completed for gentoo

Action: sync for repo: gentoo, returned code = 0


emerge --sync 1.58s user 0.51s system 75% cpu 2.750 total


--
Neil Bothwick

Do Roman paramedics refer to IV's as "4's"?

On Sunday, 31 July 2022 21:43:12 BST Neil Bothwick wrote:

> I switched to using git for syncing, from github, and haven't looked
> back. It is *much* faster, several times faster syncing from github than
> using rsync to sync from a local mirror, and github is always there.

One thing to note, if you do change to git syncing, is that you'll have to
delete your entire portage tree: 'rm -r /var/db/repos/portage'. That may seem
like a bad idea if you're having sync problems, but it isn't really. After
that, the sync will take just seconds, as Neil said.

You'll never look back.

--
Regards,
Peter.

Given the speed improvement in using git for --sync, I wonder why that
isn't the default in the portage section of the manual....

On Sun, Jul 31, 2022 at 4:49 PM Peter Humphrey <peter@prh.myzen.co.uk>
wrote:

> On Sunday, 31 July 2022 21:43:12 BST Neil Bothwick wrote:
>
> > I switched to using git for syncing, from github, and haven't looked
> > back. It is *much* faster, several times faster syncing from github than
> > using rsync to sync from a local mirror, and github is always there.
>
> One thing to note, if you do change to git syncing, is that you'll have to
> delete your entire portage tree: 'rm -r /var/db/repos/portage'. That may
> seem
> like a bad idea if you're having sync problems, but it isn't really. After
> that, the sync will take just seconds, as Neil said.
>
> You'll never look back.
>
> --
> Regards,
> Peter.
>
>
>
>
>

--
Lee ????
<ny6p01@gmail.com>

On 7/31/22 21:51, n952162 wrote:
> I've been running gentoo for years now, and every time I go to --sync,
> it's really a painful process.
>
> The process can take *very* before you find out if it succeeded or not.
>
> I try different repos.conf servers - one works for a while, then
> doesn't, then later, the new one doesn't work anymore and the old one
> works again.
>
> I've asked here about it multiple times and get the answers
>
> - "I don't have a problem"
>
> - "just change the server"
>
> - "keep trying"
>
> It can take several hours before it finally works
>
> It seems like a time-out problem.  Or maybe a memory problem ... In any
> case, it doesn't seem like it ought to be difficult to at least know
> what the problem is.
>
> Or?
>
>

Thanks all, for the various suggestions, I'll try each.

On Monday, 1 August 2022 06:09:07 BST n952162 wrote:
> On 7/31/22 21:51, n952162 wrote:
> > I've been running gentoo for years now, and every time I go to --sync,
> > it's really a painful process.
> >
> > The process can take *very* before you find out if it succeeded or not.
> >
> > I try different repos.conf servers - one works for a while, then
> > doesn't, then later, the new one doesn't work anymore and the old one
> > works again.
> >
> > I've asked here about it multiple times and get the answers
> >
> > - "I don't have a problem"
> >
> > - "just change the server"
> >
> > - "keep trying"
> >
> > It can take several hours before it finally works
> >
> > It seems like a time-out problem. Or maybe a memory problem ... In any
> > case, it doesn't seem like it ought to be difficult to at least know
> > what the problem is.
> >
> > Or?
>
> Thanks all, for the various suggestions, I'll try each.

Two points to consider when choosing git to sync your portage tree:

1. It used to be the case the first time you run git it would try to download
GB of commits history and take ages to do so on a slow connection. The
solution used to be to add "EGIT_CLONE_TYPE=shallow" in your make.conf, but
I'm not sure if this is still needed (I don't use git).

2. These days rsync uses hashes and gpg to check the integrity of portage and
will flag up a warning in case of file tampering, or corrupt data. As far as I
know such a solution doesn't exist with git.

On Mon, 01 Aug 2022 07:58:05 +0100, Michael wrote:

> 1. It used to be the case the first time you run git it would try to
> download GB of commits history and take ages to do so on a slow
> connection. The solution used to be to add "EGIT_CLONE_TYPE=shallow"
> in your make.conf, but I'm not sure if this is still needed (I don't
> use git).

It doesn't seem to be the case, running sync with an empty portage tree
gives

>>> Syncing repository 'gentoo' into '/var/portage'...
/usr/bin/git clone --depth 1 https://github.com/gentoo-mirror/gentoo .

% du /var/portage
608M /var/portage

It did take 32s, but most of that was the Performing Global Updates stage.


--
Neil Bothwick

"A hundred years of forgetting and it all comes rushing back..."

On 2022-08-01 07:58+0100 Michael <confabulate@kintzios.com> wrote:

> […]
>
> 2. These days rsync uses hashes and gpg to check the integrity of
> portage and will flag up a warning in case of file tampering, or
> corrupt data. As far as I know such a solution doesn't exist with
> git.

Verification can be turned on with
sync-git-verify-commit-signature = yes
in repos.conf.[1] This does not seem to be enabled by default.[2]

[1] <https://wiki.gentoo.org/wiki/Project:Portage/Repository_verification#git>
[2] <https://wiki.gentoo.org/wiki/Portage_Security#git-mirror_repositories>

On Monday, 1 August 2022 10:43:55 BST tastytea wrote:
> On 2022-08-01 07:58+0100 Michael <confabulate@kintzios.com> wrote:
> > […]
> >
> > 2. These days rsync uses hashes and gpg to check the integrity of
> > portage and will flag up a warning in case of file tampering, or
> > corrupt data. As far as I know such a solution doesn't exist with
> > git.
>
> Verification can be turned on with
> sync-git-verify-commit-signature = yes
> in repos.conf.[1] This does not seem to be enabled by default.[2]
>
> [1]
> <https://wiki.gentoo.org/wiki/Project:Portage/Repository_verification#git>
> [2] <https://wiki.gentoo.org/wiki/Portage_Security#git-mirror_repositories>

I see ... this is an improvement from what I recall it to be. Thanks for
pointing it out. Perhaps I should start using git again. :-)

On 2022-07-31, n952162 <n952162@web.de> wrote:
> I've been running gentoo for years now, and every time I go to --sync,
> it's really a painful process.
>
> The process can take *very* before you find out if it succeeded or not.

In my experience, long --sync times have always been due to a slow
rsync server. Switching to a different server fixed it for me (though
sometimes it took a few tries to find a server that responded in a
timly manner).

That said, git is an order of magnitude faster than even the fastest
rsync server.

> It can take several hours before it finally works

That seems like something other than just a slow server. When dealing
with slow servers, it never took more than 15-30 minutes.

--
Grant