Mailing List Archive

On 15/01/2021 07:56, thelma@sys-concept.com wrote:
Hello
> On both of my systems the network card speed is showing 1000
> cat /sys/class/net/enp4s0/speed 1000
>
> but when I do rsync larage file I only see about: 20 to 22MB/s
> On my home network I get about 110MB/s between PC's
>
> Both PC's have SSD and the swith is Gigabit (I think).
> How to find a the bottleneck?
>
20MB = 80Mb so it sounds like your network is a 100Mb network. What is
the perfs of your switch(s) between your systems ?


Regards,

Hogren

? 2021/1/15 ??2:56, thelma@sys-concept.com ??:
> On both of my systems the network card speed is showing 1000
> cat /sys/class/net/enp4s0/speed 1000
>
> but when I do rsync larage file I only see about: 20 to 22MB/s
> On my home network I get about 110MB/s between PC's
>
> Both PC's have SSD and the swith is Gigabit (I think).
> How to find a the bottleneck?
1000Mbps network card's maximum theoretical speed is about 125MiB/s.
It only works in short distances.

The real speed is depending on where the files are and all devices on
the link.
So where are  the files?

> -----Original Message-----
> From: thelma@sys-concept.com <thelma@sys-concept.com>
> Sent: Friday, January 15, 2021 07:57
> To: Gentoo mailing list <gentoo-user@lists.gentoo.org>
> Subject: [gentoo-user] network transfer speed
>
>
> On both of my systems the network card speed is showing 1000
> cat /sys/class/net/enp4s0/speed 1000
>
> but when I do rsync larage file I only see about: 20 to 22MB/s On my home
> network I get about 110MB/s between PC's
>
> Both PC's have SSD and the swith is Gigabit (I think).
> How to find a the bottleneck?

If the PCs attached to the switch show 1000 then the switch _is_ gigabit.

On my 1Gb home network I have an FTP transfer speed between Gentoo PCs A and B of almost 900Mbps, the other way round is almost half of that. One difference between the two systems is the disk, A uses SATA-2 disk while B has SATA-3.

Does the 'B' in 110MB/s stand for byte? If so you have 880Mbps which is not bad, the problem probably lies somewhere else. Otherwise you could check the switch error count (if you have a managed switch) or the network card error count, just to ensure you don't have a cabling/connector problem.

Have you tried other transfer methods just for comparison? I think FTP is still the fastest way to transfer files, though insecure or inconvenient as it might be. I have no experience with rsync.

raffaele

> -----Original Message-----
> From: bobwxc <bobwxc@88.com>
> Sent: Friday, January 15, 2021 08:57
> To: gentoo-user@lists.gentoo.org
> Subject: Re: [gentoo-user] network transfer speed
>
> ? 2021/1/15 ??2:56, thelma@sys-concept.com ??:
> > On both of my systems the network card speed is showing 1000
> > cat /sys/class/net/enp4s0/speed 1000
> >
> > but when I do rsync larage file I only see about: 20 to 22MB/s On my
> > home network I get about 110MB/s between PC's
> >
> > Both PC's have SSD and the swith is Gigabit (I think).
> > How to find a the bottleneck?
> 1000Mbps network card's maximum theoretical speed is about 125MiB/s.
> It only works in short distances.

Correct but that's the line speed that you'll never reach, when you take into account Ethernet frame overhead, IP (and possibly TCP) header overhead and application ( rsync, FTP, SMB, NFS) overhead you get lower figures. In my experience 900Mbps (110MiBps) on a 1000Mbps line is more realistic for 'normal' transfers.

raffaele

ST Restricted

> -----Original Message-----
> From: Hogren <hogren@iiiha.com>
> Sent: Friday, January 15, 2021 08:50
> To: gentoo-user@lists.gentoo.org
> Subject: Re: [gentoo-user] network transfer speed
>
>
> On 15/01/2021 07:56, thelma@sys-concept.com wrote:
> Hello
> > On both of my systems the network card speed is showing 1000
> > cat /sys/class/net/enp4s0/speed 1000
> >
> > but when I do rsync larage file I only see about: 20 to 22MB/s On my
> > home network I get about 110MB/s between PC's
> >
> > Both PC's have SSD and the swith is Gigabit (I think).
> > How to find a the bottleneck?
> >
> 20MB = 80Mb so it sounds like your network is a 100Mb network. What is the
> perfs of your switch(s) between your systems ?

I disagree, /sys/class/net/enp4s0/speed shows the speed negotiated by the network card with the switch, it cannot be 1000 if the switch is a only a 10/100. I think we can safely assume the network is a gigabit one.

raffaele

? 2021/1/15 ??4:27, Raffaele BELARDI ??:
>> -----Original Message-----
>> From: bobwxc <bobwxc@88.com>
>> Sent: Friday, January 15, 2021 08:57
>> To: gentoo-user@lists.gentoo.org
>> Subject: Re: [gentoo-user] network transfer speed
>>
>> ? 2021/1/15 ??2:56, thelma@sys-concept.com ??:
>>> On both of my systems the network card speed is showing 1000
>>> cat /sys/class/net/enp4s0/speed 1000
>>>
>>> but when I do rsync larage file I only see about: 20 to 22MB/s On my
>>> home network I get about 110MB/s between PC's
>>>
>>> Both PC's have SSD and the swith is Gigabit (I think).
>>> How to find a the bottleneck?
>> 1000Mbps network card's maximum theoretical speed is about 125MiB/s.
>> It only works in short distances.
> Correct but that's the line speed that you'll never reach, when you take into account Ethernet frame overhead, IP (and possibly TCP) header overhead and application ( rsync, FTP, SMB, NFS) overhead you get lower figures. In my experience 900Mbps (110MiBps) on a 1000Mbps line is more realistic for 'normal' transfers.
>
Yes, you are right. So it is just *theoretical* speed :-)

I don't know where does the file he sync from.
If you sync a file from a server in other city, for a 20 to 22MB/s speed
is very normal. But if in home, that is not good.

And for ftp and rsync.
    ftp is better for transferring a single large file once.
    rsync is better for a long-term, incremental synchronization. The
file verification of rsync may take a lot of time for first sync.

On Friday, 15 January 2021 08:42:16 GMT bobwxc wrote:
> ? 2021/1/15 ??4:27, Raffaele BELARDI ??:
> >> -----Original Message-----
> >> From: bobwxc <bobwxc@88.com>
> >> Sent: Friday, January 15, 2021 08:57
> >> To: gentoo-user@lists.gentoo.org
> >> Subject: Re: [gentoo-user] network transfer speed
> >>
> >> ? 2021/1/15 ??2:56, thelma@sys-concept.com ??:
> >>> On both of my systems the network card speed is showing 1000
> >>> cat /sys/class/net/enp4s0/speed 1000
> >>>
> >>> but when I do rsync larage file I only see about: 20 to 22MB/s On my
> >>> home network I get about 110MB/s between PC's
> >>>
> >>> Both PC's have SSD and the swith is Gigabit (I think).
> >>> How to find a the bottleneck?
> >>
> >> 1000Mbps network card's maximum theoretical speed is about 125MiB/s.
> >> It only works in short distances.
> >
> > Correct but that's the line speed that you'll never reach, when you take
> > into account Ethernet frame overhead, IP (and possibly TCP) header
> > overhead and application ( rsync, FTP, SMB, NFS) overhead you get lower
> > figures. In my experience 900Mbps (110MiBps) on a 1000Mbps line is more
> > realistic for 'normal' transfers.
> Yes, you are right. So it is just *theoretical* speed :-)
>
> I don't know where does the file he sync from.
> If you sync a file from a server in other city, for a 20 to 22MB/s speed
> is very normal. But if in home, that is not good.
>
> And for ftp and rsync.
> ftp is better for transferring a single large file once.
> rsync is better for a long-term, incremental synchronization. The
> file verification of rsync may take a lot of time for first sync.

There is a theoretical network speed as already mentioned. There is a
protocol speed, which may limit throughput if it has e.g. heavy encryption/
compression and the CPU is anaemic. Finally, there is a MoBo bus (SCSI/SATA/
USB) and the media storage limit. If using USB 1.1 or 2.0 and/or the disks
are slow or experience write amplification, you'll find this will constrain
the final transfer speed significantly.

On 15/01/2021 09:34, Raffaele BELARDI wrote:
>
> ST Restricted
>
>> -----Original Message-----
>> From: Hogren <hogren@iiiha.com>
>> Sent: Friday, January 15, 2021 08:50
>> To: gentoo-user@lists.gentoo.org
>> Subject: Re: [gentoo-user] network transfer speed
>>
>>
>> On 15/01/2021 07:56, thelma@sys-concept.com wrote:
>> Hello
>>> On both of my systems the network card speed is showing 1000
>>> cat /sys/class/net/enp4s0/speed 1000
>>>
>>> but when I do rsync larage file I only see about: 20 to 22MB/s On my
>>> home network I get about 110MB/s between PC's
>>>
>>> Both PC's have SSD and the swith is Gigabit (I think).
>>> How to find a the bottleneck?
>>>
>> 20MB = 80Mb so it sounds like your network is a 100Mb network. What is the
>> perfs of your switch(s) between your systems ?
> I disagree, /sys/class/net/enp4s0/speed shows the speed negotiated by the network card with the switch, it cannot be 1000 if the switch is a only a 10/100. I think we can safely assume the network is a gigabit one.
>
> raffaele

Yes, I thought about that after. But may be he has several switchs
between the two systems.

Hogren

On Friday, 15 January 2021 13:26:23 GMT Hogren wrote:
> On 15/01/2021 09:34, Raffaele BELARDI wrote:
> > ST Restricted
> >
> >> -----Original Message-----
> >> From: Hogren <hogren@iiiha.com>
> >> Sent: Friday, January 15, 2021 08:50
> >> To: gentoo-user@lists.gentoo.org
> >> Subject: Re: [gentoo-user] network transfer speed
> >>
> >>
> >> On 15/01/2021 07:56, thelma@sys-concept.com wrote:
> >> Hello
> >>
> >>> On both of my systems the network card speed is showing 1000
> >>> cat /sys/class/net/enp4s0/speed 1000
> >>>
> >>> but when I do rsync larage file I only see about: 20 to 22MB/s On my
> >>> home network I get about 110MB/s between PC's
> >>>
> >>> Both PC's have SSD and the swith is Gigabit (I think).
> >>> How to find a the bottleneck?
> >>
> >> 20MB = 80Mb so it sounds like your network is a 100Mb network. What is
> >> the
> >> perfs of your switch(s) between your systems ?
> >
> > I disagree, /sys/class/net/enp4s0/speed shows the speed negotiated by the
> > network card with the switch, it cannot be 1000 if the switch is a only a
> > 10/100. I think we can safely assume the network is a gigabit one.
> >
> > raffaele
>
> Yes, I thought about that after. But may be he has several switchs
> between the two systems.
>
> Hogren

There's an easy way to test the speed limits of the network Vs the limits of
the storage media. Use netcat/telnet to send a large file across from tmpfs
on host A to a tmpfs on host B. As long as tmpfs is large enough to not start
using swap, the speed will reflect what the network can achieve.

On 1/15/21 6:26 AM, Hogren wrote:
>
> On 15/01/2021 09:34, Raffaele BELARDI wrote:
>>
>> ST Restricted
>>
>>> -----Original Message-----
>>> From: Hogren <hogren@iiiha.com>
>>> Sent: Friday, January 15, 2021 08:50
>>> To: gentoo-user@lists.gentoo.org
>>> Subject: Re: [gentoo-user] network transfer speed
>>>
>>>
>>> On 15/01/2021 07:56, thelma@sys-concept.com wrote:
>>> Hello
>>>> On both of my systems the network card speed is showing 1000
>>>> cat /sys/class/net/enp4s0/speed   1000
>>>>
>>>> but when I do rsync larage file I only see about: 20 to 22MB/s On my
>>>> home network I get about 110MB/s between PC's
>>>>
>>>> Both PC's have SSD  and the swith is Gigabit (I think).
>>>> How to find a the bottleneck?
>>>>
>>> 20MB = 80Mb so it sounds like your network is a 100Mb network. What is the
>>> perfs of your switch(s) between your systems ?
>> I disagree, /sys/class/net/enp4s0/speed shows the speed negotiated by the network card with the switch, it cannot be 1000 if the switch is a only a 10/100. I think we can safely assume the network is a gigabit one.
>>
>> raffaele
>
> Yes, I thought about that after. But may be he has several switchs between the two systems.
>
> Hogren

I just checked the remote location and there are two swiches:
- D-link Green Technology (I think it is DSG-1005D
- Trident Gigabit Switch - TEG-S80g

They are both Gigabit switches.

On 1/15/21 2:58 AM, Michael wrote:
> On Friday, 15 January 2021 08:42:16 GMT bobwxc wrote:
>> ? 2021/1/15 ??4:27, Raffaele BELARDI ??:
>>>> -----Original Message-----
>>>> From: bobwxc <bobwxc@88.com>
>>>> Sent: Friday, January 15, 2021 08:57
>>>> To: gentoo-user@lists.gentoo.org
>>>> Subject: Re: [gentoo-user] network transfer speed
>>>>
>>>> ? 2021/1/15 ??2:56, thelma@sys-concept.com ??:
>>>>> On both of my systems the network card speed is showing 1000
>>>>> cat /sys/class/net/enp4s0/speed 1000
>>>>>
>>>>> but when I do rsync larage file I only see about: 20 to 22MB/s On my
>>>>> home network I get about 110MB/s between PC's
>>>>>
>>>>> Both PC's have SSD and the swith is Gigabit (I think).
>>>>> How to find a the bottleneck?
>>>>
>>>> 1000Mbps network card's maximum theoretical speed is about 125MiB/s.
>>>> It only works in short distances.
>>>
>>> Correct but that's the line speed that you'll never reach, when you take
>>> into account Ethernet frame overhead, IP (and possibly TCP) header
>>> overhead and application ( rsync, FTP, SMB, NFS) overhead you get lower
>>> figures. In my experience 900Mbps (110MiBps) on a 1000Mbps line is more
>>> realistic for 'normal' transfers.
>> Yes, you are right. So it is just *theoretical* speed :-)
>>
>> I don't know where does the file he sync from.
>> If you sync a file from a server in other city, for a 20 to 22MB/s speed
>> is very normal. But if in home, that is not good.
>>
>> And for ftp and rsync.
>> ftp is better for transferring a single large file once.
>> rsync is better for a long-term, incremental synchronization. The
>> file verification of rsync may take a lot of time for first sync.
>
> There is a theoretical network speed as already mentioned. There is a
> protocol speed, which may limit throughput if it has e.g. heavy encryption/
> compression and the CPU is anaemic. Finally, there is a MoBo bus (SCSI/SATA/
> USB) and the media storage limit. If using USB 1.1 or 2.0 and/or the disks
> are slow or experience write amplification, you'll find this will constrain
> the final transfer speed significantly.

The computers on this network are 2-meters apart and they both use SSD Drive (so USB limitation doesn't come under consideration).
Like I said, on my home network when I transfer the 24GB file I get about 110MiBps transfer, so I was expecting the same in remote location).
Some units are connected to a router Ausus RT-AC66U B1 but these ports are gigabit too.

On 1/15/21 11:51 AM, thelma@sys-concept.com wrote:
> On 1/15/21 2:58 AM, Michael wrote:
>> On Friday, 15 January 2021 08:42:16 GMT bobwxc wrote:
>>> ? 2021/1/15 ??4:27, Raffaele BELARDI ??:
>>>>> -----Original Message-----
>>>>> From: bobwxc <bobwxc@88.com>
>>>>> Sent: Friday, January 15, 2021 08:57
>>>>> To: gentoo-user@lists.gentoo.org
>>>>> Subject: Re: [gentoo-user] network transfer speed
>>>>>
>>>>> ? 2021/1/15 ??2:56, thelma@sys-concept.com ??:
>>>>>> On both of my systems the network card speed is showing 1000
>>>>>> cat /sys/class/net/enp4s0/speed 1000
>>>>>>
>>>>>> but when I do rsync larage file I only see about: 20 to 22MB/s On my
>>>>>> home network I get about 110MB/s between PC's
>>>>>>
>>>>>> Both PC's have SSD and the swith is Gigabit (I think).
>>>>>> How to find a the bottleneck?
>>>>> 1000Mbps network card's maximum theoretical speed is about 125MiB/s.
>>>>> It only works in short distances.
>>>> Correct but that's the line speed that you'll never reach, when you take
>>>> into account Ethernet frame overhead, IP (and possibly TCP) header
>>>> overhead and application ( rsync, FTP, SMB, NFS) overhead you get lower
>>>> figures. In my experience 900Mbps (110MiBps) on a 1000Mbps line is more
>>>> realistic for 'normal' transfers.
>>> Yes, you are right. So it is just *theoretical* speed :-)
>>>
>>> I don't know where does the file he sync from.
>>> If you sync a file from a server in other city, for a 20 to 22MB/s speed
>>> is very normal. But if in home, that is not good.
>>>
>>> And for ftp and rsync.
>>> ftp is better for transferring a single large file once.
>>> rsync is better for a long-term, incremental synchronization. The
>>> file verification of rsync may take a lot of time for first sync.
>> There is a theoretical network speed as already mentioned. There is a
>> protocol speed, which may limit throughput if it has e.g. heavy encryption/
>> compression and the CPU is anaemic. Finally, there is a MoBo bus (SCSI/SATA/
>> USB) and the media storage limit. If using USB 1.1 or 2.0 and/or the disks
>> are slow or experience write amplification, you'll find this will constrain
>> the final transfer speed significantly.
> The computers on this network are 2-meters apart and they both use SSD Drive (so USB limitation doesn't come under consideration).
> Like I said, on my home network when I transfer the 24GB file I get about 110MiBps transfer, so I was expecting the same in remote location).
> Some units are connected to a router Ausus RT-AC66U B1 but these ports are gigabit too.
When you say the computers are remote, is it possible the file is
passing through your local computer on the way between the two remote
machines?  Where are you actually running the rsync command?

On 1/15/21 9:55 AM, Jack wrote:
> On 1/15/21 11:51 AM, thelma@sys-concept.com wrote:
>> On 1/15/21 2:58 AM, Michael wrote:
>>> On Friday, 15 January 2021 08:42:16 GMT bobwxc wrote:
>>>> ? 2021/1/15 ??4:27, Raffaele BELARDI ??:
>>>>>> -----Original Message-----
>>>>>> From: bobwxc <bobwxc@88.com>
>>>>>> Sent: Friday, January 15, 2021 08:57
>>>>>> To: gentoo-user@lists.gentoo.org
>>>>>> Subject: Re: [gentoo-user] network transfer speed
>>>>>>
>>>>>> ? 2021/1/15 ??2:56, thelma@sys-concept.com ??:
>>>>>>> On both of my systems the network card speed is showing 1000
>>>>>>> cat /sys/class/net/enp4s0/speed   1000
>>>>>>>
>>>>>>> but when I do rsync larage file I only see about: 20 to 22MB/s On my
>>>>>>> home network I get about 110MB/s between PC's
>>>>>>>
>>>>>>> Both PC's have SSD  and the swith is Gigabit (I think).
>>>>>>> How to find a the bottleneck?
>>>>>> 1000Mbps network card's maximum theoretical speed is about 125MiB/s.
>>>>>> It only works in short distances.
>>>>> Correct but that's the line speed that you'll never reach, when you take
>>>>> into account Ethernet frame overhead, IP (and possibly TCP) header
>>>>> overhead and application ( rsync, FTP, SMB, NFS) overhead you get lower
>>>>> figures. In my experience 900Mbps (110MiBps) on a 1000Mbps line is more
>>>>> realistic for 'normal' transfers.
>>>> Yes, you are right. So it is just *theoretical* speed :-)
>>>>
>>>> I don't know where does the file he sync from.
>>>> If you sync a file from a server in other city, for a 20 to 22MB/s speed
>>>> is very normal. But if in home, that is not good.
>>>>
>>>> And for ftp and rsync.
>>>>       ftp is better for transferring a single large file once.
>>>>       rsync is better for a long-term, incremental synchronization. The
>>>> file verification of rsync may take a lot of time for first sync.
>>> There is a theoretical network speed as already mentioned.  There is a
>>> protocol speed, which may limit throughput if it has e.g. heavy encryption/
>>> compression and the CPU is anaemic.  Finally, there is a MoBo bus (SCSI/SATA/
>>> USB) and the media storage limit.  If using USB 1.1 or 2.0 and/or the disks
>>> are slow or experience write amplification, you'll find this will constrain
>>> the final transfer speed significantly.
>> The computers on this network are 2-meters apart and they both use SSD Drive (so USB limitation doesn't come under consideration).
>> Like I said, on my home network when I transfer the 24GB file I get about 110MiBps transfer, so I was expecting the same in remote location).
>> Some units are connected to a router Ausus RT-AC66U B1 but these ports are gigabit too.
> When you say the computers are remote, is it possible the file is passing through your local computer on the way between the two remote machines?  Where are you actually running the rsync command?

I ssh over VPN to remote computers and run "rsync" there. Will it effect the speed?

On 1/15/21 9:55 AM, Jack wrote:
[snip]
>>>>
>>>> I don't know where does the file he sync from.
>>>> If you sync a file from a server in other city, for a 20 to 22MB/s speed
>>>> is very normal. But if in home, that is not good.
>>>>
>>>> And for ftp and rsync.
>>>>       ftp is better for transferring a single large file once.
>>>>       rsync is better for a long-term, incremental synchronization. The
>>>> file verification of rsync may take a lot of time for first sync.
>>> There is a theoretical network speed as already mentioned.  There is a
>>> protocol speed, which may limit throughput if it has e.g. heavy encryption/
>>> compression and the CPU is anaemic.  Finally, there is a MoBo bus (SCSI/SATA/
>>> USB) and the media storage limit.  If using USB 1.1 or 2.0 and/or the disks
>>> are slow or experience write amplification, you'll find this will constrain
>>> the final transfer speed significantly.
>> The computers on this network are 2-meters apart and they both use SSD Drive (so USB limitation doesn't come under consideration).
>> Like I said, on my home network when I transfer the 24GB file I get about 110MiBps transfer, so I was expecting the same in remote location).
>> Some units are connected to a router Ausus RT-AC66U B1 but these ports are gigabit too.
> When you say the computers are remote, is it possible the file is passing through your local computer on the way between the two remote machines?  Where are you actually running the rsync command?

The fact that I'm logged via ssh over VPN to a remote network should not have any influence over network speed.
I just made a loop:
Network A ==> Internet ==> Network B
ssh back to Network A over internet and run "rsync" I got same speed (as if I run the command locally) on Network A 112MB/s

So the limiting factor is somewhere else.

On 1/15/21 1:11 AM, Raffaele BELARDI wrote:
>> -----Original Message-----
>> From: thelma@sys-concept.com <thelma@sys-concept.com>
>> Sent: Friday, January 15, 2021 07:57
>> To: Gentoo mailing list <gentoo-user@lists.gentoo.org>
>> Subject: [gentoo-user] network transfer speed
>>
>>
>> On both of my systems the network card speed is showing 1000
>> cat /sys/class/net/enp4s0/speed 1000
>>
>> but when I do rsync larage file I only see about: 20 to 22MB/s On my home
>> network I get about 110MB/s between PC's
>>
>> Both PC's have SSD and the swith is Gigabit (I think).
>> How to find a the bottleneck?
>
> If the PCs attached to the switch show 1000 then the switch _is_ gigabit.
>
> On my 1Gb home network I have an FTP transfer speed between Gentoo PCs A and B of almost 900Mbps, the other way round is almost half of that. One difference between the two systems is the disk, A uses SATA-2 disk while B has SATA-3.
>
> Does the 'B' in 110MB/s stand for byte? If so you have 880Mbps which is not bad, the problem probably lies somewhere else. Otherwise you could check the switch error count (if you have a managed switch) or the network card error count, just to ensure you don't have a cabling/connector problem.
>
> Have you tried other transfer methods just for comparison? I think FTP is still the fastest way to transfer files, though insecure or inconvenient as it might be. I have no experience with rsync.
>
> raffaele

On a remote network I run ethtool on both cards and I got both 1000Mb/s speed

1.)
ethtool net0
Settings for net0:
Supported ports: [ TP MII ]
Supported link modes: 10baseT/Half 10baseT/Full
100baseT/Half 100baseT/Full
1000baseT/Half 1000baseT/Full
Supported pause frame use: No
Supports auto-negotiation: Yes
Advertised link modes: 10baseT/Half 10baseT/Full
100baseT/Half 100baseT/Full
1000baseT/Half 1000baseT/Full
Advertised pause frame use: Symmetric Receive-only
Advertised auto-negotiation: Yes
Link partner advertised link modes: 10baseT/Half 10baseT/Full
100baseT/Half 100baseT/Full
1000baseT/Full
Link partner advertised pause frame use: Symmetric
Link partner advertised auto-negotiation: Yes
Speed: 1000Mb/s
Duplex: Full
Port: MII
PHYAD: 0
Transceiver: internal
Auto-negotiation: on
MDI-X: on (auto)

2.)
ethtool enp4s0
Settings for enp4s0:
Supported ports: [ TP ]
Supported link modes: 10baseT/Half 10baseT/Full
100baseT/Half 100baseT/Full
1000baseT/Full
Supported pause frame use: Symmetric
Supports auto-negotiation: Yes
Advertised link modes: 10baseT/Half 10baseT/Full
100baseT/Half 100baseT/Full
1000baseT/Full
Advertised pause frame use: Symmetric
Advertised auto-negotiation: Yes
Speed: 1000Mb/s
Duplex: Full
Port: Twisted Pair
PHYAD: 1
Transceiver: internal
Auto-negotiation: on
MDI-X: on (auto)

On 16/1/21 6:56 am, thelma@sys-concept.com wrote:
> On 1/15/21 1:11 AM, Raffaele BELARDI wrote:
>>> -----Original Message-----
>>> From: thelma@sys-concept.com <thelma@sys-concept.com>
>>> Sent: Friday, January 15, 2021 07:57
>>> To: Gentoo mailing list <gentoo-user@lists.gentoo.org>
>>> Subject: [gentoo-user] network transfer speed
>>>
>>>
>>> On both of my systems the network card speed is showing 1000
>>> cat /sys/class/net/enp4s0/speed 1000
>>>
>>> but when I do rsync larage file I only see about: 20 to 22MB/s On my home
>>> network I get about 110MB/s between PC's
>>>
>>> Both PC's have SSD and the swith is Gigabit (I think).
>>> How to find a the bottleneck?
>> If the PCs attached to the switch show 1000 then the switch _is_ gigabit.
>>
>> On my 1Gb home network I have an FTP transfer speed between Gentoo PCs A and B of almost 900Mbps, the other way round is almost half of that. One difference between the two systems is the disk, A uses SATA-2 disk while B has SATA-3.
>>
>> Does the 'B' in 110MB/s stand for byte? If so you have 880Mbps which is not bad, the problem probably lies somewhere else. Otherwise you could check the switch error count (if you have a managed switch) or the network card error count, just to ensure you don't have a cabling/connector problem.
>>
>> Have you tried other transfer methods just for comparison? I think FTP is still the fastest way to transfer files, though insecure or inconvenient as it might be. I have no experience with rsync.
>>
>> raffaele
> On a remote network I run ethtool on both cards and I got both 1000Mb/s speed
>
> 1.)
> ethtool net0
> Settings for net0:
> Supported ports: [ TP MII ]
> Supported link modes: 10baseT/Half 10baseT/Full
> 100baseT/Half 100baseT/Full
> 1000baseT/Half 1000baseT/Full
> Supported pause frame use: No
> Supports auto-negotiation: Yes
> Advertised link modes: 10baseT/Half 10baseT/Full
> 100baseT/Half 100baseT/Full
> 1000baseT/Half 1000baseT/Full
> Advertised pause frame use: Symmetric Receive-only
> Advertised auto-negotiation: Yes
> Link partner advertised link modes: 10baseT/Half 10baseT/Full
> 100baseT/Half 100baseT/Full
> 1000baseT/Full
> Link partner advertised pause frame use: Symmetric
> Link partner advertised auto-negotiation: Yes
> Speed: 1000Mb/s
> Duplex: Full
> Port: MII
> PHYAD: 0
> Transceiver: internal
> Auto-negotiation: on
> MDI-X: on (auto)
>
> 2.)
> ethtool enp4s0
> Settings for enp4s0:
> Supported ports: [ TP ]
> Supported link modes: 10baseT/Half 10baseT/Full
> 100baseT/Half 100baseT/Full
> 1000baseT/Full
> Supported pause frame use: Symmetric
> Supports auto-negotiation: Yes
> Advertised link modes: 10baseT/Half 10baseT/Full
> 100baseT/Half 100baseT/Full
> 1000baseT/Full
> Advertised pause frame use: Symmetric
> Advertised auto-negotiation: Yes
> Speed: 1000Mb/s
> Duplex: Full
> Port: Twisted Pair
> PHYAD: 1
> Transceiver: internal
> Auto-negotiation: on
> MDI-X: on (auto)
>
What brand/type of interface (Realtek, int/ext USB ?)  There is an
obscure bug that drops throughput to ~1/3 in Realtek RTL8153 chip on
USB2 and the r8152 kernel driver.  Some interfaces (e.g., most raspberry
pi) connect via internal USB with its own throughput problems.  Probably
not the problem in this case but worth mentioning.

Rsync is not a good tool for measuring throughput as its optimisations
are problematic - and this can show up when comparing local to remote
tests.  If on a file system it will always transfer the whole file (and
it considers a network filesystem local) while on a network it will use
the transfer algorithm which can vary depending on commandline switches
used and where the remote file lives, and this can change between tests
depending on conditions.  Whole files have little overhead, whereas the
algorithm overhead can actually be much slower (hence why it tries to do
it this way) Try and use a largish file created with /dev/random and dd
as well as a simple netcat type transfer mechanism to measure.  Note
that encryption can severely limit transfers due to cpu limiting if on a
low powered system and rsync over ssh uses encrytion.  And if a secure
VPN is in the mix its orders of magnitude worse.  Also note that
something like an inline IDS on a link can drop throughput by 50% or
more (measured on an older enterprise Cisco device) which further
complicates local/remote via internet comparisons.


Good luck in finding the problem - its really not simple and easy.

BillK

On Fri, Jan 15, 2021 at 3:40 PM William Kenworthy <billk@iinet.net.au>
wrote:

>
>
> Rsync is not a good tool for measuring throughput as its optimisations
> are problematic - and this can show up when comparing local to remote
> tests.
>
>
> Yup. A better option is to use something like net-misc/iperf. That will
give you a much better idea of the raw network performance. It's also
available on Windows so you can use it to test Linux<->Windows and
Windows<->Windows speeds as well as Linux<->Linux speeds.

--
Manuel A. McLure WW1FA <manuel@mclure.org> <http://www.mclure.org>
...for in Ulthar, according to an ancient and significant law,
no man may kill a cat. -- H.P. Lovecraft

> On a remote network I run ethtool on both cards and I got both 1000Mb/s
> speed
>
>
As the 20 odd MB/s you're getting is above what is possible on 100M
ethernet, you can rule out any ethernet interfaces at 100M.

Can you describe the network between the two systems with the slow transfer?

If there is a fast WAN from one side of the globe to the other it could be
latency related. OpenSSH used to have a fixed internal window size that
made it slow on high bandwidth high latency links, and I notice the hpn USE
flag still exists in the openssh ebuild, which implies the issue with
openssh still exists. Rsync can use either ssh or its own protocol, so if
there's a high latency link between the two boxes and rsync is using ssh,
you could investigate rebuilding openssh with +hpn.

What does ping show the latency as?

Otherwise i'd be thinking about packet loss. First place to start for that
is on the endpoint interfaces;
ifconfig enp35s0f0 | grep err
RX errors 0 dropped 0 overruns 0 frame 0
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

On Friday, 15 January 2021 22:43:36 GMT thelma@sys-concept.com wrote:

> The fact that I'm logged via ssh over VPN to a remote network should not
> have any influence over network speed.

It may influence speed if you're trying to push a large file through the
tunnel. TCP over TCP tends to choke due to retransmissions:

http://sites.inka.de/bigred/devel/tcp-tcp.html

Is the VPN you mention using a TCP or UDP tunnel?


> I just made a loop:
> Network A ==> Internet ==> Network B
> ssh back to Network A over internet and run "rsync" I got same speed (as if
> I run the command locally) on Network A 112MB/s
>
> So the limiting factor is somewhere else.

I'm sorry, but I fail to understand with any clarity what runs where and how
when you test things locally, Vs remotely. I mean:

- Network topology;
- Network Protocols;
- Applications & application protocols;
- Relevant services on each peer;
- Actions on each peer;
- Results per action.

As already mentioned iperf or netcat/telnet results will confirm if this is
purely a network issue, ISPs performing deep packet inspection/throttling
affecting throughput asymmetrically, etc.

SSDs are typically faster than spinning disks, but not always as fast as
ramdisks/tmpfs, especially if write amplification takes place, TRIM kicks in,
etc.

Stacking network + application protocols can also have an adverse effect.
Applications like rsync which compare file names, sizes, hashes and what not,
do not offer a reliable speed comparison.

Methodically testing each component of the transmission system should get you
an answer at the end.

On 1/16/21 11:55 AM, Michael wrote:
> On Friday, 15 January 2021 22:43:36 GMT thelma@sys-concept.com wrote:
>
>> The fact that I'm logged via ssh over VPN to a remote network should not
>> have any influence over network speed.
>
> It may influence speed if you're trying to push a large file through the
> tunnel. TCP over TCP tends to choke due to retransmissions:
>
> http://sites.inka.de/bigred/devel/tcp-tcp.html
>
> Is the VPN you mention using a TCP or UDP tunnel?

The VPN is using UDP tunnel.

>
>> I just made a loop:
>> Network A ==> Internet ==> Network B
>> ssh back to Network A over internet and run "rsync" I got same speed (as if
>> I run the command locally) on Network A 112MB/s
>>
>> So the limiting factor is somewhere else.
>
> I'm sorry, but I fail to understand with any clarity what runs where and how
> when you test things locally, Vs remotely. I mean:

What I did is log-in to a remote network over ssh, from remote network I logged back to my network PC using ssh and tested the transfer speed.

> - Network topology;
> - Network Protocols;
> - Applications & application protocols;
> - Relevant services on each peer;
> - Actions on each peer;
> - Results per action.
>
> As already mentioned iperf or netcat/telnet results will confirm if this is
> purely a network issue, ISPs performing deep packet inspection/throttling
> affecting throughput asymmetrically, etc.
>
> SSDs are typically faster than spinning disks, but not always as fast as
> ramdisks/tmpfs, especially if write amplification takes place, TRIM kicks in,
> etc.
>
> Stacking network + application protocols can also have an adverse effect.
> Applications like rsync which compare file names, sizes, hashes and what not,
> do not offer a reliable speed comparison.
>
> Methodically testing each component of the transmission system should get you
> an answer at the end.

On a remote network there are 5-pcs
- two of them are Windows PC
- 3-pcs are Gentoo

I was just testing Getnoo PC's
One small PC is about 20-30meters away from the switch and connected by CAT5 (or CAT5e)
that PC is network card is capable of 1000-speed according to "ethtool" but negotiated the speed of 100 according to:
/sys/class/net/enp4s0/speed

The 2-PCs closer to the switch, negotiated the speed of 1000 according to:
/sys/class/net/enpxxxx/speed

I have to check (this week) if they are both connected directly to a switch and or firewall Ausus RT-AC66U and the light on the switch.
I'll check out/investigate the use of "net-misc/iperf" to check the speed.

On 1/15/21 9:00 PM, Adam Carter wrote:
>> On a remote network I run ethtool on both cards and I got both 1000Mb/s
>> speed
>>
>>
> As the 20 odd MB/s you're getting is above what is possible on 100M
> ethernet, you can rule out any ethernet interfaces at 100M.

1.) One Gentoo PC (that is about 20-30meters away from the switch) negotiated the speed of only 100 despite being capable of doing 1000.
I'll have to buy a new switch and make a new CAT5e cable to test it.
But but it will take some time.

> Can you describe the network between the two systems with the slow transfer?

2.) The two Gentoo PC that are meters away from the switch are my concern firs.
One is a server, another small PC run 24/7 and both negotiated speed of 1000 with the switch.

I have to re-test the transfer speed between these to boxes first, and check to light on the switch if it is green and/or orange

> If there is a fast WAN from one side of the globe to the other it could be
> latency related. OpenSSH used to have a fixed internal window size that
> made it slow on high bandwidth high latency links, and I notice the hpn USE
> flag still exists in the openssh ebuild, which implies the issue with
> openssh still exists. Rsync can use either ssh or its own protocol, so if
> there's a high latency link between the two boxes and rsync is using ssh,
> you could investigate rebuilding openssh with +hpn.
>
> What does ping show the latency as?
>
> Otherwise i'd be thinking about packet loss. First place to start for that
> is on the endpoint interfaces;
> ifconfig enp35s0f0 | grep err
> RX errors 0 dropped 0 overruns 0 frame 0
> TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

I'll test the above for errors tomorrow.

On 2021.01.16 15:36, thelma@sys-concept.com wrote:
[snip.....]
> 2.) The two Gentoo PC that are meters away from the switch are my
> concern firs.
> One is a server, another small PC run 24/7 and both negotiated speed
> of 1000 with the switch.

I may be way off base here, but if the switch is connected to a router,
packets from one PC go to the switch and then to everything else
connected to it, including both the other PC and the router. Is there
any chance the router is passing packets back to the switch to get to
the second PC? I can imagine that causing lots of problems. However,
I would hope it is smart enough to know it doesn't need to do so, since
both PCs show up on the same router port.

On Saturday, 16 January 2021 20:59:58 GMT Jack wrote:
> On 2021.01.16 15:36, thelma@sys-concept.com wrote:
> [snip.....]
>
> > 2.) The two Gentoo PC that are meters away from the switch are my
> > concern firs.
> > One is a server, another small PC run 24/7 and both negotiated speed
> > of 1000 with the switch.
>
> I may be way off base here, but if the switch is connected to a router,
> packets from one PC go to the switch and then to everything else
> connected to it, including both the other PC and the router. Is there
> any chance the router is passing packets back to the switch to get to
> the second PC? I can imagine that causing lots of problems. However,
> I would hope it is smart enough to know it doesn't need to do so, since
> both PCs show up on the same router port.

Unmanaged witches for the consumer market tend to last fewer years than they
used to. This is because their power supply as well as the components on the
circuit board are now manufactured as cheaply as possible. Good for repeat
sales, bad for the consumer.

The switch may fail outright, in which case the network fault is obvious. Or,
it may start failing intermittently. Typical failure modes are for one or
more ports to stop working altogether, or downgrade the port speed - as
observed by the OP. Removing the ethernet cable, powering down the switch,
re-inserting the ethernet cable and powering up the switch may restore the
correct operation on the port.

It was usually the case electrolytic capacitors on the circuit board could be
replaced to fix the switch, but more recently some manufactures no longer use
electrolytic capacitors - so I'm not sure repairs are even possible.

> -----Original Message-----
> From: Jack <ostroffjh@users.sourceforge.net>
> Sent: Saturday, January 16, 2021 22:00
> To: gentoo-user@lists.gentoo.org
> Subject: Re: [gentoo-user] network transfer speed
>
> I may be way off base here, but if the switch is connected to a router,
> packets from one PC go to the switch and then to everything else connected
> to it, including both the other PC and the router. Is there any chance the
> router is passing packets back to the switch to get to the second PC? I can
> imagine that causing lots of problems. However, I would hope it is smart
> enough to know it doesn't need to do so, since both PCs show up on the
> same router port.

A switch uses the Ethernet MAC destination address to forward a packet only on the 'interested' ports. What you describe would be a 'hub' [1], I don't think it's easy to find one of those on recent networks.

Raffaele

[1] https://en.wikipedia.org/wiki/Ethernet_hub