Mailing List Archive

Hi Frank,

Are you sure the traffic does not go out to the internet or take an
unexpected path? Check ping and traceroute to ensure the path is
expected and round-trip times are low (as you'd expect on a LAN).
Verify traceroute in both directions.

Another idea, perhaps something is misconfigured and the firewall
thinks that A and B hosts are on the same subnet and it sends out an
ICMPv6 Redirect packet for each packet transiting the firewall to
signal the source of a better/direct path. Generating these packets
might be CPU-intensive for the firewall and slow down the transfer.

Additionally, perform a packet capture when doing an IPv4 and IPv6
transfer and compare the two. Search for differences, look for TCP
window size values, window scaling values, they might be entirely
different in v4 and v6. Check also if there are some related packets
such as some ICMP errors or maybe some retransmits or duplicate ACKs
which you may see with v6 and not with v4.

Also, run top or atop or htop on the firewall to see the CPU usage and
compare the usage during IPv4 and IPv6 transfers to see if it's
significantly different, perhaps IPv6 uses more CPU cycles and you can
identify with top what process or task requires more CPU to have a
better understanding and fix it.

Best regards,
Andras


On Wed, Oct 7, 2015 at 1:07 AM, Frank Steiner
<fsteiner-mail1@bio.ifi.lmu.de> wrote:
> Hi all,
>
> I've encountered a strange speed problem with ipv6 forwarding. We are using a routing firewall running SLES 11 sp3 at our chair. It has two 10GB network cards with 10G uplinks. We have a subnet behind the firewall and one in front of it and the firewall is configured to forward all traffic between the networks (and has ips from the network on the according devices of course). All hosts/networks have public ipv4 and public ipv6 IPs.
>
> Now I'm at host A behind the firewall and copy a file from host B outside the firewall. Works with ~ 112MB/s (the hosts have 1 GB uplinks) when I explicitely use the ipv4 address of B in the scp/wget or whatever. When I use the ipv6 address of B (which is the default when I use the host name), the transfer rate drops to ~ 1 MB/s.
>
> When copying from A to B via ipv6 adresses I get ~ 15 MB/s.
>
> But (let's assume the firewall ist host F) when I copy from A to F, F to A, B to F, F to B, always using ipv6 addresses, I always get the full transfer speed of ~ 112 MB/s.
>
> Thus, both directions from and to the firewall from both subnets are working at full speed when using ipv6 adresses. Only the forwarding through the firewall is slow with ipv6 adresses, while it's fast with ipv4.
>
> I've no idea where to start looking. I flushed all ip6tables rules with no change, /proc/sys/net/ipv6/conf/all/forwarding is "1", default route is set for ipv4 and ipv6.
>
> Any ideas what could be wrong with my setup?
>
> cu,
> Frank
>
> --
> Dipl.-Inform. Frank Steiner Web: http://www.bio.ifi.lmu.de/~steiner/
> Lehrstuhl f. Bioinformatik Mail: http://www.bio.ifi.lmu.de/~steiner/m/
> LMU, Amalienstr. 17 Phone: +49 89 2180-4049
> 80333 Muenchen, Germany Fax: +49 89 2180-99-4049
> * Rekursion kann man erst verstehen, wenn man Rekursion verstanden hat. *

Hi,

Andras Toth wrote

> Hi Frank,
>
> Are you sure the traffic does not go out to the internet or take an
> unexpected path? Check ping and traceroute to ensure the path is
> expected and round-trip times are low (as you'd expect on a LAN).
> Verify traceroute in both directions.

I got one step further. tracerout shows that route from inside (A)
outside (B) is A->F->B with F being my firewall.

But route from B to A goes through the router. I've setup all hosts
in the subnet in front of the firewall to route their packets through
the router R that our data center configured for this subnet.

Thus it's B->R->F->A. The same happens for ipv4, no ->R-> when
sending from A to B, but via R from B to A. While it's fast for
ipv4, it's slow for ipv6. So I added a route for the internal
subnet to the routing table of B so that the trace now shows
B->F->A. And then the copying between A and B is at full speed
of 112MB/s.

Now I thought maybe the router could be slow/misconfigured for ipv6
as ipv4 is so fast even with routing B to A via R.
I tested another ipv6 subnet outside the firewall, which is
not connected to my firewall but to the same router. Say there is
a host named C in this subnet.
As C is not in a common subnet with F, the traceroute for both
directions is A<->F<->R<->C. And this connection is fast! Copying
between A and C happens with 112 MB/s.

Two things I could imagine now: either the router is somehow
misconfigured only for our ipv6 subnet.

Or is it possible that ipv6 has a problem with the route from A->B
being different than B->A? Sth. similar as you described here maybe?

> Another idea, perhaps something is misconfigured and the firewall
> thinks that A and B hosts are on the same subnet and it sends out an
> ICMPv6 Redirect packet for each packet transiting the firewall to
> signal the source of a better/direct path. Generating these packets
> might be CPU-intensive for the firewall and slow down the transfer.

Of course the fix for our subnet is simple by adding the additional
configuration route, but I'd like to understand what goes wrong as
soon as B routes to A via R and F and not just F. And why this is
only a problem for ipv6.

cu,
Frank

--
Dipl.-Inform. Frank Steiner Web: http://www.bio.ifi.lmu.de/~steiner/
Lehrstuhl f. Bioinformatik Mail: http://www.bio.ifi.lmu.de/~steiner/m/
LMU, Amalienstr. 17 Phone: +49 89 2180-4049
80333 Muenchen, Germany Fax: +49 89 2180-99-4049
* Rekursion kann man erst verstehen, wenn man Rekursion verstanden hat. *

> I got one step further. tracerout shows that route from inside (A)
> outside (B) is A->F->B with F being my firewall.
>
> But route from B to A goes through the router. I've setup all hosts
> in the subnet in front of the firewall to route their packets through
> the router R that our data center configured for this subnet.
>
> Thus it's B->R->F->A. The same happens for ipv4, no ->R-> when
> sending from A to B, but via R from B to A. While it's fast for
> ipv4, it's slow for ipv6. So I added a route for the internal
> subnet to the routing table of B so that the trace now shows
> B->F->A. And then the copying between A and B is at full speed
> of 112MB/s.

Hi Frank,

So, R, B, and F all have legs on a common network segment, right? And B
probably points to R for default gateway? Does B have routes in its
table so that it knows to point to F in order to reach B? If not, it is
sending packets to R, who is probably returning ICMP redirects to B.
Perhaps B is dropping them? A tcpdump on R, B, and F might help show
you what's going on.

--Matt

Matt Rowley wrote:
>
> So, R, B, and F all have legs on a common network segment, right? And B
> probably points to R for default gateway? Does B have routes in its
> table so that it knows to point to F in order to reach B? If not, it is

Sorry, that should be "knows to point to F in order to reach A" not B.

:)

Sorry, clearly needed more coffee before replying!

cheers,
Matt

Matt Rowley wrote

>> I got one step further. tracerout shows that route from inside (A)
>> outside (B) is A->F->B with F being my firewall.
>>
>> But route from B to A goes through the router. I've setup all hosts
>> in the subnet in front of the firewall to route their packets through
>> the router R that our data center configured for this subnet.
>>
>> Thus it's B->R->F->A. The same happens for ipv4, no ->R-> when
>> sending from A to B, but via R from B to A. While it's fast for
>> ipv4, it's slow for ipv6. So I added a route for the internal
>> subnet to the routing table of B so that the trace now shows
>> B->F->A. And then the copying between A and B is at full speed
>> of 112MB/s.
>
> Hi Frank,
>
> So, R, B, and F all have legs on a common network segment, right?

Yes!

> And B points to R for default gateway?

Right.

> Does B have routes in its table so that it knows to point to F in
> order to reach A?

That's what I tried to describe above. By default it doesn't and that's
when the traffic slow. When I add such a route, the traffic is fast.
I just don't why it's slow without that route for ipv6 only, while
ipv4 has no problems routing through R first. Well, maybe...

> If not, it is sending packets to R, who is probably returning ICMP
> redirects to B. Perhaps B is dropping them?

...sth. like that. As far as I understand, ipv6 tries to dynamically
configure better routes while ipv4 doesn't.

> A tcpdump on R, B, and F might help show you what's going on.

I'm not really familiar with tcpdump, but I will figure out!

cu,
Frank


--
Dipl.-Inform. Frank Steiner Web: http://www.bio.ifi.lmu.de/~steiner/
Lehrstuhl f. Bioinformatik Mail: http://www.bio.ifi.lmu.de/~steiner/m/
LMU, Amalienstr. 17 Phone: +49 89 2180-4049
80333 Muenchen, Germany Fax: +49 89 2180-99-4049
* Rekursion kann man erst verstehen, wenn man Rekursion verstanden hat. *

Please remove me from this email conversation
----- Original Message -----
From: Frank Steiner &lt;fsteiner-mail1@bio.ifi.lmu.de&gt;
To: Matt Rowley &lt;matt@arin.net&gt;
Cc: ipv6-ops@lists.cluenet.de
Sent: Fri, 09 Oct 2015 07:22:09 -0600 (MDT)
Subject: Re: Strange speed problems with ipv6 forwarding

Matt Rowley wrote

&gt;&gt; I got one step further. tracerout shows that route from inside (A)
&gt;&gt; outside (B) is A-&gt;F-&gt;B with F being my firewall.
&gt;&gt;
&gt;&gt; But route from B to A goes through the router. I've setup all hosts
&gt;&gt; in the subnet in front of the firewall to route their packets through
&gt;&gt; the router R that our data center configured for this subnet.
&gt;&gt;
&gt;&gt; Thus it's B-&gt;R-&gt;F-&gt;A. The same happens for ipv4, no -&gt;R-&gt; when
&gt;&gt; sending from A to B, but via R from B to A. While it's fast for
&gt;&gt; ipv4, it's slow for ipv6. So I added a route for the internal
&gt;&gt; subnet to the routing table of B so that the trace now shows
&gt;&gt; B-&gt;F-&gt;A. And then the copying between A and B is at full speed
&gt;&gt; of 112MB/s.
&gt;
&gt; Hi Frank,
&gt;
&gt; So, R, B, and F all have legs on a common network segment, right?

Yes!

&gt; And B points to R for default gateway?

Right.

&gt; Does B have routes in its table so that it knows to point to F in
&gt; order to reach A?

That's what I tried to describe above. By default it doesn't and that's
when the traffic slow. When I add such a route, the traffic is fast.
I just don't why it's slow without that route for ipv6 only, while
ipv4 has no problems routing through R first. Well, maybe...

&gt; If not, it is sending packets to R, who is probably returning ICMP
&gt; redirects to B. Perhaps B is dropping them?

...sth. like that. As far as I understand, ipv6 tries to dynamically
configure better routes while ipv4 doesn't.

&gt; A tcpdump on R, B, and F might help show you what's going on.

I'm not really familiar with tcpdump, but I will figure out!

cu,
Frank


--
Dipl.-Inform. Frank Steiner Web: http://www.bio.ifi.lmu.de/~steiner/
Lehrstuhl f. Bioinformatik Mail: http://www.bio.ifi.lmu.de/~steiner/m/
LMU, Amalienstr. 17 Phone: +49 89 2180-4049
80333 Muenchen, Germany Fax: +49 89 2180-99-4049
* Rekursion kann man erst verstehen, wenn man Rekursion verstanden hat. *

Hi Frank,

As Matt and me earlier pointed out it sounds like the router is
sending ICMP Redirects because they are on a common segment and the
router tries to tell the host a better route exist. You can read more
about redirects here:
http://www.cisco.com/c/en/us/support/docs/ip/routing-information-protocol-rip/13714-43.html

The slowness occurs as the router cannot forward packets in its
dedicated fast circuits (ASIC or data-plane) because it needs to send
an ICMP Redirect packet to each packet on the same subnet. This is
done in the router CPU (control-plane) which is slower than the
dedicated circuits in the router data-plane.

Try disabling IPv6 redirects, on Cisco routers it's generally "no ipv6
redirects" on the routed interfaces, such as the L3 ports or VLAN
Interfaces (SVI). Alternatively fix your topology and routing to
ensure the return traffic does not go through the router in the same
subnet.

You most likely have "no ip redirects" configured already on the
router for v4, that's why for IPv4 it is fast, because the router
won't have to generate ICMP packets for v4 traffic and they go through
the data-plane and CPU does not slow them down.

Let me know if this solves the problem and if you have any other questions.

Best regards,
Andras


On Sat, Oct 10, 2015 at 12:22 AM, Frank Steiner
<fsteiner-mail1@bio.ifi.lmu.de> wrote:
> Matt Rowley wrote
>
>>> I got one step further. tracerout shows that route from inside (A)
>>> outside (B) is A->F->B with F being my firewall.
>>>
>>> But route from B to A goes through the router. I've setup all hosts
>>> in the subnet in front of the firewall to route their packets through
>>> the router R that our data center configured for this subnet.
>>>
>>> Thus it's B->R->F->A. The same happens for ipv4, no ->R-> when
>>> sending from A to B, but via R from B to A. While it's fast for
>>> ipv4, it's slow for ipv6. So I added a route for the internal
>>> subnet to the routing table of B so that the trace now shows
>>> B->F->A. And then the copying between A and B is at full speed
>>> of 112MB/s.
>>
>> Hi Frank,
>>
>> So, R, B, and F all have legs on a common network segment, right?
>
> Yes!
>
>> And B points to R for default gateway?
>
> Right.
>
>> Does B have routes in its table so that it knows to point to F in
>> order to reach A?
>
> That's what I tried to describe above. By default it doesn't and that's
> when the traffic slow. When I add such a route, the traffic is fast.
> I just don't why it's slow without that route for ipv6 only, while
> ipv4 has no problems routing through R first. Well, maybe...
>
>> If not, it is sending packets to R, who is probably returning ICMP
>> redirects to B. Perhaps B is dropping them?
>
> ...sth. like that. As far as I understand, ipv6 tries to dynamically
> configure better routes while ipv4 doesn't.
>
>> A tcpdump on R, B, and F might help show you what's going on.
>
> I'm not really familiar with tcpdump, but I will figure out!
>
> cu,
> Frank
>
>
> --
> Dipl.-Inform. Frank Steiner Web: http://www.bio.ifi.lmu.de/~steiner/
> Lehrstuhl f. Bioinformatik Mail: http://www.bio.ifi.lmu.de/~steiner/m/
> LMU, Amalienstr. 17 Phone: +49 89 2180-4049
> 80333 Muenchen, Germany Fax: +49 89 2180-99-4049
> * Rekursion kann man erst verstehen, wenn man Rekursion verstanden hat. *

Andras Toth wrote

> Hi Frank,
>
> As Matt and me earlier pointed out it sounds like the router is
> sending ICMP Redirects because they are on a common segment and the
> router tries to tell the host a better route exist. You can read more
> about redirects here:
> http://www.cisco.com/c/en/us/support/docs/ip/routing-information-protocol-rip/13714-43.html

Well, it does sound reasonable and like my configuration is bad
because it prevents the better route. I can fix this for the static
hosts, but we also allow laptops with dhcp in this segment, and
they get their information from a dhcp server that is not in my
control, and the same...


> The slowness occurs as the router cannot forward packets in its
> dedicated fast circuits (ASIC or data-plane) because it needs to send
> an ICMP Redirect packet to each packet on the same subnet. This is
> done in the router CPU (control-plane) which is slower than the
> dedicated circuits in the router data-plane.
>
> Try disabling IPv6 redirects, on Cisco routers it's generally "no ipv6
> redirects" on the routed interfaces, such as the L3 ports or VLAN
> Interfaces (SVI). Alternatively fix your topology and routing to
> ensure the return traffic does not go through the router in the same
> subnet.

...holds for the router, it's managed by our data center. I will write
them a mail about this issue and ask for a change in the config, but
I don't know if they will care about nit :-)

Anyway, your explanations helped to understand what's going on and
as I can easily fix this for my own hosts that suffered from this
problem, you both helped me a lot, thank you very much!

If our data center is willing to change the configuration, I'll
let you know if it helped!

cu,
Frank

--
Dipl.-Inform. Frank Steiner Web: http://www.bio.ifi.lmu.de/~steiner/
Lehrstuhl f. Bioinformatik Mail: http://www.bio.ifi.lmu.de/~steiner/m/
LMU, Amalienstr. 17 Phone: +49 89 2180-4049
80333 Muenchen, Germany Fax: +49 89 2180-99-4049
* Rekursion kann man erst verstehen, wenn man Rekursion verstanden hat. *

Andras Toth wrote

> Try disabling IPv6 redirects, on Cisco routers it's generally "no ipv6
> redirects" on the routed interfaces, such as the L3 ports or VLAN
> Interfaces (SVI). Alternatively fix your topology and routing to
> ensure the return traffic does not go through the router in the same
> subnet.

You were both right! The data center people disabled ipv6 redirects
and the speed immediately changed to the full 112 MB/s!

Thanks for all the explanations and solution proposals, it didn't
only solve the problem, I've also learned a lot :-)

cu,
Frank


--
Dipl.-Inform. Frank Steiner Web: http://www.bio.ifi.lmu.de/~steiner/
Lehrstuhl f. Bioinformatik Mail: http://www.bio.ifi.lmu.de/~steiner/m/
LMU, Amalienstr. 17 Phone: +49 89 2180-4049
80333 Muenchen, Germany Fax: +49 89 2180-99-4049
* Rekursion kann man erst verstehen, wenn man Rekursion verstanden hat. *