Dear colleagues,
I've got some serious trouble debugging a problem with some of our
multilink bundles.
I already moved them to an separate LNS, to have a better debugging
chance.
I'm talking about bundles all coming from a large national carrier. All of
them are directly terminated on the LNS, the carrier sends the tunnels to.
No forwarding, no sgbp on these bundles.
At this debugging level, there are only two bundles with two links each,
but the problem is exactly the same when there are more bundles (same
carrier).
Continuously pinging the CPEs at the other end of the bundle shows peaks
of >1000ms (up to >9000ms) and sometimes single to few packets get lost.
debug ppp multilink events at these times says:
Mar 23 13:56:54: Vi128 MLP: Lost fragment timeout, seq 7BEAD
Mar 23 13:56:54: Vi128 MLP: Discard reassembled packet
Mar 23 13:56:54: Vi148 MLP: Lost fragment timeout, seq 6A3CF
Mar 23 13:56:54: Vi148 MLP: Discard reassembled packet
Mar 23 13:57:04: Vi148 MLP: Lost fragment timeout, seq 6A41F
Mar 23 13:57:04: Vi148 MLP: Discard reassembled packet
Mar 23 13:57:10: Vi128 MLP: Lost fragment timeout, seq 7BFDF
Mar 23 13:57:10: Vi128 MLP: Discard reassembled packet
Mar 23 13:57:11: Vi128 MLP: Lost fragment timeout, seq 7BFE1
Mar 23 13:57:11: Vi128 MLP: Discard reassembled packet
Mar 23 13:57:12: Vi128 MLP: Lost fragment timeout, seq 7BFEA
Mar 23 13:57:12: Vi128 MLP: Begin bit lost, discard fragment 7BFEB
lost fragment counters increase at these times, reordered counter steadily
increases all the time.
What drives me crazy is that it's not one single bundle that is having the
problem at a time, but it's ALL of the bundles on this LNS at the SAME
time. It's not a constant problem and happens from time to time. Completly
unpredictable (duration and interval)!
When forwardings those links to a linux LNS running rp-l2tpd, those links
get bundled just fine, no such problem, everything ok.
So I wonder whether there is a common buffer that all bundles share or if
you guys know of any other thing I could check?
I already tried setting
ppp multilink queue depth qos 3
ppp multilink queue depth fifo 3 / 50 / 255
and/or
ppp multilink slippage mru 16
without any luck or significant change.
BTW: where can I check the status of these buffers?
Any hints, tipps, tricks or criticism would be appreciated ;-)
Thanks in advance,
John
P.S.: The individual links themselves are just fine, when using multipath
instead of multilink or just forwarding them to the linux box to let it
bundle them, everything is fine. Just when the 7200 should do it...
everything goes crazy...
Cisco IOS Software, 7200 Software (C7200-A3JK9S-M), Version 12.4(25b),
RELEASE SOFTWARE (fc1)
Cisco 7204VXR (NPE300) processor (revision D) with 229376K/65536K bytes of
memory.
Processor board ID 28711625
R7000 CPU at 262MHz, Implementation 39, Rev 2.1, 256KB L2 Cache
4 slot VXR midplane, Version 2.7
lns3#show int Vi128
Virtual-Access128 is up, line protocol is up
Hardware is Virtual Access interface
Interface is unnumbered. Using address of Loopback11 (3.3.3.3)
MTU 1454 bytes, BW 2000000 Kbit/sec, DLY 100000 usec,
reliability 255/255, txload 1/255, rxload 1/255
Encapsulation PPP, LCP Open, multilink Open
Listen: IPV6CP
Open: IPCP
MLP Bundle vaccess, cloned from AAA, Virtual-Template1
Vaccess status 0x40, loopback not set
Keepalive set (10 sec)
DTR is pulsed for 5 seconds on reset
Last input 00:00:42, output never, output hang never
Last clearing of "show interface" counters 02:52:23
Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0
Queueing strategy: fifo
Output queue: 0/40 (size/max)
5 minute input rate 5000 bits/sec, 9 packets/sec
5 minute output rate 6000 bits/sec, 9 packets/sec
65649 packets input, 4366426 bytes, 0 no buffer
Received 0 broadcasts, 0 runts, 0 giants, 0 throttles
80 input errors, 0 CRC, 18 frame, 0 overrun, 0 ignored, 21 abort
65710 packets output, 5113033 bytes, 0 underruns
0 output errors, 0 collisions, 0 interface resets
0 unknown protocol drops
0 output buffer failures, 0 output buffers swapped out
0 carrier transitions
lns3#show int Vi148
Virtual-Access148 is up, line protocol is up
Hardware is Virtual Access interface
Interface is unnumbered. Using address of Loopback11 (3.3.3.3)
MTU 1454 bytes, BW 2000000 Kbit/sec, DLY 100000 usec,
reliability 255/255, txload 1/255, rxload 1/255
Encapsulation PPP, LCP Open, multilink Open
Listen: IPV6CP
Open: IPCP
MLP Bundle vaccess, cloned from AAA, Virtual-Template1
Vaccess status 0x40, loopback not set
Keepalive set (10 sec)
DTR is pulsed for 5 seconds on reset
Last input 00:00:14, output never, output hang never
Last clearing of "show interface" counters 02:56:51
Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0
Queueing strategy: fifo
Output queue: 0/40 (size/max)
5 minute input rate 1000 bits/sec, 3 packets/sec
5 minute output rate 2000 bits/sec, 3 packets/sec
45163 packets input, 3066362 bytes, 0 no buffer
Received 0 broadcasts, 0 runts, 0 giants, 0 throttles
47 input errors, 0 CRC, 13 frame, 0 overrun, 0 ignored, 10 abort
45715 packets output, 3720617 bytes, 0 underruns
0 output errors, 0 collisions, 0 interface resets
0 unknown protocol drops
0 output buffer failures, 0 output buffers swapped out
0 carrier transitions
lns3#show ppp multilink interface vi128
Virtual-Access128, bundle name is user1@realm/0002.b641.8893
Username is user1@realm
Endpoint discriminator is 0002.b641.8893
Bundle up for 16:30:44, total bandwidth 2000000, load 1/255
Receive buffer limit 48768 bytes, frag timeout 1000 ms
Using relaxed lost fragment detection algorithm.
0/0 fragments/bytes in reassembly list
41 lost fragments, 37978 reordered
39/1245 discarded fragments/bytes, 0 lost received
0x7DA00 received sequence, 0x3ED96 sent sequence
Member links: 2 (max not set, min not set)
upstream:Vi126 (2.2.2.2), since 16:30:44, unsequenced
upstream:Vi144 (2.2.2.2), since 16:30:42, unsequenced
lns3#show ppp multilink interface vi148
Virtual-Access148, bundle name is user2@realm/0002.b641.888d
Username is user2@realm
Endpoint discriminator is 0002.b641.888d
Bundle up for 16:30:51, total bandwidth 2000000, load 1/255
Receive buffer limit 48768 bytes, frag timeout 1000 ms
Using relaxed lost fragment detection algorithm.
0/0 fragments/bytes in reassembly list
24 lost fragments, 27218 reordered
23/774 discarded fragments/bytes, 0 lost received
0x6B390 received sequence, 0x363CB sent sequence
Member links: 2 (max not set, min not set)
upstream:Vi146 (1.1.1.1), since 16:30:51, unsequenced
upstream:Vi147 (1.1.1.1), since 16:30:51, unsequenced
interface Virtual-Template1
description Default Template for L2TP Termination & Multilink Bundles (!)
mtu 1454
ip unnumbered Loopback10
no ip redirects
no ip proxy-arp
ip tcp adjust-mss 1414
no logging event link-status
no snmp trap link-status
ipv6 enable
ipv6 verify unicast reverse-path
no peer default ip address
ppp authentication pap chap callin
ppp ipcp dns 141.1.1.1 8.8.8.8
ppp multilink
ppp multilink fragment disable
end
_______________________________________________
cisco-bba mailing list
cisco-bba@puck.nether.net
https://puck.nether.net/mailman/listinfo/cisco-bba
I've got some serious trouble debugging a problem with some of our
multilink bundles.
I already moved them to an separate LNS, to have a better debugging
chance.
I'm talking about bundles all coming from a large national carrier. All of
them are directly terminated on the LNS, the carrier sends the tunnels to.
No forwarding, no sgbp on these bundles.
At this debugging level, there are only two bundles with two links each,
but the problem is exactly the same when there are more bundles (same
carrier).
Continuously pinging the CPEs at the other end of the bundle shows peaks
of >1000ms (up to >9000ms) and sometimes single to few packets get lost.
debug ppp multilink events at these times says:
Mar 23 13:56:54: Vi128 MLP: Lost fragment timeout, seq 7BEAD
Mar 23 13:56:54: Vi128 MLP: Discard reassembled packet
Mar 23 13:56:54: Vi148 MLP: Lost fragment timeout, seq 6A3CF
Mar 23 13:56:54: Vi148 MLP: Discard reassembled packet
Mar 23 13:57:04: Vi148 MLP: Lost fragment timeout, seq 6A41F
Mar 23 13:57:04: Vi148 MLP: Discard reassembled packet
Mar 23 13:57:10: Vi128 MLP: Lost fragment timeout, seq 7BFDF
Mar 23 13:57:10: Vi128 MLP: Discard reassembled packet
Mar 23 13:57:11: Vi128 MLP: Lost fragment timeout, seq 7BFE1
Mar 23 13:57:11: Vi128 MLP: Discard reassembled packet
Mar 23 13:57:12: Vi128 MLP: Lost fragment timeout, seq 7BFEA
Mar 23 13:57:12: Vi128 MLP: Begin bit lost, discard fragment 7BFEB
lost fragment counters increase at these times, reordered counter steadily
increases all the time.
What drives me crazy is that it's not one single bundle that is having the
problem at a time, but it's ALL of the bundles on this LNS at the SAME
time. It's not a constant problem and happens from time to time. Completly
unpredictable (duration and interval)!
When forwardings those links to a linux LNS running rp-l2tpd, those links
get bundled just fine, no such problem, everything ok.
So I wonder whether there is a common buffer that all bundles share or if
you guys know of any other thing I could check?
I already tried setting
ppp multilink queue depth qos 3
ppp multilink queue depth fifo 3 / 50 / 255
and/or
ppp multilink slippage mru 16
without any luck or significant change.
BTW: where can I check the status of these buffers?
Any hints, tipps, tricks or criticism would be appreciated ;-)
Thanks in advance,
John
P.S.: The individual links themselves are just fine, when using multipath
instead of multilink or just forwarding them to the linux box to let it
bundle them, everything is fine. Just when the 7200 should do it...
everything goes crazy...
Cisco IOS Software, 7200 Software (C7200-A3JK9S-M), Version 12.4(25b),
RELEASE SOFTWARE (fc1)
Cisco 7204VXR (NPE300) processor (revision D) with 229376K/65536K bytes of
memory.
Processor board ID 28711625
R7000 CPU at 262MHz, Implementation 39, Rev 2.1, 256KB L2 Cache
4 slot VXR midplane, Version 2.7
lns3#show int Vi128
Virtual-Access128 is up, line protocol is up
Hardware is Virtual Access interface
Interface is unnumbered. Using address of Loopback11 (3.3.3.3)
MTU 1454 bytes, BW 2000000 Kbit/sec, DLY 100000 usec,
reliability 255/255, txload 1/255, rxload 1/255
Encapsulation PPP, LCP Open, multilink Open
Listen: IPV6CP
Open: IPCP
MLP Bundle vaccess, cloned from AAA, Virtual-Template1
Vaccess status 0x40, loopback not set
Keepalive set (10 sec)
DTR is pulsed for 5 seconds on reset
Last input 00:00:42, output never, output hang never
Last clearing of "show interface" counters 02:52:23
Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0
Queueing strategy: fifo
Output queue: 0/40 (size/max)
5 minute input rate 5000 bits/sec, 9 packets/sec
5 minute output rate 6000 bits/sec, 9 packets/sec
65649 packets input, 4366426 bytes, 0 no buffer
Received 0 broadcasts, 0 runts, 0 giants, 0 throttles
80 input errors, 0 CRC, 18 frame, 0 overrun, 0 ignored, 21 abort
65710 packets output, 5113033 bytes, 0 underruns
0 output errors, 0 collisions, 0 interface resets
0 unknown protocol drops
0 output buffer failures, 0 output buffers swapped out
0 carrier transitions
lns3#show int Vi148
Virtual-Access148 is up, line protocol is up
Hardware is Virtual Access interface
Interface is unnumbered. Using address of Loopback11 (3.3.3.3)
MTU 1454 bytes, BW 2000000 Kbit/sec, DLY 100000 usec,
reliability 255/255, txload 1/255, rxload 1/255
Encapsulation PPP, LCP Open, multilink Open
Listen: IPV6CP
Open: IPCP
MLP Bundle vaccess, cloned from AAA, Virtual-Template1
Vaccess status 0x40, loopback not set
Keepalive set (10 sec)
DTR is pulsed for 5 seconds on reset
Last input 00:00:14, output never, output hang never
Last clearing of "show interface" counters 02:56:51
Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0
Queueing strategy: fifo
Output queue: 0/40 (size/max)
5 minute input rate 1000 bits/sec, 3 packets/sec
5 minute output rate 2000 bits/sec, 3 packets/sec
45163 packets input, 3066362 bytes, 0 no buffer
Received 0 broadcasts, 0 runts, 0 giants, 0 throttles
47 input errors, 0 CRC, 13 frame, 0 overrun, 0 ignored, 10 abort
45715 packets output, 3720617 bytes, 0 underruns
0 output errors, 0 collisions, 0 interface resets
0 unknown protocol drops
0 output buffer failures, 0 output buffers swapped out
0 carrier transitions
lns3#show ppp multilink interface vi128
Virtual-Access128, bundle name is user1@realm/0002.b641.8893
Username is user1@realm
Endpoint discriminator is 0002.b641.8893
Bundle up for 16:30:44, total bandwidth 2000000, load 1/255
Receive buffer limit 48768 bytes, frag timeout 1000 ms
Using relaxed lost fragment detection algorithm.
0/0 fragments/bytes in reassembly list
41 lost fragments, 37978 reordered
39/1245 discarded fragments/bytes, 0 lost received
0x7DA00 received sequence, 0x3ED96 sent sequence
Member links: 2 (max not set, min not set)
upstream:Vi126 (2.2.2.2), since 16:30:44, unsequenced
upstream:Vi144 (2.2.2.2), since 16:30:42, unsequenced
lns3#show ppp multilink interface vi148
Virtual-Access148, bundle name is user2@realm/0002.b641.888d
Username is user2@realm
Endpoint discriminator is 0002.b641.888d
Bundle up for 16:30:51, total bandwidth 2000000, load 1/255
Receive buffer limit 48768 bytes, frag timeout 1000 ms
Using relaxed lost fragment detection algorithm.
0/0 fragments/bytes in reassembly list
24 lost fragments, 27218 reordered
23/774 discarded fragments/bytes, 0 lost received
0x6B390 received sequence, 0x363CB sent sequence
Member links: 2 (max not set, min not set)
upstream:Vi146 (1.1.1.1), since 16:30:51, unsequenced
upstream:Vi147 (1.1.1.1), since 16:30:51, unsequenced
interface Virtual-Template1
description Default Template for L2TP Termination & Multilink Bundles (!)
mtu 1454
ip unnumbered Loopback10
no ip redirects
no ip proxy-arp
ip tcp adjust-mss 1414
no logging event link-status
no snmp trap link-status
ipv6 enable
ipv6 verify unicast reverse-path
no peer default ip address
ppp authentication pap chap callin
ppp ipcp dns 141.1.1.1 8.8.8.8
ppp multilink
ppp multilink fragment disable
end
_______________________________________________
cisco-bba mailing list
cisco-bba@puck.nether.net
https://puck.nether.net/mailman/listinfo/cisco-bba