Mailing List Archive

TCP and PPPoE problems with one type of CPE
We recently upgraded all of our Cisco 7200 broadband routers to NPE-G2's
and some NPE-G1's. The code train we're using is 12.2SB.

In all of our moves/upgrades, everything has gone extremely well.

We have however run into one problem one one particular type of client.
A Zywall5 firewall.

For whatever reason this box will connect/auth fine, it passes ICMP,
UDP, and ESP traffic just fine, but TCP is completely sporadic or
doesnt' work at all. MTU/MSS issue? Well maybe.

We have thousands of other customers both with MSS adjusted and non-MSS
adjusted connections and all works well. We've tried adjusting the MSS
and not for the Zywalls without any luck at all.

What's weird is that when the Zywall first connects it appears to work
fine for just a little while, then hoses up again.

Traffic through the unit (non-TCP) works fine. Even if you browse to
the unit from the Internet to the management web interface on the public
IP, it just hangs up (most of the time).

Even if you try the CLI from telnet on the public side it just hangs.
(it works ok via it's VPN connection however)

We have one customer who has a dozen of these things deployed. They
claim they worked fine on our old routers (pre 12.2SB) and just stopped
working after the upgrades. Keep in mind, this is the only one with the
issue. Even our packet captures show TCP acks make it to the the
Zywall, but then the Zywall just fails to respond.

Just curious if anyone else has run into a similar problem. Zyxel
doesn't seem to be much help and the customer claims they've got the
latest firmware.

--
Robert Blayzor
INOC
rblayzor@inoc.net
http://www.inoc.net/~rblayzor/

NOTICE: alloc: /dev/null: filesystem full
_______________________________________________
cisco-bba mailing list
cisco-bba@puck.nether.net
https://puck.nether.net/mailman/listinfo/cisco-bba
Re: TCP and PPPoE problems with one type of CPE [ In reply to ]
Frank Bulk wrote:
> Doesn't sounds like an easy one to solve.
>
> If I was in that position I would be asking the customer to push back on
> ZyXEL tech support for assistance, assuring the customer of my willingness
> to work with them and ZyXEL tech support.


I actually found out what was causing this. It has to do with TCP
header compression. By default I believe the 7200 (12.2 SB) tcp header
compression is set passive and will only send compressed data to the
connecting PPPoE host if it sends compressed data.

Our packet traces showed this to be the case. When the session came up
a connection initiated without compression was replied to by the Zywall
with TCP header compression in use. From that point on all the TCP
headers from the 7200 side were sent compressed. At that point, the
Zyxel would reply back with something like "VJ compressed TCP (unknown
direction)". (at least that's the flags/indication from Wireshark). But
the Zyxel is clearly complaining about the TCP header compression from
the 7200 end of the connection.

If we turn off header compression completely on the 7200 side, things
work well again. Keep in mind that none of our other CPE's are seeing
this problem. The strange thing is this has worked until we
moved/upgraded things to 12.2SB series routers. So I'm not sure what
changed in header compression on the Cisco end that caused the Zyxels to
start acting up. But it appears to be a Zyxel problem, every other
PPPoE client we've tried works fine.


--
Robert Blayzor
INOC
rblayzor@inoc.net
http://www.inoc.net/~rblayzor/

NOTICE: alloc: /dev/null: filesystem full
_______________________________________________
cisco-bba mailing list
cisco-bba@puck.nether.net
https://puck.nether.net/mailman/listinfo/cisco-bba
Re: TCP and PPPoE problems with one type of CPE [ In reply to ]
Frank Bulk wrote:
> Interesting...were you able to turn off header compression for just that PPPoE connection using RADIUS attributes, or did you have turn it off for that PPPoE terminating interface?

Our RADIUS server sent VJ back to the router, but the router on PPP
interfaces is passive by default. If we implicitly turn it off, then
the Zyxel doesn't send compressed headers.


> What happened when you turned TCP header compression explicitly on?

Forced on would be bad. Since I have thousands of customers behind this
virtual template there was no easy way to turn it on for just this one
customer.

By default I think our RADIUS server sets VJ on (thus making compression
optional), but none of the clients ever really use it. (since it's
broadband after all). I think our new policy will be to just turn off
header compression completely. I think this was popular with lower
speed connections like dialup and ISDN, but now with our minimum pipe
being 512-768K, it's not really an issue anymore. In fact, I think it'd
be rather taxing on the BRAS if we did compress all that broadband since
the word on the street is it causes the packets to become processed
switched.


--
Robert Blayzor
INOC
rblayzor@inoc.net
http://www.inoc.net/~rblayzor/

Design: The activity of preparing for a design review.
_______________________________________________
cisco-bba mailing list
cisco-bba@puck.nether.net
https://puck.nether.net/mailman/listinfo/cisco-bba
Re: TCP and PPPoE problems with one type of CPE [ In reply to ]
You just need to go back two months on this listserv to see that having the
VJ compression attribute on the RADIUS profile was a major CPU penalty for
our 7206VXR....we changed things such that only if the NAS was our RAS box
would the VJ compression attribute be used. This was a leftover item from
our dial-up days.

Frank

-----Original Message-----
From: Robert Blayzor [mailto:rblayzor@inoc.net]
Sent: Saturday, June 23, 2007 5:54 PM
To: Frank Bulk
Cc: cisco-bba@puck.nether.net
Subject: Re: [cisco-bba] TCP and PPPoE problems with one type of CPE

Frank Bulk wrote:
> Interesting...were you able to turn off header compression for just that
PPPoE connection using RADIUS attributes, or did you have turn it off for
that PPPoE terminating interface?

Our RADIUS server sent VJ back to the router, but the router on PPP
interfaces is passive by default. If we implicitly turn it off, then
the Zyxel doesn't send compressed headers.


> What happened when you turned TCP header compression explicitly on?

Forced on would be bad. Since I have thousands of customers behind this
virtual template there was no easy way to turn it on for just this one
customer.

By default I think our RADIUS server sets VJ on (thus making compression
optional), but none of the clients ever really use it. (since it's
broadband after all). I think our new policy will be to just turn off
header compression completely. I think this was popular with lower
speed connections like dialup and ISDN, but now with our minimum pipe
being 512-768K, it's not really an issue anymore. In fact, I think it'd
be rather taxing on the BRAS if we did compress all that broadband since
the word on the street is it causes the packets to become processed
switched.


--
Robert Blayzor
INOC
rblayzor@inoc.net
http://www.inoc.net/~rblayzor/

Design: The activity of preparing for a design review.

_______________________________________________
cisco-bba mailing list
cisco-bba@puck.nether.net
https://puck.nether.net/mailman/listinfo/cisco-bba
Re: TCP and PPPoE problems with one type of CPE [ In reply to ]
Frank Bulk wrote:
> You just need to go back two months on this listserv to see that having the
> VJ compression attribute on the RADIUS profile was a major CPU penalty for
> our 7206VXR....we changed things such that only if the NAS was our RAS box
> would the VJ compression attribute be used. This was a leftover item from
> our dial-up days.


Yep, that's exactly how it kind of lingered around here. Popular in the
dialup days, and still is. I checked several of our NAS devices and VJ
is on almost every connection (if not every).

Carried over into DSL also when DSL first came around and most ISP's
were only offering 256-384K at first. But now that the norm is mostly
1-3Mbit downstream, TCP header compression probably hurts more than it
helps.

I did notice that even though we were set passive, not a lot of PPPoE
clients actually used header compression, in fact very few. It was
enabled on the virtual interface, but "not compressing".

Still though, the Zyxel issue was indeed strange.

--
Robert Blayzor
INOC
rblayzor@inoc.net
http://www.inoc.net/~rblayzor/

FreeBSD, Putting the 'Operating' back into OS!
_______________________________________________
cisco-bba mailing list
cisco-bba@puck.nether.net
https://puck.nether.net/mailman/listinfo/cisco-bba
Re: TCP and PPPoE problems with one type of CPE [ In reply to ]
We had the same problem back in 2004....

>X-BrightmailFiltered: true
>X-Brightmail-Tracker: AAAAAA==
>Date: Tue, 7 Dec 2004 16:59:59 -0800
>From: Dennis Peng <dpeng@cisco.com>
>To: Clayton Zekelman <clayton@mnsi.net>
>Cc: Arie Vayner <ariev@netvision.net.il>, cisco-bba@puck.nether.net
>Subject: Re: [cisco-bba] LNS issues on a 7301
>User-Agent: Mutt/1.4.1i
>X-Spam-Checker-Version: SpamAssassin 3.0.1 (2004-10-22) on e450
>X-Spam-Level:
>X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=unavailable
> version=3.0.1
>
>Clayton Zekelman [clayton@mnsi.net] wrote:
> >
> >
> > It may have indirectly been MTU issues.
> >
> > Somewhere along the way, IOS started treating the RADIUS
> Framed-Compression=Van-Jacobsen-TCP-IP attribute differently.
> >
> > Users who had that attribute set in the RADIUS users file were
> failing. Everyone without it (our DEFAULT entry) was fine. It was
> one of those things we always set during the dialup days, and it
> never appeared to cause a problem on DSL until we went to 12.3(11)T.
>
>For the archives, this problem exists in 12.3(7)T or later. I've
>opened up a bug on this, CSCsa49007. The release note says:
>
>In IOS releases 12.3(7)T or later, if the command <CmdBold>ip tcp
>header-compression<NoCmdBold> is configured on VPDN virtual-access
>interfaces, TCP traffic will fail to pass. The command may also be
>enabled via the RADIUS Framed-Compression attribute. VJ header
>compression does not have to be negotiated successfully for TCP
>traffic to be affected, it just needs to be configured. The workaround
>is to either remove the <CmdBold>ip tcp header-compression<NoCmdBold>
>command, filter the Framed-Compression attribute using the RADIUS
>Attribute Filtering feature in IOS, or remove the attribute from the
>user's RADIUS profile.
>
>If you have RADIUS profiles with the Framed-Compression set to VJ
>header compression, or would like to use VJ header compression on your
>LNS (eg you are aggregating modem links), please open up a TAC case
>and request that they link the case to this bug. I'd greatly appreciate
>your support in documenting your requests/needs.
>
> > I suspect that somehow when the TCP/IP header was decompressed
> with VJ, it grew larger and broke things due to the PPPoE packet
> size limitations.
> >
> > Either way:
> > 12.3(9) - VJ "on" causes no problems.
> > 12.3(11)T - VJ "on" breaks things.
> >
> > I'd rather avoid using ip tcp adjust-mss. From what I remember
> in earlier releases, it blocked creation of Virtual-Access
> Subinterfaces, and caused some performance hits. I don't know if
> this is still true. Since my L2TP tunnels come in on an ATM
> interface, I don't hit the maximum L2TP packet size over Ethernet issue.
>
>TCP adjust-mss is sub-interface capable in 12.3(3) and later
>(CSCea31101). However, anyone using the command on the sub-interface
>should make sure they have the fix for CSCee75569, in 12.3(10) or
>later. Otherwise you will only see TCP adjust-mss sporadically working
>which will cause performance problems.
>
>Dennis
>
> > I opened a TAC case, but the front line guy is still stuck on
> having me make sure I have "enable vpdn" turned on (despite the
> fact that the syntax is "vpdn enable" - which of course I must have
> on to even be able to encounter this problem...<sigh>).
> >
> > I'll send him what I found in the morning.
> >
> >
> > ----- Original Message ---------------
> >
> > Subject: RE: [cisco-bba] LNS issues on a 7301
> > From: "Arie Vayner" <ariev@netvision.net.il>
> > Date: Fri, 3 Dec 2004 10:17:00 +0200
> > To: "Clayton Zekelman" <clayton@MNSi.Net>,
> > <cisco-bba@puck.nether.net>
> >
> > >You may be hitting some MTU issues?
> > >Try using tcp mss-adjust
> > >
> > >Arie
> > >
> > >-----Original Message-----
> > >From: cisco-bba-bounces@puck.nether.net
> > >[mailto:cisco-bba-bounces@puck.nether.net] On Behalf Of Clayton Zekelman
> > >Sent: Thursday, December 02, 2004 5:19 PM
> > >To: cisco-bba@puck.nether.net
> > >Subject: [cisco-bba] LNS issues on a 7301
> > >
> > >
> > >
> > >We just purchased a new 7301 running 12.3(11)T as an LNS.
> > >
> > >Customers are coming in via L2TP tunnels from a Juniper ERX-1400 on an
> > >ATM OC3c interface.
> > >
> > >We just cut over from a 6400 NRP-1 running 12.2(15)T1, and now some
> > >customers who terminate in PTA's are having connectivity issues. These
> > >customers were also working fine on a 6400 NRP-1 running 12.3(9) a few
> > >weeks back.
> > >
> > >Currently, it looks as if the customers we tunnel THROUGH the 7301 work
> > >just fine. Customers terminating are reporting various issues.
> > >Interesting to note that we can ping most of the users from outside our
> > >network just fine. The ones we can't ping I suspect have ping responses
> > >turned off on their CPE routers.
> > >
> > >As a quick fix, I'm tunneling the users through to another LNS. Any
> > >suggestions?
> > >
> > >Here's our Virtual Template:
> > >
> > >interface Virtual-Template1
> > > mtu 1492
> > > ip unnumbered GigabitEthernet0/0
> > > no logging event link-status
> > > load-interval 30
> > > peer default ip address pool dynamic1
> > > ppp authentication pap ppp_local
> > > ppp authorization ppp_local
> > > ppp ipcp dns 216.8.137.204 216.8.137.203
> > >
> > >
> > >
> > >---
> > >Clayton Zekelman
> > >Managed Network Systems Inc. (MNSi)
> > >344-300 Tecumseh Rd. E.
> > >Windsor, Ontario
> > >N8X 5E8
> > >
> > >tel. 519-985-8410
> > >fax. 519-258-3009
> > >
> > >_______________________________________________
> > >cisco-bba mailing list
> > >cisco-bba@puck.nether.net
> > >https://puck.nether.net/mailman/listinfo/cisco-bba
> > >
> > >
> > >
> >
> > _______________________________________________
> > cisco-bba mailing list
> > cisco-bba@puck.nether.net
> > https://puck.nether.net/mailman/listinfo/cisco-bba

---
Clayton Zekelman
Managed Network Systems Inc. (MNSi)
344-300 Tecumseh Rd. E.
Windsor, Ontario
N8X 5E8

tel. 519-985-8410
fax. 519-985-8409

_______________________________________________
cisco-bba mailing list
cisco-bba@puck.nether.net
https://puck.nether.net/mailman/listinfo/cisco-bba