Mailing List Archive: X.75 (LAPB-TA) sessions getting "stuck"? (AS5300, 12.3)

X.75 (LAPB-TA) sessions getting "stuck"? (AS5300, 12.3)

Oct 2, 2009, 4:08 AM

Post #1 of 7 (7078 views)

Hi,

I have this AS5300 that is handling incoming X.75/LAPB-TA calls, and
every few weeks, it seems to get stuck in weird ways.

Incoming calls connect fine, and the initial few bytes ("username, password")
are exchanged fine. After that, nothing sent by the AS5300 to the ISDN
side arrives at the caller location - but "debug lapb-ta packets" *does*
log the packets just fine...:

Oct 2 12:56:01: %LINK-3-UPDOWN: Interface Serial0:0, changed state to up
Oct 2 12:56:01: %ISDN-6-CONNECT: Interface Serial0:0 is now connected to 8912003682 N/A
Oct 2 12:56:04: LAPB-TA: Autodetect trying to detect LAPB on Se0:0
Oct 2 12:56:04: sampled pkt: 2 bytes: 1 3F.. match
Oct 2 12:56:04: LAPBTA: get_ll_config: Serial0:0
Oct 2 12:56:04: LAPBTA127: vty allocated for Se0:0
Oct 2 12:56:04: LAPBTA127: process 81
Oct 2 12:56:04: Se0:0: LAPB-TA started
Oct 2 12:56:04: LAPBTA: service change: LAPB physical layer up, context 62CC3A58 interface up, protocol down
Oct 2 12:56:04: LAPBTA: service change: , context 62CC3A58 up
Oct 2 12:56:04: LAPB-TA: Se0:0, 14 sent
Oct 2 12:56:04: LAPB-TA: Se0:0, 1 rcvd
Oct 2 12:56:04: LAPB-TA: Se0:0, 3 rcvd
Oct 2 12:56:04: LAPB-TA: Se0:0, 4 rcvd
Oct 2 12:56:04: LAPB-TA: Se0:0, 3 sent
Oct 2 12:56:04: LAPB-TA: Se0:0, 5 rcvd
Oct 2 12:56:04: LAPB-TA: Se0:0, 5 sent
Oct 2 12:56:04: LAPB-TA: Se0:0, 2 rcvd
Oct 2 12:56:04: LAPB-TA: Se0:0, 6 sent
Oct 2 12:56:04: LAPB-TA: Se0:0, 10 sent
Oct 2 12:56:05: %LINEPROTO-5-UPDOWN: Line protocol on Interface Serial0:0, changed state to up
Oct 2 12:56:07: LAPB-TA: Se0:0, 5 sent
Oct 2 12:56:07: LAPB-TA: Se0:0, 21 sent

- about here, it breaks. Nothing ever arrives at the calling side anymore
(and nothing the other side shows up as "rcvd" either).

Oct 2 12:56:07: LAPB-TA: Se0:0, 4 sent
Oct 2 12:56:17: LAPB-TA: Se0:0, 21 sent
Oct 2 12:56:17: LAPB-TA: Se0:0, 4 sent
Oct 2 12:56:27: LAPB-TA: Se0:0, 21 sent
Oct 2 12:56:27: LAPB-TA: Se0:0, 4 sent
Oct 2 12:56:37: LAPB-TA: Se0:0, 21 sent
Oct 2 12:56:37: LAPB-TA: Se0:0, 4 sent
Oct 2 12:57:11: LAPBTA127: stopped, removing lapb
Oct 2 12:57:11: LAPBTA: service change: LAPB deleted, context 62CC3A58 interface up, protocol down, disconnecting
Oct 2 12:57:11: LAPBTA: removing context 62CC3A58
Oct 2 12:57:11: %ISDN-6-DISCONNECT: Interface Serial0:0 disconnected from 8912003682 , call lasted 70 seconds
Oct 2 12:57:11: LAPBTA: close 62DDA294, line 127
Oct 2 12:57:11: %LINK-3-UPDOWN: Interface Serial0:0, changed state to down

... nothing unusual in the log, it just seems as if "LAPB-TA" loses contact
to "the ISDN side of things". No real idea how to debug this further...

If I reboot the box, everything is back to working. "shut / no shut" on
the serial0:15 interface did *not* change anything.

This is currently running 12.3(26), which is the latest 12.3 release.

I'd bet willing to try an upgrade to 12.4, but this brings up the next
question - is the 5300 supported in 12.4? The IOS files that can be
found on the FTP server in 12.4/12.4.25b/as5300/ are all named "as5350-...",
so I'm wondering whether they will work on the 5300 as well, or whether
this is incompatible, and the directory naming is just weird.

(We could also downgrade to 12.2, if this is a new problem in 12.3...)

gert

--
USENET is *not* the non-clickable part of WWW!
//www.muc.de/~gert/
Gert Doering - Munich, Germany gert@greenie.muc.de
fax: +49-89-35655025 gert@net.informatik.tu-muenchen.de

Re: X.75 (LAPB-TA) sessions getting "stuck"? (AS5300, 12.3) [ In reply to ]

Aaron at cisco

Oct 2, 2009, 11:40 AM

Post #2 of 7 (6856 views)

Permalink

Hi Gert,

> I have this AS5300 that is handling incoming X.75/LAPB-TA calls, and
> every few weeks, it seems to get stuck in weird ways.

Well, since I was the one who talked you into using the 5300 for
this, I suppose I bear some responsibility here.

> Incoming calls connect fine, and the initial few bytes ("username, password")
> are exchanged fine. After that, nothing sent by the AS5300 to the ISDN
> side arrives at the caller location - but "debug lapb-ta packets" *does*
> log the packets just fine...:
>
> Oct 2 12:56:01: %LINK-3-UPDOWN: Interface Serial0:0, changed state to up
> Oct 2 12:56:01: %ISDN-6-CONNECT: Interface Serial0:0 is now connected to 8912003682 N/A
> Oct 2 12:56:04: LAPB-TA: Autodetect trying to detect LAPB on Se0:0
> Oct 2 12:56:04: sampled pkt: 2 bytes: 1 3F.. match
> Oct 2 12:56:04: LAPBTA: get_ll_config: Serial0:0
> Oct 2 12:56:04: LAPBTA127: vty allocated for Se0:0
> Oct 2 12:56:04: LAPBTA127: process 81
> Oct 2 12:56:04: Se0:0: LAPB-TA started
> Oct 2 12:56:04: LAPBTA: service change: LAPB physical layer up, context 62CC3A58 interface up, protocol down
> Oct 2 12:56:04: LAPBTA: service change: , context 62CC3A58 up
> Oct 2 12:56:04: LAPB-TA: Se0:0, 14 sent
> Oct 2 12:56:04: LAPB-TA: Se0:0, 1 rcvd
> Oct 2 12:56:04: LAPB-TA: Se0:0, 3 rcvd
> Oct 2 12:56:04: LAPB-TA: Se0:0, 4 rcvd
> Oct 2 12:56:04: LAPB-TA: Se0:0, 3 sent
> Oct 2 12:56:04: LAPB-TA: Se0:0, 5 rcvd
> Oct 2 12:56:04: LAPB-TA: Se0:0, 5 sent
> Oct 2 12:56:04: LAPB-TA: Se0:0, 2 rcvd
> Oct 2 12:56:04: LAPB-TA: Se0:0, 6 sent
> Oct 2 12:56:04: LAPB-TA: Se0:0, 10 sent
> Oct 2 12:56:05: %LINEPROTO-5-UPDOWN: Line protocol on Interface Serial0:0, changed state to up
> Oct 2 12:56:07: LAPB-TA: Se0:0, 5 sent
> Oct 2 12:56:07: LAPB-TA: Se0:0, 21 sent
>
> - about here, it breaks. Nothing ever arrives at the calling side anymore
> (and nothing the other side shows up as "rcvd" either).
>
> Oct 2 12:56:07: LAPB-TA: Se0:0, 4 sent
> Oct 2 12:56:17: LAPB-TA: Se0:0, 21 sent
> Oct 2 12:56:17: LAPB-TA: Se0:0, 4 sent
> Oct 2 12:56:27: LAPB-TA: Se0:0, 21 sent
> Oct 2 12:56:27: LAPB-TA: Se0:0, 4 sent
> Oct 2 12:56:37: LAPB-TA: Se0:0, 21 sent
> Oct 2 12:56:37: LAPB-TA: Se0:0, 4 sent
> Oct 2 12:57:11: LAPBTA127: stopped, removing lapb
> Oct 2 12:57:11: LAPBTA: service change: LAPB deleted, context 62CC3A58 interface up, protocol down, disconnecting
> Oct 2 12:57:11: LAPBTA: removing context 62CC3A58
> Oct 2 12:57:11: %ISDN-6-DISCONNECT: Interface Serial0:0 disconnected from 8912003682 , call lasted 70 seconds
> Oct 2 12:57:11: LAPBTA: close 62DDA294, line 127
> Oct 2 12:57:11: %LINK-3-UPDOWN: Interface Serial0:0, changed state to down
>
> ... nothing unusual in the log, it just seems as if "LAPB-TA" loses contact
> to "the ISDN side of things". No real idea how to debug this further...

My generic approach would be something like this:

Compare and contrast what happens when things are working as desired,
what what is happening when things are not.

Things to compare:

- debugs:
debug isdn q931
debug lapb-ta data
debug lapb-ta error
debug lapb-ta event
debug modem

- during that little 1-minute window where things are stuck, get:

show line vty <n> <= on the VTY used by the LAPB-TA session
show interface serial0:<n> <= on the B channel used by the call

compare how the vty and the serial interface look, with how they
look when things are working right

> If I reboot the box, everything is back to working. "shut / no shut" on
> the serial0:15 interface did *not* change anything.

I'm guessing that you've got some problem with a vty, but that's just
a guess. If so, then maybe you could do a "clear line" on the vty
to get things unstuck.

> This is currently running 12.3(26), which is the latest 12.3 release.
>
>
> I'd bet willing to try an upgrade to 12.4, but this brings up the next
> question - is the 5300 supported in 12.4?

Nope. 12.3 mainline is the last branch to support the 5300.

> The IOS files that can be
> found on the FTP server in 12.4/12.4.25b/as5300/ are all named "as5350-...",
> so I'm wondering whether they will work on the 5300 as well, or whether
> this is incompatible, and the directory naming is just weird.

The 5350 is an entirely different system from the 5300, and neither's
images will boot on the other.

> (We could also downgrade to 12.2, if this is a new problem in 12.3...)
>
> gert

Yeah, if you just want to try different code versions, you could try
(among stuff that's still on CCO):

12.3(3i)
12.2(15)T7

Good luck,

Aaron
_______________________________________________
cisco-nas mailing list
cisco-nas@puck.nether.net
https://puck.nether.net/mailman/listinfo/cisco-nas

Re: X.75 (LAPB-TA) sessions getting "stuck"? (AS5300, 12.3) [ In reply to ]

gert at greenie

Oct 2, 2009, 1:44 PM

Post #3 of 7 (6848 views)

Permalink

Hi,

On Fri, Oct 02, 2009 at 11:40:33AM -0700, Aaron Leonard wrote:
> > I have this AS5300 that is handling incoming X.75/LAPB-TA calls, and
> > every few weeks, it seems to get stuck in weird ways.
>
> Well, since I was the one who talked you into using the 5300 for
> this, I suppose I bear some responsibility here.

Well, originally it was my idea :-) - but I had certainly hoped for your
help here.

[..]
> Compare and contrast what happens when things are working as desired,
> what what is happening when things are not.

Things look perfectly normal from the debug output - the call connects,
the first few bytes (encapsulated in X.75) are exchanged normally, and
then "something" stops in the chain

data -> AS5300(X.75) -> AS5300(ISDN B-Channel) -> ISDN network -> caller

the "debug lapb-ta packets" claims that packets are X.75 encapsulated,
and the caller claims "nothing arriving here". I have no means to
figure out whether the X.75 layer in the AS5300 is not passing the
frames properly to the ISDN layer, or they get lost "somewhere in the
ISDN network".

... which might very well be, given that ISDN Data is something a number
of Telcos don't really support any longer... "who wants this when you
can have DSL".

> - during that little 1-minute window where things are stuck, get:
>
> show line vty <n> <= on the VTY used by the LAPB-TA session
> show interface serial0:<n> <= on the B channel used by the call
>
> compare how the vty and the serial interface look, with how they
> look when things are working right

It's not actually a "1-minute window where things are stuck" - maybe that
was misleading from the log and from my description. When they are stuck,
they stay stuck - a reload was needed to bring it back to working.

But I'll definitely give "show line" and "show int" a very close look
when/if it happens next time.

> > If I reboot the box, everything is back to working. "shut / no shut" on
> > the serial0:15 interface did *not* change anything.
>
> I'm guessing that you've got some problem with a vty, but that's just
> a guess. If so, then maybe you could do a "clear line" on the vty
> to get things unstuck.

The weird thing is, the effect persists across calls.

So for the next call, it will behave identically - it will connect,
exchange a few frames just fine, and then get stuck again.

What I did not test yet was to do multiple calls in parallel, to see
whether the effect only hits a specific vty or "all of them". Thanks
for that idea.

> > This is currently running 12.3(26), which is the latest 12.3 release.
> >
> > I'd bet willing to try an upgrade to 12.4, but this brings up the next
> > question - is the 5300 supported in 12.4?
>
> Nope. 12.3 mainline is the last branch to support the 5300.

OK - so I'm not going to try this :)

> > The IOS files that can be
> > found on the FTP server in 12.4/12.4.25b/as5300/ are all named "as5350-...",
> > so I'm wondering whether they will work on the 5300 as well, or whether
> > this is incompatible, and the directory naming is just weird.
>
> The 5350 is an entirely different system from the 5300, and neither's
> images will boot on the other.

Interesting artefact, to have 5350 IOS in the 5300 subdirectory, then :-)

> > (We could also downgrade to 12.2, if this is a new problem in 12.3...)
>
> Yeah, if you just want to try different code versions, you could try
> (among stuff that's still on CCO):
>
> 12.3(3i)
> 12.2(15)T7

Under normal circumstances, I'd try 12.2 "main line" next - but since
you've recommended these two versions, what's special about them? "Major
surgery between 12.3(3) and 12.3(4)"?

gert

--
USENET is *not* the non-clickable part of WWW!
//www.muc.de/~gert/
Gert Doering - Munich, Germany gert@greenie.muc.de
fax: +49-89-35655025 gert@net.informatik.tu-muenchen.de

Re: X.75 (LAPB-TA) sessions getting "stuck"? (AS5300, 12.3) [ In reply to ]

Aaron at cisco

Oct 2, 2009, 2:16 PM

Post #4 of 7 (6865 views)

Permalink

> The weird thing is, the effect persists across calls.
>
> So for the next call, it will behave identically - it will connect,
> exchange a few frames just fine, and then get stuck again.
>
> What I did not test yet was to do multiple calls in parallel, to see
> whether the effect only hits a specific vty or "all of them". Thanks
> for that idea.

I think the algorithm will always hit the first free vty. So, if
let's say vty 0 is in use and vty 1 is (somehow) messed up, but free,
then the next LAPB-TA call would hit vty 1. If, right after that,
another LAPB-TA call were to come in, it might hit vty 2 and maybe be
OK.

So if you could get the bad vty out of harms way (e.g. by removing
"transport input lapb-ta" from it), then this might get you going
again.

>>> (We could also downgrade to 12.2, if this is a new problem in 12.3...)
>> Yeah, if you just want to try different code versions, you could try
>> (among stuff that's still on CCO):
>>
>> 12.3(3i)
>> 12.2(15)T7
>
> Under normal circumstances, I'd try 12.2 "main line" next - but since
> you've recommended these two versions, what's special about them? "Major
> surgery between 12.3(3) and 12.3(4)"?

Actually, the really nice code for the 5300 was the 12.2XB train.
Ah, 12.2(2)XB11 ... good times, good times. (But you can't get it
any more.) 12.2(15)T7 is the closest that we have to 12.2(2)XB11, so
that's why I mentioned that.

12.2 mainline, on the other hand, was never used much on the 5300
- it didn't get the XB goodness. It didn't support V.92, which we
made a big deal of back in the day, although looking back at it,
it didn't really add up to much. So I suppose you could try 12.2
if you like.

When you come down to it, LAPB-TA never got exercised very much.
So I wouldn't expect to see much difference in different versions.

Best,

Aaron
_______________________________________________
cisco-nas mailing list
cisco-nas@puck.nether.net
https://puck.nether.net/mailman/listinfo/cisco-nas

Re: X.75 (LAPB-TA) sessions getting "stuck"? (AS5300, 12.3) [ In reply to ]

gert at greenie

Nov 3, 2009, 8:16 AM

Post #5 of 7 (6689 views)

Permalink

Hi,

so now it had happened again... (the X.75-got-stuck thing).

On Fri, Oct 02, 2009 at 02:16:44PM -0700, Aaron Leonard wrote:
> I think the algorithm will always hit the first free vty. So, if
> let's say vty 0 is in use and vty 1 is (somehow) messed up, but free,
> then the next LAPB-TA call would hit vty 1. If, right after that,
> another LAPB-TA call were to come in, it might hit vty 2 and maybe be
> OK.
>
> So if you could get the bad vty out of harms way (e.g. by removing
> "transport input lapb-ta" from it), then this might get you going
> again.

I think it is not stuck to the VTY, but to "the first, second, ...
parallel X.75 session".

For a while, I could make X.75 work again by calling on two different lines
in parallel (the first one would reliably get stuck, the second worked
for a few calls) - so it wasn't "all of the X.75 is broken, just the
first call".

I tried disabling the first few VTYs, but that didn't change anything -
the first call was still stuck, but on a higher VTY number now.

After a few tests, I now have two stuck X.75 sessions in parallel,
and have no more test devices to make further calls...

Is there any trick to reset the internal state of the X.75 protocol
machinery?

Like "remove lapb-ta autodetection from all ISDN interfaces" or so
(which didn't have any effect, btw)?

gert

--
USENET is *not* the non-clickable part of WWW!
//www.muc.de/~gert/
Gert Doering - Munich, Germany gert@greenie.muc.de
fax: +49-89-35655025 gert@net.informatik.tu-muenchen.de

Re: X.75 (LAPB-TA) sessions getting "stuck"? (AS5300, 12.3) [ In reply to ]

gert at greenie

Nov 3, 2009, 8:26 AM

Post #6 of 7 (6665 views)

Permalink

Hi,

On Tue, Nov 03, 2009 at 05:16:35PM +0100, Gert Doering wrote:
> Like "remove lapb-ta autodetection from all ISDN interfaces" or so
> (which didn't have any effect, btw)?

Ah, now that's interesting.

A "not-yet-messed-up" B-Channel interface has:

AS53-k02-Test#sh int s0:3
Serial0:3 is down, line protocol is down
Input queue: 0/10/0/0 (size/max/drops/flushes); Total output drops: 0

while a "messed-up" B-Channel interface has:

AS53-k02-Test#sh int s0:1
Serial0:1 is down, line protocol is down
Input queue: 6/10/0/0 (size/max/drops/flushes); Total output drops: 58

if I increase the input hold queue to 100, there is still a packet
stuck in there:

Input queue: 6/100/0/0 (size/max/drops/flushes); Total output drops: 58

... but the X.75 calls are working again.

After every further call, more packets get stuck in the input queue...

Input queue: 11/100/0/0 (size/max/drops/flushes); Total output drops: 58

(I'm not sure yet whether it's "one stuck packet per call" or has some
more random element to it).

So. Next question :-) - is there a way to figure out what sort of
"packets" are hanging in the input queue, and to get rid of them?

(I remember security advisories with input queues getting wedged, but
that was related to IP protocols, not X.75 protocol handling...)

gert
--
USENET is *not* the non-clickable part of WWW!
//www.muc.de/~gert/
Gert Doering - Munich, Germany gert@greenie.muc.de
fax: +49-89-35655025 gert@net.informatik.tu-muenchen.de

Re: X.75 (LAPB-TA) sessions getting "stuck"? (AS5300, 12.3) [ In reply to ]

Aaron at cisco

Nov 3, 2009, 1:47 PM

Post #7 of 7 (6675 views)

Permalink

Good sleuthwork, Gert. Looks like you're suffering from a input buffer
leak.

"show buffer input-interface <interface-name> dump" would dump out the
packets which can then be hand decoded (or munged into a format amenable
to being read by Wireshark.)

As to whether we could figure out some kind of workaround or bugfix ...
at some point, you'd want to have a TAC case open (not that the 5300 is
supported any longer) and get a DE assigned to figure out the bug.

Aaron

------------------------------------------------------------------------

Gert Doering wrote:
> Hi,
>
> On Tue, Nov 03, 2009 at 05:16:35PM +0100, Gert Doering wrote:
>
>> Like "remove lapb-ta autodetection from all ISDN interfaces" or so
>> (which didn't have any effect, btw)?
>>
>
> Ah, now that's interesting.
>
> A "not-yet-messed-up" B-Channel interface has:
>
> AS53-k02-Test#sh int s0:3
> Serial0:3 is down, line protocol is down
> Input queue: 0/10/0/0 (size/max/drops/flushes); Total output drops: 0
>
> while a "messed-up" B-Channel interface has:
>
> AS53-k02-Test#sh int s0:1
> Serial0:1 is down, line protocol is down
> Input queue: 6/10/0/0 (size/max/drops/flushes); Total output drops: 58
>
> if I increase the input hold queue to 100, there is still a packet
> stuck in there:
>
> Input queue: 6/100/0/0 (size/max/drops/flushes); Total output drops: 58
>
> ... but the X.75 calls are working again.
>
> After every further call, more packets get stuck in the input queue...
>
> Input queue: 11/100/0/0 (size/max/drops/flushes); Total output drops: 58
>
> (I'm not sure yet whether it's "one stuck packet per call" or has some
> more random element to it).
>
>
> So. Next question :-) - is there a way to figure out what sort of
> "packets" are hanging in the input queue, and to get rid of them?
>
> (I remember security advisories with input queues getting wedged, but
> that was related to IP protocols, not X.75 protocol handling...)
>
> gert
>
_______________________________________________
cisco-nas mailing list
cisco-nas@puck.nether.net
https://puck.nether.net/mailman/listinfo/cisco-nas