Mailing List Archive

[nsp] 12.0(22)S - avoid it
Hello,

A few days ago, I upgraded five GSR from 12.0(19)S2 to the brand new
12.0(22)S (the one that merges the S and ST trains)... What a mistake !

1/ Some STM-16 links between involved GSR became to flap. Got "Loss of
signal" one side and "Loss of frame" other side, got a _huge_ amount of
"Input errors" even when doing a "loopback internal"... The links flaped
more when they where taken out of production, interresting.

2/ Bogus BGP advertisements appeared. A "show ip bgp neighbor X
advertised" was correct, but the peer actually didn't received any of
the prefixes. "clear soft out" or "clear out" didn't solved the issue, I
had to reset the session. This happen on different routers, on different
IX...

3/ A "neighbor X weight Z" (where X can be an actual BGP peer or a peer
group) is accepted by the router but not shown in a "show run" nor
applied. But an inbound route-map with a "set weight" was still working.

I also noticed some strange latencies in ssh or telnet (you known, when
the charaters you type are echoed slowly). The averaged CPU usage was 8%
before, 11% after the upgrade. So is the memory usage, 20MB more than
with the previous version...

Well, all that I can say is that this version has some issues (and, yes, I
was foolish to put in production a 12.0S version with no digit behind the
'S' ;-) ). Now reverting to 12.0(21)ST3. Too bad, the new SSO (successor
of RPR+) was very promising...

If someone had any issue with 12.0(21)ST3, please tell me!

--
Antoine Versini
Re: [nsp] 12.0(22)S - avoid it [ In reply to ]
Unfortunately I heard about 12.0(22)S actually is 12.0(22)ST after I have
upgraded our core routers to 12.0(22)S, this was intended to solve the
dCAR on dot1q interfaces bug. A 7576 crashed 14 hours after
upgrade right in the middle of peak time with output stuck and cbus
complex on FE interface. We have reverted back to 12.0(21)S3.

I am very suprised that Cisco has decided to merge 12.0ST train into S
train without telling people exactly what they have done since stability
is supposed to be the top priority for S-train and people upgrade to the
latest version to avoid known bugs and hoping to improve stability.

In your case, 12.0(21)S3 is still the most stable version from my
experience, I would rather run 12.0(21)S3 and move the affected interface
to another FE port on a separate VIP card to work around the dCAR bug.

Regards,

Leon Chang, CCNP, CCDP
Senior Network Engineer
The Internet Group Ltd. (IHUG), Auckland, New Zealand AS7657
PH: +64 9 359 2710
---------------------------------------------------------------------

On Sat, 24 Aug 2002, Antoine Versini wrote:

> Hello,
>
> A few days ago, I upgraded five GSR from 12.0(19)S2 to the brand new
> 12.0(22)S (the one that merges the S and ST trains)... What a mistake !
>
> 1/ Some STM-16 links between involved GSR became to flap. Got "Loss of
> signal" one side and "Loss of frame" other side, got a _huge_ amount of
> "Input errors" even when doing a "loopback internal"... The links flaped
> more when they where taken out of production, interresting.
>
> 2/ Bogus BGP advertisements appeared. A "show ip bgp neighbor X
> advertised" was correct, but the peer actually didn't received any of
> the prefixes. "clear soft out" or "clear out" didn't solved the issue, I
> had to reset the session. This happen on different routers, on different
> IX...
>
> 3/ A "neighbor X weight Z" (where X can be an actual BGP peer or a peer
> group) is accepted by the router but not shown in a "show run" nor
> applied. But an inbound route-map with a "set weight" was still working.
>
> I also noticed some strange latencies in ssh or telnet (you known, when
> the charaters you type are echoed slowly). The averaged CPU usage was 8%
> before, 11% after the upgrade. So is the memory usage, 20MB more than
> with the previous version...
>
> Well, all that I can say is that this version has some issues (and, yes, I
> was foolish to put in production a 12.0S version with no digit behind the
> 'S' ;-) ). Now reverting to 12.0(21)ST3. Too bad, the new SSO (successor
> of RPR+) was very promising...
>
> If someone had any issue with 12.0(21)ST3, please tell me!
>
> --
> Antoine Versini
>
> _______________________________________________
> cisco-nsp mailing list real_name)s@puck.nether.net
> http://puck.nether.net/mailman/listinfo/cisco-nsp
> archive at http://puck.nether.net/pipermail/cisco-nsp/
>
Re: [nsp] 12.0(22)S - avoid it [ In reply to ]
I believe that 12.0(21)S4 is out which may fix some problems
that were in S3 that people may have encountered.


On Sat, Aug 24, 2002 at 11:23:33PM +1200, Leon Chang wrote:
> Unfortunately I heard about 12.0(22)S actually is 12.0(22)ST after I have
> upgraded our core routers to 12.0(22)S, this was intended to solve the
> dCAR on dot1q interfaces bug. A 7576 crashed 14 hours after
> upgrade right in the middle of peak time with output stuck and cbus
> complex on FE interface. We have reverted back to 12.0(21)S3.
>
> I am very suprised that Cisco has decided to merge 12.0ST train into S
> train without telling people exactly what they have done since stability
> is supposed to be the top priority for S-train and people upgrade to the
> latest version to avoid known bugs and hoping to improve stability.

I'll leave this one alone.

- jared

--
Jared Mauch | pgp key available via finger from jared@puck.nether.net
clue++; | http://puck.nether.net/~jared/ My statements are only mine.
Re: [nsp] 12.0(22)S - avoid it [ In reply to ]
On Sat, Aug 24, 2002 at 11:22:42AM -0400, Jared Mauch wrote:
> I believe that 12.0(21)S4 is out which may fix some problems
> that were in S3 that people may have encountered.

s/may/do/g

:-)

The PA-E3/T3 bug was a grave one...


Best regards,
Daniel
Re: [nsp] 12.0(22)S - avoid it [ In reply to ]
> Unfortunately I heard about 12.0(22)S actually is 12.0(22)ST after I have
> upgraded our core routers to 12.0(22)S, this was intended to solve the
> dCAR on dot1q interfaces bug. A 7576 crashed 14 hours after
> upgrade right in the middle of peak time with output stuck and cbus
> complex on FE interface. We have reverted back to 12.0(21)S3.

We are running a 7507 with 12.0(22)S for 3 weeks, 3 days, 5 hours, 43
minutes now ...
2 full feeds and 5 other sessions with 5 routers in the european IX-es.
1 VIP4-80 w/ 256/64 with a PA-A3-OC3 and a GEIP+.

So far, the only problems we've got were because the GEIP+ has only 64M of
DRAM, wich is really a BIG problem, as the DCEF ussualy crashes and
doesn't come back...

First we didn't wanted to run 12.0(22)S, but it was the only S-whatever
release which worked with our GEIP+. In all other versions as soon as we
enabled any kind of multicast (ospf for example) on the interface, it
goes to line protocol down and stays like that until we remove the cables
and put them back ... strange thing isn't it ? :(


> I am very suprised that Cisco has decided to merge 12.0ST train into S
> train without telling people exactly what they have done since stability
> is supposed to be the top priority for S-train and people upgrade to the
> latest version to avoid known bugs and hoping to improve stability.

Cisco said that the S series will be continued only by 12.0.21Sx
updates... I don't really know if this is for marketing reasons, as 12.2S
probably will soon be available ... (or am I mistaking ?)

--

George Boulescu
Senior Network Engineer
RoEduNet Bucharest NOC
CCAI, CCNA
Re: [nsp] 12.0(22)S - avoid it [ In reply to ]
Hi,

On Sat, Aug 24, 2002 at 11:23:33PM +1200, Leon Chang wrote:
> In your case, 12.0(21)S3 is still the most stable version from my
> experience, I would rather run 12.0(21)S3 and move the affected interface
> to another FE port on a separate VIP card to work around the dCAR bug.

12.0(21)S3 has issues with PA-E3 and PA-T3 on 7200/7500 platforms - for
some people, those lines just *stop working* completely.

12.0(21)S4 fixes that (tested myself) but the interface counters are
finally completely broken.

gert,
waiting for 12.0(21)S5

--
USENET is *not* the non-clickable part of WWW!
//www.muc.de/~gert/
Gert Doering - Munich, Germany gert@greenie.muc.de
fax: +49-89-35655025 gert.doering@physik.tu-muenchen.de
Re: [nsp] 12.0(22)S - avoid it [ In reply to ]
Hi,

On Sat, Aug 24, 2002 at 11:37:43PM +0300, George Boulescu wrote:
> Cisco said that the S series will be continued only by 12.0.21Sx
> updates... I don't really know if this is for marketing reasons, as 12.2S
> probably will soon be available ... (or am I mistaking ?)

Rumors say "not before November".

gert
--
USENET is *not* the non-clickable part of WWW!
//www.muc.de/~gert/
Gert Doering - Munich, Germany gert@greenie.muc.de
fax: +49-89-35655025 gert.doering@physik.tu-muenchen.de
Re: [nsp] 12.0(22)S - avoid it [ In reply to ]
Hello,

Gert Doering wrote:
>
> Hi,
>
...
> 12.0(21)S3 has issues with PA-E3 and PA-T3 on 7200/7500 platforms - for
> some people, those lines just *stop working* completely.

Can anyone please provide a bug ID for the PA-T3 problem mentioned
above?

Thank you


>
> 12.0(21)S4 fixes that (tested myself) but the interface counters are
> finally completely broken.
>
> gert,
> waiting for 12.0(21)S5
RE: [nsp] 12.0(22)S - avoid it [ In reply to ]
I have been trying to find this as well. Please forward it on if you have
it.

Thanks,

nate

-----Original Message-----
From: Steven W. Raymond [mailto:steven_raymond@eli.net]
Sent: Wednesday, September 04, 2002 2:25 PM
To: Gert Doering
Cc: cisco-nsp@puck.nether.net; sraymond@eli.net
Subject: Re: [nsp] 12.0(22)S - avoid it


Hello,

Gert Doering wrote:
>
> Hi,
>
...
> 12.0(21)S3 has issues with PA-E3 and PA-T3 on 7200/7500 platforms -
> for some people, those lines just *stop working* completely.

Can anyone please provide a bug ID for the PA-T3 problem mentioned above?

Thank you


>
> 12.0(21)S4 fixes that (tested myself) but the interface counters are
> finally completely broken.
>
> gert,
> waiting for 12.0(21)S5
_______________________________________________
cisco-nsp mailing list real_name)s@puck.nether.net
http://puck.nether.net/mailman/listinfo/cisco-nsp
archive at http://puck.nether.net/pipermail/cisco-nsp/
Re: [nsp] 12.0(22)S - avoid it [ In reply to ]
> > 12.0(21)S3 has issues with PA-E3 and PA-T3 on 7200/7500 platforms - for
> > some people, those lines just *stop working* completely.
>
> Can anyone please provide a bug ID for the PA-T3 problem mentioned
> above?

CSCin13850. Experienced it personally. Not nice at all.

Steinar Haug, Nethelp consulting, sthaug@nethelp.no