Mailing List Archive

ISSU offlined mpc - why?
Hello,

    I did an issu upgrade on an MX240 with dual routing engines, going
from 15.1R6 to 17.4R2, and I had reviewed the fine manuals particularly
concerning advance steps to ensure nonstop routing and so forth. But
during the upgrade (near the very end), the router did offline my MPC 3D
16x 10GE, resulting in loss of routing for 5 minutes. According to
juniper in their docs that state that:

"Unified ISSU is supported with Junos OS Release 17.4R1 for MX Series
routers with MPC-3D-16XGE-SFPP"

    my expectations were that the card would stay online and there
would be little to no operational impact, but this clearly wasn't the
case. I am just wondering if anyone can tell me why this might have been
the case and in the future what steps I would need to take to ensure
this both otherwise stays on the air during a future software update?


Thank you.

Mike-

_______________________________________________
juniper-nsp mailing list juniper-nsp@puck.nether.net
https://puck.nether.net/mailman/listinfo/juniper-nsp
Re: ISSU offlined mpc - why? [ In reply to ]
On Wed, Sep 01, 2021 at 09:44:08AM -0700, Mike via juniper-nsp wrote:
> "Unified ISSU is supported with Junos OS Release 17.4R1 for MX Series
> routers with MPC-3D-16XGE-SFPP"
>
> ??? my expectations were that the card would stay online and there
> would be little to no operational impact, but this clearly wasn't the
> case. I am just wondering if anyone can tell me why this might have been
> the case and in the future what steps I would need to take to ensure
> this both otherwise stays on the air during a future software update?

Eventually during the ISSU process, the line card software needs to be
upgraded. During that part, each line card goes offline one at a
time. If you have multiple line cards, design your network such that
redundant network paths are connected across different cards to
prevent a total outage when each line card is upgraded one-by-one.
_______________________________________________
juniper-nsp mailing list juniper-nsp@puck.nether.net
https://puck.nether.net/mailman/listinfo/juniper-nsp
Re: ISSU offlined mpc - why? [ In reply to ]
On Wed, 1 Sept 2021 at 20:35, Chuck Anderson via juniper-nsp
<juniper-nsp@puck.nether.net> wrote:

> Eventually during the ISSU process, the line card software needs to be
> upgraded. During that part, each line card goes offline one at a
> time. If you have multiple line cards, design your network such that
> redundant network paths are connected across different cards to
> prevent a total outage when each line card is upgraded one-by-one.

Disclaimer: I do not use ISSU nor do I plan to use it, it is complex
and I've been hurt before with non-obvious failure modes after the box
is left in an unknown state, this is vendor agnostic fear I have and I
am currently not seeking help to overcome it.

https://www.juniper.net/documentation/us/en/software/junos/high-availability/topics/topic-map/unified-issu-enhanced-mode.html
Enhanced mode is an in-service software upgrade (ISSU) option
available on MPC8E, MPC9E, and MPC11E line cards that eliminates
packet loss during the unified ISSU process

The changes in hardware that affect MPC behaviour are mostly the
microcode that the lookup engines run, in trio these are collections
of identical cores called PPE (packet processing engine). In theory we
could put one PPE at a time out of service, and restart it with new
ucode, until we're all done, having N-1/N of nominal PPS capacity
during upgrade (I think EA has 96 PPE, so 98.95% during upgrade),
And in fact this is what 'hyper mode' does, Trio has ucode compatible
history of over a decade, without running hyper mode the PPEs are
running the old +decade old collection of code, with hyper mode the
PPEs are running new rewrite of the ucode. Newer hardware is hyper
mode only, older hardware it is operator choice which generation of
ucode to run.

--
++ytti
_______________________________________________
juniper-nsp mailing list juniper-nsp@puck.nether.net
https://puck.nether.net/mailman/listinfo/juniper-nsp
Re: ISSU offlined mpc - why? [ In reply to ]
> Hello,
>
> ???? I did an issu upgrade on an MX240 with dual routing engines, going?
> from 15.1R6 to 17.4R2, and I had reviewed the fine manuals particularly?
> concerning advance steps to ensure nonstop routing and so forth. But?
> during the upgrade (near the very end), the router did offline my MPC 3D?
> 16x 10GE, resulting in loss of routing for 5 minutes. According to?
> juniper in their docs that state that:
>
> "Unified ISSU is supported with Junos OS Release 17.4R1 for MX Series?
> routers with MPC-3D-16XGE-SFPP"

I think that might be a misunderstanding. Shouldnt it read "is supported starting with Junos Release 17.4R1"?

(That would make sense since some of the cards listed on that paragraph, e.g.?MPC7EQ-MRATE-RB, were only supported since JunOS 16+)

Because then it would be clear - since you are coming from 15.1R6 this release wouldnt support it and jumping to 17.4R2 would
cause the MPC reboot.

-Jonas
_______________________________________________
juniper-nsp mailing list juniper-nsp@puck.nether.net
https://puck.nether.net/mailman/listinfo/juniper-nsp
Re: ISSU offlined mpc - why? [ In reply to ]
On 9/1/21 18:44, Mike via juniper-nsp wrote:

> Hello,
>
>     I did an issu upgrade on an MX240 with dual routing engines, going
> from 15.1R6 to 17.4R2, and I had reviewed the fine manuals
> particularly concerning advance steps to ensure nonstop routing and so
> forth. But during the upgrade (near the very end), the router did
> offline my MPC 3D 16x 10GE, resulting in loss of routing for 5
> minutes. According to juniper in their docs that state that:
>
> "Unified ISSU is supported with Junos OS Release 17.4R1 for MX Series
> routers with MPC-3D-16XGE-SFPP"
>
>     my expectations were that the card would stay online and there
> would be little to no operational impact, but this clearly wasn't the
> case. I am just wondering if anyone can tell me why this might have
> been the case and in the future what steps I would need to take to
> ensure this both otherwise stays on the air during a future software
> update?

I haven't thought about ISSU since 1 year after it first made an
appearance across vendor platforms.

Are people still actually considering it, or even trying to use it?

It didn't seem - to me anyway - that it had any chance of delivering on
its promise more than a decade ago. I can't fathom that this situation
has changed in 2021, but I could be wrong.

Mark.
_______________________________________________
juniper-nsp mailing list juniper-nsp@puck.nether.net
https://puck.nether.net/mailman/listinfo/juniper-nsp
Re: ISSU offlined mpc - why? [ In reply to ]
On 9/2/21 14:56, Jerry Jones wrote:

> I view ISSU as a script to simplify upgrades - not provide for hitless.
>
> Terrible namig on Junipers part. You can perform manually with less
> downtime if any.

This is what we do, because you would be scheduling an outage window anyway.

Works, and is reliable.

Mark.

_______________________________________________
juniper-nsp mailing list juniper-nsp@puck.nether.net
https://puck.nether.net/mailman/listinfo/juniper-nsp