Mailing List Archive

Along with this - I would suggest looking at Port Checker (
https://apps.juniper.net/home/port-checker/index.html ) to make sure
your port combinations are valid.

Kevin

On Thu, Jun 8, 2023 at 9:16?AM Mark Tinka via juniper-nsp
<juniper-nsp@puck.nether.net> wrote:
>
> So, we decided to give the MX304 another sniff, and needed to find out
> why Juniper charge a license for 16x 100Gbps ports per line card, and
> yet the data sheet suggests the box can handle 48x 100Gbps ports
> chassis-wide.
>
> Well, turns out that if you deploy it with redundant RE's, you get 32x
> 100Gbps ports (2x line cards of 16x ports each).
>
> However, to take another 16 ports that gets you to 48x 100Gbps ports,
> you need to sacrifice one RE to use its slot :-).
>
> Mark.
> _______________________________________________
> juniper-nsp mailing list juniper-nsp@puck.nether.net
> https://puck.nether.net/mailman/listinfo/juniper-nsp
_______________________________________________
juniper-nsp mailing list juniper-nsp@puck.nether.net
https://puck.nether.net/mailman/listinfo/juniper-nsp

On 6/8/23 17:18, Kevin Shymkiw wrote:
> Along with this - I would suggest looking at Port Checker (
> https://apps.juniper.net/home/port-checker/index.html ) to make sure
> your port combinations are valid.

We've had ample experience with Juniper's MPC7E, MX204, PTX1000 and
PTX10001 to know how they structure this from a philosophical
standpoint. So not a major drama there.

It's just interesting to me that the data sheet does not mention needing
to sacrifice an RE to get to the chassis' advertised full port
compliment. Unless the data sheet was updated and I missed it.

Mark.
_______________________________________________
juniper-nsp mailing list juniper-nsp@puck.nether.net
https://puck.nether.net/mailman/listinfo/juniper-nsp

Hello good afternoon.

Please have a look at the following documentation:


https://community.juniper.net/blogs/reema-ray/2023/03/28/mx304-deepdive


It will have everything you need to do with it, including the pictures.

Our first boxes are arriving this next month in Brazil.

By the specs of the new chipset (TRIO6) it's a very good box. A lot of enhancements.

And it supports 400G already ( ZR and ZR+ need to check ) ( 16 x 100 or 4 x 400 ) per LMIC.

Best regards

Giuliano



-----Original Message-----
From: juniper-nsp <juniper-nsp-bounces@puck.nether.net> On Behalf Of Mark Tinka via juniper-nsp
Sent: Thursday, June 8, 2023 12:25 PM
To: Kevin Shymkiw <kshymkiw@gmail.com>
Cc: juniper-nsp <juniper-nsp@puck.nether.net>
Subject: Re: [j-nsp] MX304 Port Layout



On 6/8/23 17:18, Kevin Shymkiw wrote:
> Along with this - I would suggest looking at Port Checker (
> https://apps/
> .juniper.net%2Fhome%2Fport-checker%2Findex.html&data=05%7C01%7Cgiuliano%40wztech.com.br%7C28f16ddcdd4440a77d9d08db68348d0b%7C584787b077bd4312bf8815412b8ae504%7C1%7C0%7C638218347183708615%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=KbDjdprQEMTa1yXQSPrF%2BytprEB0JAtWWRJtkX2QYAs%3D&reserved=0 ) to make sure your port combinations are valid.

We've had ample experience with Juniper's MPC7E, MX204, PTX1000 and
PTX10001 to know how they structure this from a philosophical standpoint. So not a major drama there.

It's just interesting to me that the data sheet does not mention needing to sacrifice an RE to get to the chassis' advertised full port compliment. Unless the data sheet was updated and I missed it.

Mark.
_______________________________________________
juniper-nsp mailing list juniper-nsp@puck.nether.net
https://puck.nether.net/mailman/listinfo/juniper-nsp

WZTECH is registered trademark of WZTECH NETWORKS.
Copyright © 2023 WZTECH NETWORKS. All Rights Reserved.

IMPORTANTE:
As informações deste e-mail e o conteúdo dos eventuais documentos anexos são confidenciais e para conhecimento exclusivo do destinatário. Se o leitor desta mensagem não for o seu destinatário, fica desde já notificado de que não poderá divulgar, distribuir ou, sob qualquer forma, dar conhecimento a terceiros das informações e do conteúdo dos documentos anexos. Neste caso, favor comunicar imediatamente o remetente, respondendo este e-mail ou telefonando ao mesmo, e em seguida apague-o.

CONFIDENTIALITY NOTICE:
The information transmitted in this email message and any attachments are solely for the intended recipient and may contain confidential or privileged information. If you are not the intended recipient, any review, transmission, dissemination or other use of this information is prohibited. If you have received this communication in error, please notify the sender immediately and delete the material from any computer, including any copies.
_______________________________________________
juniper-nsp mailing list juniper-nsp@puck.nether.net
https://puck.nether.net/mailman/listinfo/juniper-nsp

On 6/8/23 17:35, Giuliano C. Medalha wrote:

> Hello good afternoon.
>
> Please have a look at the following documentation:
>
>
> https://community.juniper.net/blogs/reema-ray/2023/03/28/mx304-deepdive

Thanks, this is most useful!


> It will have everything you need to do with it, including the pictures.
>
> Our first boxes are arriving this next month in Brazil.
>
> By the specs of the new chipset (TRIO6) it's a very good box. A lot of enhancements.

Trio capacity aside, based on our experience with the MPC7E, MX204 and
MX10003, we expect it to be fairly straight forward.

What is holding us back is the cost. The license for each 16-port line
card is eye-watering. While I don't see anything comparable in ASR99xx
Cisco-land (in terms of form factor and 100Gbps port density), those
prices are certainly going to force Juniper customers to look at other
options. They would do well to get that under control.


> And it supports 400G already ( ZR and ZR+ need to check ) ( 16 x 100 or 4 x 400 ) per LMIC.

The LMIC won't care whether it's ZR, ZR+, FR4 or DR4. It will be
compatible with whatever pluggable is used, as long as it can do 400Gbps.

Unless, of course, you mean whether Juniper provide an interface into
the optic for use-cases where you are plugging into a ROADM... that, I
don't know.

Are you intending to use this router for long-distance applications?

Mark.
_______________________________________________
juniper-nsp mailing list juniper-nsp@puck.nether.net
https://puck.nether.net/mailman/listinfo/juniper-nsp

> Hello good afternoon.
>
> Please have a look at the following documentation:
>
>
> https://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fcommunity.juniper.net%2Fblogs%2Freema-ray%2F2023%2F03%2F28%2Fmx304-deepdive&data=05%7C01%7Cgiuliano%40wztech.com.br%7Cc7b64b057b6b488bff0d08db683a0b28%7C584787b077bd4312bf8815412b8ae504%7C1%7C0%7C638218370748741098%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=fhF2D3TSWDQljPw0jepN5ZsB%2FdOSUq4zIx%2F9w1TkkhE%3D&reserved=0<https://community.juniper.net/blogs/reema-ray/2023/03/28/mx304-deepdive>

Thanks, this is most useful!


> It will have everything you need to do with it, including the pictures.
>
> Our first boxes are arriving this next month in Brazil.
>
> By the specs of the new chipset (TRIO6) it's a very good box. A lot of enhancements.

Trio capacity aside, based on our experience with the MPC7E, MX204 and
MX10003, we expect it to be fairly straight forward.

What is holding us back is the cost. The license for each 16-port line
card is eye-watering. While I don't see anything comparable in ASR99xx
Cisco-land (in terms of form factor and 100Gbps port density), those
prices are certainly going to force Juniper customers to look at other
options. They would do well to get that under control.


but you have the flex model. With license for capacity and features. Advanced and Premium.

fib is better now - 12M

sampling rate for ipfix is better too.

but you have other parameters for mpls and bng too




> And it supports 400G already ( ZR and ZR+ need to check ) ( 16 x 100 or 4 x 400 ) per LMIC.

The LMIC won't care whether it's ZR, ZR+, FR4 or DR4. It will be compatible with whatever pluggable is used, as long as it can do 400Gbps.



But you need the correct drivers on Junos

And the chassis must support temperature and power budgets

Juniper now has good prices ( common optics ) for 400G ( JCO part numbers )



Unless, of course, you mean whether Juniper provide an interface into
the optic for use-cases where you are plugging into a ROADM... that, I
don't know.

Are you intending to use this router for long-distance applications?



Low 40 km or maximum 80 km direct with ZR high power ( end of the year )

Thanks

Mark.

WZTECH is registered trademark of WZTECH NETWORKS.
Copyright (c) 2023 WZTECH NETWORKS. All Rights Reserved.

IMPORTANTE:
As informa??es deste e-mail e o conte?do dos eventuais documentos anexos s?o confidenciais e para conhecimento exclusivo do destinat?rio. Se o leitor desta mensagem n?o for o seu destinat?rio, fica desde j? notificado de que n?o poder? divulgar, distribuir ou, sob qualquer forma, dar conhecimento a terceiros das informa??es e do conte?do dos documentos anexos. Neste caso, favor comunicar imediatamente o remetente, respondendo este e-mail ou telefonando ao mesmo, e em seguida apague-o.

CONFIDENTIALITY NOTICE:
The information transmitted in this email message and any attachments are solely for the intended recipient and may contain confidential or privileged information. If you are not the intended recipient, any review, transmission, dissemination or other use of this information is prohibited. If you have received this communication in error, please notify the sender immediately and delete the material from any computer, including any copies.
_______________________________________________
juniper-nsp mailing list juniper-nsp@puck.nether.net
https://puck.nether.net/mailman/listinfo/juniper-nsp

On 6/8/23 18:39, Giuliano C. Medalha wrote:

> but you have the flex model. With license for capacity and features.
>  Advanced and Premium.

Which isn't a new thing with vendors. The prices are just terrible, even
with discounts.


> fib is better now - 12M
>
> sampling rate for ipfix is better too.
>
> but you have other parameters for mpls and bng too

Yes, like I said, increased Trio capacity aside, it's straight forward.
Just not sure the price is justified. I expect a premium for custom
silicon over Broadcom, but that seems excessive.


> But you need the correct drivers on Junos

Yes, that is assumed, of course, especially if you want to talk to a
ROADM. There is varying support amongst vendors, but as with everything,
they will converge in time.


> Juniper now has good prices ( common optics ) for 400G ( JCO part
> numbers )

Mixing $vendor_name and "good optics prices" has always ended in tears.


> Low 40 km or maximum 80 km direct with ZR high power ( end of the year )

Okay.

Our use-case is 100Gbps customer edge, so within the data centre.

We operate an optical network for anything longer than 80km.

Mark.
_______________________________________________
juniper-nsp mailing list juniper-nsp@puck.nether.net
https://puck.nether.net/mailman/listinfo/juniper-nsp

On 2023-06-08 17:18, Kevin Shymkiw via juniper-nsp wrote:

> Along with this - I would suggest looking at Port Checker (
> https://apps.juniper.net/home/port-checker/index.html ) to make sure
> your port combinations are valid.

The port checker claims an interresting "feature": if you have
anything in port 3, then *all* the other ports in that port group
must also be occupied. So if you use all those four ports for
e.g. 100GE, everything is fine, but if you then want to stop using
either of ports 0, 1 or 2, the configuration becomes invalid...

(And similarly for ports 5, 8 and 14 in their respective groups.)

I hope that's a bug in the port checker, not actual behaviour by
the MX304...


--
Thomas Bellman, National Supercomputer Centre, Linköping Univ., Sweden
"We don't understand the software, and sometimes we don't understand
the hardware, but we can *see* the blinking lights!"

No, that is not quite right. We have 2 chassis of MX304 in Production today and 1 spare all with Redundant REs You do not need all the ports filled in a port group. I know since we mixed in some 40G and 40G is ONLY supported on the bottom row of ports so we have a mix and had to break stuff out leaving empty ports because of that limitation, and it is running just fine. But you do have to be careful which type of optics get plugged into which ports. IE Port 0/2 vs Port 1/3 in a grouping if you are not using 100G optics.

The big issue we ran into is if you have redundant REs then there is a super bad bug that after 6 hours (1 of our 3 would lock up after reboot quickly and the other 2 would take a very long time) to 8 days will lock the entire chassis up solid where we had to pull the REs physical out to reboot them. It is fixed now, but they had to manually poke new firmware into the ASICs on each RE when they were in a half-powered state, Was a very complex procedure with tech support and the MX304 engineering team. It took about 3 hours to do all 3 MX304s one RE at a time. We have not seen an update with this built-in yet. (We just did this back at the end of April)


-----Original Message-----
From: juniper-nsp <juniper-nsp-bounces@puck.nether.net> On Behalf Of Thomas Bellman via juniper-nsp
Sent: Thursday, June 8, 2023 2:09 PM
To: juniper-nsp <juniper-nsp@puck.nether.net>
Subject: Re: [EXT] [j-nsp] MX304 Port Layout

On 2023-06-08 17:18, Kevin Shymkiw via juniper-nsp wrote:

> Along with this - I would suggest looking at Port Checker (
> https://apps.juniper.net/home/port-checker/index.html ) to make sure
> your port combinations are valid.

The port checker claims an interresting "feature": if you have anything in port 3, then *all* the other ports in that port group must also be occupied. So if you use all those four ports for e.g. 100GE, everything is fine, but if you then want to stop using either of ports 0, 1 or 2, the configuration becomes invalid...

(And similarly for ports 5, 8 and 14 in their respective groups.)

I hope that's a bug in the port checker, not actual behaviour by the MX304...


--
Thomas Bellman, National Supercomputer Centre, Linköping Univ., Sweden "We don't understand the software, and sometimes we don't understand the hardware, but we can *see* the blinking lights!"

_______________________________________________
juniper-nsp mailing list juniper-nsp@puck.nether.net
https://puck.nether.net/mailman/listinfo/juniper-nsp

On 6/9/23 00:03, Litterick, Jeff (BIT) via juniper-nsp wrote:

> The big issue we ran into is if you have redundant REs then there is a super bad bug that after 6 hours (1 of our 3 would lock up after reboot quickly and the other 2 would take a very long time) to 8 days will lock the entire chassis up solid where we had to pull the REs physical out to reboot them. It is fixed now, but they had to manually poke new firmware into the ASICs on each RE when they were in a half-powered state, Was a very complex procedure with tech support and the MX304 engineering team. It took about 3 hours to do all 3 MX304s one RE at a time. We have not seen an update with this built-in yet. (We just did this back at the end of April)

Oh dear, that's pretty nasty. So did they say new units shipping today
would come with the RE's already fixed?

We've been suffering a somewhat similar issue on the PTX1000, where a
bug was introduced via regression in Junos 21.4, 22.1 and 22.2 that
causes CPU queues to get filled up by unknown MAC address frames, and
are not cleared. It takes 64 days for this packet accumulation to grow
to a point where the queues get exhausted, causing a host loopback wedge.

You would see an error like this in the logs:

<date> <time> <hostname> alarmd[27630]: Alarm set: FPC id=150995048,
color=RED, class=CHASSIS, reason=FPC 0 Major Errors
<date> <time> <hostname> fpc0 Performing action cmalarm for error
/fpc/0/pfe/0/cm/0/Host_Loopback/0/HOST_LOOPBACK_MAKE_CMERROR_ID[1]
(0x20002) in module: Host Loopback with scope: pfe category: functional
level: major
<date> <time> <hostname> fpc0 Cmerror Op Set: Host Loopback: HOST
LOOPBACK WEDGE DETECTED IN PATH ID 1  (URI:
/fpc/0/pfe/0/cm/0/Host_Loopback/0/HOST_LOOPBACK_MAKE_CMERROR_ID[1])
Apr 1 03:52:28  PTX1000 fpc0 CMError:
/fpc/0/pfe/0/cm/0/Host_Loopback/0/HOST_LOOPBACK_MAKE_CMERROR_ID[3]
(0x20004), in module: Host Loopback with scope: pfe category: functional
level: major

This causes the router to drop all control plane traffic, which,
basically, makes it unusable. One has to reboot the box to get it back
up and running, until it happens again 64 days later.

The issue is resolved in Junos 21.4R3-S4, 22.4R2, 23.2R1 and 23.3R1.

However, these releases are not shipping yet, so Juniper gave us a
workaround SLAX script that automatically runs and clears the CPU queues
before the 64 days are up.

We are currently running Junos 22.1R3.9 on this platform, and will move
to 22.4R2 in a few weeks to permanently fix this.

Junos 20.2, 20.3 and 20.4 are not affected, nor is anything after 23.2R1.

I understand it may also affect the QFX and MX, but I don't have details
on that.

Mark.

_______________________________________________
juniper-nsp mailing list juniper-nsp@puck.nether.net
https://puck.nether.net/mailman/listinfo/juniper-nsp

Hi Jeff,

Thank you very mush for sharing this information. Do you know in what
publicly available release it's going to be fixed? Knowing PR number
would be the best but I guess it may be internal-only.

Kind regards,
Andrey

Litterick, Jeff (BIT) via juniper-nsp ?????(?) 2023-06-08 18:03:
> No, that is not quite right. We have 2 chassis of MX304 in Production
> today and 1 spare all with Redundant REs You do not need all the
> ports filled in a port group. I know since we mixed in some 40G and
> 40G is ONLY supported on the bottom row of ports so we have a mix and
> had to break stuff out leaving empty ports because of that limitation,
> and it is running just fine. But you do have to be careful which
> type of optics get plugged into which ports. IE Port 0/2 vs Port 1/3
> in a grouping if you are not using 100G optics.
>
> The big issue we ran into is if you have redundant REs then there is a
> super bad bug that after 6 hours (1 of our 3 would lock up after
> reboot quickly and the other 2 would take a very long time) to 8 days
> will lock the entire chassis up solid where we had to pull the REs
> physical out to reboot them. It is fixed now, but they had to
> manually poke new firmware into the ASICs on each RE when they were in
> a half-powered state, Was a very complex procedure with tech support
> and the MX304 engineering team. It took about 3 hours to do all 3
> MX304s one RE at a time. We have not seen an update with this
> built-in yet. (We just did this back at the end of April)
>
>
_______________________________________________
juniper-nsp mailing list juniper-nsp@puck.nether.net
https://puck.nether.net/mailman/listinfo/juniper-nsp

Hi Mark,

Not sure why it's eye-watering. The price of fully populated MX304 is
basically the same as it's predecessor MX10003 but it provides 3.2T BW
capacity vs 2.4T. If you compare with MX204, then MX304 is about 20%
expensive for the same total BW, but MX204 doesn't have redundant RE and
if you use it in redundant chassis configuration you will have to spend
some BW on "fabric" links, effectively leveling the price if calculated
for the same BW. I'm just comparing numbers, not considering any real
topology, which is another can of worms. Most probably it's not worth to
try to scale MX204s to more than a pair of devices, at least I wouldn't
do it and consider it ;)
I'd rather call eye-watering prices for MPC7 and MPC10 to upgrade
existing MX480 routers if you still to use their low-speed ports. Two
MPC10s with SCB3s upgrade cost more than MX304, but gives 30% less BW
capacity. For MPC7 this ratio is even worse.
This brings a question, does anybody have an experience with HQoS on
MX304? I mean just per-subinterface queueing on an interface to a
switch, not BNG subscribers CoS which is probably another big topic. At
least I'm not dare yet to try MX304 in BNG role, maybe later ;)

Kind regards,
Andrey

Mark Tinka via juniper-nsp ?????(?) 2023-06-08 12:04:

> Trio capacity aside, based on our experience with the MPC7E, MX204 and
> MX10003, we expect it to be fairly straight forward.
>
> What is holding us back is the cost. The license for each 16-port line
> card is eye-watering. While I don't see anything comparable in ASR99xx
> Cisco-land (in terms of form factor and 100Gbps port density), those
> prices are certainly going to force Juniper customers to look at other
> options. They would do well to get that under control.
>
>
_______________________________________________
juniper-nsp mailing list juniper-nsp@puck.nether.net
https://puck.nether.net/mailman/listinfo/juniper-nsp

On Fri, 9 Jun 2023 at 16:58, Andrey Kostin via juniper-nsp
<juniper-nsp@puck.nether.net> wrote:

> Not sure why it's eye-watering. The price of fully populated MX304 is
> basically the same as it's predecessor MX10003 but it provides 3.2T BW
> capacity vs 2.4T. If you compare with MX204, then MX304 is about 20%
> expensive for the same total BW, but MX204 doesn't have redundant RE and
> if you use it in redundant chassis configuration you will have to spend
> some BW on "fabric" links, effectively leveling the price if calculated
> for the same BW. I'm just comparing numbers, not considering any real

That's not it, RE doesn't attach to fabric serdes.

You are right that the MX304 is the successor of MX10003 not MX201.

MX80, M104 and MX201 are unique in that they are true pizzabox Trios.
They have exactly 1 trio, and both WAN and FAB side connect to WAN
ports (not sure if MX201 just leaves them unconnected) Therefore say
40G Trio in linecard mode is 80G Trio in pizza mode (albeit PPS stays
the same) as you're not wasting capacity to non-revenue fabric ports.
This single Trio design makes the box very cost effective, as not only
do you just have one Trio and double the capacity per Trio, but you
also don't have any fabric chip and fabric serdes.

MX304 however has Trio in the linecard, so it really is very much a
normal chassis box. And having multiple Trios it needs fabric.

I do think Juniper and the rest of the vendors keep struggling to
identify 'few to many' markets, and are only good at identifying 'many
to few' markets. MX304 and ever denser 512x112G serdes chips represent
this.

I expect many people in this list have no need for more performance
than single Trio YT in any pop at all, yet they need ports. And they
are not adequately addressed by vendors. But they do need the deep
features of NPU.

I keep hoping that someone is so disruptive that they take the
nvidia/gpu approach to npu. That is, you can buy Trio PCI from newegg
for 2 grand, and can program it as you wish. I think this market
remains unidentified and even adjusting to cannibalization would
increase market size.
I can't understand why JNPR is not trying this, they've lost for 20
years to inflation in valuation, what do they have to lose?

--
++ytti
_______________________________________________
juniper-nsp mailing list juniper-nsp@puck.nether.net
https://puck.nether.net/mailman/listinfo/juniper-nsp

On 6/9/23 15:57, Andrey Kostin wrote:

> Hi Mark,
>
> Not sure why it's eye-watering. The price of fully populated MX304 is
> basically the same as it's predecessor MX10003 but it provides 3.2T BW
> capacity vs 2.4T.

That's true, but the premium being paid for 400Gbps capability that some
houses may not yet need is probably what is pushing that price up in
comparison to the MX10003, which does not support 400Gbps.

But to be fair, it will come down to the discounts you can negotiate
with Juniper. I'm perhaps more concerned because we got good pricing on
the MX10003, even when we did a like-for-somewhat-like comparison with
the MX304.

As much as we have struggled with Cisco in the past, this is forcing me
to see what is available in their ASR99xx boxes. But off-the-bat, form
factor and port density is poor on the Cisco side, compared to the MX304
and MX10003.


> If you compare with MX204, then MX304 is about 20% expensive for the
> same total BW, but MX204 doesn't have redundant RE and if you use it
> in redundant chassis configuration you will have to spend some BW on
> "fabric" links, effectively leveling the price if calculated for the
> same BW. I'm just comparing numbers, not considering any real
> topology, which is another can of worms. Most probably it's not worth
> to try to scale MX204s to more than a pair of devices, at least I
> wouldn't do it and consider it ;)

The use-case for MX204 and MX304 is very very different. As you say,
MX304 is a better alternative for the MX10003 (which I am getting
conflicting information about re: sale availability from Juniper).

We use the MX204 extensively, but only for peering and routing for
value-added services too small to plug into a larger MX.


> I'd rather call eye-watering prices for MPC7 and MPC10 to upgrade
> existing MX480 routers if you still to use their low-speed ports. Two
> MPC10s with SCB3s upgrade cost more than MX304, but gives 30% less BW
> capacity. For MPC7 this ratio is even worse.

Agreed - the MPC7 and MPC10's only make sense for large capacity
aggregation or backbone links, not as an access port for 100Gbps
customers. The MX10003 and MX304 are better boxes for 100Gbps access for
customers.

Conversely, trying to use the MX304 or MX10003 as a core box is too
costly, since you are paying the premium for edge features in Trio, when
all you need is basic Ethernet, IS-IS and MPLS.

So the MPC7/MPC10 vs. MX304/10003 use-cases are clearly defined, if
money is an object.


> This brings a question, does anybody have an experience with HQoS on
> MX304? I mean just per-subinterface queueing on an interface to a
> switch, not BNG subscribers CoS which is probably another big topic.
> At least I'm not dare yet to try MX304 in BNG role, maybe later ;)

In this world where the kind of traffic you will be pushing through an
MX304 most likely being majority off-net content, do you really need
H-QoS :-)?

Mark.
_______________________________________________
juniper-nsp mailing list juniper-nsp@puck.nether.net
https://puck.nether.net/mailman/listinfo/juniper-nsp

On 6/9/23 16:12, Saku Ytti wrote:

> I expect many people in this list have no need for more performance
> than single Trio YT in any pop at all, yet they need ports. And they
> are not adequately addressed by vendors. But they do need the deep
> features of NPU.

This.

There is sufficient performance in Trio today (even a single Trio chip
on the board) that people are willing to take an oversubscribed box or
line card because in real life, they will run out of ports long before
they run out of aggregate forwarding capacity.

The MX204, even though it's a pizza box, is a good example of how it
could do with 8x 100Gbps ports, even though Trio on it will only forward
400Gbps. Most use-cases will require another MX204 chassis, just for
ports, before the existing one has hit anywhere close to capacity.

Really, folk are just chasing the Trio capability, otherwise they'd have
long solved their port-count problems by choosing any Broadcom-based box
on the market. Juniper know this, and they are using it against their
customers, knowingly or otherwise. Cisco was good at this back in the
day, over-subscribing line cards on their switches and routers. Juniper
have always been a little more purist, but the market can't handle it
because the rate of traffic growth is being out-paced by what a single
Trio chip can do for a couple of ports, in the edge.


> I keep hoping that someone is so disruptive that they take the
> nvidia/gpu approach to npu. That is, you can buy Trio PCI from newegg
> for 2 grand, and can program it as you wish. I think this market
> remains unidentified and even adjusting to cannibalization would
> increase market size.
> I can't understand why JNPR is not trying this, they've lost for 20
> years to inflation in valuation, what do they have to lose?

Well, the story is that Cisco are doing this with Meta and Microsoft on
their C8000 platform, and apparently, doing billions of US$ in business
on the back of that.

Mark.
_______________________________________________
juniper-nsp mailing list juniper-nsp@puck.nether.net
https://puck.nether.net/mailman/listinfo/juniper-nsp

On Fri, 9 Jun 2023 at 17:26, Mark Tinka <mark@tinka.africa> wrote:

> Well, the story is that Cisco are doing this with Meta and Microsoft on
> their C8000 platform, and apparently, doing billions of US$ in business
> on the back of that.

I'm not convinced at all that leaba is being sold. I think it's sold
conditionally when customers would otherwise be lost.

I am reminder of this:
https://www.servethehome.com/this-is-a-broadcom-tomahawk-4-64-port-400gbe-switch-chip-lga8371-intel-amd-ampere/

LGA8371 socketed BRCM TH4. Ostensibly this allows a lot more switches
to appear in the market, as the switch maker doesn't need to be
friendly with BRCM. They make the switch, the customer buys the chip
and sockets it. Wouldn't surprise me if FB, AMZN and the likes would
have pressed for something like this, so they could use cheaper
sources to make the rest of the switch, sources which BRCM didn't want
to play ball with.

But NPU from newegg and community writes code that doesn't exist, and
I think it should and there would be volume in it, but no large volume
to any single customer.

--
++ytti
_______________________________________________
juniper-nsp mailing list juniper-nsp@puck.nether.net
https://puck.nether.net/mailman/listinfo/juniper-nsp

On 6/9/23 16:35, Saku Ytti wrote:

> I'm not convinced at all that leaba is being sold. I think it's sold
> conditionally when customers would otherwise be lost.

Probably - it's a "grain of salt" situation when you hear the news.

I don't think Meta and Microsoft have not bought zero of the C8000...
but not to the degree that they would ignore more primary options, I think.


> But NPU from newegg and community writes code that doesn't exist, and
> I think it should and there would be volume in it, but no large volume
> to any single customer.

Not enough foresight from traditional OEM's to see the potential here.

Mark.
_______________________________________________
juniper-nsp mailing list juniper-nsp@puck.nether.net
https://puck.nether.net/mailman/listinfo/juniper-nsp

Saku Ytti ?????(?) 2023-06-09 10:12:
> On Fri, 9 Jun 2023 at 16:58, Andrey Kostin via juniper-nsp
> <juniper-nsp@puck.nether.net> wrote:
>
>> Not sure why it's eye-watering. The price of fully populated MX304 is
>> basically the same as it's predecessor MX10003 but it provides 3.2T BW
>> capacity vs 2.4T. If you compare with MX204, then MX304 is about 20%
>> expensive for the same total BW, but MX204 doesn't have redundant RE
>> and
>> if you use it in redundant chassis configuration you will have to
>> spend
>> some BW on "fabric" links, effectively leveling the price if
>> calculated
>> for the same BW. I'm just comparing numbers, not considering any real
>
> That's not it, RE doesn't attach to fabric serdes.

Sorry, I mixed two different points. I wanted to say that redundant RE
adds more cost to MX304, unrelated to forwarding BW. But if you want to
have MX204s in redundant configuration, some ports have to be sacrificed
for connectivity between them. We have two MX204s running in pair with
2x100G taken for links between them and remaining BW is 6x100G for
actual forwarding in/out. In this case it's kind of at the same level
for price/100G value.

>
> I expect many people in this list have no need for more performance
> than single Trio YT in any pop at all, yet they need ports. And they
> are not adequately addressed by vendors. But they do need the deep
> features of NPU.

I agree, and that's why I asked about HQoS experience, just to add more
inexpensive low-speed switch ports via trunk but still be able to treat
them more like separate ports from a router perspective.

> I keep hoping that someone is so disruptive that they take the
> nvidia/gpu approach to npu. That is, you can buy Trio PCI from newegg
> for 2 grand, and can program it as you wish. I think this market
> remains unidentified and even adjusting to cannibalization would
> increase market size.
> I can't understand why JNPR is not trying this, they've lost for 20
> years to inflation in valuation, what do they have to lose?

I'm not in this market, have no qualification and resources for
development. The demand in such devices should be really massive to
justify a process like this.

Kind regards,
Andrey
_______________________________________________
juniper-nsp mailing list juniper-nsp@puck.nether.net
https://puck.nether.net/mailman/listinfo/juniper-nsp

Mark Tinka ?????(?) 2023-06-09 10:26:
> On 6/9/23 16:12, Saku Ytti wrote:
>
>> I expect many people in this list have no need for more performance
>> than single Trio YT in any pop at all, yet they need ports. And they
>> are not adequately addressed by vendors. But they do need the deep
>> features of NPU.
>
> This.
>
> There is sufficient performance in Trio today (even a single Trio chip
> on the board) that people are willing to take an oversubscribed box or
> line card because in real life, they will run out of ports long before
> they run out of aggregate forwarding capacity.
>
> The MX204, even though it's a pizza box, is a good example of how it
> could do with 8x 100Gbps ports, even though Trio on it will only
> forward 400Gbps. Most use-cases will require another MX204 chassis,
> just for ports, before the existing one has hit anywhere close to
> capacity.

Agree, there is a gap between 204 and 304, but don't forget that they
belong to different generations. 304 is shiny new with a next level
performance that's replacing MX10k3. The previous generation was
announced to retire, but life of MX204 was extended because Juniper
realized that they don't have anything atm to replace it and probably
will lose revenue. Maybe this gap was caused by covid that slowed down
the new platform. And possibly we may see a single NPU model based on
the new gen chip, because chips for 204 are finite. At least it would be
logical to make it, considering success of MX204.
>
> Really, folk are just chasing the Trio capability, otherwise they'd
> have long solved their port-count problems by choosing any
> Broadcom-based box on the market. Juniper know this, and they are
> using it against their customers, knowingly or otherwise. Cisco was
> good at this back in the day, over-subscribing line cards on their
> switches and routers. Juniper have always been a little more purist,
> but the market can't handle it because the rate of traffic growth is
> being out-paced by what a single Trio chip can do for a couple of
> ports, in the edge.

I think that it's not rational to make another chipset with lower
bandwidth, easier to limit an existing more powerful chip. Then it leads
to MX5/MX10/MX40/MX80 hardware and licensing model. It could be a single
Trio6 with up to 1.6T in access ports and 1.6T in uplink ports with low
features. Maybe it will come, who knows, let's watch ;)

Kind regards,
Andrey
_______________________________________________
juniper-nsp mailing list juniper-nsp@puck.nether.net
https://puck.nether.net/mailman/listinfo/juniper-nsp

On Fri, 9 Jun 2023 at 18:46, Andrey Kostin <ankost@podolsk.ru> wrote:

> I'm not in this market, have no qualification and resources for
> development. The demand in such devices should be really massive to
> justify a process like this.

Are you not? You use a lot of open source software, because someone
else did the hard work, and you have something practical.

The same would be the thesis here, You order the PCI NPU from newegg,
and you have an ecosystem of practical software to pull from various
sources. Maybe you'll contribute something back, maybe not.

Very typical network is a border router or two, which needs features
and performance, then switches to connect to compute. People who have
no resources or competence to write software could still be users in
this market.
--
++ytti
_______________________________________________
juniper-nsp mailing list juniper-nsp@puck.nether.net
https://puck.nether.net/mailman/listinfo/juniper-nsp

Saku Ytti ?????(?) 2023-06-09 10:35:

> LGA8371 socketed BRCM TH4. Ostensibly this allows a lot more switches
> to appear in the market, as the switch maker doesn't need to be
> friendly with BRCM. They make the switch, the customer buys the chip
> and sockets it. Wouldn't surprise me if FB, AMZN and the likes would
> have pressed for something like this, so they could use cheaper
> sources to make the rest of the switch, sources which BRCM didn't want
> to play ball with.

Can anything else be inserted in this socket? If not, then what's the
point? For server CPUs there are many models with different clocking and
number of cores, so socket provides a flexibility. If there is only one
chip that fits the socket, then the socket is a redundant part.

Kind regards,
Andrey
_______________________________________________
juniper-nsp mailing list juniper-nsp@puck.nether.net
https://puck.nether.net/mailman/listinfo/juniper-nsp

Not sure, but anything shipped before May most likely would be affected, if not into May a bit. Since we were one of the first if not the first customer to get the fixed applied to the equipment we got at the end of March.

We never knew the real root cause outside that when it happened the primary RE would lock up-sold and not respond to any command or input or allow access to any management port and there were no crash dumps or logs. The Backup RE would NOT Take over and get stuck in a loop trying to take over the mastership, but the backup RE would still respond to management, but even a reboot of it would not allow it to take mastership. The only solution was a full power plug removal or RE removal from the chassis for a power reset. But they were able to find this in the lab at Juniper right after we reported it and they worked on a fix and got it to use about 1.5 weeks later. We got lucky in that one of the 3 boxes would never last more than 6 hours after a reboot before the lockup of the master RE (No matter if it was RE0 or RE1 as master) The other 2 boxes could go a week or more before locking up. So we were a good test to see if the fixed work since in the lab it would take up to 8 days before locking up.

FYI: The one that would lock up inside 6 hours was our spare and had no traffic at all or even a optic plugged into any port and not even test traffic which the other 2 had going. We did not go into production until 2 weeks after the fix was applied to make sure.

This problem would only surface also if you have more then one RE plugged into the system. Even if failover was not configured. It was just the presence of the 2nd RE that would trigger it. I understand that the engineering team is now fully regressive testing all releases with multiple REs now. I guess that was not true before we found the bug.

-----Original Message-----
From: juniper-nsp <juniper-nsp-bounces@puck.nether.net> On Behalf Of Mark Tinka via juniper-nsp
Sent: Thursday, June 8, 2023 10:53 PM
To: juniper-nsp@puck.nether.net
Subject: Re: [EXT] [j-nsp] MX304 Port Layout



On 6/9/23 00:03, Litterick, Jeff (BIT) via juniper-nsp wrote:

> The big issue we ran into is if you have redundant REs then there is a super bad bug that after 6 hours (1 of our 3 would lock up after reboot quickly and the other 2 would take a very long time) to 8 days will lock the entire chassis up solid where we had to pull the REs physical out to reboot them. It is fixed now, but they had to manually poke new firmware into the ASICs on each RE when they were in a half-powered state, Was a very complex procedure with tech support and the MX304 engineering team. It took about 3 hours to do all 3 MX304s one RE at a time. We have not seen an update with this built-in yet. (We just did this back at the end of April)

Oh dear, that's pretty nasty. So did they say new units shipping today would come with the RE's already fixed?

We've been suffering a somewhat similar issue on the PTX1000, where a bug was introduced via regression in Junos 21.4, 22.1 and 22.2 that causes CPU queues to get filled up by unknown MAC address frames, and are not cleared. It takes 64 days for this packet accumulation to grow to a point where the queues get exhausted, causing a host loopback wedge.

You would see an error like this in the logs:

<date> <time> <hostname> alarmd[27630]: Alarm set: FPC id=150995048, color=RED, class=CHASSIS, reason=FPC 0 Major Errors <date> <time> <hostname> fpc0 Performing action cmalarm for error /fpc/0/pfe/0/cm/0/Host_Loopback/0/HOST_LOOPBACK_MAKE_CMERROR_ID[1]
(0x20002) in module: Host Loopback with scope: pfe category: functional
level: major
<date> <time> <hostname> fpc0 Cmerror Op Set: Host Loopback: HOST LOOPBACK WEDGE DETECTED IN PATH ID 1  (URI:
/fpc/0/pfe/0/cm/0/Host_Loopback/0/HOST_LOOPBACK_MAKE_CMERROR_ID[1])
Apr 1 03:52:28  PTX1000 fpc0 CMError:
/fpc/0/pfe/0/cm/0/Host_Loopback/0/HOST_LOOPBACK_MAKE_CMERROR_ID[3]
(0x20004), in module: Host Loopback with scope: pfe category: functional
level: major

This causes the router to drop all control plane traffic, which, basically, makes it unusable. One has to reboot the box to get it back up and running, until it happens again 64 days later.

The issue is resolved in Junos 21.4R3-S4, 22.4R2, 23.2R1 and 23.3R1.

However, these releases are not shipping yet, so Juniper gave us a workaround SLAX script that automatically runs and clears the CPU queues before the 64 days are up.

We are currently running Junos 22.1R3.9 on this platform, and will move to 22.4R2 in a few weeks to permanently fix this.

Junos 20.2, 20.3 and 20.4 are not affected, nor is anything after 23.2R1.

I understand it may also affect the QFX and MX, but I don't have details on that.

Mark.

_______________________________________________
juniper-nsp mailing list juniper-nsp@puck.nether.net https://puck.nether.net/mailman/listinfo/juniper-nsp
_______________________________________________
juniper-nsp mailing list juniper-nsp@puck.nether.net
https://puck.nether.net/mailman/listinfo/juniper-nsp

On Fri, 9 Jun 2023 at 19:15, Andrey Kostin <ankost@podolsk.ru> wrote:

> Can anything else be inserted in this socket? If not, then what's the
> point? For server CPUs there are many models with different clocking and
> number of cores, so socket provides a flexibility. If there is only one
> chip that fits the socket, then the socket is a redundant part.

Not that I know. I think the point may be decouplement. BRCM doesn't
want to do business with just everyone. This allows someone to build
the switch, without providing the chips. Then customers can buy a
switch from this vendor and chip directly from BRCM.
I could imagine some big players like FB and AMZN designing their own
switch, having some random shop actually build it. But Broadcom saying
'no, we don't do business with you'. This way they could actually get
the switch from anywhere, while having a direct chip relationship with
BRCM.



--
++ytti
_______________________________________________
juniper-nsp mailing list juniper-nsp@puck.nether.net
https://puck.nether.net/mailman/listinfo/juniper-nsp

This is why we got the MX304. It was a test to replace our MX10008 Chassis, which we bought a few of because we had to get at a reasonable price into 100G at high density at multiple sites a few years back now. Though we really only need 4 line cards, with 2 being for redundancy. The MX1004 was not available at the time back then (Wish it had been. The MX10008 is a heavy beast indeed and we had to use fork lifts to move them around into the data centers). But after handling the MX304 we will most likely for 400G go to the MX10004 line for the future and just use the MX304 at very small edge sites if needed. Mainly due to full FPC redundancy requirements at many of our locations. And yes we had multiple full FPC failures in the past on the MX10008 line. We went through at first an RMA cycle with multiple line cards which in the end was due to just 1 line cards causing full FPC failure on a different line card in the chassis around every 3 months or so. Only having everything redundant across both FPCs allowed us not to have serious downtime.


-----Original Message-----
From: juniper-nsp <juniper-nsp-bounces@puck.nether.net> On Behalf Of Andrey Kostin via juniper-nsp
Sent: Friday, June 9, 2023 11:09 AM
To: Mark Tinka <mark@tinka.africa>
Cc: Saku Ytti <saku@ytti.fi>; juniper-nsp <juniper-nsp@puck.nether.net>
Subject: Re: [EXT] [j-nsp] MX304 Port Layout

Mark Tinka ?????(?) 2023-06-09 10:26:
> On 6/9/23 16:12, Saku Ytti wrote:
>
>> I expect many people in this list have no need for more performance
>> than single Trio YT in any pop at all, yet they need ports. And they
>> are not adequately addressed by vendors. But they do need the deep
>> features of NPU.
>
> This.
>
> There is sufficient performance in Trio today (even a single Trio chip
> on the board) that people are willing to take an oversubscribed box or
> line card because in real life, they will run out of ports long before
> they run out of aggregate forwarding capacity.
>
> The MX204, even though it's a pizza box, is a good example of how it
> could do with 8x 100Gbps ports, even though Trio on it will only
> forward 400Gbps. Most use-cases will require another MX204 chassis,
> just for ports, before the existing one has hit anywhere close to
> capacity.

Agree, there is a gap between 204 and 304, but don't forget that they belong to different generations. 304 is shiny new with a next level performance that's replacing MX10k3. The previous generation was announced to retire, but life of MX204 was extended because Juniper realized that they don't have anything atm to replace it and probably will lose revenue. Maybe this gap was caused by covid that slowed down the new platform. And possibly we may see a single NPU model based on the new gen chip, because chips for 204 are finite. At least it would be logical to make it, considering success of MX204.
>
> Really, folk are just chasing the Trio capability, otherwise they'd
> have long solved their port-count problems by choosing any
> Broadcom-based box on the market. Juniper know this, and they are
> using it against their customers, knowingly or otherwise. Cisco was
> good at this back in the day, over-subscribing line cards on their
> switches and routers. Juniper have always been a little more purist,
> but the market can't handle it because the rate of traffic growth is
> being out-paced by what a single Trio chip can do for a couple of
> ports, in the edge.

I think that it's not rational to make another chipset with lower bandwidth, easier to limit an existing more powerful chip. Then it leads to MX5/MX10/MX40/MX80 hardware and licensing model. It could be a single
Trio6 with up to 1.6T in access ports and 1.6T in uplink ports with low features. Maybe it will come, who knows, let's watch ;)

Kind regards,
Andrey
_______________________________________________
juniper-nsp mailing list juniper-nsp@puck.nether.net https://puck.nether.net/mailman/listinfo/juniper-nsp
_______________________________________________
juniper-nsp mailing list juniper-nsp@puck.nether.net
https://puck.nether.net/mailman/listinfo/juniper-nsp

Thank you very much, Jeff, for sharing your experience. Will watch
closely Release Notes for upcoming Junos releases. And kudos to Juniper
for finding and fixing it, 1,5 week is very fast reaction!.

Kind regards,
Andrey

Litterick, Jeff (BIT) ?????(?) 2023-06-09 12:41:
> This is why we got the MX304. It was a test to replace our MX10008
> Chassis, which we bought a few of because we had to get at a
> reasonable price into 100G at high density at multiple sites a few
> years back now. Though we really only need 4 line cards, with 2 being
> for redundancy. The MX1004 was not available at the time back then
> (Wish it had been. The MX10008 is a heavy beast indeed and we had to
> use fork lifts to move them around into the data centers). But
> after handling the MX304 we will most likely for 400G go to the
> MX10004 line for the future and just use the MX304 at very small edge
> sites if needed. Mainly due to full FPC redundancy requirements at
> many of our locations. And yes we had multiple full FPC failures in
> the past on the MX10008 line. We went through at first an RMA cycle
> with multiple line cards which in the end was due to just 1 line cards
> causing full FPC failure on a different line card in the chassis
> around every 3 months or so. Only having everything redundant across
> both FPCs allowed us not to have serious downtime.
>
>
_______________________________________________
juniper-nsp mailing list juniper-nsp@puck.nether.net
https://puck.nether.net/mailman/listinfo/juniper-nsp