Mailing List Archive

Paschal,

It is not supported, nor is it recommended for redundancy in a routed setup. Please describe your (desired) topology, that way the community can discuss alternatives.

Thanks,

Ryan Hamel

________________________________
From: NANOG <nanog-bounces+ryan=rkhtech.org@nanog.org> on behalf of Pascal Masha <pascalmasha@gmail.com>
Sent: Monday, August 21, 2023 7:41 AM
To: nanog@nanog.org <nanog@nanog.org>
Subject: MX204 Virtual Chassis Setup

Caution: This is an external email and may be malicious. Please take care when clicking links or opening attachments.

Hello,

Does the MX204 support virtual chassis setup?

Regards,
Paschal Masha

On 8/21/23 16:51, Ryan Hamel wrote:

> Paschal,
>
> It is not supported, nor is it recommended for redundancy in a routed
> setup. Please describe your (desired) topology, that way the community
> can discuss alternatives.

Sounds like the OP wants to build a chassis-based system out of
non-redundant hardware.

Mark.

No, but they do however work just great as an active-active pair of routers
when cross linked and iBGP peered to each other and everything downstream
connected to each one.

Chris

On Mon, Aug 21, 2023 at 9:43?AM Pascal Masha <pascalmasha@gmail.com> wrote:

> Hello,
>
> Does the MX204 support virtual chassis setup?
>
> Regards,
> Paschal Masha
>

Thanks just wanted to know whether it was a supported feature.

On Tue, 22 Aug 2023, 21:00 Chris, <chris@noskillz.com> wrote:

> No, but they do however work just great as an active-active pair of
> routers when cross linked and iBGP peered to each other and everything
> downstream connected to each one.
>
> Chris
>
> On Mon, Aug 21, 2023 at 9:43?AM Pascal Masha <pascalmasha@gmail.com>
> wrote:
>
>> Hello,
>>
>> Does the MX204 support virtual chassis setup?
>>
>> Regards,
>> Paschal Masha
>>
>

On 8/23/23 08:00, Pascal Masha wrote:

> Thanks just wanted to know whether it was a supported feature.

What would have been nice is if Juniper oversubscribed the face plate of
this platform, as most people are more likely to run out of ports than
they would the 400Gbps forwarding capacity of Trio.

In some cases, we deploy more of these in the same PoP just because we
need more ports, not because we need more capacity; and a chassis would
not make sense for the function, yet.

Mark.

>
> What would have been nice is if Juniper oversubscribed the face plate of
> this platform, as most people are more likely to run out of ports than
> they would the 400Gbps forwarding capacity of Trio.
>

You're restricted to 400G because they did fixed lane allocations to the EA
chip on the PFE to each port group. Doing an MRATE setup to let you access
all 480G would have increased electrical complexity, and dramatically
increased the price point of the box. There are tradeoffs. The more
flexibility you want, the more expensive the box is going to be.

I'm not sure they allow oversubscription on anything in the MX line anymore
honestly. I could be wrong, I've been face down in a specific subset of
equipment for a while, someone please correct me if I am.

On Wed, Aug 23, 2023 at 2:11?AM Mark Tinka <mark@tinka.africa> wrote:

>
>
> On 8/23/23 08:00, Pascal Masha wrote:
>
> > Thanks just wanted to know whether it was a supported feature.
>
> What would have been nice is if Juniper oversubscribed the face plate of
> this platform, as most people are more likely to run out of ports than
> they would the 400Gbps forwarding capacity of Trio.
>
> In some cases, we deploy more of these in the same PoP just because we
> need more ports, not because we need more capacity; and a chassis would
> not make sense for the function, yet.
>
> Mark.
>

Does Fusion not make sense in this case? I've not had a ton of experience
with it, but it does well to add a crazy port count to an otherwise very
port limited device.

-Matt

On Wed, Aug 23, 2023 at 9:01?AM Tom Beecher <beecher@beecher.cc> wrote:

> What would have been nice is if Juniper oversubscribed the face plate of
>> this platform, as most people are more likely to run out of ports than
>> they would the 400Gbps forwarding capacity of Trio.
>>
>
> You're restricted to 400G because they did fixed lane allocations to the
> EA chip on the PFE to each port group. Doing an MRATE setup to let you
> access all 480G would have increased electrical complexity, and
> dramatically increased the price point of the box. There are tradeoffs. The
> more flexibility you want, the more expensive the box is going to be.
>
> I'm not sure they allow oversubscription on anything in the MX line
> anymore honestly. I could be wrong, I've been face down in a specific
> subset of equipment for a while, someone please correct me if I am.
>
> On Wed, Aug 23, 2023 at 2:11?AM Mark Tinka <mark@tinka.africa> wrote:
>
>>
>>
>> On 8/23/23 08:00, Pascal Masha wrote:
>>
>> > Thanks just wanted to know whether it was a supported feature.
>>
>> What would have been nice is if Juniper oversubscribed the face plate of
>> this platform, as most people are more likely to run out of ports than
>> they would the 400Gbps forwarding capacity of Trio.
>>
>> In some cases, we deploy more of these in the same PoP just because we
>> need more ports, not because we need more capacity; and a chassis would
>> not make sense for the function, yet.
>>
>> Mark.
>>
>

--
Matt Erculiani, NREMT
ERCUL-ARIN

On 8/23/23 17:01, Tom Beecher wrote:

> I'm not sure they allow oversubscription on anything in the MX line
> anymore honestly. I could be wrong, I've been face down in a specific
> subset of equipment for a while, someone please correct me if I am.

On the new ACX line, yes.

If I look at the MPC7E, MPC10E, MX10003 and MX304, no oversubscription
is allowed.

Even the LC2103 MPC on the MX10003 which has more ports than Trio
capacity, won't let you use more than 1.2Tbps (3x Trio 3 chips on it).

We don't mess around with any other MX products, so not sure (although
we are still yet to deploy the MPC10E's and the MX304).

Mark.

On Wednesday, 23 August, 2023 16:33, "Mark Tinka" <mark@tinka.africa> said:

[faceplate oversubscription]

> On the new ACX line, yes.

Not Trio, and different PLM :)

> We don't mess around with any other MX products, so not sure (although
> we are still yet to deploy the MPC10E's and the MX304).

MX304 (well, strictly LMIC16) has the same restriction, and a need for another entry in the magic port checker (https://apps.juniper.net/home/port-checker/index.html) for restrictions beyond "SUM(port-speeds) <= 1.6T".

They make sense once you've looked at the block diagram for the thing and followed the lines, but things like "4x10G breakout can only go in odd-numbered ports, and you have to leave the corresponding next-lowest even-numbered port empty" are not instantly obvious.

Thanks,
Tim.

some of these port capabilities are weird to me.  like on the
ACX7100-48L you can do 4x100 or 8x50, but ONLY one 40g ?!

me@7100> show chassis pic pic-slot 0 fpc-slot 0 | find 400
  48     0       1x400G 1x100G 1x40G 4x100G 2x100G 8x50G 2x50G 4x25G
4x10G 3x100G
  49     0       1x400G 1x100G 1x40G 4x100G 2x100G 8x50G 2x50G 4x25G
4x10G 3x100G
  50     0       1x400G 1x100G 1x40G 4x100G 2x100G 8x50G 2x50G 4x25G
4x10G 3x100G
  51     0       1x400G 1x100G 1x40G 4x100G 2x100G 8x50G 2x50G 4x25G
4x10G 3x100G
  52     0       1x400G 1x100G 1x40G 4x100G 2x100G 8x50G 2x50G 4x25G
4x10G 3x100G
  53     0       1x400G 1x100G 1x40G 4x100G 2x100G 8x50G 2x50G 4x25G
4x10G 3x100G
  54     NA      1x10G




On 8/23/2023 11:29 AM, tim@pelican.org wrote:
> On Wednesday, 23 August, 2023 16:33, "Mark Tinka" <mark@tinka.africa> said:
>
> [faceplate oversubscription]
>
>> On the new ACX line, yes.
> Not Trio, and different PLM :)
>
>> We don't mess around with any other MX products, so not sure (although
>> we are still yet to deploy the MPC10E's and the MX304).
> MX304 (well, strictly LMIC16) has the same restriction, and a need for another entry in the magic port checker (https://apps.juniper.net/home/port-checker/index.html) for restrictions beyond "SUM(port-speeds) <= 1.6T".
>
> They make sense once you've looked at the block diagram for the thing and followed the lines, but things like "4x10G breakout can only go in odd-numbered ports, and you have to leave the corresponding next-lowest even-numbered port empty" are not instantly obvious.
>
> Thanks,
> Tim.
>
>
--
-Aaron

On 8/23/23 18:29, tim@pelican.org wrote:

> Not Trio, and different PLM :)

Yes, aware... I was just speaking in general for what is likely to be a
very popular platform :-).


> MX304 (well, strictly LMIC16) has the same restriction, and a need for another entry in the magic port checker (https://apps.juniper.net/home/port-checker/index.html) for restrictions beyond "SUM(port-speeds) <= 1.6T".

Yep.

That trick they did where you can live with one RE and get 3 MIC's in
the MX304 is... well, I guess everyone will have their own opinion.


> They make sense once you've looked at the block diagram for the thing and followed the lines, but things like "4x10G breakout can only go in odd-numbered ports, and you have to leave the corresponding next-lowest even-numbered port empty" are not instantly obvious.

They do take some getting used to. But this is what comes with all the
flexibility operators often seek.

Mark.

Aaron Gould ?????(?) 2023-08-23 12:38:
> some of these port capabilities are weird to me.  like on the
> ACX7100-48L you can do 4x100 or 8x50, but ONLY one 40g ?!
>
> me@7100> show chassis pic pic-slot 0 fpc-slot 0 | find 400
>   48     0       1x400G 1x100G 1x40G 4x100G 2x100G 8x50G 2x50G 4x25G
> 4x10G 3x100G
>   49     0       1x400G 1x100G 1x40G 4x100G 2x100G 8x50G 2x50G 4x25G
> 4x10G 3x100G
>   50     0       1x400G 1x100G 1x40G 4x100G 2x100G 8x50G 2x50G 4x25G
> 4x10G 3x100G
>   51     0       1x400G 1x100G 1x40G 4x100G 2x100G 8x50G 2x50G 4x25G
> 4x10G 3x100G
>   52     0       1x400G 1x100G 1x40G 4x100G 2x100G 8x50G 2x50G 4x25G
> 4x10G 3x100G
>   53     0       1x400G 1x100G 1x40G 4x100G 2x100G 8x50G 2x50G 4x25G
> 4x10G 3x100G
>   54     NA      1x10G
>

Probably because 40G is a product 10G lanes. There are only 4 lanes
available, and the speed of a single lane can vary. So, 40G is the max
speed for the lowest single lane's speed.

Kind regards,
Andrey

On 8/23/23 17:14, Matt Erculiani wrote:

> Does Fusion not make sense in this case? I've not had a ton of
> experience with it, but it does well to add a crazy port count to an
> otherwise very port limited device.

In small edge PoP's, we attach an Arista 1U switch with tons of 1/10Gbps
ports to an MX204 via 802.1Q. Works a treat. I've never been convinced
by vendor-specific satellite systems :-).

On another note, the potential issue we might run into is pressure on
control plane memory on the MX204 for us that run BGP Add-Paths. You can
always upgrade the RE on an MX240/480/960, but the MX204 is fixed (and
last time I checked, fiddling with Juniper RE memory was generally
frowned upon).

Luckily, the MX10003 ships with 64GB of RAM, since it is now EoL.

The MX304 ships with 128GB of RAM, so anybody running Add-Paths on that
box won't have an issue there.

Mark.

>
> On another note, the potential issue we might run into is pressure on
> control plane memory on the MX204 for us that run BGP Add-Paths. You can
> always upgrade the RE on an MX240/480/960, but the MX204 is fixed (and
> last time I checked, fiddling with Juniper RE memory was generally
> frowned upon).
>

In my experience and testing with them, you have a decent bit of headroom
past the published RIB/FIB limits before they'll fall over.

On Fri, Aug 25, 2023 at 11:35?AM Mark Tinka <mark@tinka.africa> wrote:

>
>
> On 8/23/23 17:14, Matt Erculiani wrote:
>
> > Does Fusion not make sense in this case? I've not had a ton of
> > experience with it, but it does well to add a crazy port count to an
> > otherwise very port limited device.
>
> In small edge PoP's, we attach an Arista 1U switch with tons of 1/10Gbps
> ports to an MX204 via 802.1Q. Works a treat. I've never been convinced
> by vendor-specific satellite systems :-).
>
> On another note, the potential issue we might run into is pressure on
> control plane memory on the MX204 for us that run BGP Add-Paths. You can
> always upgrade the RE on an MX240/480/960, but the MX204 is fixed (and
> last time I checked, fiddling with Juniper RE memory was generally
> frowned upon).
>
> Luckily, the MX10003 ships with 64GB of RAM, since it is now EoL.
>
> The MX304 ships with 128GB of RAM, so anybody running Add-Paths on that
> box won't have an issue there.
>
> Mark.
>

On 8/25/23 19:16, Tom Beecher wrote:

> In my experience and testing with them, you have a decent bit of
> headroom past the published RIB/FIB limits before they'll fall over.

They are holding up pretty well for us, mainly because we do a lot more
BGP on MX480's than on MX204's. We use the MX204's mainly for peering
and CDN gateways. Where we use them for edge customers, it's a handful
of BGP sessions.

On MX480 16GB RE's running two full BGP feeds but hundreds of customer
sessions, Add-Paths really eats into RAM. We've had to upgrade some of
the busier routers from 16GB to 64GB RE's, especially on later versions
of code where ROV can also bite into memory on boxes carrying lots of
BGP sessions.

Mark.

No VC here, unsure if it works, but yeah, we like them and deploy them in pairs for metro-e (ce) and cbh for vlans carried over mpls pw

Reliable for us


Aaron

> On Aug 25, 2023, at 4:40 PM, Mark Tinka <mark@tinka.africa> wrote:
>
> ?
>
>> On 8/25/23 19:16, Tom Beecher wrote:
>>
>> In my experience and testing with them, you have a decent bit of headroom past the published RIB/FIB limits before they'll fall over.
>
> They are holding up pretty well for us, mainly because we do a lot more BGP on MX480's than on MX204's. We use the MX204's mainly for peering and CDN gateways. Where we use them for edge customers, it's a handful of BGP sessions.
>
> On MX480 16GB RE's running two full BGP feeds but hundreds of customer sessions, Add-Paths really eats into RAM. We've had to upgrade some of the busier routers from 16GB to 64GB RE's, especially on later versions of code where ROV can also bite into memory on boxes carrying lots of BGP sessions.
>
> Mark.

>
> On MX480 16GB RE's running two full BGP feeds but hundreds of customer
> sessions, Add-Paths really eats into RAM.
>

It would, sure. Instead of storing a single prefix/next-hop with flags in
memory, you now have to store every prefix/next-hop that you are announcing
as well.

On Fri, Aug 25, 2023 at 5:39?PM Mark Tinka <mark@tinka.africa> wrote:

>
>
> On 8/25/23 19:16, Tom Beecher wrote:
>
> > In my experience and testing with them, you have a decent bit of
> > headroom past the published RIB/FIB limits before they'll fall over.
>
> They are holding up pretty well for us, mainly because we do a lot more
> BGP on MX480's than on MX204's. We use the MX204's mainly for peering
> and CDN gateways. Where we use them for edge customers, it's a handful
> of BGP sessions.
>
> On MX480 16GB RE's running two full BGP feeds but hundreds of customer
> sessions, Add-Paths really eats into RAM. We've had to upgrade some of
> the busier routers from 16GB to 64GB RE's, especially on later versions
> of code where ROV can also bite into memory on boxes carrying lots of
> BGP sessions.
>
> Mark.
>

On 8/26/23 00:54, Tom Beecher wrote:

> It would, sure. Instead of storing a single prefix/next-hop with flags
> in memory, you now have to store every prefix/next-hop that you are
> announcing as well.

Indeed.

But it has been worth it. The load balancing from PE-to-PE has been
fantastic, especially when coupled with BGP Multipath.

No more messing about with LOCAL_PREF for multi-homed customers, and it
works just as well with different (but equal-length) AS_PATH's.

Mark.

I sincerely doubt there is much demand for *new* 40G these days.

Look at the population of 40G members on major IXes.

People have either one 10G, 2 x 10G, or 100G.

40G was a dead-end 9 years ago and much so more now.



On Wed, Aug 23, 2023 at 9:38?AM Aaron Gould <aaron1@gvtc.com> wrote:

> some of these port capabilities are weird to me. like on the
> ACX7100-48L you can do 4x100 or 8x50, but ONLY one 40g ?!
>
> me@7100> show chassis pic pic-slot 0 fpc-slot 0 | find 400
> 48 0 1x400G 1x100G 1x40G 4x100G 2x100G 8x50G 2x50G 4x25G
> 4x10G 3x100G
> 49 0 1x400G 1x100G 1x40G 4x100G 2x100G 8x50G 2x50G 4x25G
> 4x10G 3x100G
> 50 0 1x400G 1x100G 1x40G 4x100G 2x100G 8x50G 2x50G 4x25G
> 4x10G 3x100G
> 51 0 1x400G 1x100G 1x40G 4x100G 2x100G 8x50G 2x50G 4x25G
> 4x10G 3x100G
> 52 0 1x400G 1x100G 1x40G 4x100G 2x100G 8x50G 2x50G 4x25G
> 4x10G 3x100G
> 53 0 1x400G 1x100G 1x40G 4x100G 2x100G 8x50G 2x50G 4x25G
> 4x10G 3x100G
> 54 NA 1x10G
>
>
>
>
> On 8/23/2023 11:29 AM, tim@pelican.org wrote:
> > On Wednesday, 23 August, 2023 16:33, "Mark Tinka" <mark@tinka.africa>
> said:
> >
> > [faceplate oversubscription]
> >
> >> On the new ACX line, yes.
> > Not Trio, and different PLM :)
> >
> >> We don't mess around with any other MX products, so not sure (although
> >> we are still yet to deploy the MPC10E's and the MX304).
> > MX304 (well, strictly LMIC16) has the same restriction, and a need for
> another entry in the magic port checker (
> https://apps.juniper.net/home/port-checker/index.html) for restrictions
> beyond "SUM(port-speeds) <= 1.6T".
> >
> > They make sense once you've looked at the block diagram for the thing
> and followed the lines, but things like "4x10G breakout can only go in
> odd-numbered ports, and you have to leave the corresponding next-lowest
> even-numbered port empty" are not instantly obvious.
> >
> > Thanks,
> > Tim.
> >
> >
> --
> -Aaron
>
>

On 8/27/23 04:52, Eric Kuhnke wrote:

> I sincerely doubt there is much demand for *new* 40G these days.
>
> Look at the population of 40G members on major IXes.
>
> People have either one 10G, 2 x 10G, or 100G.
>
> 40G was a dead-end 9 years ago and much so more now.

We have customers that sometimes ask for 40Gbps interconnects. I always
tell our Pre-Sales team that those are the ones who "led the way", back
in the day. Sadly, they were a bit too early :-).

Mark.

Well, or they simply found a potential deal on hardware that came with 40 gig ports. 40 gigs is still a lot of bits to a lot of people.




-----
Mike Hammett
Intelligent Computing Solutions

Midwest Internet Exchange

The Brothers WISP

----- Original Message -----

From: "Mark Tinka" <mark@tinka.africa>
To: nanog@nanog.org
Sent: Saturday, August 26, 2023 10:59:36 PM
Subject: Re: MX204 Virtual Chassis Setup



On 8/27/23 04:52, Eric Kuhnke wrote:

> I sincerely doubt there is much demand for *new* 40G these days.
>
> Look at the population of 40G members on major IXes.
>
> People have either one 10G, 2 x 10G, or 100G.
>
> 40G was a dead-end 9 years ago and much so more now.

We have customers that sometimes ask for 40Gbps interconnects. I always
tell our Pre-Sales team that those are the ones who "led the way", back
in the day. Sadly, they were a bit too early :-).

Mark.

On 8/28/23 03:05, Mike Hammett wrote:
> Well, or they simply found a potential deal on hardware that came with
> 40 gig ports. 40 gigs is still a lot of bits to a lot of people.

For internal use, sure.

But when connecting to another AS, the chances of them supporting 40Gbps
in one or more places is inconsistent to slim.

Exchange points may be an exception.

Mark.

(Enterprise AS for context)

This hasn’t been my experience in the US, however we mostly deal in tier 2 markets (I.e. Detroit, Miami, Dallas, etc…) and we have plenty of 40G private interconnects. I don’t doubt 40G is going away, I’ve just never had trouble using it around here.

The only time we’ve been asked to run something other than 40G was because we like to run our ports very hot (latency insensitive traffic) and some networks do not tolerate consistently high utilization of their ports.

Different story in Japan, it’s 100G+ or nothing. You just have to find someone willing to peer with you in the first place…

Sent from my iPhone

> On Aug 27, 2023, at 23:43, Mark Tinka <mark@tinka.africa> wrote:
> ?
>
> On 8/28/23 03:05, Mike Hammett wrote:
>> Well, or they simply found a potential deal on hardware that came with 40 gig ports. 40 gigs is still a lot of bits to a lot of people.
>
> For internal use, sure.
>
> But when connecting to another AS, the chances of them supporting 40Gbps in one or more places is inconsistent to slim.
>
> Exchange points may be an exception.
>
> Mark.

On 8/28/23 07:55, Daniel Marks wrote:
> (Enterprise AS for context)
>
> This hasn’t been my experience in the US, however we mostly deal in
> tier 2 markets (I.e. Detroit, Miami, Dallas, etc…) and we have plenty
> of 40G private interconnects. I don’t doubt 40G is going away, I’ve
> just never had trouble using it around here.

This would be expected, since most 40Gbps hardware I have seen in Europe
and Africa is in the enterprise and exchange point space. If service
providers have them, I'd imagine that inventory is far lower than 100Gbps.

Mark.