Mailing List Archive: Jumbo frames / mismatch MTU

Jumbo frames / mismatch MTU

Apr 23, 2021, 6:23 AM

Post #1 of 8 (1626 views)

Hello,

I have a network configured with a QFX core, 10 ACX2200 & 10 SRX300s. The ACXs are connected to the QFX via 10G links and the SRX300s are connected via a Transparent LAN type service. The ACX & SRX have LDP,OSPF & BGP running to support MPLS on the QFX facing interfaces. The QFX is setup as a simple layer 2 device and is not running MPLS.

Each ACX & SRX has a 10G or 1G interface configured, connected to the QFX with an IP from 10.200.1.0/24. Each router has a loopback lo0 with a /32 from 10.200.2.0/24. All routers dynamically share loopback IPs via LDP & OSPF. All routers have BGP full mesh and establish BGP sessions to the loopback addresses once they are learned.

The SRX devices are limited to an MTU of 1600 due to the TLS carrier they are using to connect back to the QFX.

I need to support 9K frames from one ACX to another over this network. The QFX is configured for MTU of 9192 on all interfaces. When I configure a couple ACXs with 9192 MTU the OSPF & LDP sessions go away.

I can ping ACX to ACX with 9k packets just fine.

For some reason LDP or OSPF don?t get established the ACXs never learn the lo0 IPs of their peers which causes BGP to fail.

I?ve enabled mtu-discovery in OSPF & LDP to no avail
Is there some other setting I?m missing?

Am I allowed to have most routers running MTU 1600 and a couple running 9k? I only really need one ACX <-> QFX <-> ACX path to support 9k for SAN replication between buildings

Currently the ACX interface is:

xe-0/3/0

mtu 1600;

unit 0 {

family inet {

sampling {

input;

output;

}

address 10.200.1.1/24;

}

family mpls;

}

lo0

unit 0 {

family inet {

address 10.200.2.1/32;

}

}

protocols ldp

interface ge-0/0/0.0;

interface xe-0/3/0.0;

interface lo0.0;

protocols ospf

traffic-engineering;

export [ export-direct export-statics ];

area 0.0.0.0 {

interface lo0.0;

interface xe-0/3/0.0;

interface ge-0/0/0.0;

}

I?ve backed out the mtu-discovery for LDP & OSPF.

Everything is working. If I ?set mtu 9192? everything breaks

_______________________________________________
juniper-nsp mailing list juniper-nsp@puck.nether.net
https://puck.nether.net/mailman/listinfo/juniper-nsp

Re: Jumbo frames / mismatch MTU [ In reply to ]

hibaysal at gmail

Apr 23, 2021, 9:03 AM

Post #2 of 8 (1626 views)

Permalink

Hi,

Juniper counts layer 2 the header as well for MTU, unlike most of the vendors.

Juniper side MTU has to be set 14 bytes bigger and your OSPF and LDP should come up

Good luck!!

Sent from my iPhone

> On 23 Apr 2021, at 15:36, Matthew Crocker <matthew@corp.crocker.com> wrote:
>
> ?
> Hello,
>
> I have a network configured with a QFX core, 10 ACX2200 & 10 SRX300s. The ACXs are connected to the QFX via 10G links and the SRX300s are connected via a Transparent LAN type service. The ACX & SRX have LDP,OSPF & BGP running to support MPLS on the QFX facing interfaces. The QFX is setup as a simple layer 2 device and is not running MPLS.
>
> Each ACX & SRX has a 10G or 1G interface configured, connected to the QFX with an IP from 10.200.1.0/24. Each router has a loopback lo0 with a /32 from 10.200.2.0/24. All routers dynamically share loopback IPs via LDP & OSPF. All routers have BGP full mesh and establish BGP sessions to the loopback addresses once they are learned.
>
> The SRX devices are limited to an MTU of 1600 due to the TLS carrier they are using to connect back to the QFX.
>
> I need to support 9K frames from one ACX to another over this network. The QFX is configured for MTU of 9192 on all interfaces. When I configure a couple ACXs with 9192 MTU the OSPF & LDP sessions go away.
>
> I can ping ACX to ACX with 9k packets just fine.
>
> For some reason LDP or OSPF don’t get established the ACXs never learn the lo0 IPs of their peers which causes BGP to fail.
>
> I’ve enabled mtu-discovery in OSPF & LDP to no avail
> Is there some other setting I’m missing?
>
> Am I allowed to have most routers running MTU 1600 and a couple running 9k? I only really need one ACX <-> QFX <-> ACX path to support 9k for SAN replication between buildings
>
> Currently the ACX interface is:
>
> xe-0/3/0
>
> mtu 1600;
>
> unit 0 {
>
> family inet {
>
> sampling {
>
> input;
>
> output;
>
> }
>
> address 10.200.1.1/24;
>
> }
>
> family mpls;
>
> }
>
>
>
> lo0
>
> unit 0 {
>
> family inet {
>
> address 10.200.2.1/32;
>
> }
>
> }
>
>
> protocols ldp
>
> interface ge-0/0/0.0;
>
> interface xe-0/3/0.0;
>
> interface lo0.0;
>
>
> protocols ospf
>
> traffic-engineering;
>
> export [ export-direct export-statics ];
>
> area 0.0.0.0 {
>
> interface lo0.0;
>
> interface xe-0/3/0.0;
>
> interface ge-0/0/0.0;
>
> }
>
> I’ve backed out the mtu-discovery for LDP & OSPF.
>
> Everything is working. If I ‘set mtu 9192’ everything breaks
>
> _______________________________________________
> juniper-nsp mailing list juniper-nsp@puck.nether.net
> https://puck.nether.net/mailman/listinfo/juniper-nsp
_______________________________________________
juniper-nsp mailing list juniper-nsp@puck.nether.net
https://puck.nether.net/mailman/listinfo/juniper-nsp

Re: Jumbo frames / mismatch MTU [ In reply to ]

saku at ytti

Apr 23, 2021, 9:16 AM

Post #3 of 8 (1626 views)

Permalink

Hey,

> Juniper counts layer 2 the header as well for MTU, unlike most of the
> vendors.
> Juniper side MTU has to be set 14 bytes bigger and your OSPF and LDP
> should come up
> Good luck!!
>

Not sure this is the reason. Considering he is stating his network is all
JNPR, and considering modern CSCO and NOK count MTU the same way as JNPR
(doesn't make it any less wrong, ofc).

I don't think there is enough information to answer OP, but one possibility
is that he runs VLAN tagging on the QFX core, and the frames sent by edge
to core are too large to allow for additional 4B.

--
++ytti
_______________________________________________
juniper-nsp mailing list juniper-nsp@puck.nether.net
https://puck.nether.net/mailman/listinfo/juniper-nsp

Re: Jumbo frames / mismatch MTU [ In reply to ]

me at falz

Apr 23, 2021, 9:55 AM

Post #4 of 8 (1626 views)

Permalink

On Fri, Apr 23, 2021 at 8:35 AM Matthew Crocker
<matthew@corp.crocker.com> wrote:
> Currently the ACX interface is:
>
> xe-0/3/0
> mtu 1600;
> unit 0 {
> family inet {
> sampling {
> input;
> output;
> }
> address 10.200.1.1/24;
> }
> family mpls;
> }

You could also `set interfaces xe-0/0/3.0 family inet mtu <value>` to
force the IP mtu without having to calculate from the interface MTU.
We've done this in spots where routers/switches have different
interface MTUs (9192 / 9216) to keep the value the same at layer3.
_______________________________________________
juniper-nsp mailing list juniper-nsp@puck.nether.net
https://puck.nether.net/mailman/listinfo/juniper-nsp

Re: Jumbo frames / mismatch MTU [ In reply to ]

aaron1 at gvtc

Apr 23, 2021, 11:37 AM

Post #5 of 8 (1626 views)

Permalink

Yeah, crazy how there's many mtu differences...apparently cosmetic since we
all know ethernet always has a header !

IOS (classic and XE) doesn't include eth header

IOS-XR like MX - does include eth header

For some reason on my MX104's I did...
set interfaces ae10 mtu 9192

here's some things I've flipped on acx5048 for interfaces and vpls or
l2circuits....

set interfaces ge-0/0/47 mtu 9216

set protocols l2circuit neighbor 10.10.10.3 interface ge-0/0/47.100 mtu 1500

set routing-instances 500 protocols vpls mtu 9216

you can ignore mtu in ospf, although, I prefer to fix instead of ignore.

Also, I recall having something weird, where I ospf neighbored over a 3rd
part tls service, and when they made changes to the transport, that my ospf
started having issues establishing neighborship

(also, your mention of ldp dropping, probably has everything to do with
broken ospf, as I understand ldp needs to the /32 loopbacks to fully
establish... the initial ldp is link local mcast, but the ldp step 2 is a
tcp session to the adjacent loopbacks... hence IGP must be up)

-Aaron

_______________________________________________
juniper-nsp mailing list juniper-nsp@puck.nether.net
https://puck.nether.net/mailman/listinfo/juniper-nsp

Re: Jumbo frames / mismatch MTU [ In reply to ]

cra at alum

Apr 23, 2021, 1:29 PM

Post #6 of 8 (1626 views)

Permalink

On Fri, Apr 23, 2021 at 01:23:32PM +0000, Matthew Crocker wrote:
> The SRX devices are limited to an MTU of 1600 due to the TLS carrier they are using to connect back to the QFX.
>
> I need to support 9K frames from one ACX to another over this network. The QFX is configured for MTU of 9192 on all interfaces. When I configure a couple ACXs with 9192 MTU the OSPF & LDP sessions go away.
>
> I can ping ACX to ACX with 9k packets just fine.

> Everything is working. If I ‘set mtu 9192’ everything breaks

Just set the IP MTU in addition to the physical L2 "mtu". That way
you don't have to care about calculating any possible differences
caused by encapsulation overhead:

set interface ... family inet mtu 9000

But how are you planning on overcoming the 1600 limitation on the TLS carrier?
_______________________________________________
juniper-nsp mailing list juniper-nsp@puck.nether.net
https://puck.nether.net/mailman/listinfo/juniper-nsp

Re: Jumbo frames / mismatch MTU [ In reply to ]

saku at ytti

Apr 24, 2021, 12:19 AM

Post #7 of 8 (1624 views)

Permalink

On Fri, 23 Apr 2021 at 23:34, Chuck Anderson <cra@alum.wpi.edu> wrote:

Just set the IP MTU in addition to the physical L2 "mtu". That way
> you don't have to care about calculating any possible differences
> caused by encapsulation overhead:

I agree, but ideally you'd also know what you are doing. But it's good to
set the physical (IFD) MTU as high as possible, while always setting IFA
MTU. This is so that when you get that 1 service which does require higher
MTU, it is a non-disruptive change to enable it.

When people say the MTU is 'ethernet headers' or 'L2' I don't think this is
correct, since FCS is missing and FCS, imho, is blatantly part of the L2
Ethernet. The problem is that Ethernet is L1 and L2 and there isn't very
clear separation necessarily. I believe the reason why FCS was omitted is
purely for programmer convenience, not user convenience. The programmer
writing the code for the rewrite engine has visibility to the entire L2,
except FCS, because FCS is emitted typically later by the PHY, not by the
rewrite engine.
So this implementation detail is now exposed to users, and we just have to
know that 'yeah MTU is L2 but not FCS', which I have to say, grinds my
gears a little bit.

IMHO:
L1: Preamble(7) SFD1(1) <L2> IFG(12)
L2: DMAC(6) SMAC(6) ETYPE(2) <L3>(46..) FCS(4)

We do carry ethernet L2 over some other L1 than ethernet, in which case we
don't do preamble, sfd, ifg, we may or may not carry FCS in this case.

--
++ytti
_______________________________________________
juniper-nsp mailing list juniper-nsp@puck.nether.net
https://puck.nether.net/mailman/listinfo/juniper-nsp

Re: Jumbo frames / mismatch MTU [ In reply to ]

matthew at corp

Apr 24, 2021, 12:58 PM

Post #8 of 8 (1618 views)

Permalink

I set the interface MTU to 9192 and the family inet mtu to 1586 and everything is working.

I don?t think I?ll need to worry about the 1600 byte limitation for the TLS devices as the only thing that will be greater than 1500 bytes is the SAN replication traffic going between two ACX and will be encapsulated in MPLS. ?family mpls? on the two AXCs is 9166. Clearly if machines on the other side of the TLS link need to talk to the SAN they will have to do so on a different SAN interface that is set to 1500

Currently the ACX 2200 are acting as the border between the QFX/TLS WAN and the local LAN in each building. I haven?t enabled jumbo frames on the LAN side of the ACX yet as I need coordinate with the LAN manager during a maintenance window.

From: juniper-nsp <juniper-nsp-bounces@puck.nether.net> on behalf of Chuck Anderson <cra@alum.wpi.edu>
Date: Friday, April 23, 2021 at 4:32 PM
To: juniper-nsp@puck.nether.net <juniper-nsp@puck.nether.net>
Subject: Re: [j-nsp] Jumbo frames / mismatch MTU
CAUTION: This email originated from outside of Crocker. Do not click links or open attachments unless you recognize the sender and know the content is safe.

On Fri, Apr 23, 2021 at 01:23:32PM +0000, Matthew Crocker wrote:
> The SRX devices are limited to an MTU of 1600 due to the TLS carrier they are using to connect back to the QFX.
>
> I need to support 9K frames from one ACX to another over this network. The QFX is configured for MTU of 9192 on all interfaces. When I configure a couple ACXs with 9192 MTU the OSPF & LDP sessions go away.
>
> I can ping ACX to ACX with 9k packets just fine.

> Everything is working. If I ?set mtu 9192? everything breaks

Just set the IP MTU in addition to the physical L2 "mtu". That way
you don't have to care about calculating any possible differences
caused by encapsulation overhead:

set interface ... family inet mtu 9000

But how are you planning on overcoming the 1600 limitation on the TLS carrier?
_______________________________________________
juniper-nsp mailing list juniper-nsp@puck.nether.net
https://puck.nether.net/mailman/listinfo/juniper-nsp
_______________________________________________
juniper-nsp mailing list juniper-nsp@puck.nether.net
https://puck.nether.net/mailman/listinfo/juniper-nsp