Mailing List Archive

Are you sure your tester is capable of generating that volume of traffic?

Dave

On Mon, 21 Jan 2019 at 20:09, Jason Lixfeld <jason-jnsp@lixfeld.ca> wrote:

> Hi all,
>
> I’m doing some RFC2544 tests through an MX204. The tester is connected to
> et-0/0/2, and the test destination is somewhere out there via et-0/0/0. 64
> byte packets seem to be getting dropped, and I’m trying to find where on
> the box those drops are being recorded.
>
> I’ve distilled the test down to generating 100 million 64 byte (UDP)
> packets to the destination, but the counters on et-0/0/2 read as though
> they’ve only received about 76.6% of those packets.
>
> If I change the test to send 100 million 100 byte packets, the counters on
> et-0/0/2 account for all packets.
>
> I’ve tried looking at various output to find a counter that registers the
> missing packets, but I’m not having any luck.
>
> Aside from 'show interface et-0/0/2 extensive’, I’ve looked here with no
> luck:
>
> show interface queue et-0/0/2
> show pfe statistics traffic detail
> show pfe statistics exceptions
> show pfe statistics error
>
> Somewhere else I should be looking?
> _______________________________________________
> juniper-nsp mailing list juniper-nsp@puck.nether.net
> https://puck.nether.net/mailman/listinfo/juniper-nsp
>
_______________________________________________
juniper-nsp mailing list juniper-nsp@puck.nether.net
https://puck.nether.net/mailman/listinfo/juniper-nsp

On Mon, 21 Jan 2019 at 22:09, Jason Lixfeld <jason-jnsp@lixfeld.ca> wrote:

> I’ve distilled the test down to generating 100 million 64 byte (UDP) packets to the destination, but the counters on et-0/0/2 read as though they’ve only received about 76.6% of those packets.

This caught my eye.

84B / 64B = 76.19% - awfully close to what you're seeing.

What are you actually doing? How are measuring rates?

JunOS SNMP and 'show int' measures _l3 speed_, which is 20/84 = 23.8%.
So worst case you see 23.8Gbps out of 100Gbps interface, when it's
fully congested, because you're measuring wrong thing.

SNMP standard says L2 rate, which is 76.19Gbps out of 100Gbps, which
is also measuring wrong thing. So everyone is looking at stats, which
are wrong, unless they post-process and approximate the bps to be bps
+ (pps* overhead).

You actually want to measure true L1 rate (preamble 1, sfd 7, dmac 6,
smac 6, etype 2, payload 46, crc 4, ifg 12).

MX204 is single Eagle trio, certainly can't do wire-rate on all ports,
but can do in one port.

--
++ytti
_______________________________________________
juniper-nsp mailing list juniper-nsp@puck.nether.net
https://puck.nether.net/mailman/listinfo/juniper-nsp

Hey,

> On Jan 21, 2019, at 3:38 PM, Dave Bell <me@geordish.org> wrote:
>
> Are you sure your tester is capable of generating that volume of traffic?

Yes. I’m quite sure.

_______________________________________________
juniper-nsp mailing list juniper-nsp@puck.nether.net
https://puck.nether.net/mailman/listinfo/juniper-nsp

> On Jan 22, 2019, at 4:49 AM, Saku Ytti <saku@ytti.fi> wrote:
>
> On Mon, 21 Jan 2019 at 22:09, Jason Lixfeld <jason-jnsp@lixfeld.ca> wrote:
>
>> I’ve distilled the test down to generating 100 million 64 byte (UDP) packets to the destination, but the counters on et-0/0/2 read as though they’ve only received about 76.6% of those packets.
>
> What are you actually doing?

Trying to making sure the counters work :)

Transmitting exactly 100 million 64 byte UDP packets. SPORT: 49184 DPORT: 7.

> How are measuring rates?

I’m not interested in rate at this point. I’m interested in correlating the number of packets the tester sent vs. the number that it’s directly connected MX interface received. As it stands, there’s a discrepancy, and that’s fine, I’m trying to find the counter(s) on the box that will ultimately account for all 100 million packets.


_______________________________________________
juniper-nsp mailing list juniper-nsp@puck.nether.net
https://puck.nether.net/mailman/listinfo/juniper-nsp

> Jason Lixfeld
> Sent: Monday, January 21, 2019 8:09 PM
>
> Hi all,
>
> I’m doing some RFC2544 tests through an MX204. The tester is connected to
> et-0/0/2, and the test destination is somewhere out there via et-0/0/0. 64
> byte packets seem to be getting dropped, and I’m trying to find where on
> the box those drops are being recorded.
>
> I’ve distilled the test down to generating 100 million 64 byte (UDP) packets to
> the destination, but the counters on et-0/0/2 read as though they’ve only
> received about 76.6% of those packets.
>
> If I change the test to send 100 million 100 byte packets, the counters on et-
> 0/0/2 account for all packets.
>
> I’ve tried looking at various output to find a counter that registers the missing
> packets, but I’m not having any luck.
>
> Aside from 'show interface et-0/0/2 extensive’, I’ve looked here with no
> luck:
>
> show interface queue et-0/0/2
> show pfe statistics traffic detail
> show pfe statistics exceptions
> show pfe statistics error
>
> Somewhere else I should be looking?
>
Maybe any of the show commands in the below, if they show any drops?
https://kb.juniper.net/InfoCenter/index?page=content&id=KB26519&cat=FIREWALL&actp=LIST

I appreciate you're just concerned with packet count at the moment, but what is interesting is that if you change the rate at which you blast the packet at the box (bigger packets = slower PPS rate) the counters align all of a sudden.
That sort of indicates that for the 64B stream the packets are dropped by the platform -do you get the confirmation on the RX end of the tester about the missing packets? Not sure if this is about misaligned counters or actually about dropped packets?

How I read your test is that presumably this is 40G in and 40G out the same PFE (back to back packets @ 64B or 100B packets)
So we should just consider single PFE performance but still the resulting PPS rate is noooowhere near the theoretical PPS budget.
How are the PFEs on 204 linked together (any sacrifices in the PFE BW/PPS budget to account for the fabric)? On MPC7E all 4 PFEs would be connected via fabric.
So nothing really adds up here... shouldn't be happening -not at these rates

adam


_______________________________________________
juniper-nsp mailing list juniper-nsp@puck.nether.net
https://puck.nether.net/mailman/listinfo/juniper-nsp

You're looking in the wrong place :)

You might better understand if you look here: https://en.wikipedia.org/wiki/Ethernet_frame

> -----Original Message-----
> From: juniper-nsp [mailto:juniper-nsp-bounces@puck.nether.net] On Behalf
> Of Jason Lixfeld
> Sent: Tuesday, 22 January 2019 6:09 AM
> To: Juniper List <juniper-nsp@puck.nether.net>
> Subject: [j-nsp] Finding drops
>
> Hi all,
>
> I’m doing some RFC2544 tests through an MX204. The tester is connected to
> et-0/0/2, and the test destination is somewhere out there via et-0/0/0. 64
> byte packets seem to be getting dropped, and I’m trying to find where on
> the box those drops are being recorded.
>
> I’ve distilled the test down to generating 100 million 64 byte (UDP) packets to
> the destination, but the counters on et-0/0/2 read as though they’ve only
> received about 76.6% of those packets.
>
> If I change the test to send 100 million 100 byte packets, the counters on et-
> 0/0/2 account for all packets.
>
> I’ve tried looking at various output to find a counter that registers the missing
> packets, but I’m not having any luck.
>
> Aside from 'show interface et-0/0/2 extensive’, I’ve looked here with no
> luck:
>
> show interface queue et-0/0/2
> show pfe statistics traffic detail
> show pfe statistics exceptions
> show pfe statistics error
>
> Somewhere else I should be looking?
> _______________________________________________
> juniper-nsp mailing list juniper-nsp@puck.nether.net
> https://puck.nether.net/mailman/listinfo/juniper-nsp
_______________________________________________
juniper-nsp mailing list juniper-nsp@puck.nether.net
https://puck.nether.net/mailman/listinfo/juniper-nsp

My 2 cents: it could be interesting to check if running the system in hyper-mode makes a difference (that should normally be expected).

> Le 22 janv. 2019 à 20:42, adamv0025@netconsultings.com a écrit :
>
> That sort of indicates that for the 64B stream the packets are dropped by the platform -do you get the confirmation on the RX end of the tester about the missing packets? Not sure if this is about misaligned counters or actually about dropped packets?
>
> How I read your test is that presumably this is 40G in and 40G out the same PFE (back to back packets @ 64B or 100B packets)
> So we should just consider single PFE performance but still the resulting PPS rate is noooowhere near the theoretical PPS budget.
> How are the PFEs on 204 linked together (any sacrifices in the PFE BW/PPS budget to account for the fabric)? On MPC7E all 4 PFEs would be connected via fabric.
> So nothing really adds up here... shouldn't be happening -not at these rates

_______________________________________________
juniper-nsp mailing list juniper-nsp@puck.nether.net
https://puck.nether.net/mailman/listinfo/juniper-nsp

On Mon, 21 Jan 2019 at 20:09, Jason Lixfeld <jason-jnsp@lixfeld.ca> wrote:
>
> Hi all,
>
> I’m doing some RFC2544 tests through an MX204. The tester is connected to et-0/0/2, and the test destination is somewhere out there via et-0/0/0. 64 byte packets seem to be getting dropped, and I’m trying to find where on the box those drops are being recorded.
>
> I’ve distilled the test down to generating 100 million 64 byte (UDP) packets to the destination, but the counters on et-0/0/2 read as though they’ve only received about 76.6% of those packets.
>
> If I change the test to send 100 million 100 byte packets, the counters on et-0/0/2 account for all packets.
>
> I’ve tried looking at various output to find a counter that registers the missing packets, but I’m not having any luck.
>
> Aside from 'show interface et-0/0/2 extensive’, I’ve looked here with no luck:
>
> show interface queue et-0/0/2
> show pfe statistics traffic detail
> show pfe statistics exceptions
> show pfe statistics error
>
> Somewhere else I should be looking?

Hi Jason,

To me there are two variables here; speed and packet size. You are
sending 64B packets and experiancing issues but at what rate? 40Gbps
line rate (circa 59.4Mpps)? When you are sending 100B packets you are
not experiancing issues, at what rate is this, 40Gbps (circa 41.6Mpps)
too?

When sending larger packets (100B) the PPS rate would be lower to
achieve line rate @ 40Gbps so two things have changed between the
working and non-working that tests. To try and reduce this to a single
variable (packet size only) does you tester support any sort of packet
pacing, i.e. can you sending 64B packets but at less than line rate?
For example, can you transmit at a rate of 10Gbps of 64B packets
(circa 14.4Mpps) instead of 40Gpbs of 64B packets (circa 59.5Mpps) and
see if all the packets are counted on your ingress interface?

On a side note, when you say that the packet counters on the MX
receiving interface are missing packets, what about the transmitting
interface and the receiving RFC2544 tester? Do they also show the
packets as missing (so they were dropped) or do they show the correct
number of packets received and it's just the MX receiving interface
that isn't accounting for all packets BUT all packets are forwarded?

Cheers,
James.
_______________________________________________
juniper-nsp mailing list juniper-nsp@puck.nether.net
https://puck.nether.net/mailman/listinfo/juniper-nsp

Hey,

> On Jan 22, 2019, at 2:42 PM, adamv0025@netconsultings.com wrote:
>
> Maybe any of the show commands in the below, if they show any drops?
> https://kb.juniper.net/InfoCenter/index?page=content&id=KB26519&cat=FIREWALL&actp=LIST <https://kb.juniper.net/InfoCenter/index?page=content&id=KB26519&cat=FIREWALL&actp=LIST>

Nope. It’s all clean.

> I appreciate you're just concerned with packet count at the moment, but what is interesting is that if you change the rate at which you blast the packet at the box (bigger packets = slower PPS rate) the counters align all of a sudden.
> That sort of indicates that for the 64B stream the packets are dropped by the platform -do you get the confirmation on the RX end of the tester about the missing packets? Not sure if this is about misaligned counters or actually about dropped packets?

The Rx tester shows 76M packets, so it’s only seeing what et-0/0/2 says it’s seeing.

> How I read your test is that presumably this is 40G in and 40G out the same PFE (back to back packets @ 64B or 100B packets)

It’s 100G in/out.

> So we should just consider single PFE performance but still the resulting PPS rate is noooowhere near the theoretical PPS budget.
> How are the PFEs on 204 linked together (any sacrifices in the PFE BW/PPS budget to account for the fabric)? On MPC7E all 4 PFEs would be connected via fabric.
> So nothing really adds up here... shouldn't be happening -not at these rates
>
> adam
>
>

_______________________________________________
juniper-nsp mailing list juniper-nsp@puck.nether.net
https://puck.nether.net/mailman/listinfo/juniper-nsp

> On Jan 22, 2019, at 4:06 PM, Olivier Benghozi <olivier.benghozi@wifirst.fr> wrote:
>
> My 2 cents: it could be interesting to check if running the system in hyper-mode makes a difference (that should normally be expected).

Same results after enabling hyper-mode

_______________________________________________
juniper-nsp mailing list juniper-nsp@puck.nether.net
https://puck.nether.net/mailman/listinfo/juniper-nsp

On Tue, 22 Jan 2019 at 20:17, Jason Lixfeld <jason-jnsp@lixfeld.ca> wrote:


> Transmitting exactly 100 million 64 byte UDP packets. SPORT: 49184 DPORT: 7.

Ok so ingress interface shows 100M packets coming in, but egress
interface shows only 76M packets going out?

And nothing in 'show int egress extensive'?

What about
start shell pfe network fpc0
show jnh 0 exception terse
show jnh ifd <interface index> stream



--
++ytti
_______________________________________________
juniper-nsp mailing list juniper-nsp@puck.nether.net
https://puck.nether.net/mailman/listinfo/juniper-nsp

As per the requestor original message, I read that the packets are not
counted on INGRESS
There is a diff between received packets and the receive counter

Regards
alexander

-----Urspr?ngliche Nachricht-----
Von: juniper-nsp [mailto:juniper-nsp-bounces@puck.nether.net] Im Auftrag von
Saku Ytti
Gesendet: Mittwoch, 23. Januar 2019 14:59
An: Jason Lixfeld
Cc: Juniper List
Betreff: Re: [j-nsp] Finding drops

On Tue, 22 Jan 2019 at 20:17, Jason Lixfeld <jason-jnsp@lixfeld.ca> wrote:


> Transmitting exactly 100 million 64 byte UDP packets. SPORT: 49184
DPORT: 7.

Ok so ingress interface shows 100M packets coming in, but egress
interface shows only 76M packets going out?

And nothing in 'show int egress extensive'?

What about
start shell pfe network fpc0
show jnh 0 exception terse
show jnh ifd <interface index> stream



--
++ytti
_______________________________________________
juniper-nsp mailing list juniper-nsp@puck.nether.net
https://puck.nether.net/mailman/listinfo/juniper-nsp

_______________________________________________
juniper-nsp mailing list juniper-nsp@puck.nether.net
https://puck.nether.net/mailman/listinfo/juniper-nsp

This is somewhat embarrassing. I was looking at the wrong side of the test when I initially observed the issue and I didn’t clue into that till now, so some of the previous claims are false.

Just for completeness, here’s the actual test topology:

[ Tx Tester ] - et-0/0/2 - [ mx1 ] - et-0/0/0 - [ mx2 ] - et-0/0/2 - [ Rx Tester ]

...

> On Jan 23, 2019, at 8:58 AM, Saku Ytti <saku@ytti.fi> wrote:
>
> On Tue, 22 Jan 2019 at 20:17, Jason Lixfeld <jason-jnsp@lixfeld.ca> wrote:
>
>
>> Transmitting exactly 100 million 64 byte UDP packets. SPORT: 49184 DPORT: 7.
>
> Ok so ingress interface shows 100M packets coming in, but egress
> interface shows only 76M packets going out?

Not quite. Tester is sending 100M packets, ingress interface (et-0/0/2 @ mx1) shows 86 million packets coming in

> And nothing in 'show int egress extensive'?

Now that I’m looking at the right box, yes! More importantly, on et-0/0/2 @ mx1:

…
Traffic statistics:
Input packets: 85405847 0 pps
…
Input errors:
Errors: 0, Drops: 0, Framing errors: 0, Runts: 0, Policed discards: 0, L3 incompletes: 0, L2 channel errors: 0, L2 mismatch timeouts: 0, FIFO errors: 0, Resource errors: 14594154
…

85405847+14594154 = 100000001

So now I see an error counter that has incremented and that accounts for the missing packets. This is what I was after.



Incidentally, Olivier brought up hyper mode, so here are the results of that.

et-0/0/2 @ mx1:
…
Traffic statistics:
Input packets: 100000017 0 pps

et-0/0/0 @ mx1:
…
Traffic statistics:
Output packets: 76645888 8 pps
…
Output errors:
Carrier transitions: 1, Errors: 0, Drops: 23355576, Collisions: 0, Aged packets: 0, FIFO errors: 0, HS link CRC errors: 0, MTU errors: 0, Resource errors: 0
Egress queues: 8 supported, 4 in use
Queue counters: Queued packets Transmitted packets Dropped packets
0 100000097 76644521 23355576
1 0 0 0
2 0 0 0
3 1348 1348 0

Thanks all! Now I can try and look into those error counters.
_______________________________________________
juniper-nsp mailing list juniper-nsp@puck.nether.net
https://puck.nether.net/mailman/listinfo/juniper-nsp

On Wed, 23 Jan 2019 at 17:01, Jason Lixfeld <jason-jnsp@lixfeld.ca> wrote:

> Now that I’m looking at the right box, yes! More importantly, on et-0/0/2 @ mx1:
>
> Input errors:
> Errors: 0, Drops: 0, Framing errors: 0, Runts: 0, Policed discards: 0, L3 incompletes: 0, L2 channel errors: 0, L2 mismatch timeouts: 0, FIFO errors: 0, Resource errors: 14594154
> …
>
> 85405847+14594154 = 100000001
> Queue counters: Queued packets Transmitted packets Dropped packets
> 0 100000097 76644521 23355576
> 3 1348 1348 0

Have you changed class-of-service config from standard? My first guess
would be that queue0 depth is insufficient, but even that is bit odd
as you have same rate ingress and egress, so no queueing should occur,
unless someone other interface is competing for same egress as well.


--
++ytti
_______________________________________________
juniper-nsp mailing list juniper-nsp@puck.nether.net
https://puck.nether.net/mailman/listinfo/juniper-nsp

> On Jan 23, 2019, at 10:23 AM, Saku Ytti <saku@ytti.fi> wrote:
>
> On Wed, 23 Jan 2019 at 17:01, Jason Lixfeld <jason-jnsp@lixfeld.ca> wrote:
>
>> Now that I’m looking at the right box, yes! More importantly, on et-0/0/2 @ mx1:
>>
>> Input errors:
>> Errors: 0, Drops: 0, Framing errors: 0, Runts: 0, Policed discards: 0, L3 incompletes: 0, L2 channel errors: 0, L2 mismatch timeouts: 0, FIFO errors: 0, Resource errors: 14594154
>> …
>>
>> 85405847+14594154 = 100000001
>> Queue counters: Queued packets Transmitted packets Dropped packets
>> 0 100000097 76644521 23355576
>> 3 1348 1348 0
>
> Have you changed class-of-service config from standard? My first guess
> would be that queue0 depth is insufficient, but even that is bit odd
> as you have same rate ingress and egress, so no queueing should occur,
> unless someone other interface is competing for same egress as well.

There’s been no adjustment of CoS from factory default, and there’s no other interface competing for that egress.

At this point I’ll take it up with JTAC.

>
> --
> ++ytti

_______________________________________________
juniper-nsp mailing list juniper-nsp@puck.nether.net
https://puck.nether.net/mailman/listinfo/juniper-nsp

Hey Jason,

> Jason Lixfeld
> Sent: Wednesday, January 23, 2019 3:02 PM
>
> This is somewhat embarrassing. I was looking at the wrong side of the test
> when I initially observed the issue and I didn’t clue into that till now, so some
> of the previous claims are false.
>
> Just for completeness, here’s the actual test topology:
>
> [ Tx Tester ] - et-0/0/2 - [ mx1 ] - et-0/0/0 - [ mx2 ] - et-0/0/2 - [ Rx Tester ]
>
Is the test stream unidirectional please? -say from left (the mx1 side) to right (mx2 side) please? Or bidirectional please?
Now that you're looking at the right router.
Can you please run the: show pfe statistics traffic fpc 0 | match "cell|drops"
on mx1
If it shows info cell drops then that means the PFE can't cope with the PPS rate.

Since as Saku confirmed both interfaces et-0/0/2 and et-0/0/0 on mx1 are on the same PFE then the packet processing computational load for ingress and egress processing is not spread across two PFEs but rather executed on a single PFE which has to handle 200Gbps (100in+100out) worth of traffic @ 64bps, can't be bothered to calculate the pps rate there, but my guess is that the PFE can't handle the resulting PPS rate (as it is most likely above the PFE's overall (in+out) PPS budget) which is not that high on Gen3 (applies to most NPUs out there with 100g ports).
If the chip is rated for 800G(400in+400out) extrapolating from my notes on MPC7 testing the 204PFE then should cope with ~200in+~200out @64bit (if your traffic is bidirectional you'd be at the limit.

> Incidentally, Olivier brought up hyper mode, so here are the results of that.
>
> et-0/0/2 @ mx1:
> Input packets: 100000017 0 pps
>
> et-0/0/0 @ mx1:
> Output packets: 76645888 8 pps
> …
This is an interesting result,
Seems like hyper-mode helped with the ingress computation bit
-is the flow-control disabled on all interfaces involved with this test please? (we don't want the mx2 to send pause frames to et-0/0/0 on mx1 when it can't cope with the ingress PPS rate, skewing the results)

adam


_______________________________________________
juniper-nsp mailing list juniper-nsp@puck.nether.net
https://puck.nether.net/mailman/listinfo/juniper-nsp

On Thu, 24 Jan 2019 at 12:52, <adamv0025@netconsultings.com> wrote:

> If it shows info cell drops then that means the PFE can't cope with the PPS rate.

It can't drop cells, as packets are never cellified, as platform does
not have FAB side at all. There is only WAN side, so all is local
switched.

> Since as Saku confirmed both interfaces et-0/0/2 and et-0/0/0 on mx1 are on the same PFE then the packet processing computational load for ingress and egress processing is not spread across two PFEs but rather executed on a single PFE which has to handle 200Gbps (100in+100out) worth of traffic @ 64bps, can't be bothered to calculate the pps rate there, but my guess is that the PFE can't handle the resulting PPS rate (as it is most likely above the PFE's overall (in+out) PPS budget) which is not that high on Gen3 (applies to most NPUs out there with 100g ports).
> If the chip is rated for 800G(400in+400out) extrapolating from my notes on MPC7 testing the 204PFE then should cope with ~200in+~200out @64bit (if your traffic is bidirectional you'd be at the limit.

EAChip (mqss) on fabric platforms has 400Gbps WAN + 400Gbps FAB, MX204
is fabless, so it's 800Gbps WAN. This test should pass with modest
luss load.

> -is the flow-control disabled on all interfaces involved with this test please? (we don't want the mx2 to send pause frames to et-0/0/0 on mx1 when it can't cope with the ingress PPS rate, skewing the results)

This is really good proposal, flow-control is on by default.

--
++ytti
_______________________________________________
juniper-nsp mailing list juniper-nsp@puck.nether.net
https://puck.nether.net/mailman/listinfo/juniper-nsp

Hey Adam,

> On Jan 24, 2019, at 5:51 AM, <adamv0025@netconsultings.com> <adamv0025@netconsultings.com> wrote:
>
> Is the test stream unidirectional please? -say from left (the mx1 side) to right (mx2 side) please? Or bidirectional please?

It’s been bi-directional, in that the Rx Tester is set to loopback. More or less only so I could see the number of packets received out the far side. This tester won’t count ingress packets unless it’s in loopback mode, or actually running some test.

In any event, turning the loopback off doesn’t have any effect on the results.

> Now that you're looking at the right router.
> Can you please run the: show pfe statistics traffic fpc 0 | match "cell|drops"
> on mx1
> If it shows info cell drops then that means the PFE can't cope with the PPS rate.

No matches here. As Saku eluded to.

> Since as Saku confirmed both interfaces et-0/0/2 and et-0/0/0 on mx1 are on the same PFE then the packet processing computational load for ingress and egress processing is not spread across two PFEs but rather executed on a single PFE which has to handle 200Gbps (100in+100out) worth of traffic @ 64bps, can't be bothered to calculate the pps rate there, but my guess is that the PFE can't handle the resulting PPS rate (as it is most likely above the PFE's overall (in+out) PPS budget) which is not that high on Gen3 (applies to most NPUs out there with 100g ports).
>

> If the chip is rated for 800G(400in+400out) extrapolating from my notes on MPC7 testing the 204PFE then should cope with ~200in+~200out @64bit (if your traffic is bidirectional you'd be at the limit.

Are there no specs published that provide the max number of pps @ 64 bytes the MPC7?

> -is the flow-control disabled on all interfaces involved with this test please? (we don't want the mx2 to send pause frames to et-0/0/0 on mx1 when it can't cope with the ingress PPS rate, skewing the results)


It is disabled.

jlixfeld@mx1# wildcard range show interfaces et-0/0/[0,2] | display set | match flow
set interfaces et-0/0/0 ether-options no-flow-control
set interfaces et-0/0/2 ether-options no-flow-control

[edit]
jlixfeld@mx1#

jlixfeld@mx2# wildcard range show interfaces et-0/0/[0,2] | display set | match flow
set interfaces et-0/0/0 ether-options no-flow-control
set interfaces et-0/0/2 ether-options no-flow-control

[edit]
jlixfeld@mx2#
_______________________________________________
juniper-nsp mailing list juniper-nsp@puck.nether.net
https://puck.nether.net/mailman/listinfo/juniper-nsp

Hi,

Just to close the loop on this, according to JTAC, the throughput issues observed are addressed in KB33477 (basically, wire speed can be achieved on > 96 byte packets).

> On Jan 24, 2019, at 9:43 AM, Jason Lixfeld <jason-jnsp@lixfeld.ca> wrote:
>
> Hey Adam,
>
>> On Jan 24, 2019, at 5:51 AM, <adamv0025@netconsultings.com> <adamv0025@netconsultings.com> wrote:
>>
>> Is the test stream unidirectional please? -say from left (the mx1 side) to right (mx2 side) please? Or bidirectional please?
>
> It’s been bi-directional, in that the Rx Tester is set to loopback. More or less only so I could see the number of packets received out the far side. This tester won’t count ingress packets unless it’s in loopback mode, or actually running some test.
>
> In any event, turning the loopback off doesn’t have any effect on the results.
>
>> Now that you're looking at the right router.
>> Can you please run the: show pfe statistics traffic fpc 0 | match "cell|drops"
>> on mx1
>> If it shows info cell drops then that means the PFE can't cope with the PPS rate.
>
> No matches here. As Saku eluded to.
>
>> Since as Saku confirmed both interfaces et-0/0/2 and et-0/0/0 on mx1 are on the same PFE then the packet processing computational load for ingress and egress processing is not spread across two PFEs but rather executed on a single PFE which has to handle 200Gbps (100in+100out) worth of traffic @ 64bps, can't be bothered to calculate the pps rate there, but my guess is that the PFE can't handle the resulting PPS rate (as it is most likely above the PFE's overall (in+out) PPS budget) which is not that high on Gen3 (applies to most NPUs out there with 100g ports).
>>
>
>> If the chip is rated for 800G(400in+400out) extrapolating from my notes on MPC7 testing the 204PFE then should cope with ~200in+~200out @64bit (if your traffic is bidirectional you'd be at the limit.
>
> Are there no specs published that provide the max number of pps @ 64 bytes the MPC7?
>
>> -is the flow-control disabled on all interfaces involved with this test please? (we don't want the mx2 to send pause frames to et-0/0/0 on mx1 when it can't cope with the ingress PPS rate, skewing the results)
>
>
> It is disabled.
>
> jlixfeld@mx1# wildcard range show interfaces et-0/0/[0,2] | display set | match flow
> set interfaces et-0/0/0 ether-options no-flow-control
> set interfaces et-0/0/2 ether-options no-flow-control
>
> [edit]
> jlixfeld@mx1#
>
> jlixfeld@mx2# wildcard range show interfaces et-0/0/[0,2] | display set | match flow
> set interfaces et-0/0/0 ether-options no-flow-control
> set interfaces et-0/0/2 ether-options no-flow-control
>
> [edit]
> jlixfeld@mx2#
> _______________________________________________
> juniper-nsp mailing list juniper-nsp@puck.nether.net
> https://puck.nether.net/mailman/listinfo/juniper-nsp

_______________________________________________
juniper-nsp mailing list juniper-nsp@puck.nether.net
https://puck.nether.net/mailman/listinfo/juniper-nsp

Hi Jason,
Sorry missed the previous email,

Hmm interesting, so it's capped at the WA block then not on the ASIC, good to know.
On MPC7s we did not run into this issue but rather to the chip overload problem
From my notes:
MPC7 24x10GE per PFE that is 2x212.773Mpps (425.546Mpps two directions) -> 17.73Mpps per 10GE
MPC7 20x10GE per PFE that is 2x212.773Mpps (425.546Mpps two directions) -> 21.28Mpps per 10GE
Just for reference:
Line-rate 10Gbps@L2(64B)@L1(84B) is 29.761Mpps (two directions

Under NDA you can get what juniper tested internally, some graphs etc..
But it's better to do your own testing (in a specific use case)

See the full linerate performance is not there so that you can send 64b packets at 100G no one would do that -but it's terher to give you some headroom to send packets of normal size and do some processing while maintaining the performance.
So yes with 100GE capable chips you have to be careful nowadays on what features you enable.

adam
> -----Original Message-----
> From: Jason Lixfeld <jason-jnsp@lixfeld.ca>
> Sent: Wednesday, January 30, 2019 5:05 PM
> To: adamv0025@netconsultings.com
> Cc: Saku Ytti <saku@ytti.fi>; Juniper List <juniper-nsp@puck.nether.net>
> Subject: Re: [j-nsp] Finding drops
>
> Hi,
>
> Just to close the loop on this, according to JTAC, the throughput issues
> observed are addressed in KB33477 (basically, wire speed can be achieved on
> > 96 byte packets).
>
> > On Jan 24, 2019, at 9:43 AM, Jason Lixfeld <jason-jnsp@lixfeld.ca> wrote:
> >
> > Hey Adam,
> >
> >> On Jan 24, 2019, at 5:51 AM, <adamv0025@netconsultings.com>
> <adamv0025@netconsultings.com> wrote:
> >>
> >> Is the test stream unidirectional please? -say from left (the mx1 side) to
> right (mx2 side) please? Or bidirectional please?
> >
> > It’s been bi-directional, in that the Rx Tester is set to loopback. More or less
> only so I could see the number of packets received out the far side. This
> tester won’t count ingress packets unless it’s in loopback mode, or actually
> running some test.
> >
> > In any event, turning the loopback off doesn’t have any effect on the
> results.
> >
> >> Now that you're looking at the right router.
> >> Can you please run the: show pfe statistics traffic fpc 0 | match
> "cell|drops"
> >> on mx1
> >> If it shows info cell drops then that means the PFE can't cope with the PPS
> rate.
> >
> > No matches here. As Saku eluded to.
> >
> >> Since as Saku confirmed both interfaces et-0/0/2 and et-0/0/0 on mx1 are
> on the same PFE then the packet processing computational load for ingress
> and egress processing is not spread across two PFEs but rather executed on a
> single PFE which has to handle 200Gbps (100in+100out) worth of traffic @
> 64bps, can't be bothered to calculate the pps rate there, but my guess is that
> the PFE can't handle the resulting PPS rate (as it is most likely above the PFE's
> overall (in+out) PPS budget) which is not that high on Gen3 (applies to most
> NPUs out there with 100g ports).
> >>
> >
> >> If the chip is rated for 800G(400in+400out) extrapolating from my notes
> on MPC7 testing the 204PFE then should cope with ~200in+~200out @64bit
> (if your traffic is bidirectional you'd be at the limit.
> >
> > Are there no specs published that provide the max number of pps @ 64
> bytes the MPC7?
> >
> >> -is the flow-control disabled on all interfaces involved with this
> >> test please? (we don't want the mx2 to send pause frames to et-0/0/0
> >> on mx1 when it can't cope with the ingress PPS rate, skewing the
> >> results)
> >
> >
> > It is disabled.
> >
> > jlixfeld@mx1# wildcard range show interfaces et-0/0/[0,2] | display
> > set | match flow set interfaces et-0/0/0 ether-options no-flow-control
> > set interfaces et-0/0/2 ether-options no-flow-control
> >
> > [edit]
> > jlixfeld@mx1#
> >
> > jlixfeld@mx2# wildcard range show interfaces et-0/0/[0,2] | display
> > set | match flow set interfaces et-0/0/0 ether-options no-flow-control
> > set interfaces et-0/0/2 ether-options no-flow-control
> >
> > [edit]
> > jlixfeld@mx2#
> > _______________________________________________
> > juniper-nsp mailing list juniper-nsp@puck.nether.net
> > https://puck.nether.net/mailman/listinfo/juniper-nsp


_______________________________________________
juniper-nsp mailing list juniper-nsp@puck.nether.net
https://puck.nether.net/mailman/listinfo/juniper-nsp

Interesting to see that Hyper-mode is useless on MX204, by the way (it's expected to do something on MPC7).

> Le 31 janv. 2019 à 16:46, adamv0025@netconsultings.com a écrit :
>
> Hmm interesting, so it's capped at the WA block then not on the ASIC, good to know.
> On MPC7s we did not run into this issue but rather to the chip overload problem
> From my notes:
> MPC7 24x10GE per PFE that is 2x212.773Mpps (425.546Mpps two directions) -> 17.73Mpps per 10GE
> MPC7 20x10GE per PFE that is 2x212.773Mpps (425.546Mpps two directions) -> 21.28Mpps per 10GE
> Just for reference:
> Line-rate 10Gbps@L2(64B)@L1(84B) is 29.761Mpps (two directions
>
> Under NDA you can get what juniper tested internally, some graphs etc..
> But it's better to do your own testing (in a specific use case)
>
> See the full linerate performance is not there so that you can send 64b packets at 100G no one would do that -but it's terher to give you some headroom to send packets of normal size and do some processing while maintaining the performance.
> So yes with 100GE capable chips you have to be careful nowadays on what features you enable.

_______________________________________________
juniper-nsp mailing list juniper-nsp@puck.nether.net
https://puck.nether.net/mailman/listinfo/juniper-nsp