Mailing List Archive: Layer 2 packet forwarded in hardware(PP)

Layer 2 packet forwarded in hardware(PP)

Aug 13, 2018, 8:43 AM

Post #1 of 6 (1967 views)

Hello everyone,

I currently see increased CPU% one one of our MLXe line cards:

#sh cpu lp

15:48:40 GMT+01 Mon Aug 13 2018

SLOT #: LP CPU UTILIZATION in %:
in 1 second: in 5 seconds: in 60 seconds: in 300
seconds:
1: 6 5 4 4
2: 37 30 30 30
3: 1 1 1 1
4: 1 1 1 1

Traffic hitting the CPU with "Layer 2 packet forwarded in hardware(PP)".
Does anyone know what this means? If the packet is being forwarded in
hardware, why does it hit the LP cpu?

[ppcr_rx_packet]: Packet received
Time stamp : 23 day(s) 00h 42m 13s:,
TM Header: [ 00be 0113 0013 ]
Type: Fabric Unicast(0x00000000) Size: 190 Class: 0 Src sys port: 275
Dest Port: 0 Drop Prec: 0 Ing Q Sig: 1 Out mirr dis: 0x0 Excl src: 1
Sys mc: 1
**********************************************************************
Packet size: 182, XPP reason code: 0x00000000
00: a054 2003 3850 0e0b-0028 506e 0000 0000 FID = 0xa054
10: 000c 2939 c197 000c-29f5 951b 0800 4500 Offset = 0x10
20: 0212 ef53 4000 4011-2139 0a0a 0a1b 0a0a VLAN = 3595(0x0e0b)
30: 0a20 b1c5 08af 01fe-20a7 7b22 5365 7665 CAM = 0x142837(R)
40: 7269 7479 2220 3a20-2274 7261 6365 222c SFLOW = 0
50: 2253 7263 2220 3a20-225c 7530 3034 355c DBL TAG = 0
60: 7530 3036 455c 7530-3037 345c 7530 3036
70: 395c 7530 3037 345c-7530 3037 395c 7530
Pri CPU MON SRC PType US BRD DAV SAV DPV SV ER TXA SAS Tag MVID
0 1 0 1/1 3 0 0 1 1 1 0 0 0 0 1 0

10.10.10.27 -> 10.10.10.32 UDP [45509 -> 2223]
**********************************************************************
[ppcr_tx_packet] ACTION: Drop packet(reason: Layer 2 packet forwarded in
hardware(PP))

I also see arp packets hitting the LP CPU like below. Interesting that first packet is being
classified as "Fabric Unicast" while the second one is "Multicast" and
actually both are broadcast arp packets to ffff ffff ffff? Finally, ARP
packets should not hit the LP CPU as those are pure L2 VLANS?

[ppcr_rx_packet]: Packet received
Time stamp : 23 day(s) 00h 42m 13s:,
TM Header: [ 0052 6113 0010 ]
Type: Fabric Unicast(0x00000000) Size: 82 Class: 3 Src sys port: 275
Dest Port: 0 Drop Prec: 0 Ing Q Sig: 1 Out mirr dis: 0x0 Excl src: 0
Sys mc: 0
**********************************************************************
Packet size: 76, XPP reason code: 0x00000000
00: 05f2 c002 5050 6e11-0c55 6ea6 0600 0000 FID = 0x05f2
10: ffff ffff ffff 000c-29da 8cf3 0806 0001 Offset = 0x10
20: 0800 0604 0001 000c-29da 8cf3 ac10 0505 VLAN = 3601(0x0e11)
30: 0000 0000 0000 ac10-0501 0000 0000 0000 CAM = 0x0ab753(R)
40: 0000 0000 0000 0000-0000 0000 0000 7530 SFLOW = 0
50: 3037 7530 3037 7530-3037 225c 7530 3035 DBL TAG = 0
60: 325c 7530 3036 355c-7530 3036 345c 7530
70: 3036 395c 7530 3037-335c 7530 3034 335c
Pri CPU MON SRC PType US BRD DAV SAV DPV SV ER TXA SAS Tag MVID
3 0 0 2/2 2 0 1 0 1 0 0 0 0 0 1 0

000c.29da.8cf3 -> Broadcast ARP Req 172.16.5.5 -> 172.16.5.1
**********************************************************************
[ppcr_tx_packet] ACTION: Forward packet using fid 0xa057
[xpp10ge_cpu_forward_debug]: Forward LP packet
Time stamp : 23 day(s) 00h 42m 13s:,
TM Header: [ 1012 0057 a057 ]
Type: Multicast(0x00000000) Size: 18 Mcast ID: 0x7a0 Src Port: 1
Drp Pri: 1 Snp: 1 Exclude Src: 1 Cls: 0x00000003
**********************************************************************
00: a057 c002 5050 6e11-0000 0098 0600 0000 FID = 0xa057
10: ffff ffff ffff 000c-29da 8cf3 0806 0001 Offset = 0x10
20: 0800 0604 0001 000c-29da 8cf3 ac10 0505 VLAN = 3601(0x0e11)
30: 0000 0000 0000 ac10-0501 0000 0000 0000 CAM = 0x00004c
40: 0000 0000 0000 0000-0000 0000 0000 60c0 SFLOW = 0
50: 0000 0028 0601 2a03-2880 f21c 80c4 face DBL TAG = 0
60: b00c 0000 43fe 2a01-07a0 0002 101f 0225
70: 90ff fee9 c3a4 01bb-bcd9 da6f 5abd 98d4
Pri CPU MON SRC PType US BRD DAV SAV DPV SV ER TXA SAS Tag MVID
3 0 0 1/1 2 0 1 0 1 0 0 0 0 0 1 0

000c.29da.8cf3 -> Broadcast ARP Req 172.16.5.5 -> 172.16.5.1
**********************************************************************

There is a LAG on those ports:
Port [Sys P] [Port P] [ Key ] [Act][Tio][Agg][Syn][Col][Dis][Def][Exp][Ope]
1/1 1 1 107 Yes L Agg Syn Col Dis No No Ope
1/2 1 1 107 Yes L Agg Syn Col Dis No No Ope
1/6 1 1 107 Yes L Agg Syn Col Dis No No Ope
1/15 1 1 107 Yes L Agg Syn Col Dis No No Ope
2/1 1 1 107 Yes L Agg Syn Col Dis No No Ope
2/2 1 1 107 Yes L Agg Syn Col Dis No No Ope
2/6 1 1 107 Yes L Agg Syn Col Dis No No Ope
2/15 1 1 107 Yes L Agg Syn Col Dis No No Ope

Any ideas about this one?

Best regards,

Franz Georg
_______________________________________________
foundry-nsp mailing list
foundry-nsp@puck.nether.net
http://puck.nether.net/mailman/listinfo/foundry-nsp

Re: Layer 2 packet forwarded in hardware(PP) [ In reply to ]

lists at openunix

Sep 10, 2018, 3:10 AM

Post #2 of 6 (1957 views)

Permalink

I see on another device traffic hitting the CPU that looks like the same
packet hits twice with different reasons.
Does anyone have an idea why this happens?
I wonder why it is being claffied as multicast only once (and finally it
doesn't look like multicast)?

[ppcr_rx_packet]: Packet received
Time stamp : 83 day(s) 06h 07m 07s:,
TM Header: [ 0151 0a25 0000 ]
Type: Fabric Unicast(0x00000000) Size: 337 Class: 0 Src sys port: 2597
Dest Port: 0 Drop Prec: 0 Ing Q Sig: 0 Out mirr dis: 0x0 Excl src: 0 Sys mc: 0
**********************************************************************
Packet size: 331, XPP reason code: 0x0000a8a9
00: 05f0 0003 1850 0e34-78cb fffe 0000 0000 FID = 0x05f0
10: 000c 29ab f8be 000c-297e ba9a 0800 4500 Offset = 0x10
20: 012d 030c 4000 4011-1ff6 0a40 0125 0a40 VLAN = 3636(0x0e34)
30: 011a 8f2b 1f93 0119-f6e8 7b22 7322 3a22 CAM = 0x05ffff(R)
40: 7375 6974 6522 2c22-7461 7267 6574 223a SFLOW = 0
50: 2275 7365 7222 2c22-6465 7669 6365 223a DBL TAG = 0
60: 6661 6c73 652c 2274-7970 6522 3a22 7573
70: 6572 6261 6e6b 222c-2269 6422 3a22 3562
Pri CPU MON SRC PType US BRD DAV SAV DPV SV ER TXA SAS Tag MVID
0 0 0 11/4 3 0 0 0 1 1 0 0 0 0 1 0

10.64.1.37 -> 10.64.1.26 UDP [36651 -> 8083]
**********************************************************************
[ppcr_tx_packet] ACTION: Forward packet using fid 0xa3df
[xpp10ge_cpu_forward_debug]: Forward LP packet
Time stamp : 83 day(s) 06h 07m 07s:,
TM Header: [ 1002 03df a3df ]
Type: Multicast(0x00000000) Size: 2 Mcast ID: 0xfa3 Src Port: 15
Drp Pri: 3 Snp: 1 Exclude Src: 1 Cls: 0x00000003
**********************************************************************
00: a3df 0003 1850 0e34-84c0 0296 0000 0000 FID = 0xa3df
10: 000c 29ab f8be 000c-297e ba9a 0800 4500 Offset = 0x10
20: 012d 030c 4000 4011-1ff6 0a40 0125 0a40 VLAN = 3636(0x0e34)
30: 011a 8f2b 1f93 0119-f6e8 7b22 7322 3a22 CAM = 0x00014b
40: 7375 6974 6522 2c22-7461 7267 6574 223a SFLOW = 0
50: 2275 7365 7222 2c22-6465 7669 6365 223a DBL TAG = 0
60: 6661 6c73 652c 2274-7970 6522 3a22 7573
70: 6572 6261 6e6b 222c-2269 6422 3a22 3562
Pri CPU MON SRC PType US BRD DAV SAV DPV SV ER TXA SAS Tag MVID
0 0 0 12/4 3 0 0 0 1 1 0 0 0 0 1 0

10.64.1.37 -> 10.64.1.26 UDP [36651 -> 8083]
**********************************************************************

_______________________________________________
foundry-nsp mailing list
foundry-nsp@puck.nether.net
http://puck.nether.net/mailman/listinfo/foundry-nsp

Re: Layer 2 packet forwarded in hardware(PP) [ In reply to ]

ekoyle+puck.nether.net at gmail

Sep 10, 2018, 7:22 AM

Post #3 of 6 (1957 views)

Permalink

You can enable cpu-protection on the vlan IIRC, I don't remember all the
caveats; definitely look at the manual before enabling.

--
Eldon

On Mon, Sep 10, 2018, 04:11 Franz Georg Köhler <lists@openunix.de> wrote:

> I see on another device traffic hitting the CPU that looks like the same
> packet hits twice with different reasons.
> Does anyone have an idea why this happens?
> I wonder why it is being claffied as multicast only once (and finally it
> doesn't look like multicast)?
>
> [ppcr_rx_packet]: Packet received
> Time stamp : 83 day(s) 06h 07m 07s:,
> TM Header: [ 0151 0a25 0000 ]
> Type: Fabric Unicast(0x00000000) Size: 337 Class: 0 Src sys port: 2597
> Dest Port: 0 Drop Prec: 0 Ing Q Sig: 0 Out mirr dis: 0x0 Excl src: 0 Sys
> mc: 0
> **********************************************************************
> Packet size: 331, XPP reason code: 0x0000a8a9
> 00: 05f0 0003 1850 0e34-78cb fffe 0000 0000 FID = 0x05f0
> 10: 000c 29ab f8be 000c-297e ba9a 0800 4500 Offset = 0x10
> 20: 012d 030c 4000 4011-1ff6 0a40 0125 0a40 VLAN = 3636(0x0e34)
> 30: 011a 8f2b 1f93 0119-f6e8 7b22 7322 3a22 CAM = 0x05ffff(R)
> 40: 7375 6974 6522 2c22-7461 7267 6574 223a SFLOW = 0
> 50: 2275 7365 7222 2c22-6465 7669 6365 223a DBL TAG = 0
> 60: 6661 6c73 652c 2274-7970 6522 3a22 7573
> 70: 6572 6261 6e6b 222c-2269 6422 3a22 3562
> Pri CPU MON SRC PType US BRD DAV SAV DPV SV ER TXA SAS Tag MVID
> 0 0 0 11/4 3 0 0 0 1 1 0 0 0 0 1 0
>
> 10.64.1.37 -> 10.64.1.26 UDP [36651 -> 8083]
> **********************************************************************
> [ppcr_tx_packet] ACTION: Forward packet using fid 0xa3df
> [xpp10ge_cpu_forward_debug]: Forward LP packet
> Time stamp : 83 day(s) 06h 07m 07s:,
> TM Header: [ 1002 03df a3df ]
> Type: Multicast(0x00000000) Size: 2 Mcast ID: 0xfa3 Src Port: 15
> Drp Pri: 3 Snp: 1 Exclude Src: 1 Cls: 0x00000003
> **********************************************************************
> 00: a3df 0003 1850 0e34-84c0 0296 0000 0000 FID = 0xa3df
> 10: 000c 29ab f8be 000c-297e ba9a 0800 4500 Offset = 0x10
> 20: 012d 030c 4000 4011-1ff6 0a40 0125 0a40 VLAN = 3636(0x0e34)
> 30: 011a 8f2b 1f93 0119-f6e8 7b22 7322 3a22 CAM = 0x00014b
> 40: 7375 6974 6522 2c22-7461 7267 6574 223a SFLOW = 0
> 50: 2275 7365 7222 2c22-6465 7669 6365 223a DBL TAG = 0
> 60: 6661 6c73 652c 2274-7970 6522 3a22 7573
> 70: 6572 6261 6e6b 222c-2269 6422 3a22 3562
> Pri CPU MON SRC PType US BRD DAV SAV DPV SV ER TXA SAS Tag MVID
> 0 0 0 12/4 3 0 0 0 1 1 0 0 0 0 1 0
>
> 10.64.1.37 -> 10.64.1.26 UDP [36651 -> 8083]
> **********************************************************************
>
>
> _______________________________________________
> foundry-nsp mailing list
> foundry-nsp@puck.nether.net
> http://puck.nether.net/mailman/listinfo/foundry-nsp
>

Re: Layer 2 packet forwarded in hardware(PP) [ In reply to ]

lists at openunix

Sep 13, 2018, 4:57 AM

Post #4 of 6 (1955 views)

Permalink

On Mo, Sep 10, 2018 at 08:22:19 -0600, Eldon Koyle <ekoyle+puck.nether.net@gmail.com> wrote:
> You can enable cpu-protection on the vlan IIRC, I don't remember all the
> caveats; definitely look at the manual before enabling.

Also with CPU protection enabled on the VLAN I see packets hitting the
CPU with "reason: Layer 2 packet forwarded in hardware(PP)".

Therefore, I still wonder what does that mean exactly: Why is the packet
hitting CPU if it is being forwarded in hardware?

_______________________________________________
foundry-nsp mailing list
foundry-nsp@puck.nether.net
http://puck.nether.net/mailman/listinfo/foundry-nsp

Re: Layer 2 packet forwarded in hardware(PP) [ In reply to ]

ekoyle+puck.nether.net at gmail

Sep 13, 2018, 7:23 AM

Post #5 of 6 (1955 views)

Permalink

Have you tried running dm pstat? It can sometimes help identify the type
of traffic causing issues. First run is a throwaway, it shows counts since
the previous run.

I think dynamic arp inspection can still run without any L3 configured. It
is also possible that you have some kind of hardware entry that is
flip-flopping (duplicate MAC, some kind of multicast, etc.) or a lot of
broadcast/unknown unicast traffic. Also, multicast is treated like
broadcast if you don't enable igmp snooping, which will eat CPU.

It would take an insane amount of ARP traffic to generate that much load on
the lp cpu, so that may be a red herring.

--
Eldon

On Thu, Sep 13, 2018, 05:58 Franz Georg Köhler <lists@openunix.de> wrote:

> On Mo, Sep 10, 2018 at 08:22:19 -0600, Eldon Koyle <
> ekoyle+puck.nether.net@gmail.com> wrote:
> > You can enable cpu-protection on the vlan IIRC, I don't remember all the
> > caveats; definitely look at the manual before enabling.
>
> Also with CPU protection enabled on the VLAN I see packets hitting the
> CPU with "reason: Layer 2 packet forwarded in hardware(PP)".
>
> Therefore, I still wonder what does that mean exactly: Why is the packet
> hitting CPU if it is being forwarded in hardware?
>
>
>
> _______________________________________________
> foundry-nsp mailing list
> foundry-nsp@puck.nether.net
> http://puck.nether.net/mailman/listinfo/foundry-nsp
>

Re: Layer 2 packet forwarded in hardware(PP) [ In reply to ]

lists at openunix

Sep 14, 2018, 1:50 AM

Post #6 of 6 (1953 views)

Permalink

On Do, Sep 13, 2018 at 08:23:06 -0600, Eldon Koyle <ekoyle+puck.nether.net@gmail.com> wrote:
> Have you tried running dm pstat? It can sometimes help identify the type
> of traffic causing issues. First run is a throwaway, it shows counts since
> the previous run.

Thank you, Eldon.
I think this is a hint into the right direction, it looks like a lot of
mac learning is taking place:

MODULE 2
----------

Total Packets Received: 8696

MPLS uplink packets received: 0
VPLS packets received: 0
VLL packets received: 0
L3 VPN packets received: 0
Other MPLS packets received: 0

ARP packets received: 2139
ARP request packets received: 2107
ARP response packets received: 2107

IPV4 packets received: 234
IPv4 unicast packets routed: 0
IPv4 protocol packets received: 121
GRE tunnel packets received: 0
6to4 tunnel packets received: 0

IPV6 packets received: 56
IPv6 unicast packets routed: 0
IPv6 protocol packets received: 29

IPv4 multicast packets routed: 0
IPv6 multicast packets routed: 0

L2VPN endpoint packets received: 0
VPLS endpoint packets received: 0
VLL endpoint packets received: 0
Local-VLL endpoint packets received: 0

L2 packets received: 7705
L2 known unicast packets forwarded: 0
L2 unknown unicast packets flooded: 1002
L2 broadcast Packets flooded: 2147
L2 multicast Packets flooded: 129
Packets received for SA learning: 4721

Other packets received: 0

Total Packets dropped: 4937

Packet drop causes:
1 ( 0-Unknown cause)
4179 ( 1-Layer 2 packet forwarded in hardware(PP))
669 ( 9-Packet received for SFLOW sampling(PP))
13 (45-IP TTL too small(PE))
5 (53-IPv4 protocol drop(PFE))
37 (54-IP no route(PFE))
17 (56-Layer 3 invalid FID(PFE))
14 (57-Ipv6 protocol drop(PFE))
2 (67-DA learned on source port(OD))

ARP packets captured for DAI: 2139
ARP packets failed DAI: 0

Debugging says a lot of those lines:

# debug mac learning vlan 3595
learning: debugging is on
Sep 14 10:09:25.761 info - macmgr_ipc_learn_da_entry. Learn DA 000c.2939.c197 IPC received
Sep 14 10:09:25.761 info - macmgr_ipc_learn_da_entry. Learn DA 000c.2939.c197 IPC received
Sep 14 10:09:25.761 info - macmgr_ipc_learn_da_entry. Learn DA 000c.2939.c197 IPC received
Sep 14 10:09:25.761 info - macmgr_ipc_learn_da_entry. Learn DA 000c.2939.c197 IPC received

However, we do not know that mac:

#sh mac-address 000c.2939.c197
Total active entries from all ports = 3089
Type Code - ST:Static SEC:Secure 1x:Dot1x NA: NotAvail A:Allow D:Deny
CCL: Cluster Client Local CCR:Cluster Client Remote CL:Local CR:Remote

It looks like the system cannot learn the DA 000c 2939 c197:

[ppcr_rx_packet]: Packet received
Time stamp : 31 day(s) 18h 37m 42s:,
TM Header: [ 00be 0196 0013 ]
Type: Fabric Unicast(0x00000000) Size: 190 Class: 0 Src sys port: 406
Dest Port: 0 Drop Prec: 0 Ing Q Sig: 1 Out mirr dis: 0x0 Excl src: 1 Sys mc: 1
**********************************************************************
Packet size: 182, XPP reason code: 0x00000000
00: a054 2003 3850 0e0b-0020 008a 0000 0000 FID = 0xa054
10: 000c 2939 c197 000c-29f5 951b 0800 4500 Offset = 0x10
20: 0306 b617 4000 4011-5981 0a0a 0a1b 0a0a VLAN = 3595(0x0e0b)
30: 0a20 eb71 08af 02f2-b549 7b22 5365 7665 CAM = 0x100045(R)
40: 7269 7479 2220 3a20-2269 6e66 6f22 2c22 SFLOW = 0
50: 5461 6722 203a 2022-5c75 3030 3533 5c75 DBL TAG = 0
60: 3030 3635 5c75 3030-3732 5c75 3030 3736
70: 5c75 3030 3635 5c75-3030 3732 222c 2253
Pri CPU MON SRC PType US BRD DAV SAV DPV SV ER TXA SAS Tag MVID
0 1 0 1/1 3 0 0 1 1 1 0 0 0 0 1 0

10.10.10.27 -> 10.10.10.32 UDP [60273 -> 2223]
**********************************************************************
[ppcr_tx_packet] ACTION: Drop packet(reason: Layer 2 packet forwarded in hardware(PP))

I have know set "unknown-unicast-mac-entry" in the VLAN configuration
and this caused LP CPU to drop back to normal values because those
packets are not hitting the CPU any more. From the manual:
? Using the unknown-unicast-mac-entry command will forward Layer 2
unknown unicast traffic without going to the CPU.

The unknown-unicast-mac-entry command must be configured with the
vlan-cpu-protection command, as shown in the example above.

I guess the mac address in question is not in use and traffic towards
that address gets always flooded across the network.
Message "Layer 2 packet forwarded in hardware(PP)" still sounds a bit
bizarre to me. Maybe this means that packet is being forwarded in
hardware after it has been processed by CPU for learning?

Best regards,

Franz Georg
_______________________________________________
foundry-nsp mailing list
foundry-nsp@puck.nether.net
http://puck.nether.net/mailman/listinfo/foundry-nsp