Mailing List Archive

NCS-5501 - EVPN L2VPN BVI mac-address weirdness
Hi folks,

Wondering if anybody came across an issue with NCS-5501 models doing
EVPN Active/Active with Anycast IRB gateways. We have a multitude of
BVI's, with L2VPN and EVPN configured between a pair ov NCSs and using
a ESI for Bundle-Ether40 connected to downstream Nexus's.

The config i'm using below:

<config>
interface Bundle-Ether40.1017 l2transport
description CUSTOMER: TEST-BVI-MAC-ISSUE - 001
encapsulation dot1q 1017
rewrite ingress tag pop 1 symmetric


evpn
evi 11017
description CUSTOMER: TEST-BVI-MAC-ISSUE - 001
control-word-disable
advertise-mac


l2vpn
bridge group GRP-000111-00
bridge-domain BD--000111-00
description CUSTOMER: TEST-BVI-MAC-ISSUE - 001
interface Bundle-Ether40.1017
routed interface BVI1017
evi 11017


interface BVI101017
description CUSTOMER: TEST-BVI-MAC-ISSUE - 001
vrf TEST-BVI
ipv4 address 172.31.175.1 255.255.255.0
mac-address 0000.ff00.ffaa
</config>

The behaviour we're seeing and causing us some grief, is that whilst
on our Nexus's we see the mac address of 0000.ff00.ffaa in vlan 1017,
we're also seeing a second generate mac address 9ce1.7685.2000 closely
resembling that of NCS #1's Bundle-Ether40's BIA of 9ce1.7685.24df.
This behaviour seems to be causing fluctuations with reachability to
some applications, so far most notable are citrix orientated apps.
From our packet captures, the netscalers fluctuates between knowing
the NCS's as the gateway of 172.31.175.1 with 0000.ff00.ffaa
mac-address and then randomly gets the mac-address as 9ce1.7685.2000
instead. Regardless which mac-address we configure on the BVI, the
fluctuations keeps happening.

When we change the mac address to 9ce1.7685.2000 on the BVI of both
NCSs, the issue appears to resolve itself and works, except that a
failure introduced on the 1st NCS breaks the connectivity and we see
the same happening with the 2nd NCS's BE40 of 7c31.0e21.5cdf and some
mac address of 7c31.0e21.5000.....fluctuating between that mac address
and 9ce1.7685.2000

Our Cisco tac guy seems to be scratching his head as well.. Any
thoughts or experiences like this?

Cheers,
_______________________________________________
cisco-nsp mailing list cisco-nsp@puck.nether.net
https://puck.nether.net/mailman/listinfo/cisco-nsp
archive at http://puck.nether.net/pipermail/cisco-nsp/
Re: NCS-5501 - EVPN L2VPN BVI mac-address weirdness [ In reply to ]
Hi Drikus,

Did you ever resolve this?

We saw issues with MAC addresses on NCS55K too, but not related to EVPN.

For example, one can use the commands 'interface foo; mac xxxx.xxxx.xxxx" to set a custom MAC on a physical interface, XR commits the config but on these Broadcom chips it doesn't actually do anything. CLI output shows the custom MAC but a packet capture shows the BIA.

We had another issue with the MAC address of CDP/LLDP frames addresses on bundle interface (I can't remember the exact details right now, but I think the source MAC address of these frames sometimes had the logical bundle MAC and sometimes had the member link MAC, and it was inconsistent).

So it seems these chips have issues with MAC address consistency. I'm wondering if there is some relation to what you are seeing.

Cheers,
James.
_______________________________________________
cisco-nsp mailing list cisco-nsp@puck.nether.net
https://puck.nether.net/mailman/listinfo/cisco-nsp
archive at http://puck.nether.net/pipermail/cisco-nsp/
Re: NCS-5501 - EVPN L2VPN BVI mac-address weirdness [ In reply to ]
James Bensley wrote on 21/12/2021 07:26:
> For example, one can use the commands 'interface foo; mac
> xxxx.xxxx.xxxx" to set a custom MAC on a physical interface, XR
> commits the config but on these Broadcom chips it doesn't actually do
> anything. CLI output shows the custom MAC but a packet capture shows
> the BIA.
also, ncs5k will occasionally change the mac address on a BE interface
after upgrade from 6.x to 7.x.

Nick
_______________________________________________
cisco-nsp mailing list cisco-nsp@puck.nether.net
https://puck.nether.net/mailman/listinfo/cisco-nsp
archive at http://puck.nether.net/pipermail/cisco-nsp/
Re: NCS-5501 - EVPN L2VPN BVI mac-address weirdness [ In reply to ]
Heya James,

We still have a TAC case open with Cisco, they seem to be scratching
their heads as to why this is happening, the usual rigmarole, but did
provide 2x bug id's which seems to relate -- CSCvs91974 & CSCvk45012.
Right now we've seen the issue mainly affected by some of our
customers using mac-based forwarding techniques used by vendors, with
Citrix being our main pain at the moment. We're connecting up a couple
of Fortigates to the NCSs soon as well, perhaps i'll disable LLDP just
to be safe, thanks on that.

For the most part, Cisco and XR BU seems to be interested in resolving
it, though they want explanations as to why our customers need to use
mac-based forwarding in their solutions..

I'll reply once i have some concrete info from them.....


On Tue, Dec 21, 2021 at 6:26 PM James Bensley
<jwbensley+cisco-nsp@gmail.com> wrote:
>
> Hi Drikus,
>
> Did you ever resolve this?
>
> We saw issues with MAC addresses on NCS55K too, but not related to EVPN.
>
> For example, one can use the commands 'interface foo; mac xxxx.xxxx.xxxx" to set a custom MAC on a physical interface, XR commits the config but on these Broadcom chips it doesn't actually do anything. CLI output shows the custom MAC but a packet capture shows the BIA.
>
> We had another issue with the MAC address of CDP/LLDP frames addresses on bundle interface (I can't remember the exact details right now, but I think the source MAC address of these frames sometimes had the logical bundle MAC and sometimes had the member link MAC, and it was inconsistent).
>
> So it seems these chips have issues with MAC address consistency. I'm wondering if there is some relation to what you are seeing.
>
> Cheers,
> James.
_______________________________________________
cisco-nsp mailing list cisco-nsp@puck.nether.net
https://puck.nether.net/mailman/listinfo/cisco-nsp
archive at http://puck.nether.net/pipermail/cisco-nsp/
Re: NCS-5501 - EVPN L2VPN BVI mac-address weirdness [ In reply to ]
On 12/21/21 09:26, James Bensley wrote:

> Hi Drikus,
>
> Did you ever resolve this?
>
> We saw issues with MAC addresses on NCS55K too, but not related to EVPN.
>
> For example, one can use the commands 'interface foo; mac xxxx.xxxx.xxxx" to set a custom MAC on a physical interface, XR commits the config but on these Broadcom chips it doesn't actually do anything. CLI output shows the custom MAC but a packet capture shows the BIA.
>
> We had another issue with the MAC address of CDP/LLDP frames addresses on bundle interface (I can't remember the exact details right now, but I think the source MAC address of these frames sometimes had the logical bundle MAC and sometimes had the member link MAC, and it was inconsistent).
>
> So it seems these chips have issues with MAC address consistency. I'm wondering if there is some relation to what you are seeing.

The joys of merchant silicon - completely unpredictable; especially on
platforms that you are used to being so.

Given the rising(?) cost of hardware, and the declining revenue from
customers, merchant silicon is only going to become more and more
relevant. We can't still be playing these games at this stage in the game.

I have lowered my guard, a little, and am now willing to test some boxes
from traditional vendors (Juniper + Nokia) that are shipping on the back
of Broadcom J2 chips. But I've already had to dump Nokia because they
say they won't support a chip-restricted feature which Juniper claim
they will.

I don't know who's lying, or telling the half-truth.

So much confusion, in this space.

Mark.
_______________________________________________
cisco-nsp mailing list cisco-nsp@puck.nether.net
https://puck.nether.net/mailman/listinfo/cisco-nsp
archive at http://puck.nether.net/pipermail/cisco-nsp/
Re: NCS-5501 - EVPN L2VPN BVI mac-address weirdness [ In reply to ]
hey,

> But I've already had to dump Nokia because they say they won't support a
> chip-restricted feature which Juniper claim they will.

Which feature?

Vendors, including Nokia, have worked around BCM limitations before by
playing tricks like recirculating packet twice (disabling some of the
frontplate) for e-tree, some multicast stuff, packet mirroring, OAM
loopback etc.

--
tarko
_______________________________________________
cisco-nsp mailing list cisco-nsp@puck.nether.net
https://puck.nether.net/mailman/listinfo/cisco-nsp
archive at http://puck.nether.net/pipermail/cisco-nsp/
Re: NCS-5501 - EVPN L2VPN BVI mac-address weirdness [ In reply to ]
On 12/23/21 13:41, Tarko Tikan wrote:

>
> Which feature?

Egress policing.

Mark.
_______________________________________________
cisco-nsp mailing list cisco-nsp@puck.nether.net
https://puck.nether.net/mailman/listinfo/cisco-nsp
archive at http://puck.nether.net/pipermail/cisco-nsp/