Mailing List Archive

MX VRRP on VXLAN enviroment
I have a scenario here where I use EVPN-VXLAN with qfx5120 switches
and until then I was using the gateways on the switches, but as the
switch does not have the possibility to use any kind of firewall on
the irb interfaces, I had the idea to migrate the networks to two
routers MX80.
But I caught a problem with these routers, when using VRRP over VXLAN.
I configured the two MX80 routers with VRRP in IPv4 and IPv6,
sometimes IPv4 dies and the virtual IP stops responding, generating
timeout in network accesses.
Apparently it seems that the mac address of the virtual IP and the
table of mac's of the VXLAN are lost, causing the problem.
Does anyone happen to have a scenario like this and faced this problem?
_______________________________________________
juniper-nsp mailing list juniper-nsp@puck.nether.net
https://puck.nether.net/mailman/listinfo/juniper-nsp
Re: MX VRRP on VXLAN enviroment [ In reply to ]
Hi,

> On 20/07/2021, at 12:23 AM, Cristian Cardoso via juniper-nsp <juniper-nsp@puck.nether.net> wrote:
>
> I have a scenario here where I use EVPN-VXLAN with qfx5120 switches
> and until then I was using the gateways on the switches, but as the
> switch does not have the possibility to use any kind of firewall on
> the irb interfaces, I had the idea to migrate the networks to two
> routers MX80.
> But I caught a problem with these routers, when using VRRP over VXLAN.
> I configured the two MX80 routers with VRRP in IPv4 and IPv6,
> sometimes IPv4 dies and the virtual IP stops responding, generating
> timeout in network accesses.
> Apparently it seems that the mac address of the virtual IP and the
> table of mac's of the VXLAN are lost, causing the problem.
> Does anyone happen to have a scenario like this and faced this problem?

I’m not sure exactly what is causing your problem, though there are certainly a lot of curly edge cases in EVPN where this sort of thing can happen.
Do you have a specific need to run VRRP?

One of the benefits of EVPN is “virtual-gateway-address” which advertises the gateway address from all IRBs with that configured - and they are all “active” rather than VRRP’s active/standby.
If you don’t have a need for VRRP specifically, this might be a better solution for you.

Incidentally, by default it uses the VRRP group 1 MAC, but it is not running VRRP at all. You can configure it to use a different MAC, if required (i.e. if you have other devices on that broadcast domain running VRRP group 1 perhaps).

--
Nathan Ward

_______________________________________________
juniper-nsp mailing list juniper-nsp@puck.nether.net
https://puck.nether.net/mailman/listinfo/juniper-nsp
Re: MX VRRP on VXLAN enviroment [ In reply to ]
I had several problems using the virtual gateway via EVPN on the
switches, even the function of being switches and not routers. In my
scenario it is important to have a minimal firewall on the interfaces,
and in the models I have here, this is not possible.
My idea of using VRRP on the routers above the VXLAN switches, is just
to let the switches do only the VXLAN and thus remove the functions of
the layer 3 gateway from the qfx.
Using a more "simple" Layer 3 scenario for routing.

Em seg., 19 de jul. de 2021 às 09:41, Nathan Ward
<juniper-nsp@daork.net> escreveu:
>
> Hi,
>
> > On 20/07/2021, at 12:23 AM, Cristian Cardoso via juniper-nsp <juniper-nsp@puck.nether.net> wrote:
> >
> > I have a scenario here where I use EVPN-VXLAN with qfx5120 switches
> > and until then I was using the gateways on the switches, but as the
> > switch does not have the possibility to use any kind of firewall on
> > the irb interfaces, I had the idea to migrate the networks to two
> > routers MX80.
> > But I caught a problem with these routers, when using VRRP over VXLAN.
> > I configured the two MX80 routers with VRRP in IPv4 and IPv6,
> > sometimes IPv4 dies and the virtual IP stops responding, generating
> > timeout in network accesses.
> > Apparently it seems that the mac address of the virtual IP and the
> > table of mac's of the VXLAN are lost, causing the problem.
> > Does anyone happen to have a scenario like this and faced this problem?
>
> I’m not sure exactly what is causing your problem, though there are certainly a lot of curly edge cases in EVPN where this sort of thing can happen.
> Do you have a specific need to run VRRP?
>
> One of the benefits of EVPN is “virtual-gateway-address” which advertises the gateway address from all IRBs with that configured - and they are all “active” rather than VRRP’s active/standby.
> If you don’t have a need for VRRP specifically, this might be a better solution for you.
>
> Incidentally, by default it uses the VRRP group 1 MAC, but it is not running VRRP at all. You can configure it to use a different MAC, if required (i.e. if you have other devices on that broadcast domain running VRRP group 1 perhaps).
>
> --
> Nathan Ward
>
_______________________________________________
juniper-nsp mailing list juniper-nsp@puck.nether.net
https://puck.nether.net/mailman/listinfo/juniper-nsp
Re: MX VRRP on VXLAN enviroment [ In reply to ]
In theory when the VRRP falls over, the promoted MX80 should send a
frame with the VRRP virtual MAC as source, causing it to be learnt on a
different switch.  That switch should then send an EVPN type-2 UPDATE
for the MAC and the remaining switches will update their local MAC
tables with VXLAN-encap next-hop.  Can you verify any of that?  Like
does the MAC get learnt from the other MX80 properly?  Does the EVPN
update get generated?


On 19/07/2021 13:56, Cristian Cardoso via juniper-nsp wrote:
> I had several problems using the virtual gateway via EVPN on the
> switches, even the function of being switches and not routers. In my
> scenario it is important to have a minimal firewall on the interfaces,
> and in the models I have here, this is not possible.
> My idea of using VRRP on the routers above the VXLAN switches, is just
> to let the switches do only the VXLAN and thus remove the functions of
> the layer 3 gateway from the qfx.
> Using a more "simple" Layer 3 scenario for routing.
>
> Em seg., 19 de jul. de 2021 às 09:41, Nathan Ward
> <juniper-nsp@daork.net> escreveu:
>> Hi,
>>
>>> On 20/07/2021, at 12:23 AM, Cristian Cardoso via juniper-nsp <juniper-nsp@puck.nether.net> wrote:
>>>
>>> I have a scenario here where I use EVPN-VXLAN with qfx5120 switches
>>> and until then I was using the gateways on the switches, but as the
>>> switch does not have the possibility to use any kind of firewall on
>>> the irb interfaces, I had the idea to migrate the networks to two
>>> routers MX80.
>>> But I caught a problem with these routers, when using VRRP over VXLAN.
>>> I configured the two MX80 routers with VRRP in IPv4 and IPv6,
>>> sometimes IPv4 dies and the virtual IP stops responding, generating
>>> timeout in network accesses.
>>> Apparently it seems that the mac address of the virtual IP and the
>>> table of mac's of the VXLAN are lost, causing the problem.
>>> Does anyone happen to have a scenario like this and faced this problem?
>> I’m not sure exactly what is causing your problem, though there are certainly a lot of curly edge cases in EVPN where this sort of thing can happen.
>> Do you have a specific need to run VRRP?
>>
>> One of the benefits of EVPN is “virtual-gateway-address” which advertises the gateway address from all IRBs with that configured - and they are all “active” rather than VRRP’s active/standby.
>> If you don’t have a need for VRRP specifically, this might be a better solution for you.
>>
>> Incidentally, by default it uses the VRRP group 1 MAC, but it is not running VRRP at all. You can configure it to use a different MAC, if required (i.e. if you have other devices on that broadcast domain running VRRP group 1 perhaps).
>>
>> --
>> Nathan Ward
>>
> _______________________________________________
> juniper-nsp mailing list juniper-nsp@puck.nether.net
> https://puck.nether.net/mailman/listinfo/juniper-nsp
_______________________________________________
juniper-nsp mailing list juniper-nsp@puck.nether.net
https://puck.nether.net/mailman/listinfo/juniper-nsp
Re: MX VRRP on VXLAN enviroment [ In reply to ]
Take a look on this KB
https://kb.juniper.net/InfoCenter/index?page=content&id=KB35676&cat=EX9208&actp=LIST

The default duplicate-mac-detection settings are far to high

Nitzan

On Mon, Jul 19, 2021 at 4:50 PM Cathal Mooney via juniper-nsp <
juniper-nsp@puck.nether.net> wrote:

> In theory when the VRRP falls over, the promoted MX80 should send a
> frame with the VRRP virtual MAC as source, causing it to be learnt on a
> different switch. That switch should then send an EVPN type-2 UPDATE
> for the MAC and the remaining switches will update their local MAC
> tables with VXLAN-encap next-hop. Can you verify any of that? Like
> does the MAC get learnt from the other MX80 properly? Does the EVPN
> update get generated?
>
>
> On 19/07/2021 13:56, Cristian Cardoso via juniper-nsp wrote:
> > I had several problems using the virtual gateway via EVPN on the
> > switches, even the function of being switches and not routers. In my
> > scenario it is important to have a minimal firewall on the interfaces,
> > and in the models I have here, this is not possible.
> > My idea of using VRRP on the routers above the VXLAN switches, is just
> > to let the switches do only the VXLAN and thus remove the functions of
> > the layer 3 gateway from the qfx.
> > Using a more "simple" Layer 3 scenario for routing.
> >
> > Em seg., 19 de jul. de 2021 às 09:41, Nathan Ward
> > <juniper-nsp@daork.net> escreveu:
> >> Hi,
> >>
> >>> On 20/07/2021, at 12:23 AM, Cristian Cardoso via juniper-nsp <
> juniper-nsp@puck.nether.net> wrote:
> >>>
> >>> I have a scenario here where I use EVPN-VXLAN with qfx5120 switches
> >>> and until then I was using the gateways on the switches, but as the
> >>> switch does not have the possibility to use any kind of firewall on
> >>> the irb interfaces, I had the idea to migrate the networks to two
> >>> routers MX80.
> >>> But I caught a problem with these routers, when using VRRP over VXLAN.
> >>> I configured the two MX80 routers with VRRP in IPv4 and IPv6,
> >>> sometimes IPv4 dies and the virtual IP stops responding, generating
> >>> timeout in network accesses.
> >>> Apparently it seems that the mac address of the virtual IP and the
> >>> table of mac's of the VXLAN are lost, causing the problem.
> >>> Does anyone happen to have a scenario like this and faced this problem?
> >> I’m not sure exactly what is causing your problem, though there are
> certainly a lot of curly edge cases in EVPN where this sort of thing can
> happen.
> >> Do you have a specific need to run VRRP?
> >>
> >> One of the benefits of EVPN is “virtual-gateway-address” which
> advertises the gateway address from all IRBs with that configured - and
> they are all “active” rather than VRRP’s active/standby.
> >> If you don’t have a need for VRRP specifically, this might be a better
> solution for you.
> >>
> >> Incidentally, by default it uses the VRRP group 1 MAC, but it is not
> running VRRP at all. You can configure it to use a different MAC, if
> required (i.e. if you have other devices on that broadcast domain running
> VRRP group 1 perhaps).
> >>
> >> --
> >> Nathan Ward
> >>
> > _______________________________________________
> > juniper-nsp mailing list juniper-nsp@puck.nether.net
> > https://puck.nether.net/mailman/listinfo/juniper-nsp
> _______________________________________________
> juniper-nsp mailing list juniper-nsp@puck.nether.net
> https://puck.nether.net/mailman/listinfo/juniper-nsp
>
_______________________________________________
juniper-nsp mailing list juniper-nsp@puck.nether.net
https://puck.nether.net/mailman/listinfo/juniper-nsp
Re: MX VRRP on VXLAN enviroment [ In reply to ]
Hi
Thanks for the tip, I'll set it up here.

Em seg., 19 de jul. de 2021 às 14:36, Nitzan Tzelniker via juniper-nsp
<juniper-nsp@puck.nether.net> escreveu:
>
> Take a look on this KB
> https://kb.juniper.net/InfoCenter/index?page=content&id=KB35676&cat=EX9208&actp=LIST
>
> The default duplicate-mac-detection settings are far to high
>
> Nitzan
>
> On Mon, Jul 19, 2021 at 4:50 PM Cathal Mooney via juniper-nsp <
> juniper-nsp@puck.nether.net> wrote:
>
> > In theory when the VRRP falls over, the promoted MX80 should send a
> > frame with the VRRP virtual MAC as source, causing it to be learnt on a
> > different switch. That switch should then send an EVPN type-2 UPDATE
> > for the MAC and the remaining switches will update their local MAC
> > tables with VXLAN-encap next-hop. Can you verify any of that? Like
> > does the MAC get learnt from the other MX80 properly? Does the EVPN
> > update get generated?
> >
> >
> > On 19/07/2021 13:56, Cristian Cardoso via juniper-nsp wrote:
> > > I had several problems using the virtual gateway via EVPN on the
> > > switches, even the function of being switches and not routers. In my
> > > scenario it is important to have a minimal firewall on the interfaces,
> > > and in the models I have here, this is not possible.
> > > My idea of using VRRP on the routers above the VXLAN switches, is just
> > > to let the switches do only the VXLAN and thus remove the functions of
> > > the layer 3 gateway from the qfx.
> > > Using a more "simple" Layer 3 scenario for routing.
> > >
> > > Em seg., 19 de jul. de 2021 às 09:41, Nathan Ward
> > > <juniper-nsp@daork.net> escreveu:
> > >> Hi,
> > >>
> > >>> On 20/07/2021, at 12:23 AM, Cristian Cardoso via juniper-nsp <
> > juniper-nsp@puck.nether.net> wrote:
> > >>>
> > >>> I have a scenario here where I use EVPN-VXLAN with qfx5120 switches
> > >>> and until then I was using the gateways on the switches, but as the
> > >>> switch does not have the possibility to use any kind of firewall on
> > >>> the irb interfaces, I had the idea to migrate the networks to two
> > >>> routers MX80.
> > >>> But I caught a problem with these routers, when using VRRP over VXLAN.
> > >>> I configured the two MX80 routers with VRRP in IPv4 and IPv6,
> > >>> sometimes IPv4 dies and the virtual IP stops responding, generating
> > >>> timeout in network accesses.
> > >>> Apparently it seems that the mac address of the virtual IP and the
> > >>> table of mac's of the VXLAN are lost, causing the problem.
> > >>> Does anyone happen to have a scenario like this and faced this problem?
> > >> I’m not sure exactly what is causing your problem, though there are
> > certainly a lot of curly edge cases in EVPN where this sort of thing can
> > happen.
> > >> Do you have a specific need to run VRRP?
> > >>
> > >> One of the benefits of EVPN is “virtual-gateway-address” which
> > advertises the gateway address from all IRBs with that configured - and
> > they are all “active” rather than VRRP’s active/standby.
> > >> If you don’t have a need for VRRP specifically, this might be a better
> > solution for you.
> > >>
> > >> Incidentally, by default it uses the VRRP group 1 MAC, but it is not
> > running VRRP at all. You can configure it to use a different MAC, if
> > required (i.e. if you have other devices on that broadcast domain running
> > VRRP group 1 perhaps).
> > >>
> > >> --
> > >> Nathan Ward
> > >>
> > > _______________________________________________
> > > juniper-nsp mailing list juniper-nsp@puck.nether.net
> > > https://puck.nether.net/mailman/listinfo/juniper-nsp
> > _______________________________________________
> > juniper-nsp mailing list juniper-nsp@puck.nether.net
> > https://puck.nether.net/mailman/listinfo/juniper-nsp
> >
> _______________________________________________
> juniper-nsp mailing list juniper-nsp@puck.nether.net
> https://puck.nether.net/mailman/listinfo/juniper-nsp
_______________________________________________
juniper-nsp mailing list juniper-nsp@puck.nether.net
https://puck.nether.net/mailman/listinfo/juniper-nsp
Re: MX VRRP on VXLAN enviroment [ In reply to ]
Cristian Cardoso via juniper-nsp ????? 2021-07-19 14:15:
> Hi
> Thanks for the tip, I'll set it up here.
>

Are you trying to setup MX80 as end-host, without including it in EVPN?
If so, then you can extend EVPN to MX80 and run virtual-gateway from it.
No need for VRRP in this case.

Kind regards,
Andrey
_______________________________________________
juniper-nsp mailing list juniper-nsp@puck.nether.net
https://puck.nether.net/mailman/listinfo/juniper-nsp
Re: MX VRRP on VXLAN enviroment [ In reply to ]
Hi Andrey

My idea was to keep only with VRRP to be something simpler for the
team to manage.
I set the config that Nitzan Tzelniker suggested and so far I haven't
seen more occurrences of arp suppression problems in VXLAN.
Of course, the last time this happened it took about 2 weeks to end up
happening.
I did several reboots between one MX80 and another and so far there
was no problem with the gateway dying for the network that is in VRRP.

Em sex., 23 de jul. de 2021 às 14:34, Andrey Kostin
<ankost@podolsk.ru> escreveu:
>
> Cristian Cardoso via juniper-nsp ????? 2021-07-19 14:15:
> > Hi
> > Thanks for the tip, I'll set it up here.
> >
>
> Are you trying to setup MX80 as end-host, without including it in EVPN?
> If so, then you can extend EVPN to MX80 and run virtual-gateway from it.
> No need for VRRP in this case.
>
> Kind regards,
> Andrey
_______________________________________________
juniper-nsp mailing list juniper-nsp@puck.nether.net
https://puck.nether.net/mailman/listinfo/juniper-nsp
Re: MX VRRP on VXLAN enviroment [ In reply to ]
Hi Andrey

In my case, what you said happened, as I modified the arp suppression
configuration of evpn-vxlan, since this was silently dropping mac's and
dropping VRRPv4 only, in IPv6 this did not happen.

set protocols evpn duplicate-mac-detection detection-threshold 20
set protocols evpn duplicate-mac-detection detection-window 5
set protocols evpn duplicate-mac-detection auto-recovery-time 5

With the above configurations, I never had a problem with VRRPv4 crashing
in my environment.
Environment with VRRP is already working since the email that responds in
2021 without any drop or problems.

kind regards,
Cristian Cardoso



Em qua., 14 de dez. de 2022 às 12:26, Andrey Kostin <ankost@podolsk.ru>
escreveu:

> Cristian Cardoso ?????(?) 2021-07-26 21:37:
> > Hi Andrey
> >
> > My idea was to keep only with VRRP to be something simpler for the
> > team to manage.
> > I set the config that Nitzan Tzelniker suggested and so far I haven't
> > seen more occurrences of arp suppression problems in VXLAN.
> > Of course, the last time this happened it took about 2 weeks to end up
> > happening.
> > I did several reboots between one MX80 and another and so far there
> > was no problem with the gateway dying for the network that is in VRRP.
> >
>
> Hi Cristian,
>
> Late reply, you probably already solved your problem, but I faced the
> same issue and I remembered that before I saw that connecting VXLAN
> switches to a traditional switches with connected VRRP hosts broke VRRP
> for them.
> The issue is with VRRPv4 MAC that is by default used as virtual-gateway
> MAC in EVPN/VXLAN. Looks like Broadcom switches process it even if L3
> gateway isn't configured. In my case the symptoms were exactly like
> yours, some intermittent connectivity issues. I suspect that VRRP MAC is
> processed by switches in some cases, probably when packets are sent
> between switches via VXLAN tunnel. For now I just disabled VRRPv4 and
> seems it helped. Going to test using manually configured MAC for VRRPv4.
>
> Kind regards,
> Andrey
>
_______________________________________________
juniper-nsp mailing list juniper-nsp@puck.nether.net
https://puck.nether.net/mailman/listinfo/juniper-nsp
Re: MX VRRP on VXLAN enviroment [ In reply to ]
Hi Cristian,

I tried to reproduce the issue by reverting the configuration, but it
didn't occur. It's still unclear to me why only v4 was affected and v6
was not. Furthermore, in stable state (which it was in my case) vrrp
backup is silent so no mac move events can happen and I confirmed it
with vrrp statistics that I collected before disabling VRRP, backup
router wasn't sending anything.
After re-activating the interface on backup router I also didn't see any
vrrp packet sent from it. Eventually I ended up with configuring
exception for VRRP MACs and will watch how it goes:

https://www.juniper.net/documentation/us/en/software/junos/evpn-vxlan/multicast-l2/topics/ref/statement/exclusive-mac-edit-protocols-l2-learning-global-mac-move.html

Kind regards,
Andrey

Cristian Cardoso ?????(?) 2022-12-14 11:20:
> Hi Andrey
>
> In my case, what you said happened, as I modified the arp suppression
> configuration of evpn-vxlan, since this was silently dropping mac's
> and dropping VRRPv4 only, in IPv6 this did not happen.
>
> set protocols evpn duplicate-mac-detection detection-threshold 20
> set protocols evpn duplicate-mac-detection detection-window 5
> set protocols evpn duplicate-mac-detection auto-recovery-time 5
>
> With the above configurations, I never had a problem with VRRPv4
> crashing in my environment.
>
> Environment with VRRP is already working since the email that responds
> in 2021 without any drop or problems.
>
> kind regards,
>
> Cristian Cardoso
>
_______________________________________________
juniper-nsp mailing list juniper-nsp@puck.nether.net
https://puck.nether.net/mailman/listinfo/juniper-nsp
Re: MX VRRP on VXLAN enviroment [ In reply to ]
Hello Cristian
I once faced a similar issue and I came across this PR mentioned in this
KB. I think it’s worth checking.

https://supportportal.juniper.net/s/article/Junos-18-4R3-3-18-4R3-S1-3-18-4R3-S2-19-1R2-8-19-1R2-S1-1-19-2R2-is-not-recommended-to-be-deployed-on-MX-Series-platform-with-Trio-MPCs-for-EVPN-with-MPLS-VXLAN?language=en_US



Alert Description

With EVPN with MPLS/VXLAN environment, on an MX/EX Series router with
Trio-based MPCs, unicast traffic will be dropped when the destination is
reachable over an integrated routing and bridging (IRB) interface
participating in the EVPN instance. PR1497203 is tracking this issue.



On Mon, Jul 19, 2021 at 5:27 AM Cristian Cardoso via juniper-nsp <
juniper-nsp@puck.nether.net> wrote:

> I have a scenario here where I use EVPN-VXLAN with qfx5120 switches
> and until then I was using the gateways on the switches, but as the
> switch does not have the possibility to use any kind of firewall on
> the irb interfaces, I had the idea to migrate the networks to two
> routers MX80.
> But I caught a problem with these routers, when using VRRP over VXLAN.
> I configured the two MX80 routers with VRRP in IPv4 and IPv6,
> sometimes IPv4 dies and the virtual IP stops responding, generating
> timeout in network accesses.
> Apparently it seems that the mac address of the virtual IP and the
> table of mac's of the VXLAN are lost, causing the problem.
> Does anyone happen to have a scenario like this and faced this problem?
> _______________________________________________
> juniper-nsp mailing list juniper-nsp@puck.nether.net
> https://puck.nether.net/mailman/listinfo/juniper-nsp
>
--
Thanks, Aftab
_______________________________________________
juniper-nsp mailing list juniper-nsp@puck.nether.net
https://puck.nether.net/mailman/listinfo/juniper-nsp