Thank you Sebastian for sharing your very valuable experience.
Kind regards,
Andrey
Sebastian Wiesinger ????? 2019-03-22 04:39:
> * Andrey Kostin <ankost@podolsk.ru> [2019-03-15 20:50]:
>> I'm interested to hear about experience of running EVPN/VXLAN,
>> particularly
>> with QFX10k as L3 gateway and QFX5k as spine/leaves. As per docs, it
>> should
>> be immune to any single switch downtime, so might be a candidate to
>> really
>> redundant design.
>
> All right here it goes:
>
> I can't speak for QFX10k as spine but we have QFX5100 Leaf/Spine
> setups with EVPN/VXLAN running right now. Switch downtime is no
> problem at all, we unplugged a running switch, shut down ports,
> unplugged cables between leaf & spine or leaf & client all while there
> was storage traffic (NFS) active in the setup. Worst thing that
> happend was that IOPS went down from 400k/s to 100k/s for 1-3 seconds.
>
> What did bother us was that you are limited (at least on QFX5100) in
> the amount of "VLANs" (VNIs). We were testing with 30 client
> full-trunk ports per leaf and with that amount you can only provision
> around 500 VLANs before you get errors and basically it seems you run
> out of memory for bridge domains on the switch. This seems to be a
> limitation by the chips used in the QFX5100, at least that's what I
> got when I asked about it.
>
> You can check if you know where:
>
> root@SW-A:RE:0% ifsmon -Id | grep IFBD
> IFBD : 12884 0
>
> root@SW-A:RE:0% ifsmon -Id | grep Bridge
> Bridge Domain : 3502 0
>
> These numbers combined need to be <= 16382.
>
> And if you get over the limit these nice errors occur:
>
> dcf_ng_get_vxlan_ifbd_hw_token: Max vxlan ifbd hw token reached 16382
> ifbd_create_node: VXLAN IFBD hw token couldn't be allocated for
> <xe-...>
>
> Workaround is to decrease VLANs or trunk config.
>
> Also you absolutely NEED LACP from servers to the fabric. 17.4 has
> enhancements which will put the client ports in LACP standby when the
> leaf gets separated from all spines.
>
>> As a downside I see the more complex configuration at least. Adding
>> vlan means adding routing instance etc. There are also other
>> questions, about convergence, scalability, how stable it is and code
>> maturity.
>
> We have it automated with Ansible. Management access happens over OOB
> (Mgmt) ports and everything is pushed by Ansible playbooks. Ansible
> generates configuration from templates and pushes it to the switches
> via netconf. I never would want to do this by hand. This demands a
> certain level of structuring by every team (network, people doing the
> cabling, server team) but it works out well for structured setups.
>
> Our switch config looks like this:
>
> --------------------------------------------------------------------------
> user@sw1-spine-pod1> show configuration
> ## Last commit: 2019-03-11 03:13:49 CET by user
> ## Image name: jinstall-host-qfx-5-flex-17.4R2-S2.3-signed.tgz
>
> version 17.4R1-S3.3;
> groups {
> /* Created by Ansible */
> evpn-defaults { /* OMITTED */ };
> /* Created by Ansible */
> evpn-spine-defaults { /* OMITTED */ };
> /* Created by Ansible */
> evpn-spine-1 { /* OMITTED */ };
> /* Created by Ansible - Empty group for maintenance operations */
> client-interfaces;
> }
> apply-groups [ evpn-defaults evpn-spine-defaults evpn-spine-1 ];
> --------------------------------------------------------------------------
>
> So everything Ansible does is contained in apply-groups and is hidden.
> You can
> immediately spot if something is configured by hand.
>
> For code we're currently running on the 17.4 train which works mostly
> fine, we had a few problems with third party 40G optics but these
> should be fixed in the newest 17.4 service release.
>
> Also we had a problem where new Spine/Leaf links did not come up but
> these vanished after rebooting/upgrading the spines.
>
> In daily operations it proves to be quite stable.
>
>
> Best Regards
>
> Sebastian
_______________________________________________
juniper-nsp mailing list juniper-nsp@puck.nether.net
https://puck.nether.net/mailman/listinfo/juniper-nsp
Kind regards,
Andrey
Sebastian Wiesinger ????? 2019-03-22 04:39:
> * Andrey Kostin <ankost@podolsk.ru> [2019-03-15 20:50]:
>> I'm interested to hear about experience of running EVPN/VXLAN,
>> particularly
>> with QFX10k as L3 gateway and QFX5k as spine/leaves. As per docs, it
>> should
>> be immune to any single switch downtime, so might be a candidate to
>> really
>> redundant design.
>
> All right here it goes:
>
> I can't speak for QFX10k as spine but we have QFX5100 Leaf/Spine
> setups with EVPN/VXLAN running right now. Switch downtime is no
> problem at all, we unplugged a running switch, shut down ports,
> unplugged cables between leaf & spine or leaf & client all while there
> was storage traffic (NFS) active in the setup. Worst thing that
> happend was that IOPS went down from 400k/s to 100k/s for 1-3 seconds.
>
> What did bother us was that you are limited (at least on QFX5100) in
> the amount of "VLANs" (VNIs). We were testing with 30 client
> full-trunk ports per leaf and with that amount you can only provision
> around 500 VLANs before you get errors and basically it seems you run
> out of memory for bridge domains on the switch. This seems to be a
> limitation by the chips used in the QFX5100, at least that's what I
> got when I asked about it.
>
> You can check if you know where:
>
> root@SW-A:RE:0% ifsmon -Id | grep IFBD
> IFBD : 12884 0
>
> root@SW-A:RE:0% ifsmon -Id | grep Bridge
> Bridge Domain : 3502 0
>
> These numbers combined need to be <= 16382.
>
> And if you get over the limit these nice errors occur:
>
> dcf_ng_get_vxlan_ifbd_hw_token: Max vxlan ifbd hw token reached 16382
> ifbd_create_node: VXLAN IFBD hw token couldn't be allocated for
> <xe-...>
>
> Workaround is to decrease VLANs or trunk config.
>
> Also you absolutely NEED LACP from servers to the fabric. 17.4 has
> enhancements which will put the client ports in LACP standby when the
> leaf gets separated from all spines.
>
>> As a downside I see the more complex configuration at least. Adding
>> vlan means adding routing instance etc. There are also other
>> questions, about convergence, scalability, how stable it is and code
>> maturity.
>
> We have it automated with Ansible. Management access happens over OOB
> (Mgmt) ports and everything is pushed by Ansible playbooks. Ansible
> generates configuration from templates and pushes it to the switches
> via netconf. I never would want to do this by hand. This demands a
> certain level of structuring by every team (network, people doing the
> cabling, server team) but it works out well for structured setups.
>
> Our switch config looks like this:
>
> --------------------------------------------------------------------------
> user@sw1-spine-pod1> show configuration
> ## Last commit: 2019-03-11 03:13:49 CET by user
> ## Image name: jinstall-host-qfx-5-flex-17.4R2-S2.3-signed.tgz
>
> version 17.4R1-S3.3;
> groups {
> /* Created by Ansible */
> evpn-defaults { /* OMITTED */ };
> /* Created by Ansible */
> evpn-spine-defaults { /* OMITTED */ };
> /* Created by Ansible */
> evpn-spine-1 { /* OMITTED */ };
> /* Created by Ansible - Empty group for maintenance operations */
> client-interfaces;
> }
> apply-groups [ evpn-defaults evpn-spine-defaults evpn-spine-1 ];
> --------------------------------------------------------------------------
>
> So everything Ansible does is contained in apply-groups and is hidden.
> You can
> immediately spot if something is configured by hand.
>
> For code we're currently running on the 17.4 train which works mostly
> fine, we had a few problems with third party 40G optics but these
> should be fixed in the newest 17.4 service release.
>
> Also we had a problem where new Spine/Leaf links did not come up but
> these vanished after rebooting/upgrading the spines.
>
> In daily operations it proves to be quite stable.
>
>
> Best Regards
>
> Sebastian
_______________________________________________
juniper-nsp mailing list juniper-nsp@puck.nether.net
https://puck.nether.net/mailman/listinfo/juniper-nsp