Mailing List Archive

Cisco NCS VxLAN Experience
Hello everyone,

A customer of mine's interested in acquiring some NCS boxes, in order to
aggregate all their servers with few NCSes as possible and p2p connect
between them (actually between few small DCs), using VxLAN.

On paper it looks good. NCS seems to offer good port density, reasonable
price (as long you stick with not too complicated scenarios) and supports
VxLAN. As all of know, real world experience might be pretty different. Is
anyone here use NCS for VxLAN connectivity and would like to share his/her
thoughts and experience? That would be much appreciated.

Thank you in advance.

Best regards and happy new year.
_______________________________________________
cisco-nsp mailing list cisco-nsp@puck.nether.net
https://puck.nether.net/mailman/listinfo/cisco-nsp
archive at http://puck.nether.net/pipermail/cisco-nsp/
Re: Cisco NCS VxLAN Experience [ In reply to ]
On Wed, 2020-01-08 at 18:53 +0200, Alex K. wrote:
> A customer of mine's interested in acquiring some NCS boxes, in order
> to aggregate all their servers with few NCSes as possible and p2p
> connect between them (actually between few small DCs), using VxLAN.

I can't seem to find anything on NCS 5500 and VxLAN. It's mentioned a
few times in tech updates and Cisco Live presentations but nothing
more. What role would it have in a VxLAN network? Do you have more info
on that?

> On paper it looks good. NCS seems to offer good port density,
> reasonable price (as long you stick with not too complicated
> scenarios) and supports VxLAN.

For us it seemed deceptively cheap when looking at CapEx but ended up
being much more expensive when looking at recurring license costs. Make
sure you look at e.g. 3/5/7 years when calculating costs.

> As all of know, real world experience might be pretty different. Is
> anyone here use NCS for VxLAN connectivity and would like to share
> his/her thoughts and experience? That would be much appreciated.

We haven't looked at anything VxLAN related, but we did try and
shoehorn in an NCS 55A1-24H as a DC gateway router with ACI on the
south side and an existing MPLS network on the north side.

All in all it didn't work for us. My notes from the tests are below.

We specifically tested with a Bundle-Ether interface downwards for
redundancy. I think, but I haven't tested, that the limitations are the
same also for just single stand-alone physical interfaces.

- It doesn't support HSRP, despite what at least one data sheet
says [1]. Configuration guides say nothing about HSRP and even
though you can configure it, it doesn't work properly. So no
seamless migration from existing HSRP gateways.

- VRRP group IDs cannot be reused between subinterfaces on the
same bundle-ether. This seems to be an XR limitation according
to [2].

- You can only configure 16 different VRRD group IDs on different
subinterfaces on the same bundle-ether. After that it complains
in the log. Others have noted this for NCS 5504, see [3].

LC/0/0/CPU0:Jul 15 14:33:10.759 CEST: vlan_ea[126]: %PLATFORM-DNX_VLAN_EA-3-MAX_VRRP : Only 16 VRRP sessions can be configured under a main interface BE1 (0x800002c).
RP/0/RP0/CPU0:Jul 15 14:33:10.765 CEST: vrrp[1195]: %IP-FHRP-3-ERR_MAX_MAC : Maximum MAC address filters for interface Bundle-Ether1 exceeded

- You could create "l2tranport" subinterfaces and use BVI's, and
on these you can reuse VRRP group ID. This will give you up to
750 "subinterfaces" per physical interface, but NCS doesn't do
Netflow on BVI's. See e.g. [4]. It's not just not supported,
it doesn't accept the "flow" commands there.

- Regarding multicast, the platform only supports PIM-SSM (with
SSM mapping though) and doesn't do mVPN profile 0 (default
MDT, GRE, PIM in the core). Keep that in mind if you need to
interface with existing things like we did. See [5].

All in all we found the NCS 5500 platform to not be suitable as a DC
gateway for us. It could probably work fine as a core P-router or in a
specific edge scenario though.

[1]: https://www.cisco.com/c/en/us/products/collateral/routers/network-convergence-system-5500-series/datasheet-c78-737935.html
[2]: https://community.cisco.com/t5/xr-os-and-platforms/asr9k-4-2-3-to-4-3-4-upgrade-vrrp-issue/td-p/2448500
[3]: https://community.cisco.com/t5/xr-os-and-platforms/ncs5504-vrrp-sessions-limit/td-p/3675139
[4]: https://www.cisco.com/c/en/us/td/docs/iosxr/ncs5500/netflow/63x/b-netflow-cg-ncs5500-63x/b-netflow-cg-ncs5500-63x_chapter_010.html
[5]: https://www.cisco.com/c/en/us/td/docs/iosxr/ncs5500/multicast/61x/b-ncs-5500-multicast-configuration-guide-61x/b-ncs-5500-multicast-configuration-guide-61x_chapter_01.html

--
Peter


_______________________________________________
cisco-nsp mailing list cisco-nsp@puck.nether.net
https://puck.nether.net/mailman/listinfo/cisco-nsp
archive at http://puck.nether.net/pipermail/cisco-nsp/
Re: Cisco NCS VxLAN Experience [ In reply to ]
Hello,

Nexus 9000 is more suitable for this task. Even the price is much lower.


On 1/8/20 6:53 PM, Alex K. wrote:
> Hello everyone,
>
> A customer of mine's interested in acquiring some NCS boxes, in order to
> aggregate all their servers with few NCSes as possible and p2p connect
> between them (actually between few small DCs), using VxLAN.
>
> On paper it looks good. NCS seems to offer good port density, reasonable
> price (as long you stick with not too complicated scenarios) and supports
> VxLAN. As all of know, real world experience might be pretty different. Is
> anyone here use NCS for VxLAN connectivity and would like to share his/her
> thoughts and experience? That would be much appreciated.
>
> Thank you in advance.
>
> Best regards and happy new year.
> _______________________________________________
> cisco-nsp mailing list cisco-nsp@puck.nether.net
> https://puck.nether.net/mailman/listinfo/cisco-nsp
> archive at http://puck.nether.net/pipermail/cisco-nsp/

--
Best regards,
Adrian Minta


_______________________________________________
cisco-nsp mailing list cisco-nsp@puck.nether.net
https://puck.nether.net/mailman/listinfo/cisco-nsp
archive at http://puck.nether.net/pipermail/cisco-nsp/
Re: Cisco NCS VxLAN Experience [ In reply to ]
On 09/01/2020 15:56, Adrian Minta wrote:
> Nexus 9000 is more suitable for this task. Even the price is much lower.


This. Stick with the data centre products for data centre tasks.

If you need NCS for some other reason as well, that's fine, but start
with the suitable product range as Adrian suggests.

Regards,

--
Tom
_______________________________________________
cisco-nsp mailing list cisco-nsp@puck.nether.net
https://puck.nether.net/mailman/listinfo/cisco-nsp
archive at http://puck.nether.net/pipermail/cisco-nsp/
Re: Cisco NCS VxLAN Experience [ In reply to ]
Hi,

On Thu, Jan 09, 2020 at 04:34:53PM +0000, Tom Hill wrote:
> On 09/01/2020 15:56, Adrian Minta wrote:
> > Nexus 9000 is more suitable for this task. Even the price is much lower.
>
> This. Stick with the data centre products for data centre tasks.
>
> If you need NCS for some other reason as well, that's fine, but start
> with the suitable product range as Adrian suggests.

And *this* is why we're not buying Cisco anymore.

gert
--
"If was one thing all people took for granted, was conviction that if you
feed honest figures into a computer, honest figures come out. Never doubted
it myself till I met a computer with a sense of humor."
Robert A. Heinlein, The Moon is a Harsh Mistress

Gert Doering - Munich, Germany gert@greenie.muc.de
Re: Cisco NCS VxLAN Experience [ In reply to ]
So you are saying that a vendor shouldn't have certain products for certain domains? I'm unsure what your reasons is to be honest.

Best Regards,
Ted Pelas Johansson
On 9 Jan 2020, 18:30 +0100, Gert Doering <gert@greenie.muc.de>, wrote:
> Hi,
>
> On Thu, Jan 09, 2020 at 04:34:53PM +0000, Tom Hill wrote:
> > On 09/01/2020 15:56, Adrian Minta wrote:
> > > Nexus 9000 is more suitable for this task. Even the price is much lower.
> >
> > This. Stick with the data centre products for data centre tasks.
> >
> > If you need NCS for some other reason as well, that's fine, but start
> > with the suitable product range as Adrian suggests.
>
> And *this* is why we're not buying Cisco anymore.
>
> gert
> --
> "If was one thing all people took for granted, was conviction that if you
> feed honest figures into a computer, honest figures come out. Never doubted
> it myself till I met a computer with a sense of humor."
> Robert A. Heinlein, The Moon is a Harsh Mistress
>
> Gert Doering - Munich, Germany gert@greenie.muc.de
>
> _______________________________________________
> cisco-nsp mailing list cisco-nsp@puck.nether.net
> https://puck.nether.net/mailman/listinfo/cisco-nsp
> archive at http://puck.nether.net/pipermail/cisco-nsp/
_______________________________________________
cisco-nsp mailing list cisco-nsp@puck.nether.net
https://puck.nether.net/mailman/listinfo/cisco-nsp
archive at http://puck.nether.net/pipermail/cisco-nsp/
Re: Cisco NCS VxLAN Experience [ In reply to ]
Hi,

On Thu, Jan 09, 2020 at 07:01:32PM +0100, Ted Pelas Johansson wrote:
> So you are saying that a vendor shouldn't have certain products for certain domains? I'm unsure what your reasons is to be honest.

Cisco has a zillion products that mainly differenciate in "which of
the advertised features are unusable or broken" and "what operating
system do we use this week?".

Ditching out half the products and using the engineering capacity
freed by this might results in half the product families shipping
actually useful products.

I have some amount of understanding for "product <x> cannot do <y>,
because the hardware cannot do it". Trade-offs need to be made, and
restrictions need to be clearly documented (before buying).

I do have *null* understanding for "we have cisco proprietary protocols
that our customers are actively using (HSRP, EIGRP) but we do not
support this on <arbitrary product> because we can, buy something
else!" (EIGRP on IOS XR on NCS5k, HSRPv2 with IPv6 on ASR920).

Or "yes, we can see that this <control plane feature> would be trivially
to implement, but you can only have it if you buy the *same box* coloured
differently" (static MPLS labels on 6500 vs. 7600 come to mind).

I only have very limited understanding for "for your set of requirements,
we have four different products that all look the same, and possibly
even share the same chipset, but because of BU differenciation, none
of the products will do *all* you need - so if you need feature X, Y
and Z, you need to buy from *two* of our nice BUs!".


In many real-world networks, there is no hard-cut differenciation between
"this part is 'datacenter', that part is 'enterprise' and the interconnects
are 'service-provide'". We have it all, and we need to be able to support
it with reasonable effort.

Buying a pair of Nexus, a pair of ASR9k and possibly a pair of ASR1000
*per site* just because different Cisco BUs can't agree on how to interop
is just too annoying. If I want non-interoperable gear with different
OSes and totally different bugs, I can just buy from a different vendor
instead (who might actually value our money).

gert
--
"If was one thing all people took for granted, was conviction that if you
feed honest figures into a computer, honest figures come out. Never doubted
it myself till I met a computer with a sense of humor."
Robert A. Heinlein, The Moon is a Harsh Mistress

Gert Doering - Munich, Germany gert@greenie.muc.de
Re: Cisco NCS VxLAN Experience [ In reply to ]
On 9/Jan/20 20:19, Gert Doering wrote:

> Cisco has a zillion products that mainly differenciate in "which of
> the advertised features are unusable or broken" and "what operating
> system do we use this week?".
>
> Ditching out half the products and using the engineering capacity
> freed by this might results in half the product families shipping
> actually useful products.

I couldn't agree more.

Some use-cases are unique, but by and large, the Internet is the
Internet, and creating distinct commercial lines so you can re-sell the
same products and services several times over is not going to work anymore.

The only area where costs aren't reducing is in vendor equipment and
services. The pressure on revenue from Internet services of any kind are
very high, and getting even higher. Customers have grown accustomed to
expecting all services for free, or close to free, and the companies
providing those service have taken the bait, putting a strain on each
other in the same industry and creating a race to the bottom.

Equipment vendors can't keep charging network operators like it was back
in the day. If they continue to do so, we will walk. Simple.

Mark.
Re: Cisco NCS VxLAN Experience [ In reply to ]
On 9/Jan/20 20:19, Gert Doering wrote:

>> Cisco has a zillion products that mainly differenciate in "which of
>> the advertised features are unusable or broken" and "what operating
>> system do we use this week?".
>>
>> Ditching out half the products and using the engineering capacity
>> freed by this might results in half the product families shipping
>> actually useful products.

On 10/01/2020, 08:51, Mark wrote:

>I couldn't agree more.

>Some use-cases are unique, but by and large, the Internet is the
>Internet, and creating distinct commercial lines so you can re-sell the
>same products and services several times over is not going to work anymore.

>The only area where costs aren't reducing is in vendor equipment and
>services. The pressure on revenue from Internet services of any kind are
>very high, and getting even higher. Customers have grown accustomed to
>expecting all services for free, or close to free, and the companies
>providing those service have taken the bait, putting a strain on each
>other in the same industry and creating a race to the bottom.

>Equipment vendors can't keep charging network operators like it was back
>in the day. If they continue to do so, we will walk. Simple.

>Mark.

Completely agree with both of you. We've been trying to make this case with Cisco for many years.

Additionally The NCS line, both the DC and SP products, are based on Broadcom chipsets which are heavily limited in their capabilities, particularly egress TCAM capabilities are limited in such a way that it makes it almost unusable in a service-provider environment. Add to that the pain of having to deal with lack of feature parity between the platforms, and even releasing platforms for the service provider environment which out of the box don't support essential services (the NCS 520 springs to mind, a Service Provider NID that doesn't support Clocking or IGMP snooping).

The mixture of IOS XE and XR is also highly annoying, XE at this point is an enterprise operating system and the End-of-life of the 15.5S series spells the end for service provider use of IOS XE in my mind, where the 920 lines goes after that is a big question mark, our experience with IOS XE 16 (16.3/16.6 and 16.9) is that it's a largely unusable blob of crap.

We've been trying to raise these points with Cisco for a while, but I have a feeling they mostly don't care about the SP market at this stage and are focused on the more lucrative "cloud" and enterprise markets

Kind regards,
Sigurbjörn

_______________________________________________
cisco-nsp mailing list cisco-nsp@puck.nether.net
https://puck.nether.net/mailman/listinfo/cisco-nsp
archive at http://puck.nether.net/pipermail/cisco-nsp/
Re: Cisco NCS VxLAN Experience [ In reply to ]
On 10/Jan/20 13:09, Sigurbjörn Birkir Lárusson wrote:

>
>
> Additionally The NCS line, both the DC and SP products, are based on Broadcom chipsets which are heavily limited in their capabilities, particularly egress TCAM capabilities are limited in such a way that it makes it almost unusable in a service-provider environment.

This is actually a very important point that has been on mind, but I've
neglected to eloquently share with my peers.

Broadcom levels the playing field amongst traditional and new vendors.
If Cisco and Juniper have the same access to Broadcom chips as do newer
market entrants such as Arista and Arrcus, what are we really paying the
traditional, expensive vendors for when ordering boxes shipping with the
same chips across the board?

Established code in routing protocols from the traditional vendors would
be quite mature, granted. However, if the limitations in Broadcom chips
apply to all vendors that use them, no amount of software hackery will
fix that. So if both Arista and Cisco are struggling to deliver the same
features due to a limitation in the Broadcom chip, do you want to spend
5 times the amount of money to figure that out, or 5 times less?

The way I see it, new vendors who are investing in off-the-shelf chips
provide a better commercial option to network operators who don't mind
such silicon in the short-to-medium (and, perhaps, even long) term
because those vendors are not encumbered by the days when they made
their own silicon.

I don't, in all honesty, see any real benefit in using traditional,
expensive vendors that are pushing out boxes based on merchant silicon
for a sales and pricing process steeped in their in-house silicon
history, when their "superior" software will be as broken as that from a
newer vendor due to the merchant silicon limitations, i.e., if I don't
want to spend real money, I won't do it with a  traditional vendor.

Which brings me back to - if a Juniper MX480 appears everywhere
(peering, transit, edge, data centre, metro, enterprise, security,
analytics, e.t.c.), why should I expend any energy being told you need
an ASR9000 for this, but a Nexus 7000 for that?

Mark.



_______________________________________________
cisco-nsp mailing list cisco-nsp@puck.nether.net
https://puck.nether.net/mailman/listinfo/cisco-nsp
archive at http://puck.nether.net/pipermail/cisco-nsp/
Re: Cisco NCS VxLAN Experience [ In reply to ]
_______________________________________________
cisco-nsp mailing list cisco-nsp@puck.nether.net
https://puck.nether.net/mailman/listinfo/cisco-nsp
archive at http://puck.nether.net/pipermail/cisco-nsp/
Re: Cisco NCS VxLAN Experience [ In reply to ]
_______________________________________________
cisco-nsp mailing list cisco-nsp@puck.nether.net
https://puck.nether.net/mailman/listinfo/cisco-nsp
archive at http://puck.nether.net/pipermail/cisco-nsp/
Re: Cisco NCS VxLAN Experience [ In reply to ]
On 09/01/2020 18:19, Gert Doering wrote:
> Cisco has a zillion products that mainly differenciate in "which of
> the advertised features are unusable or broken" and "what operating
> system do we use this week?".
>
> Ditching out half the products and using the engineering capacity
> freed by this might results in half the product families shipping
> actually useful products.


To some extent, I would agree. Though I do not agree entirely.

At a certain level of "one size fits all", you reach a point where no,
actually that box isn't suitable for every function. Either it's
designed for a different use-case, or the bill of materials is too high
for areas where lighter feature sets will suffice.

Differentiation of hardware features vs. hardware cost, is why we don't
all run the same laptop, phone, car, bicycle, etc.

What I did find promising was that the Cisco 8000 has dropped its
suffix, suggesting that we might find it aligning NCS, ASR, and - who
knows - perhaps even some Nexus areas? It's an interesting shift anyway.

Regards,

--
Tom
_______________________________________________
cisco-nsp mailing list cisco-nsp@puck.nether.net
https://puck.nether.net/mailman/listinfo/cisco-nsp
archive at http://puck.nether.net/pipermail/cisco-nsp/
Re: Cisco NCS VxLAN Experience [ In reply to ]
On 10/Jan/20 16:52, Brian Turnbow wrote:
> You neglected to mention that they also mostly provide feature parity across the platforms that use that silicon
> So box a and box b will do the sames things in the same way, with the same commands.
> So no we at business unit service provider with not allow datacenter features in our boxes...

Madness.

But it's fine; while the BU's are waving their gentleman sausages
around, we, the customers, will keep walking.

Mark.
_______________________________________________
cisco-nsp mailing list cisco-nsp@puck.nether.net
https://puck.nether.net/mailman/listinfo/cisco-nsp
archive at http://puck.nether.net/pipermail/cisco-nsp/
Re: Cisco NCS VxLAN Experience [ In reply to ]
On 10/Jan/20 16:57, Tom Hill wrote:

> To some extent, I would agree. Though I do not agree entirely.
>
> At a certain level of "one size fits all", you reach a point where no,
> actually that box isn't suitable for every function. Either it's
> designed for a different use-case, or the bill of materials is too high
> for areas where lighter feature sets will suffice.
>
> Differentiation of hardware features vs. hardware cost, is why we don't
> all run the same laptop, phone, car, bicycle, etc.

We all understand and agree that some kind of specialization is
necessary for certain use-cases.

There isn't any equipment vendor that ships only one piece of hardware,
and we can understand why, and why that makes sense.

But what Cisco are doing goes above and beyond what we can accept.

Mark.
_______________________________________________
cisco-nsp mailing list cisco-nsp@puck.nether.net
https://puck.nether.net/mailman/listinfo/cisco-nsp
archive at http://puck.nether.net/pipermail/cisco-nsp/
Re: Cisco NCS VxLAN Experience [ In reply to ]
> Mark Tinka
> Sent: Friday, January 10, 2020 11:38 AM
>
> However, if the limitations in Broadcom chips apply to
> all vendors that use them, no amount of software hackery will fix that. So if
> both Arista and Cisco are struggling to deliver the same features due to a
> limitation in the Broadcom chip, do you want to spend
> 5 times the amount of money to figure that out, or 5 times less?
>

Now I guess the underlying assumption is that all vendors just take the of the shelf chip and plug it as is inside their box of choice.
But I think that's not quite the case, as each network gear vendor works with chip vendors on various customization even adding custom peripherals etc.. experimenting with the chip u-code...
So yes, while at the core the bare silicon is the same the differences in u-code and peripherals might make it into quite a distinct products in the end.
And I'd guess the cost is somewhat proportional to the RND spent on developing the particular solution?

adam

_______________________________________________
cisco-nsp mailing list cisco-nsp@puck.nether.net
https://puck.nether.net/mailman/listinfo/cisco-nsp
archive at http://puck.nether.net/pipermail/cisco-nsp/