Mailing List Archive

Linux router network cards
I'm looking around for networking cards to build a linux based
router. It needs to be able to do XDP, multiqueues, have good in-kernel
driver support and be able to handle 10Gbe with good offloading for
dealing with high packets per second.

What features should I be looking for to really optimize things for a
three transit setup, with full tables.

Something like the Intel XL710-QDA2 card maybe?

--
micah
RE: Linux router network cards [ In reply to ]
Hi Micah,

Take a look at the Mellanox ConnectX 5 series of cards. They handle DPDK, PVRDMA (basically SR-IOV that allows live migration between hosts), and can even process packets within the NIC for some models. They did a fantastic presentation at AusNOG 2019 which showed off a lot of the features. We tried some out with Vmware and could get 20Gbps throughput (limited by the 2x 10G NICs we had configured) to a VM running Linux with DPDK+VPP.

The slidedeck for the presentation is here:
https://www.ausnog.net/sites/default/files/ausnog-2019/presentations/1.9_Rhod_Brown_AusNOG2019.pdf

It's heavily targeting virtualised workloads but some of the feature sets apply to bare-metal uses too.

Regards,
Philip Loenneker | Senior Network Engineer | TasmaNet

-----Original Message-----
From: NANOG <nanog-bounces+philip.loenneker=tasmanet.com.au@nanog.org> On Behalf Of micah anderson
Sent: Wednesday, 21 October 2020 2:37 PM
To: nanog@nanog.org
Subject: Linux router network cards


I'm looking around for networking cards to build a linux based router. It needs to be able to do XDP, multiqueues, have good in-kernel driver support and be able to handle 10Gbe with good offloading for dealing with high packets per second.

What features should I be looking for to really optimize things for a three transit setup, with full tables.

Something like the Intel XL710-QDA2 card maybe?

--
micah
Re: Linux router network cards [ In reply to ]
On 2020-10-20 22:37, Philip Loenneker wrote:
> Take a look at the Mellanox ConnectX 5 series of cards. They handle DPDK, PVRDMA (basically SR-IOV that allows live migration between hosts), and can even process packets within the NIC for some models. They did a fantastic presentation at AusNOG 2019 which showed off a lot of the features. We tried some out with Vmware and could get 20Gbps throughput (limited by the 2x 10G NICs we had configured) to a VM running Linux with DPDK+VPP.

Plus Mellanox introduced the SwitchDev capability which provides for
offloading flow management to the hardware.
Re: Linux router network cards [ In reply to ]
I wonder if they are going to get CUDA cores on the next version since they
are owned by NVIDIA now. That would be a powerful little package.


On Wed, Oct 21, 2020 at 1:42 AM Raymond Burkholder <ray@oneunified.net>
wrote:

> On 2020-10-20 22:37, Philip Loenneker wrote:
> > Take a look at the Mellanox ConnectX 5 series of cards. They handle
> DPDK, PVRDMA (basically SR-IOV that allows live migration between hosts),
> and can even process packets within the NIC for some models. They did a
> fantastic presentation at AusNOG 2019 which showed off a lot of the
> features. We tried some out with Vmware and could get 20Gbps throughput
> (limited by the 2x 10G NICs we had configured) to a VM running Linux with
> DPDK+VPP.
>
> Plus Mellanox introduced the SwitchDev capability which provides for
> offloading flow management to the hardware.
>
Re: Linux router network cards [ In reply to ]
Hi micah,

I think this was shared in the past and may be useful with regards to
what you expect in terms of performance:
https://blog.apnic.net/2020/04/30/how-to-build-an-xdp-based-bgp-peering-router/
.

BR,

Marinos

On 21-Oct-20 6:37 AM, micah anderson wrote:
> I'm looking around for networking cards to build a linux based
> router. It needs to be able to do XDP, multiqueues, have good in-kernel
> driver support and be able to handle 10Gbe with good offloading for
> dealing with high packets per second.
>
> What features should I be looking for to really optimize things for a
> three transit setup, with full tables.
>
> Something like the Intel XL710-QDA2 card maybe?
>
RE: Linux router network cards [ In reply to ]
Thanks for the reply.

Philip Loenneker <Philip.Loenneker@tasmanet.com.au> writes:
> Take a look at the Mellanox ConnectX 5 series of cards. They handle
> DPDK, PVRDMA (basically SR-IOV that allows live migration between
> hosts), and can even process packets within the NIC for some

From what I can tell, SR-IOV/PVRDMA aren't really useful for me in
building a router that wont be doing any virtualization.

If the card can do DPDK, can it do XDP?

> The slidedeck for the presentation is here:
> https://www.ausnog.net/sites/default/files/ausnog-2019/presentations/1.9_Rhod_Brown_AusNOG2019.pdf
>
> It's heavily targeting virtualised workloads but some of the feature sets apply to bare-metal uses too.

Yeah, this wont be a virtualized environment, just a router passing
packets, dropping them, handling bgp and collecting flows.

--
micah
RE: Linux router network cards [ In reply to ]
Chelsio cards are probably what you are looking for.

https://www.chelsio.com/terminator-6-asic/

It's closer to an asic than a traditional nic as the router/firewall rules
are pushed directly into the hardware.

I don't know how good they are with linux and they seem to be compatible.
https://www.chelsio.com/linux/

You will need to mess around a bit and fiddle here and there. If you don't
mind using FreeBSD instead of linux, you could achieve a smoother and more
integrated experience.

Jean

-----Original Message-----
From: NANOG <nanog-bounces+jean=ddostest.me@nanog.org> On Behalf Of micah
anderson
Sent: Thursday, October 22, 2020 5:31 PM
To: Philip Loenneker <Philip.Loenneker@tasmanet.com.au>; NANOG
<nanog@nanog.org>
Subject: RE: Linux router network cards


Thanks for the reply.

Philip Loenneker <Philip.Loenneker@tasmanet.com.au> writes:
> Take a look at the Mellanox ConnectX 5 series of cards. They handle
> DPDK, PVRDMA (basically SR-IOV that allows live migration between
> hosts), and can even process packets within the NIC for some

From what I can tell, SR-IOV/PVRDMA aren't really useful for me in building
a router that wont be doing any virtualization.

If the card can do DPDK, can it do XDP?

> The slidedeck for the presentation is here:
> https://www.ausnog.net/sites/default/files/ausnog-2019/presentations/1
> .9_Rhod_Brown_AusNOG2019.pdf
>
> It's heavily targeting virtualised workloads but some of the feature sets
apply to bare-metal uses too.

Yeah, this wont be a virtualized environment, just a router passing packets,
dropping them, handling bgp and collecting flows.

--
micah
Re: Linux router network cards [ In reply to ]
I use DANOS with Intel XL710 10G NICs in DPDK mode for linux based routing.

If you're doing routing protocols, allocate 2 CPU cores to the control
plane and then a CPU core per 10G/1G interface for the dataplane, plus an
extra core for good measure. So for a 4 x 10G router taking in full routes,
2 cores for control plane, 5 cores for the dataplane. Those cores should be
Intel Xeon E5-2600v3/4 or newer and faster the clocks, the better.

Similar CPU core allocations if you choose TNSR.

On Thu, Oct 22, 2020 at 3:21 PM Jean St-Laurent via NANOG <nanog@nanog.org>
wrote:

> Chelsio cards are probably what you are looking for.
>
> https://www.chelsio.com/terminator-6-asic/
>
> It's closer to an asic than a traditional nic as the router/firewall rules
> are pushed directly into the hardware.
>
> I don't know how good they are with linux and they seem to be compatible.
> https://www.chelsio.com/linux/
>
> You will need to mess around a bit and fiddle here and there. If you don't
> mind using FreeBSD instead of linux, you could achieve a smoother and more
> integrated experience.
>
> Jean
>
> -----Original Message-----
> From: NANOG <nanog-bounces+jean=ddostest.me@nanog.org> On Behalf Of micah
> anderson
> Sent: Thursday, October 22, 2020 5:31 PM
> To: Philip Loenneker <Philip.Loenneker@tasmanet.com.au>; NANOG
> <nanog@nanog.org>
> Subject: RE: Linux router network cards
>
>
> Thanks for the reply.
>
> Philip Loenneker <Philip.Loenneker@tasmanet.com.au> writes:
> > Take a look at the Mellanox ConnectX 5 series of cards. They handle
> > DPDK, PVRDMA (basically SR-IOV that allows live migration between
> > hosts), and can even process packets within the NIC for some
>
> From what I can tell, SR-IOV/PVRDMA aren't really useful for me in building
> a router that wont be doing any virtualization.
>
> If the card can do DPDK, can it do XDP?
>
> > The slidedeck for the presentation is here:
> > https://www.ausnog.net/sites/default/files/ausnog-2019/presentations/1
> > .9_Rhod_Brown_AusNOG2019.pdf
> >
> > It's heavily targeting virtualised workloads but some of the feature sets
> apply to bare-metal uses too.
>
> Yeah, this wont be a virtualized environment, just a router passing
> packets,
> dropping them, handling bgp and collecting flows.
>
> --
> micah
>
>
RE: Linux router network cards [ In reply to ]
micah anderson <micah@riseup.net> writes:

> Thanks for the reply.
>
> Philip Loenneker <Philip.Loenneker@tasmanet.com.au> writes:
>> Take a look at the Mellanox ConnectX 5 series of cards. They handle
>> DPDK, PVRDMA (basically SR-IOV that allows live migration between
>> hosts), and can even process packets within the NIC for some
>
> From what I can tell, SR-IOV/PVRDMA aren't really useful for me in
> building a router that wont be doing any virtualization.
>
> If the card can do DPDK, can it do XDP?

The Connect-X 5 has excellent XDP support - it's what we used for the
XDP paper:

https://github.com/xdp-project/xdp-paper#hardware

-Toke
RE: Linux router network cards [ In reply to ]
Hi Jared,



This project looks very interesting.



Can you share with us which software or package do you use in DANOS for routing? Is it a kind of command wrapper on top of FRR?



Also, it seems stable, but I am sure you already faced some minor or weird bugs. How is the support handle with DANOS? Is it community driven?



Thanks for sharing



Jean



From: NANOG <nanog-bounces+jean=ddostest.me@nanog.org> On Behalf Of Jared Geiger
Sent: Friday, October 23, 2020 12:30 AM
To: NANOG <nanog@nanog.org>
Subject: Re: Linux router network cards



I use DANOS with Intel XL710 10G NICs in DPDK mode for linux based routing.



If you're doing routing protocols, allocate 2 CPU cores to the control plane and then a CPU core per 10G/1G interface for the dataplane, plus an extra core for good measure. So for a 4 x 10G router taking in full routes, 2 cores for control plane, 5 cores for the dataplane. Those cores should be Intel Xeon E5-2600v3/4 or newer and faster the clocks, the better.



Similar CPU core allocations if you choose TNSR.



On Thu, Oct 22, 2020 at 3:21 PM Jean St-Laurent via NANOG <nanog@nanog.org <mailto:nanog@nanog.org> > wrote:

Chelsio cards are probably what you are looking for.

https://www.chelsio.com/terminator-6-asic/

It's closer to an asic than a traditional nic as the router/firewall rules
are pushed directly into the hardware.

I don't know how good they are with linux and they seem to be compatible.
https://www.chelsio.com/linux/

You will need to mess around a bit and fiddle here and there. If you don't
mind using FreeBSD instead of linux, you could achieve a smoother and more
integrated experience.

Jean

-----Original Message-----
From: NANOG <nanog-bounces+jean=ddostest.me@nanog.org <mailto:ddostest.me@nanog.org> > On Behalf Of micah
anderson
Sent: Thursday, October 22, 2020 5:31 PM
To: Philip Loenneker <Philip.Loenneker@tasmanet.com.au <mailto:Philip.Loenneker@tasmanet.com.au> >; NANOG
<nanog@nanog.org <mailto:nanog@nanog.org> >
Subject: RE: Linux router network cards


Thanks for the reply.

Philip Loenneker <Philip.Loenneker@tasmanet.com.au <mailto:Philip.Loenneker@tasmanet.com.au> > writes:
> Take a look at the Mellanox ConnectX 5 series of cards. They handle
> DPDK, PVRDMA (basically SR-IOV that allows live migration between
> hosts), and can even process packets within the NIC for some

From what I can tell, SR-IOV/PVRDMA aren't really useful for me in building
a router that wont be doing any virtualization.

If the card can do DPDK, can it do XDP?

> The slidedeck for the presentation is here:
> https://www.ausnog.net/sites/default/files/ausnog-2019/presentations/1
> .9_Rhod_Brown_AusNOG2019.pdf
>
> It's heavily targeting virtualised workloads but some of the feature sets
apply to bare-metal uses too.

Yeah, this wont be a virtualized environment, just a router passing packets,
dropping them, handling bgp and collecting flows.

--
micah
Re: Linux router network cards [ In reply to ]
DANOS is a full Network Operating System https://www.danosproject.org/
managed by the Linux Foundation so it is open source. AT&T is the main
contributor and consumer of it so far. It evolved from Vyatta, the linux
NOS that has been around and passed ownership through a couple companies
since the early 2000s. It is using FRR for the routing control plane and
the Danos CLI wraps it and the other software packages together (VPNs, DNS,
DHCP, CGNAT, etc) into a single config file to manage. Support for the
opensource version is very responsive via github, Atlassan issue tracker,
and the Matrix chat room.

If you want a commercially supported version, you can go with Danos Vyatta
Edition from IP Infusion. It uses IPI's routing engine control plane
instead of FRR but is very similar.

On Fri, Oct 23, 2020 at 3:57 AM Jean St-Laurent via NANOG <nanog@nanog.org>
wrote:

> Hi Jared,
>
>
>
> This project looks very interesting.
>
>
>
> Can you share with us which software or package do you use in DANOS for
> routing? Is it a kind of command wrapper on top of FRR?
>
>
>
> Also, it seems stable, but I am sure you already faced some minor or weird
> bugs. How is the support handle with DANOS? Is it community driven?
>
>
>
> Thanks for sharing
>
>
>
> Jean
>
>
>
> *From:* NANOG <nanog-bounces+jean=ddostest.me@nanog.org> *On Behalf Of *Jared
> Geiger
> *Sent:* Friday, October 23, 2020 12:30 AM
> *To:* NANOG <nanog@nanog.org>
> *Subject:* Re: Linux router network cards
>
>
>
> I use DANOS with Intel XL710 10G NICs in DPDK mode for linux based routing.
>
>
>
> If you're doing routing protocols, allocate 2 CPU cores to the control
> plane and then a CPU core per 10G/1G interface for the dataplane, plus an
> extra core for good measure. So for a 4 x 10G router taking in full routes,
> 2 cores for control plane, 5 cores for the dataplane. Those cores should be
> Intel Xeon E5-2600v3/4 or newer and faster the clocks, the better.
>
>
>
> Similar CPU core allocations if you choose TNSR.
>
>
>
> On Thu, Oct 22, 2020 at 3:21 PM Jean St-Laurent via NANOG <nanog@nanog.org>
> wrote:
>
> Chelsio cards are probably what you are looking for.
>
> https://www.chelsio.com/terminator-6-asic/
>
> It's closer to an asic than a traditional nic as the router/firewall rules
> are pushed directly into the hardware.
>
> I don't know how good they are with linux and they seem to be compatible.
> https://www.chelsio.com/linux/
>
> You will need to mess around a bit and fiddle here and there. If you don't
> mind using FreeBSD instead of linux, you could achieve a smoother and more
> integrated experience.
>
> Jean
>
> -----Original Message-----
> From: NANOG <nanog-bounces+jean=ddostest.me@nanog.org> On Behalf Of micah
> anderson
> Sent: Thursday, October 22, 2020 5:31 PM
> To: Philip Loenneker <Philip.Loenneker@tasmanet.com.au>; NANOG
> <nanog@nanog.org>
> Subject: RE: Linux router network cards
>
>
> Thanks for the reply.
>
> Philip Loenneker <Philip.Loenneker@tasmanet.com.au> writes:
> > Take a look at the Mellanox ConnectX 5 series of cards. They handle
> > DPDK, PVRDMA (basically SR-IOV that allows live migration between
> > hosts), and can even process packets within the NIC for some
>
> From what I can tell, SR-IOV/PVRDMA aren't really useful for me in building
> a router that wont be doing any virtualization.
>
> If the card can do DPDK, can it do XDP?
>
> > The slidedeck for the presentation is here:
> > https://www.ausnog.net/sites/default/files/ausnog-2019/presentations/1
> > .9_Rhod_Brown_AusNOG2019.pdf
> >
> > It's heavily targeting virtualised workloads but some of the feature sets
> apply to bare-metal uses too.
>
> Yeah, this wont be a virtualized environment, just a router passing
> packets,
> dropping them, handling bgp and collecting flows.
>
> --
> micah
>
>
Re: Linux router network cards [ In reply to ]
In addition to Jared's advice, I would recommend calculating PCI-Express
bandwidth bus points for whatever platform one is using.

For instance using the Intel X710-DA4, which could be capable in a maximal
scenario of 80Gbps of traffic, ensure it's in at least a PCI-E 3.0 x4 slot.
And calculate the total number of PCI-E 3.0 x1 (or PCI-E 4.0 if a very new
system) lanes that exist and are connected to the CPU. Big difference
between some options for Ryzen and Threadripper vs Intel CPUs, towards the
lower end of the cost range.

With recent Linux kernels if you have an Intel 510 or 710 series two or
four port card in a slot that can't support its full capability, you'll get
a warning in dmesg at boot time.



On Thu, Oct 22, 2020 at 9:30 PM Jared Geiger <jared@compuwizz.net> wrote:

> I use DANOS with Intel XL710 10G NICs in DPDK mode for linux based routing.
>
> If you're doing routing protocols, allocate 2 CPU cores to the control
> plane and then a CPU core per 10G/1G interface for the dataplane, plus an
> extra core for good measure. So for a 4 x 10G router taking in full routes,
> 2 cores for control plane, 5 cores for the dataplane. Those cores should be
> Intel Xeon E5-2600v3/4 or newer and faster the clocks, the better.
>
> Similar CPU core allocations if you choose TNSR.
>
> On Thu, Oct 22, 2020 at 3:21 PM Jean St-Laurent via NANOG <nanog@nanog.org>
> wrote:
>
>> Chelsio cards are probably what you are looking for.
>>
>> https://www.chelsio.com/terminator-6-asic/
>>
>> It's closer to an asic than a traditional nic as the router/firewall rules
>> are pushed directly into the hardware.
>>
>> I don't know how good they are with linux and they seem to be compatible.
>> https://www.chelsio.com/linux/
>>
>> You will need to mess around a bit and fiddle here and there. If you don't
>> mind using FreeBSD instead of linux, you could achieve a smoother and more
>> integrated experience.
>>
>> Jean
>>
>> -----Original Message-----
>> From: NANOG <nanog-bounces+jean=ddostest.me@nanog.org> On Behalf Of micah
>> anderson
>> Sent: Thursday, October 22, 2020 5:31 PM
>> To: Philip Loenneker <Philip.Loenneker@tasmanet.com.au>; NANOG
>> <nanog@nanog.org>
>> Subject: RE: Linux router network cards
>>
>>
>> Thanks for the reply.
>>
>> Philip Loenneker <Philip.Loenneker@tasmanet.com.au> writes:
>> > Take a look at the Mellanox ConnectX 5 series of cards. They handle
>> > DPDK, PVRDMA (basically SR-IOV that allows live migration between
>> > hosts), and can even process packets within the NIC for some
>>
>> From what I can tell, SR-IOV/PVRDMA aren't really useful for me in
>> building
>> a router that wont be doing any virtualization.
>>
>> If the card can do DPDK, can it do XDP?
>>
>> > The slidedeck for the presentation is here:
>> > https://www.ausnog.net/sites/default/files/ausnog-2019/presentations/1
>> > .9_Rhod_Brown_AusNOG2019.pdf
>> >
>> > It's heavily targeting virtualised workloads but some of the feature
>> sets
>> apply to bare-metal uses too.
>>
>> Yeah, this wont be a virtualized environment, just a router passing
>> packets,
>> dropping them, handling bgp and collecting flows.
>>
>> --
>> micah
>>
>>
RE: Linux router network cards [ In reply to ]
And do not use an Intel CPU.

Intel only has 4x PCIe lanes that are shared out into whatever configuration they claim to have and are totally unsuitable for use in a computer that actually has to be able to do high-speed I/O.

--
Be decisive. Make a decision, right or wrong. The road of life is paved with flat squirrels who could not make a decision.

>-----Original Message-----
>From: NANOG <nanog-bounces+kmedcalf=dessus.com@nanog.org> On Behalf Of
>Eric Kuhnke
>Sent: Saturday, 24 October, 2020 06:22
>To: Jared Geiger <jared@compuwizz.net>
>Cc: NANOG <nanog@nanog.org>
>Subject: Re: Linux router network cards
>
>In addition to Jared's advice, I would recommend calculating PCI-Express
>bandwidth bus points for whatever platform one is using.
>
>
>For instance using the Intel X710-DA4, which could be capable in a
>maximal scenario of 80Gbps of traffic, ensure it's in at least a PCI-E
>3.0 x4 slot. And calculate the total number of PCI-E 3.0 x1 (or PCI-E 4.0
>if a very new system) lanes that exist and are connected to the CPU. Big
>difference between some options for Ryzen and Threadripper vs Intel CPUs,
>towards the lower end of the cost range.
>
>
>With recent Linux kernels if you have an Intel 510 or 710 series two or
>four port card in a slot that can't support its full capability, you'll
>get a warning in dmesg at boot time.
>
>
>
>On Thu, Oct 22, 2020 at 9:30 PM Jared Geiger <jared@compuwizz.net
><mailto:jared@compuwizz.net> > wrote:
>
>
> I use DANOS with Intel XL710 10G NICs in DPDK mode for linux based
>routing.
>
> If you're doing routing protocols, allocate 2 CPU cores to the
>control plane and then a CPU core per 10G/1G interface for the dataplane,
>plus an extra core for good measure. So for a 4 x 10G router taking in
>full routes, 2 cores for control plane, 5 cores for the dataplane. Those
>cores should be Intel Xeon E5-2600v3/4 or newer and faster the clocks,
>the better.
>
> Similar CPU core allocations if you choose TNSR.
>
> On Thu, Oct 22, 2020 at 3:21 PM Jean St-Laurent via NANOG
><nanog@nanog.org <mailto:nanog@nanog.org> > wrote:
>
>
> Chelsio cards are probably what you are looking for.
>
> https://www.chelsio.com/terminator-6-asic/
>
> It's closer to an asic than a traditional nic as the
>router/firewall rules
> are pushed directly into the hardware.
>
> I don't know how good they are with linux and they seem to be
>compatible.
> https://www.chelsio.com/linux/
>
> You will need to mess around a bit and fiddle here and there.
>If you don't
> mind using FreeBSD instead of linux, you could achieve a
>smoother and more
> integrated experience.
>
> Jean
>
> -----Original Message-----
> From: NANOG <nanog-bounces+jean=ddostest.me@nanog.org
><mailto:ddostest.me@nanog.org> > On Behalf Of micah
> anderson
> Sent: Thursday, October 22, 2020 5:31 PM
> To: Philip Loenneker <Philip.Loenneker@tasmanet.com.au
><mailto:Philip.Loenneker@tasmanet.com.au> >; NANOG
> <nanog@nanog.org <mailto:nanog@nanog.org> >
> Subject: RE: Linux router network cards
>
>
> Thanks for the reply.
>
> Philip Loenneker <Philip.Loenneker@tasmanet.com.au
><mailto:Philip.Loenneker@tasmanet.com.au> > writes:
> > Take a look at the Mellanox ConnectX 5 series of cards. They
>handle
> > DPDK, PVRDMA (basically SR-IOV that allows live migration
>between
> > hosts), and can even process packets within the NIC for some
>
> From what I can tell, SR-IOV/PVRDMA aren't really useful for
>me in building
> a router that wont be doing any virtualization.
>
> If the card can do DPDK, can it do XDP?
>
> > The slidedeck for the presentation is here:
> > https://www.ausnog.net/sites/default/files/ausnog-
>2019/presentations/1
> > .9_Rhod_Brown_AusNOG2019.pdf
> >
> > It's heavily targeting virtualised workloads but some of the
>feature sets
> apply to bare-metal uses too.
>
> Yeah, this wont be a virtualized environment, just a router
>passing packets,
> dropping them, handling bgp and collecting flows.
>
> --
> micah
>
>
Re: Linux router network cards [ In reply to ]
? 24 octobre 2020 09:55 -06, Keith Medcalf:

> And do not use an Intel CPU.
>
> Intel only has 4x PCIe lanes that are shared out into whatever
> configuration they claim to have and are totally unsuitable for use in
> a computer that actually has to be able to do high-speed I/O.

That's likely to be incorrect. Intel CPU usually have 48 lanes for the
Skylake generation. The 4 lanes limitation only applies to what is
connected over DMI to the PCH, which is usually used for low-bandwidth
stuff (1G NIC, SATA, 1x PCIe slots). Look at your motherboard manual to
check how many lanes are affected to each component.
--
Make sure every module hides something.
- The Elements of Programming Style (Kernighan & Plauger)
Re: Linux router network cards [ In reply to ]
Not true, Intel Xeon E5 v3/4 have many more lanes than that. For example,
this one has 40
https://ark.intel.com/content/www/us/en/ark/products/91754/intel-xeon-processor-e5-2680-v4-35m-cache-2-40-ghz.html

Put two in the system and thats 80 lanes of PCIe Gen3.

Even newer procs in the bronze/silver/gold/platinum lineup have more lanes.

On Sat, Oct 24, 2020 at 8:57 AM Keith Medcalf <kmedcalf@dessus.com> wrote:

>
> And do not use an Intel CPU.
>
> Intel only has 4x PCIe lanes that are shared out into whatever
> configuration they claim to have and are totally unsuitable for use in a
> computer that actually has to be able to do high-speed I/O.
>
> --
> Be decisive. Make a decision, right or wrong. The road of life is paved
> with flat squirrels who could not make a decision.
>
> >-----Original Message-----
> >From: NANOG <nanog-bounces+kmedcalf=dessus.com@nanog.org> On Behalf Of
> >Eric Kuhnke
> >Sent: Saturday, 24 October, 2020 06:22
> >To: Jared Geiger <jared@compuwizz.net>
> >Cc: NANOG <nanog@nanog.org>
> >Subject: Re: Linux router network cards
> >
> >In addition to Jared's advice, I would recommend calculating PCI-Express
> >bandwidth bus points for whatever platform one is using.
> >
> >
> >For instance using the Intel X710-DA4, which could be capable in a
> >maximal scenario of 80Gbps of traffic, ensure it's in at least a PCI-E
> >3.0 x4 slot. And calculate the total number of PCI-E 3.0 x1 (or PCI-E 4.0
> >if a very new system) lanes that exist and are connected to the CPU. Big
> >difference between some options for Ryzen and Threadripper vs Intel CPUs,
> >towards the lower end of the cost range.
> >
> >
> >With recent Linux kernels if you have an Intel 510 or 710 series two or
> >four port card in a slot that can't support its full capability, you'll
> >get a warning in dmesg at boot time.
> >
> >
> >
> >On Thu, Oct 22, 2020 at 9:30 PM Jared Geiger <jared@compuwizz.net
> ><mailto:jared@compuwizz.net> > wrote:
> >
> >
> > I use DANOS with Intel XL710 10G NICs in DPDK mode for linux based
> >routing.
> >
> > If you're doing routing protocols, allocate 2 CPU cores to the
> >control plane and then a CPU core per 10G/1G interface for the dataplane,
> >plus an extra core for good measure. So for a 4 x 10G router taking in
> >full routes, 2 cores for control plane, 5 cores for the dataplane. Those
> >cores should be Intel Xeon E5-2600v3/4 or newer and faster the clocks,
> >the better.
> >
> > Similar CPU core allocations if you choose TNSR.
> >
> > On Thu, Oct 22, 2020 at 3:21 PM Jean St-Laurent via NANOG
> ><nanog@nanog.org <mailto:nanog@nanog.org> > wrote:
> >
> >
> > Chelsio cards are probably what you are looking for.
> >
> > https://www.chelsio.com/terminator-6-asic/
> >
> > It's closer to an asic than a traditional nic as the
> >router/firewall rules
> > are pushed directly into the hardware.
> >
> > I don't know how good they are with linux and they seem to
> be
> >compatible.
> > https://www.chelsio.com/linux/
> >
> > You will need to mess around a bit and fiddle here and
> there.
> >If you don't
> > mind using FreeBSD instead of linux, you could achieve a
> >smoother and more
> > integrated experience.
> >
> > Jean
> >
> > -----Original Message-----
> > From: NANOG <nanog-bounces+jean=ddostest.me@nanog.org
> ><mailto:ddostest.me@nanog.org> > On Behalf Of micah
> > anderson
> > Sent: Thursday, October 22, 2020 5:31 PM
> > To: Philip Loenneker <Philip.Loenneker@tasmanet.com.au
> ><mailto:Philip.Loenneker@tasmanet.com.au> >; NANOG
> > <nanog@nanog.org <mailto:nanog@nanog.org> >
> > Subject: RE: Linux router network cards
> >
> >
> > Thanks for the reply.
> >
> > Philip Loenneker <Philip.Loenneker@tasmanet.com.au
> ><mailto:Philip.Loenneker@tasmanet.com.au> > writes:
> > > Take a look at the Mellanox ConnectX 5 series of cards.
> They
> >handle
> > > DPDK, PVRDMA (basically SR-IOV that allows live migration
> >between
> > > hosts), and can even process packets within the NIC for
> some
> >
> > From what I can tell, SR-IOV/PVRDMA aren't really useful
> for
> >me in building
> > a router that wont be doing any virtualization.
> >
> > If the card can do DPDK, can it do XDP?
> >
> > > The slidedeck for the presentation is here:
> > > https://www.ausnog.net/sites/default/files/ausnog-
> >2019/presentations/1
> > > .9_Rhod_Brown_AusNOG2019.pdf
> > >
> > > It's heavily targeting virtualised workloads but some of
> the
> >feature sets
> > apply to bare-metal uses too.
> >
> > Yeah, this wont be a virtualized environment, just a router
> >passing packets,
> > dropping them, handling bgp and collecting flows.
> >
> > --
> > micah
> >
> >
>
>
>
>
>
Re: Linux router network cards [ In reply to ]
If building a lower end/low cost router this is absolutely a consideration.
In single socket regular ATX form factor, and products in the price range
of $165 for a motherboard and $250-400 price range for a CPU.

Comparing the PCI-E lanes available on an Intel Core i7 series to something
AMD zen/zen2 based (Ryzen), the AMD has greatly more. Some of the Intel
single socket core i5/i7 products have just enough PCI-E lanes for their
own onboard gigabit NIC and one PCI-E 3.0 x16 GPU for gaming purposes.

Would absolutely be a consideration if trying to build something with 8 to
12 10GbE interfaces capable of bursty traffic, but not flows and traffic
levels that would require line rate on all ports simultaneously.

On Sun, Oct 25, 2020 at 10:13 AM Vincent Bernat <bernat@luffy.cx> wrote:

> ? 24 octobre 2020 09:55 -06, Keith Medcalf:
>
> > And do not use an Intel CPU.
> >
> > Intel only has 4x PCIe lanes that are shared out into whatever
> > configuration they claim to have and are totally unsuitable for use in
> > a computer that actually has to be able to do high-speed I/O.
>
> That's likely to be incorrect. Intel CPU usually have 48 lanes for the
> Skylake generation. The 4 lanes limitation only applies to what is
> connected over DMI to the PCH, which is usually used for low-bandwidth
> stuff (1G NIC, SATA, 1x PCIe slots). Look at your motherboard manual to
> check how many lanes are affected to each component.
> --
> Make sure every module hides something.
> - The Elements of Programming Style (Kernighan & Plauger)
>