Mailing List Archive

SDN Internet Router (sir)
https://github.com/dbarrosop/sir


I came across this over the weekend. Given that the project was abandoned six years ago, are there any other efforts with a similar goal (more intelligently placing routes into FIBs of low-FIB capacity devices?




-----
Mike Hammett
Intelligent Computing Solutions

Midwest Internet Exchange

The Brothers WISP
Re: SDN Internet Router (sir) [ In reply to ]
It’s not a problem, due to cheap, plentiful high-speed memory and rapid prefix search silicon in backbone routers. The entire Internet routing table consumes at most a few gigabytes when fully structured (and only a few hundred Mbytes stored flat). That’s less memory than your average laptop sports.


Even in the worst case scenario, where every network decides to announce only its most specific prefixes, the BGP backbone would temporarily enter an oscillating state that generates a large number of routing updates into the inter-domain routing space. In this case, BGP route damping will quickly suppress the crazies while the backbone stabilizes.


Small routers should not be taking full tables, since there is no point to them being in the default free zone. For large routers, neither memory nor CPU speed are an issue. High-speed routers operating in the default-free zone have a critical path in the forwarding decision for each packet: it needs to take less than the inter-packet arrival time for minimum-sized IP packets.


This is easy to achieve with today’s hardware. A router line card with an aggregate line rate across all of its point-to-point interfaces of 10Tbps (readily available in today’s gear) can process packets with just a handful of cycles in the FIB Ternary Content Addressable Memory (TCAM) using ASIC-assisted lookups. TCAM is the most expensive component you’re paying for in such a router. It’s not cheap, but backbone routers don’t need to be cheap. They just need to not be memory-constrained.

-mel via cell

On Jan 3, 2023, at 7:47 AM, Mike Hammett <nanog@ics-il.net> wrote:

?
https://github.com/dbarrosop/sir

I came across this over the weekend. Given that the project was abandoned six years ago, are there any other efforts with a similar goal (more intelligently placing routes into FIBs of low-FIB capacity devices?



-----
Mike Hammett
Intelligent Computing Solutions<http://www.ics-il.com/>
[http://www.ics-il.com/images/fbicon.png]<https://www.facebook.com/ICSIL>[http://www.ics-il.com/images/googleicon.png]<https://plus.google.com/+IntelligentComputingSolutionsDeKalb>[http://www.ics-il.com/images/linkedinicon.png]<https://www.linkedin.com/company/intelligent-computing-solutions>[http://www.ics-il.com/images/twittericon.png]<https://twitter.com/ICSIL>
Midwest Internet Exchange<http://www.midwest-ix.com/>
[http://www.ics-il.com/images/fbicon.png]<https://www.facebook.com/mdwestix>[http://www.ics-il.com/images/linkedinicon.png]<https://www.linkedin.com/company/midwest-internet-exchange>[http://www.ics-il.com/images/twittericon.png]<https://twitter.com/mdwestix>
The Brothers WISP<http://www.thebrotherswisp.com/>
[http://www.ics-il.com/images/fbicon.png]<https://www.facebook.com/thebrotherswisp>[http://www.ics-il.com/images/youtubeicon.png]<https://www.youtube.com/channel/UCXSdfxQv7SpoRQYNyLwntZg>
Re: SDN Internet Router (sir) [ In reply to ]
https://github.com/sflow-rt/active-routes

Inspired by SIR, but uses Bird multi-table capability to separate RIB/FIB
routes.

On Tue, Jan 3, 2023 at 7:47 AM Mike Hammett <nanog@ics-il.net> wrote:

> https://github.com/dbarrosop/sir
>
> I came across this over the weekend. Given that the project was abandoned
> six years ago, are there any other efforts with a similar goal (more
> intelligently placing routes into FIBs of low-FIB capacity devices?
>
>
>
> -----
> Mike Hammett
> Intelligent Computing Solutions <http://www.ics-il.com/>
> <https://www.facebook.com/ICSIL>
> <https://plus.google.com/+IntelligentComputingSolutionsDeKalb>
> <https://www.linkedin.com/company/intelligent-computing-solutions>
> <https://twitter.com/ICSIL>
> Midwest Internet Exchange <http://www.midwest-ix.com/>
> <https://www.facebook.com/mdwestix>
> <https://www.linkedin.com/company/midwest-internet-exchange>
> <https://twitter.com/mdwestix>
> The Brothers WISP <http://www.thebrotherswisp.com/>
> <https://www.facebook.com/thebrotherswisp>
> <https://www.youtube.com/channel/UCXSdfxQv7SpoRQYNyLwntZg>
>
Re: SDN Internet Router (sir) [ In reply to ]
The problems of today are the same as the problems of yesterday, just with bigger numbers.




-----
Mike Hammett
Intelligent Computing Solutions

Midwest Internet Exchange

The Brothers WISP

----- Original Message -----

From: "Mel Beckman" <mel@beckman.org>
To: "Mike Hammett" <nanog@ics-il.net>
Cc: "NANOG" <nanog@nanog.org>
Sent: Tuesday, January 3, 2023 10:57:34 AM
Subject: Re: SDN Internet Router (sir)


It’s not a problem, due to cheap, plentiful high-speed memory and rapid prefix search silicon in backbone routers. The entire Internet routing table consumes at most a few gigabytes when fully structured (and only a few hundred Mbytes stored flat). That’s less memory than your average laptop sports.


Even in the worst case scenario, where every network decides to announce only its most specific prefixes, the BGP backbone would temporarily enter an oscillating state that generates a large number of routing updates into the inter-domain routing space. In this case, BGP route damping will quickly suppress the crazies while the backbone stabilizes.


Small routers should not be taking full tables, since there is no point to them being in the default free zone. For large routers, neither memory nor CPU speed are an issue. High-speed routers operating in the default-free zone have a critical path in the forwarding decision for each packet: it needs to take less than the inter-packet arrival time for minimum-sized IP packets.


This is easy to achieve with today’s hardware. A router line card with an aggregate line rate across all of its point-to-point interfaces of 10Tbps (readily available in today’s gear) can process packets with just a handful of cycles in the FIB Ternary Content Addressable Memory (TCAM) using ASIC-assisted lookups. TCAM is the most expensive component you’re paying for in such a router. It’s not cheap, but backbone routers don’t need to be cheap. They just need to not be memory-constrained.

-mel via cell



On Jan 3, 2023, at 7:47 AM, Mike Hammett <nanog@ics-il.net> wrote:




<blockquote>


https://github.com/dbarrosop/sir


I came across this over the weekend. Given that the project was abandoned six years ago, are there any other efforts with a similar goal (more intelligently placing routes into FIBs of low-FIB capacity devices?




-----
Mike Hammett
Intelligent Computing Solutions

Midwest Internet Exchange

The Brothers WISP


</blockquote>
Re: SDN Internet Router (sir) [ In reply to ]
There are likely more networks with 10 gigabit or less total external capacity than there are with more.


Creating imaginary lines in the sand doesn't really help anyone.




-----
Mike Hammett
Intelligent Computing Solutions

Midwest Internet Exchange

The Brothers WISP

----- Original Message -----

From: "Mel Beckman" <mel@beckman.org>
To: "Mike Hammett" <nanog@ics-il.net>
Cc: "NANOG" <nanog@nanog.org>
Sent: Tuesday, January 3, 2023 10:57:34 AM
Subject: Re: SDN Internet Router (sir)


It’s not a problem, due to cheap, plentiful high-speed memory and rapid prefix search silicon in backbone routers. The entire Internet routing table consumes at most a few gigabytes when fully structured (and only a few hundred Mbytes stored flat). That’s less memory than your average laptop sports.


Even in the worst case scenario, where every network decides to announce only its most specific prefixes, the BGP backbone would temporarily enter an oscillating state that generates a large number of routing updates into the inter-domain routing space. In this case, BGP route damping will quickly suppress the crazies while the backbone stabilizes.


Small routers should not be taking full tables, since there is no point to them being in the default free zone. For large routers, neither memory nor CPU speed are an issue. High-speed routers operating in the default-free zone have a critical path in the forwarding decision for each packet: it needs to take less than the inter-packet arrival time for minimum-sized IP packets.


This is easy to achieve with today’s hardware. A router line card with an aggregate line rate across all of its point-to-point interfaces of 10Tbps (readily available in today’s gear) can process packets with just a handful of cycles in the FIB Ternary Content Addressable Memory (TCAM) using ASIC-assisted lookups. TCAM is the most expensive component you’re paying for in such a router. It’s not cheap, but backbone routers don’t need to be cheap. They just need to not be memory-constrained.

-mel via cell



On Jan 3, 2023, at 7:47 AM, Mike Hammett <nanog@ics-il.net> wrote:




<blockquote>


https://github.com/dbarrosop/sir


I came across this over the weekend. Given that the project was abandoned six years ago, are there any other efforts with a similar goal (more intelligently placing routes into FIBs of low-FIB capacity devices?




-----
Mike Hammett
Intelligent Computing Solutions

Midwest Internet Exchange

The Brothers WISP


</blockquote>
Re: SDN Internet Router (sir) [ In reply to ]
If your BGP router supports sflow or netflow and includes ASN in those, you
can use
akvorado, as-stat, or elastiflow.

Le mar. 3 janv. 2023 à 12:16, Peter Phaal <peter.phaal@gmail.com> a écrit :

> https://github.com/sflow-rt/active-routes
>
> Inspired by SIR, but uses Bird multi-table capability to separate RIB/FIB
> routes.
>
> On Tue, Jan 3, 2023 at 7:47 AM Mike Hammett <nanog@ics-il.net> wrote:
>
>> https://github.com/dbarrosop/sir
>>
>> I came across this over the weekend. Given that the project was abandoned
>> six years ago, are there any other efforts with a similar goal (more
>> intelligently placing routes into FIBs of low-FIB capacity devices?
>>
>>
>>
>> -----
>> Mike Hammett
>> Intelligent Computing Solutions <http://www.ics-il.com/>
>> <https://www.facebook.com/ICSIL>
>> <https://plus.google.com/+IntelligentComputingSolutionsDeKalb>
>> <https://www.linkedin.com/company/intelligent-computing-solutions>
>> <https://twitter.com/ICSIL>
>> Midwest Internet Exchange <http://www.midwest-ix.com/>
>> <https://www.facebook.com/mdwestix>
>> <https://www.linkedin.com/company/midwest-internet-exchange>
>> <https://twitter.com/mdwestix>
>> The Brothers WISP <http://www.thebrotherswisp.com/>
>> <https://www.facebook.com/thebrotherswisp>
>> <https://www.youtube.com/channel/UCXSdfxQv7SpoRQYNyLwntZg>
>>
>
Re: SDN Internet Router (sir) [ In reply to ]
Disagree that it’s a line in the sand. It’s use the right tool for the job.

If a device is low FIB, it’s that way for a reason. There are plenty of
ways to massage that with policy and software, depending on capabilities ,
but at the end of the day, trying to sort 10 pounds of shit to store in a 5
pound bag is eventually going to end up the same way.

On Tue, Jan 3, 2023 at 13:18 Mike Hammett <nanog@ics-il.net> wrote:

> There are likely more networks with 10 gigabit or less total external
> capacity than there are with more.
>
> Creating imaginary lines in the sand doesn't really help anyone.
>
>
>
>
> -----
> Mike Hammett
> Intelligent Computing Solutions <http://www.ics-il.com/>
> <https://www.facebook.com/ICSIL>
> <https://plus.google.com/+IntelligentComputingSolutionsDeKalb>
> <https://www.linkedin.com/company/intelligent-computing-solutions>
> <https://twitter.com/ICSIL>
> Midwest Internet Exchange <http://www.midwest-ix.com/>
> <https://www.facebook.com/mdwestix>
> <https://www.linkedin.com/company/midwest-internet-exchange>
> <https://twitter.com/mdwestix>
> The Brothers WISP <http://www.thebrotherswisp.com/>
> <https://www.facebook.com/thebrotherswisp>
> <https://www.youtube.com/channel/UCXSdfxQv7SpoRQYNyLwntZg>
> ------------------------------
> *From: *"Mel Beckman" <mel@beckman.org>
> *To: *"Mike Hammett" <nanog@ics-il.net>
> *Cc: *"NANOG" <nanog@nanog.org>
> *Sent: *Tuesday, January 3, 2023 10:57:34 AM
> *Subject: *Re: SDN Internet Router (sir)
>
> It’s not a problem, due to cheap, plentiful high-speed memory and rapid
> prefix search silicon in backbone routers. The entire Internet routing
> table consumes at most a few gigabytes when fully structured (and only a
> few hundred Mbytes stored flat). That’s less memory than your average
> laptop sports.
>
>
> Even in the worst case scenario, where every network decides to announce
> only its most specific prefixes, the BGP backbone would temporarily enter
> an oscillating state that generates a large number of routing updates into
> the inter-domain routing space. In this case, BGP route damping will
> quickly suppress the crazies while the backbone stabilizes.
>
>
> Small routers should not be taking full tables, since there is no point to
> them being in the default free zone. For large routers, neither memory nor
> CPU speed are an issue. High-speed routers operating in the default-free
> zone have a critical path in the forwarding decision for each packet: it
> needs to take less than the inter-packet arrival time for minimum-sized IP
> packets.
>
>
> This is easy to achieve with today’s hardware. A router line card with an
> aggregate line rate across all of its point-to-point interfaces of 10Tbps
> (readily available in today’s gear) can process packets with just a handful
> of cycles in the FIB Ternary Content Addressable Memory (TCAM) using
> ASIC-assisted lookups. TCAM is the most expensive component you’re paying
> for in such a router. It’s not cheap, but backbone routers don’t need
> to be cheap. They just need to not be memory-constrained.
>
> -mel via cell
>
> On Jan 3, 2023, at 7:47 AM, Mike Hammett <nanog@ics-il.net> wrote:
>
> ?
> https://github.com/dbarrosop/sir
>
> I came across this over the weekend. Given that the project was abandoned
> six years ago, are there any other efforts with a similar goal (more
> intelligently placing routes into FIBs of low-FIB capacity devices?
>
>
>
> -----
> Mike Hammett
> Intelligent Computing Solutions <http://www.ics-il.com/>
> <https://www.facebook.com/ICSIL>
> <https://plus.google.com/+IntelligentComputingSolutionsDeKalb>
> <https://www.linkedin.com/company/intelligent-computing-solutions>
> <https://twitter.com/ICSIL>
> Midwest Internet Exchange <http://www.midwest-ix.com/>
> <https://www.facebook.com/mdwestix>
> <https://www.linkedin.com/company/midwest-internet-exchange>
> <https://twitter.com/mdwestix>
> The Brothers WISP <http://www.thebrotherswisp.com/>
> <https://www.facebook.com/thebrotherswisp>
> <https://www.youtube.com/channel/UCXSdfxQv7SpoRQYNyLwntZg>
>
>
>
Re: SDN Internet Router (sir) [ In reply to ]
On Wed, Jan 4, 2023 at 8:37 AM Tom Beecher <beecher@beecher.cc> wrote:

> Disagree that it’s a line in the sand. It’s use the right tool for the
> job.
>
> If a device is low FIB, it’s that way for a reason. There are plenty of
> ways to massage that with policy and software, depending on capabilities ,
> but at the end of the day, trying to sort 10 pounds of shit to store in a 5
> pound bag is eventually going to end up the same way.
>

Some of the reasoning behind 'i need/want to do SDN things' is 'low fib
device' sort of reasonings. Some is: "I just want a forwarding decision
that's not entirely LPM oriented"
(or to be fair: "My LPM is not JUST longest prefix, but also some other
data")

For folk that are looking for software alternatives you might look at
faucet:
https://github.com/faucetsdn/faucet

which still seems to be actively developed.
Re: SDN Internet Router (sir) [ In reply to ]
"The right tool for the job" gets into a religious argument in assuming that one's way to do the job is the only reasonable way to do the job.


Large networks historically have a very poor (IMO) model of gigantic iron in a few locations, which results in sub-optimal routing for the rest of their network between those large POPs. I've heard time and time again that someone buying service from a major network in say New Orleans has a first hop of Dallas or Atlanta. I agree that full-route capable routers need to be in the large, central locations, but it isn't cost effective to have them at every POP, especially if you're a last-mile provider.


I'd go into more examples of where it doesn't make sense to have full-route routers everywhere, but I'm afraid that the Internet would then focus on the examples instead of the core idea of intelligently putting routes into the FIBs of low-FIB routers throughout my network.




-----
Mike Hammett
Intelligent Computing Solutions

Midwest Internet Exchange

The Brothers WISP

----- Original Message -----

From: "Tom Beecher" <beecher@beecher.cc>
To: "Mike Hammett" <nanog@ics-il.net>
Cc: "Mel Beckman" <mel@beckman.org>, "NANOG" <nanog@nanog.org>
Sent: Wednesday, January 4, 2023 7:36:58 AM
Subject: Re: SDN Internet Router (sir)


Disagree that it’s a line in the sand. It’s use the right tool for the job.


If a device is low FIB, it’s that way for a reason. There are plenty of ways to massage that with policy and software, depending on capabilities , but at the end of the day, trying to sort 10 pounds of shit to store in a 5 pound bag is eventually going to end up the same way.



On Tue, Jan 3, 2023 at 13:18 Mike Hammett < nanog@ics-il.net > wrote:




There are likely more networks with 10 gigabit or less total external capacity than there are with more.


Creating imaginary lines in the sand doesn't really help anyone.







-----
Mike Hammett
Intelligent Computing Solutions

Midwest Internet Exchange

The Brothers WISP



From: "Mel Beckman" < mel@beckman.org >
To: "Mike Hammett" < nanog@ics-il.net >
Cc: "NANOG" < nanog@nanog.org >
Sent: Tuesday, January 3, 2023 10:57:34 AM
Subject: Re: SDN Internet Router (sir)


It’s not a problem, due to cheap, plentiful high-speed memory and rapid prefix search silicon in backbone routers. The entire Internet routing table consumes at most a few gigabytes when fully structured (and only a few hundred Mbytes stored flat). That’s less memory than your average laptop sports.


Even in the worst case scenario, where every network decides to announce only its most specific prefixes, the BGP backbone would temporarily enter an oscillating state that generates a large number of routing updates into the inter-domain routing space. In this case, BGP route damping will quickly suppress the crazies while the backbone stabilizes.


Small routers should not be taking full tables, since there is no point to them being in the default free zone. For large routers, neither memory nor CPU speed are an issue. High-speed routers operating in the default-free zone have a critical path in the forwarding decision for each packet: it needs to take less than the inter-packet arrival time for minimum-sized IP packets.


This is easy to achieve with today’s hardware. A router line card with an aggregate line rate across all of its point-to-point interfaces of 10Tbps (readily available in today’s gear) can process packets with just a handful of cycles in the FIB Ternary Content Addressable Memory (TCAM) using ASIC-assisted lookups. TCAM is the most expensive component you’re paying for in such a router. It’s not cheap, but backbone routers don’t need to be cheap. They just need to not be memory-constrained.

-mel via cell


<blockquote>
On Jan 3, 2023, at 7:47 AM, Mike Hammett < nanog@ics-il.net > wrote:




<blockquote>


https://github.com/dbarrosop/sir


I came across this over the weekend. Given that the project was abandoned six years ago, are there any other efforts with a similar goal (more intelligently placing routes into FIBs of low-FIB capacity devices?




-----
Mike Hammett
Intelligent Computing Solutions

Midwest Internet Exchange

The Brothers WISP


</blockquote>


</blockquote>
Re: SDN Internet Router (sir) [ In reply to ]
>
> "The right tool for the job" gets into a religious argument in assuming
> that one's way to do the job is the only reasonable way to do the job


I disagree that it's religious. I completely agree there are locations in
networks that having full DFZ capable routers doesn't make technical or
economic sense. But there have long been different products for those
different use cases.

To perhaps explain my viewpoint better,(and perhaps I didn't properly
comprehend the problem you're aiming to solve) :

If you are trying to use SDN stuff to shuffle routes on and off a box
because you have the wrong sized routers in place, then I would argue
you're doing it wrong.

If you are trying to use SDN stuff to (as Christopher mentioned) make
decisions that are not strictly LPM, and the equipment you have cannot do
that, then that's different and entirely reasonable.

If the second use case is more of what you were asking, then I apologize
for misunderstanding.



On Thu, Jan 5, 2023 at 9:57 AM Mike Hammett <nanog@ics-il.net> wrote:

> "The right tool for the job" gets into a religious argument in assuming
> that one's way to do the job is the only reasonable way to do the job.
>
> Large networks historically have a very poor (IMO) model of gigantic iron
> in a few locations, which results in sub-optimal routing for the rest of
> their network between those large POPs. I've heard time and time again that
> someone buying service from a major network in say New Orleans has a first
> hop of Dallas or Atlanta. I agree that full-route capable routers need to
> be in the large, central locations, but it isn't cost effective to have
> them at every POP, especially if you're a last-mile provider.
>
> I'd go into more examples of where it doesn't make sense to have
> full-route routers everywhere, but I'm afraid that the Internet would then
> focus on the examples instead of the core idea of intelligently putting
> routes into the FIBs of low-FIB routers throughout my network.
>
>
>
> -----
> Mike Hammett
> Intelligent Computing Solutions <http://www.ics-il.com/>
> <https://www.facebook.com/ICSIL>
> <https://plus.google.com/+IntelligentComputingSolutionsDeKalb>
> <https://www.linkedin.com/company/intelligent-computing-solutions>
> <https://twitter.com/ICSIL>
> Midwest Internet Exchange <http://www.midwest-ix.com/>
> <https://www.facebook.com/mdwestix>
> <https://www.linkedin.com/company/midwest-internet-exchange>
> <https://twitter.com/mdwestix>
> The Brothers WISP <http://www.thebrotherswisp.com/>
> <https://www.facebook.com/thebrotherswisp>
> <https://www.youtube.com/channel/UCXSdfxQv7SpoRQYNyLwntZg>
> ------------------------------
> *From: *"Tom Beecher" <beecher@beecher.cc>
> *To: *"Mike Hammett" <nanog@ics-il.net>
> *Cc: *"Mel Beckman" <mel@beckman.org>, "NANOG" <nanog@nanog.org>
> *Sent: *Wednesday, January 4, 2023 7:36:58 AM
> *Subject: *Re: SDN Internet Router (sir)
>
> Disagree that it’s a line in the sand. It’s use the right tool for the
> job.
>
> If a device is low FIB, it’s that way for a reason. There are plenty of
> ways to massage that with policy and software, depending on capabilities ,
> but at the end of the day, trying to sort 10 pounds of shit to store in a 5
> pound bag is eventually going to end up the same way.
>
> On Tue, Jan 3, 2023 at 13:18 Mike Hammett <nanog@ics-il.net> wrote:
>
>> There are likely more networks with 10 gigabit or less total external
>> capacity than there are with more.
>>
>> Creating imaginary lines in the sand doesn't really help anyone.
>>
>>
>>
>>
>> -----
>> Mike Hammett
>> Intelligent Computing Solutions <http://www.ics-il.com/>
>> <https://www.facebook.com/ICSIL>
>> <https://plus.google.com/+IntelligentComputingSolutionsDeKalb>
>> <https://www.linkedin.com/company/intelligent-computing-solutions>
>> <https://twitter.com/ICSIL>
>> Midwest Internet Exchange <http://www.midwest-ix.com/>
>> <https://www.facebook.com/mdwestix>
>> <https://www.linkedin.com/company/midwest-internet-exchange>
>> <https://twitter.com/mdwestix>
>> The Brothers WISP <http://www.thebrotherswisp.com/>
>> <https://www.facebook.com/thebrotherswisp>
>> <https://www.youtube.com/channel/UCXSdfxQv7SpoRQYNyLwntZg>
>> ------------------------------
>> *From: *"Mel Beckman" <mel@beckman.org>
>> *To: *"Mike Hammett" <nanog@ics-il.net>
>> *Cc: *"NANOG" <nanog@nanog.org>
>> *Sent: *Tuesday, January 3, 2023 10:57:34 AM
>> *Subject: *Re: SDN Internet Router (sir)
>>
>> It’s not a problem, due to cheap, plentiful high-speed memory and rapid
>> prefix search silicon in backbone routers. The entire Internet routing
>> table consumes at most a few gigabytes when fully structured (and only a
>> few hundred Mbytes stored flat). That’s less memory than your average
>> laptop sports.
>>
>>
>> Even in the worst case scenario, where every network decides to announce
>> only its most specific prefixes, the BGP backbone would temporarily enter
>> an oscillating state that generates a large number of routing updates into
>> the inter-domain routing space. In this case, BGP route damping will
>> quickly suppress the crazies while the backbone stabilizes.
>>
>>
>> Small routers should not be taking full tables, since there is no point
>> to them being in the default free zone. For large routers, neither memory
>> nor CPU speed are an issue. High-speed routers operating in the
>> default-free zone have a critical path in the forwarding decision for each
>> packet: it needs to take less than the inter-packet arrival time for
>> minimum-sized IP packets.
>>
>>
>> This is easy to achieve with today’s hardware. A router line card with an
>> aggregate line rate across all of its point-to-point interfaces of 10Tbps
>> (readily available in today’s gear) can process packets with just a handful
>> of cycles in the FIB Ternary Content Addressable Memory (TCAM) using
>> ASIC-assisted lookups. TCAM is the most expensive component you’re paying
>> for in such a router. It’s not cheap, but backbone routers don’t need
>> to be cheap. They just need to not be memory-constrained.
>>
>> -mel via cell
>>
>> On Jan 3, 2023, at 7:47 AM, Mike Hammett <nanog@ics-il.net> wrote:
>>
>> ?
>> https://github.com/dbarrosop/sir
>>
>> I came across this over the weekend. Given that the project was abandoned
>> six years ago, are there any other efforts with a similar goal (more
>> intelligently placing routes into FIBs of low-FIB capacity devices?
>>
>>
>>
>> -----
>> Mike Hammett
>> Intelligent Computing Solutions <http://www.ics-il.com/>
>> <https://www.facebook.com/ICSIL>
>> <https://plus.google.com/+IntelligentComputingSolutionsDeKalb>
>> <https://www.linkedin.com/company/intelligent-computing-solutions>
>> <https://twitter.com/ICSIL>
>> Midwest Internet Exchange <http://www.midwest-ix.com/>
>> <https://www.facebook.com/mdwestix>
>> <https://www.linkedin.com/company/midwest-internet-exchange>
>> <https://twitter.com/mdwestix>
>> The Brothers WISP <http://www.thebrotherswisp.com/>
>> <https://www.facebook.com/thebrotherswisp>
>> <https://www.youtube.com/channel/UCXSdfxQv7SpoRQYNyLwntZg>
>>
>>
>>
>
Re: SDN Internet Router (sir) [ In reply to ]
Then please bless the world with the right way.


You acknowledge that not every router in a network needs to be fully DFZ capable, but then crap on my desire to have more than a default route in one.




-----
Mike Hammett
Intelligent Computing Solutions

Midwest Internet Exchange

The Brothers WISP

----- Original Message -----

From: "Tom Beecher" <beecher@beecher.cc>
To: "Mike Hammett" <nanog@ics-il.net>
Cc: "Mel Beckman" <mel@beckman.org>, "NANOG" <nanog@nanog.org>
Sent: Thursday, January 5, 2023 9:55:38 AM
Subject: Re: SDN Internet Router (sir)




"The right tool for the job" gets into a religious argument in assuming that one's way to do the job is the only reasonable way to do the job




I disagree that it's religious. I completely agree there are locations in networks that having full DFZ capable routers doesn't make technical or economic sense. But there have long been different products for those different use cases.


To perhaps explain my viewpoint better,(and perhaps I didn't properly comprehend the problem you're aiming to solve) :


If you are trying to use SDN stuff to shuffle routes on and off a box because you have the wrong sized routers in place, then I would argue you're doing it wrong.


If you are trying to use SDN stuff to (as Christopher mentioned) make decisions that are not strictly LPM, and the equipment you have cannot do that, then that's different and entirely reasonable.


If the second use case is more of what you were asking, then I apologize for misunderstanding.





On Thu, Jan 5, 2023 at 9:57 AM Mike Hammett < nanog@ics-il.net > wrote:

<blockquote>



"The right tool for the job" gets into a religious argument in assuming that one's way to do the job is the only reasonable way to do the job.


Large networks historically have a very poor (IMO) model of gigantic iron in a few locations, which results in sub-optimal routing for the rest of their network between those large POPs. I've heard time and time again that someone buying service from a major network in say New Orleans has a first hop of Dallas or Atlanta. I agree that full-route capable routers need to be in the large, central locations, but it isn't cost effective to have them at every POP, especially if you're a last-mile provider.


I'd go into more examples of where it doesn't make sense to have full-route routers everywhere, but I'm afraid that the Internet would then focus on the examples instead of the core idea of intelligently putting routes into the FIBs of low-FIB routers throughout my network.




-----
Mike Hammett
Intelligent Computing Solutions

Midwest Internet Exchange

The Brothers WISP



From: "Tom Beecher" < beecher@beecher.cc >
To: "Mike Hammett" < nanog@ics-il.net >
Cc: "Mel Beckman" < mel@beckman.org >, "NANOG" < nanog@nanog.org >
Sent: Wednesday, January 4, 2023 7:36:58 AM
Subject: Re: SDN Internet Router (sir)


Disagree that it’s a line in the sand. It’s use the right tool for the job.


If a device is low FIB, it’s that way for a reason. There are plenty of ways to massage that with policy and software, depending on capabilities , but at the end of the day, trying to sort 10 pounds of shit to store in a 5 pound bag is eventually going to end up the same way.



On Tue, Jan 3, 2023 at 13:18 Mike Hammett < nanog@ics-il.net > wrote:

<blockquote>


There are likely more networks with 10 gigabit or less total external capacity than there are with more.


Creating imaginary lines in the sand doesn't really help anyone.







-----
Mike Hammett
Intelligent Computing Solutions

Midwest Internet Exchange

The Brothers WISP



From: "Mel Beckman" < mel@beckman.org >
To: "Mike Hammett" < nanog@ics-il.net >
Cc: "NANOG" < nanog@nanog.org >
Sent: Tuesday, January 3, 2023 10:57:34 AM
Subject: Re: SDN Internet Router (sir)


It’s not a problem, due to cheap, plentiful high-speed memory and rapid prefix search silicon in backbone routers. The entire Internet routing table consumes at most a few gigabytes when fully structured (and only a few hundred Mbytes stored flat). That’s less memory than your average laptop sports.


Even in the worst case scenario, where every network decides to announce only its most specific prefixes, the BGP backbone would temporarily enter an oscillating state that generates a large number of routing updates into the inter-domain routing space. In this case, BGP route damping will quickly suppress the crazies while the backbone stabilizes.


Small routers should not be taking full tables, since there is no point to them being in the default free zone. For large routers, neither memory nor CPU speed are an issue. High-speed routers operating in the default-free zone have a critical path in the forwarding decision for each packet: it needs to take less than the inter-packet arrival time for minimum-sized IP packets.


This is easy to achieve with today’s hardware. A router line card with an aggregate line rate across all of its point-to-point interfaces of 10Tbps (readily available in today’s gear) can process packets with just a handful of cycles in the FIB Ternary Content Addressable Memory (TCAM) using ASIC-assisted lookups. TCAM is the most expensive component you’re paying for in such a router. It’s not cheap, but backbone routers don’t need to be cheap. They just need to not be memory-constrained.

-mel via cell


<blockquote>
On Jan 3, 2023, at 7:47 AM, Mike Hammett < nanog@ics-il.net > wrote:


</blockquote>

<blockquote>


https://github.com/dbarrosop/sir


I came across this over the weekend. Given that the project was abandoned six years ago, are there any other efforts with a similar goal (more intelligently placing routes into FIBs of low-FIB capacity devices?




-----
Mike Hammett
Intelligent Computing Solutions

Midwest Internet Exchange

The Brothers WISP


</blockquote>


</blockquote>


</blockquote>
Re: SDN Internet Router (sir) [ In reply to ]
Initially, my thought was to use community filtering to push just IXes, customers, and defaults throughout the network, but that's obviously still sub-optimal.


I'd be surprised if a last mile network had a ton of traffic going to any more than a few hundred prefixes.




-----
Mike Hammett
Intelligent Computing Solutions

Midwest Internet Exchange

The Brothers WISP

----- Original Message -----

From: "Mike Hammett" <nanog@ics-il.net>
To: "Tom Beecher" <beecher@beecher.cc>
Cc: "NANOG" <nanog@nanog.org>
Sent: Thursday, January 5, 2023 10:02:40 AM
Subject: Re: SDN Internet Router (sir)


Then please bless the world with the right way.


You acknowledge that not every router in a network needs to be fully DFZ capable, but then crap on my desire to have more than a default route in one.




-----
Mike Hammett
Intelligent Computing Solutions

Midwest Internet Exchange

The Brothers WISP

----- Original Message -----

From: "Tom Beecher" <beecher@beecher.cc>
To: "Mike Hammett" <nanog@ics-il.net>
Cc: "Mel Beckman" <mel@beckman.org>, "NANOG" <nanog@nanog.org>
Sent: Thursday, January 5, 2023 9:55:38 AM
Subject: Re: SDN Internet Router (sir)




"The right tool for the job" gets into a religious argument in assuming that one's way to do the job is the only reasonable way to do the job




I disagree that it's religious. I completely agree there are locations in networks that having full DFZ capable routers doesn't make technical or economic sense. But there have long been different products for those different use cases.


To perhaps explain my viewpoint better,(and perhaps I didn't properly comprehend the problem you're aiming to solve) :


If you are trying to use SDN stuff to shuffle routes on and off a box because you have the wrong sized routers in place, then I would argue you're doing it wrong.


If you are trying to use SDN stuff to (as Christopher mentioned) make decisions that are not strictly LPM, and the equipment you have cannot do that, then that's different and entirely reasonable.


If the second use case is more of what you were asking, then I apologize for misunderstanding.





On Thu, Jan 5, 2023 at 9:57 AM Mike Hammett < nanog@ics-il.net > wrote:

<blockquote>



"The right tool for the job" gets into a religious argument in assuming that one's way to do the job is the only reasonable way to do the job.


Large networks historically have a very poor (IMO) model of gigantic iron in a few locations, which results in sub-optimal routing for the rest of their network between those large POPs. I've heard time and time again that someone buying service from a major network in say New Orleans has a first hop of Dallas or Atlanta. I agree that full-route capable routers need to be in the large, central locations, but it isn't cost effective to have them at every POP, especially if you're a last-mile provider.


I'd go into more examples of where it doesn't make sense to have full-route routers everywhere, but I'm afraid that the Internet would then focus on the examples instead of the core idea of intelligently putting routes into the FIBs of low-FIB routers throughout my network.




-----
Mike Hammett
Intelligent Computing Solutions

Midwest Internet Exchange

The Brothers WISP



From: "Tom Beecher" < beecher@beecher.cc >
To: "Mike Hammett" < nanog@ics-il.net >
Cc: "Mel Beckman" < mel@beckman.org >, "NANOG" < nanog@nanog.org >
Sent: Wednesday, January 4, 2023 7:36:58 AM
Subject: Re: SDN Internet Router (sir)


Disagree that it’s a line in the sand. It’s use the right tool for the job.


If a device is low FIB, it’s that way for a reason. There are plenty of ways to massage that with policy and software, depending on capabilities , but at the end of the day, trying to sort 10 pounds of shit to store in a 5 pound bag is eventually going to end up the same way.



On Tue, Jan 3, 2023 at 13:18 Mike Hammett < nanog@ics-il.net > wrote:

<blockquote>


There are likely more networks with 10 gigabit or less total external capacity than there are with more.


Creating imaginary lines in the sand doesn't really help anyone.







-----
Mike Hammett
Intelligent Computing Solutions

Midwest Internet Exchange

The Brothers WISP



From: "Mel Beckman" < mel@beckman.org >
To: "Mike Hammett" < nanog@ics-il.net >
Cc: "NANOG" < nanog@nanog.org >
Sent: Tuesday, January 3, 2023 10:57:34 AM
Subject: Re: SDN Internet Router (sir)


It’s not a problem, due to cheap, plentiful high-speed memory and rapid prefix search silicon in backbone routers. The entire Internet routing table consumes at most a few gigabytes when fully structured (and only a few hundred Mbytes stored flat). That’s less memory than your average laptop sports.


Even in the worst case scenario, where every network decides to announce only its most specific prefixes, the BGP backbone would temporarily enter an oscillating state that generates a large number of routing updates into the inter-domain routing space. In this case, BGP route damping will quickly suppress the crazies while the backbone stabilizes.


Small routers should not be taking full tables, since there is no point to them being in the default free zone. For large routers, neither memory nor CPU speed are an issue. High-speed routers operating in the default-free zone have a critical path in the forwarding decision for each packet: it needs to take less than the inter-packet arrival time for minimum-sized IP packets.


This is easy to achieve with today’s hardware. A router line card with an aggregate line rate across all of its point-to-point interfaces of 10Tbps (readily available in today’s gear) can process packets with just a handful of cycles in the FIB Ternary Content Addressable Memory (TCAM) using ASIC-assisted lookups. TCAM is the most expensive component you’re paying for in such a router. It’s not cheap, but backbone routers don’t need to be cheap. They just need to not be memory-constrained.

-mel via cell


<blockquote>
On Jan 3, 2023, at 7:47 AM, Mike Hammett < nanog@ics-il.net > wrote:


</blockquote>

<blockquote>


https://github.com/dbarrosop/sir


I came across this over the weekend. Given that the project was abandoned six years ago, are there any other efforts with a similar goal (more intelligently placing routes into FIBs of low-FIB capacity devices?




-----
Mike Hammett
Intelligent Computing Solutions

Midwest Internet Exchange

The Brothers WISP


</blockquote>


</blockquote>


</blockquote>
Re: SDN Internet Router (sir) [ In reply to ]
?Mike,

Your original question was:

“Given that the project was abandoned six years ago, are there any other efforts with a similar goal (more intelligently placing routes into FIBs of low-FIB capacity devices?”

People then, respectfully, tried to clarify your request or explain why placing routes in a low-FIB capacity device isn’t seen as being beneficial. Only now have you added the desire to simply have “more than a default route” in such a router.

You can, of course, have more than a default route today - e.g., through local pref and BGP communities for things such as company routes. You haven’t said what you define as “more intelligently”, so perhaps you can more clearly explain the problem you see with the current BGP capabilities via some examples.

-mel

On Jan 5, 2023, at 8:02 AM, Mike Hammett <nanog@ics-il.net> wrote:

?
Then please bless the world with the right way.

You acknowledge that not every router in a network needs to be fully DFZ capable, but then crap on my desire to have more than a default route in one.



-----
Mike Hammett
Intelligent Computing Solutions<http://www.ics-il.com/>
[http://www.ics-il.com/images/fbicon.png]<https://www.facebook.com/ICSIL>[http://www.ics-il.com/images/googleicon.png]<https://plus.google.com/+IntelligentComputingSolutionsDeKalb>[http://www.ics-il.com/images/linkedinicon.png]<https://www.linkedin.com/company/intelligent-computing-solutions>[http://www.ics-il.com/images/twittericon.png]<https://twitter.com/ICSIL>
Midwest Internet Exchange<http://www.midwest-ix.com/>
[http://www.ics-il.com/images/fbicon.png]<https://www.facebook.com/mdwestix>[http://www.ics-il.com/images/linkedinicon.png]<https://www.linkedin.com/company/midwest-internet-exchange>[http://www.ics-il.com/images/twittericon.png]<https://twitter.com/mdwestix>
The Brothers WISP<http://www.thebrotherswisp.com/>
[http://www.ics-il.com/images/fbicon.png]<https://www.facebook.com/thebrotherswisp>[http://www.ics-il.com/images/youtubeicon.png]<https://www.youtube.com/channel/UCXSdfxQv7SpoRQYNyLwntZg>
________________________________
From: "Tom Beecher" <beecher@beecher.cc>
To: "Mike Hammett" <nanog@ics-il.net>
Cc: "Mel Beckman" <mel@beckman.org>, "NANOG" <nanog@nanog.org>
Sent: Thursday, January 5, 2023 9:55:38 AM
Subject: Re: SDN Internet Router (sir)

"The right tool for the job" gets into a religious argument in assuming that one's way to do the job is the only reasonable way to do the job

I disagree that it's religious. I completely agree there are locations in networks that having full DFZ capable routers doesn't make technical or economic sense. But there have long been different products for those different use cases.

To perhaps explain my viewpoint better,(and perhaps I didn't properly comprehend the problem you're aiming to solve) :

If you are trying to use SDN stuff to shuffle routes on and off a box because you have the wrong sized routers in place, then I would argue you're doing it wrong.

If you are trying to use SDN stuff to (as Christopher mentioned) make decisions that are not strictly LPM, and the equipment you have cannot do that, then that's different and entirely reasonable.

If the second use case is more of what you were asking, then I apologize for misunderstanding.



On Thu, Jan 5, 2023 at 9:57 AM Mike Hammett <nanog@ics-il.net<mailto:nanog@ics-il.net>> wrote:
"The right tool for the job" gets into a religious argument in assuming that one's way to do the job is the only reasonable way to do the job.

Large networks historically have a very poor (IMO) model of gigantic iron in a few locations, which results in sub-optimal routing for the rest of their network between those large POPs. I've heard time and time again that someone buying service from a major network in say New Orleans has a first hop of Dallas or Atlanta. I agree that full-route capable routers need to be in the large, central locations, but it isn't cost effective to have them at every POP, especially if you're a last-mile provider.

I'd go into more examples of where it doesn't make sense to have full-route routers everywhere, but I'm afraid that the Internet would then focus on the examples instead of the core idea of intelligently putting routes into the FIBs of low-FIB routers throughout my network.



-----
Mike Hammett
Intelligent Computing Solutions<http://www.ics-il.com/>
[http://www.ics-il.com/images/fbicon.png]<https://www.facebook.com/ICSIL>[http://www.ics-il.com/images/googleicon.png]<https://plus.google.com/+IntelligentComputingSolutionsDeKalb>[http://www.ics-il.com/images/linkedinicon.png]<https://www.linkedin.com/company/intelligent-computing-solutions>[http://www.ics-il.com/images/twittericon.png]<https://twitter.com/ICSIL>
Midwest Internet Exchange<http://www.midwest-ix.com/>
[http://www.ics-il.com/images/fbicon.png]<https://www.facebook.com/mdwestix>[http://www.ics-il.com/images/linkedinicon.png]<https://www.linkedin.com/company/midwest-internet-exchange>[http://www.ics-il.com/images/twittericon.png]<https://twitter.com/mdwestix>
The Brothers WISP<http://www.thebrotherswisp.com/>
[http://www.ics-il.com/images/fbicon.png]<https://www.facebook.com/thebrotherswisp>[http://www.ics-il.com/images/youtubeicon.png]<https://www.youtube.com/channel/UCXSdfxQv7SpoRQYNyLwntZg>
________________________________
From: "Tom Beecher" <beecher@beecher.cc<mailto:beecher@beecher.cc>>
To: "Mike Hammett" <nanog@ics-il.net<mailto:nanog@ics-il.net>>
Cc: "Mel Beckman" <mel@beckman.org<mailto:mel@beckman.org>>, "NANOG" <nanog@nanog.org<mailto:nanog@nanog.org>>
Sent: Wednesday, January 4, 2023 7:36:58 AM
Subject: Re: SDN Internet Router (sir)

Disagree that it’s a line in the sand. It’s use the right tool for the job.

If a device is low FIB, it’s that way for a reason. There are plenty of ways to massage that with policy and software, depending on capabilities , but at the end of the day, trying to sort 10 pounds of shit to store in a 5 pound bag is eventually going to end up the same way.

On Tue, Jan 3, 2023 at 13:18 Mike Hammett <nanog@ics-il.net<mailto:nanog@ics-il.net>> wrote:
There are likely more networks with 10 gigabit or less total external capacity than there are with more.

Creating imaginary lines in the sand doesn't really help anyone.




-----
Mike Hammett
Intelligent Computing Solutions<http://www.ics-il.com/>
[http://www.ics-il.com/images/fbicon.png]<https://www.facebook.com/ICSIL>[http://www.ics-il.com/images/googleicon.png]<https://plus.google.com/+IntelligentComputingSolutionsDeKalb>[http://www.ics-il.com/images/linkedinicon.png]<https://www.linkedin.com/company/intelligent-computing-solutions>[http://www.ics-il.com/images/twittericon.png]<https://twitter.com/ICSIL>
Midwest Internet Exchange<http://www.midwest-ix.com/>
[http://www.ics-il.com/images/fbicon.png]<https://www.facebook.com/mdwestix>[http://www.ics-il.com/images/linkedinicon.png]<https://www.linkedin.com/company/midwest-internet-exchange>[http://www.ics-il.com/images/twittericon.png]<https://twitter.com/mdwestix>
The Brothers WISP<http://www.thebrotherswisp.com/>
[http://www.ics-il.com/images/fbicon.png]<https://www.facebook.com/thebrotherswisp>[http://www.ics-il.com/images/youtubeicon.png]<https://www.youtube.com/channel/UCXSdfxQv7SpoRQYNyLwntZg>
________________________________
From: "Mel Beckman" <mel@beckman.org<mailto:mel@beckman.org>>
To: "Mike Hammett" <nanog@ics-il.net<mailto:nanog@ics-il.net>>
Cc: "NANOG" <nanog@nanog.org<mailto:nanog@nanog.org>>
Sent: Tuesday, January 3, 2023 10:57:34 AM
Subject: Re: SDN Internet Router (sir)


It’s not a problem, due to cheap, plentiful high-speed memory and rapid prefix search silicon in backbone routers. The entire Internet routing table consumes at most a few gigabytes when fully structured (and only a few hundred Mbytes stored flat). That’s less memory than your average laptop sports.


Even in the worst case scenario, where every network decides to announce only its most specific prefixes, the BGP backbone would temporarily enter an oscillating state that generates a large number of routing updates into the inter-domain routing space. In this case, BGP route damping will quickly suppress the crazies while the backbone stabilizes.


Small routers should not be taking full tables, since there is no point to them being in the default free zone. For large routers, neither memory nor CPU speed are an issue. High-speed routers operating in the default-free zone have a critical path in the forwarding decision for each packet: it needs to take less than the inter-packet arrival time for minimum-sized IP packets.


This is easy to achieve with today’s hardware. A router line card with an aggregate line rate across all of its point-to-point interfaces of 10Tbps (readily available in today’s gear) can process packets with just a handful of cycles in the FIB Ternary Content Addressable Memory (TCAM) using ASIC-assisted lookups. TCAM is the most expensive component you’re paying for in such a router. It’s not cheap, but backbone routers don’t need to be cheap. They just need to not be memory-constrained.

-mel via cell

On Jan 3, 2023, at 7:47 AM, Mike Hammett <nanog@ics-il.net<mailto:nanog@ics-il.net>> wrote:

?
https://github.com/dbarrosop/sir

I came across this over the weekend. Given that the project was abandoned six years ago, are there any other efforts with a similar goal (more intelligently placing routes into FIBs of low-FIB capacity devices?



-----
Mike Hammett
Intelligent Computing Solutions<http://www.ics-il.com/>
[http://www.ics-il.com/images/fbicon.png]<https://www.facebook.com/ICSIL>[http://www.ics-il.com/images/googleicon.png]<https://plus.google.com/+IntelligentComputingSolutionsDeKalb>[http://www.ics-il.com/images/linkedinicon.png]<https://www.linkedin.com/company/intelligent-computing-solutions>[http://www.ics-il.com/images/twittericon.png]<https://twitter.com/ICSIL>
Midwest Internet Exchange<http://www.midwest-ix.com/>
[http://www.ics-il.com/images/fbicon.png]<https://www.facebook.com/mdwestix>[http://www.ics-il.com/images/linkedinicon.png]<https://www.linkedin.com/company/midwest-internet-exchange>[http://www.ics-il.com/images/twittericon.png]<https://twitter.com/mdwestix>
The Brothers WISP<http://www.thebrotherswisp.com/>
[http://www.ics-il.com/images/fbicon.png]<https://www.facebook.com/thebrotherswisp>[http://www.ics-il.com/images/youtubeicon.png]<https://www.youtube.com/channel/UCXSdfxQv7SpoRQYNyLwntZg>
Re: SDN Internet Router (sir) [ In reply to ]
BGP knows nothing about the importance of a given prefix.




-----
Mike Hammett
Intelligent Computing Solutions

Midwest Internet Exchange

The Brothers WISP

----- Original Message -----

From: "Mel Beckman" <mel@beckman.org>
To: "Mike Hammett" <nanog@ics-il.net>
Cc: "Tom Beecher" <beecher@beecher.cc>, "NANOG" <nanog@nanog.org>
Sent: Thursday, January 5, 2023 10:30:35 AM
Subject: Re: SDN Internet Router (sir)


Mike,


Your original question was:


“Given that the project was abandoned six years ago, are there any other efforts with a similar goal (more intelligently placing routes into FIBs of low-FIB capacity devices?”


People then, respectfully, tried to clarify your request or explain why placing routes in a low-FIB capacity device isn’t seen as being beneficial. Only now have you added the desire to simply have “more than a default route” in such a router.


You can, of course, have more than a default route today - e.g., through local pref and BGP communities for things such as company routes. You haven’t said what you define as “more intelligently”, so perhaps you can more clearly explain the problem you see with the current BGP capabilities via some examples.


-mel



On Jan 5, 2023, at 8:02 AM, Mike Hammett <nanog@ics-il.net> wrote:




<blockquote>


Then please bless the world with the right way.


You acknowledge that not every router in a network needs to be fully DFZ capable, but then crap on my desire to have more than a default route in one.




-----
Mike Hammett
Intelligent Computing Solutions

Midwest Internet Exchange

The Brothers WISP

----- Original Message -----

From: "Tom Beecher" <beecher@beecher.cc>
To: "Mike Hammett" <nanog@ics-il.net>
Cc: "Mel Beckman" <mel@beckman.org>, "NANOG" <nanog@nanog.org>
Sent: Thursday, January 5, 2023 9:55:38 AM
Subject: Re: SDN Internet Router (sir)



<blockquote>
"The right tool for the job" gets into a religious argument in assuming that one's way to do the job is the only reasonable way to do the job
</blockquote>



I disagree that it's religious. I completely agree there are locations in networks that having full DFZ capable routers doesn't make technical or economic sense. But there have long been different products for those different use cases.


To perhaps explain my viewpoint better,(and perhaps I didn't properly comprehend the problem you're aiming to solve) :


If you are trying to use SDN stuff to shuffle routes on and off a box because you have the wrong sized routers in place, then I would argue you're doing it wrong.


If you are trying to use SDN stuff to (as Christopher mentioned) make decisions that are not strictly LPM, and the equipment you have cannot do that, then that's different and entirely reasonable.


If the second use case is more of what you were asking, then I apologize for misunderstanding.





On Thu, Jan 5, 2023 at 9:57 AM Mike Hammett < nanog@ics-il.net > wrote:

<blockquote>



"The right tool for the job" gets into a religious argument in assuming that one's way to do the job is the only reasonable way to do the job.


Large networks historically have a very poor (IMO) model of gigantic iron in a few locations, which results in sub-optimal routing for the rest of their network between those large POPs. I've heard time and time again that someone buying service from a major network in say New Orleans has a first hop of Dallas or Atlanta. I agree that full-route capable routers need to be in the large, central locations, but it isn't cost effective to have them at every POP, especially if you're a last-mile provider.


I'd go into more examples of where it doesn't make sense to have full-route routers everywhere, but I'm afraid that the Internet would then focus on the examples instead of the core idea of intelligently putting routes into the FIBs of low-FIB routers throughout my network.




-----
Mike Hammett
Intelligent Computing Solutions

Midwest Internet Exchange

The Brothers WISP



From: "Tom Beecher" < beecher@beecher.cc >
To: "Mike Hammett" < nanog@ics-il.net >
Cc: "Mel Beckman" < mel@beckman.org >, "NANOG" < nanog@nanog.org >
Sent: Wednesday, January 4, 2023 7:36:58 AM
Subject: Re: SDN Internet Router (sir)


Disagree that it’s a line in the sand. It’s use the right tool for the job.


If a device is low FIB, it’s that way for a reason. There are plenty of ways to massage that with policy and software, depending on capabilities , but at the end of the day, trying to sort 10 pounds of shit to store in a 5 pound bag is eventually going to end up the same way.



On Tue, Jan 3, 2023 at 13:18 Mike Hammett < nanog@ics-il.net > wrote:

<blockquote>


There are likely more networks with 10 gigabit or less total external capacity than there are with more.


Creating imaginary lines in the sand doesn't really help anyone.







-----
Mike Hammett
Intelligent Computing Solutions

Midwest Internet Exchange

The Brothers WISP



From: "Mel Beckman" < mel@beckman.org >
To: "Mike Hammett" < nanog@ics-il.net >
Cc: "NANOG" < nanog@nanog.org >
Sent: Tuesday, January 3, 2023 10:57:34 AM
Subject: Re: SDN Internet Router (sir)


It’s not a problem, due to cheap, plentiful high-speed memory and rapid prefix search silicon in backbone routers. The entire Internet routing table consumes at most a few gigabytes when fully structured (and only a few hundred Mbytes stored flat). That’s less memory than your average laptop sports.


Even in the worst case scenario, where every network decides to announce only its most specific prefixes, the BGP backbone would temporarily enter an oscillating state that generates a large number of routing updates into the inter-domain routing space. In this case, BGP route damping will quickly suppress the crazies while the backbone stabilizes.


Small routers should not be taking full tables, since there is no point to them being in the default free zone. For large routers, neither memory nor CPU speed are an issue. High-speed routers operating in the default-free zone have a critical path in the forwarding decision for each packet: it needs to take less than the inter-packet arrival time for minimum-sized IP packets.


This is easy to achieve with today’s hardware. A router line card with an aggregate line rate across all of its point-to-point interfaces of 10Tbps (readily available in today’s gear) can process packets with just a handful of cycles in the FIB Ternary Content Addressable Memory (TCAM) using ASIC-assisted lookups. TCAM is the most expensive component you’re paying for in such a router. It’s not cheap, but backbone routers don’t need to be cheap. They just need to not be memory-constrained.

-mel via cell


<blockquote>
On Jan 3, 2023, at 7:47 AM, Mike Hammett < nanog@ics-il.net > wrote:


</blockquote>

<blockquote>


https://github.com/dbarrosop/sir


I came across this over the weekend. Given that the project was abandoned six years ago, are there any other efforts with a similar goal (more intelligently placing routes into FIBs of low-FIB capacity devices?




-----
Mike Hammett
Intelligent Computing Solutions

Midwest Internet Exchange

The Brothers WISP


</blockquote>


</blockquote>


</blockquote>


</blockquote>
Re: SDN Internet Router (sir) [ In reply to ]
On Thu, Jan 5, 2023 at 11:18 AM Mike Hammett <nanog@ics-il.net> wrote:

> Initially, my thought was to use community filtering to push just IXes,
> customers, and defaults throughout the network, but that's obviously still
> sub-optimal.
>
> I'd be surprised if a last mile network had a ton of traffic going to any
> more than a few hundred prefixes.
>

I think in a low-fib box at the edge of your network your choices are:
"the easy choice, get default, follow that"

"send some limited set of prefixes to the device, and default, so you MAY
choose better for the initial hop away"

you certainly can do the second with communities, or route-filters
(prefix-list) on the senders, or....
you can choose what prefixes make the cut (get the community(ies)) based on
traffic volumes or expected destination locality:
"do not go east to go west!"

these things will introduce toil and SOME suboptimal routing in some
instances... perhaps it's better than per flow choosing left/right though
and the support calls related to that choice.

In your NOLA / DFW / ATL example it's totally possible that the networks in
question do something like:
"low fib box in tier-2 city (NOLA), dfz capable/core devices in tier-1
city (DFW/ATL), and send default from left/right to NOLA"

Could they send more prefixes than default? sure... do they want to deal
with the toil that induces? (probably not says your example).

SDN isn't really an answer to this, though.. I don't think. Unless you
envision that to lower the toil ?
Re: SDN Internet Router (sir) [ In reply to ]
I'm not concerned with which technology or buzzword gets the job done, only that the job is done.


Communities certainly work. I could tag each of my peers (not a bad practice in the first place) with a different community, or set of communities and only allow say non-route server peers and customers to put routes into the limited-FIB boxes. However, how many routes can a given router hold in its FIB? 10K? 100k? 87,500? I'd have to make some manual choices about what gets included and what doesn't. That's not that big of a deal.


I'm getting almost 11k prefixes from my Amazon, Cloudflare, and Google peers. How many of those 11k do I have significant bits to? 1k at most? Less than 10% of the routes I'm putting in via communities are ones I actually care about. I get 142k from my HE peer (oddly, I just noticed my HE IPV6 peer is down, so time to remedy that. See, something good does come from arguing on the Internet!). That would be at or exceeding the FIB size of many routers, yet to what gain? I of course do have to make an administration somewhere to decide what gets included and what doesn't. Of course whatever is done manually will result in sub-optimal routing. I'll be concerning myself with useless prefixes and ignoring ones from transit (excluded from this whole thing) altogether.


Does something exist to make more intelligent choices than I can? Yes, at least two of them are in the public domain. I've had offlist responses about others home-brewing their own, similar solutions. Are they reliable? I don't know. Are they worth it? I'd assume so, but that depends greatly on if they're reliable.


Looking briefly at the couple of things out there, they're evaluating the top X prefixes in terms of traffic reported by s-flow, where X is the number I define, and those get pushed into the FIB. One recalculates every hour, one does so more quickly. How much is appropriate? I'm not sure. I can't imagine it would *NEED* to be done all of that often, given the traffic/prefix density an eyeball network will have. Default routes carry the rest. Default routes could be handled outside of this process, such that if this process fails, you just get some sub-optimal routing until repaired. Maybe it doesn't filter properly and sends a bunch of routes. Then just have a prefix limit set on the box. Maybe it sends the wrong prefixes. No harm, no foul. If you're routing sub-optimally internally, when it does hit a real router with a full FIB, it gets handled appropriately.




I would just be looking for solutions that influence what's in the FIB and let the rest of the router work as the rest of the router would.




-----
Mike Hammett
Intelligent Computing Solutions

Midwest Internet Exchange

The Brothers WISP

----- Original Message -----

From: "Christopher Morrow" <morrowc.lists@gmail.com>
To: "Mike Hammett" <nanog@ics-il.net>
Cc: "Tom Beecher" <beecher@beecher.cc>, "NANOG" <nanog@nanog.org>
Sent: Thursday, January 5, 2023 12:27:08 PM
Subject: Re: SDN Internet Router (sir)







On Thu, Jan 5, 2023 at 11:18 AM Mike Hammett < nanog@ics-il.net > wrote:





Initially, my thought was to use community filtering to push just IXes, customers, and defaults throughout the network, but that's obviously still sub-optimal.


I'd be surprised if a last mile network had a ton of traffic going to any more than a few hundred prefixes.





I think in a low-fib box at the edge of your network your choices are:
"the easy choice, get default, follow that"



"send some limited set of prefixes to the device, and default, so you MAY choose better for the initial hop away"


you certainly can do the second with communities, or route-filters (prefix-list) on the senders, or....
you can choose what prefixes make the cut (get the community(ies)) based on traffic volumes or expected destination locality:
"do not go east to go west!"


these things will introduce toil and SOME suboptimal routing in some instances... perhaps it's better than per flow choosing left/right though and the support calls related to that choice.


In your NOLA / DFW / ATL example it's totally possible that the networks in question do something like:
"low fib box in tier-2 city (NOLA), dfz capable/core devices in tier-1 city (DFW/ATL), and send default from left/right to NOLA"

Could they send more prefixes than default? sure... do they want to deal with the toil that induces? (probably not says your example).


SDN isn't really an answer to this, though.. I don't think. Unless you envision that to lower the toil ?
Re: SDN Internet Router (sir) [ In reply to ]
Lots of 1M tcam fib limits in older gear.......

So yeah, its the same problem, bigger numbers and still not solved in
any sort of non-painful or expensive way.

I think Ill explore the google path and paper on it again.

Joe

Mike Hammett wrote:
> Then please bless the world with the right way.
>
> You acknowledge that not every router in a network needs to be fully
> DFZ capable, but then crap on my desire to have more than a default
> route in one.
>
>
>
> -----
> Mike Hammett
> Intelligent Computing Solutions <http://www.ics-il.com/>
> <https://www.facebook.com/ICSIL><https://plus.google.com/+IntelligentComputingSolutionsDeKalb><https://www.linkedin.com/company/intelligent-computing-solutions><https://twitter.com/ICSIL>
> Midwest Internet Exchange <http://www.midwest-ix.com/>
> <https://www.facebook.com/mdwestix><https://www.linkedin.com/company/midwest-internet-exchange><https://twitter.com/mdwestix>
> The Brothers WISP <http://www.thebrotherswisp.com/>
> <https://www.facebook.com/thebrotherswisp><https://www.youtube.com/channel/UCXSdfxQv7SpoRQYNyLwntZg>
> ------------------------------------------------------------------------
> *From: *"Tom Beecher" <beecher@beecher.cc>
> *To: *"Mike Hammett" <nanog@ics-il.net>
> *Cc: *"Mel Beckman" <mel@beckman.org>, "NANOG" <nanog@nanog.org>
> *Sent: *Thursday, January 5, 2023 9:55:38 AM
> *Subject: *Re: SDN Internet Router (sir)
>
> "The right tool for the job" gets into a religious argument in
> assuming that one's way to do the job is the only reasonable way
> to do the job
>
>
> I disagree that it's religious. I completely agree there are locations
> in networks that having full DFZ capable routers doesn't make
> technical or economic sense. But there have long been different
> products for those different use cases.
>
> To perhaps explain my viewpoint better,(and perhaps I didn't properly
> comprehend the problem you're aiming to solve) :
>
> If you are trying to use SDN stuff to shuffle routes on and off a box
> because you have the wrong sized routers in place, then I would argue
> you're doing it wrong.
>
> If you are trying to use SDN stuff to (as Christopher mentioned) make
> decisions that are not strictly LPM, and the equipment you have cannot
> do that, then that's different and entirely reasonable.
>
> If the second use case is more of what you were asking, then I
> apologize for misunderstanding.
>
>
> On Thu, Jan 5, 2023 at 9:57 AM Mike Hammett <nanog@ics-il.net
> <mailto:nanog@ics-il.net>> wrote:
>
> "The right tool for the job" gets into a religious argument in
> assuming that one's way to do the job is the only reasonable way
> to do the job.
>
> Large networks historically have a very poor (IMO) model of
> gigantic iron in a few locations, which results in sub-optimal
> routing for the rest of their network between those large POPs.
> I've heard time and time again that someone buying service from a
> major network in say New Orleans has a first hop of Dallas or
> Atlanta. I agree that full-route capable routers need to be in the
> large, central locations, but it isn't cost effective to have them
> at every POP, especially if you're a last-mile provider.
>
> I'd go into more examples of where it doesn't make sense to have
> full-route routers everywhere, but I'm afraid that the Internet
> would then focus on the examples instead of the core idea of
> intelligently putting routes into the FIBs of low-FIB routers
> throughout my network.
>
>
>
> -----
> Mike Hammett
> Intelligent Computing Solutions <http://www.ics-il.com/>
> <https://www.facebook.com/ICSIL><https://plus.google.com/+IntelligentComputingSolutionsDeKalb><https://www.linkedin.com/company/intelligent-computing-solutions><https://twitter.com/ICSIL>
> Midwest Internet Exchange <http://www.midwest-ix.com/>
> <https://www.facebook.com/mdwestix><https://www.linkedin.com/company/midwest-internet-exchange><https://twitter.com/mdwestix>
> The Brothers WISP <http://www.thebrotherswisp.com/>
> <https://www.facebook.com/thebrotherswisp><https://www.youtube.com/channel/UCXSdfxQv7SpoRQYNyLwntZg>
> ------------------------------------------------------------------------
> *From: *"Tom Beecher" <beecher@beecher.cc <mailto:beecher@beecher.cc>>
> *To: *"Mike Hammett" <nanog@ics-il.net <mailto:nanog@ics-il.net>>
> *Cc: *"Mel Beckman" <mel@beckman.org <mailto:mel@beckman.org>>,
> "NANOG" <nanog@nanog.org <mailto:nanog@nanog.org>>
> *Sent: *Wednesday, January 4, 2023 7:36:58 AM
> *Subject: *Re: SDN Internet Router (sir)
>
> Disagree that it’s a line in the sand. It’s use the right tool for
> the job.
>
> If a device is low FIB, it’s that way for a reason. There are
> plenty of ways to massage that with policy and software, depending
> on capabilities , but at the end of the day, trying to sort 10
> pounds of shit to store in a 5 pound bag is eventually going to
> end up the same way.
>
> On Tue, Jan 3, 2023 at 13:18 Mike Hammett <nanog@ics-il.net
> <mailto:nanog@ics-il.net>> wrote:
>
> There are likely more networks with 10 gigabit or less total
> external capacity than there are with more.
>
> Creating imaginary lines in the sand doesn't really help anyone.
>
>
>
>
> -----
> Mike Hammett
> Intelligent Computing Solutions <http://www.ics-il.com/>
> <https://www.facebook.com/ICSIL><https://plus.google.com/+IntelligentComputingSolutionsDeKalb><https://www.linkedin.com/company/intelligent-computing-solutions><https://twitter.com/ICSIL>
> Midwest Internet Exchange <http://www.midwest-ix.com/>
> <https://www.facebook.com/mdwestix><https://www.linkedin.com/company/midwest-internet-exchange><https://twitter.com/mdwestix>
> The Brothers WISP <http://www.thebrotherswisp.com/>
> <https://www.facebook.com/thebrotherswisp><https://www.youtube.com/channel/UCXSdfxQv7SpoRQYNyLwntZg>
> ------------------------------------------------------------------------
> *From: *"Mel Beckman" <mel@beckman.org <mailto:mel@beckman.org>>
> *To: *"Mike Hammett" <nanog@ics-il.net <mailto:nanog@ics-il.net>>
> *Cc: *"NANOG" <nanog@nanog.org <mailto:nanog@nanog.org>>
> *Sent: *Tuesday, January 3, 2023 10:57:34 AM
> *Subject: *Re: SDN Internet Router (sir)
>
> It’s not a problem, due to cheap, plentiful high-speed memory
> and rapid prefix search silicon in backbone routers. The
> entire Internet routing table consumes at most a few gigabytes
> when fully structured (and only a few hundred Mbytes stored
> flat). That’s less memory than your average laptop sports.
>
>
> Even in the worst case scenario, where every network decides
> to announce only its most specific prefixes, the BGP backbone
> would temporarily enter an oscillating state that generates a
> large number of routing updates into the inter-domain routing
> space. In this case, BGP route damping will quickly suppress
> the crazies while the backbone stabilizes.
>
>
> Small routers should not be taking full tables, since there is
> no point to them being in the default free zone. For large
> routers, neither memory nor CPU speed are an issue. High-speed
> routers operating in the default-free zone have a critical
> path in the forwarding decision for each packet: it needs to
> take less than the inter-packet arrival time for minimum-sized
> IP packets.
>
>
> This is easy to achieve with today’s hardware. A router line
> card with an aggregate line rate across all of its
> point-to-point interfaces of 10Tbps (readily available in
> today’s gear) can process packets with just a handful of
> cycles in the FIB Ternary Content Addressable Memory (TCAM)
> using ASIC-assisted lookups. TCAM is the most expensive
> component you’re paying for in such a router. It’s not cheap,
> but backbone routers don’t need to be cheap. They just need to
> not be memory-constrained.
>
>
> -mel via cell
>
> On Jan 3, 2023, at 7:47 AM, Mike Hammett <nanog@ics-il.net
> <mailto:nanog@ics-il.net>> wrote:
>
> ?
> https://github.com/dbarrosop/sir
>
> I came across this over the weekend. Given that the
> project was abandoned six years ago, are there any other
> efforts with a similar goal (more intelligently placing
> routes into FIBs of low-FIB capacity devices?
>
>
>
> -----
> Mike Hammett
> Intelligent Computing Solutions <http://www.ics-il.com/>
> <https://www.facebook.com/ICSIL><https://plus.google.com/+IntelligentComputingSolutionsDeKalb><https://www.linkedin.com/company/intelligent-computing-solutions><https://twitter.com/ICSIL>
> Midwest Internet Exchange <http://www.midwest-ix.com/>
> <https://www.facebook.com/mdwestix><https://www.linkedin.com/company/midwest-internet-exchange><https://twitter.com/mdwestix>
> The Brothers WISP <http://www.thebrotherswisp.com/>
> <https://www.facebook.com/thebrotherswisp><https://www.youtube.com/channel/UCXSdfxQv7SpoRQYNyLwntZg>
>
>
>
>
Re: SDN Internet Router (sir) [ In reply to ]
Very true.

https://www.cidr-report.org/cgi-bin/plota?file=%2fvar%2fdata%2fbgp%2fas2.0%2fbgp%2dactive%2etxt&descr=Active%20BGP%20entries%20%28FIB%29&ylabel=Active%20BGP%20entries%20%28FIB%29&with=step


"big enough" equipment from not that long ago couldn't carry a full table today (or tomorrow).



-----
Mike Hammett
Intelligent Computing Solutions

Midwest Internet Exchange

The Brothers WISP

----- Original Message -----

From: "Joe Maimon" <jmaimon@jmaimon.com>
To: "Mike Hammett" <nanog@ics-il.net>, "Tom Beecher" <beecher@beecher.cc>
Cc: "NANOG" <nanog@nanog.org>
Sent: Thursday, January 5, 2023 2:22:22 PM
Subject: Re: SDN Internet Router (sir)

Lots of 1M tcam fib limits in older gear.......

So yeah, its the same problem, bigger numbers and still not solved in
any sort of non-painful or expensive way.

I think Ill explore the google path and paper on it again.

Joe

Mike Hammett wrote:
> Then please bless the world with the right way.
>
> You acknowledge that not every router in a network needs to be fully
> DFZ capable, but then crap on my desire to have more than a default
> route in one.
>
>
>
> -----
> Mike Hammett
> Intelligent Computing Solutions <http://www.ics-il.com/>
> <https://www.facebook.com/ICSIL><https://plus.google.com/+IntelligentComputingSolutionsDeKalb><https://www.linkedin.com/company/intelligent-computing-solutions><https://twitter.com/ICSIL>
> Midwest Internet Exchange <http://www.midwest-ix.com/>
> <https://www.facebook.com/mdwestix><https://www.linkedin.com/company/midwest-internet-exchange><https://twitter.com/mdwestix>
> The Brothers WISP <http://www.thebrotherswisp.com/>
> <https://www.facebook.com/thebrotherswisp><https://www.youtube.com/channel/UCXSdfxQv7SpoRQYNyLwntZg>
> ------------------------------------------------------------------------
> *From: *"Tom Beecher" <beecher@beecher.cc>
> *To: *"Mike Hammett" <nanog@ics-il.net>
> *Cc: *"Mel Beckman" <mel@beckman.org>, "NANOG" <nanog@nanog.org>
> *Sent: *Thursday, January 5, 2023 9:55:38 AM
> *Subject: *Re: SDN Internet Router (sir)
>
> "The right tool for the job" gets into a religious argument in
> assuming that one's way to do the job is the only reasonable way
> to do the job
>
>
> I disagree that it's religious. I completely agree there are locations
> in networks that having full DFZ capable routers doesn't make
> technical or economic sense. But there have long been different
> products for those different use cases.
>
> To perhaps explain my viewpoint better,(and perhaps I didn't properly
> comprehend the problem you're aiming to solve) :
>
> If you are trying to use SDN stuff to shuffle routes on and off a box
> because you have the wrong sized routers in place, then I would argue
> you're doing it wrong.
>
> If you are trying to use SDN stuff to (as Christopher mentioned) make
> decisions that are not strictly LPM, and the equipment you have cannot
> do that, then that's different and entirely reasonable.
>
> If the second use case is more of what you were asking, then I
> apologize for misunderstanding.
>
>
> On Thu, Jan 5, 2023 at 9:57 AM Mike Hammett <nanog@ics-il.net
> <mailto:nanog@ics-il.net>> wrote:
>
> "The right tool for the job" gets into a religious argument in
> assuming that one's way to do the job is the only reasonable way
> to do the job.
>
> Large networks historically have a very poor (IMO) model of
> gigantic iron in a few locations, which results in sub-optimal
> routing for the rest of their network between those large POPs.
> I've heard time and time again that someone buying service from a
> major network in say New Orleans has a first hop of Dallas or
> Atlanta. I agree that full-route capable routers need to be in the
> large, central locations, but it isn't cost effective to have them
> at every POP, especially if you're a last-mile provider.
>
> I'd go into more examples of where it doesn't make sense to have
> full-route routers everywhere, but I'm afraid that the Internet
> would then focus on the examples instead of the core idea of
> intelligently putting routes into the FIBs of low-FIB routers
> throughout my network.
>
>
>
> -----
> Mike Hammett
> Intelligent Computing Solutions <http://www.ics-il.com/>
> <https://www.facebook.com/ICSIL><https://plus.google.com/+IntelligentComputingSolutionsDeKalb><https://www.linkedin.com/company/intelligent-computing-solutions><https://twitter.com/ICSIL>
> Midwest Internet Exchange <http://www.midwest-ix.com/>
> <https://www.facebook.com/mdwestix><https://www.linkedin.com/company/midwest-internet-exchange><https://twitter.com/mdwestix>
> The Brothers WISP <http://www.thebrotherswisp.com/>
> <https://www.facebook.com/thebrotherswisp><https://www.youtube.com/channel/UCXSdfxQv7SpoRQYNyLwntZg>
> ------------------------------------------------------------------------
> *From: *"Tom Beecher" <beecher@beecher.cc <mailto:beecher@beecher.cc>>
> *To: *"Mike Hammett" <nanog@ics-il.net <mailto:nanog@ics-il.net>>
> *Cc: *"Mel Beckman" <mel@beckman.org <mailto:mel@beckman.org>>,
> "NANOG" <nanog@nanog.org <mailto:nanog@nanog.org>>
> *Sent: *Wednesday, January 4, 2023 7:36:58 AM
> *Subject: *Re: SDN Internet Router (sir)
>
> Disagree that it’s a line in the sand. It’s use the right tool for
> the job.
>
> If a device is low FIB, it’s that way for a reason. There are
> plenty of ways to massage that with policy and software, depending
> on capabilities , but at the end of the day, trying to sort 10
> pounds of shit to store in a 5 pound bag is eventually going to
> end up the same way.
>
> On Tue, Jan 3, 2023 at 13:18 Mike Hammett <nanog@ics-il.net
> <mailto:nanog@ics-il.net>> wrote:
>
> There are likely more networks with 10 gigabit or less total
> external capacity than there are with more.
>
> Creating imaginary lines in the sand doesn't really help anyone.
>
>
>
>
> -----
> Mike Hammett
> Intelligent Computing Solutions <http://www.ics-il.com/>
> <https://www.facebook.com/ICSIL><https://plus.google.com/+IntelligentComputingSolutionsDeKalb><https://www.linkedin.com/company/intelligent-computing-solutions><https://twitter.com/ICSIL>
> Midwest Internet Exchange <http://www.midwest-ix.com/>
> <https://www.facebook.com/mdwestix><https://www.linkedin.com/company/midwest-internet-exchange><https://twitter.com/mdwestix>
> The Brothers WISP <http://www.thebrotherswisp.com/>
> <https://www.facebook.com/thebrotherswisp><https://www.youtube.com/channel/UCXSdfxQv7SpoRQYNyLwntZg>
> ------------------------------------------------------------------------
> *From: *"Mel Beckman" <mel@beckman.org <mailto:mel@beckman.org>>
> *To: *"Mike Hammett" <nanog@ics-il.net <mailto:nanog@ics-il.net>>
> *Cc: *"NANOG" <nanog@nanog.org <mailto:nanog@nanog.org>>
> *Sent: *Tuesday, January 3, 2023 10:57:34 AM
> *Subject: *Re: SDN Internet Router (sir)
>
> It’s not a problem, due to cheap, plentiful high-speed memory
> and rapid prefix search silicon in backbone routers. The
> entire Internet routing table consumes at most a few gigabytes
> when fully structured (and only a few hundred Mbytes stored
> flat). That’s less memory than your average laptop sports.
>
>
> Even in the worst case scenario, where every network decides
> to announce only its most specific prefixes, the BGP backbone
> would temporarily enter an oscillating state that generates a
> large number of routing updates into the inter-domain routing
> space. In this case, BGP route damping will quickly suppress
> the crazies while the backbone stabilizes.
>
>
> Small routers should not be taking full tables, since there is
> no point to them being in the default free zone. For large
> routers, neither memory nor CPU speed are an issue. High-speed
> routers operating in the default-free zone have a critical
> path in the forwarding decision for each packet: it needs to
> take less than the inter-packet arrival time for minimum-sized
> IP packets.
>
>
> This is easy to achieve with today’s hardware. A router line
> card with an aggregate line rate across all of its
> point-to-point interfaces of 10Tbps (readily available in
> today’s gear) can process packets with just a handful of
> cycles in the FIB Ternary Content Addressable Memory (TCAM)
> using ASIC-assisted lookups. TCAM is the most expensive
> component you’re paying for in such a router. It’s not cheap,
> but backbone routers don’t need to be cheap. They just need to
> not be memory-constrained.
>
>
> -mel via cell
>
> On Jan 3, 2023, at 7:47 AM, Mike Hammett <nanog@ics-il.net
> <mailto:nanog@ics-il.net>> wrote:
>
>
> https://github.com/dbarrosop/sir
>
> I came across this over the weekend. Given that the
> project was abandoned six years ago, are there any other
> efforts with a similar goal (more intelligently placing
> routes into FIBs of low-FIB capacity devices?
>
>
>
> -----
> Mike Hammett
> Intelligent Computing Solutions <http://www.ics-il.com/>
> <https://www.facebook.com/ICSIL><https://plus.google.com/+IntelligentComputingSolutionsDeKalb><https://www.linkedin.com/company/intelligent-computing-solutions><https://twitter.com/ICSIL>
> Midwest Internet Exchange <http://www.midwest-ix.com/>
> <https://www.facebook.com/mdwestix><https://www.linkedin.com/company/midwest-internet-exchange><https://twitter.com/mdwestix>
> The Brothers WISP <http://www.thebrotherswisp.com/>
> <https://www.facebook.com/thebrotherswisp><https://www.youtube.com/channel/UCXSdfxQv7SpoRQYNyLwntZg>
>
>
>
>
Re: SDN Internet Router (sir) [ In reply to ]
Mike Hammett wrote:
> I'm not concerned with which technology or buzzword gets the job done,
> only that the job is done.
>
>
>
> Looking briefly at the couple of things out there, they're evaluating
> the top X prefixes in terms of traffic reported by s-flow, where X is
> the number I define, and those get pushed into the FIB. One
> recalculates every hour, one does so more quickly. How much is
> appropriate? I'm not sure. I can't imagine it would *NEED* to be done
> all of that often, given the traffic/prefix density an eyeball network
> will have. Default routes carry the rest. Default routes could be
> handled outside of this process, such that if this process fails, you
> just get some sub-optimal routing until repaired. Maybe it doesn't
> filter properly and sends a bunch of routes. Then just have a prefix
> limit set on the box. Maybe it sends the wrong prefixes. No harm, no
> foul. If you're routing sub-optimally internally, when it does hit a
> real router with a full FIB, it gets handled appropriately.

Unless it loops.

The rest sounds nice. But flow caching got a bad rap back in the early
worm days. But thats because the situation was a little worse back then.
Cache the wrong routes or run out of cache, router dies. So long as
thats not the case automating optimization is an extremely valuable goal.

>
>
> I would just be looking for solutions that influence what's in the FIB
> and let the rest of the router work as the rest of the router would.

The problem comes when the router wont work at all without the FIB
routes, like in the olden days.
>
>
>
> -----
> Mike Hammett
> Intelligent Computing Solutions <http://www.ics-il.com/>
> <https://www.facebook.com/ICSIL><https://plus.google.com/+IntelligentComputingSolutionsDeKalb><https://www.linkedin.com/company/intelligent-computing-solutions><https://twitter.com/ICSIL>
> Midwest Internet Exchange <http://www.midwest-ix.com/>
> <https://www.facebook.com/mdwestix><https://www.linkedin.com/company/midwest-internet-exchange><https://twitter.com/mdwestix>
> The Brothers WISP <http://www.thebrotherswisp.com/>
> <https://www.facebook.com/thebrotherswisp><https://www.youtube.com/channel/UCXSdfxQv7SpoRQYNyLwntZg>
> ------------------------------------------------------------------------
> *From: *"Christopher Morrow" <morrowc.lists@gmail.com>
> *To: *"Mike Hammett" <nanog@ics-il.net>
> *Cc: *"Tom Beecher" <beecher@beecher.cc>, "NANOG" <nanog@nanog.org>
> *Sent: *Thursday, January 5, 2023 12:27:08 PM
> *Subject: *Re: SDN Internet Router (sir)
>
>
>
> On Thu, Jan 5, 2023 at 11:18 AM Mike Hammett <nanog@ics-il.net
> <mailto:nanog@ics-il.net>> wrote:
>
> Initially, my thought was to use community filtering to push just
> IXes, customers, and defaults throughout the network, but that's
> obviously still sub-optimal.
>
> I'd be surprised if a last mile network had a ton of traffic going
> to any more than a few hundred prefixes.
>
>
> I think in a low-fib box at the edge of your network your choices are:
> "the easy choice, get default, follow that"
>
> "send some limited set of prefixes to the device, and default, so
> you MAY choose better for the initial hop away"
>
> you certainly can do the second with communities, or route-filters
> (prefix-list) on the senders, or....
> you can choose what prefixes make the cut (get the community(ies))
> based on traffic volumes or expected destination locality:
> "do not go east to go west!"
>
> these things will introduce toil and SOME suboptimal routing in some
> instances... perhaps it's better than per flow choosing left/right
> though and the support calls related to that choice.
>
> In your NOLA / DFW / ATL example it's totally possible that the
> networks in question do something like:
> "low fib box in tier-2 city (NOLA), dfz capable/core devices in
> tier-1 city (DFW/ATL), and send default from left/right to NOLA"
>
> Could they send more prefixes than default? sure... do they want to
> deal with the toil that induces? (probably not says your example).
>
> SDN isn't really an answer to this, though.. I don't think. Unless you
> envision that to lower the toil ?
>
Re: SDN Internet Router (sir) [ In reply to ]
I guess I wasn't around for those days.


As far as running out, again, assuming the tooling works correctly, I'd think to target fewer routes than you could hold. Maybe 1k routes is all one would need to get a significant percent of the traffic. A lot of room to mess up if you can hold 100k, 500k routes.





-----
Mike Hammett
Intelligent Computing Solutions

Midwest Internet Exchange

The Brothers WISP

----- Original Message -----

From: "Joe Maimon" <jmaimon@jmaimon.com>
To: "Mike Hammett" <nanog@ics-il.net>, "Christopher Morrow" <morrowc.lists@gmail.com>
Cc: "NANOG" <nanog@nanog.org>
Sent: Thursday, January 5, 2023 2:30:40 PM
Subject: Re: SDN Internet Router (sir)



Mike Hammett wrote:
> I'm not concerned with which technology or buzzword gets the job done,
> only that the job is done.
>
>
>
> Looking briefly at the couple of things out there, they're evaluating
> the top X prefixes in terms of traffic reported by s-flow, where X is
> the number I define, and those get pushed into the FIB. One
> recalculates every hour, one does so more quickly. How much is
> appropriate? I'm not sure. I can't imagine it would *NEED* to be done
> all of that often, given the traffic/prefix density an eyeball network
> will have. Default routes carry the rest. Default routes could be
> handled outside of this process, such that if this process fails, you
> just get some sub-optimal routing until repaired. Maybe it doesn't
> filter properly and sends a bunch of routes. Then just have a prefix
> limit set on the box. Maybe it sends the wrong prefixes. No harm, no
> foul. If you're routing sub-optimally internally, when it does hit a
> real router with a full FIB, it gets handled appropriately.

Unless it loops.

The rest sounds nice. But flow caching got a bad rap back in the early
worm days. But thats because the situation was a little worse back then.
Cache the wrong routes or run out of cache, router dies. So long as
thats not the case automating optimization is an extremely valuable goal.

>
>
> I would just be looking for solutions that influence what's in the FIB
> and let the rest of the router work as the rest of the router would.

The problem comes when the router wont work at all without the FIB
routes, like in the olden days.
>
>
>
> -----
> Mike Hammett
> Intelligent Computing Solutions <http://www.ics-il.com/>
> <https://www.facebook.com/ICSIL><https://plus.google.com/+IntelligentComputingSolutionsDeKalb><https://www.linkedin.com/company/intelligent-computing-solutions><https://twitter.com/ICSIL>
> Midwest Internet Exchange <http://www.midwest-ix.com/>
> <https://www.facebook.com/mdwestix><https://www.linkedin.com/company/midwest-internet-exchange><https://twitter.com/mdwestix>
> The Brothers WISP <http://www.thebrotherswisp.com/>
> <https://www.facebook.com/thebrotherswisp><https://www.youtube.com/channel/UCXSdfxQv7SpoRQYNyLwntZg>
> ------------------------------------------------------------------------
> *From: *"Christopher Morrow" <morrowc.lists@gmail.com>
> *To: *"Mike Hammett" <nanog@ics-il.net>
> *Cc: *"Tom Beecher" <beecher@beecher.cc>, "NANOG" <nanog@nanog.org>
> *Sent: *Thursday, January 5, 2023 12:27:08 PM
> *Subject: *Re: SDN Internet Router (sir)
>
>
>
> On Thu, Jan 5, 2023 at 11:18 AM Mike Hammett <nanog@ics-il.net
> <mailto:nanog@ics-il.net>> wrote:
>
> Initially, my thought was to use community filtering to push just
> IXes, customers, and defaults throughout the network, but that's
> obviously still sub-optimal.
>
> I'd be surprised if a last mile network had a ton of traffic going
> to any more than a few hundred prefixes.
>
>
> I think in a low-fib box at the edge of your network your choices are:
> "the easy choice, get default, follow that"
>
> "send some limited set of prefixes to the device, and default, so
> you MAY choose better for the initial hop away"
>
> you certainly can do the second with communities, or route-filters
> (prefix-list) on the senders, or....
> you can choose what prefixes make the cut (get the community(ies))
> based on traffic volumes or expected destination locality:
> "do not go east to go west!"
>
> these things will introduce toil and SOME suboptimal routing in some
> instances... perhaps it's better than per flow choosing left/right
> though and the support calls related to that choice.
>
> In your NOLA / DFW / ATL example it's totally possible that the
> networks in question do something like:
> "low fib box in tier-2 city (NOLA), dfz capable/core devices in
> tier-1 city (DFW/ATL), and send default from left/right to NOLA"
>
> Could they send more prefixes than default? sure... do they want to
> deal with the toil that induces? (probably not says your example).
>
> SDN isn't really an answer to this, though.. I don't think. Unless you
> envision that to lower the toil ?
>
Re: SDN Internet Router (sir) [ In reply to ]
Mike,

I’m not sure I understand what you mean by “suboptimal“ routing. Even though the Internet uses AS path length for routing, many of those path lengths are bogus, and don’t really represent any kind of path performance value. For example, a single AS might hide many hops in an MPLS network as a single hop, obscuring asymmetric routing and other uglies. Prepending also occurs when destinations are trying to enforce their own engineering policies, which often conflict with yours or mine.

So what do you mean by “suboptimal“? Are you thinking that the “best” path in BGP tables actually meant you were getting a performance benefit? Because that’s definitely not the case in today’s Internet. Were were you thinking that you would be going along less congested paths? That’s really at the mercy of the traffic engineering of backbone providers over which we have no control.

I generally populate local router FIBs to merel choose an exit point for purposes of load balancing, and nothing more.

-mel

On Jan 5, 2023, at 12:38 PM, Mike Hammett <nanog@ics-il.net> wrote:

?
I guess I wasn't around for those days.

As far as running out, again, assuming the tooling works correctly, I'd think to target fewer routes than you could hold. Maybe 1k routes is all one would need to get a significant percent of the traffic. A lot of room to mess up if you can hold 100k, 500k routes.



-----
Mike Hammett
Intelligent Computing Solutions<http://www.ics-il.com/>
[http://www.ics-il.com/images/fbicon.png]<https://www.facebook.com/ICSIL>[http://www.ics-il.com/images/googleicon.png]<https://plus.google.com/+IntelligentComputingSolutionsDeKalb>[http://www.ics-il.com/images/linkedinicon.png]<https://www.linkedin.com/company/intelligent-computing-solutions>[http://www.ics-il.com/images/twittericon.png]<https://twitter.com/ICSIL>
Midwest Internet Exchange<http://www.midwest-ix.com/>
[http://www.ics-il.com/images/fbicon.png]<https://www.facebook.com/mdwestix>[http://www.ics-il.com/images/linkedinicon.png]<https://www.linkedin.com/company/midwest-internet-exchange>[http://www.ics-il.com/images/twittericon.png]<https://twitter.com/mdwestix>
The Brothers WISP<http://www.thebrotherswisp.com/>
[http://www.ics-il.com/images/fbicon.png]<https://www.facebook.com/thebrotherswisp>[http://www.ics-il.com/images/youtubeicon.png]<https://www.youtube.com/channel/UCXSdfxQv7SpoRQYNyLwntZg>
________________________________
From: "Joe Maimon" <jmaimon@jmaimon.com>
To: "Mike Hammett" <nanog@ics-il.net>, "Christopher Morrow" <morrowc.lists@gmail.com>
Cc: "NANOG" <nanog@nanog.org>
Sent: Thursday, January 5, 2023 2:30:40 PM
Subject: Re: SDN Internet Router (sir)



Mike Hammett wrote:
> I'm not concerned with which technology or buzzword gets the job done,
> only that the job is done.
>
>
>
> Looking briefly at the couple of things out there, they're evaluating
> the top X prefixes in terms of traffic reported by s-flow, where X is
> the number I define, and those get pushed into the FIB. One
> recalculates every hour, one does so more quickly. How much is
> appropriate? I'm not sure. I can't imagine it would *NEED* to be done
> all of that often, given the traffic/prefix density an eyeball network
> will have. Default routes carry the rest. Default routes could be
> handled outside of this process, such that if this process fails, you
> just get some sub-optimal routing until repaired. Maybe it doesn't
> filter properly and sends a bunch of routes. Then just have a prefix
> limit set on the box. Maybe it sends the wrong prefixes. No harm, no
> foul. If you're routing sub-optimally internally, when it does hit a
> real router with a full FIB, it gets handled appropriately.

Unless it loops.

The rest sounds nice. But flow caching got a bad rap back in the early
worm days. But thats because the situation was a little worse back then.
Cache the wrong routes or run out of cache, router dies. So long as
thats not the case automating optimization is an extremely valuable goal.

>
>
> I would just be looking for solutions that influence what's in the FIB
> and let the rest of the router work as the rest of the router would.

The problem comes when the router wont work at all without the FIB
routes, like in the olden days.
>
>
>
> -----
> Mike Hammett
> Intelligent Computing Solutions <http://www.ics-il.com/>
> <https://www.facebook.com/ICSIL><https://plus.google.com/+IntelligentComputingSolutionsDeKalb><https://www.linkedin.com/company/intelligent-computing-solutions><https://twitter.com/ICSIL>
> Midwest Internet Exchange <http://www.midwest-ix.com/>
> <https://www.facebook.com/mdwestix><https://www.linkedin.com/company/midwest-internet-exchange><https://twitter.com/mdwestix>
> The Brothers WISP <http://www.thebrotherswisp.com/>
> <https://www.facebook.com/thebrotherswisp><https://www.youtube.com/channel/UCXSdfxQv7SpoRQYNyLwntZg>
> ------------------------------------------------------------------------
> *From: *"Christopher Morrow" <morrowc.lists@gmail.com>
> *To: *"Mike Hammett" <nanog@ics-il.net>
> *Cc: *"Tom Beecher" <beecher@beecher.cc>, "NANOG" <nanog@nanog.org>
> *Sent: *Thursday, January 5, 2023 12:27:08 PM
> *Subject: *Re: SDN Internet Router (sir)
>
>
>
> On Thu, Jan 5, 2023 at 11:18 AM Mike Hammett <nanog@ics-il.net
> <mailto:nanog@ics-il.net>> wrote:
>
> Initially, my thought was to use community filtering to push just
> IXes, customers, and defaults throughout the network, but that's
> obviously still sub-optimal.
>
> I'd be surprised if a last mile network had a ton of traffic going
> to any more than a few hundred prefixes.
>
>
> I think in a low-fib box at the edge of your network your choices are:
> "the easy choice, get default, follow that"
>
> "send some limited set of prefixes to the device, and default, so
> you MAY choose better for the initial hop away"
>
> you certainly can do the second with communities, or route-filters
> (prefix-list) on the senders, or....
> you can choose what prefixes make the cut (get the community(ies))
> based on traffic volumes or expected destination locality:
> "do not go east to go west!"
>
> these things will introduce toil and SOME suboptimal routing in some
> instances... perhaps it's better than per flow choosing left/right
> though and the support calls related to that choice.
>
> In your NOLA / DFW / ATL example it's totally possible that the
> networks in question do something like:
> "low fib box in tier-2 city (NOLA), dfz capable/core devices in
> tier-1 city (DFW/ATL), and send default from left/right to NOLA"
>
> Could they send more prefixes than default? sure... do they want to
> deal with the toil that induces? (probably not says your example).
>
> SDN isn't really an answer to this, though.. I don't think. Unless you
> envision that to lower the toil ?
>
Re: SDN Internet Router (sir) [ In reply to ]
I hesitated to get too specific in examples because someone is going to drag the conversation into the weeds.

Let's take the the Dallas - New Orleans - Atlanta example where I have a connection from New Orleans to Dallas and a connection from New Orleans to Atlanta.


Let's say I peer with Netflix in both markets. Netflix chooses to serve me out of Atlanta, for whatever reason. Say my default route sends my traffic to Dallas. That's not where Netflix wanted it, so now I have to go from Dallas to Atlanta, whether that's my circuit or across the public Internet. Potentially, it's on MPLS and it rides back through the New Orleans router to get back to Atlanta. That's a long trip when I already had a better path, the less-than-full-fib router just didn't know about it. Given that Netflix is a sizable amount of traffic in an eyeball ISP, that's a lot of traffic to be going the wrong way. If the website for Viktor's Arctic Plunge in Siberia was hosted in Atlanta, I wouldn't give two craps that the traffic went the wrong way because A), I'll probably never go there and B) when someone does, it won't be meaningfully enough traffic to accommodate.


Someone's going to tell me to put a full-table router in New Orleans. Maybe I should. Okay, so maybe I have a POP in Ashford, Alabama. It has transport to New Orleans and Atlanta. There aren't enough grains of sugar in Ashford, Alabama to justify a current-generation, full table router. Now I'm even closer to Atlanta, but default may point to New Orleans.




-----
Mike Hammett
Intelligent Computing Solutions

Midwest Internet Exchange

The Brothers WISP

----- Original Message -----

From: "Mel Beckman" <mel@beckman.org>
To: "Mike Hammett" <nanog@ics-il.net>
Cc: "Joe Maimon" <jmaimon@jmaimon.com>, "NANOG" <nanog@nanog.org>
Sent: Thursday, January 5, 2023 2:54:27 PM
Subject: Re: SDN Internet Router (sir)


Mike,


I’m not sure I understand what you mean by “suboptimal“ routing. Even though the Internet uses AS path length for routing, many of those path lengths are bogus, and don’t really represent any kind of path performance value. For example, a single AS might hide many hops in an MPLS network as a single hop, obscuring asymmetric routing and other uglies. Prepending also occurs when destinations are trying to enforce their own engineering policies, which often conflict with yours or mine.


So what do you mean by “suboptimal“? Are you thinking that the “best” path in BGP tables actually meant you were getting a performance benefit? Because that’s definitely not the case in today’s Internet. Were were you thinking that you would be going along less congested paths? That’s really at the mercy of the traffic engineering of backbone providers over which we have no control.


I generally populate local router FIBs to merel choose an exit point for purposes of load balancing, and nothing more.


-mel




On Jan 5, 2023, at 12:38 PM, Mike Hammett <nanog@ics-il.net> wrote:




<blockquote>


I guess I wasn't around for those days.


As far as running out, again, assuming the tooling works correctly, I'd think to target fewer routes than you could hold. Maybe 1k routes is all one would need to get a significant percent of the traffic. A lot of room to mess up if you can hold 100k, 500k routes.





-----
Mike Hammett
Intelligent Computing Solutions

Midwest Internet Exchange

The Brothers WISP

----- Original Message -----

From: "Joe Maimon" <jmaimon@jmaimon.com>
To: "Mike Hammett" <nanog@ics-il.net>, "Christopher Morrow" <morrowc.lists@gmail.com>
Cc: "NANOG" <nanog@nanog.org>
Sent: Thursday, January 5, 2023 2:30:40 PM
Subject: Re: SDN Internet Router (sir)



Mike Hammett wrote:
> I'm not concerned with which technology or buzzword gets the job done,
> only that the job is done.
>
>
>
> Looking briefly at the couple of things out there, they're evaluating
> the top X prefixes in terms of traffic reported by s-flow, where X is
> the number I define, and those get pushed into the FIB. One
> recalculates every hour, one does so more quickly. How much is
> appropriate? I'm not sure. I can't imagine it would *NEED* to be done
> all of that often, given the traffic/prefix density an eyeball network
> will have. Default routes carry the rest. Default routes could be
> handled outside of this process, such that if this process fails, you
> just get some sub-optimal routing until repaired. Maybe it doesn't
> filter properly and sends a bunch of routes. Then just have a prefix
> limit set on the box. Maybe it sends the wrong prefixes. No harm, no
> foul. If you're routing sub-optimally internally, when it does hit a
> real router with a full FIB, it gets handled appropriately.

Unless it loops.

The rest sounds nice. But flow caching got a bad rap back in the early
worm days. But thats because the situation was a little worse back then.
Cache the wrong routes or run out of cache, router dies. So long as
thats not the case automating optimization is an extremely valuable goal.

>
>
> I would just be looking for solutions that influence what's in the FIB
> and let the rest of the router work as the rest of the router would.

The problem comes when the router wont work at all without the FIB
routes, like in the olden days.
>
>
>
> -----
> Mike Hammett
> Intelligent Computing Solutions <http://www.ics-il.com/>
> <https://www.facebook.com/ICSIL><https://plus.google.com/+IntelligentComputingSolutionsDeKalb><https://www.linkedin.com/company/intelligent-computing-solutions><https://twitter.com/ICSIL>
> Midwest Internet Exchange <http://www.midwest-ix.com/>
> <https://www.facebook.com/mdwestix><https://www.linkedin.com/company/midwest-internet-exchange><https://twitter.com/mdwestix>
> The Brothers WISP <http://www.thebrotherswisp.com/>
> <https://www.facebook.com/thebrotherswisp><https://www.youtube.com/channel/UCXSdfxQv7SpoRQYNyLwntZg>
> ------------------------------------------------------------------------
> *From: *"Christopher Morrow" <morrowc.lists@gmail.com>
> *To: *"Mike Hammett" <nanog@ics-il.net>
> *Cc: *"Tom Beecher" <beecher@beecher.cc>, "NANOG" <nanog@nanog.org>
> *Sent: *Thursday, January 5, 2023 12:27:08 PM
> *Subject: *Re: SDN Internet Router (sir)
>
>
>
> On Thu, Jan 5, 2023 at 11:18 AM Mike Hammett <nanog@ics-il.net
> <mailto:nanog@ics-il.net>> wrote:
>
> Initially, my thought was to use community filtering to push just
> IXes, customers, and defaults throughout the network, but that's
> obviously still sub-optimal.
>
> I'd be surprised if a last mile network had a ton of traffic going
> to any more than a few hundred prefixes.
>
>
> I think in a low-fib box at the edge of your network your choices are:
> "the easy choice, get default, follow that"
>
> "send some limited set of prefixes to the device, and default, so
> you MAY choose better for the initial hop away"
>
> you certainly can do the second with communities, or route-filters
> (prefix-list) on the senders, or....
> you can choose what prefixes make the cut (get the community(ies))
> based on traffic volumes or expected destination locality:
> "do not go east to go west!"
>
> these things will introduce toil and SOME suboptimal routing in some
> instances... perhaps it's better than per flow choosing left/right
> though and the support calls related to that choice.
>
> In your NOLA / DFW / ATL example it's totally possible that the
> networks in question do something like:
> "low fib box in tier-2 city (NOLA), dfz capable/core devices in
> tier-1 city (DFW/ATL), and send default from left/right to NOLA"
>
> Could they send more prefixes than default? sure... do they want to
> deal with the toil that induces? (probably not says your example).
>
> SDN isn't really an answer to this, though.. I don't think. Unless you
> envision that to lower the toil ?
>



</blockquote>
Re: SDN Internet Router (sir) [ In reply to ]
This is not a green grass problem space.

https://www.cisco.com/c/en/us/products/ios-nx-os-software/performance-routing-pfr/index.html

And you could probably envision how you could create your own internal
scheme of route reflectors/servers, community tags, probers and updaters
to achieve something similar.

Most likely Mike is referring to the sub-optimal result where a large
percentage of a router's traffic is taking extra internal hops or worse,
maybe even egressing from the AS into a less than optimal path, not
because the AS does not have the correct route for the most likely as
perceived by BGP optimal path, but that the traffic handling device was
not able to be configured to accept any such routes, because doing such
statically is not likely to achieve the results and more likely to
result in crashed routers one unexpected fine morning.

Nanogers pointed me at this some time back, I think its germaine

https://blog.google/products/google-cloud/making-google-cloud-faster-more-available-and-cost-effective-extending-sdn-public-internet-espresso/

RIB/FIB static configuration limitation tip:

Apply the same logic on all similar capacity devices to cut down on the
RIBFIB, because thats the best way to minimize loops. And a guaranteed
loop free path for the default route. Policy or tag tunnel or whatever.

Joe


Mel Beckman wrote:
> Mike,
>
> I’m not sure I understand what you mean by “suboptimal“ routing. Even
> though the Internet uses AS path length for routing, many of those
> path lengths are bogus, and don’t really represent any kind of path
> performance value. For example, a single AS might hide many hops in an
> MPLS network as a single hop, obscuring asymmetric routing and other
> uglies. Prepending also occurs when destinations are trying to enforce
> their own engineering policies, which often conflict with yours or mine.
>
> So what do you mean by “suboptimal“? Are you thinking that the “best”
> path in BGP tables actually meant you were getting a performance
> benefit? Because that’s definitely not the case in today’s Internet.
> Were were you thinking that you would be going along less congested
> paths? That’s really at the mercy of the traffic engineering of
> backbone providers over which we have no control.
>
> I generally populate local router FIBs to merel choose an exit point
> for purposes of load balancing, and nothing more.
>
> -mel
>
>> On Jan 5, 2023, at 12:38 PM, Mike Hammett <nanog@ics-il.net> wrote:
>>
>> ?
>> I guess I wasn't around for those days.
>>
>> As far as running out, again, assuming the tooling works correctly,
>> I'd think to target fewer routes than you could hold. Maybe 1k routes
>> is all one would need to get a significant percent of the traffic. A
>> lot of room to mess up if you can hold 100k, 500k routes.
>>
>>
>>
>> -----
>> Mike Hammett
>> Intelligent Computing Solutions <http://www.ics-il.com/>
>> <https://www.facebook.com/ICSIL><https://plus.google.com/+IntelligentComputingSolutionsDeKalb><https://www.linkedin.com/company/intelligent-computing-solutions><https://twitter.com/ICSIL>
>> Midwest Internet Exchange <http://www.midwest-ix.com/>
>> <https://www.facebook.com/mdwestix><https://www.linkedin.com/company/midwest-internet-exchange><https://twitter.com/mdwestix>
>> The Brothers WISP <http://www.thebrotherswisp.com/>
>> <https://www.facebook.com/thebrotherswisp><https://www.youtube.com/channel/UCXSdfxQv7SpoRQYNyLwntZg>
>> ------------------------------------------------------------------------
>> *From: *"Joe Maimon" <jmaimon@jmaimon.com>
>> *To: *"Mike Hammett" <nanog@ics-il.net>, "Christopher Morrow"
>> <morrowc.lists@gmail.com>
>> *Cc: *"NANOG" <nanog@nanog.org>
>> *Sent: *Thursday, January 5, 2023 2:30:40 PM
>> *Subject: *Re: SDN Internet Router (sir)
>>
>>
>>
>> Mike Hammett wrote:
>> > I'm not concerned with which technology or buzzword gets the job done,
>> > only that the job is done.
>> >
>> >
>> >
>> > Looking briefly at the couple of things out there, they're evaluating
>> > the top X prefixes in terms of traffic reported by s-flow, where X is
>> > the number I define, and those get pushed into the FIB. One
>> > recalculates every hour, one does so more quickly. How much is
>> > appropriate? I'm not sure. I can't imagine it would *NEED* to be done
>> > all of that often, given the traffic/prefix density an eyeball network
>> > will have. Default routes carry the rest. Default routes could be
>> > handled outside of this process, such that if this process fails, you
>> > just get some sub-optimal routing until repaired. Maybe it doesn't
>> > filter properly and sends a bunch of routes. Then just have a prefix
>> > limit set on the box. Maybe it sends the wrong prefixes. No harm, no
>> > foul. If you're routing sub-optimally internally, when it does hit a
>> > real router with a full FIB, it gets handled appropriately.
>>
>> Unless it loops.
>>
>> The rest sounds nice. But flow caching got a bad rap back in the early
>> worm days. But thats because the situation was a little worse back then.
>> Cache the wrong routes or run out of cache, router dies. So long as
>> thats not the case automating optimization is an extremely valuable goal.
>>
>> >
>> >
>> > I would just be looking for solutions that influence what's in the FIB
>> > and let the rest of the router work as the rest of the router would.
>>
>> The problem comes when the router wont work at all without the FIB
>> routes, like in the olden days.
>> >
>> >
>> >
>> > -----
>> > Mike Hammett
>> > Intelligent Computing Solutions <http://www.ics-il.com/>
>> >
>> <https://www.facebook.com/ICSIL><https://plus.google.com/+IntelligentComputingSolutionsDeKalb><https://www.linkedin.com/company/intelligent-computing-solutions><https://twitter.com/ICSIL>
>> > Midwest Internet Exchange <http://www.midwest-ix.com/>
>> >
>> <https://www.facebook.com/mdwestix><https://www.linkedin.com/company/midwest-internet-exchange><https://twitter.com/mdwestix>
>> > The Brothers WISP <http://www.thebrotherswisp.com/>
>> >
>> <https://www.facebook.com/thebrotherswisp><https://www.youtube.com/channel/UCXSdfxQv7SpoRQYNyLwntZg>
>> >
>> ------------------------------------------------------------------------
>> > *From: *"Christopher Morrow" <morrowc.lists@gmail.com>
>> > *To: *"Mike Hammett" <nanog@ics-il.net>
>> > *Cc: *"Tom Beecher" <beecher@beecher.cc>, "NANOG" <nanog@nanog.org>
>> > *Sent: *Thursday, January 5, 2023 12:27:08 PM
>> > *Subject: *Re: SDN Internet Router (sir)
>> >
>> >
>> >
>> > On Thu, Jan 5, 2023 at 11:18 AM Mike Hammett <nanog@ics-il.net
>> > <mailto:nanog@ics-il.net>> wrote:
>> >
>> > Initially, my thought was to use community filtering to push just
>> > IXes, customers, and defaults throughout the network, but that's
>> > obviously still sub-optimal.
>> >
>> > I'd be surprised if a last mile network had a ton of traffic going
>> > to any more than a few hundred prefixes.
>> >
>> >
>> > I think in a low-fib box at the edge of your network your choices are:
>> > "the easy choice, get default, follow that"
>> >
>> > "send some limited set of prefixes to the device, and default, so
>> > you MAY choose better for the initial hop away"
>> >
>> > you certainly can do the second with communities, or route-filters
>> > (prefix-list) on the senders, or....
>> > you can choose what prefixes make the cut (get the community(ies))
>> > based on traffic volumes or expected destination locality:
>> > "do not go east to go west!"
>> >
>> > these things will introduce toil and SOME suboptimal routing in some
>> > instances... perhaps it's better than per flow choosing left/right
>> > though and the support calls related to that choice.
>> >
>> > In your NOLA / DFW / ATL example it's totally possible that the
>> > networks in question do something like:
>> > "low fib box in tier-2 city (NOLA), dfz capable/core devices in
>> > tier-1 city (DFW/ATL), and send default from left/right to NOLA"
>> >
>> > Could they send more prefixes than default? sure... do they want to
>> > deal with the toil that induces? (probably not says your example).
>> >
>> > SDN isn't really an answer to this, though.. I don't think. Unless you
>> > envision that to lower the toil ?
>> >
>>
>>
Re: SDN Internet Router (sir) [ In reply to ]
Mike,

Thanks for that useful example. On a side note, Netflix is a thorn in all our sides :) You could put a localpref filter route to override the default for Netflix prefixes, but this impacts resilience. Since you peer with Netflix, I suspect we probably agree that Netflix’s ideas on traffic engineering are pretty one sided.

I think it’s safe to say that BGP, which has scaled amazingly well, didn’t anticipate some of the big gorilla content systems. I don’t really see, though, how injecting FIB entries helps more than other methods. And as others have pointed out, the risk of creating routing loops is significant.

Perhaps it is time to migrate to a new version of BGP. Projects like MBGP and FP-7‘s 4WARD are working on new follow-on routing models, but nothing is on the immediate horizon. I think we all thought we should finish IPv6 migration first :)

-mel via cell

On Jan 5, 2023, at 1:11 PM, Mike Hammett <nanog@ics-il.net> wrote:

?
I hesitated to get too specific in examples because someone is going to drag the conversation into the weeds.

Let's take the the Dallas - New Orleans - Atlanta example where I have a connection from New Orleans to Dallas and a connection from New Orleans to Atlanta.

Let's say I peer with Netflix in both markets. Netflix chooses to serve me out of Atlanta, for whatever reason. Say my default route sends my traffic to Dallas. That's not where Netflix wanted it, so now I have to go from Dallas to Atlanta, whether that's my circuit or across the public Internet. Potentially, it's on MPLS and it rides back through the New Orleans router to get back to Atlanta. That's a long trip when I already had a better path, the less-than-full-fib router just didn't know about it. Given that Netflix is a sizable amount of traffic in an eyeball ISP, that's a lot of traffic to be going the wrong way. If the website for Viktor's Arctic Plunge in Siberia was hosted in Atlanta, I wouldn't give two craps that the traffic went the wrong way because A), I'll probably never go there and B) when someone does, it won't be meaningfully enough traffic to accommodate.

Someone's going to tell me to put a full-table router in New Orleans. Maybe I should. Okay, so maybe I have a POP in Ashford, Alabama. It has transport to New Orleans and Atlanta. There aren't enough grains of sugar in Ashford, Alabama to justify a current-generation, full table router. Now I'm even closer to Atlanta, but default may point to New Orleans.



-----
Mike Hammett
Intelligent Computing Solutions<http://www.ics-il.com/>
[http://www.ics-il.com/images/fbicon.png]<https://www.facebook.com/ICSIL>[http://www.ics-il.com/images/googleicon.png]<https://plus.google.com/+IntelligentComputingSolutionsDeKalb>[http://www.ics-il.com/images/linkedinicon.png]<https://www.linkedin.com/company/intelligent-computing-solutions>[http://www.ics-il.com/images/twittericon.png]<https://twitter.com/ICSIL>
Midwest Internet Exchange<http://www.midwest-ix.com/>
[http://www.ics-il.com/images/fbicon.png]<https://www.facebook.com/mdwestix>[http://www.ics-il.com/images/linkedinicon.png]<https://www.linkedin.com/company/midwest-internet-exchange>[http://www.ics-il.com/images/twittericon.png]<https://twitter.com/mdwestix>
The Brothers WISP<http://www.thebrotherswisp.com/>
[http://www.ics-il.com/images/fbicon.png]<https://www.facebook.com/thebrotherswisp>[http://www.ics-il.com/images/youtubeicon.png]<https://www.youtube.com/channel/UCXSdfxQv7SpoRQYNyLwntZg>
________________________________
From: "Mel Beckman" <mel@beckman.org>
To: "Mike Hammett" <nanog@ics-il.net>
Cc: "Joe Maimon" <jmaimon@jmaimon.com>, "NANOG" <nanog@nanog.org>
Sent: Thursday, January 5, 2023 2:54:27 PM
Subject: Re: SDN Internet Router (sir)

Mike,

I’m not sure I understand what you mean by “suboptimal“ routing. Even though the Internet uses AS path length for routing, many of those path lengths are bogus, and don’t really represent any kind of path performance value. For example, a single AS might hide many hops in an MPLS network as a single hop, obscuring asymmetric routing and other uglies. Prepending also occurs when destinations are trying to enforce their own engineering policies, which often conflict with yours or mine.

So what do you mean by “suboptimal“? Are you thinking that the “best” path in BGP tables actually meant you were getting a performance benefit? Because that’s definitely not the case in today’s Internet. Were were you thinking that you would be going along less congested paths? That’s really at the mercy of the traffic engineering of backbone providers over which we have no control.

I generally populate local router FIBs to merel choose an exit point for purposes of load balancing, and nothing more.

-mel

On Jan 5, 2023, at 12:38 PM, Mike Hammett <nanog@ics-il.net> wrote:

?
I guess I wasn't around for those days.

As far as running out, again, assuming the tooling works correctly, I'd think to target fewer routes than you could hold. Maybe 1k routes is all one would need to get a significant percent of the traffic. A lot of room to mess up if you can hold 100k, 500k routes.



-----
Mike Hammett
Intelligent Computing Solutions<http://www.ics-il.com/>
[http://www.ics-il.com/images/fbicon.png]<https://www.facebook.com/ICSIL>[http://www.ics-il.com/images/googleicon.png]<https://plus.google.com/+IntelligentComputingSolutionsDeKalb>[http://www.ics-il.com/images/linkedinicon.png]<https://www.linkedin.com/company/intelligent-computing-solutions>[http://www.ics-il.com/images/twittericon.png]<https://twitter.com/ICSIL>
Midwest Internet Exchange<http://www.midwest-ix.com/>
[http://www.ics-il.com/images/fbicon.png]<https://www.facebook.com/mdwestix>[http://www.ics-il.com/images/linkedinicon.png]<https://www.linkedin.com/company/midwest-internet-exchange>[http://www.ics-il.com/images/twittericon.png]<https://twitter.com/mdwestix>
The Brothers WISP<http://www.thebrotherswisp.com/>
[http://www.ics-il.com/images/fbicon.png]<https://www.facebook.com/thebrotherswisp>[http://www.ics-il.com/images/youtubeicon.png]<https://www.youtube.com/channel/UCXSdfxQv7SpoRQYNyLwntZg>
________________________________
From: "Joe Maimon" <jmaimon@jmaimon.com>
To: "Mike Hammett" <nanog@ics-il.net>, "Christopher Morrow" <morrowc.lists@gmail.com>
Cc: "NANOG" <nanog@nanog.org>
Sent: Thursday, January 5, 2023 2:30:40 PM
Subject: Re: SDN Internet Router (sir)



Mike Hammett wrote:
> I'm not concerned with which technology or buzzword gets the job done,
> only that the job is done.
>
>
>
> Looking briefly at the couple of things out there, they're evaluating
> the top X prefixes in terms of traffic reported by s-flow, where X is
> the number I define, and those get pushed into the FIB. One
> recalculates every hour, one does so more quickly. How much is
> appropriate? I'm not sure. I can't imagine it would *NEED* to be done
> all of that often, given the traffic/prefix density an eyeball network
> will have. Default routes carry the rest. Default routes could be
> handled outside of this process, such that if this process fails, you
> just get some sub-optimal routing until repaired. Maybe it doesn't
> filter properly and sends a bunch of routes. Then just have a prefix
> limit set on the box. Maybe it sends the wrong prefixes. No harm, no
> foul. If you're routing sub-optimally internally, when it does hit a
> real router with a full FIB, it gets handled appropriately.

Unless it loops.

The rest sounds nice. But flow caching got a bad rap back in the early
worm days. But thats because the situation was a little worse back then.
Cache the wrong routes or run out of cache, router dies. So long as
thats not the case automating optimization is an extremely valuable goal.

>
>
> I would just be looking for solutions that influence what's in the FIB
> and let the rest of the router work as the rest of the router would.

The problem comes when the router wont work at all without the FIB
routes, like in the olden days.
>
>
>
> -----
> Mike Hammett
> Intelligent Computing Solutions <http://www.ics-il.com/>
> <https://www.facebook.com/ICSIL><https://plus.google.com/+IntelligentComputingSolutionsDeKalb><https://www.linkedin.com/company/intelligent-computing-solutions><https://twitter.com/ICSIL>
> Midwest Internet Exchange <http://www.midwest-ix.com/>
> <https://www.facebook.com/mdwestix><https://www.linkedin.com/company/midwest-internet-exchange><https://twitter.com/mdwestix>
> The Brothers WISP <http://www.thebrotherswisp.com/>
> <https://www.facebook.com/thebrotherswisp><https://www.youtube.com/channel/UCXSdfxQv7SpoRQYNyLwntZg>
> ------------------------------------------------------------------------
> *From: *"Christopher Morrow" <morrowc.lists@gmail.com>
> *To: *"Mike Hammett" <nanog@ics-il.net>
> *Cc: *"Tom Beecher" <beecher@beecher.cc>, "NANOG" <nanog@nanog.org>
> *Sent: *Thursday, January 5, 2023 12:27:08 PM
> *Subject: *Re: SDN Internet Router (sir)
>
>
>
> On Thu, Jan 5, 2023 at 11:18 AM Mike Hammett <nanog@ics-il.net
> <mailto:nanog@ics-il.net>> wrote:
>
> Initially, my thought was to use community filtering to push just
> IXes, customers, and defaults throughout the network, but that's
> obviously still sub-optimal.
>
> I'd be surprised if a last mile network had a ton of traffic going
> to any more than a few hundred prefixes.
>
>
> I think in a low-fib box at the edge of your network your choices are:
> "the easy choice, get default, follow that"
>
> "send some limited set of prefixes to the device, and default, so
> you MAY choose better for the initial hop away"
>
> you certainly can do the second with communities, or route-filters
> (prefix-list) on the senders, or....
> you can choose what prefixes make the cut (get the community(ies))
> based on traffic volumes or expected destination locality:
> "do not go east to go west!"
>
> these things will introduce toil and SOME suboptimal routing in some
> instances... perhaps it's better than per flow choosing left/right
> though and the support calls related to that choice.
>
> In your NOLA / DFW / ATL example it's totally possible that the
> networks in question do something like:
> "low fib box in tier-2 city (NOLA), dfz capable/core devices in
> tier-1 city (DFW/ATL), and send default from left/right to NOLA"
>
> Could they send more prefixes than default? sure... do they want to
> deal with the toil that induces? (probably not says your example).
>
> SDN isn't really an answer to this, though.. I don't think. Unless you
> envision that to lower the toil ?
>
Re: SDN Internet Router (sir) [ In reply to ]
And here is another interesting approach Ive left open in my browser
window for who knows how long

https://inog.net/files/iNOG14v_oliver_sourcerouting.pdf

The problem with BGP is that local actors can exact global costs
trivially by consuming as many routing slots as they can get away with,
add together BGP path decisions and Most Specific traffic-engineering is
the goto knob. Sometimes you just want to say this is the route, do not
accept any more specifics, unless this route is no longer the route. But
you want that done automatically and correctly, reliably.

This is also why all the multi-homing approaches that do not involve
global routing havent really taken off in any way to blunt table growth.
And likely wont.

See the aggregation factor in the routing report for how bad this is.

There have been lots of BGP protocol and feature updates, but unless
your going to uniformly run new systems and enterprise systems that
support all of them, its hard to decide to build your entire routing
strategy around them.

That BGP unlike EIGRP never tried to tie together performance indicators
with routing metrics feature or misdesign, you could debate that but it
was always intentional. And opex has pretty much fallen down on the side
of against IGP->BGP redistribution of prefixes, let alone performance
metrics.

That eBGP prefix has no good reliable way of indicating that an
advertised route sucks so bad that you should never attempt to use it
unless as last resort, thats why we have AS-paths wrapping screen lines.

"finish IPv6 migration"? Letting IPv6 migration state factor as decision
input on anything not directly related to IPv6 migration was never
logical, just naively optimistic, and should be stamped out wherever
encountered. If its good, use it now and Ipv6 will adopt it as well. If
it isnt, why wait to find out?

Joe

Mel Beckman wrote:
> Mike,
>
> Thanks for that useful example. On a side note, Netflix is a thorn in
> all our sides :) You could put a localpref filter route to override
> the default for Netflix prefixes, but this impacts resilience. Since
> you peer with Netflix, I suspect we probably agree that Netflix’s
> ideas on traffic engineering are pretty one sided.
>
> I think it’s safe to say that BGP, which has scaled amazingly well,
> didn’t anticipate some of the big gorilla content systems. I don’t
> really see, though, how injecting FIB entries helps more than other
> methods. And as others have pointed out, the risk of creating routing
> loops is significant.
>
> Perhaps it is time to migrate to a new version of BGP. Projects like
> MBGP and FP-7‘s 4WARD are working on new follow-on routing models, but
> nothing is on the immediate horizon. I think we all thought we should
> finish IPv6 migration first :)
>
> -mel via cell
>
>> On Jan 5, 2023, at 1:11 PM, Mike Hammett <nanog@ics-il.net> wrote:
>>
>> ?
>> I hesitated to get too specific in examples because someone is going
>> to drag the conversation into the weeds.
>>
>> Let's take the the Dallas - New Orleans - Atlanta example where I
>> have a connection from New Orleans to Dallas and a connection from
>> New Orleans to Atlanta.
>>
>> Let's say I peer with Netflix in both markets. Netflix chooses to
>> serve me out of Atlanta, for whatever reason. Say my default route
>> sends my traffic to Dallas. That's not where Netflix wanted it, so
>> now I have to go from Dallas to Atlanta, whether that's my circuit or
>> across the public Internet. Potentially, it's on MPLS and it rides
>> back through the New Orleans router to get back to Atlanta. That's a
>> long trip when I already had a better path, the less-than-full-fib
>> router just didn't know about it. Given that Netflix is a sizable
>> amount of traffic in an eyeball ISP, that's a lot of traffic to be
>> going the wrong way. If the website for Viktor's Arctic Plunge in
>> Siberia was hosted in Atlanta, I wouldn't give two craps that the
>> traffic went the wrong way because A), I'll probably never go there
>> and B) when someone does, it won't be meaningfully enough traffic to
>> accommodate.
>>
>> Someone's going to tell me to put a full-table router in New Orleans.
>> Maybe I should. Okay, so maybe I have a POP in Ashford, Alabama. It
>> has transport to New Orleans and Atlanta. There aren't enough grains
>> of sugar in Ashford, Alabama to justify a current-generation, full
>> table router. Now I'm even closer to Atlanta, but default may point
>> to New Orleans.
>>
>>
>>
>> -----
>> Mike Hammett
>> Intelligent Computing Solutions <http://www.ics-il.com/>
>> <https://www.facebook.com/ICSIL><https://plus.google.com/+IntelligentComputingSolutionsDeKalb><https://www.linkedin.com/company/intelligent-computing-solutions><https://twitter.com/ICSIL>
>> Midwest Internet Exchange <http://www.midwest-ix.com/>
>> <https://www.facebook.com/mdwestix><https://www.linkedin.com/company/midwest-internet-exchange><https://twitter.com/mdwestix>
>> The Brothers WISP <http://www.thebrotherswisp.com/>
>> <https://www.facebook.com/thebrotherswisp><https://www.youtube.com/channel/UCXSdfxQv7SpoRQYNyLwntZg>
>> ------------------------------------------------------------------------
>> *From: *"Mel Beckman" <mel@beckman.org>
>> *To: *"Mike Hammett" <nanog@ics-il.net>
>> *Cc: *"Joe Maimon" <jmaimon@jmaimon.com>, "NANOG" <nanog@nanog.org>
>> *Sent: *Thursday, January 5, 2023 2:54:27 PM
>> *Subject: *Re: SDN Internet Router (sir)
>>
>> Mike,
>>
>> I’m not sure I understand what you mean by “suboptimal“ routing. Even
>> though the Internet uses AS path length for routing, many of those
>> path lengths are bogus, and don’t really represent any kind of path
>> performance value. For example, a single AS might hide many hops in
>> an MPLS network as a single hop, obscuring asymmetric routing and
>> other uglies. Prepending also occurs when destinations are trying to
>> enforce their own engineering policies, which often conflict with
>> yours or mine.
>>
>> So what do you mean by “suboptimal“? Are you thinking that the “best”
>> path in BGP tables actually meant you were getting a performance
>> benefit? Because that’s definitely not the case in today’s Internet.
>> Were were you thinking that you would be going along less congested
>> paths? That’s really at the mercy of the traffic engineering of
>> backbone providers over which we have no control.
>>
>> I generally populate local router FIBs to merel choose an exit point
>> for purposes of load balancing, and nothing more.
>>
>> -mel
>>
>> On Jan 5, 2023, at 12:38 PM, Mike Hammett <nanog@ics-il.net> wrote:
>>
>> ?
>> I guess I wasn't around for those days.
>>
>> As far as running out, again, assuming the tooling works
>> correctly, I'd think to target fewer routes than you could hold.
>> Maybe 1k routes is all one would need to get a significant
>> percent of the traffic. A lot of room to mess up if you can hold
>> 100k, 500k routes.
>>
>>
>>
>> -----
>> Mike Hammett
>> Intelligent Computing Solutions <http://www.ics-il.com/>
>> <https://www.facebook.com/ICSIL><https://plus.google.com/+IntelligentComputingSolutionsDeKalb><https://www.linkedin.com/company/intelligent-computing-solutions><https://twitter.com/ICSIL>
>> Midwest Internet Exchange <http://www.midwest-ix.com/>
>> <https://www.facebook.com/mdwestix><https://www.linkedin.com/company/midwest-internet-exchange><https://twitter.com/mdwestix>
>> The Brothers WISP <http://www.thebrotherswisp.com/>
>> <https://www.facebook.com/thebrotherswisp><https://www.youtube.com/channel/UCXSdfxQv7SpoRQYNyLwntZg>
>> ------------------------------------------------------------------------
>> *From: *"Joe Maimon" <jmaimon@jmaimon.com>
>> *To: *"Mike Hammett" <nanog@ics-il.net>, "Christopher Morrow"
>> <morrowc.lists@gmail.com>
>> *Cc: *"NANOG" <nanog@nanog.org>
>> *Sent: *Thursday, January 5, 2023 2:30:40 PM
>> *Subject: *Re: SDN Internet Router (sir)
>>
>>
>>
>> Mike Hammett wrote:
>> > I'm not concerned with which technology or buzzword gets the
>> job done,
>> > only that the job is done.
>> >
>> >
>> >
>> > Looking briefly at the couple of things out there, they're
>> evaluating
>> > the top X prefixes in terms of traffic reported by s-flow,
>> where X is
>> > the number I define, and those get pushed into the FIB. One
>> > recalculates every hour, one does so more quickly. How much is
>> > appropriate? I'm not sure. I can't imagine it would *NEED* to
>> be done
>> > all of that often, given the traffic/prefix density an eyeball
>> network
>> > will have. Default routes carry the rest. Default routes could be
>> > handled outside of this process, such that if this process
>> fails, you
>> > just get some sub-optimal routing until repaired. Maybe it doesn't
>> > filter properly and sends a bunch of routes. Then just have a
>> prefix
>> > limit set on the box. Maybe it sends the wrong prefixes. No
>> harm, no
>> > foul. If you're routing sub-optimally internally, when it does
>> hit a
>> > real router with a full FIB, it gets handled appropriately.
>>
>> Unless it loops.
>>
>> The rest sounds nice. But flow caching got a bad rap back in the
>> early
>> worm days. But thats because the situation was a little worse
>> back then.
>> Cache the wrong routes or run out of cache, router dies. So long as
>> thats not the case automating optimization is an extremely
>> valuable goal.
>>
>> >
>> >
>> > I would just be looking for solutions that influence what's in
>> the FIB
>> > and let the rest of the router work as the rest of the router
>> would.
>>
>> The problem comes when the router wont work at all without the FIB
>> routes, like in the olden days.
>> >
>> >
>> >
>> > -----
>> > Mike Hammett
>> > Intelligent Computing Solutions <http://www.ics-il.com/>
>> >
>> <https://www.facebook.com/ICSIL><https://plus.google.com/+IntelligentComputingSolutionsDeKalb><https://www.linkedin.com/company/intelligent-computing-solutions><https://twitter.com/ICSIL>
>> > Midwest Internet Exchange <http://www.midwest-ix.com/>
>> >
>> <https://www.facebook.com/mdwestix><https://www.linkedin.com/company/midwest-internet-exchange><https://twitter.com/mdwestix>
>> > The Brothers WISP <http://www.thebrotherswisp.com/>
>> >
>> <https://www.facebook.com/thebrotherswisp><https://www.youtube.com/channel/UCXSdfxQv7SpoRQYNyLwntZg>
>> >
>> ------------------------------------------------------------------------
>> > *From: *"Christopher Morrow" <morrowc.lists@gmail.com>
>> > *To: *"Mike Hammett" <nanog@ics-il.net>
>> > *Cc: *"Tom Beecher" <beecher@beecher.cc>, "NANOG" <nanog@nanog.org>
>> > *Sent: *Thursday, January 5, 2023 12:27:08 PM
>> > *Subject: *Re: SDN Internet Router (sir)
>> >
>> >
>> >
>> > On Thu, Jan 5, 2023 at 11:18 AM Mike Hammett <nanog@ics-il.net
>> > <mailto:nanog@ics-il.net>> wrote:
>> >
>> > Initially, my thought was to use community filtering to
>> push just
>> > IXes, customers, and defaults throughout the network, but
>> that's
>> > obviously still sub-optimal.
>> >
>> > I'd be surprised if a last mile network had a ton of
>> traffic going
>> > to any more than a few hundred prefixes.
>> >
>> >
>> > I think in a low-fib box at the edge of your network your
>> choices are:
>> > "the easy choice, get default, follow that"
>> >
>> > "send some limited set of prefixes to the device, and
>> default, so
>> > you MAY choose better for the initial hop away"
>> >
>> > you certainly can do the second with communities, or route-filters
>> > (prefix-list) on the senders, or....
>> > you can choose what prefixes make the cut (get the community(ies))
>> > based on traffic volumes or expected destination locality:
>> > "do not go east to go west!"
>> >
>> > these things will introduce toil and SOME suboptimal routing in
>> some
>> > instances... perhaps it's better than per flow choosing left/right
>> > though and the support calls related to that choice.
>> >
>> > In your NOLA / DFW / ATL example it's totally possible that the
>> > networks in question do something like:
>> > "low fib box in tier-2 city (NOLA), dfz capable/core devices in
>> > tier-1 city (DFW/ATL), and send default from left/right to NOLA"
>> >
>> > Could they send more prefixes than default? sure... do they
>> want to
>> > deal with the toil that induces? (probably not says your example).
>> >
>> > SDN isn't really an answer to this, though.. I don't think.
>> Unless you
>> > envision that to lower the toil ?
>> >
>>
>>
>>
RE: SDN Internet Router (sir) [ In reply to ]
I love that we can't even get a full week into the new year without beating the "let's overhaul BGP" drum. Some things never change. <3

Chris


-----Original Message-----
From: NANOG <nanog-bounces+chris.wright=commnetbroadband.com@nanog.org> On Behalf Of Joe Maimon
Sent: Thursday, January 5, 2023 5:51 PM
To: Mel Beckman <mel@beckman.org>; Mike Hammett <nanog@ics-il.net>
Cc: NANOG <nanog@nanog.org>
Subject: Re: SDN Internet Router (sir)

And here is another interesting approach Ive left open in my browser window for who knows how long

https://inog.net/files/iNOG14v_oliver_sourcerouting.pdf

The problem with BGP is that local actors can exact global costs trivially by consuming as many routing slots as they can get away with, add together BGP path decisions and Most Specific traffic-engineering is the goto knob. Sometimes you just want to say this is the route, do not accept any more specifics, unless this route is no longer the route. But you want that done automatically and correctly, reliably.

This is also why all the multi-homing approaches that do not involve global routing havent really taken off in any way to blunt table growth.
And likely wont.

See the aggregation factor in the routing report for how bad this is.

There have been lots of BGP protocol and feature updates, but unless your going to uniformly run new systems and enterprise systems that support all of them, its hard to decide to build your entire routing strategy around them.

That BGP unlike EIGRP never tried to tie together performance indicators with routing metrics feature or misdesign, you could debate that but it was always intentional. And opex has pretty much fallen down on the side of against IGP->BGP redistribution of prefixes, let alone performance metrics.

That eBGP prefix has no good reliable way of indicating that an advertised route sucks so bad that you should never attempt to use it unless as last resort, thats why we have AS-paths wrapping screen lines.

"finish IPv6 migration"? Letting IPv6 migration state factor as decision input on anything not directly related to IPv6 migration was never logical, just naively optimistic, and should be stamped out wherever encountered. If its good, use it now and Ipv6 will adopt it as well. If it isnt, why wait to find out?

Joe

Mel Beckman wrote:
> Mike,
>
> Thanks for that useful example. On a side note, Netflix is a thorn in
> all our sides :) You could put a localpref filter route to override
> the default for Netflix prefixes, but this impacts resilience. Since
> you peer with Netflix, I suspect we probably agree that Netflix’s
> ideas on traffic engineering are pretty one sided.
>
> I think it’s safe to say that BGP, which has scaled amazingly well,
> didn’t anticipate some of the big gorilla content systems. I don’t
> really see, though, how injecting FIB entries helps more than other
> methods. And as others have pointed out, the risk of creating routing
> loops is significant.
>
> Perhaps it is time to migrate to a new version of BGP. Projects like
> MBGP and FP-7‘s 4WARD are working on new follow-on routing models, but
> nothing is on the immediate horizon. I think we all thought we should
> finish IPv6 migration first :)
>
> -mel via cell
>
>> On Jan 5, 2023, at 1:11 PM, Mike Hammett <nanog@ics-il.net> wrote:
>>
>> ?
>> I hesitated to get too specific in examples because someone is going
>> to drag the conversation into the weeds.
>>
>> Let's take the the Dallas - New Orleans - Atlanta example where I
>> have a connection from New Orleans to Dallas and a connection from
>> New Orleans to Atlanta.
>>
>> Let's say I peer with Netflix in both markets. Netflix chooses to
>> serve me out of Atlanta, for whatever reason. Say my default route
>> sends my traffic to Dallas. That's not where Netflix wanted it, so
>> now I have to go from Dallas to Atlanta, whether that's my circuit or
>> across the public Internet. Potentially, it's on MPLS and it rides
>> back through the New Orleans router to get back to Atlanta. That's a
>> long trip when I already had a better path, the less-than-full-fib
>> router just didn't know about it. Given that Netflix is a sizable
>> amount of traffic in an eyeball ISP, that's a lot of traffic to be
>> going the wrong way. If the website for Viktor's Arctic Plunge in
>> Siberia was hosted in Atlanta, I wouldn't give two craps that the
>> traffic went the wrong way because A), I'll probably never go there
>> and B) when someone does, it won't be meaningfully enough traffic to
>> accommodate.
>>
>> Someone's going to tell me to put a full-table router in New Orleans.
>> Maybe I should. Okay, so maybe I have a POP in Ashford, Alabama. It
>> has transport to New Orleans and Atlanta. There aren't enough grains
>> of sugar in Ashford, Alabama to justify a current-generation, full
>> table router. Now I'm even closer to Atlanta, but default may point
>> to New Orleans.
>>
>>
>>
>> -----
>> Mike Hammett
>> Intelligent Computing Solutions <http://www.ics-il.com/>
>> <https://www.facebook.com/ICSIL><https://plus.google.com/+IntelligentComputingSolutionsDeKalb><https://www.linkedin.com/company/intelligent-computing-solutions><https://twitter.com/ICSIL>
>> Midwest Internet Exchange <http://www.midwest-ix.com/>
>> <https://www.facebook.com/mdwestix><https://www.linkedin.com/company/midwest-internet-exchange><https://twitter.com/mdwestix>
>> The Brothers WISP <http://www.thebrotherswisp.com/>
>> <https://www.facebook.com/thebrotherswisp><https://www.youtube.com/channel/UCXSdfxQv7SpoRQYNyLwntZg>
>> ------------------------------------------------------------------------
>> *From: *"Mel Beckman" <mel@beckman.org>
>> *To: *"Mike Hammett" <nanog@ics-il.net>
>> *Cc: *"Joe Maimon" <jmaimon@jmaimon.com>, "NANOG" <nanog@nanog.org>
>> *Sent: *Thursday, January 5, 2023 2:54:27 PM
>> *Subject: *Re: SDN Internet Router (sir)
>>
>> Mike,
>>
>> I’m not sure I understand what you mean by “suboptimal“ routing. Even
>> though the Internet uses AS path length for routing, many of those
>> path lengths are bogus, and don’t really represent any kind of path
>> performance value. For example, a single AS might hide many hops in
>> an MPLS network as a single hop, obscuring asymmetric routing and
>> other uglies. Prepending also occurs when destinations are trying to
>> enforce their own engineering policies, which often conflict with
>> yours or mine.
>>
>> So what do you mean by “suboptimal“? Are you thinking that the “best”
>> path in BGP tables actually meant you were getting a performance
>> benefit? Because that’s definitely not the case in today’s Internet.
>> Were were you thinking that you would be going along less congested
>> paths? That’s really at the mercy of the traffic engineering of
>> backbone providers over which we have no control.
>>
>> I generally populate local router FIBs to merel choose an exit point
>> for purposes of load balancing, and nothing more.
>>
>> -mel
>>
>> On Jan 5, 2023, at 12:38 PM, Mike Hammett <nanog@ics-il.net> wrote:
>>
>> ?
>> I guess I wasn't around for those days.
>>
>> As far as running out, again, assuming the tooling works
>> correctly, I'd think to target fewer routes than you could hold.
>> Maybe 1k routes is all one would need to get a significant
>> percent of the traffic. A lot of room to mess up if you can hold
>> 100k, 500k routes.
>>
>>
>>
>> -----
>> Mike Hammett
>> Intelligent Computing Solutions <http://www.ics-il.com/>
>> <https://www.facebook.com/ICSIL><https://plus.google.com/+IntelligentComputingSolutionsDeKalb><https://www.linkedin.com/company/intelligent-computing-solutions><https://twitter.com/ICSIL>
>> Midwest Internet Exchange <http://www.midwest-ix.com/>
>> <https://www.facebook.com/mdwestix><https://www.linkedin.com/company/midwest-internet-exchange><https://twitter.com/mdwestix>
>> The Brothers WISP <http://www.thebrotherswisp.com/>
>> <https://www.facebook.com/thebrotherswisp><https://www.youtube.com/channel/UCXSdfxQv7SpoRQYNyLwntZg>
>> ------------------------------------------------------------------------
>> *From: *"Joe Maimon" <jmaimon@jmaimon.com>
>> *To: *"Mike Hammett" <nanog@ics-il.net>, "Christopher Morrow"
>> <morrowc.lists@gmail.com>
>> *Cc: *"NANOG" <nanog@nanog.org>
>> *Sent: *Thursday, January 5, 2023 2:30:40 PM
>> *Subject: *Re: SDN Internet Router (sir)
>>
>>
>>
>> Mike Hammett wrote:
>> > I'm not concerned with which technology or buzzword gets the
>> job done,
>> > only that the job is done.
>> >
>> >
>> >
>> > Looking briefly at the couple of things out there, they're
>> evaluating
>> > the top X prefixes in terms of traffic reported by s-flow,
>> where X is
>> > the number I define, and those get pushed into the FIB. One
>> > recalculates every hour, one does so more quickly. How much is
>> > appropriate? I'm not sure. I can't imagine it would *NEED* to
>> be done
>> > all of that often, given the traffic/prefix density an eyeball
>> network
>> > will have. Default routes carry the rest. Default routes could be
>> > handled outside of this process, such that if this process
>> fails, you
>> > just get some sub-optimal routing until repaired. Maybe it doesn't
>> > filter properly and sends a bunch of routes. Then just have a
>> prefix
>> > limit set on the box. Maybe it sends the wrong prefixes. No
>> harm, no
>> > foul. If you're routing sub-optimally internally, when it does
>> hit a
>> > real router with a full FIB, it gets handled appropriately.
>>
>> Unless it loops.
>>
>> The rest sounds nice. But flow caching got a bad rap back in the
>> early
>> worm days. But thats because the situation was a little worse
>> back then.
>> Cache the wrong routes or run out of cache, router dies. So long as
>> thats not the case automating optimization is an extremely
>> valuable goal.
>>
>> >
>> >
>> > I would just be looking for solutions that influence what's in
>> the FIB
>> > and let the rest of the router work as the rest of the router
>> would.
>>
>> The problem comes when the router wont work at all without the FIB
>> routes, like in the olden days.
>> >
>> >
>> >
>> > -----
>> > Mike Hammett
>> > Intelligent Computing Solutions <http://www.ics-il.com/>
>> >
>> <https://www.facebook.com/ICSIL><https://plus.google.com/+IntelligentComputingSolutionsDeKalb><https://www.linkedin.com/company/intelligent-computing-solutions><https://twitter.com/ICSIL>
>> > Midwest Internet Exchange <http://www.midwest-ix.com/>
>> >
>> <https://www.facebook.com/mdwestix><https://www.linkedin.com/company/midwest-internet-exchange><https://twitter.com/mdwestix>
>> > The Brothers WISP <http://www.thebrotherswisp.com/>
>> >
>> <https://www.facebook.com/thebrotherswisp><https://www.youtube.com/channel/UCXSdfxQv7SpoRQYNyLwntZg>
>> >
>> ------------------------------------------------------------------------
>> > *From: *"Christopher Morrow" <morrowc.lists@gmail.com>
>> > *To: *"Mike Hammett" <nanog@ics-il.net>
>> > *Cc: *"Tom Beecher" <beecher@beecher.cc>, "NANOG" <nanog@nanog.org>
>> > *Sent: *Thursday, January 5, 2023 12:27:08 PM
>> > *Subject: *Re: SDN Internet Router (sir)
>> >
>> >
>> >
>> > On Thu, Jan 5, 2023 at 11:18 AM Mike Hammett <nanog@ics-il.net
>> > <mailto:nanog@ics-il.net>> wrote:
>> >
>> > Initially, my thought was to use community filtering to
>> push just
>> > IXes, customers, and defaults throughout the network, but
>> that's
>> > obviously still sub-optimal.
>> >
>> > I'd be surprised if a last mile network had a ton of
>> traffic going
>> > to any more than a few hundred prefixes.
>> >
>> >
>> > I think in a low-fib box at the edge of your network your
>> choices are:
>> > "the easy choice, get default, follow that"
>> >
>> > "send some limited set of prefixes to the device, and
>> default, so
>> > you MAY choose better for the initial hop away"
>> >
>> > you certainly can do the second with communities, or route-filters
>> > (prefix-list) on the senders, or....
>> > you can choose what prefixes make the cut (get the community(ies))
>> > based on traffic volumes or expected destination locality:
>> > "do not go east to go west!"
>> >
>> > these things will introduce toil and SOME suboptimal routing in
>> some
>> > instances... perhaps it's better than per flow choosing left/right
>> > though and the support calls related to that choice.
>> >
>> > In your NOLA / DFW / ATL example it's totally possible that the
>> > networks in question do something like:
>> > "low fib box in tier-2 city (NOLA), dfz capable/core devices in
>> > tier-1 city (DFW/ATL), and send default from left/right to NOLA"
>> >
>> > Could they send more prefixes than default? sure... do they
>> want to
>> > deal with the toil that induces? (probably not says your example).
>> >
>> > SDN isn't really an answer to this, though.. I don't think.
>> Unless you
>> > envision that to lower the toil ?
>> >
>>
>>
>>
Re: SDN Internet Router (sir) [ In reply to ]
Having wanted something similar recently, let me clarify what my desire
was.

I had a 1M FIB device I needed to get some additional life out of, running
ipv4 and ipv6. It also was running short on memory. This particular
device had 3 connections to the rest of the net which were running BGP, one
of which was a peering connection at an exchange so quite a few bgp
sessions on that one link.

What I would have liked to see is a solution to offload the BGP itself onto
a VM which would then aggregate the routes before installing into the FIB.
Just aggregating prefixes in my case would have taken care of what I
wanted to do. Instead, I ended up pointing default at the transit
providers and taking only the peering routes, and hoped the preferred
transit provider didn't end up with odd breakage too often before we could
get a replacement device sourced and installed which took way too long.

There are quite a lot of multihomed situations I've seen where the edge
device simply can't do a full routing table and buying something that can
isn't really possible due to budget or other constraints. Pointing default
at one of the upstreams works until connectivity breaks from that upstream
to the rest of the net. A solution which could dynamically look at the
bgp reachability and install a default route to whichever upstream was
better connected and then install additional routes overriding that default
to some predefined maximum number of prefixes would be useful. That way
you could gain the benefits of multi homing without needing a large fib
device. With some intelligent rules you could also ensure that the
overriding routes first covered any prefixes that didn't seem reachable
through the default provider, then additional rules could be prioritized
based on various other metrics so you'd get similar performance to a full
internet routing table with significantly fewer routing entries.

In the end though, I do expect that the hassle of setting up and managing
a solution like this is likely to result in most people deciding that it
isn't worth the extra complexity just to avoid upgrading a low fib device
where a larger one is really needed.







On Thu, Jan 5, 2023, 9:31 AM Mel Beckman <mel@beckman.org> wrote:

> ?Mike,
>
> Your original question was:
>
> “Given that the project was abandoned six years ago, are there any other
> efforts with a similar goal (more intelligently placing routes into FIBs of
> low-FIB capacity devices?”
>
> People then, respectfully, tried to clarify your request or explain why
> placing routes in a low-FIB capacity device isn’t seen as being beneficial.
> Only now have you added the desire to simply have “more than a default
> route” in such a router.
>
> You can, of course, have more than a default route today - e.g., through
> local pref and BGP communities for things such as company routes. You
> haven’t said what you define as “more intelligently”, so perhaps you can
> more clearly explain the problem you see with the current BGP capabilities
> via some examples.
>
> -mel
>
> On Jan 5, 2023, at 8:02 AM, Mike Hammett <nanog@ics-il.net> wrote:
>
> ?
> Then please bless the world with the right way.
>
> You acknowledge that not every router in a network needs to be fully DFZ
> capable, but then crap on my desire to have more than a default route in
> one.
>
>
>
> -----
> Mike Hammett
> Intelligent Computing Solutions <http://www.ics-il.com/>
> <https://www.facebook.com/ICSIL>
> <https://plus.google.com/+IntelligentComputingSolutionsDeKalb>
> <https://www.linkedin.com/company/intelligent-computing-solutions>
> <https://twitter.com/ICSIL>
> Midwest Internet Exchange <http://www.midwest-ix.com/>
> <https://www.facebook.com/mdwestix>
> <https://www.linkedin.com/company/midwest-internet-exchange>
> <https://twitter.com/mdwestix>
> The Brothers WISP <http://www.thebrotherswisp.com/>
> <https://www.facebook.com/thebrotherswisp>
> <https://www.youtube.com/channel/UCXSdfxQv7SpoRQYNyLwntZg>
> ------------------------------
> *From: *"Tom Beecher" <beecher@beecher.cc>
> *To: *"Mike Hammett" <nanog@ics-il.net>
> *Cc: *"Mel Beckman" <mel@beckman.org>, "NANOG" <nanog@nanog.org>
> *Sent: *Thursday, January 5, 2023 9:55:38 AM
> *Subject: *Re: SDN Internet Router (sir)
>
> "The right tool for the job" gets into a religious argument in assuming
>> that one's way to do the job is the only reasonable way to do the job
>
>
> I disagree that it's religious. I completely agree there are locations in
> networks that having full DFZ capable routers doesn't make technical or
> economic sense. But there have long been different products for those
> different use cases.
>
> To perhaps explain my viewpoint better,(and perhaps I didn't properly
> comprehend the problem you're aiming to solve) :
>
> If you are trying to use SDN stuff to shuffle routes on and off a box
> because you have the wrong sized routers in place, then I would argue
> you're doing it wrong.
>
> If you are trying to use SDN stuff to (as Christopher mentioned) make
> decisions that are not strictly LPM, and the equipment you have cannot do
> that, then that's different and entirely reasonable.
>
> If the second use case is more of what you were asking, then I apologize
> for misunderstanding.
>
>
>
> On Thu, Jan 5, 2023 at 9:57 AM Mike Hammett <nanog@ics-il.net> wrote:
>
>> "The right tool for the job" gets into a religious argument in assuming
>> that one's way to do the job is the only reasonable way to do the job.
>>
>> Large networks historically have a very poor (IMO) model of gigantic iron
>> in a few locations, which results in sub-optimal routing for the rest of
>> their network between those large POPs. I've heard time and time again that
>> someone buying service from a major network in say New Orleans has a first
>> hop of Dallas or Atlanta. I agree that full-route capable routers need to
>> be in the large, central locations, but it isn't cost effective to have
>> them at every POP, especially if you're a last-mile provider.
>>
>> I'd go into more examples of where it doesn't make sense to have
>> full-route routers everywhere, but I'm afraid that the Internet would then
>> focus on the examples instead of the core idea of intelligently putting
>> routes into the FIBs of low-FIB routers throughout my network.
>>
>>
>>
>> -----
>> Mike Hammett
>> Intelligent Computing Solutions <http://www.ics-il.com/>
>> <https://www.facebook.com/ICSIL>
>> <https://plus.google.com/+IntelligentComputingSolutionsDeKalb>
>> <https://www.linkedin.com/company/intelligent-computing-solutions>
>> <https://twitter.com/ICSIL>
>> Midwest Internet Exchange <http://www.midwest-ix.com/>
>> <https://www.facebook.com/mdwestix>
>> <https://www.linkedin.com/company/midwest-internet-exchange>
>> <https://twitter.com/mdwestix>
>> The Brothers WISP <http://www.thebrotherswisp.com/>
>> <https://www.facebook.com/thebrotherswisp>
>> <https://www.youtube.com/channel/UCXSdfxQv7SpoRQYNyLwntZg>
>> ------------------------------
>> *From: *"Tom Beecher" <beecher@beecher.cc>
>> *To: *"Mike Hammett" <nanog@ics-il.net>
>> *Cc: *"Mel Beckman" <mel@beckman.org>, "NANOG" <nanog@nanog.org>
>> *Sent: *Wednesday, January 4, 2023 7:36:58 AM
>> *Subject: *Re: SDN Internet Router (sir)
>>
>> Disagree that it’s a line in the sand. It’s use the right tool for the
>> job.
>>
>> If a device is low FIB, it’s that way for a reason. There are plenty of
>> ways to massage that with policy and software, depending on capabilities ,
>> but at the end of the day, trying to sort 10 pounds of shit to store in a 5
>> pound bag is eventually going to end up the same way.
>>
>> On Tue, Jan 3, 2023 at 13:18 Mike Hammett <nanog@ics-il.net> wrote:
>>
>>> There are likely more networks with 10 gigabit or less total external
>>> capacity than there are with more.
>>>
>>> Creating imaginary lines in the sand doesn't really help anyone.
>>>
>>>
>>>
>>>
>>> -----
>>> Mike Hammett
>>> Intelligent Computing Solutions <http://www.ics-il.com/>
>>> <https://www.facebook.com/ICSIL>
>>> <https://plus.google.com/+IntelligentComputingSolutionsDeKalb>
>>> <https://www.linkedin.com/company/intelligent-computing-solutions>
>>> <https://twitter.com/ICSIL>
>>> Midwest Internet Exchange <http://www.midwest-ix.com/>
>>> <https://www.facebook.com/mdwestix>
>>> <https://www.linkedin.com/company/midwest-internet-exchange>
>>> <https://twitter.com/mdwestix>
>>> The Brothers WISP <http://www.thebrotherswisp.com/>
>>> <https://www.facebook.com/thebrotherswisp>
>>> <https://www.youtube.com/channel/UCXSdfxQv7SpoRQYNyLwntZg>
>>> ------------------------------
>>> *From: *"Mel Beckman" <mel@beckman.org>
>>> *To: *"Mike Hammett" <nanog@ics-il.net>
>>> *Cc: *"NANOG" <nanog@nanog.org>
>>> *Sent: *Tuesday, January 3, 2023 10:57:34 AM
>>> *Subject: *Re: SDN Internet Router (sir)
>>>
>>> It’s not a problem, due to cheap, plentiful high-speed memory and rapid
>>> prefix search silicon in backbone routers. The entire Internet routing
>>> table consumes at most a few gigabytes when fully structured (and only a
>>> few hundred Mbytes stored flat). That’s less memory than your average
>>> laptop sports.
>>>
>>>
>>> Even in the worst case scenario, where every network decides to announce
>>> only its most specific prefixes, the BGP backbone would temporarily enter
>>> an oscillating state that generates a large number of routing updates into
>>> the inter-domain routing space. In this case, BGP route damping will
>>> quickly suppress the crazies while the backbone stabilizes.
>>>
>>>
>>> Small routers should not be taking full tables, since there is no point
>>> to them being in the default free zone. For large routers, neither memory
>>> nor CPU speed are an issue. High-speed routers operating in the
>>> default-free zone have a critical path in the forwarding decision for each
>>> packet: it needs to take less than the inter-packet arrival time for
>>> minimum-sized IP packets.
>>>
>>>
>>> This is easy to achieve with today’s hardware. A router line card with
>>> an aggregate line rate across all of its point-to-point interfaces of
>>> 10Tbps (readily available in today’s gear) can process packets with just a
>>> handful of cycles in the FIB Ternary Content Addressable Memory (TCAM)
>>> using ASIC-assisted lookups. TCAM is the most expensive component you’re
>>> paying for in such a router. It’s not cheap, but backbone routers
>>> don’t need to be cheap. They just need to not be memory-constrained.
>>>
>>> -mel via cell
>>>
>>> On Jan 3, 2023, at 7:47 AM, Mike Hammett <nanog@ics-il.net> wrote:
>>>
>>> ?
>>> https://github.com/dbarrosop/sir
>>>
>>> I came across this over the weekend. Given that the project was
>>> abandoned six years ago, are there any other efforts with a similar goal
>>> (more intelligently placing routes into FIBs of low-FIB capacity devices?
>>>
>>>
>>>
>>> -----
>>> Mike Hammett
>>> Intelligent Computing Solutions <http://www.ics-il.com/>
>>> <https://www.facebook.com/ICSIL>
>>> <https://plus.google.com/+IntelligentComputingSolutionsDeKalb>
>>> <https://www.linkedin.com/company/intelligent-computing-solutions>
>>> <https://twitter.com/ICSIL>
>>> Midwest Internet Exchange <http://www.midwest-ix.com/>
>>> <https://www.facebook.com/mdwestix>
>>> <https://www.linkedin.com/company/midwest-internet-exchange>
>>> <https://twitter.com/mdwestix>
>>> The Brothers WISP <http://www.thebrotherswisp.com/>
>>> <https://www.facebook.com/thebrotherswisp>
>>> <https://www.youtube.com/channel/UCXSdfxQv7SpoRQYNyLwntZg>
>>>
>>>
>>>
>>
>
Re: SDN Internet Router (sir) [ In reply to ]
I don't understand where the routing loop opportunity comes from. Ideally, you'd take the existing set of available routes and only send on the ones with high traffic levels, not necessarily injecting additional entries beyond what the table had, which would only make the too many routes problem worse. These things happen all of the time, just with different inputs (route reflectors, route servers, etc.). Those use traditional routing metrics. S-flow would just be another metric by which to filter on.

I don't know how the linked-to projects function in that regard.



-----
Mike Hammett
[ http://www.ics-il.com/ | Intelligent Computing Solutions ]
[ https://www.facebook.com/ICSIL ] [ https://plus.google.com/+IntelligentComputingSolutionsDeKalb ] [ https://www.linkedin.com/company/intelligent-computing-solutions ] [ https://twitter.com/ICSIL ]
[ http://www.midwest-ix.com/ | Midwest Internet Exchange ]
[ https://www.facebook.com/mdwestix ] [ https://www.linkedin.com/company/midwest-internet-exchange ] [ https://twitter.com/mdwestix ]
[ http://www.thebrotherswisp.com/ | The Brothers WISP ]
[ https://www.facebook.com/thebrotherswisp ] [ https://www.youtube.com/channel/UCXSdfxQv7SpoRQYNyLwntZg ]


From: "Mel Beckman" <mel@beckman.org>
To: "Mike Hammett" <nanog@ics-il.net>
Cc: "Joe Maimon" <jmaimon@jmaimon.com>, "NANOG" <nanog@nanog.org>
Sent: Thursday, January 5, 2023 4:36:09 PM
Subject: Re: SDN Internet Router (sir)

Mike,

Thanks for that useful example. On a side note, Netflix is a thorn in all our sides :) You could put a localpref filter route to override the default for Netflix prefixes, but this impacts resilience. Since you peer with Netflix, I suspect we probably agree that Netflix’s ideas on traffic engineering are pretty one sided.

I think it’s safe to say that BGP, which has scaled amazingly well, didn’t anticipate some of the big gorilla content systems. I don’t really see, though, how injecting FIB entries helps more than other methods. And as others have pointed out, the risk of creating routing loops is significant.

Perhaps it is time to migrate to a new version of BGP. Projects like MBGP and FP-7‘s 4WARD are working on new follow-on routing models, but nothing is on the immediate horizon. I think we all thought we should finish IPv6 migration first :)

-mel via cell



On Jan 5, 2023, at 1:11 PM, Mike Hammett <nanog@ics-il.net> wrote:





BQ_BEGIN

I hesitated to get too specific in examples because someone is going to drag the conversation into the weeds.

Let's take the the Dallas - New Orleans - Atlanta example where I have a connection from New Orleans to Dallas and a connection from New Orleans to Atlanta.

Let's say I peer with Netflix in both markets. Netflix chooses to serve me out of Atlanta, for whatever reason. Say my default route sends my traffic to Dallas. That's not where Netflix wanted it, so now I have to go from Dallas to Atlanta, whether that's my circuit or across the public Internet. Potentially, it's on MPLS and it rides back through the New Orleans router to get back to Atlanta. That's a long trip when I already had a better path, the less-than-full-fib router just didn't know about it. Given that Netflix is a sizable amount of traffic in an eyeball ISP, that's a lot of traffic to be going the wrong way. If the website for Viktor's Arctic Plunge in Siberia was hosted in Atlanta, I wouldn't give two craps that the traffic went the wrong way because A), I'll probably never go there and B) when someone does, it won't be meaningfully enough traffic to accommodate.

Someone's going to tell me to put a full-table router in New Orleans. Maybe I should. Okay, so maybe I have a POP in Ashford, Alabama. It has transport to New Orleans and Atlanta. There aren't enough grains of sugar in Ashford, Alabama to justify a current-generation, full table router. Now I'm even closer to Atlanta, but default may point to New Orleans.



-----
Mike Hammett
[ http://www.ics-il.com/ | Intelligent
Computing Solutions ]
[ https://www.facebook.com/ICSIL ] [ https://plus.google.com/+IntelligentComputingSolutionsDeKalb ] [ https://www.linkedin.com/company/intelligent-computing-solutions ] [ https://twitter.com/ICSIL ]
[ http://www.midwest-ix.com/ | Midwest
Internet Exchange ]
[ https://www.facebook.com/mdwestix ] [ https://www.linkedin.com/company/midwest-internet-exchange ] [ https://twitter.com/mdwestix ]
[ http://www.thebrotherswisp.com/ | The
Brothers WISP ]
[ https://www.facebook.com/thebrotherswisp ] [ https://www.youtube.com/channel/UCXSdfxQv7SpoRQYNyLwntZg ]

From: "Mel Beckman" <mel@beckman.org>
To: "Mike Hammett" <nanog@ics-il.net>
Cc: "Joe Maimon" <jmaimon@jmaimon.com>, "NANOG" <nanog@nanog.org>
Sent: Thursday, January 5, 2023 2:54:27 PM
Subject: Re: SDN Internet Router (sir)

Mike,

I’m not sure I understand what you mean by “suboptimal“ routing. Even though the Internet uses AS path length for routing, many of those path lengths are bogus, and don’t really represent any kind of path performance value. For example, a single AS might hide many hops in an MPLS network as a single hop, obscuring asymmetric routing and other uglies. Prepending also occurs when destinations are trying to enforce their own engineering policies, which often conflict with yours or mine.

So what do you mean by “suboptimal“? Are you thinking that the “best” path in BGP tables actually meant you were getting a performance benefit? Because that’s definitely not the case in today’s Internet. Were were you thinking that you would be going along less congested paths? That’s really at the mercy of the traffic engineering of backbone providers over which we have no control.

I generally populate local router FIBs to merel choose an exit point for purposes of load balancing, and nothing more.

-mel


BQ_BEGIN
On Jan 5, 2023, at 12:38 PM, Mike Hammett <nanog@ics-il.net> wrote:


BQ_END


BQ_BEGIN

I guess I wasn't around for those days.

As far as running out, again, assuming the tooling works correctly, I'd think to target fewer routes than you could hold. Maybe 1k routes is all one would need to get a significant percent of the traffic. A lot of room to mess up if you can hold 100k, 500k routes.



-----
Mike Hammett
[ http://www.ics-il.com/ | Intelligent
Computing Solutions ]
[ https://www.facebook.com/ICSIL ] [ https://plus.google.com/+IntelligentComputingSolutionsDeKalb ] [ https://www.linkedin.com/company/intelligent-computing-solutions ] [ https://twitter.com/ICSIL ]
[ http://www.midwest-ix.com/ | Midwest
Internet Exchange ]
[ https://www.facebook.com/mdwestix ] [ https://www.linkedin.com/company/midwest-internet-exchange ] [ https://twitter.com/mdwestix ]
[ http://www.thebrotherswisp.com/ | The
Brothers WISP ]
[ https://www.facebook.com/thebrotherswisp ] [ https://www.youtube.com/channel/UCXSdfxQv7SpoRQYNyLwntZg ]

From: "Joe Maimon" <jmaimon@jmaimon.com>
To: "Mike Hammett" <nanog@ics-il.net>, "Christopher Morrow" <morrowc.lists@gmail.com>
Cc: "NANOG" <nanog@nanog.org>
Sent: Thursday, January 5, 2023 2:30:40 PM
Subject: Re: SDN Internet Router (sir)



Mike Hammett wrote:
> I'm not concerned with which technology or buzzword gets the job done,
> only that the job is done.
>
>
>
> Looking briefly at the couple of things out there, they're evaluating
> the top X prefixes in terms of traffic reported by s-flow, where X is
> the number I define, and those get pushed into the FIB. One
> recalculates every hour, one does so more quickly. How much is
> appropriate? I'm not sure. I can't imagine it would *NEED* to be done
> all of that often, given the traffic/prefix density an eyeball network
> will have. Default routes carry the rest. Default routes could be
> handled outside of this process, such that if this process fails, you
> just get some sub-optimal routing until repaired. Maybe it doesn't
> filter properly and sends a bunch of routes. Then just have a prefix
> limit set on the box. Maybe it sends the wrong prefixes. No harm, no
> foul. If you're routing sub-optimally internally, when it does hit a
> real router with a full FIB, it gets handled appropriately.

Unless it loops.

The rest sounds nice. But flow caching got a bad rap back in the early
worm days. But thats because the situation was a little worse back then.
Cache the wrong routes or run out of cache, router dies. So long as
thats not the case automating optimization is an extremely valuable goal.

>
>
> I would just be looking for solutions that influence what's in the FIB
> and let the rest of the router work as the rest of the router would.

The problem comes when the router wont work at all without the FIB
routes, like in the olden days.
>
>
>
> -----
> Mike Hammett
> Intelligent Computing Solutions <http://www.ics-il.com/>
> <https://www.facebook.com/ICSIL><https://plus.google.com/+IntelligentComputingSolutionsDeKalb><https://www.linkedin.com/company/intelligent-computing-solutions><https://twitter.com/ICSIL>
> Midwest Internet Exchange <http://www.midwest-ix.com/>
> <https://www.facebook.com/mdwestix><https://www.linkedin.com/company/midwest-internet-exchange><https://twitter.com/mdwestix>
> The Brothers WISP <http://www.thebrotherswisp.com/>
> <https://www.facebook.com/thebrotherswisp><https://www.youtube.com/channel/UCXSdfxQv7SpoRQYNyLwntZg>
> ------------------------------------------------------------------------
> *From: *"Christopher Morrow" <morrowc.lists@gmail.com>
> *To: *"Mike Hammett" <nanog@ics-il.net>
> *Cc: *"Tom Beecher" <beecher@beecher.cc>, "NANOG" <nanog@nanog.org>
> *Sent: *Thursday, January 5, 2023 12:27:08 PM
> *Subject: *Re: SDN Internet Router (sir)
>
>
>
> On Thu, Jan 5, 2023 at 11:18 AM Mike Hammett <nanog@ics-il.net
> <mailto:nanog@ics-il.net>> wrote:
>
> Initially, my thought was to use community filtering to push just
> IXes, customers, and defaults throughout the network, but that's
> obviously still sub-optimal.
>
> I'd be surprised if a last mile network had a ton of traffic going
> to any more than a few hundred prefixes.
>
>
> I think in a low-fib box at the edge of your network your choices are:
> "the easy choice, get default, follow that"
>
> "send some limited set of prefixes to the device, and default, so
> you MAY choose better for the initial hop away"
>
> you certainly can do the second with communities, or route-filters
> (prefix-list) on the senders, or....
> you can choose what prefixes make the cut (get the community(ies))
> based on traffic volumes or expected destination locality:
> "do not go east to go west!"
>
> these things will introduce toil and SOME suboptimal routing in some
> instances... perhaps it's better than per flow choosing left/right
> though and the support calls related to that choice.
>
> In your NOLA / DFW / ATL example it's totally possible that the
> networks in question do something like:
> "low fib box in tier-2 city (NOLA), dfz capable/core devices in
> tier-1 city (DFW/ATL), and send default from left/right to NOLA"
>
> Could they send more prefixes than default? sure... do they want to
> deal with the toil that induces? (probably not says your example).
>
> SDN isn't really an answer to this, though.. I don't think. Unless you
> envision that to lower the toil ?
>



BQ_END



BQ_END
Re: SDN Internet Router (sir) [ In reply to ]
I didn't know where the conversation on routing flows differently came from and just ignored it. Now I realize the thread has SDN in the title. SDN can mean many things and if someone didn't properly evaluate the link in the original post, their understanding could be tainted by their experiences.



-----
Mike Hammett
[ http://www.ics-il.com/ | Intelligent Computing Solutions ]
[ https://www.facebook.com/ICSIL ] [ https://plus.google.com/+IntelligentComputingSolutionsDeKalb ] [ https://www.linkedin.com/company/intelligent-computing-solutions ] [ https://twitter.com/ICSIL ]
[ http://www.midwest-ix.com/ | Midwest Internet Exchange ]
[ https://www.facebook.com/mdwestix ] [ https://www.linkedin.com/company/midwest-internet-exchange ] [ https://twitter.com/mdwestix ]
[ http://www.thebrotherswisp.com/ | The Brothers WISP ]
[ https://www.facebook.com/thebrotherswisp ] [ https://www.youtube.com/channel/UCXSdfxQv7SpoRQYNyLwntZg ]


From: "Mike Hammett" <nanog@ics-il.net>
To: "NANOG" <nanog@nanog.org>
Sent: Tuesday, January 3, 2023 9:43:58 AM
Subject: SDN Internet Router (sir)

https://github.com/dbarrosop/sir

I came across this over the weekend. Given that the project was abandoned six years ago, are there any other efforts with a similar goal (more intelligently placing routes into FIBs of low-FIB capacity devices?



-----
Mike Hammett
[ http://www.ics-il.com/ | Intelligent Computing Solutions ]
[ https://www.facebook.com/ICSIL ] [ https://plus.google.com/+IntelligentComputingSolutionsDeKalb ] [ https://www.linkedin.com/company/intelligent-computing-solutions ] [ https://twitter.com/ICSIL ]
[ http://www.midwest-ix.com/ | Midwest Internet Exchange ]
[ https://www.facebook.com/mdwestix ] [ https://www.linkedin.com/company/midwest-internet-exchange ] [ https://twitter.com/mdwestix ]
[ http://www.thebrotherswisp.com/ | The Brothers WISP ]
[ https://www.facebook.com/thebrotherswisp ] [ https://www.youtube.com/channel/UCXSdfxQv7SpoRQYNyLwntZg ]
Re: SDN Internet Router (sir) [ In reply to ]
Christopher Morrow wrote:

> Some of the reasoning behind 'i need/want to do SDN things' is 'low fib
> device' sort of reasonings.

What?

SDN is a poor alternative for those who can't construct a
network with fully automated QoS guarantee.

Even with SDN, QoS guarantee implies QoS routing requiring
dedicated routing table entry for each flow, which will not
shrink but bloat routing tables regardless of whether you
call it FIB or not.

Masataka Ohta
Re: SDN Internet Router (sir) [ In reply to ]
Thanks for this example.

It sounds like you are describing egress peer engineering, but kinda in
reverse.

In 'traditional' EPE, the routers have all the routes, and you are using
the external controller to perform the performance tests that matter to
you, and signal the network where to take the traffic based on those tests.

It seems like you want to do the same thing , but instead of having the
controller signal the network where to carry bits, you want the controller
to signal the networks what routes are present, and direct the bits that
way.

Do I have this right?

On Thu, Jan 5, 2023 at 4:12 PM Mike Hammett <nanog@ics-il.net> wrote:

> I hesitated to get too specific in examples because someone is going to
> drag the conversation into the weeds.
>
> Let's take the the Dallas - New Orleans - Atlanta example where I have a
> connection from New Orleans to Dallas and a connection from New Orleans to
> Atlanta.
>
> Let's say I peer with Netflix in both markets. Netflix chooses to serve me
> out of Atlanta, for whatever reason. Say my default route sends my traffic
> to Dallas. That's not where Netflix wanted it, so now I have to go from
> Dallas to Atlanta, whether that's my circuit or across the public Internet.
> Potentially, it's on MPLS and it rides back through the New Orleans router
> to get back to Atlanta. That's a long trip when I already had a better
> path, the less-than-full-fib router just didn't know about it. Given that
> Netflix is a sizable amount of traffic in an eyeball ISP, that's a lot of
> traffic to be going the wrong way. If the website for Viktor's Arctic
> Plunge in Siberia was hosted in Atlanta, I wouldn't give two craps that the
> traffic went the wrong way because A), I'll probably never go there and B)
> when someone does, it won't be meaningfully enough traffic to accommodate.
>
> Someone's going to tell me to put a full-table router in New Orleans.
> Maybe I should. Okay, so maybe I have a POP in Ashford, Alabama. It has
> transport to New Orleans and Atlanta. There aren't enough grains of sugar
> in Ashford, Alabama to justify a current-generation, full table router. Now
> I'm even closer to Atlanta, but default may point to New Orleans.
>
>
>
> -----
> Mike Hammett
> Intelligent Computing Solutions <http://www.ics-il.com/>
> <https://www.facebook.com/ICSIL>
> <https://plus.google.com/+IntelligentComputingSolutionsDeKalb>
> <https://www.linkedin.com/company/intelligent-computing-solutions>
> <https://twitter.com/ICSIL>
> Midwest Internet Exchange <http://www.midwest-ix.com/>
> <https://www.facebook.com/mdwestix>
> <https://www.linkedin.com/company/midwest-internet-exchange>
> <https://twitter.com/mdwestix>
> The Brothers WISP <http://www.thebrotherswisp.com/>
> <https://www.facebook.com/thebrotherswisp>
> <https://www.youtube.com/channel/UCXSdfxQv7SpoRQYNyLwntZg>
> ------------------------------
> *From: *"Mel Beckman" <mel@beckman.org>
> *To: *"Mike Hammett" <nanog@ics-il.net>
> *Cc: *"Joe Maimon" <jmaimon@jmaimon.com>, "NANOG" <nanog@nanog.org>
> *Sent: *Thursday, January 5, 2023 2:54:27 PM
> *Subject: *Re: SDN Internet Router (sir)
>
> Mike,
>
> I’m not sure I understand what you mean by “suboptimal“ routing. Even
> though the Internet uses AS path length for routing, many of those path
> lengths are bogus, and don’t really represent any kind of path performance
> value. For example, a single AS might hide many hops in an MPLS network as
> a single hop, obscuring asymmetric routing and other uglies. Prepending
> also occurs when destinations are trying to enforce their own engineering
> policies, which often conflict with yours or mine.
>
> So what do you mean by “suboptimal“? Are you thinking that the “best” path
> in BGP tables actually meant you were getting a performance benefit?
> Because that’s definitely not the case in today’s Internet. Were were you
> thinking that you would be going along less congested paths? That’s really
> at the mercy of the traffic engineering of backbone providers over which we
> have no control.
>
> I generally populate local router FIBs to merel choose an exit point for
> purposes of load balancing, and nothing more.
>
> -mel
>
> On Jan 5, 2023, at 12:38 PM, Mike Hammett <nanog@ics-il.net> wrote:
>
> ?
> I guess I wasn't around for those days.
>
> As far as running out, again, assuming the tooling works correctly, I'd
> think to target fewer routes than you could hold. Maybe 1k routes is all
> one would need to get a significant percent of the traffic. A lot of room
> to mess up if you can hold 100k, 500k routes.
>
>
>
> -----
> Mike Hammett
> Intelligent Computing Solutions <http://www.ics-il.com/>
> <https://www.facebook.com/ICSIL>
> <https://plus.google.com/+IntelligentComputingSolutionsDeKalb>
> <https://www.linkedin.com/company/intelligent-computing-solutions>
> <https://twitter.com/ICSIL>
> Midwest Internet Exchange <http://www.midwest-ix.com/>
> <https://www.facebook.com/mdwestix>
> <https://www.linkedin.com/company/midwest-internet-exchange>
> <https://twitter.com/mdwestix>
> The Brothers WISP <http://www.thebrotherswisp.com/>
> <https://www.facebook.com/thebrotherswisp>
> <https://www.youtube.com/channel/UCXSdfxQv7SpoRQYNyLwntZg>
> ------------------------------
> *From: *"Joe Maimon" <jmaimon@jmaimon.com>
> *To: *"Mike Hammett" <nanog@ics-il.net>, "Christopher Morrow" <
> morrowc.lists@gmail.com>
> *Cc: *"NANOG" <nanog@nanog.org>
> *Sent: *Thursday, January 5, 2023 2:30:40 PM
> *Subject: *Re: SDN Internet Router (sir)
>
>
>
> Mike Hammett wrote:
> > I'm not concerned with which technology or buzzword gets the job done,
> > only that the job is done.
> >
> >
> >
> > Looking briefly at the couple of things out there, they're evaluating
> > the top X prefixes in terms of traffic reported by s-flow, where X is
> > the number I define, and those get pushed into the FIB. One
> > recalculates every hour, one does so more quickly. How much is
> > appropriate? I'm not sure. I can't imagine it would *NEED* to be done
> > all of that often, given the traffic/prefix density an eyeball network
> > will have. Default routes carry the rest. Default routes could be
> > handled outside of this process, such that if this process fails, you
> > just get some sub-optimal routing until repaired. Maybe it doesn't
> > filter properly and sends a bunch of routes. Then just have a prefix
> > limit set on the box. Maybe it sends the wrong prefixes. No harm, no
> > foul. If you're routing sub-optimally internally, when it does hit a
> > real router with a full FIB, it gets handled appropriately.
>
> Unless it loops.
>
> The rest sounds nice. But flow caching got a bad rap back in the early
> worm days. But thats because the situation was a little worse back then.
> Cache the wrong routes or run out of cache, router dies. So long as
> thats not the case automating optimization is an extremely valuable goal.
>
> >
> >
> > I would just be looking for solutions that influence what's in the FIB
> > and let the rest of the router work as the rest of the router would.
>
> The problem comes when the router wont work at all without the FIB
> routes, like in the olden days.
> >
> >
> >
> > -----
> > Mike Hammett
> > Intelligent Computing Solutions <http://www.ics-il.com/>
> > <https://www.facebook.com/ICSIL><
> https://plus.google.com/+IntelligentComputingSolutionsDeKalb><
> https://www.linkedin.com/company/intelligent-computing-solutions><
> https://twitter.com/ICSIL>
> > Midwest Internet Exchange <http://www.midwest-ix.com/>
> > <https://www.facebook.com/mdwestix><
> https://www.linkedin.com/company/midwest-internet-exchange><
> https://twitter.com/mdwestix>
> > The Brothers WISP <http://www.thebrotherswisp.com/>
> > <https://www.facebook.com/thebrotherswisp><
> https://www.youtube.com/channel/UCXSdfxQv7SpoRQYNyLwntZg>
> > ------------------------------------------------------------------------
> > *From: *"Christopher Morrow" <morrowc.lists@gmail.com>
> > *To: *"Mike Hammett" <nanog@ics-il.net>
> > *Cc: *"Tom Beecher" <beecher@beecher.cc>, "NANOG" <nanog@nanog.org>
> > *Sent: *Thursday, January 5, 2023 12:27:08 PM
> > *Subject: *Re: SDN Internet Router (sir)
> >
> >
> >
> > On Thu, Jan 5, 2023 at 11:18 AM Mike Hammett <nanog@ics-il.net
> > <mailto:nanog@ics-il.net>> wrote:
> >
> > Initially, my thought was to use community filtering to push just
> > IXes, customers, and defaults throughout the network, but that's
> > obviously still sub-optimal.
> >
> > I'd be surprised if a last mile network had a ton of traffic going
> > to any more than a few hundred prefixes.
> >
> >
> > I think in a low-fib box at the edge of your network your choices are:
> > "the easy choice, get default, follow that"
> >
> > "send some limited set of prefixes to the device, and default, so
> > you MAY choose better for the initial hop away"
> >
> > you certainly can do the second with communities, or route-filters
> > (prefix-list) on the senders, or....
> > you can choose what prefixes make the cut (get the community(ies))
> > based on traffic volumes or expected destination locality:
> > "do not go east to go west!"
> >
> > these things will introduce toil and SOME suboptimal routing in some
> > instances... perhaps it's better than per flow choosing left/right
> > though and the support calls related to that choice.
> >
> > In your NOLA / DFW / ATL example it's totally possible that the
> > networks in question do something like:
> > "low fib box in tier-2 city (NOLA), dfz capable/core devices in
> > tier-1 city (DFW/ATL), and send default from left/right to NOLA"
> >
> > Could they send more prefixes than default? sure... do they want to
> > deal with the toil that induces? (probably not says your example).
> >
> > SDN isn't really an answer to this, though.. I don't think. Unless you
> > envision that to lower the toil ?
> >
>
>
>
>
Re: SDN Internet Router (sir) [ In reply to ]
On Fri, 6 Jan 2023, 17:07 Masataka Ohta, <mohta@necom830.hpcl.titech.ac.jp>
wrote:

> Christopher Morrow wrote:
>
> > Some of the reasoning behind 'i need/want to do SDN things' is 'low fib
> > device' sort of reasonings.
>
> What?
>
> SDN is a poor alternative for those who can't construct a
> network with fully automated QoS guarantee.
>
> Even with SDN, QoS guarantee implies QoS routing requiring
> dedicated routing table entry for each flow, which will not
> shrink but bloat routing tables regardless of whether you
> call it FIB or not.


I'm going to have to "what?" you right back there. You don't need to
program each individual flow, just each FEC.

SDN does not imply QoS routing, it's just one aspect of it. Some use it for
classifying guest traffic etc.

M

>
>
Re: SDN Internet Router (sir) [ In reply to ]
Maybe?

I don't need any additional performance tests, though. Just watching which prefixes are the top talkers and leaving the rest to default.


I'm not looking at this to do what a BGP optimizier would do and find the best tested path to the top talkers and then massage BGP to get it routed that way. Determine the top talkers, then let BGP do its thing for those top talkers.


I don't want to manually say X traffic from Y POP manually goes here, but I don't want to just leave it to default routing either. Something in the middle.




-----
Mike Hammett
Intelligent Computing Solutions

Midwest Internet Exchange

The Brothers WISP

----- Original Message -----

From: "Tom Beecher" <beecher@beecher.cc>
To: "Mike Hammett" <nanog@ics-il.net>
Cc: "Mel Beckman" <mel@beckman.org>, "NANOG" <nanog@nanog.org>
Sent: Friday, January 6, 2023 9:16:19 AM
Subject: Re: SDN Internet Router (sir)


Thanks for this example.


It sounds like you are describing egress peer engineering, but kinda in reverse.


In 'traditional' EPE, the routers have all the routes, and you are using the external controller to perform the performance tests that matter to you, and signal the network where to take the traffic based on those tests.


It seems like you want to do the same thing , but instead of having the controller signal the network where to carry bits, you want the controller to signal the networks what routes are present, and direct the bits that way.


Do I have this right?


On Thu, Jan 5, 2023 at 4:12 PM Mike Hammett < nanog@ics-il.net > wrote:






I hesitated to get too specific in examples because someone is going to drag the conversation into the weeds.

Let's take the the Dallas - New Orleans - Atlanta example where I have a connection from New Orleans to Dallas and a connection from New Orleans to Atlanta.


Let's say I peer with Netflix in both markets. Netflix chooses to serve me out of Atlanta, for whatever reason. Say my default route sends my traffic to Dallas. That's not where Netflix wanted it, so now I have to go from Dallas to Atlanta, whether that's my circuit or across the public Internet. Potentially, it's on MPLS and it rides back through the New Orleans router to get back to Atlanta. That's a long trip when I already had a better path, the less-than-full-fib router just didn't know about it. Given that Netflix is a sizable amount of traffic in an eyeball ISP, that's a lot of traffic to be going the wrong way. If the website for Viktor's Arctic Plunge in Siberia was hosted in Atlanta, I wouldn't give two craps that the traffic went the wrong way because A), I'll probably never go there and B) when someone does, it won't be meaningfully enough traffic to accommodate.


Someone's going to tell me to put a full-table router in New Orleans. Maybe I should. Okay, so maybe I have a POP in Ashford, Alabama. It has transport to New Orleans and Atlanta. There aren't enough grains of sugar in Ashford, Alabama to justify a current-generation, full table router. Now I'm even closer to Atlanta, but default may point to New Orleans.




-----
Mike Hammett
Intelligent Computing Solutions

Midwest Internet Exchange

The Brothers WISP



From: "Mel Beckman" < mel@beckman.org >
To: "Mike Hammett" < nanog@ics-il.net >
Cc: "Joe Maimon" < jmaimon@jmaimon.com >, "NANOG" < nanog@nanog.org >
Sent: Thursday, January 5, 2023 2:54:27 PM
Subject: Re: SDN Internet Router (sir)


Mike,


I’m not sure I understand what you mean by “suboptimal“ routing. Even though the Internet uses AS path length for routing, many of those path lengths are bogus, and don’t really represent any kind of path performance value. For example, a single AS might hide many hops in an MPLS network as a single hop, obscuring asymmetric routing and other uglies. Prepending also occurs when destinations are trying to enforce their own engineering policies, which often conflict with yours or mine.


So what do you mean by “suboptimal“? Are you thinking that the “best” path in BGP tables actually meant you were getting a performance benefit? Because that’s definitely not the case in today’s Internet. Were were you thinking that you would be going along less congested paths? That’s really at the mercy of the traffic engineering of backbone providers over which we have no control.


I generally populate local router FIBs to merel choose an exit point for purposes of load balancing, and nothing more.


-mel



<blockquote>
On Jan 5, 2023, at 12:38 PM, Mike Hammett < nanog@ics-il.net > wrote:




<blockquote>


I guess I wasn't around for those days.


As far as running out, again, assuming the tooling works correctly, I'd think to target fewer routes than you could hold. Maybe 1k routes is all one would need to get a significant percent of the traffic. A lot of room to mess up if you can hold 100k, 500k routes.





-----
Mike Hammett
Intelligent Computing Solutions

Midwest Internet Exchange

The Brothers WISP



From: "Joe Maimon" < jmaimon@jmaimon.com >
To: "Mike Hammett" < nanog@ics-il.net >, "Christopher Morrow" < morrowc.lists@gmail.com >
Cc: "NANOG" < nanog@nanog.org >
Sent: Thursday, January 5, 2023 2:30:40 PM
Subject: Re: SDN Internet Router (sir)



Mike Hammett wrote:
> I'm not concerned with which technology or buzzword gets the job done,
> only that the job is done.
>
>
>
> Looking briefly at the couple of things out there, they're evaluating
> the top X prefixes in terms of traffic reported by s-flow, where X is
> the number I define, and those get pushed into the FIB. One
> recalculates every hour, one does so more quickly. How much is
> appropriate? I'm not sure. I can't imagine it would *NEED* to be done
> all of that often, given the traffic/prefix density an eyeball network
> will have. Default routes carry the rest. Default routes could be
> handled outside of this process, such that if this process fails, you
> just get some sub-optimal routing until repaired. Maybe it doesn't
> filter properly and sends a bunch of routes. Then just have a prefix
> limit set on the box. Maybe it sends the wrong prefixes. No harm, no
> foul. If you're routing sub-optimally internally, when it does hit a
> real router with a full FIB, it gets handled appropriately.

Unless it loops.

The rest sounds nice. But flow caching got a bad rap back in the early
worm days. But thats because the situation was a little worse back then.
Cache the wrong routes or run out of cache, router dies. So long as
thats not the case automating optimization is an extremely valuable goal.

>
>
> I would just be looking for solutions that influence what's in the FIB
> and let the rest of the router work as the rest of the router would.

The problem comes when the router wont work at all without the FIB
routes, like in the olden days.
>
>
>
> -----
> Mike Hammett
> Intelligent Computing Solutions < http://www.ics-il.com/ >
> < https://www.facebook.com/ICSIL >< https://plus.google.com/+IntelligentComputingSolutionsDeKalb >< https://www.linkedin.com/company/intelligent-computing-solutions >< https://twitter.com/ICSIL >
> Midwest Internet Exchange < http://www.midwest-ix.com/ >
> < https://www.facebook.com/mdwestix >< https://www.linkedin.com/company/midwest-internet-exchange >< https://twitter.com/mdwestix >
> The Brothers WISP < http://www.thebrotherswisp.com/ >
> < https://www.facebook.com/thebrotherswisp >< https://www.youtube.com/channel/UCXSdfxQv7SpoRQYNyLwntZg >
> ------------------------------------------------------------------------
> *From: *"Christopher Morrow" < morrowc.lists@gmail.com >
> *To: *"Mike Hammett" < nanog@ics-il.net >
> *Cc: *"Tom Beecher" < beecher@beecher.cc >, "NANOG" < nanog@nanog.org >
> *Sent: *Thursday, January 5, 2023 12:27:08 PM
> *Subject: *Re: SDN Internet Router (sir)
>
>
>
> On Thu, Jan 5, 2023 at 11:18 AM Mike Hammett < nanog@ics-il.net
> <mailto: nanog@ics-il.net >> wrote:
>
> Initially, my thought was to use community filtering to push just
> IXes, customers, and defaults throughout the network, but that's
> obviously still sub-optimal.
>
> I'd be surprised if a last mile network had a ton of traffic going
> to any more than a few hundred prefixes.
>
>
> I think in a low-fib box at the edge of your network your choices are:
> "the easy choice, get default, follow that"
>
> "send some limited set of prefixes to the device, and default, so
> you MAY choose better for the initial hop away"
>
> you certainly can do the second with communities, or route-filters
> (prefix-list) on the senders, or....
> you can choose what prefixes make the cut (get the community(ies))
> based on traffic volumes or expected destination locality:
> "do not go east to go west!"
>
> these things will introduce toil and SOME suboptimal routing in some
> instances... perhaps it's better than per flow choosing left/right
> though and the support calls related to that choice.
>
> In your NOLA / DFW / ATL example it's totally possible that the
> networks in question do something like:
> "low fib box in tier-2 city (NOLA), dfz capable/core devices in
> tier-1 city (DFW/ATL), and send default from left/right to NOLA"
>
> Could they send more prefixes than default? sure... do they want to
> deal with the toil that induces? (probably not says your example).
>
> SDN isn't really an answer to this, though.. I don't think. Unless you
> envision that to lower the toil ?
>



</blockquote>


</blockquote>
Re: SDN Internet Router (sir) [ In reply to ]
On Fri, 6 Jan 2023, 11:25 Forrest Christian (List Account), <
lists@packetflux.com> wrote:

> In the end though, I do expect that the hassle of setting up and managing
> a solution like this is likely to result in most people deciding that it
> isn't worth the extra complexity just to avoid upgrading a low fib device
> where a larger one is really needed.
>

Quite the contrary, nearly 10 years ago (just before SIR was released) I
was doing this precisely because the lower fib box was a good tradeoff
between ports and cost, and needed something to do IXP/PNI peering with.
Only instead of running the sflow analysis on the box, I was exporting it
elsewhere and pushing prefix filters every once in a while to make sure the
highest traffic prefixes were served locally.

Ultimately, it's part of the TCO of your network, and when traffic volumes
are high, you look for any opportunity to reduce that CapEx cost of a fully
high FIB router.

It sounds like the idea still has value!

M
Re: SDN Internet Router (sir) [ In reply to ]
I suspect it always will have value, whether it's peering routers, POP routers, multi-homed customer routers, etc.




-----
Mike Hammett
Intelligent Computing Solutions

Midwest Internet Exchange

The Brothers WISP

----- Original Message -----

From: "Matthew Walster via NANOG" <nanog@nanog.org>
To: "Forrest Christian (List Account)" <lists@packetflux.com>
Cc: "NANOG" <nanog@nanog.org>
Sent: Friday, January 6, 2023 9:34:16 AM
Subject: Re: SDN Internet Router (sir)






On Fri, 6 Jan 2023, 11:25 Forrest Christian (List Account), < lists@packetflux.com > wrote:




In the end though, I do expect that the hassle of setting up and managing a solution like this is likely to result in most people deciding that it isn't worth the extra complexity just to avoid upgrading a low fib device where a larger one is really needed.




Quite the contrary, nearly 10 years ago (just before SIR was released) I was doing this precisely because the lower fib box was a good tradeoff between ports and cost, and needed something to do IXP/PNI peering with. Only instead of running the sflow analysis on the box, I was exporting it elsewhere and pushing prefix filters every once in a while to make sure the highest traffic prefixes were served locally.


Ultimately, it's part of the TCO of your network, and when traffic volumes are high, you look for any opportunity to reduce that CapEx cost of a fully high FIB router.


It sounds like the idea still has value!


M



<blockquote>

</blockquote>
Re: SDN Internet Router (sir) [ In reply to ]
Gotcha.

Setup a Quagga/Bird box. Do your top talker analysis , use that box to
inject the routes you deem important with communities. On your routers ,
create policy structure to only take a default plus those communities.

Obviously lots of devils in the details of the implementation , but
something like that is all you need to do.

On Fri, Jan 6, 2023 at 10:29 Mike Hammett <nanog@ics-il.net> wrote:

> Maybe?
>
> I don't need any additional performance tests, though. Just watching which
> prefixes are the top talkers and leaving the rest to default.
>
> I'm not looking at this to do what a BGP optimizier would do and find the
> best tested path to the top talkers and then massage BGP to get it routed
> that way. Determine the top talkers, then let BGP do its thing for those
> top talkers.
>
> I don't want to manually say X traffic from Y POP manually goes here, but
> I don't want to just leave it to default routing either. Something in the
> middle.
>
>
>
> -----
> Mike Hammett
> Intelligent Computing Solutions <http://www.ics-il.com/>
> <https://www.facebook.com/ICSIL>
> <https://plus.google.com/+IntelligentComputingSolutionsDeKalb>
> <https://www.linkedin.com/company/intelligent-computing-solutions>
> <https://twitter.com/ICSIL>
> Midwest Internet Exchange <http://www.midwest-ix.com/>
> <https://www.facebook.com/mdwestix>
> <https://www.linkedin.com/company/midwest-internet-exchange>
> <https://twitter.com/mdwestix>
> The Brothers WISP <http://www.thebrotherswisp.com/>
> <https://www.facebook.com/thebrotherswisp>
> <https://www.youtube.com/channel/UCXSdfxQv7SpoRQYNyLwntZg>
> ------------------------------
> *From: *"Tom Beecher" <beecher@beecher.cc>
> *To: *"Mike Hammett" <nanog@ics-il.net>
> *Cc: *"Mel Beckman" <mel@beckman.org>, "NANOG" <nanog@nanog.org>
> *Sent: *Friday, January 6, 2023 9:16:19 AM
>
> *Subject: *Re: SDN Internet Router (sir)
>
> Thanks for this example.
>
> It sounds like you are describing egress peer engineering, but kinda in
> reverse.
>
> In 'traditional' EPE, the routers have all the routes, and you are using
> the external controller to perform the performance tests that matter to
> you, and signal the network where to take the traffic based on those tests.
>
> It seems like you want to do the same thing , but instead of having the
> controller signal the network where to carry bits, you want the controller
> to signal the networks what routes are present, and direct the bits that
> way.
>
> Do I have this right?
>
> On Thu, Jan 5, 2023 at 4:12 PM Mike Hammett <nanog@ics-il.net> wrote:
>
>> I hesitated to get too specific in examples because someone is going to
>> drag the conversation into the weeds.
>>
>> Let's take the the Dallas - New Orleans - Atlanta example where I have a
>> connection from New Orleans to Dallas and a connection from New Orleans to
>> Atlanta.
>>
>> Let's say I peer with Netflix in both markets. Netflix chooses to serve
>> me out of Atlanta, for whatever reason. Say my default route sends my
>> traffic to Dallas. That's not where Netflix wanted it, so now I have to go
>> from Dallas to Atlanta, whether that's my circuit or across the public
>> Internet. Potentially, it's on MPLS and it rides back through the New
>> Orleans router to get back to Atlanta. That's a long trip when I already
>> had a better path, the less-than-full-fib router just didn't know about it.
>> Given that Netflix is a sizable amount of traffic in an eyeball ISP, that's
>> a lot of traffic to be going the wrong way. If the website for Viktor's
>> Arctic Plunge in Siberia was hosted in Atlanta, I wouldn't give two craps
>> that the traffic went the wrong way because A), I'll probably never go
>> there and B) when someone does, it won't be meaningfully enough traffic to
>> accommodate.
>>
>> Someone's going to tell me to put a full-table router in New Orleans.
>> Maybe I should. Okay, so maybe I have a POP in Ashford, Alabama. It has
>> transport to New Orleans and Atlanta. There aren't enough grains of sugar
>> in Ashford, Alabama to justify a current-generation, full table router. Now
>> I'm even closer to Atlanta, but default may point to New Orleans.
>>
>>
>>
>> -----
>> Mike Hammett
>> Intelligent Computing Solutions <http://www.ics-il.com/>
>> <https://www.facebook.com/ICSIL>
>> <https://plus.google.com/+IntelligentComputingSolutionsDeKalb>
>> <https://www.linkedin.com/company/intelligent-computing-solutions>
>> <https://twitter.com/ICSIL>
>> Midwest Internet Exchange <http://www.midwest-ix.com/>
>> <https://www.facebook.com/mdwestix>
>> <https://www.linkedin.com/company/midwest-internet-exchange>
>> <https://twitter.com/mdwestix>
>> The Brothers WISP <http://www.thebrotherswisp.com/>
>> <https://www.facebook.com/thebrotherswisp>
>> <https://www.youtube.com/channel/UCXSdfxQv7SpoRQYNyLwntZg>
>> ------------------------------
>> *From: *"Mel Beckman" <mel@beckman.org>
>> *To: *"Mike Hammett" <nanog@ics-il.net>
>> *Cc: *"Joe Maimon" <jmaimon@jmaimon.com>, "NANOG" <nanog@nanog.org>
>> *Sent: *Thursday, January 5, 2023 2:54:27 PM
>> *Subject: *Re: SDN Internet Router (sir)
>>
>> Mike,
>>
>> I’m not sure I understand what you mean by “suboptimal“ routing. Even
>> though the Internet uses AS path length for routing, many of those path
>> lengths are bogus, and don’t really represent any kind of path performance
>> value. For example, a single AS might hide many hops in an MPLS network as
>> a single hop, obscuring asymmetric routing and other uglies. Prepending
>> also occurs when destinations are trying to enforce their own engineering
>> policies, which often conflict with yours or mine.
>>
>> So what do you mean by “suboptimal“? Are you thinking that the “best”
>> path in BGP tables actually meant you were getting a performance benefit?
>> Because that’s definitely not the case in today’s Internet. Were were you
>> thinking that you would be going along less congested paths? That’s really
>> at the mercy of the traffic engineering of backbone providers over which we
>> have no control.
>>
>> I generally populate local router FIBs to merel choose an exit point for
>> purposes of load balancing, and nothing more.
>>
>> -mel
>>
>> On Jan 5, 2023, at 12:38 PM, Mike Hammett <nanog@ics-il.net> wrote:
>>
>> ?
>> I guess I wasn't around for those days.
>>
>> As far as running out, again, assuming the tooling works correctly, I'd
>> think to target fewer routes than you could hold. Maybe 1k routes is all
>> one would need to get a significant percent of the traffic. A lot of room
>> to mess up if you can hold 100k, 500k routes.
>>
>>
>>
>> -----
>> Mike Hammett
>> Intelligent Computing Solutions <http://www.ics-il.com/>
>> <https://www.facebook.com/ICSIL>
>> <https://plus.google.com/+IntelligentComputingSolutionsDeKalb>
>> <https://www.linkedin.com/company/intelligent-computing-solutions>
>> <https://twitter.com/ICSIL>
>> Midwest Internet Exchange <http://www.midwest-ix.com/>
>> <https://www.facebook.com/mdwestix>
>> <https://www.linkedin.com/company/midwest-internet-exchange>
>> <https://twitter.com/mdwestix>
>> The Brothers WISP <http://www.thebrotherswisp.com/>
>> <https://www.facebook.com/thebrotherswisp>
>> <https://www.youtube.com/channel/UCXSdfxQv7SpoRQYNyLwntZg>
>> ------------------------------
>> *From: *"Joe Maimon" <jmaimon@jmaimon.com>
>> *To: *"Mike Hammett" <nanog@ics-il.net>, "Christopher Morrow" <
>> morrowc.lists@gmail.com>
>> *Cc: *"NANOG" <nanog@nanog.org>
>> *Sent: *Thursday, January 5, 2023 2:30:40 PM
>> *Subject: *Re: SDN Internet Router (sir)
>>
>>
>>
>> Mike Hammett wrote:
>> > I'm not concerned with which technology or buzzword gets the job done,
>> > only that the job is done.
>> >
>> >
>> >
>> > Looking briefly at the couple of things out there, they're evaluating
>> > the top X prefixes in terms of traffic reported by s-flow, where X is
>> > the number I define, and those get pushed into the FIB. One
>> > recalculates every hour, one does so more quickly. How much is
>> > appropriate? I'm not sure. I can't imagine it would *NEED* to be done
>> > all of that often, given the traffic/prefix density an eyeball network
>> > will have. Default routes carry the rest. Default routes could be
>> > handled outside of this process, such that if this process fails, you
>> > just get some sub-optimal routing until repaired. Maybe it doesn't
>> > filter properly and sends a bunch of routes. Then just have a prefix
>> > limit set on the box. Maybe it sends the wrong prefixes. No harm, no
>> > foul. If you're routing sub-optimally internally, when it does hit a
>> > real router with a full FIB, it gets handled appropriately.
>>
>> Unless it loops.
>>
>> The rest sounds nice. But flow caching got a bad rap back in the early
>> worm days. But thats because the situation was a little worse back then.
>> Cache the wrong routes or run out of cache, router dies. So long as
>> thats not the case automating optimization is an extremely valuable goal.
>>
>> >
>> >
>> > I would just be looking for solutions that influence what's in the FIB
>> > and let the rest of the router work as the rest of the router would.
>>
>> The problem comes when the router wont work at all without the FIB
>> routes, like in the olden days.
>> >
>> >
>> >
>> > -----
>> > Mike Hammett
>> > Intelligent Computing Solutions <http://www.ics-il.com/>
>> > <https://www.facebook.com/ICSIL><
>> https://plus.google.com/+IntelligentComputingSolutionsDeKalb><
>> https://www.linkedin.com/company/intelligent-computing-solutions><
>> https://twitter.com/ICSIL>
>> > Midwest Internet Exchange <http://www.midwest-ix.com/>
>> > <https://www.facebook.com/mdwestix><
>> https://www.linkedin.com/company/midwest-internet-exchange><
>> https://twitter.com/mdwestix>
>> > The Brothers WISP <http://www.thebrotherswisp.com/>
>> > <https://www.facebook.com/thebrotherswisp><
>> https://www.youtube.com/channel/UCXSdfxQv7SpoRQYNyLwntZg>
>> > ------------------------------------------------------------------------
>> > *From: *"Christopher Morrow" <morrowc.lists@gmail.com>
>> > *To: *"Mike Hammett" <nanog@ics-il.net>
>> > *Cc: *"Tom Beecher" <beecher@beecher.cc>, "NANOG" <nanog@nanog.org>
>> > *Sent: *Thursday, January 5, 2023 12:27:08 PM
>> > *Subject: *Re: SDN Internet Router (sir)
>> >
>> >
>> >
>> > On Thu, Jan 5, 2023 at 11:18 AM Mike Hammett <nanog@ics-il.net
>> > <mailto:nanog@ics-il.net>> wrote:
>> >
>> > Initially, my thought was to use community filtering to push just
>> > IXes, customers, and defaults throughout the network, but that's
>> > obviously still sub-optimal.
>> >
>> > I'd be surprised if a last mile network had a ton of traffic going
>> > to any more than a few hundred prefixes.
>> >
>> >
>> > I think in a low-fib box at the edge of your network your choices are:
>> > "the easy choice, get default, follow that"
>> >
>> > "send some limited set of prefixes to the device, and default, so
>> > you MAY choose better for the initial hop away"
>> >
>> > you certainly can do the second with communities, or route-filters
>> > (prefix-list) on the senders, or....
>> > you can choose what prefixes make the cut (get the community(ies))
>> > based on traffic volumes or expected destination locality:
>> > "do not go east to go west!"
>> >
>> > these things will introduce toil and SOME suboptimal routing in some
>> > instances... perhaps it's better than per flow choosing left/right
>> > though and the support calls related to that choice.
>> >
>> > In your NOLA / DFW / ATL example it's totally possible that the
>> > networks in question do something like:
>> > "low fib box in tier-2 city (NOLA), dfz capable/core devices in
>> > tier-1 city (DFW/ATL), and send default from left/right to NOLA"
>> >
>> > Could they send more prefixes than default? sure... do they want to
>> > deal with the toil that induces? (probably not says your example).
>> >
>> > SDN isn't really an answer to this, though.. I don't think. Unless you
>> > envision that to lower the toil ?
>> >
>>
>>
>>
>>
>
Re: SDN Internet Router (sir) [ In reply to ]
Right.


Only I'm not the guy to build that solution.


What I originally linked to (and another link or two contributed since then) seem to be people that already built that solution.




-----
Mike Hammett
Intelligent Computing Solutions

Midwest Internet Exchange

The Brothers WISP

----- Original Message -----

From: "Tom Beecher" <beecher@beecher.cc>
To: "Mike Hammett" <nanog@ics-il.net>
Cc: "Mel Beckman" <mel@beckman.org>, "NANOG" <nanog@nanog.org>
Sent: Friday, January 6, 2023 9:51:43 AM
Subject: Re: SDN Internet Router (sir)


Gotcha.


Setup a Quagga/Bird box. Do your top talker analysis , use that box to inject the routes you deem important with communities. On your routers , create policy structure to only take a default plus those communities.


Obviously lots of devils in the details of the implementation , but something like that is all you need to do.



On Fri, Jan 6, 2023 at 10:29 Mike Hammett < nanog@ics-il.net > wrote:




Maybe?

I don't need any additional performance tests, though. Just watching which prefixes are the top talkers and leaving the rest to default.


I'm not looking at this to do what a BGP optimizier would do and find the best tested path to the top talkers and then massage BGP to get it routed that way. Determine the top talkers, then let BGP do its thing for those top talkers.


I don't want to manually say X traffic from Y POP manually goes here, but I don't want to just leave it to default routing either. Something in the middle.




-----
Mike Hammett
Intelligent Computing Solutions

Midwest Internet Exchange

The Brothers WISP



From: "Tom Beecher" < beecher@beecher.cc >
To: "Mike Hammett" < nanog@ics-il.net >
Cc: "Mel Beckman" < mel@beckman.org >, "NANOG" < nanog@nanog.org >
Sent: Friday, January 6, 2023 9:16:19 AM




Subject: Re: SDN Internet Router (sir)


Thanks for this example.


It sounds like you are describing egress peer engineering, but kinda in reverse.


In 'traditional' EPE, the routers have all the routes, and you are using the external controller to perform the performance tests that matter to you, and signal the network where to take the traffic based on those tests.


It seems like you want to do the same thing , but instead of having the controller signal the network where to carry bits, you want the controller to signal the networks what routes are present, and direct the bits that way.


Do I have this right?


On Thu, Jan 5, 2023 at 4:12 PM Mike Hammett < nanog@ics-il.net > wrote:

<blockquote>




I hesitated to get too specific in examples because someone is going to drag the conversation into the weeds.

Let's take the the Dallas - New Orleans - Atlanta example where I have a connection from New Orleans to Dallas and a connection from New Orleans to Atlanta.


Let's say I peer with Netflix in both markets. Netflix chooses to serve me out of Atlanta, for whatever reason. Say my default route sends my traffic to Dallas. That's not where Netflix wanted it, so now I have to go from Dallas to Atlanta, whether that's my circuit or across the public Internet. Potentially, it's on MPLS and it rides back through the New Orleans router to get back to Atlanta. That's a long trip when I already had a better path, the less-than-full-fib router just didn't know about it. Given that Netflix is a sizable amount of traffic in an eyeball ISP, that's a lot of traffic to be going the wrong way. If the website for Viktor's Arctic Plunge in Siberia was hosted in Atlanta, I wouldn't give two craps that the traffic went the wrong way because A), I'll probably never go there and B) when someone does, it won't be meaningfully enough traffic to accommodate.


Someone's going to tell me to put a full-table router in New Orleans. Maybe I should. Okay, so maybe I have a POP in Ashford, Alabama. It has transport to New Orleans and Atlanta. There aren't enough grains of sugar in Ashford, Alabama to justify a current-generation, full table router. Now I'm even closer to Atlanta, but default may point to New Orleans.




-----
Mike Hammett
Intelligent Computing Solutions

Midwest Internet Exchange

The Brothers WISP



From: "Mel Beckman" < mel@beckman.org >
To: "Mike Hammett" < nanog@ics-il.net >
Cc: "Joe Maimon" < jmaimon@jmaimon.com >, "NANOG" < nanog@nanog.org >
Sent: Thursday, January 5, 2023 2:54:27 PM
Subject: Re: SDN Internet Router (sir)


Mike,


I’m not sure I understand what you mean by “suboptimal“ routing. Even though the Internet uses AS path length for routing, many of those path lengths are bogus, and don’t really represent any kind of path performance value. For example, a single AS might hide many hops in an MPLS network as a single hop, obscuring asymmetric routing and other uglies. Prepending also occurs when destinations are trying to enforce their own engineering policies, which often conflict with yours or mine.


So what do you mean by “suboptimal“? Are you thinking that the “best” path in BGP tables actually meant you were getting a performance benefit? Because that’s definitely not the case in today’s Internet. Were were you thinking that you would be going along less congested paths? That’s really at the mercy of the traffic engineering of backbone providers over which we have no control.


I generally populate local router FIBs to merel choose an exit point for purposes of load balancing, and nothing more.


-mel



<blockquote>
On Jan 5, 2023, at 12:38 PM, Mike Hammett < nanog@ics-il.net > wrote:




<blockquote>


I guess I wasn't around for those days.


As far as running out, again, assuming the tooling works correctly, I'd think to target fewer routes than you could hold. Maybe 1k routes is all one would need to get a significant percent of the traffic. A lot of room to mess up if you can hold 100k, 500k routes.





-----
Mike Hammett
Intelligent Computing Solutions

Midwest Internet Exchange

The Brothers WISP



From: "Joe Maimon" < jmaimon@jmaimon.com >
To: "Mike Hammett" < nanog@ics-il.net >, "Christopher Morrow" < morrowc.lists@gmail.com >
Cc: "NANOG" < nanog@nanog.org >
Sent: Thursday, January 5, 2023 2:30:40 PM
Subject: Re: SDN Internet Router (sir)



Mike Hammett wrote:
> I'm not concerned with which technology or buzzword gets the job done,
> only that the job is done.
>
>
>
> Looking briefly at the couple of things out there, they're evaluating
> the top X prefixes in terms of traffic reported by s-flow, where X is
> the number I define, and those get pushed into the FIB. One
> recalculates every hour, one does so more quickly. How much is
> appropriate? I'm not sure. I can't imagine it would *NEED* to be done
> all of that often, given the traffic/prefix density an eyeball network
> will have. Default routes carry the rest. Default routes could be
> handled outside of this process, such that if this process fails, you
> just get some sub-optimal routing until repaired. Maybe it doesn't
> filter properly and sends a bunch of routes. Then just have a prefix
> limit set on the box. Maybe it sends the wrong prefixes. No harm, no
> foul. If you're routing sub-optimally internally, when it does hit a
> real router with a full FIB, it gets handled appropriately.

Unless it loops.

The rest sounds nice. But flow caching got a bad rap back in the early
worm days. But thats because the situation was a little worse back then.
Cache the wrong routes or run out of cache, router dies. So long as
thats not the case automating optimization is an extremely valuable goal.

>
>
> I would just be looking for solutions that influence what's in the FIB
> and let the rest of the router work as the rest of the router would.

The problem comes when the router wont work at all without the FIB
routes, like in the olden days.
>
>
>
> -----
> Mike Hammett
> Intelligent Computing Solutions < http://www.ics-il.com/ >
> < https://www.facebook.com/ICSIL >< https://plus.google.com/+IntelligentComputingSolutionsDeKalb >< https://www.linkedin.com/company/intelligent-computing-solutions >< https://twitter.com/ICSIL >
> Midwest Internet Exchange < http://www.midwest-ix.com/ >
> < https://www.facebook.com/mdwestix >< https://www.linkedin.com/company/midwest-internet-exchange >< https://twitter.com/mdwestix >
> The Brothers WISP < http://www.thebrotherswisp.com/ >
> < https://www.facebook.com/thebrotherswisp >< https://www.youtube.com/channel/UCXSdfxQv7SpoRQYNyLwntZg >
> ------------------------------------------------------------------------
> *From: *"Christopher Morrow" < morrowc.lists@gmail.com >
> *To: *"Mike Hammett" < nanog@ics-il.net >
> *Cc: *"Tom Beecher" < beecher@beecher.cc >, "NANOG" < nanog@nanog.org >
> *Sent: *Thursday, January 5, 2023 12:27:08 PM
> *Subject: *Re: SDN Internet Router (sir)
>
>
>
> On Thu, Jan 5, 2023 at 11:18 AM Mike Hammett < nanog@ics-il.net
> <mailto: nanog@ics-il.net >> wrote:
>
> Initially, my thought was to use community filtering to push just
> IXes, customers, and defaults throughout the network, but that's
> obviously still sub-optimal.
>
> I'd be surprised if a last mile network had a ton of traffic going
> to any more than a few hundred prefixes.
>
>
> I think in a low-fib box at the edge of your network your choices are:
> "the easy choice, get default, follow that"
>
> "send some limited set of prefixes to the device, and default, so
> you MAY choose better for the initial hop away"
>
> you certainly can do the second with communities, or route-filters
> (prefix-list) on the senders, or....
> you can choose what prefixes make the cut (get the community(ies))
> based on traffic volumes or expected destination locality:
> "do not go east to go west!"
>
> these things will introduce toil and SOME suboptimal routing in some
> instances... perhaps it's better than per flow choosing left/right
> though and the support calls related to that choice.
>
> In your NOLA / DFW / ATL example it's totally possible that the
> networks in question do something like:
> "low fib box in tier-2 city (NOLA), dfz capable/core devices in
> tier-1 city (DFW/ATL), and send default from left/right to NOLA"
>
> Could they send more prefixes than default? sure... do they want to
> deal with the toil that induces? (probably not says your example).
>
> SDN isn't really an answer to this, though.. I don't think. Unless you
> envision that to lower the toil ?
>



</blockquote>


</blockquote>


</blockquote>
Re: SDN Internet Router (sir) [ In reply to ]
On Fri, 6 Jan 2023, 18:38 Mike Hammett, <nanog@ics-il.net> wrote:

> I suspect it always will have value, whether it's peering routers, POP
> routers, multi-homed customer routers, etc.
>

Indeed. It's not "clean" but it is an acceptable tradeoff if you know what
you're doing, and how traffic sloshes around etc.

I wrote a tool once that took a number of BGP feeds and aggregated the
prefixes based on the next-hop values, which was *amazingly* good at
reducing FIB sizes, but consumed so much CPU and memory, not to mention the
latency of updates during any sizeable churn event, that it proved less
useful than just precomputing based on historical traffic flows and
updating the lists semi-frequently.

The idea of Juniper's EPE etc is very attractive, and largely matches what
I had done back then, but does it with a lot more finesse. Ultimately, it's
a tradeoff between CapEx of the high FIB router and the OpEx of the
engineers who have to maintain the often hacky solution ;)


M
Re: SDN Internet Router (sir) [ In reply to ]
>
> What I originally linked to (and another link or two contributed since
> then) seem to be people that already built that solution.


What's been shared isn't the same as what I described. faucet / sir are
doing the decision part and the signaling part together.

I am saying separate them. BGP doesn't know how to make the *decision* you
want, but it does know how to process *signals* that you send it.


On Fri, Jan 6, 2023 at 10:58 AM Mike Hammett <nanog@ics-il.net> wrote:

> Right.
>
> Only I'm not the guy to build that solution.
>
> What I originally linked to (and another link or two contributed since
> then) seem to be people that already built that solution.
>
>
>
> -----
> Mike Hammett
> Intelligent Computing Solutions <http://www.ics-il.com/>
> <https://www.facebook.com/ICSIL>
> <https://plus.google.com/+IntelligentComputingSolutionsDeKalb>
> <https://www.linkedin.com/company/intelligent-computing-solutions>
> <https://twitter.com/ICSIL>
> Midwest Internet Exchange <http://www.midwest-ix.com/>
> <https://www.facebook.com/mdwestix>
> <https://www.linkedin.com/company/midwest-internet-exchange>
> <https://twitter.com/mdwestix>
> The Brothers WISP <http://www.thebrotherswisp.com/>
> <https://www.facebook.com/thebrotherswisp>
> <https://www.youtube.com/channel/UCXSdfxQv7SpoRQYNyLwntZg>
> ------------------------------
> *From: *"Tom Beecher" <beecher@beecher.cc>
> *To: *"Mike Hammett" <nanog@ics-il.net>
> *Cc: *"Mel Beckman" <mel@beckman.org>, "NANOG" <nanog@nanog.org>
> *Sent: *Friday, January 6, 2023 9:51:43 AM
> *Subject: *Re: SDN Internet Router (sir)
>
> Gotcha.
>
> Setup a Quagga/Bird box. Do your top talker analysis , use that box to
> inject the routes you deem important with communities. On your routers ,
> create policy structure to only take a default plus those communities.
>
> Obviously lots of devils in the details of the implementation , but
> something like that is all you need to do.
>
> On Fri, Jan 6, 2023 at 10:29 Mike Hammett <nanog@ics-il.net> wrote:
>
>> Maybe?
>>
>> I don't need any additional performance tests, though. Just watching
>> which prefixes are the top talkers and leaving the rest to default.
>>
>> I'm not looking at this to do what a BGP optimizier would do and find the
>> best tested path to the top talkers and then massage BGP to get it routed
>> that way. Determine the top talkers, then let BGP do its thing for those
>> top talkers.
>>
>> I don't want to manually say X traffic from Y POP manually goes here, but
>> I don't want to just leave it to default routing either. Something in the
>> middle.
>>
>>
>>
>> -----
>> Mike Hammett
>> Intelligent Computing Solutions <http://www.ics-il.com/>
>> <https://www.facebook.com/ICSIL>
>> <https://plus.google.com/+IntelligentComputingSolutionsDeKalb>
>> <https://www.linkedin.com/company/intelligent-computing-solutions>
>> <https://twitter.com/ICSIL>
>> Midwest Internet Exchange <http://www.midwest-ix.com/>
>> <https://www.facebook.com/mdwestix>
>> <https://www.linkedin.com/company/midwest-internet-exchange>
>> <https://twitter.com/mdwestix>
>> The Brothers WISP <http://www.thebrotherswisp.com/>
>> <https://www.facebook.com/thebrotherswisp>
>> <https://www.youtube.com/channel/UCXSdfxQv7SpoRQYNyLwntZg>
>> ------------------------------
>> *From: *"Tom Beecher" <beecher@beecher.cc>
>> *To: *"Mike Hammett" <nanog@ics-il.net>
>> *Cc: *"Mel Beckman" <mel@beckman.org>, "NANOG" <nanog@nanog.org>
>> *Sent: *Friday, January 6, 2023 9:16:19 AM
>>
>> *Subject: *Re: SDN Internet Router (sir)
>>
>> Thanks for this example.
>>
>> It sounds like you are describing egress peer engineering, but kinda in
>> reverse.
>>
>> In 'traditional' EPE, the routers have all the routes, and you are using
>> the external controller to perform the performance tests that matter to
>> you, and signal the network where to take the traffic based on those tests.
>>
>> It seems like you want to do the same thing , but instead of having the
>> controller signal the network where to carry bits, you want the controller
>> to signal the networks what routes are present, and direct the bits that
>> way.
>>
>> Do I have this right?
>>
>> On Thu, Jan 5, 2023 at 4:12 PM Mike Hammett <nanog@ics-il.net> wrote:
>>
>>> I hesitated to get too specific in examples because someone is going to
>>> drag the conversation into the weeds.
>>>
>>> Let's take the the Dallas - New Orleans - Atlanta example where I have a
>>> connection from New Orleans to Dallas and a connection from New Orleans to
>>> Atlanta.
>>>
>>> Let's say I peer with Netflix in both markets. Netflix chooses to serve
>>> me out of Atlanta, for whatever reason. Say my default route sends my
>>> traffic to Dallas. That's not where Netflix wanted it, so now I have to go
>>> from Dallas to Atlanta, whether that's my circuit or across the public
>>> Internet. Potentially, it's on MPLS and it rides back through the New
>>> Orleans router to get back to Atlanta. That's a long trip when I already
>>> had a better path, the less-than-full-fib router just didn't know about it.
>>> Given that Netflix is a sizable amount of traffic in an eyeball ISP, that's
>>> a lot of traffic to be going the wrong way. If the website for Viktor's
>>> Arctic Plunge in Siberia was hosted in Atlanta, I wouldn't give two craps
>>> that the traffic went the wrong way because A), I'll probably never go
>>> there and B) when someone does, it won't be meaningfully enough traffic to
>>> accommodate.
>>>
>>> Someone's going to tell me to put a full-table router in New Orleans.
>>> Maybe I should. Okay, so maybe I have a POP in Ashford, Alabama. It has
>>> transport to New Orleans and Atlanta. There aren't enough grains of sugar
>>> in Ashford, Alabama to justify a current-generation, full table router. Now
>>> I'm even closer to Atlanta, but default may point to New Orleans.
>>>
>>>
>>>
>>> -----
>>> Mike Hammett
>>> Intelligent Computing Solutions <http://www.ics-il.com/>
>>> <https://www.facebook.com/ICSIL>
>>> <https://plus.google.com/+IntelligentComputingSolutionsDeKalb>
>>> <https://www.linkedin.com/company/intelligent-computing-solutions>
>>> <https://twitter.com/ICSIL>
>>> Midwest Internet Exchange <http://www.midwest-ix.com/>
>>> <https://www.facebook.com/mdwestix>
>>> <https://www.linkedin.com/company/midwest-internet-exchange>
>>> <https://twitter.com/mdwestix>
>>> The Brothers WISP <http://www.thebrotherswisp.com/>
>>> <https://www.facebook.com/thebrotherswisp>
>>> <https://www.youtube.com/channel/UCXSdfxQv7SpoRQYNyLwntZg>
>>> ------------------------------
>>> *From: *"Mel Beckman" <mel@beckman.org>
>>> *To: *"Mike Hammett" <nanog@ics-il.net>
>>> *Cc: *"Joe Maimon" <jmaimon@jmaimon.com>, "NANOG" <nanog@nanog.org>
>>> *Sent: *Thursday, January 5, 2023 2:54:27 PM
>>> *Subject: *Re: SDN Internet Router (sir)
>>>
>>> Mike,
>>>
>>> I’m not sure I understand what you mean by “suboptimal“ routing. Even
>>> though the Internet uses AS path length for routing, many of those path
>>> lengths are bogus, and don’t really represent any kind of path performance
>>> value. For example, a single AS might hide many hops in an MPLS network as
>>> a single hop, obscuring asymmetric routing and other uglies. Prepending
>>> also occurs when destinations are trying to enforce their own engineering
>>> policies, which often conflict with yours or mine.
>>>
>>> So what do you mean by “suboptimal“? Are you thinking that the “best”
>>> path in BGP tables actually meant you were getting a performance benefit?
>>> Because that’s definitely not the case in today’s Internet. Were were you
>>> thinking that you would be going along less congested paths? That’s really
>>> at the mercy of the traffic engineering of backbone providers over which we
>>> have no control.
>>>
>>> I generally populate local router FIBs to merel choose an exit point for
>>> purposes of load balancing, and nothing more.
>>>
>>> -mel
>>>
>>> On Jan 5, 2023, at 12:38 PM, Mike Hammett <nanog@ics-il.net> wrote:
>>>
>>> ?
>>> I guess I wasn't around for those days.
>>>
>>> As far as running out, again, assuming the tooling works correctly, I'd
>>> think to target fewer routes than you could hold. Maybe 1k routes is all
>>> one would need to get a significant percent of the traffic. A lot of room
>>> to mess up if you can hold 100k, 500k routes.
>>>
>>>
>>>
>>> -----
>>> Mike Hammett
>>> Intelligent Computing Solutions <http://www.ics-il.com/>
>>> <https://www.facebook.com/ICSIL>
>>> <https://plus.google.com/+IntelligentComputingSolutionsDeKalb>
>>> <https://www.linkedin.com/company/intelligent-computing-solutions>
>>> <https://twitter.com/ICSIL>
>>> Midwest Internet Exchange <http://www.midwest-ix.com/>
>>> <https://www.facebook.com/mdwestix>
>>> <https://www.linkedin.com/company/midwest-internet-exchange>
>>> <https://twitter.com/mdwestix>
>>> The Brothers WISP <http://www.thebrotherswisp.com/>
>>> <https://www.facebook.com/thebrotherswisp>
>>> <https://www.youtube.com/channel/UCXSdfxQv7SpoRQYNyLwntZg>
>>> ------------------------------
>>> *From: *"Joe Maimon" <jmaimon@jmaimon.com>
>>> *To: *"Mike Hammett" <nanog@ics-il.net>, "Christopher Morrow" <
>>> morrowc.lists@gmail.com>
>>> *Cc: *"NANOG" <nanog@nanog.org>
>>> *Sent: *Thursday, January 5, 2023 2:30:40 PM
>>> *Subject: *Re: SDN Internet Router (sir)
>>>
>>>
>>>
>>> Mike Hammett wrote:
>>> > I'm not concerned with which technology or buzzword gets the job done,
>>> > only that the job is done.
>>> >
>>> >
>>> >
>>> > Looking briefly at the couple of things out there, they're evaluating
>>> > the top X prefixes in terms of traffic reported by s-flow, where X is
>>> > the number I define, and those get pushed into the FIB. One
>>> > recalculates every hour, one does so more quickly. How much is
>>> > appropriate? I'm not sure. I can't imagine it would *NEED* to be done
>>> > all of that often, given the traffic/prefix density an eyeball network
>>> > will have. Default routes carry the rest. Default routes could be
>>> > handled outside of this process, such that if this process fails, you
>>> > just get some sub-optimal routing until repaired. Maybe it doesn't
>>> > filter properly and sends a bunch of routes. Then just have a prefix
>>> > limit set on the box. Maybe it sends the wrong prefixes. No harm, no
>>> > foul. If you're routing sub-optimally internally, when it does hit a
>>> > real router with a full FIB, it gets handled appropriately.
>>>
>>> Unless it loops.
>>>
>>> The rest sounds nice. But flow caching got a bad rap back in the early
>>> worm days. But thats because the situation was a little worse back then.
>>> Cache the wrong routes or run out of cache, router dies. So long as
>>> thats not the case automating optimization is an extremely valuable goal.
>>>
>>> >
>>> >
>>> > I would just be looking for solutions that influence what's in the FIB
>>> > and let the rest of the router work as the rest of the router would.
>>>
>>> The problem comes when the router wont work at all without the FIB
>>> routes, like in the olden days.
>>> >
>>> >
>>> >
>>> > -----
>>> > Mike Hammett
>>> > Intelligent Computing Solutions <http://www.ics-il.com/>
>>> > <https://www.facebook.com/ICSIL><
>>> https://plus.google.com/+IntelligentComputingSolutionsDeKalb><
>>> https://www.linkedin.com/company/intelligent-computing-solutions><
>>> https://twitter.com/ICSIL>
>>> > Midwest Internet Exchange <http://www.midwest-ix.com/>
>>> > <https://www.facebook.com/mdwestix><
>>> https://www.linkedin.com/company/midwest-internet-exchange><
>>> https://twitter.com/mdwestix>
>>> > The Brothers WISP <http://www.thebrotherswisp.com/>
>>> > <https://www.facebook.com/thebrotherswisp><
>>> https://www.youtube.com/channel/UCXSdfxQv7SpoRQYNyLwntZg>
>>> >
>>> ------------------------------------------------------------------------
>>> > *From: *"Christopher Morrow" <morrowc.lists@gmail.com>
>>> > *To: *"Mike Hammett" <nanog@ics-il.net>
>>> > *Cc: *"Tom Beecher" <beecher@beecher.cc>, "NANOG" <nanog@nanog.org>
>>> > *Sent: *Thursday, January 5, 2023 12:27:08 PM
>>> > *Subject: *Re: SDN Internet Router (sir)
>>> >
>>> >
>>> >
>>> > On Thu, Jan 5, 2023 at 11:18 AM Mike Hammett <nanog@ics-il.net
>>> > <mailto:nanog@ics-il.net>> wrote:
>>> >
>>> > Initially, my thought was to use community filtering to push just
>>> > IXes, customers, and defaults throughout the network, but that's
>>> > obviously still sub-optimal.
>>> >
>>> > I'd be surprised if a last mile network had a ton of traffic going
>>> > to any more than a few hundred prefixes.
>>> >
>>> >
>>> > I think in a low-fib box at the edge of your network your choices are:
>>> > "the easy choice, get default, follow that"
>>> >
>>> > "send some limited set of prefixes to the device, and default, so
>>> > you MAY choose better for the initial hop away"
>>> >
>>> > you certainly can do the second with communities, or route-filters
>>> > (prefix-list) on the senders, or....
>>> > you can choose what prefixes make the cut (get the community(ies))
>>> > based on traffic volumes or expected destination locality:
>>> > "do not go east to go west!"
>>> >
>>> > these things will introduce toil and SOME suboptimal routing in some
>>> > instances... perhaps it's better than per flow choosing left/right
>>> > though and the support calls related to that choice.
>>> >
>>> > In your NOLA / DFW / ATL example it's totally possible that the
>>> > networks in question do something like:
>>> > "low fib box in tier-2 city (NOLA), dfz capable/core devices in
>>> > tier-1 city (DFW/ATL), and send default from left/right to NOLA"
>>> >
>>> > Could they send more prefixes than default? sure... do they want to
>>> > deal with the toil that induces? (probably not says your example).
>>> >
>>> > SDN isn't really an answer to this, though.. I don't think. Unless you
>>> > envision that to lower the toil ?
>>> >
>>>
>>>
>>>
>>>
>>
>
Re: SDN Internet Router (sir) [ In reply to ]
Matthew Walster wrote:

> SDN does not imply QoS routing,

As long as the shortest path is comfortable enough, no, it
does not have to.

> it's just one aspect of it. Some use it for
> classifying guest traffic etc.

If special path is provided for guest or otherwise
prioritized traffic, that's QoS routing.

Anyway, prioritization needs more, not less,
routing table entries.

Masataka Ohta
Re: SDN Internet Router (sir) [ In reply to ]
On Sat, 7 Jan 2023, 03:17 Masataka Ohta, <mohta@necom830.hpcl.titech.ac.jp>
wrote:

> Matthew Walster wrote:
>
> > it's just one aspect of it. Some use it for
> > classifying guest traffic etc.
>
> If special path is provided for guest or otherwise
> prioritized traffic, that's QoS routing.
>

No... It's action based. You can send it a different route, you can
replicate it, you can drop it, you can mutate it... You can send it to a
different destination for stateful filtering when it doesn't match an
expected pattern!

SDN is not just QoS routing, please stop saying that.

Anyway, prioritization needs more, not less,
> routing table entries.
>

Nope, not true. Had 1000 routes, only 100 available in FIB. So you filter
to the top 50 doing traffic and default route the rest of the traffic. Less
entries.

M

>
Re: SDN Internet Router (sir) [ In reply to ]
Freertr folks have just published (I didn’t look into the details of their implementation though):
“rare/freertr just got fib compression.. in our nren, the v4 table can be compressed from 900k to 260k, the v6 table from 160k to 52k... the tofino2 asic with our dataplane code ( https://lnkd.in/dJrHVZqE ) can accomodate 520k v4 and 130k v6”

On a none related note - they are also implementing RIFT, we plan testing against Junos and Python OpenSource implementations during IETF 116 hackathon.

Cheers,
Jeff
On Jan 6, 2023, at 17:13, Matthew Walster via NANOG <nanog@nanog.org> wrote:

?

On Fri, 6 Jan 2023, 18:38 Mike Hammett, <nanog@ics-il.net> wrote:
I suspect it always will have value, whether it's peering routers, POP routers, multi-homed customer routers, etc.

Indeed. It's not "clean" but it is an acceptable tradeoff if you know what you're doing, and how traffic sloshes around etc.
I wrote a tool once that took a number of BGP feeds and aggregated the prefixes based on the next-hop values, which was *amazingly* good at reducing FIB sizes, but consumed so much CPU and memory, not to mention the latency of updates during any sizeable churn event, that it proved less useful than just precomputing based on historical traffic flows and updating the lists semi-frequently.
The idea of Juniper's EPE etc is very attractive, and largely matches what I had done back then, but does it with a lot more finesse. Ultimately, it's a tradeoff between CapEx of the high FIB router and the OpEx of the engineers who have to maintain the often hacky solution ;)

M
Re: SDN Internet Router (sir) [ In reply to ]
You might want to search for “policy based add-path”, same idea (BGP listener + flow collector), different issue (60M+ entries BGP RIB), all clouds use some version of that, not sure about open sourcing it though
Cheers,
Jeff
On Jan 6, 2023, at 17:00, Mike Hammett <nanog@ics-il.net> wrote:

?Right.
Only I'm not the guy to build that solution.
What I originally linked to (and another link or two contributed since then) seem to be people that already built that solution.



-----
Mike Hammett
http://www.ics-il.com/"]Intelligent Computing Solutions
https://www.facebook.com/ICSIL"]https://plus.google.com/+IntelligentComputingSolutionsDeKalb"]https://www.linkedin.com/company/intelligent-computing-solutions"]https://twitter.com/ICSIL"]
http://www.midwest-ix.com/"]Midwest Internet Exchange
https://www.facebook.com/mdwestix"]https://www.linkedin.com/company/midwest-internet-exchange"]https://twitter.com/mdwestix"]
http://www.thebrotherswisp.com/"]The Brothers WISP
https://www.facebook.com/thebrotherswisp"]https://www.youtube.com/channel/UCXSdfxQv7SpoRQYNyLwntZg"]

From: "Tom Beecher" <beecher@beecher.cc>
To: "Mike Hammett" <nanog@ics-il.net>
Cc: "Mel Beckman" <mel@beckman.org>, "NANOG" <nanog@nanog.org>
Sent: Friday, January 6, 2023 9:51:43 AM
Subject: Re: SDN Internet Router (sir)

Gotcha.
Setup a Quagga/Bird box. Do your top talker analysis , use that box to inject the routes you deem important with communities. On your routers , create policy structure to only take a default plus those communities.
Obviously lots of devils in the details of the implementation , but something like that is all you need to do.
On Fri, Jan 6, 2023 at 10:29 Mike Hammett <nanog@ics-il.net> wrote:
Maybe?

I don't need any additional performance tests, though. Just watching which prefixes are the top talkers and leaving the rest to default.
I'm not looking at this to do what a BGP optimizier would do and find the best tested path to the top talkers and then massage BGP to get it routed that way. Determine the top talkers, then let BGP do its thing for those top talkers.
I don't want to manually say X traffic from Y POP manually goes here, but I don't want to just leave it to default routing either. Something in the middle.



-----
Mike Hammett
http://www.ics-il.com/"]Intelligent Computing Solutions
https://www.facebook.com/ICSIL"]https://plus.google.com/+IntelligentComputingSolutionsDeKalb"]https://www.linkedin.com/company/intelligent-computing-solutions"]https://twitter.com/ICSIL"]
http://www.midwest-ix.com/"]Midwest Internet Exchange
https://www.facebook.com/mdwestix"]https://www.linkedin.com/company/midwest-internet-exchange"]https://twitter.com/mdwestix"]
http://www.thebrotherswisp.com/"]The Brothers WISP
https://www.facebook.com/thebrotherswisp"]https://www.youtube.com/channel/UCXSdfxQv7SpoRQYNyLwntZg"]

From: "Tom Beecher" <beecher@beecher.cc>
To: "Mike Hammett" <nanog@ics-il.net>
Cc: "Mel Beckman" <mel@beckman.org>, "NANOG" <nanog@nanog.org>
Sent: Friday, January 6, 2023 9:16:19 AM
Subject: Re: SDN Internet Router (sir)

Thanks for this example.
It sounds like you are describing egress peer engineering, but kinda in reverse.
In 'traditional' EPE, the routers have all the routes, and you are using the external controller to perform the performance tests that matter to you, and signal the network where to take the traffic based on those tests.
It seems like you want to do the same thing , but instead of having the controller signal the network where to carry bits, you want the controller to signal the networks what routes are present, and direct the bits that way.
Do I have this right?
On Thu, Jan 5, 2023 at 4:12 PM Mike Hammett <nanog@ics-il.net> wrote:
I hesitated to get too specific in examples because someone is going to drag the conversation into the weeds.
Let's take the the Dallas - New Orleans - Atlanta example where I have a connection from New Orleans to Dallas and a connection from New Orleans to Atlanta.
Let's say I peer with Netflix in both markets. Netflix chooses to serve me out of Atlanta, for whatever reason. Say my default route sends my traffic to Dallas. That's not where Netflix wanted it, so now I have to go from Dallas to Atlanta, whether that's my circuit or across the public Internet. Potentially, it's on MPLS and it rides back through the New Orleans router to get back to Atlanta. That's a long trip when I already had a better path, the less-than-full-fib router just didn't know about it. Given that Netflix is a sizable amount of traffic in an eyeball ISP, that's a lot of traffic to be going the wrong way. If the website for Viktor's Arctic Plunge in Siberia was hosted in Atlanta, I wouldn't give two craps that the traffic went the wrong way because A), I'll probably never go there and B) when someone does, it won't be meaningfully enough traffic to accommodate.
Someone's going to tell me to put a full-table router in New Orleans. Maybe I should. Okay, so maybe I have a POP in Ashford, Alabama. It has transport to New Orleans and Atlanta. There aren't enough grains of sugar in Ashford, Alabama to justify a current-generation, full table router. Now I'm even closer to Atlanta, but default may point to New Orleans.



-----
Mike Hammett
http://www.ics-il.com/"]Intelligent Computing Solutions
https://www.facebook.com/ICSIL"]https://plus.google.com/+IntelligentComputingSolutionsDeKalb"]https://www.linkedin.com/company/intelligent-computing-solutions"]https://twitter.com/ICSIL"]
http://www.midwest-ix.com/"]Midwest Internet Exchange
https://www.facebook.com/mdwestix"]https://www.linkedin.com/company/midwest-internet-exchange"]https://twitter.com/mdwestix"]
http://www.thebrotherswisp.com/"]The Brothers WISP
https://www.facebook.com/thebrotherswisp"]https://www.youtube.com/channel/UCXSdfxQv7SpoRQYNyLwntZg"]

From: "Mel Beckman" <mel@beckman.org>
To: "Mike Hammett" <nanog@ics-il.net>
Cc: "Joe Maimon" <jmaimon@jmaimon.com>, "NANOG" <nanog@nanog.org>
Sent: Thursday, January 5, 2023 2:54:27 PM
Subject: Re: SDN Internet Router (sir)

Mike,
I’m not sure I understand what you mean by “suboptimal“ routing. Even though the Internet uses AS path length for routing, many of those path lengths are bogus, and don’t really represent any kind of path performance value. For example, a single AS might hide many hops in an MPLS network as a single hop, obscuring asymmetric routing and other uglies. Prepending also occurs when destinations are trying to enforce their own engineering policies, which often conflict with yours or mine.
So what do you mean by “suboptimal“? Are you thinking that the “best” path in BGP tables actually meant you were getting a performance benefit? Because that’s definitely not the case in today’s Internet. Were were you thinking that you would be going along less congested paths? That’s really at the mercy of the traffic engineering of backbone providers over which we have no control.
I generally populate local router FIBs to merel choose an exit point for purposes of load balancing, and nothing more.
-mel
On Jan 5, 2023, at 12:38 PM, Mike Hammett <nanog@ics-il.net> wrote:

? I guess I wasn't around for those days.
As far as running out, again, assuming the tooling works correctly, I'd think to target fewer routes than you could hold. Maybe 1k routes is all one would need to get a significant percent of the traffic. A lot of room to mess up if you can hold 100k, 500k routes.



-----
Mike Hammett
http://www.ics-il.com/"]Intelligent Computing Solutions
https://www.facebook.com/ICSIL"]https://plus.google.com/+IntelligentComputingSolutionsDeKalb"]https://www.linkedin.com/company/intelligent-computing-solutions"]https://twitter.com/ICSIL"]
http://www.midwest-ix.com/"]Midwest Internet Exchange
https://www.facebook.com/mdwestix"]https://www.linkedin.com/company/midwest-internet-exchange"]https://twitter.com/mdwestix"]
http://www.thebrotherswisp.com/"]The Brothers WISP
https://www.facebook.com/thebrotherswisp"]https://www.youtube.com/channel/UCXSdfxQv7SpoRQYNyLwntZg"]

From: "Joe Maimon" <jmaimon@jmaimon.com>
To: "Mike Hammett" <nanog@ics-il.net>, "Christopher Morrow" <morrowc.lists@gmail.com>
Cc: "NANOG" <nanog@nanog.org>
Sent: Thursday, January 5, 2023 2:30:40 PM
Subject: Re: SDN Internet Router (sir)



Mike Hammett wrote:
> I'm not concerned with which technology or buzzword gets the job done,
> only that the job is done.
>
>
>
> Looking briefly at the couple of things out there, they're evaluating
> the top X prefixes in terms of traffic reported by s-flow, where X is
> the number I define, and those get pushed into the FIB. One
> recalculates every hour, one does so more quickly. How much is
> appropriate? I'm not sure. I can't imagine it would *NEED* to be done
> all of that often, given the traffic/prefix density an eyeball network
> will have. Default routes carry the rest. Default routes could be
> handled outside of this process, such that if this process fails, you
> just get some sub-optimal routing until repaired. Maybe it doesn't
> filter properly and sends a bunch of routes. Then just have a prefix
> limit set on the box. Maybe it sends the wrong prefixes. No harm, no
> foul. If you're routing sub-optimally internally, when it does hit a
> real router with a full FIB, it gets handled appropriately.

Unless it loops.

The rest sounds nice. But flow caching got a bad rap back in the early
worm days. But thats because the situation was a little worse back then.
Cache the wrong routes or run out of cache, router dies. So long as
thats not the case automating optimization is an extremely valuable goal.

>
>
> I would just be looking for solutions that influence what's in the FIB
> and let the rest of the router work as the rest of the router would.

The problem comes when the router wont work at all without the FIB
routes, like in the olden days.
>
>
>
> -----
> Mike Hammett
> Intelligent Computing Solutions <http://www.ics-il.com/"]http://www.ics-il.com/>
> <https://www.facebook.com/ICSIL"]https://www.facebook.com/ICSIL><https://plus.google.com/+IntelligentComputingSolutionsDeKalb"]https://plus.google.com/+IntelligentComputingSolutionsDeKalb><https://www.linkedin.com/company/intelligent-computing-solutions"]https://www.linkedin.com/company/intelligent-computing-solutions><https://twitter.com/ICSIL"]https://twitter.com/ICSIL>
> Midwest Internet Exchange <http://www.midwest-ix.com/"]http://www.midwest-ix.com/>
> <https://www.facebook.com/mdwestix"]https://www.facebook.com/mdwestix><https://www.linkedin.com/company/midwest-internet-exchange"]https://www.linkedin.com/company/midwest-internet-exchange><https://twitter.com/mdwestix"]https://twitter.com/mdwestix>
> The Brothers WISP <http://www.thebrotherswisp.com/"]http://www.thebrotherswisp.com/>
> <https://www.facebook.com/thebrotherswisp"]https://www.facebook.com/thebrotherswisp><https://www.youtube.com/channel/UCXSdfxQv7SpoRQYNyLwntZg"]https://www.youtube.com/channel/UCXSdfxQv7SpoRQYNyLwntZg>
> ------------------------------------------------------------------------
> *From: *"Christopher Morrow" <morrowc.lists@gmail.com>
> *To: *"Mike Hammett" <nanog@ics-il.net>
> *Cc: *"Tom Beecher" <beecher@beecher.cc>, "NANOG" <nanog@nanog.org>
> *Sent: *Thursday, January 5, 2023 12:27:08 PM
> *Subject: *Re: SDN Internet Router (sir)
>
>
>
> On Thu, Jan 5, 2023 at 11:18 AM Mike Hammett <nanog@ics-il.net
> <mailto:nanog@ics-il.net>> wrote:
>
> Initially, my thought was to use community filtering to push just
> IXes, customers, and defaults throughout the network, but that's
> obviously still sub-optimal.
>
> I'd be surprised if a last mile network had a ton of traffic going
> to any more than a few hundred prefixes.
>
>
> I think in a low-fib box at the edge of your network your choices are:
> "the easy choice, get default, follow that"
>
> "send some limited set of prefixes to the device, and default, so
> you MAY choose better for the initial hop away"
>
> you certainly can do the second with communities, or route-filters
> (prefix-list) on the senders, or....
> you can choose what prefixes make the cut (get the community(ies))
> based on traffic volumes or expected destination locality:
> "do not go east to go west!"
>
> these things will introduce toil and SOME suboptimal routing in some
> instances... perhaps it's better than per flow choosing left/right
> though and the support calls related to that choice.
>
> In your NOLA / DFW / ATL example it's totally possible that the
> networks in question do something like:
> "low fib box in tier-2 city (NOLA), dfz capable/core devices in
> tier-1 city (DFW/ATL), and send default from left/right to NOLA"
>
> Could they send more prefixes than default? sure... do they want to
> deal with the toil that induces? (probably not says your example).
>
> SDN isn't really an answer to this, though.. I don't think. Unless you
> envision that to lower the toil ?
>





Re: SDN Internet Router (sir) [ In reply to ]
Matthew Walster wrote:

> No... It's action based. You can send it a different route, you can
> replicate it, you can drop it, you can mutate it...

Replication is a poor alternative for multicast.

For other actions, why, do you think, they are performed?

Just for fun? Or to differentiate treatment of some packets,
that is, prioritization?

> You can send it to a
> different destination for stateful filtering when it doesn't match an
> expected pattern!

Unless pattern is as simple as having certain port number,
stateful filtering almost always needs all packets including
those matching expected pattern, I'm afraid.

> SDN is not just QoS routing, please stop saying that.

See above.

> Nope, not true. Had 1000 routes, only 100 available in FIB. So you filter
> to the top 50 doing traffic and default route the rest of the traffic. Less
> entries.

If default route is acceptable, just rely on it along with
50 non default routes with plain IP routers.

Masataka Ohta
Re: SDN Internet Router (sir) [ In reply to ]
On Sat, 7 Jan 2023, 20:52 Masataka Ohta, <mohta@necom830.hpcl.titech.ac.jp>
wrote:

> Matthew Walster wrote:
>
> > No... It's action based. You can send it a different route, you can
> > replicate it, you can drop it, you can mutate it...
>
> Replication is a poor alternative for multicast.
>

You conveniently ignore things like IDS, port mirroring, things like that.

For other actions, why, do you think, they are performed?
>
> Just for fun? Or to differentiate treatment of some packets,
> that is, prioritization?
>

No. There are far more actions than for prioritisation.

What if you want to make sure certain classes of traffic do not flow over a
link, because it is unencrypted and/or sensitive, but you're happy to send
as much TLS wrapped data as you like?

What if you want to sample some flows in an ERSPAN like mechanism?

What if you want to urgently drop a set of flows based on a known DDOS
signature?

> You can send it to a
> > different destination for stateful filtering when it doesn't match an
> > expected pattern!
>
> Unless pattern is as simple as having certain port number,
> stateful filtering almost always needs all packets including
> those matching expected pattern, I'm afraid.
>

Or a certain set of IP addresses. Policy based routing.

> SDN is not just QoS routing, please stop saying that.
>
> See above.
>
> > Nope, not true. Had 1000 routes, only 100 available in FIB. So you filter
> > to the top 50 doing traffic and default route the rest of the traffic.
> Less
> > entries.
>
> If default route is acceptable, just rely on it along with
> 50 non default routes with plain IP routers.


That's what OP is suggesting. That's what SIR is. Classifying prefixes by
traffic and only keeping the ones with the highest volume of traffic,
discarding the rest, relying on the default route to infill.

M
Re: SDN Internet Router (sir) [ In reply to ]
Matthew Walster wrote:

>>> No... It's action based. You can send it a different route, you can
>>> replicate it, you can drop it, you can mutate it...
>>
>> Replication is a poor alternative for multicast.

> You conveniently ignore things like IDS, port mirroring, things like that.

Wrong. Instead, you conveniently ignore that such forwarding
requires a link between an SDN router and a monitoring device
have the same or larger MTU than an incoming link of the SDN
router, which means the router and the monitoring device must
be tightly coupled effectively to be a single device.

Sometimes, packet loss possibility between them often requires
they must actually be the same device.

> No. There are far more actions than for prioritisation.

Just for fun? I'm afraid I already mentioned so.

> What if you want to make sure certain classes of traffic do not flow over a
> link, because it is unencrypted and/or sensitive, but you're happy to send
> as much TLS wrapped data as you like?

You are wrongly assuming TLS wrapped packets can be identified
packet by packet, as I wrote:

>> Unless pattern is as simple as having certain port number,
>> stateful filtering almost always needs all packets including
>> those matching expected pattern, I'm afraid.

So?

> What if you want to sample some flows in an ERSPAN like mechanism?

See above for MTU issues.

> What if you want to urgently drop a set of flows based on a known DDOS
> signature?

Urgently? Even though a DDOS signature is known in advance?

Why?

>> Unless pattern is as simple as having certain port number,
>> stateful filtering almost always needs all packets including
>> those matching expected pattern, I'm afraid.
>>
>
> Or a certain set of IP addresses. Policy based routing.

That's even simpler than port number to be treated by
having or not having proper routing table entries.

>> If default route is acceptable, just rely on it along with
>> 50 non default routes with plain IP routers.

> That's what OP is suggesting.

With plain IP routers?

> That's what SIR is. Classifying prefixes by
> traffic and only keeping the ones with the highest volume of traffic,
> discarding the rest, relying on the default route to infill.

Given the connectionless nature of the Internet, route change based
on volume of traffic averaged over certain period of time is rather
harmful than useful.

Masataka Ohta
Re: SDN Internet Router (sir) [ In reply to ]
It depends on the number of these other routers. For a last-mile provider, you may have hundreds or even thousands of POPs that only connect to other parts of your network and customers.




-----
Mike Hammett
Intelligent Computing Solutions

Midwest Internet Exchange

The Brothers WISP

----- Original Message -----

From: "Matthew Walster" <matthew@walster.org>
To: "Mike Hammett" <nanog@ics-il.net>
Cc: "NANOG" <nanog@nanog.org>, "Forrest Christian (List Account)" <lists@packetflux.com>
Sent: Friday, January 6, 2023 10:10:56 AM
Subject: Re: SDN Internet Router (sir)







On Fri, 6 Jan 2023, 18:38 Mike Hammett, < nanog@ics-il.net > wrote:




I suspect it always will have value, whether it's peering routers, POP routers, multi-homed customer routers, etc.




Indeed. It's not "clean" but it is an acceptable tradeoff if you know what you're doing, and how traffic sloshes around etc.


I wrote a tool once that took a number of BGP feeds and aggregated the prefixes based on the next-hop values, which was *amazingly* good at reducing FIB sizes, but consumed so much CPU and memory, not to mention the latency of updates during any sizeable churn event, that it proved less useful than just precomputing based on historical traffic flows and updating the lists semi-frequently.


The idea of Juniper's EPE etc is very attractive, and largely matches what I had done back then, but does it with a lot more finesse. Ultimately, it's a tradeoff between CapEx of the high FIB router and the OpEx of the engineers who have to maintain the often hacky solution ;)




M
Re: SDN Internet Router (sir) [ In reply to ]
From what perspective?


How often do Netflix, Cloudflare, Akamai, Google, etc. change what prefixes they're using in a given area? (Rhetorical). But I'd imagine it's not that much. I know one particular geography where you can measure the time between those changes in years. Not all may be so static, but they aren't THAT variable.




-----
Mike Hammett
Intelligent Computing Solutions

Midwest Internet Exchange

The Brothers WISP

----- Original Message -----

From: "Masataka Ohta" <mohta@necom830.hpcl.titech.ac.jp>
To: "Matthew Walster" <matthew@walster.org>
Cc: "nanog list" <nanog@nanog.org>
Sent: Saturday, January 7, 2023 8:44:59 AM
Subject: Re: SDN Internet Router (sir)

Matthew Walster wrote:

>>> No... It's action based. You can send it a different route, you can
>>> replicate it, you can drop it, you can mutate it...
>>
>> Replication is a poor alternative for multicast.

> You conveniently ignore things like IDS, port mirroring, things like that.

Wrong. Instead, you conveniently ignore that such forwarding
requires a link between an SDN router and a monitoring device
have the same or larger MTU than an incoming link of the SDN
router, which means the router and the monitoring device must
be tightly coupled effectively to be a single device.

Sometimes, packet loss possibility between them often requires
they must actually be the same device.

> No. There are far more actions than for prioritisation.

Just for fun? I'm afraid I already mentioned so.

> What if you want to make sure certain classes of traffic do not flow over a
> link, because it is unencrypted and/or sensitive, but you're happy to send
> as much TLS wrapped data as you like?

You are wrongly assuming TLS wrapped packets can be identified
packet by packet, as I wrote:

>> Unless pattern is as simple as having certain port number,
>> stateful filtering almost always needs all packets including
>> those matching expected pattern, I'm afraid.

So?

> What if you want to sample some flows in an ERSPAN like mechanism?

See above for MTU issues.

> What if you want to urgently drop a set of flows based on a known DDOS
> signature?

Urgently? Even though a DDOS signature is known in advance?

Why?

>> Unless pattern is as simple as having certain port number,
>> stateful filtering almost always needs all packets including
>> those matching expected pattern, I'm afraid.
>>
>
> Or a certain set of IP addresses. Policy based routing.

That's even simpler than port number to be treated by
having or not having proper routing table entries.

>> If default route is acceptable, just rely on it along with
>> 50 non default routes with plain IP routers.

> That's what OP is suggesting.

With plain IP routers?

> That's what SIR is. Classifying prefixes by
> traffic and only keeping the ones with the highest volume of traffic,
> discarding the rest, relying on the default route to infill.

Given the connectionless nature of the Internet, route change based
on volume of traffic averaged over certain period of time is rather
harmful than useful.

Masataka Ohta
Re: SDN Internet Router (sir) [ In reply to ]
" With plain IP routers?"


Yes, or, well, relatively plain, depending on the implementation.


The originally linked project used Arista.





-----
Mike Hammett
Intelligent Computing Solutions

Midwest Internet Exchange

The Brothers WISP

----- Original Message -----

From: "Masataka Ohta" <mohta@necom830.hpcl.titech.ac.jp>
To: "Matthew Walster" <matthew@walster.org>
Cc: "nanog list" <nanog@nanog.org>
Sent: Saturday, January 7, 2023 8:44:59 AM
Subject: Re: SDN Internet Router (sir)

Matthew Walster wrote:

>>> No... It's action based. You can send it a different route, you can
>>> replicate it, you can drop it, you can mutate it...
>>
>> Replication is a poor alternative for multicast.

> You conveniently ignore things like IDS, port mirroring, things like that.

Wrong. Instead, you conveniently ignore that such forwarding
requires a link between an SDN router and a monitoring device
have the same or larger MTU than an incoming link of the SDN
router, which means the router and the monitoring device must
be tightly coupled effectively to be a single device.

Sometimes, packet loss possibility between them often requires
they must actually be the same device.

> No. There are far more actions than for prioritisation.

Just for fun? I'm afraid I already mentioned so.

> What if you want to make sure certain classes of traffic do not flow over a
> link, because it is unencrypted and/or sensitive, but you're happy to send
> as much TLS wrapped data as you like?

You are wrongly assuming TLS wrapped packets can be identified
packet by packet, as I wrote:

>> Unless pattern is as simple as having certain port number,
>> stateful filtering almost always needs all packets including
>> those matching expected pattern, I'm afraid.

So?

> What if you want to sample some flows in an ERSPAN like mechanism?

See above for MTU issues.

> What if you want to urgently drop a set of flows based on a known DDOS
> signature?

Urgently? Even though a DDOS signature is known in advance?

Why?

>> Unless pattern is as simple as having certain port number,
>> stateful filtering almost always needs all packets including
>> those matching expected pattern, I'm afraid.
>>
>
> Or a certain set of IP addresses. Policy based routing.

That's even simpler than port number to be treated by
having or not having proper routing table entries.

>> If default route is acceptable, just rely on it along with
>> 50 non default routes with plain IP routers.

> That's what OP is suggesting.

With plain IP routers?

> That's what SIR is. Classifying prefixes by
> traffic and only keeping the ones with the highest volume of traffic,
> discarding the rest, relying on the default route to infill.

Given the connectionless nature of the Internet, route change based
on volume of traffic averaged over certain period of time is rather
harmful than useful.

Masataka Ohta
Re: SDN Internet Router (sir) [ In reply to ]
Mike Hammett wrote:

> " With plain IP routers?"

> Yes, or, well, relatively plain, depending on the implementation.

As completely plain routers have no difficulty to treat a
default route, it is a waste of money and effort to try to
have not so plain routers to do so regardless of whether the
routers are SDN ones or not.

Masataka Ohta
Re: SDN Internet Router (sir) [ In reply to ]
I don't even know where this conversation has gone anymore.




-----
Mike Hammett
Intelligent Computing Solutions

Midwest Internet Exchange

The Brothers WISP

----- Original Message -----

From: "Masataka Ohta" <mohta@necom830.hpcl.titech.ac.jp>
To: "Mike Hammett" <nanog@ics-il.net>
Cc: "nanog list" <nanog@nanog.org>, "Matthew Walster" <matthew@walster.org>
Sent: Wednesday, January 11, 2023 7:00:07 AM
Subject: Re: SDN Internet Router (sir)

Mike Hammett wrote:

> " With plain IP routers?"

> Yes, or, well, relatively plain, depending on the implementation.

As completely plain routers have no difficulty to treat a
default route, it is a waste of money and effort to try to
have not so plain routers to do so regardless of whether the
routers are SDN ones or not.

Masataka Ohta
Re: SDN Internet Router (sir) [ In reply to ]
On Wed, Jan 11, 2023 at 01:04:28PM -0600, Mike Hammett wrote:
> I don't even know where this conversation has gone anymore.


You have reached a terminal point in the NANOG mailing list state
machine: NOP-ARGUE

Many paths lead to this state, and it isn't unique to NANOG.

The sub-state is: I-THINK
People are now arguing about the relative merits of
their counter argument to what they *THINK* you said.

My take: having the term SDN in the Subject: doomed this conversation.