Mailing List Archive

Job,

I disagree on the fact that it is not fair to the BGP implementation ecosystem, to enforce a single piece of software to activate the no-export community by default, due to ignorance from the engineer(s) implementing the solution. It should be common sense that certain routes that should be advertised beyond the local AS, just like RFC1918 routes, and more. Also, wasn't it you that said Cisco routers had a bug in ignoring NO_EXPORT? Would you go on a rant with Cisco, even if Noction add that enabled checkbox by default?
Why are you not on your soap box about BIRD, FRrouting, OpenBGPd, Cisco, Juniper, etc... about how they can possibly allow every day screw ups to happen, but the same options like the NO_EXPORT community are available for the engineer to use? One solution would be to implement "BGP Group/Session Profiles" (ISP/RTBH/DDOS Filtering/Route Optimizers/etc) or a "BGP Session Wizard" (ask the operator questions about their intentions), then automatically generate import and export policies based on known accepted practices.
Another solution could be having the BGP daemon disclose the make, model family, and exact model of hardware it is running on, to BGP peers, and add more knobs into policy creation to match said values, and take action appropriately. That would be useful in getting around vendor specific issues, as well as belt & suspenders protection.
Ryan
On Aug 1 2020, at 9:58 am, Job Snijders <job@instituut.net> wrote:
> On Sat, Aug 01, 2020 at 06:50:55AM -0700, Ca By wrote:
> > I am not normally supporting a heavy hand in regulation, but i think it is
> > fair to say Noction and similar BGP optimizers are unsafe at any speed and
> > the FTC or similar should ban them in the USA. They harm consumers and are
> > a risk to national security / critical infrastructure
> >
> > Noction and similar could have set basic defaults (no-export, only create
> > /25 bogus routes to limit scope), but they have been clear that their greed
> > to suck up traffic does not benefit from these defaults and they wont do
> > it.
>
> Following a large scale BGP incident in March 2015, noction made it
> possible to optionally set the well-known NO_EXPORT community on route
> advertisements originated by IRP instances.
>
> "In order to further reduce the likelihood of these problems
> occurring in the future, we will be adding a feature within Noction
> IRP to give an option to tag all the more specific prefixes that it
> generates with the BGP NO_EXPORT community. This will not be enabled
> by default [snip]"
> https://www.noction.com/blog/route-optimizers
> Mar 27, 2015
>
> Due to NO_EXPORT not being set in the default configuration, there are
> probably if not certainly many unsuspecting network engineers who end up
> deploying this software - without ever even considering - to change that
> one setting in the configuration.
>
> Fast forward a few years and a few incidents, on the topic of default
> settings, following the Cloudflare/DQE/Verizon incident:
>
> "We do have no export community support and have done for many
> years. The use of more specifics is also optional. Neither replaces
> the need for filters."
> https://twitter.com/noction/status/1143177562191011840
> Jun 24, 2019
>
> Community members responded:
> "Noction have been facilitating Internet outages for years and
> years and the best thing they can say in response is that it is
> technically possible to use their product responsibly, they just
> don't ship it that way."
> https://twitter.com/PowerDNS_Bert/status/1143252745257979905
> June 24, 2019
>
> Last year Noction stated:
> "Nobody found this leak pleasant."
> https://www.noction.com/news/incident-response
> June 26, 2019
>
> Sentiment we all can agree with, change is needed!
> As far as I know, Noction IRP is the ONLY commercially available
> off-the-shelf BGP route manipulation software which - as default - does
> NOT set the BGP well-known NO_EXPORT community on the product's route
> advertisements. This is a product design decision which causes
> collateral damage.
>
> I would like to urge Noction to reconsider their position. Seek to
> migrate the existing users to use NO_EXPORT, and release a new version
> of the IRP software which sets NO_EXPORT BY DEFAULT on all generated
> routes.
>
> Kind regards,
> Job

Ryan,

The reason Noction is being singled out here as opposed to other BGP
speakers is that it inherently breaks several BGP protection mechanisms as
a means to achieve its purpose. BGP was never intended to be "optimized",
it was intended to be stable and scalable. While i'm sure there are
hundreds of operators that use these optimizers without incident, they are
a significant paint point for the rest of the internet.

They have created a platform that has the ease of use of a residential CPE,
but with the consequences of misuse of any DFZ platform. This allows users
who have little experience speaking BGP with the world to make these
mistakes because they don't know any better, whereas the other platforms
you mention require some knowledge to configure. It's not a perfect filter,
but it does create a barrier for the inept.

Since Noction has made it easy enough to configure their software so that
anyone can do it, with or without experience on the DFZ, they have SOME
responsibility to keep their software from accidentally breaking the
internet.

-Matt


On Sat, Aug 1, 2020 at 2:30 PM Ryan Hamel <ryan@rkhtech.org> wrote:

> Job,
>
> I disagree on the fact that it is not fair to the BGP implementation
> ecosystem, to enforce a single piece of software to activate the no-export
> community by default, due to ignorance from the engineer(s) implementing
> the solution. It should be common sense that certain routes that should be
> advertised beyond the local AS, just like RFC1918 routes, and more. Also,
> wasn't it you that said Cisco routers had a bug in ignoring NO_EXPORT?
> Would you go on a rant with Cisco, even if Noction add that enabled
> checkbox by default?
>
> Why are you not on your soap box about BIRD, FRrouting, OpenBGPd, Cisco,
> Juniper, etc... about how they can possibly allow every day screw ups to
> happen, but the same options like the NO_EXPORT community are available for
> the engineer to use? One solution would be to implement "BGP Group/Session
> Profiles" (ISP/RTBH/DDOS Filtering/Route Optimizers/etc) or a "BGP Session
> Wizard" (ask the operator questions about their intentions), then
> automatically generate import and export policies based on known accepted
> practices.
>
> Another solution could be having the BGP daemon disclose the make, model
> family, and exact model of hardware it is running on, to BGP peers, and add
> more knobs into policy creation to match said values, and take action
> appropriately. That would be useful in getting around vendor specific
> issues, as well as belt & suspenders protection.
>
> Ryan
> On Aug 1 2020, at 9:58 am, Job Snijders <job@instituut.net> wrote:
>
> On Sat, Aug 01, 2020 at 06:50:55AM -0700, Ca By wrote:
> > I am not normally supporting a heavy hand in regulation, but i think it
> is
> > fair to say Noction and similar BGP optimizers are unsafe at any speed
> and
> > the FTC or similar should ban them in the USA. They harm consumers and
> are
> > a risk to national security / critical infrastructure
> >
> > Noction and similar could have set basic defaults (no-export, only create
> > /25 bogus routes to limit scope), but they have been clear that their
> greed
> > to suck up traffic does not benefit from these defaults and they wont do
> > it.
>
> Following a large scale BGP incident in March 2015, noction made it
> possible to optionally set the well-known NO_EXPORT community on route
> advertisements originated by IRP instances.
>
> "In order to further reduce the likelihood of these problems
> occurring in the future, we will be adding a feature within Noction
> IRP to give an option to tag all the more specific prefixes that it
> generates with the BGP NO_EXPORT community. This will not be enabled
> by default [snip]"
> https://www.noction.com/blog/route-optimizers
> Mar 27, 2015
>
> Due to NO_EXPORT not being set in the default configuration, there are
> probably if not certainly many unsuspecting network engineers who end up
> deploying this software - without ever even considering - to change that
> one setting in the configuration.
>
> Fast forward a few years and a few incidents, on the topic of default
> settings, following the Cloudflare/DQE/Verizon incident:
>
> "We do have no export community support and have done for many
> years. The use of more specifics is also optional. Neither replaces
> the need for filters."
> https://twitter.com/noction/status/1143177562191011840
> Jun 24, 2019
>
> Community members responded:
>
> "Noction have been facilitating Internet outages for years and
> years and the best thing they can say in response is that it is
> technically possible to use their product responsibly, they just
> don't ship it that way."
> https://twitter.com/PowerDNS_Bert/status/1143252745257979905
> June 24, 2019
>
> Last year Noction stated:
>
> "Nobody found this leak pleasant."
> https://www.noction.com/news/incident-response
> June 26, 2019
>
> Sentiment we all can agree with, change is needed!
>
> As far as I know, Noction IRP is the ONLY commercially available
> off-the-shelf BGP route manipulation software which - as default - does
> NOT set the BGP well-known NO_EXPORT community on the product's route
> advertisements. This is a product design decision which causes
> collateral damage.
>
> I would like to urge Noction to reconsider their position. Seek to
> migrate the existing users to use NO_EXPORT, and release a new version
> of the IRP software which sets NO_EXPORT BY DEFAULT on all generated
> routes.
>
> Kind regards,
>
> Job
>
>

--
Matt Erculiani
ERCUL-ARIN

Was Tulix using Noction, or was it something else that caused their particular issue?




-----
Mike Hammett
Intelligent Computing Solutions
http://www.ics-il.com

Midwest-IX
http://www.midwest-ix.com

----- Original Message -----

From: "Job Snijders" <job@instituut.net>
To: nanog@nanog.org
Sent: Saturday, August 1, 2020 11:58:12 AM
Subject: Issue with Noction IRP default setting (Was: BGP route hijack by AS10990)

On Sat, Aug 01, 2020 at 06:50:55AM -0700, Ca By wrote:
> I am not normally supporting a heavy hand in regulation, but i think it is
> fair to say Noction and similar BGP optimizers are unsafe at any speed and
> the FTC or similar should ban them in the USA. They harm consumers and are
> a risk to national security / critical infrastructure
>
> Noction and similar could have set basic defaults (no-export, only create
> /25 bogus routes to limit scope), but they have been clear that their greed
> to suck up traffic does not benefit from these defaults and they wont do
> it.

Following a large scale BGP incident in March 2015, noction made it
possible to optionally set the well-known NO_EXPORT community on route
advertisements originated by IRP instances.

"In order to further reduce the likelihood of these problems
occurring in the future, we will be adding a feature within Noction
IRP to give an option to tag all the more specific prefixes that it
generates with the BGP NO_EXPORT community. This will not be enabled
by default [snip]"
https://www.noction.com/blog/route-optimizers
Mar 27, 2015

Due to NO_EXPORT not being set in the default configuration, there are
probably if not certainly many unsuspecting network engineers who end up
deploying this software - without ever even considering - to change that
one setting in the configuration.

Fast forward a few years and a few incidents, on the topic of default
settings, following the Cloudflare/DQE/Verizon incident:

"We do have no export community support and have done for many
years. The use of more specifics is also optional. Neither replaces
the need for filters."
https://twitter.com/noction/status/1143177562191011840
Jun 24, 2019

Community members responded:

"Noction have been facilitating Internet outages for years and
years and the best thing they can say in response is that it is
technically possible to use their product responsibly, they just
don't ship it that way."
https://twitter.com/PowerDNS_Bert/status/1143252745257979905
June 24, 2019

Last year Noction stated:

"Nobody found this leak pleasant."
https://www.noction.com/news/incident-response
June 26, 2019

Sentiment we all can agree with, change is needed!

As far as I know, Noction IRP is the ONLY commercially available
off-the-shelf BGP route manipulation software which - as default - does
NOT set the BGP well-known NO_EXPORT community on the product's route
advertisements. This is a product design decision which causes
collateral damage.

I would like to urge Noction to reconsider their position. Seek to
migrate the existing users to use NO_EXPORT, and release a new version
of the IRP software which sets NO_EXPORT BY DEFAULT on all generated
routes.

Kind regards,

Job

Matt,

Why are you blaming the ease of use on the vendor, for the operators lack of knowledge regarding BGP? That is like blaming a vehicle manufacturer for a person pressing the gas pedal in a car and not giving a toss about the rules of the road. The base foundation regarding the rules of the road mostly apply the same for driving a car, truck, bus, and semi/lorry truck. There is no excuse for ignorance just because the user interface is different (web browser vs. SSH client).
Adding a take on this, there are kids born after 9/11, with IP allocations and ASNs experimenting in the DFZ right now. If they can make it work, and not cause harm to other members in this community, it clearly demonstrates a lack of knowledge, or honest human error (which will never go away).
Anything that can be used, can be misused. With that said, why shouldn't ALL BGP software implementations encourage best practice? They decided RPKI validation was a good thing.
Ryan
On Aug 1 2020, at 4:12 pm, Matt Erculiani <merculiani@gmail.com> wrote:
> Ryan,
>
> The reason Noction is being singled out here as opposed to other BGP speakers is that it inherently breaks several BGP protection mechanisms as a means to achieve its purpose. BGP was never intended to be "optimized", it was intended to be stable and scalable. While i'm sure there are hundreds of operators that use these optimizers without incident, they are a significant paint point for the rest of the internet.
>
> They have created a platform that has the ease of use of a residential CPE, but with the consequences of misuse of any DFZ platform. This allows users who have little experience speaking BGP with the world to make these mistakes because they don't know any better, whereas the other platforms you mention require some knowledge to configure. It's not a perfect filter, but it does create a barrier for the inept.
>
> Since Noction has made it easy enough to configure their software so that anyone can do it, with or without experience on the DFZ, they have SOME responsibility to keep their software from accidentally breaking the internet.
>
> -Matt
>
>
> On Sat, Aug 1, 2020 at 2:30 PM Ryan Hamel <ryan@rkhtech.org (mailto:ryan@rkhtech.org)> wrote:
> > Job,
> >
> > I disagree on the fact that it is not fair to the BGP implementation ecosystem, to enforce a single piece of software to activate the no-export community by default, due to ignorance from the engineer(s) implementing the solution. It should be common sense that certain routes that should be advertised beyond the local AS, just like RFC1918 routes, and more. Also, wasn't it you that said Cisco routers had a bug in ignoring NO_EXPORT? Would you go on a rant with Cisco, even if Noction add that enabled checkbox by default?
> > Why are you not on your soap box about BIRD, FRrouting, OpenBGPd, Cisco, Juniper, etc... about how they can possibly allow every day screw ups to happen, but the same options like the NO_EXPORT community are available for the engineer to use? One solution would be to implement "BGP Group/Session Profiles" (ISP/RTBH/DDOS Filtering/Route Optimizers/etc) or a "BGP Session Wizard" (ask the operator questions about their intentions), then automatically generate import and export policies based on known accepted practices.
> > Another solution could be having the BGP daemon disclose the make, model family, and exact model of hardware it is running on, to BGP peers, and add more knobs into policy creation to match said values, and take action appropriately. That would be useful in getting around vendor specific issues, as well as belt & suspenders protection.
> > Ryan
> > On Aug 1 2020, at 9:58 am, Job Snijders <job@instituut.net (mailto:job@instituut.net)> wrote:
> > > On Sat, Aug 01, 2020 at 06:50:55AM -0700, Ca By wrote:
> > > > I am not normally supporting a heavy hand in regulation, but i think it is
> > > > fair to say Noction and similar BGP optimizers are unsafe at any speed and
> > > > the FTC or similar should ban them in the USA. They harm consumers and are
> > > > a risk to national security / critical infrastructure
> > > >
> > > > Noction and similar could have set basic defaults (no-export, only create
> > > > /25 bogus routes to limit scope), but they have been clear that their greed
> > > > to suck up traffic does not benefit from these defaults and they wont do
> > > > it.
> > >
> > > Following a large scale BGP incident in March 2015, noction made it
> > > possible to optionally set the well-known NO_EXPORT community on route
> > > advertisements originated by IRP instances.
> > >
> > > "In order to further reduce the likelihood of these problems
> > > occurring in the future, we will be adding a feature within Noction
> > > IRP to give an option to tag all the more specific prefixes that it
> > > generates with the BGP NO_EXPORT community. This will not be enabled
> > > by default [snip]"
> > > https://www.noction.com/blog/route-optimizers
> > > Mar 27, 2015
> > >
> > > Due to NO_EXPORT not being set in the default configuration, there are
> > > probably if not certainly many unsuspecting network engineers who end up
> > > deploying this software - without ever even considering - to change that
> > > one setting in the configuration.
> > >
> > > Fast forward a few years and a few incidents, on the topic of default
> > > settings, following the Cloudflare/DQE/Verizon incident:
> > >
> > > "We do have no export community support and have done for many
> > > years. The use of more specifics is also optional. Neither replaces
> > > the need for filters."
> > > https://twitter.com/noction/status/1143177562191011840
> > > Jun 24, 2019
> > >
> > > Community members responded:
> > > "Noction have been facilitating Internet outages for years and
> > > years and the best thing they can say in response is that it is
> > > technically possible to use their product responsibly, they just
> > > don't ship it that way."
> > > https://twitter.com/PowerDNS_Bert/status/1143252745257979905
> > > June 24, 2019
> > >
> > > Last year Noction stated:
> > > "Nobody found this leak pleasant."
> > > https://www.noction.com/news/incident-response
> > > June 26, 2019
> > >
> > > Sentiment we all can agree with, change is needed!
> > > As far as I know, Noction IRP is the ONLY commercially available
> > > off-the-shelf BGP route manipulation software which - as default - does
> > > NOT set the BGP well-known NO_EXPORT community on the product's route
> > > advertisements. This is a product design decision which causes
> > > collateral damage.
> > >
> > > I would like to urge Noction to reconsider their position. Seek to
> > > migrate the existing users to use NO_EXPORT, and release a new version
> > > of the IRP software which sets NO_EXPORT BY DEFAULT on all generated
> > > routes.
> > >
> > > Kind regards,
> > > Job
>
> --
> Matt Erculiani
> ERCUL-ARIN
>
>

On Sat, Aug 1, 2020 at 4:47 PM Ryan Hamel <ryan@rkhtech.org> wrote:

> Matt,
>
> Why are you blaming the ease of use on the vendor, for the operators lack
> of knowledge regarding BGP? That is like blaming a vehicle manufacturer for
> a person pressing the gas pedal in a car and not giving a toss about the
> rules of the road. The base foundation regarding the rules of the road
> mostly apply the same for driving a car, truck, bus, and semi/lorry truck.
> There is no excuse for ignorance just because the user interface is
> different (web browser vs. SSH client).
>

Vendors are responsible. The FTC slammed D-Link for being insecure and
they can slam Noction too

https://www.ftc.gov/news-events/press-releases/2019/07/d-link-agrees-make-security-enhancements-settle-ftc-litigation


Asking people in Pintos to not get in accidents is not an option.

https://www.tortmuseum.org/ford-pinto/




> Adding a take on this, there are kids born after 9/11, with IP allocations
> and ASNs experimenting in the DFZ right now. If they can make it work, and
> not cause harm to other members in this community, it clearly demonstrates
> a lack of knowledge, or honest human error (which will never go away).
>
> Anything that can be used, can be misused. With that said, why shouldn't
> ALL BGP software implementations encourage best practice? They decided RPKI
> validation was a good thing.
>
> Ryan
> On Aug 1 2020, at 4:12 pm, Matt Erculiani <merculiani@gmail.com> wrote:
>
> Ryan,
>
> The reason Noction is being singled out here as opposed to other BGP
> speakers is that it inherently breaks several BGP protection mechanisms as
> a means to achieve its purpose. BGP was never intended to be "optimized",
> it was intended to be stable and scalable. While i'm sure there are
> hundreds of operators that use these optimizers without incident, they are
> a significant paint point for the rest of the internet.
>
> They have created a platform that has the ease of use of a residential
> CPE, but with the consequences of misuse of any DFZ platform. This allows
> users who have little experience speaking BGP with the world to make these
> mistakes because they don't know any better, whereas the other platforms
> you mention require some knowledge to configure. It's not a perfect filter,
> but it does create a barrier for the inept.
>
> Since Noction has made it easy enough to configure their software so that
> anyone can do it, with or without experience on the DFZ, they have SOME
> responsibility to keep their software from accidentally breaking the
> internet.
>
> -Matt
>
>
> On Sat, Aug 1, 2020 at 2:30 PM Ryan Hamel <ryan@rkhtech.org> wrote:
>
> Job,
>
> I disagree on the fact that it is not fair to the BGP implementation
> ecosystem, to enforce a single piece of software to activate the no-export
> community by default, due to ignorance from the engineer(s) implementing
> the solution. It should be common sense that certain routes that should be
> advertised beyond the local AS, just like RFC1918 routes, and more. Also,
> wasn't it you that said Cisco routers had a bug in ignoring NO_EXPORT?
> Would you go on a rant with Cisco, even if Noction add that enabled
> checkbox by default?
>
> Why are you not on your soap box about BIRD, FRrouting, OpenBGPd, Cisco,
> Juniper, etc... about how they can possibly allow every day screw ups to
> happen, but the same options like the NO_EXPORT community are available for
> the engineer to use? One solution would be to implement "BGP Group/Session
> Profiles" (ISP/RTBH/DDOS Filtering/Route Optimizers/etc) or a "BGP Session
> Wizard" (ask the operator questions about their intentions), then
> automatically generate import and export policies based on known accepted
> practices.
>
> Another solution could be having the BGP daemon disclose the make, model
> family, and exact model of hardware it is running on, to BGP peers, and add
> more knobs into policy creation to match said values, and take action
> appropriately. That would be useful in getting around vendor specific
> issues, as well as belt & suspenders protection.
>
> Ryan
> On Aug 1 2020, at 9:58 am, Job Snijders <job@instituut.net> wrote:
>
> On Sat, Aug 01, 2020 at 06:50:55AM -0700, Ca By wrote:
> > I am not normally supporting a heavy hand in regulation, but i think it
> is
> > fair to say Noction and similar BGP optimizers are unsafe at any speed
> and
> > the FTC or similar should ban them in the USA. They harm consumers and
> are
> > a risk to national security / critical infrastructure
> >
> > Noction and similar could have set basic defaults (no-export, only create
> > /25 bogus routes to limit scope), but they have been clear that their
> greed
> > to suck up traffic does not benefit from these defaults and they wont do
> > it.
>
> Following a large scale BGP incident in March 2015, noction made it
> possible to optionally set the well-known NO_EXPORT community on route
> advertisements originated by IRP instances.
>
> "In order to further reduce the likelihood of these problems
> occurring in the future, we will be adding a feature within Noction
> IRP to give an option to tag all the more specific prefixes that it
> generates with the BGP NO_EXPORT community. This will not be enabled
> by default [snip]"
> https://www.noction.com/blog/route-optimizers
> Mar 27, 2015
>
> Due to NO_EXPORT not being set in the default configuration, there are
> probably if not certainly many unsuspecting network engineers who end up
> deploying this software - without ever even considering - to change that
> one setting in the configuration.
>
> Fast forward a few years and a few incidents, on the topic of default
> settings, following the Cloudflare/DQE/Verizon incident:
>
> "We do have no export community support and have done for many
> years. The use of more specifics is also optional. Neither replaces
> the need for filters."
> https://twitter.com/noction/status/1143177562191011840
> Jun 24, 2019
>
> Community members responded:
>
> "Noction have been facilitating Internet outages for years and
> years and the best thing they can say in response is that it is
> technically possible to use their product responsibly, they just
> don't ship it that way."
> https://twitter.com/PowerDNS_Bert/status/1143252745257979905
> June 24, 2019
>
> Last year Noction stated:
>
> "Nobody found this leak pleasant."
> https://www.noction.com/news/incident-response
> June 26, 2019
>
> Sentiment we all can agree with, change is needed!
>
> As far as I know, Noction IRP is the ONLY commercially available
> off-the-shelf BGP route manipulation software which - as default - does
> NOT set the BGP well-known NO_EXPORT community on the product's route
> advertisements. This is a product design decision which causes
> collateral damage.
>
> I would like to urge Noction to reconsider their position. Seek to
> migrate the existing users to use NO_EXPORT, and release a new version
> of the IRP software which sets NO_EXPORT BY DEFAULT on all generated
> routes.
>
> Kind regards,
>
> Job
>
>
>
> --
> Matt Erculiani
> ERCUL-ARIN
>
>

Ryan,

To continue with your analogy, this would be more similar to someone who
has never driven before walking into a dealership and buying a new car to
drive off the lot. Ultimately the responsibility is on the driver, but the
dealership should have never sold them the car in the first place. Thus,
it's reasonable to place some of the responsibility on the vendors who
introduce this equipment into the wild without truly understanding (or
worse, not caring) how easy they've made it for their users to cause havoc
when most of the defaults are left untouched.

Again, it's not like anyone consciously sets these optimizers to evil,
they're just spontaneously dangerous when left misconfigured (which is
apparently easy to do). These devices throw caution to the wind in the name
of being easy to use.

Think of all the companies that set up AD servers with "corp.com" and how
big of a security risk that is. Microsoft ended up taking action because of
the potentially catastrophic implications, but at least it only affected
the companies that made poor choices. Now imagine those same individuals
who threw security to the wind are sold one of these devices because it's a
turnkey way to make their network faster; that's exactly what we see here
and it's terrifying to think how many of those little route hijack
timebombs are out there, just waiting to ruin your day, night, or vacation
in the Bahamas.

-Matt

On Sat, Aug 1, 2020 at 6:12 PM Ca By <cb.list6@gmail.com> wrote:

>
>
> On Sat, Aug 1, 2020 at 4:47 PM Ryan Hamel <ryan@rkhtech.org> wrote:
>
>> Matt,
>>
>> Why are you blaming the ease of use on the vendor, for the operators lack
>> of knowledge regarding BGP? That is like blaming a vehicle manufacturer for
>> a person pressing the gas pedal in a car and not giving a toss about the
>> rules of the road. The base foundation regarding the rules of the road
>> mostly apply the same for driving a car, truck, bus, and semi/lorry truck.
>> There is no excuse for ignorance just because the user interface is
>> different (web browser vs. SSH client).
>>
>
> Vendors are responsible. The FTC slammed D-Link for being insecure and
> they can slam Noction too
>
>
> https://www.ftc.gov/news-events/press-releases/2019/07/d-link-agrees-make-security-enhancements-settle-ftc-litigation
>
>
> Asking people in Pintos to not get in accidents is not an option.
>
> https://www.tortmuseum.org/ford-pinto/
>
>
>
>
>> Adding a take on this, there are kids born after 9/11, with IP
>> allocations and ASNs experimenting in the DFZ right now. If they can make
>> it work, and not cause harm to other members in this community, it clearly
>> demonstrates a lack of knowledge, or honest human error (which will never
>> go away).
>>
>> Anything that can be used, can be misused. With that said, why shouldn't
>> ALL BGP software implementations encourage best practice? They decided RPKI
>> validation was a good thing.
>>
>> Ryan
>> On Aug 1 2020, at 4:12 pm, Matt Erculiani <merculiani@gmail.com> wrote:
>>
>> Ryan,
>>
>> The reason Noction is being singled out here as opposed to other BGP
>> speakers is that it inherently breaks several BGP protection mechanisms as
>> a means to achieve its purpose. BGP was never intended to be "optimized",
>> it was intended to be stable and scalable. While i'm sure there are
>> hundreds of operators that use these optimizers without incident, they are
>> a significant paint point for the rest of the internet.
>>
>> They have created a platform that has the ease of use of a residential
>> CPE, but with the consequences of misuse of any DFZ platform. This allows
>> users who have little experience speaking BGP with the world to make these
>> mistakes because they don't know any better, whereas the other platforms
>> you mention require some knowledge to configure. It's not a perfect filter,
>> but it does create a barrier for the inept.
>>
>> Since Noction has made it easy enough to configure their software so that
>> anyone can do it, with or without experience on the DFZ, they have SOME
>> responsibility to keep their software from accidentally breaking the
>> internet.
>>
>> -Matt
>>
>>
>> On Sat, Aug 1, 2020 at 2:30 PM Ryan Hamel <ryan@rkhtech.org> wrote:
>>
>> Job,
>>
>> I disagree on the fact that it is not fair to the BGP implementation
>> ecosystem, to enforce a single piece of software to activate the no-export
>> community by default, due to ignorance from the engineer(s) implementing
>> the solution. It should be common sense that certain routes that should be
>> advertised beyond the local AS, just like RFC1918 routes, and more. Also,
>> wasn't it you that said Cisco routers had a bug in ignoring NO_EXPORT?
>> Would you go on a rant with Cisco, even if Noction add that enabled
>> checkbox by default?
>>
>> Why are you not on your soap box about BIRD, FRrouting, OpenBGPd, Cisco,
>> Juniper, etc... about how they can possibly allow every day screw ups to
>> happen, but the same options like the NO_EXPORT community are available for
>> the engineer to use? One solution would be to implement "BGP Group/Session
>> Profiles" (ISP/RTBH/DDOS Filtering/Route Optimizers/etc) or a "BGP Session
>> Wizard" (ask the operator questions about their intentions), then
>> automatically generate import and export policies based on known accepted
>> practices.
>>
>> Another solution could be having the BGP daemon disclose the make, model
>> family, and exact model of hardware it is running on, to BGP peers, and add
>> more knobs into policy creation to match said values, and take action
>> appropriately. That would be useful in getting around vendor specific
>> issues, as well as belt & suspenders protection.
>>
>> Ryan
>> On Aug 1 2020, at 9:58 am, Job Snijders <job@instituut.net> wrote:
>>
>> On Sat, Aug 01, 2020 at 06:50:55AM -0700, Ca By wrote:
>> > I am not normally supporting a heavy hand in regulation, but i think it
>> is
>> > fair to say Noction and similar BGP optimizers are unsafe at any speed
>> and
>> > the FTC or similar should ban them in the USA. They harm consumers and
>> are
>> > a risk to national security / critical infrastructure
>> >
>> > Noction and similar could have set basic defaults (no-export, only
>> create
>> > /25 bogus routes to limit scope), but they have been clear that their
>> greed
>> > to suck up traffic does not benefit from these defaults and they wont do
>> > it.
>>
>> Following a large scale BGP incident in March 2015, noction made it
>> possible to optionally set the well-known NO_EXPORT community on route
>> advertisements originated by IRP instances.
>>
>> "In order to further reduce the likelihood of these problems
>> occurring in the future, we will be adding a feature within Noction
>> IRP to give an option to tag all the more specific prefixes that it
>> generates with the BGP NO_EXPORT community. This will not be enabled
>> by default [snip]"
>> https://www.noction.com/blog/route-optimizers
>> Mar 27, 2015
>>
>> Due to NO_EXPORT not being set in the default configuration, there are
>> probably if not certainly many unsuspecting network engineers who end up
>> deploying this software - without ever even considering - to change that
>> one setting in the configuration.
>>
>> Fast forward a few years and a few incidents, on the topic of default
>> settings, following the Cloudflare/DQE/Verizon incident:
>>
>> "We do have no export community support and have done for many
>> years. The use of more specifics is also optional. Neither replaces
>> the need for filters."
>> https://twitter.com/noction/status/1143177562191011840
>> Jun 24, 2019
>>
>> Community members responded:
>>
>> "Noction have been facilitating Internet outages for years and
>> years and the best thing they can say in response is that it is
>> technically possible to use their product responsibly, they just
>> don't ship it that way."
>> https://twitter.com/PowerDNS_Bert/status/1143252745257979905
>> June 24, 2019
>>
>> Last year Noction stated:
>>
>> "Nobody found this leak pleasant."
>> https://www.noction.com/news/incident-response
>> June 26, 2019
>>
>> Sentiment we all can agree with, change is needed!
>>
>> As far as I know, Noction IRP is the ONLY commercially available
>> off-the-shelf BGP route manipulation software which - as default - does
>> NOT set the BGP well-known NO_EXPORT community on the product's route
>> advertisements. This is a product design decision which causes
>> collateral damage.
>>
>> I would like to urge Noction to reconsider their position. Seek to
>> migrate the existing users to use NO_EXPORT, and release a new version
>> of the IRP software which sets NO_EXPORT BY DEFAULT on all generated
>> routes.
>>
>> Kind regards,
>>
>> Job
>>
>>
>>
>> --
>> Matt Erculiani
>> ERCUL-ARIN
>>
>>

--
Matt Erculiani
ERCUL-ARIN

All,

Watching this thread with interest got an idea - let me run it by this list
before taking it any further (ie. to IETF).

How about we learn from this and try to make BGP just a little bit safer ?

*Idea: *

In all stub (non transit) ASNs we modify BGP spec and disable automatic
iBGP to eBGP advertisement ?

*Implementation: *

Vendors to allow to define as part of global bgp configuration if given ASN
is transit or not. The default is to be discussed - no bias.

*Benefit: *

Without any issues anyone playing any tools in his network will be able to
just issue one cli and be protected from accidentally hurting others. Yet
naturally he will still be able to advertise his neworks just as today
except by explicit policy in any shape and form we would find proper
(example: "redistribute iBGP to eBGP policy-X").

We could even discuss if this should be perhaps part of BGP OPEN or BGP
capabilities too such that two sides of eBGP session must agree with each
other before bringing eBGP up.

Comments, questions, flames - all welcome :)

Cheers,
Robert.

PS. Such a definition sure can and likely will be misused (especially if we
would just settle on only a single side setting it - but that will not
cause any more harm as not having it at all.

Moreover I can already see few other good options which BGP implementation
or BGP spec can be augmented with once we know we are stub or for that
matter once it knows it is transit ....

On 1/Aug/20 18:58, Job Snijders wrote:

> Following a large scale BGP incident in March 2015, noction made it
> possible to optionally set the well-known NO_EXPORT community on route
> advertisements originated by IRP instances.
>
> "In order to further reduce the likelihood of these problems
> occurring in the future, we will be adding a feature within Noction
> IRP to give an option to tag all the more specific prefixes that it
> generates with the BGP NO_EXPORT community. This will not be enabled
> by default [snip]"
> https://www.noction.com/blog/route-optimizers
> Mar 27, 2015
>
> Due to NO_EXPORT not being set in the default configuration, there are
> probably if not certainly many unsuspecting network engineers who end up
> deploying this software - without ever even considering - to change that
> one setting in the configuration.
>
> Fast forward a few years and a few incidents, on the topic of default
> settings, following the Cloudflare/DQE/Verizon incident:
>
> "We do have no export community support and have done for many
> years. The use of more specifics is also optional. Neither replaces
> the need for filters."
> https://twitter.com/noction/status/1143177562191011840
> Jun 24, 2019
>
> Community members responded:
>
> "Noction have been facilitating Internet outages for years and
> years and the best thing they can say in response is that it is
> technically possible to use their product responsibly, they just
> don't ship it that way."
> https://twitter.com/PowerDNS_Bert/status/1143252745257979905
> June 24, 2019
>
> Last year Noction stated:
>
> "Nobody found this leak pleasant."
> https://www.noction.com/news/incident-response
> June 26, 2019
>
> Sentiment we all can agree with, change is needed!
>
> As far as I know, Noction IRP is the ONLY commercially available
> off-the-shelf BGP route manipulation software which - as default - does
> NOT set the BGP well-known NO_EXPORT community on the product's route
> advertisements. This is a product design decision which causes
> collateral damage.
>
> I would like to urge Noction to reconsider their position. Seek to
> migrate the existing users to use NO_EXPORT, and release a new version
> of the IRP software which sets NO_EXPORT BY DEFAULT on all generated
> routes.

A great first step!

Mark.

On 1/Aug/20 22:29, Ryan Hamel wrote:
> Job,
>
> I disagree on the fact that it is not fair to the BGP implementation
> ecosystem, to enforce a single piece of software to activate the
> no-export community by default, due to ignorance from the engineer(s)
> implementing the solution. It should be common sense that certain
> routes that should be advertised beyond the local AS, just like
> RFC1918 routes, and more. Also, wasn't it you that said Cisco routers
> had a bug in ignoring NO_EXPORT? Would you go on a rant with Cisco,
> even if Noction add that enabled checkbox by default?
>
> Why are you not on your soap box about BIRD, FRrouting, OpenBGPd,
> Cisco, Juniper, etc... about how they can possibly allow every day
> screw ups to happen, but the same options like the NO_EXPORT community
> are available for the engineer to use? One solution would be to
> implement "BGP Group/Session Profiles" (ISP/RTBH/DDOS Filtering/Route
> Optimizers/etc) or a "BGP Session Wizard" (ask the operator questions
> about their intentions), then automatically generate import and export
> policies based on known accepted practices.
>
> Another solution could be having the BGP daemon disclose the make,
> model family, and exact model of hardware it is running on, to BGP
> peers, and add more knobs into policy creation to match said values,
> and take action appropriately. That would be useful in getting around
> vendor specific issues, as well as belt & suspenders protection.

Most (if not all) people buying BGP optimizers aren't using them for
regular BGP-speaking routers to the rest of the Internet or their core
network.

BGP optimizers serve a unique use-case, which works in the way it does
to create an expected risk as we saw in this and past incidents. On that
basis, I think Job's request to make NO_EXPORT a mandatory default (I'd
go further to say the new default mode can be user-disabled, but with an
unmistakable warning in the UI) is not unreasonable.

Mark.

On Sun, Aug 2, 2020 at 4:34 AM Robert Raszuk <robert@raszuk.net> wrote:

> All,
>
> Watching this thread with interest got an idea - let me run it by this
> list before taking it any further (ie. to IETF).
>
> How about we learn from this and try to make BGP just a little bit safer ?
>
> *Idea: *
>
> In all stub (non transit) ASNs we modify BGP spec and disable automatic
> iBGP to eBGP advertisement ?
>

Why do you believe a stub AS was involved or that would have changed this
situation?

The whole point of Noction is for a bad isp to fake more specific routes to
downstream customers. Noction is sold to ISPs, aka transit AS, afaik



> *Implementation: *
>
> Vendors to allow to define as part of global bgp configuration if
> given ASN is transit or not. The default is to be discussed - no bias.
>

Oh. A configuration knob. Noction had knobs, the world runs of 5 year old
software with default configs.


> *Benefit: *
>
> Without any issues anyone playing any tools in his network will be able to
> just issue one cli
>

Thanks for no pretending we configure our networks with yang model apis

and be protected from accidentally hurting others. Yet naturally he will
> still be able to advertise his neworks just as today except by explicit
> policy in any shape and form we would find proper (example:
> "redistribute iBGP to eBGP policy-X").
>

XR rolls this way today, thanks Cisco. But the “any” keyword exists, so
yolo.


> We could even discuss if this should be perhaps part of BGP OPEN or BGP
> capabilities too such that two sides of eBGP session must agree with each
> other before bringing eBGP up.
>
> Comments, questions, flames - all welcome :)
>
> Cheers,
> Robert.
>
> PS. Such a definition sure can and likely will be misused (especially if
> we would just settle on only a single side setting it - but that will not
> cause any more harm as not having it at all.
>
> Moreover I can already see few other good options which BGP implementation
> or BGP spec can be augmented with once we know we are stub or for that
> matter once it knows it is transit ....
>
>

Hi Ca,

> Noction is sold to ISPs, aka transit AS, afaik

Interesting.

My impression always was by talking to Noction some time back that mainly
what they do is a flavor of performance routing. But this is not about
Noction IMHO.

If I am a non transit ASN with N upstream ISPs I want to exit not in a hot
potato style ... if I care about my services I want to exit the best
performing way to reach back customers. That's btw what Cisco PFR does or
Google's Espresso or Facebook Edge Fabric etc ...

And you have few vendors offering this as well as bunch of home grown tools
attempting to do the same. Go and mandate that all of them will do
NO-EXPORT if they insert any routes ... And we will see more and more of
those type of tools coming.

Sure we have implementations with obligatory policy on eBGP - cool. And yes
we have match "ANY" too.

So if your feedback is that to limit the iBGP routes to go out over eBGP
this is all sufficient and we do not need a bit more protection there then
case solved.

Cheers,
R.



On Sun, Aug 2, 2020 at 4:42 PM Ca By <cb.list6@gmail.com> wrote:

>
>
> On Sun, Aug 2, 2020 at 4:34 AM Robert Raszuk <robert@raszuk.net> wrote:
>
>> All,
>>
>> Watching this thread with interest got an idea - let me run it by this
>> list before taking it any further (ie. to IETF).
>>
>> How about we learn from this and try to make BGP just a little bit safer
>> ?
>>
>> *Idea: *
>>
>> In all stub (non transit) ASNs we modify BGP spec and disable automatic
>> iBGP to eBGP advertisement ?
>>
>
> Why do you believe a stub AS was involved or that would have changed this
> situation?
>
> The whole point of Noction is for a bad isp to fake more specific routes
> to downstream customers. Noction is sold to ISPs, aka transit AS, afaik
>
>
>
>> *Implementation: *
>>
>> Vendors to allow to define as part of global bgp configuration if
>> given ASN is transit or not. The default is to be discussed - no bias.
>>
>
> Oh. A configuration knob. Noction had knobs, the world runs of 5 year old
> software with default configs.
>
>
>> *Benefit: *
>>
>> Without any issues anyone playing any tools in his network will be able
>> to just issue one cli
>>
>
> Thanks for no pretending we configure our networks with yang model apis
>
> and be protected from accidentally hurting others. Yet naturally he will
>> still be able to advertise his neworks just as today except by explicit
>> policy in any shape and form we would find proper (example:
>> "redistribute iBGP to eBGP policy-X").
>>
>
> XR rolls this way today, thanks Cisco. But the “any” keyword exists, so
> yolo.
>
>
>> We could even discuss if this should be perhaps part of BGP OPEN or BGP
>> capabilities too such that two sides of eBGP session must agree with each
>> other before bringing eBGP up.
>>
>> Comments, questions, flames - all welcome :)
>>
>> Cheers,
>> Robert.
>>
>> PS. Such a definition sure can and likely will be misused (especially if
>> we would just settle on only a single side setting it - but that will not
>> cause any more harm as not having it at all.
>>
>> Moreover I can already see few other good options which BGP
>> implementation or BGP spec can be augmented with once we know we are stub
>> or for that matter once it knows it is transit ....
>>
>>

On 2/Aug/20 01:44, Ryan Hamel wrote:
> Matt,
>
> Why are you blaming the ease of use on the vendor, for the operators
> lack of knowledge regarding BGP? That is like blaming a vehicle
> manufacturer for a person pressing the gas pedal in a car and not
> giving a toss about the rules of the road. The base foundation
> regarding the rules of the road mostly apply the same for driving a
> car, truck, bus, and semi/lorry truck. There is no excuse for
> ignorance just because the user interface is different (web browser
> vs. SSH client).

Actually, there is.

One has to actually acquire knowledge about not only driving a car, but
driving it in public. That knowledge is then validated by a
gubbermint-sanctioned driver's license test. If you fail, you aren't
allowed to drive. If you are caught driving without a driver's license,
you pay the penalty.

There is no requirement for a license in order to run power into a
router and hook it up to the Internet. This is the problem I have with
the current state of how we support BGP actors.

> Adding a take on this, there are kids born after 9/11, with IP
> allocations and ASNs experimenting in the DFZ right now. If they can
> make it work, and not cause harm to other members in this community,
> it clearly demonstrates a lack of knowledge, or honest human error
> (which will never go away).

We should not be celebrating this.


>
> Anything that can be used, can be misused. With that said, why
> shouldn't ALL BGP software implementations encourage best practice?
> They decided RPKI validation was a good thing.

The larger question is we should find a way to make our industry
genuinely qualification-based, and not "free for all that decides they
want to try it out".

I don't yet know how to do that, but we certainly need to start thinking
more seriously about it. Kids born after 9/11 successfully experimenting
on a global network is not where the bar ought to be.

Mark.

And bgp "optimizer" won't do that
At best, they will let you get the less worst



On 8/2/20 6:36 PM, Robert Raszuk wrote:
> if I care about my services I want to exit the best
> performing way to reach back customers.

Mark,

I think trying to implement some kind of license requirement for DFZ
participants is a step in the wrong direction and a waste of time and
money. How would you even enforce it? If the goal is just to provide a
bigger barrier to "kids born after 9/11", why not just increase RIR fees,
or add an age requirement for individuals? And anyway, why do we need to
increase that barrier? What problem does that actually solve? Are "kids
born after 9/11" the ones propagating route leaks? I don't think they are.
But the reason for that is not that they're necessarily more skilled
operators than "adults born before 9/11" or anyone else - it's that they
are being filtered appropriately by the likes of Vultr, etc. Verizon (and
other large incumbents) could learn something from them.

Let's try to stay away from exclusivity for exclusivity's sake and actually
focus on solving the real problems we have.

On Sun, Aug 2, 2020 at 12:45 PM Mark Tinka <mark.tinka@seacom.com> wrote:

>
>
> On 2/Aug/20 01:44, Ryan Hamel wrote:
>
> Matt,
>
> Why are you blaming the ease of use on the vendor, for the operators lack
> of knowledge regarding BGP? That is like blaming a vehicle manufacturer for
> a person pressing the gas pedal in a car and not giving a toss about the
> rules of the road. The base foundation regarding the rules of the road
> mostly apply the same for driving a car, truck, bus, and semi/lorry truck.
> There is no excuse for ignorance just because the user interface is
> different (web browser vs. SSH client).
>
>
> Actually, there is.
>
> One has to actually acquire knowledge about not only driving a car, but
> driving it in public. That knowledge is then validated by a
> gubbermint-sanctioned driver's license test. If you fail, you aren't
> allowed to drive. If you are caught driving without a driver's license, you
> pay the penalty.
>
> There is no requirement for a license in order to run power into a router
> and hook it up to the Internet. This is the problem I have with the current
> state of how we support BGP actors.
>
> Adding a take on this, there are kids born after 9/11, with IP allocations
> and ASNs experimenting in the DFZ right now. If they can make it work, and
> not cause harm to other members in this community, it clearly demonstrates
> a lack of knowledge, or honest human error (which will never go away).
>
>
> We should not be celebrating this.
>
>
>
> Anything that can be used, can be misused. With that said, why shouldn't
> ALL BGP software implementations encourage best practice? They decided RPKI
> validation was a good thing.
>
>
> The larger question is we should find a way to make our industry genuinely
> qualification-based, and not "free for all that decides they want to try it
> out".
>
> I don't yet know how to do that, but we certainly need to start thinking
> more seriously about it. Kids born after 9/11 successfully experimenting on
> a global network is not where the bar ought to be.
>
> Mark.
>

On Sun, Aug 2, 2020 at 9:36 AM Robert Raszuk <robert@raszuk.net> wrote:

> Hi Ca,
>
> > Noction is sold to ISPs, aka transit AS, afaik
>
> Interesting.
>
> My impression always was by talking to Noction some time back that mainly
> what they do is a flavor of performance routing. But this is not about
> Noction IMHO.
>
> If I am a non transit ASN with N upstream ISPs I want to exit not in a hot
> potato style ... if I care about my services I want to exit the best
> performing way to reach back customers. That's btw what Cisco PFR does or
> Google's Espresso or Facebook Edge Fabric etc ...
>
> And you have few vendors offering this as well as bunch of home grown
> tools attempting to do the same. Go and mandate that all of them will do
> NO-EXPORT if they insert any routes ... And we will see more and more of
> those type of tools coming.
>
> Sure we have implementations with obligatory policy on eBGP - cool. And
> yes we have match "ANY" too.
>
> So if your feedback is that to limit the iBGP routes to go out over eBGP
> this is all sufficient and we do not need a bit more protection there then
> case solved.
>
> Cheers,
> R.
>
>
My feedback is the local_pref is complete for this behavior of setting an
outbound, including being non-transitive

FB uses local-pref for this afaik
https://research.fb.com/blog/2017/08/steering-oceans-of-content-to-the-world/


>
> On Sun, Aug 2, 2020 at 4:42 PM Ca By <cb.list6@gmail.com> wrote:
>
>>
>>
>> On Sun, Aug 2, 2020 at 4:34 AM Robert Raszuk <robert@raszuk.net> wrote:
>>
>>> All,
>>>
>>> Watching this thread with interest got an idea - let me run it by this
>>> list before taking it any further (ie. to IETF).
>>>
>>> How about we learn from this and try to make BGP just a little bit safer
>>> ?
>>>
>>> *Idea: *
>>>
>>> In all stub (non transit) ASNs we modify BGP spec and disable automatic
>>> iBGP to eBGP advertisement ?
>>>
>>
>> Why do you believe a stub AS was involved or that would have changed this
>> situation?
>>
>> The whole point of Noction is for a bad isp to fake more specific routes
>> to downstream customers. Noction is sold to ISPs, aka transit AS, afaik
>>
>>
>>
>>> *Implementation: *
>>>
>>> Vendors to allow to define as part of global bgp configuration if
>>> given ASN is transit or not. The default is to be discussed - no bias.
>>>
>>
>> Oh. A configuration knob. Noction had knobs, the world runs of 5 year old
>> software with default configs.
>>
>>
>>> *Benefit: *
>>>
>>> Without any issues anyone playing any tools in his network will be able
>>> to just issue one cli
>>>
>>
>> Thanks for no pretending we configure our networks with yang model apis
>>
>> and be protected from accidentally hurting others. Yet naturally he will
>>> still be able to advertise his neworks just as today except by explicit
>>> policy in any shape and form we would find proper (example:
>>> "redistribute iBGP to eBGP policy-X").
>>>
>>
>> XR rolls this way today, thanks Cisco. But the “any” keyword exists, so
>> yolo.
>>
>>
>>> We could even discuss if this should be perhaps part of BGP OPEN or BGP
>>> capabilities too such that two sides of eBGP session must agree with each
>>> other before bringing eBGP up.
>>>
>>> Comments, questions, flames - all welcome :)
>>>
>>> Cheers,
>>> Robert.
>>>
>>> PS. Such a definition sure can and likely will be misused (especially if
>>> we would just settle on only a single side setting it - but that will not
>>> cause any more harm as not having it at all.
>>>
>>> Moreover I can already see few other good options which BGP
>>> implementation or BGP spec can be augmented with once we know we are stub
>>> or for that matter once it knows it is transit ....
>>>
>>>

On 2/Aug/20 21:37, Ross Tajvar wrote:
> Mark,
>
> I think trying to implement some kind of license requirement for DFZ
> participants is a step in the wrong direction and a waste of time and
> money. How would you even enforce it? If the goal is just to provide a
> bigger barrier to "kids born after 9/11", why not just increase RIR
> fees, or add an age requirement for individuals? And anyway, why do we
> need to increase that barrier? What problem does that actually solve?
> Are "kids born after 9/11" the ones propagating route leaks? I don't
> think they are. But the reason for that is not that they're
> necessarily more skilled operators than "adults born before 9/11" or
> anyone else - it's that they are being filtered appropriately by the
> likes of Vultr, etc. Verizon (and other large incumbents) could learn
> something from them.
>
> Let's try to stay away from exclusivity for exclusivity's sake and
> actually focus on solving the real problems we have.

Like I said before, "guidance" rather than "regulation".

The way the Internet has worked for 4+ decades has been what has made it
so successful. However, it's starting to catch up with us, so we need to
figure it out, and not bury our heads in the sand until it hurts me or
you more directly for either us to care.

Like I also said, I don't quite know how to solve this problem yet. What
I do know is if we keep having this dance every few months each year, it
will be 2050 and we'll still be in the same place, only worse.

Before we can find a solution, we have to realize that there is a
problem. There is enough smarts in the community to find a solution.
Hopefully before some silly gubbermint (TikTok ban, anyone?) decides for us.

Mark.

I guess I missed your mention of "guidance rather than regulation", and am
still missing it, unless you're referring to another thread.

If you want to acknowledge a problem with internet governance and bring it
to this mailing list for discussion, that sounds like a good idea. But the
only "problem" I've seen you bring up in this thread is the participation
of young people, and I've yet to hear a reason why that's a bad thing. This
just sounds like gatekeeping to me.

If we want to improve routing security, then rather than making vague
claims about things "catching up with us" with no clear problem statement,
we should be focusing our efforts on basic safeguards like filtering and
RPKI OV. I don't consider that "burying my head in the sand".

On Sun, Aug 2, 2020, 5:24 PM Mark Tinka <mark.tinka@seacom.com> wrote:

>
>
> On 2/Aug/20 21:37, Ross Tajvar wrote:
> > Mark,
> >
> > I think trying to implement some kind of license requirement for DFZ
> > participants is a step in the wrong direction and a waste of time and
> > money. How would you even enforce it? If the goal is just to provide a
> > bigger barrier to "kids born after 9/11", why not just increase RIR
> > fees, or add an age requirement for individuals? And anyway, why do we
> > need to increase that barrier? What problem does that actually solve?
> > Are "kids born after 9/11" the ones propagating route leaks? I don't
> > think they are. But the reason for that is not that they're
> > necessarily more skilled operators than "adults born before 9/11" or
> > anyone else - it's that they are being filtered appropriately by the
> > likes of Vultr, etc. Verizon (and other large incumbents) could learn
> > something from them.
> >
> > Let's try to stay away from exclusivity for exclusivity's sake and
> > actually focus on solving the real problems we have.
>
> Like I said before, "guidance" rather than "regulation".
>
> The way the Internet has worked for 4+ decades has been what has made it
> so successful. However, it's starting to catch up with us, so we need to
> figure it out, and not bury our heads in the sand until it hurts me or
> you more directly for either us to care.
>
> Like I also said, I don't quite know how to solve this problem yet. What
> I do know is if we keep having this dance every few months each year, it
> will be 2050 and we'll still be in the same place, only worse.
>
> Before we can find a solution, we have to realize that there is a
> problem. There is enough smarts in the community to find a solution.
> Hopefully before some silly gubbermint (TikTok ban, anyone?) decides for
> us.
>
> Mark.
>
>

I don't think there's any requirement for it to be for downstream customers (from a BGP perspective) or any relatance to transit ASes.




Web hosting companies, their AS, no client ASes, huge optimization going on. I'd think mostly because the major eyeball ISPs have garbage peering policies and like to run their ports hot to force you to buy their transit\DIA.




-----
Mike Hammett
Intelligent Computing Solutions
http://www.ics-il.com

Midwest-IX
http://www.midwest-ix.com

----- Original Message -----

From: "Ca By" <cb.list6@gmail.com>
To: "Robert Raszuk" <robert@raszuk.net>
Cc: nanog@nanog.org
Sent: Sunday, August 2, 2020 9:42:12 AM
Subject: Re: Issue with Noction IRP default setting (Was: BGP route hijack by AS10990)







On Sun, Aug 2, 2020 at 4:34 AM Robert Raszuk < robert@raszuk.net > wrote:




All,


Watching this thread with interest got an idea - let me run it by this list before taking it any further (ie. to IETF).


How about we learn from this and try to make BGP just a little bit safer ?


Idea:


In all stub (non transit) ASNs we modify BGP spec and disable automatic iBGP to eBGP advertisement ?





Why do you believe a stub AS was involved or that would have changed this situation?


The whole point of Noction is for a bad isp to fake more specific routes to downstream customers. Noction is sold to ISPs, aka transit AS, afaik




<blockquote>




Implementation:


Vendors to allow to define as part of global bgp configuration if given ASN is transit or not. The default is to be discussed - no bias.
</blockquote>



Oh. A configuration knob. Noction had knobs, the world runs of 5 year old software with default configs.


<blockquote>





Benefit:


Without any issues anyone playing any tools in his network will be able to just issue one cli
</blockquote>



Thanks for no pretending we configure our networks with yang model apis


<blockquote>


and be protected from accidentally hurting others. Yet naturally he will still be able to advertise his neworks just as today except by explicit policy in any shape and form we would find proper (example: "redistribute iBGP to eBGP policy-X").

</blockquote>



XR rolls this way today, thanks Cisco. But the “any” keyword exists, so yolo.


<blockquote>





We could even discuss if this should be perhaps part of BGP OPEN or BGP capabilities too such that two sides of eBGP session must agree with each other before bringing eBGP up.



Comments, questions, flames - all welcome :)



Cheers,
Robert.

PS. Such a definition sure can and likely will be misused (especially if we would just settle on only a single side setting it - but that will not cause any more harm as not having it at all.


Moreover I can already see few other good options which BGP implementation or BGP spec can be augmented with once we know we are stub or for that matter once it knows it is transit ....



<blockquote>

</blockquote>

</blockquote>

On 3/Aug/20 00:03, Ross Tajvar wrote:
> I guess I missed your mention of "guidance rather than regulation",
> and am still missing it, unless you're referring to another thread.
>
> If you want to acknowledge a problem with internet governance and
> bring it to this mailing list for discussion, that sounds like a good
> idea. But the only "problem" I've seen you bring up in this thread is
> the participation of young people, and I've yet to hear a reason why
> that's a bad thing. This just sounds like gatekeeping to me.
>
> If we want to improve routing security, then rather than making vague
> claims about things "catching up with us" with no clear problem
> statement, we should be focusing our efforts on basic safeguards like
> filtering and RPKI OV. I don't consider that "burying my head in the
> sand".

You may have missed most of these fundamentals much earlier in the thread.

I'm not looking to repeat myself, so you're welcome to start from the
top and come back if you have any more questions.

Mark.

>
> Why are you not on your soap box about BIRD, FRrouting, OpenBGPd, Cisco,
> Juniper, etc... about how they can possibly allow every day screw ups to
> happen, but the same options like the NO_EXPORT community are available for
> the engineer to use? One solution would be to implement "BGP Group/Session
> Profiles" (ISP/RTBH/DDOS Filtering/Route Optimizers/etc) or a "BGP Session
> Wizard" (ask the operator questions about their intentions), then
> automatically generate import and export policies based on known accepted
> practices.
>

You seem to be implying that nobody has ever given feedback to a vendor
about their BGP implementation. That's incredibly far from the truth.
Default parameters on many NOS have been changed over the years to be safer
for 2AM compliance, or for operators with lesser experience.

It's correct that someone with any of those BGP implementations can make
configuration errors. The difference is those configuration errors are LESS
LIKELY to cause widespread disruption in the DFZ. I think back to many
years ago at the start of my career, and the first time I configured BGP on
a router with 2 upstreams. In an amazing rookie move, I created config
which I did not apply, reannouncing everything from 3356 to 1239 via
myself, and vice versa. While embracing, only a very small amount of
traffic ( <10Mbps ) and prefixes were impacted, since BGP worked as
designed, and the longer AS PATH I created was less desirable for almost
everyone.

IF you are going to create more specific announcements, be it with a BGP
"optimizer" , or with other BGP implementations, the SAFEST method to
prevent unintended consequences would be to add guardrails, like NO_EXPORT.
It's just a best practice. When you can make those good best practices a
default behavior? Even better! There is no downside to Nocton making
NO_EXPORT the default behavior, only upside to the stability of the
internet at large.

On Sat, Aug 1, 2020 at 4:31 PM Ryan Hamel <ryan@rkhtech.org> wrote:

> Job,
>
> I disagree on the fact that it is not fair to the BGP implementation
> ecosystem, to enforce a single piece of software to activate the no-export
> community by default, due to ignorance from the engineer(s) implementing
> the solution. It should be common sense that certain routes that should be
> advertised beyond the local AS, just like RFC1918 routes, and more. Also,
> wasn't it you that said Cisco routers had a bug in ignoring NO_EXPORT?
> Would you go on a rant with Cisco, even if Noction add that enabled
> checkbox by default?
>
> Why are you not on your soap box about BIRD, FRrouting, OpenBGPd, Cisco,
> Juniper, etc... about how they can possibly allow every day screw ups to
> happen, but the same options like the NO_EXPORT community are available for
> the engineer to use? One solution would be to implement "BGP Group/Session
> Profiles" (ISP/RTBH/DDOS Filtering/Route Optimizers/etc) or a "BGP Session
> Wizard" (ask the operator questions about their intentions), then
> automatically generate import and export policies based on known accepted
> practices.
>
> Another solution could be having the BGP daemon disclose the make, model
> family, and exact model of hardware it is running on, to BGP peers, and add
> more knobs into policy creation to match said values, and take action
> appropriately. That would be useful in getting around vendor specific
> issues, as well as belt & suspenders protection.
>
> Ryan
> On Aug 1 2020, at 9:58 am, Job Snijders <job@instituut.net> wrote:
>
> On Sat, Aug 01, 2020 at 06:50:55AM -0700, Ca By wrote:
> > I am not normally supporting a heavy hand in regulation, but i think it
> is
> > fair to say Noction and similar BGP optimizers are unsafe at any speed
> and
> > the FTC or similar should ban them in the USA. They harm consumers and
> are
> > a risk to national security / critical infrastructure
> >
> > Noction and similar could have set basic defaults (no-export, only create
> > /25 bogus routes to limit scope), but they have been clear that their
> greed
> > to suck up traffic does not benefit from these defaults and they wont do
> > it.
>
> Following a large scale BGP incident in March 2015, noction made it
> possible to optionally set the well-known NO_EXPORT community on route
> advertisements originated by IRP instances.
>
> "In order to further reduce the likelihood of these problems
> occurring in the future, we will be adding a feature within Noction
> IRP to give an option to tag all the more specific prefixes that it
> generates with the BGP NO_EXPORT community. This will not be enabled
> by default [snip]"
> https://www.noction.com/blog/route-optimizers
> Mar 27, 2015
>
> Due to NO_EXPORT not being set in the default configuration, there are
> probably if not certainly many unsuspecting network engineers who end up
> deploying this software - without ever even considering - to change that
> one setting in the configuration.
>
> Fast forward a few years and a few incidents, on the topic of default
> settings, following the Cloudflare/DQE/Verizon incident:
>
> "We do have no export community support and have done for many
> years. The use of more specifics is also optional. Neither replaces
> the need for filters."
> https://twitter.com/noction/status/1143177562191011840
> Jun 24, 2019
>
> Community members responded:
>
> "Noction have been facilitating Internet outages for years and
> years and the best thing they can say in response is that it is
> technically possible to use their product responsibly, they just
> don't ship it that way."
> https://twitter.com/PowerDNS_Bert/status/1143252745257979905
> June 24, 2019
>
> Last year Noction stated:
>
> "Nobody found this leak pleasant."
> https://www.noction.com/news/incident-response
> June 26, 2019
>
> Sentiment we all can agree with, change is needed!
>
> As far as I know, Noction IRP is the ONLY commercially available
> off-the-shelf BGP route manipulation software which - as default - does
> NOT set the BGP well-known NO_EXPORT community on the product's route
> advertisements. This is a product design decision which causes
> collateral damage.
>
> I would like to urge Noction to reconsider their position. Seek to
> migrate the existing users to use NO_EXPORT, and release a new version
> of the IRP software which sets NO_EXPORT BY DEFAULT on all generated
> routes.
>
> Kind regards,
>
> Job
>
>

Dear Ryan,

I have come to believe this is a Noction IRP specific issue.

On Sat, Aug 01, 2020 at 01:29:59PM -0700, Ryan Hamel wrote:
> I disagree on the fact that it is not fair to the BGP implementation
> ecosystem, to enforce a single piece of software to activate the
> no-export community by default

I am not exaggerating when I say that *ONLY* the name of this software
is mentioned when incidents like this happen. Other route manipulation
tools either use different (safer) technologies and/or mark routes with
NO_EXPORT.

Every few weeks I am in phone calls with new people who happened
originated hijacks which existed for traffic engineering purposes and
without fail it is always the same software from the same company that
originated the rogue routes.

It seems more efficient if the software were to ship with improved
default settings than me explaining the problem ad-nauseum to every new
engineer after they unsuspectingly stepped into this trap.

Not extremely dangerous by default, is it really too much to ask?

> Also, wasn't it you that said Cisco routers had a bug in ignoring
> NO_EXPORT? Would you go on a rant with Cisco, even if Noction add that
> enabled checkbox by default?

Cisco and Noction are separate companies, regardless of what Noction
does, the Cisco implementations are expected to confirm to their own
documentation and the BGP-4 specifications.

1/ Without setting NO_EXPORT by a default, route manipulation software
by default is very dangerous.

2/ Even if NO_EXPORT is set, software defects happen from time to time
and the existence of fake more-specific routes in a specific routing
domain can have dire consequences (as has been demonstrated time
after time).

Not setting NO_EXPORT as a default is setting your customers up for
failure. If your car's seatbelt accidentally breaks, it wouldn't
logically follow to also remove the airbags.

> Why are you not on your soap box about BIRD, FRrouting, OpenBGPd,
> Cisco, Juniper, etc... about how they can possibly allow every day
> screw ups to happen

It is interesting you mention these names, as all of them in recent
years went through a process to revisit some unsafe default behavior
and address it. These companies have far larger userbases, so if they
can do it, anyone can do it!

For the longest time many BGP implementations - BY DEFAULT - would
propagate any and all routes from EBGP peers to all other IGBP and EBGP
peers. The community identified this to be a root cause for many
incidents, and eventually came up with a change to the BGP-4
specification which codifies that the default should be safe instead of
dangerous. https://tools.ietf.org/html/rfc8212

- BIRD introduced support for RFC 8212 in BIRD 2 and higher
- FRRouting changed the defaults in 7.4 and higher
- Cisco IOS XR had RFC 8212 right from the start
- OpenBGPD changed its default behavior in version 6.4
- Juniper is still working on this, in the meantime a SLAX script can be
used to emulate RFC 8212 behavior: https://github.com/packetsource/rfc8212-junos

It is well understood how default settings strongly shape the success or
failure of deployments. This is no different.

Kind regards,

Job

CSCdj01351. Fixed in 1997.

Regards,
Jakob.

-----Original Message-----
Date: Sat, 1 Aug 2020 13:29:59 -0700
From: Ryan Hamel <ryan@rkhtech.org>

...
Also, wasn't it you that said Cisco routers had a bug in ignoring NO_EXPORT?
...

I was made aware of another bug in IOS-XR: CSCuv94859. Thanks Job and Ryan.
It caused some routes with NO_EXPORT to sometimes be advertised to EBGP
after an NSR switchover during a software upgrade.
It was fixed in 2015.

Regards,
Jakob.

-----Original Message-----
From: Jakob Heitz (jheitz)
Sent: Tuesday, August 4, 2020 10:24 AM
To: nanog@nanog.org
Subject: Re: Issue with Noction IRP default setting (Was: BGP route hijack by AS10990)

CSCdj01351. Fixed in 1997.

Regards,
Jakob.

-----Original Message-----
Date: Sat, 1 Aug 2020 13:29:59 -0700
From: Ryan Hamel <ryan@rkhtech.org>

...
Also, wasn't it you that said Cisco routers had a bug in ignoring NO_EXPORT?
...