Mailing List Archive

On Mon, 7 Jun 2021 at 19:32, <shahrooz@cs.umass.edu> wrote:

> We often read that the Internet (i.e. BGP) has a long convergence delay.
> But why is it so slow? And can we (researchers) do anything about it?

Create business incentives to improve it. This is a non-technical
problem, we've long had technical tools to make it fast, there just
isn't incentive to make it fast. Customers are not asking operators
for better convergence speeds.

> Please help us out to find out by answering our short anonymous survey
> (<10 minutes).

Can you tell me what have you done so far? What are the default MRAI
values for each AFI/SAFI for IOS, IOS-XR, Junos, SROS, VRP and EOS?
Then people responding don't have to check what their NOS does, they
can refer to your table and tell the default value, since this is what
>99% will be using.

Now your survey has built-in selection bias, people who answer it are
people who know what it is and who are concerned about it and have
changed it, this is not a representative group and you will start your
work with very bad data.



--
++ytti

+1 to Saku's concerns - I simply ignored the survey because I wasn't sure what MRAI was, and I wasn't sure what my values would be. But I have time to be interested right now, so a-spelunking I go...

The term "MRAI" does not appear anywhere in Arista's or Extreme's documentation. Nor does this timer interval appear in any BGP-related "show" output, on any of my platforms, that I can see.

I've found out that "out-delay" (not "delay out") is synonymous with MRAI and seems widely used.

I found "out-delay" in an Arista technote, and now I know how to override it on Aristas, and the default is zero (0). Unfortunately, "show" commands on EOS will only show you the current out-delay if it is non-zero, which makes reporting it a bit difficult.

Extreme's MLXe platform doesn't appear to support an out-delay/MRAI knob at all, at least as far as I can tell. I know there are several other current and former MLXe operators here, maybe one of them will know?

Based on my limited history with NANOG-L, I guess your initial email might have been seen by perhaps 20 people who immediately knew what you were talking about, and 2000 who didn't. (I don't actually know subscriber numbers, that's totally a WAG. And maybe more people have touched out-delay knobs than I think. Dunno.)

<snide but hopefully not too much> This illustrates the gap between academia and industry - academics can research a narrow topic, and come at issues from a theoretic standpoint using unfamiliar terminology. As a practitioner, I get told to add carrier X as a peer by end-of-day Friday, and "just make it work". I wasn't even able to understand your initial question, because I haven't spent a semester understanding the intricacies of BGP propagation - I just know it usually works, knobs exist that I shouldn't fiddle with, and that's good enough for my job. </snide>

If your work results in actionable recommendations such as "don't use BGP out-delay timers to mitigate XYZ in circumstance LMNO, do ABC instead", that's fantastic. Please keep us advised, and do post aggregated survey results here once you close the survey.

I am specifically interested in the answer to "Have you ever had to adjust BGP out-delay with any of your peers, and why?" It would be great if we could derive that answer from the survey results, but anecdotal replies here would also be helpful. All you larger(-than-me) network operators out there: when would I need to use out-delay? Why? What does it accomplish?

Good luck in reformulating your survey to get better engagement,
-Adam

Adam Thompson
Consultant, Infrastructure Services
[1593169877849]
100 - 135 Innovation Drive
Winnipeg, MB, R3T 6A8
(204) 977-6824 or 1-800-430-6404 (MB only)
athompson@merlin.mb.ca<mailto:athompson@merlin.mb.ca>
www.merlin.mb.ca<http://www.merlin.mb.ca/>

________________________________
From: NANOG <nanog-bounces+athompson=merlin.mb.ca@nanog.org> on behalf of Saku Ytti <saku@ytti.fi>
Sent: June 8, 2021 01:06
To: shahrooz@cs.umass.edu <shahrooz@cs.umass.edu>
Cc: nanog list <nanog@nanog.org>; Arun Venkataramani <arun@cs.umass.edu>
Subject: Re: A survey on BGP MRAI timer values in practice

On Mon, 7 Jun 2021 at 19:32, <shahrooz@cs.umass.edu> wrote:

> We often read that the Internet (i.e. BGP) has a long convergence delay.
> But why is it so slow? And can we (researchers) do anything about it?

Create business incentives to improve it. This is a non-technical
problem, we've long had technical tools to make it fast, there just
isn't incentive to make it fast. Customers are not asking operators
for better convergence speeds.

> Please help us out to find out by answering our short anonymous survey
> (<10 minutes).

Can you tell me what have you done so far? What are the default MRAI
values for each AFI/SAFI for IOS, IOS-XR, Junos, SROS, VRP and EOS?
Then people responding don't have to check what their NOS does, they
can refer to your table and tell the default value, since this is what
>99% will be using.

Now your survey has built-in selection bias, people who answer it are
people who know what it is and who are concerned about it and have
changed it, this is not a representative group and you will start your
work with very bad data.



--
++ytti

On Wed, 9 Jun 2021 at 01:18, Adam Thompson <athompson@merlin.mb.ca> wrote:

If your work results in actionable recommendations such as "don't use BGP
> out-delay timers to mitigate XYZ in circumstance LMNO, do ABC instead",
> that's fantastic. Please keep us advised, and do post aggregated survey
> results here once you close the survey.
>

What is actionable? What is the goal? The question as OP presented contains
some assumptions

a) better convergence is needed
b) MRAI is important part of the solution space

Neither are provable. We already know how to make DFZ convergence really
fast (or at least orders of magnitude faster than it is), that information
exists, but that isn't deployed because customers are not asking for it, so
providers are not aware that there is room for improvements.

Things don't optimise to be as good as they can be, things optimise to be
as bad as the market allows them to be. And the market accepts the DFZ
convergence.

If you do decide to optimise for DFZ convergence, without commercial
pressure, you will risk lower availability, because you'll be using
configuration less tested by other customers and everyone knows how
terrible quality every NOS is. Everyone finds novel bugs, in the same damn
protocols we've ran +20 years. It's like running Windows and Linux and
regularly finding out listing files in a directory breaks your service,
year after year after year.

For those who are interested in better convergence
- change your interface down reporting to 0 (there may be delay before
interface down is reported to system, so that optical protection works
without causing outage)
- use 'add-path' or at least 'best-external' in iBGP, so that you always
have backup eBGP route immediately available once best is invalidated
(normally you have lot of delay to find next best, once you lose your best
eBGP)
- tie your route validity to IGP, so you can invalidate your BGP the
moment IGP disappears
- ensure IGP converges fast (another topic)
- set MRAI to 0
- use PIC edge
- ensure your BGP NLRI can be as large as MTU allows
- ensure your convergence isn't bottle necked by slow peer in group
- ensure you are not dropping received TCP packets on punt path
- ensure your fast external fallover works (eBGP down, on int down) this
is quite easy to break
- then ensure everyone else in the DFZ does the same thing



But from a business POV, don't do any of this, you will have more bugs and
lower availability and your customers will be less happy.



> I *am *specifically interested in the answer to "Have you ever had to
> adjust BGP out-delay with any of your peers, and why?" It would be great
> if we could derive that answer from the survey results, but anecdotal
> replies here would also be helpful. All you larger(-than-me) network
> operators out there: when would I need to use out-delay? Why? What does
> it accomplish?
>
> Good luck in reformulating your survey to get better engagement,
> -Adam
>
> *Adam Thompson*
> Consultant, Infrastructure Services
> [image: 1593169877849]
> 100 - 135 Innovation Drive
> Winnipeg, MB, R3T 6A8
> (204) 977-6824 or 1-800-430-6404 (MB only)
> athompson@merlin.mb.ca
> www.merlin.mb.ca
>
> ------------------------------
> *From:* NANOG <nanog-bounces+athompson=merlin.mb.ca@nanog.org> on behalf
> of Saku Ytti <saku@ytti.fi>
> *Sent:* June 8, 2021 01:06
> *To:* shahrooz@cs.umass.edu <shahrooz@cs.umass.edu>
> *Cc:* nanog list <nanog@nanog.org>; Arun Venkataramani <arun@cs.umass.edu>
> *Subject:* Re: A survey on BGP MRAI timer values in practice
>
> On Mon, 7 Jun 2021 at 19:32, <shahrooz@cs.umass.edu> wrote:
>
> > We often read that the Internet (i.e. BGP) has a long convergence delay.
> > But why is it so slow? And can we (researchers) do anything about it?
>
> Create business incentives to improve it. This is a non-technical
> problem, we've long had technical tools to make it fast, there just
> isn't incentive to make it fast. Customers are not asking operators
> for better convergence speeds.
>
> > Please help us out to find out by answering our short anonymous survey
> > (<10 minutes).
>
> Can you tell me what have you done so far? What are the default MRAI
> values for each AFI/SAFI for IOS, IOS-XR, Junos, SROS, VRP and EOS?
> Then people responding don't have to check what their NOS does, they
> can refer to your table and tell the default value, since this is what
> >99% will be using.
>
> Now your survey has built-in selection bias, people who answer it are
> people who know what it is and who are concerned about it and have
> changed it, this is not a representative group and you will start your
> work with very bad data.
>
>
>
> --
> ++ytti
>


--
++ytti

In Cisco, MRAI is "advertisement-interval".
MRAI helps to reduce route update multiplication in highly redundant
networks. OTOH, it can increase the time it takes to re-advertise
a complete internet table in some router implementations.
Update multiplication due to redundant network connections causes
some receivers of the multiple updates to become slow peers.

Here's an experiment: Do something to cause a BGP route refresh, like
the equivalent of "clear bgp soft out". It will not change any routes.
It just resends everything that was already sent. See how long it takes
with MRAI=0. Then set MRAI to about half of that value and do the
refresh again. If it takes substantially longer to complete the refresh,
stick with MRAI=0.
If there is no significant difference, use MRAI of 1 or 2 seconds.

Regards,
Jakob.

-----Original Message-----
Date: Wed, 9 Jun 2021 08:53:19 +0300
From: Saku Ytti <saku@ytti.fi>

On Wed, 9 Jun 2021 at 01:18, Adam Thompson <athompson@merlin.mb.ca> wrote:

If your work results in actionable recommendations such as "don't use BGP
> out-delay timers to mitigate XYZ in circumstance LMNO, do ABC instead",
> that's fantastic. Please keep us advised, and do post aggregated survey
> results here once you close the survey.
>

What is actionable? What is the goal? The question as OP presented contains
some assumptions

a) better convergence is needed
b) MRAI is important part of the solution space

Neither are provable. We already know how to make DFZ convergence really
fast (or at least orders of magnitude faster than it is), that information
exists, but that isn't deployed because customers are not asking for it, so
providers are not aware that there is room for improvements.

Things don't optimise to be as good as they can be, things optimise to be
as bad as the market allows them to be. And the market accepts the DFZ
convergence.

If you do decide to optimise for DFZ convergence, without commercial
pressure, you will risk lower availability, because you'll be using
configuration less tested by other customers and everyone knows how
terrible quality every NOS is. Everyone finds novel bugs, in the same damn
protocols we've ran +20 years. It's like running Windows and Linux and
regularly finding out listing files in a directory breaks your service,
year after year after year.

For those who are interested in better convergence
- change your interface down reporting to 0 (there may be delay before
interface down is reported to system, so that optical protection works
without causing outage)
- use 'add-path' or at least 'best-external' in iBGP, so that you always
have backup eBGP route immediately available once best is invalidated
(normally you have lot of delay to find next best, once you lose your best
eBGP)
- tie your route validity to IGP, so you can invalidate your BGP the
moment IGP disappears
- ensure IGP converges fast (another topic)
- set MRAI to 0
- use PIC edge
- ensure your BGP NLRI can be as large as MTU allows
- ensure your convergence isn't bottle necked by slow peer in group
- ensure you are not dropping received TCP packets on punt path
- ensure your fast external fallover works (eBGP down, on int down) this
is quite easy to break
- then ensure everyone else in the DFZ does the same thing



But from a business POV, don't do any of this, you will have more bugs and
lower availability and your customers will be less happy.



> I *am *specifically interested in the answer to "Have you ever had to
> adjust BGP out-delay with any of your peers, and why?" It would be great
> if we could derive that answer from the survey results, but anecdotal
> replies here would also be helpful. All you larger(-than-me) network
> operators out there: when would I need to use out-delay? Why? What does
> it accomplish?
>
> Good luck in reformulating your survey to get better engagement,
> -Adam
>
> *Adam Thompson*
> Consultant, Infrastructure Services
> [image: 1593169877849]
> 100 - 135 Innovation Drive
> Winnipeg, MB, R3T 6A8
> (204) 977-6824 or 1-800-430-6404 (MB only)
> athompson@merlin.mb.ca
> www.merlin.mb.ca
>
> ------------------------------
> *From:* NANOG <nanog-bounces+athompson=merlin.mb.ca@nanog.org> on behalf
> of Saku Ytti <saku@ytti.fi>
> *Sent:* June 8, 2021 01:06
> *To:* shahrooz@cs.umass.edu <shahrooz@cs.umass.edu>
> *Cc:* nanog list <nanog@nanog.org>; Arun Venkataramani <arun@cs.umass.edu>
> *Subject:* Re: A survey on BGP MRAI timer values in practice
>
> On Mon, 7 Jun 2021 at 19:32, <shahrooz@cs.umass.edu> wrote:
>
> > We often read that the Internet (i.e. BGP) has a long convergence delay.
> > But why is it so slow? And can we (researchers) do anything about it?
>
> Create business incentives to improve it. This is a non-technical
> problem, we've long had technical tools to make it fast, there just
> isn't incentive to make it fast. Customers are not asking operators
> for better convergence speeds.
>
> > Please help us out to find out by answering our short anonymous survey
> > (<10 minutes).
>
> Can you tell me what have you done so far? What are the default MRAI
> values for each AFI/SAFI for IOS, IOS-XR, Junos, SROS, VRP and EOS?
> Then people responding don't have to check what their NOS does, they
> can refer to your table and tell the default value, since this is what
> >99% will be using.
>
> Now your survey has built-in selection bias, people who answer it are
> people who know what it is and who are concerned about it and have
> changed it, this is not a representative group and you will start your
> work with very bad data.
>
>
>
> --
> ++ytti
>


--
++ytti

> We already know how to make DFZ convergence really fast (or at least
> orders of magnitude faster than it is), that information exists, but
> that isn't deployed because customers are not asking for it, so
> providers are not aware that there is room for improvements.

are we confident that in the global context, not just within an isp,
there is negligible, well acceptable, oscillation?

randy

---
randy@psg.com
`gpg --locate-external-keys --auto-key-locate wkd randy@psg.com`
signatures are back, thanks to dmarc header butchery

On Wed, 9 Jun 2021 at 20:43, Randy Bush <randy@psg.com> wrote:

> are we confident that in the global context, not just within an isp,
> there is negligible, well acceptable, oscillation?

I don't understand the question, but the way I read the question it
may be unanswerable even if I did understand it. As the reader would
self-define negligible and well acceptable and answer yes/no based on
the definition they used, which might be different to the definition
writer intended.

--
++ytti

On 6/10/21 08:26, Saku Ytti wrote:

> I don't understand the question, but the way I read the question it
> may be unanswerable even if I did understand it. As the reader would
> self-define negligible and well acceptable and answer yes/no based on
> the definition they used, which might be different to the definition
> writer intended.

It's possible we've become accustom to a slow, global BGP, due to a
perception of fragility (and complexity), which favours stability over
speed.

I suppose the size of the current BGP and the nature of the FSM it lends
itself to does some to account for those perceptions.

At a per-ISP level, it is not impossible to speed up (i)BGP convergence.
On a global scale, taking the least common denominator to allow for all
manner of network we don't know about, allowing the ship a wide turn in
BGP waters, at least on a perceptive level, seems like an unsigned
social agreement amongst autonomous systems.

Ultimately, I feel we aren't talking enough about this, and hopefully,
this thread gets us to that point.

Mark.

My question at this point is, what slow global convergence? When I (or any of my downstreams) adjusts a prefix, I nearly always see global propagation in well under 60 seconds. Among networks where I can check, at least.
I understand it could be technically possible to see near-instantaneous global convergence in more like <5sec, but... on a global scale, neither I nor my customers care about the difference between 5s and 60s. Do other people need <60s propagation?
-Adam
________________________________
From: NANOG <nanog-bounces+athompson=merlin.mb.ca@nanog.org> on behalf of Mark Tinka <mark@tinka.africa>
Sent: June 10, 2021 02:36
To: nanog@nanog.org <nanog@nanog.org>
Subject: Re: A survey on BGP MRAI timer values in practice



On 6/10/21 08:26, Saku Ytti wrote:

> I don't understand the question, but the way I read the question it
> may be unanswerable even if I did understand it. As the reader would
> self-define negligible and well acceptable and answer yes/no based on
> the definition they used, which might be different to the definition
> writer intended.

It's possible we've become accustom to a slow, global BGP, due to a
perception of fragility (and complexity), which favours stability over
speed.

I suppose the size of the current BGP and the nature of the FSM it lends
itself to does some to account for those perceptions.

At a per-ISP level, it is not impossible to speed up (i)BGP convergence.
On a global scale, taking the least common denominator to allow for all
manner of network we don't know about, allowing the ship a wide turn in
BGP waters, at least on a perceptive level, seems like an unsigned
social agreement amongst autonomous systems.

Ultimately, I feel we aren't talking enough about this, and hopefully,
this thread gets us to that point.

Mark.