Mailing List Archive

Hi,

On Sat, Apr 18, 2020 at 06:06:49AM +0000, Bradley Ordner wrote:
> The carrier, keeps blaming my side so I opened a Cisco TAC case
> and they haven't got around to looking at it yet, probably because
> it really sounds like it is the carrier side. I took some packet
> captures and indeed every 60 seconds an update with the default is
> sent. Our router constantly accepts this, recalculates and enters
> it into the routing table.

If you can see the default route being announced by the carrier, it's
(very obviously so) not your router who is causing this (if you see
an outgoing route refresh request, and then an update of *everything*,
things would be different). There is no mechanism in BGP to tell the
other end "of all routes you sent me, please refresh just the default
route".

Do not let carriers bullshit you into believing "it's all your fault".


Now for "why is it sending an update" - it might be attributes changing
(due to something oscillating in the carrier network), or just a dumb
implementation on their side. Try looking at the attributes for two
consecutive updates (metric, communities, ...) and see if something
changes.

(OTOH, except for academic interest, I would not worry too much about
it either - one route recalculation every 60 seconds which leads to the
same result "out to that carrier" will not negatively affect your
network)

gert

--
"If was one thing all people took for granted, was conviction that if you
feed honest figures into a computer, honest figures come out. Never doubted
it myself till I met a computer with a sense of humor."
Robert A. Heinlein, The Moon is a Harsh Mistress

Gert Doering - Munich, Germany gert@greenie.muc.de

Hi Bradley,

From my cisco days I recall that you should not be seeing RIB being updated
over and over with the same route even if BGP keeps sending you implicit
withdraws in the form of new BGP UPDATEs. Of course I will not tell you if
the above is still identical today on all XE, NX & XR :)

However the above only works if your next hop is stable. If you are
recursing your routes over BGP (double recursion) then yes you will see
this churn going on into RIB.

But there is simple fix/test - just set a static route matching next hop of
received BGP prefixes towards your eBGP peer with interface and IP address
and see if it helps.

Cheers,
R.



On Sat, Apr 18, 2020 at 8:10 AM Bradley Ordner <bradinusa@hotmail.com>
wrote:

> I am about to leave an Enterprise environment due to Pre Covid-19
> redundancy and I just need to find the root cause of this issue before I
> leave.
>
> We recently built a Layer 2 Circuit over a Providers SDN Backbone so we
> could get a 2Gb Internet link. We peered with this neighbor and filtered a
> partial table, so we get about 30000 routes. For some reason, every 30
> seconds the default route uptime resets to 00:00 in the routing table. I
> spoke with Carrier, they made a few changes and one was the BGP
> advertisement timer. It is now set to 60 seconds and now the default route
> resets every 60 seconds.
>
> The carrier, keeps blaming my side so I opened a Cisco TAC case and they
> haven't got around to looking at it yet, probably because it really sounds
> like it is the carrier side. I took some packet captures and indeed every
> 60 seconds an update with the default is sent. Our router constantly
> accepts this, recalculates and enters it into the routing table.
>
> I can't seem to figure out if this is some type of bug or not. The router
> has been rebooted and is due for IOS upgrade shortly, but wanted to see if
> anyone has seen this or point me in the right direction.
>
> Thanks
>
> Brad
>
>
> _______________________________________________
> cisco-nsp mailing list cisco-nsp@puck.nether.net
> https://puck.nether.net/mailman/listinfo/cisco-nsp
> archive at http://puck.nether.net/pipermail/cisco-nsp/
>
_______________________________________________
cisco-nsp mailing list cisco-nsp@puck.nether.net
https://puck.nether.net/mailman/listinfo/cisco-nsp
archive at http://puck.nether.net/pipermail/cisco-nsp/

Thanks Robert & Gert. I will go back to carrier as I have checked both changing attributes and my route to peer.

I have tested this in GNS3 with different IOS as well, and everything i stable.

Brad

________________________________
From: Robert Raszuk <robert@raszuk.net>
Sent: Saturday, 18 April 2020 8:26 PM
To: Bradley Ordner <bradinusa@hotmail.com>
Cc: cisco-nsp@puck.nether.net <cisco-nsp@puck.nether.net>
Subject: Re: [c-nsp] Default Route recalculated every 60 seconds.

Hi Bradley,

From my cisco days I recall that you should not be seeing RIB being updated over and over with the same route even if BGP keeps sending you implicit withdraws in the form of new BGP UPDATEs. Of course I will not tell you if the above is still identical today on all XE, NX & XR :)

However the above only works if your next hop is stable. If you are recursing your routes over BGP (double recursion) then yes you will see this churn going on into RIB.

But there is simple fix/test - just set a static route matching next hop of received BGP prefixes towards your eBGP peer with interface and IP address and see if it helps.

Cheers,
R.



On Sat, Apr 18, 2020 at 8:10 AM Bradley Ordner <bradinusa@hotmail.com<mailto:bradinusa@hotmail.com>> wrote:
I am about to leave an Enterprise environment due to Pre Covid-19 redundancy and I just need to find the root cause of this issue before I leave.

We recently built a Layer 2 Circuit over a Providers SDN Backbone so we could get a 2Gb Internet link. We peered with this neighbor and filtered a partial table, so we get about 30000 routes. For some reason, every 30 seconds the default route uptime resets to 00:00 in the routing table. I spoke with Carrier, they made a few changes and one was the BGP advertisement timer. It is now set to 60 seconds and now the default route resets every 60 seconds.

The carrier, keeps blaming my side so I opened a Cisco TAC case and they haven't got around to looking at it yet, probably because it really sounds like it is the carrier side. I took some packet captures and indeed every 60 seconds an update with the default is sent. Our router constantly accepts this, recalculates and enters it into the routing table.

I can't seem to figure out if this is some type of bug or not. The router has been rebooted and is due for IOS upgrade shortly, but wanted to see if anyone has seen this or point me in the right direction.

Thanks

Brad


_______________________________________________
cisco-nsp mailing list cisco-nsp@puck.nether.net<mailto:cisco-nsp@puck.nether.net>
https://puck.nether.net/mailman/listinfo/cisco-nsp
archive at http://puck.nether.net/pipermail/cisco-nsp/
_______________________________________________
cisco-nsp mailing list cisco-nsp@puck.nether.net
https://puck.nether.net/mailman/listinfo/cisco-nsp
archive at http://puck.nether.net/pipermail/cisco-nsp/

On Sat, 18 Apr 2020 at 07:11, Bradley Ordner <bradinusa@hotmail.com> wrote:
>
> I am about to leave an Enterprise environment due to Pre Covid-19 redundancy and I just need to find the root cause of this issue before I leave.
>
> We recently built a Layer 2 Circuit over a Providers SDN Backbone so we could get a 2Gb Internet link. We peered with this neighbor and filtered a partial table, so we get about 30000 routes. For some reason, every 30 seconds the default route uptime resets to 00:00 in the routing table. I spoke with Carrier, they made a few changes and one was the BGP advertisement timer. It is now set to 60 seconds and now the default route resets every 60 seconds.
>
> The carrier, keeps blaming my side so I opened a Cisco TAC case and they haven't got around to looking at it yet, probably because it really sounds like it is the carrier side. I took some packet captures and indeed every 60 seconds an update with the default is sent. Our router constantly accepts this, recalculates and enters it into the routing table.
>
> I can't seem to figure out if this is some type of bug or not. The router has been rebooted and is due for IOS upgrade shortly, but wanted to see if anyone has seen this or point me in the right direction.
>
> Thanks
>
> Brad

Hi Brad,

Do you know if your provider uses IOS-XR on their PE? There was a bug
a couple of years ago in XR (I've searched on cisco.com but can't find
the BugID right now) in which XR was re-advertising the default route
every 30 or 60 seconds. We had it, and if I recall correctly it wasn't
being withdrawn, just a new BGP UPDATE was sent to supersede the
existing route, so as other posters have said, packet capture the BGP
TCP packets with your provider or use some "debug bgp" commands to see
whats really going on.

Also, maybe reconsider if need them to advertise a default route if
you're getting a partial table from them.

Cheers,
James.
_______________________________________________
cisco-nsp mailing list cisco-nsp@puck.nether.net
https://puck.nether.net/mailman/listinfo/cisco-nsp
archive at http://puck.nether.net/pipermail/cisco-nsp/

Hi James,

Interesting you mention this, as someone else on Cisco Community Page was interested in the IOS version. Possibly once this design is finished the default may come from somewhere else, but I don't think I will be there when it happens.

They have told me they have no other issues with other customers and same config, but this could be a bug between different IOS versions because I am running IOS-XE and they may be running XR as they have a ASR9K.

I am going to ask the question and see what happens. Thanks for the info.

Brad



________________________________
From: James Bensley <jwbensley+cisco-nsp@gmail.com>
Sent: Monday, 20 April 2020 6:30 PM
To: Bradley Ordner <bradinusa@hotmail.com>; cisco-nsp@puck.nether.net <cisco-nsp@puck.nether.net>
Subject: Re: [c-nsp] Default Route recalculated every 60 seconds.

On Sat, 18 Apr 2020 at 07:11, Bradley Ordner <bradinusa@hotmail.com> wrote:
>
> I am about to leave an Enterprise environment due to Pre Covid-19 redundancy and I just need to find the root cause of this issue before I leave.
>
> We recently built a Layer 2 Circuit over a Providers SDN Backbone so we could get a 2Gb Internet link. We peered with this neighbor and filtered a partial table, so we get about 30000 routes. For some reason, every 30 seconds the default route uptime resets to 00:00 in the routing table. I spoke with Carrier, they made a few changes and one was the BGP advertisement timer. It is now set to 60 seconds and now the default route resets every 60 seconds.
>
> The carrier, keeps blaming my side so I opened a Cisco TAC case and they haven't got around to looking at it yet, probably because it really sounds like it is the carrier side. I took some packet captures and indeed every 60 seconds an update with the default is sent. Our router constantly accepts this, recalculates and enters it into the routing table.
>
> I can't seem to figure out if this is some type of bug or not. The router has been rebooted and is due for IOS upgrade shortly, but wanted to see if anyone has seen this or point me in the right direction.
>
> Thanks
>
> Brad

Hi Brad,

Do you know if your provider uses IOS-XR on their PE? There was a bug
a couple of years ago in XR (I've searched on cisco.com but can't find
the BugID right now) in which XR was re-advertising the default route
every 30 or 60 seconds. We had it, and if I recall correctly it wasn't
being withdrawn, just a new BGP UPDATE was sent to supersede the
existing route, so as other posters have said, packet capture the BGP
TCP packets with your provider or use some "debug bgp" commands to see
whats really going on.

Also, maybe reconsider if need them to advertise a default route if
you're getting a partial table from them.

Cheers,
James.
_______________________________________________
cisco-nsp mailing list cisco-nsp@puck.nether.net
https://puck.nether.net/mailman/listinfo/cisco-nsp
archive at http://puck.nether.net/pipermail/cisco-nsp/

Hi,

On Mon, Apr 20, 2020 at 09:36:55AM +0000, Bradley Ordner wrote:
> They have told me they have no other issues with other customers and same config, but this could be a bug between different IOS versions because I am running IOS-XE and they may be running XR as they have a ASR9K.

Strictly speaking, there is no "issue", except that the counter for
"how old is the route?" on your side is being reset every minute.

Packet forwarding works, routing is stable, no CPU churn.

WRT "bugs between different IOS versions" - please read what I wrote
before: frequent reannouncements of a single route *can not* be triggered
by anything on your side. There is nothing in the BGP protocol which
would enable this. (If it happens for *all* routes, it could be a
soft reconfig request going awry, but this not what you see)

gert

--
"If was one thing all people took for granted, was conviction that if you
feed honest figures into a computer, honest figures come out. Never doubted
it myself till I met a computer with a sense of humor."
Robert A. Heinlein, The Moon is a Harsh Mistress

Gert Doering - Munich, Germany gert@greenie.muc.de

Thanks Gert, I will now ask them to do packet capture on their side and see if they are advertising this default to any other customer every 60 seconds.

Something else I noticed, we only accept routes less than or equal to /18. I noticed that many updates come in, for different prefixes. I can’t see how the Internet could be that unstable unless there is something wrong with their network. Wonder what is the norm when seeing so many prefixes change.

Brad




Sent from my iPhone

> On 20 Apr 2020, at 7:55 pm, Gert Doering <gert@greenie.muc.de> wrote:
>
> ?Hi,
>
>> On Mon, Apr 20, 2020 at 09:36:55AM +0000, Bradley Ordner wrote:
>> They have told me they have no other issues with other customers and same config, but this could be a bug between different IOS versions because I am running IOS-XE and they may be running XR as they have a ASR9K.
>
> Strictly speaking, there is no "issue", except that the counter for
> "how old is the route?" on your side is being reset every minute.
>
> Packet forwarding works, routing is stable, no CPU churn.
>
> WRT "bugs between different IOS versions" - please read what I wrote
> before: frequent reannouncements of a single route *can not* be triggered
> by anything on your side. There is nothing in the BGP protocol which
> would enable this. (If it happens for *all* routes, it could be a
> soft reconfig request going awry, but this not what you see)
>
> gert
>
> --
> "If was one thing all people took for granted, was conviction that if you
> feed honest figures into a computer, honest figures come out. Never doubted
> it myself till I met a computer with a sense of humor."
> Robert A. Heinlein, The Moon is a Harsh Mistress
>
> Gert Doering - Munich, Germany gert@greenie.muc.de
_______________________________________________
cisco-nsp mailing list cisco-nsp@puck.nether.net
https://puck.nether.net/mailman/listinfo/cisco-nsp
archive at http://puck.nether.net/pipermail/cisco-nsp/

Hi,

On Mon, Apr 20, 2020 at 08:54:27PM +0000, Bradley Ordner wrote:
> Thanks Gert, I will now ask them to do packet capture on their side and see if they are advertising this default to any other customer every 60 seconds.
>
> Something else I noticed, we only accept routes less than or equal to /18. I noticed that many updates come in, for different prefixes. I can???t see how the Internet could be that unstable unless there is something wrong with their network. Wonder what is the norm when seeing so many prefixes change.

The Internet is huge - 70.000 networks(!) connected together. Things
are rebuilt and changed all over the place all the time, and links and
devices fail and get repaired all over the time.

So yes, there's a constant stream of BGP updates.

Google for Geoff Huston. He's done a number of very good presentation
on the dynamics of BGP updates over time.

gert
--
"If was one thing all people took for granted, was conviction that if you
feed honest figures into a computer, honest figures come out. Never doubted
it myself till I met a computer with a sense of humor."
Robert A. Heinlein, The Moon is a Harsh Mistress

Gert Doering - Munich, Germany gert@greenie.muc.de

Just thought I would update this thread, the carrier ended up labbing this for us and found it did not occur in the lab. The only difference was the IOS on the CE router and only receiving a default, not a partial table and default.

I still couldn't believe it, so we updated our router to 16.09.05 and the default is stable.

We were originally on asr1001x-universalk9.16.06.04.SPA.bin.

I will take a packet capture (on the router) again and check if the default update is coming every sixty seconds.

Brad


________________________________
From: Gert Doering
Sent: Tuesday, April 21, 2020 7:10 AM
To: Bradley Ordner
Cc: Gert Doering; James Bensley; cisco-nsp@puck.nether.net
Subject: Re: [c-nsp] Default Route recalculated every 60 seconds.

Hi,

On Mon, Apr 20, 2020 at 08:54:27PM +0000, Bradley Ordner wrote:
> Thanks Gert, I will now ask them to do packet capture on their side and see if they are advertising this default to any other customer every 60 seconds.
>
> Something else I noticed, we only accept routes less than or equal to /18. I noticed that many updates come in, for different prefixes. I can???t see how the Internet could be that unstable unless there is something wrong with their network. Wonder what is the norm when seeing so many prefixes change.

The Internet is huge - 70.000 networks(!) connected together. Things
are rebuilt and changed all over the place all the time, and links and
devices fail and get repaired all over the time.

So yes, there's a constant stream of BGP updates.

Google for Geoff Huston. He's done a number of very good presentation
on the dynamics of BGP updates over time.

gert
--
"If was one thing all people took for granted, was conviction that if you
feed honest figures into a computer, honest figures come out. Never doubted
it myself till I met a computer with a sense of humor."
Robert A. Heinlein, The Moon is a Harsh Mistress

Gert Doering - Munich, Germany gert@greenie.muc.de
_______________________________________________
cisco-nsp mailing list cisco-nsp@puck.nether.net
https://puck.nether.net/mailman/listinfo/cisco-nsp
archive at http://puck.nether.net/pipermail/cisco-nsp/