Mailing List Archive

Confirmed Level3 carrier issue from the Orlando region --- possibly my other data centers as well (haven't checked them yet, been focusing on my ORL-FL outage).

Odd part --- I disconnected my Level3 circuit at the ORL-FL facility so that I could fully failover to my other carrier link.
Level3 is still advertising my routes, even though my link and BGP session is down.

er-02.orl-fl> show interfaces descriptions | match Level3
xe-2/0/0 down down LEVEL3 |


er-02.orl-fl> show bgp summary | match 3356
67.30.142.81 3356 0 0 0 0 27:43 Active
2001:1900:2100::26ad 3356 0 0 0 0 27:43 Idle


BGP routing table entry for 209.208.0.0/17
Versions:
Process bRIB/RIB SendTblVer
Speaker 86180864 86180864
Last Modified: May 16 05:47:52.095 for 15w1d
Paths: (25 available, best #25)
Advertised IPv4 Unicast paths to update-groups (with more than one peer):
0.1 0.2 0.8 0.9 0.10
Path #1: Received by speaker 0
Not advertised to any peer
3356 6364
129.250.8.174 (metric 9340) from 129.250.0.2 (129.250.0.2)
Origin IGP, metric 4294967294, localpref 100, valid, confed-internal
Received Path ID 0, Local Path ID 0, version 0
Community: 2914:390 2914:1003 2914:2000 2914:3000 65504:3356
Path #2: Received by speaker 0
Not advertised to any peer
3356 6364
129.250.9.50 (metric 24142) from 129.250.0.6 (129.250.0.6)
Origin IGP, metric 4294967294, localpref 100, valid, confed-internal
Received Path ID 0, Local Path ID 0, version 0
Community: 2914:390 2914:1214 2914:2213 2914:3200 65504:3356
Path #3: Received by speaker 0
Not advertised to any peer
174 6364
130.117.15.89 (metric 24174) from 129.250.0.13 (129.250.0.13)
Origin IGP, metric 4294967294, localpref 100, valid, confed-internal
Received Path ID 0, Local Path ID 0, version 0
Community: 2914:390 2914:1204 2914:2205 2914:3200 65504:174
Path #4: Received by speaker 0



Regards,

Stephen Flynn
Atlantic.Net
Direct: (321) 206-1390
sflynn@staff.atlantic.net
www.atlantic.net


-----Original Message-----
From: Outages <outages-bounces@outages.org> On Behalf Of randal k via Outages
Sent: Sunday, August 30, 2020 7:10 AM
To: outages@outages.org
Subject: [outages] CenturyLink peering issues?


CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you recognize the sender and know the content is safe.


We have been seeing serious issues with Level3/CenturyLink services over the past hour or so. Everything on-net seems OK, but transit traffic looks to have problems. Thoughts?
_______________________________________________
Outages mailing list
Outages@outages.org
https://puck.nether.net/mailman/listinfo/outages
_______________________________________________
Outages mailing list
Outages@outages.org
https://puck.nether.net/mailman/listinfo/outages

Some of our customers cannot reach our website in LA from Virginia Beach,
and connections are quite intermittent. Not sure related, but experienced
small annoying network issue around 2PM PDT from our network and got to this
point

-----Original Message-----
From: Outages [mailto:outages-bounces@outages.org] On Behalf Of randal k via
Outages
Sent: Sunday, August 30, 2020 4:10 AM
To: outages@outages.org
Subject: [outages] CenturyLink peering issues?

We have been seeing serious issues with Level3/CenturyLink services over the
past hour or so. Everything on-net seems OK, but transit traffic looks to
have problems. Thoughts?
_______________________________________________
Outages mailing list
Outages@outages.org
https://puck.nether.net/mailman/listinfo/outages

_______________________________________________
Outages mailing list
Outages@outages.org
https://puck.nether.net/mailman/listinfo/outages

Once upon a time, Stephen Flynn via Outages <outages@outages.org> said:
> Odd part --- I disconnected my Level3 circuit at the ORL-FL facility so that I could fully failover to my other carrier link.
> Level3 is still advertising my routes, even though my link and BGP session is down.

I can confirm this - I shut down IPv4 BGP with Level3 in Chicago, but
checking route-views and such, they're still advertising our routes (but
don't know how to get to us once packets hit their network)..

AS7007 all over again?
--
Chris Adams <cma@cmadams.net>
_______________________________________________
Outages mailing list
Outages@outages.org
https://puck.nether.net/mailman/listinfo/outages

It's weird you say that - I have been attempting to use Level3's well
known communities to attempt to prepend, no-export etc and have been
having limited luck ... and those communities even appear in
route-views, so I know they're being sent!

On Sun, Aug 30, 2020 at 6:10 AM Chris Adams via Outages
<outages@outages.org> wrote:
>
> Once upon a time, Stephen Flynn via Outages <outages@outages.org> said:
> > Odd part --- I disconnected my Level3 circuit at the ORL-FL facility so that I could fully failover to my other carrier link.
> > Level3 is still advertising my routes, even though my link and BGP session is down.
>
> I can confirm this - I shut down IPv4 BGP with Level3 in Chicago, but
> checking route-views and such, they're still advertising our routes (but
> don't know how to get to us once packets hit their network)..
>
> AS7007 all over again?
> --
> Chris Adams <cma@cmadams.net>
> _______________________________________________
> Outages mailing list
> Outages@outages.org
> https://puck.nether.net/mailman/listinfo/outages
_______________________________________________
Outages mailing list
Outages@outages.org
https://puck.nether.net/mailman/listinfo/outages

Ugh; seeing same thing. Have had sessions turned down for over two hours and looking glasses are still showing 3356 propagating the advertisements.

?On 8/30/20, 8:11 AM, "Outages on behalf of Chris Adams via Outages" <outages-bounces@outages.org on behalf of outages@outages.org> wrote:

Once upon a time, Stephen Flynn via Outages <outages@outages.org> said:
> Odd part --- I disconnected my Level3 circuit at the ORL-FL facility so that I could fully failover to my other carrier link.
> Level3 is still advertising my routes, even though my link and BGP session is down.

I can confirm this - I shut down IPv4 BGP with Level3 in Chicago, but
checking route-views and such, they're still advertising our routes (but
don't know how to get to us once packets hit their network)..

AS7007 all over again?
--
Chris Adams <cma@cmadams.net>
_______________________________________________
Outages mailing list
Outages@outages.org
https://puck.nether.net/mailman/listinfo/outages

_______________________________________________
Outages mailing list
Outages@outages.org
https://puck.nether.net/mailman/listinfo/outages

Once upon a time, randal k <rkohutek+outages@gmail.com> said:
> It's weird you say that - I have been attempting to use Level3's well
> known communities to attempt to prepend, no-export etc and have been
> having limited luck ... and those communities even appear in
> route-views, so I know they're being sent!

I brought my session back up, not accepting any routes, and prepending
ours (just in case the change propagated)... it took a while, but
eventually I do see the prepended routes.

But for a provider that said they shut down their link to Level3, I
still see their routes... possibly (slowly) propagating changes but not
actual withdraws?
--
Chris Adams <cma@cmadams.net>
_______________________________________________
Outages mailing list
Outages@outages.org
https://puck.nether.net/mailman/listinfo/outages

Just attempted a prepend (x4) on my Level3 advertisements --- only noticed a change within the NTT network tables.
Other large carriers received no advertisement changes (Telia, Hurricane Electric, AT&T)

I then noticed that my BGP session was constantly flapping. I've now shutdown my circuit again. Level3 is still announcing my address space.
Thank you Level3!


Regards,

Stephen Flynn
Atlantic.Net
Direct: (321) 206-1390
sflynn@staff.atlantic.net
www.atlantic.net


-----Original Message-----
From: Outages <outages-bounces@outages.org> On Behalf Of Chris Adams via Outages
Sent: Sunday, August 30, 2020 8:48 AM
To: outages@outages.org
Subject: Re: [outages] CenturyLink peering issues?


CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you recognize the sender and know the content is safe.


Once upon a time, randal k <rkohutek+outages@gmail.com> said:
> It's weird you say that - I have been attempting to use Level3's well
> known communities to attempt to prepend, no-export etc and have been
> having limited luck ... and those communities even appear in
> route-views, so I know they're being sent!

I brought my session back up, not accepting any routes, and prepending ours (just in case the change propagated)... it took a while, but eventually I do see the prepended routes.

But for a provider that said they shut down their link to Level3, I still see their routes... possibly (slowly) propagating changes but not actual withdraws?
--
Chris Adams <cma@cmadams.net>
_______________________________________________
Outages mailing list
Outages@outages.org
https://puck.nether.net/mailman/listinfo/outages
_______________________________________________
Outages mailing list
Outages@outages.org
https://puck.nether.net/mailman/listinfo/outages

Telia and Hurricane Electric are now showing cleared routing tables and sending my traffic to the correct carrier circuits.
My L3 circuit is still admin down --- it will remain this way for the remainder of the day and until I get an update from CenturyLink that the issue is resolved.



Regards,

Stephen Flynn
Atlantic.Net
Direct: (321) 206-1390
sflynn@staff.atlantic.net
www.atlantic.net


-----Original Message-----
From: Outages <outages-bounces@outages.org> On Behalf Of Stephen Flynn via Outages
Sent: Sunday, August 30, 2020 9:33 AM
To: outages@outages.org
Subject: Re: [outages] CenturyLink peering issues?


CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you recognize the sender and know the content is safe.


Just attempted a prepend (x4) on my Level3 advertisements --- only noticed a change within the NTT network tables.
Other large carriers received no advertisement changes (Telia, Hurricane Electric, AT&T)

I then noticed that my BGP session was constantly flapping. I've now shutdown my circuit again. Level3 is still announcing my address space.
Thank you Level3!


Regards,

Stephen Flynn
Atlantic.Net
Direct: (321) 206-1390
sflynn@staff.atlantic.net
www.atlantic.net


-----Original Message-----
From: Outages <outages-bounces@outages.org> On Behalf Of Chris Adams via Outages
Sent: Sunday, August 30, 2020 8:48 AM
To: outages@outages.org
Subject: Re: [outages] CenturyLink peering issues?


CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you recognize the sender and know the content is safe.


Once upon a time, randal k <rkohutek+outages@gmail.com> said:
> It's weird you say that - I have been attempting to use Level3's well
> known communities to attempt to prepend, no-export etc and have been
> having limited luck ... and those communities even appear in
> route-views, so I know they're being sent!

I brought my session back up, not accepting any routes, and prepending ours (just in case the change propagated)... it took a while, but eventually I do see the prepended routes.

But for a provider that said they shut down their link to Level3, I still see their routes... possibly (slowly) propagating changes but not actual withdraws?
--
Chris Adams <cma@cmadams.net>
_______________________________________________
Outages mailing list
Outages@outages.org
https://puck.nether.net/mailman/listinfo/outages
_______________________________________________
Outages mailing list
Outages@outages.org
https://puck.nether.net/mailman/listinfo/outages
_______________________________________________
Outages mailing list
Outages@outages.org
https://puck.nether.net/mailman/listinfo/outages