I’ve been going through our updates and it looks like another issue is that they (Crown Castle) were being denied access to POP(s) to do repairs. I’ve seen some fiber rings where the go into a downtown area where the full fiber route enters Building A then goes back out to the street then Building B and so on down the line. So the ring is snaking in and out of every building on the route. Incomes a new building owner of several buildings with no real agreement in place and they get into a dispute over payment and then they get pissed and cut you out of the building taking out anyone along the route.
Who knows what’s really going on.
-richey
From: Jason Grider <jgrider@asc.edu>
Date: Friday, April 29, 2022 at 1:26 PM
To: richey goldberg <richey.goldberg@gmail.com>, outages@outages.org <outages@outages.org>
Subject: Re: Cogent Fiber Cut Birmingham?
My understanding is that Crown Castle was doing maintenance. Questions remain about what companies were notified about said maintenance and questions abound? about why it is taking 36 hours now to repair. I know we were not notified by cogent or any other providers.
Our cogent connection is up and has routes but we are located in the same building as Cogent in that case so Cogent just needed to reestablish connections to their router and we got routes again.
I have other down connections that leave the same building that are dark fiber or long haul lit services. The dark fiber connections we have did not drop till 0200 or 0300 local. I was literally logging into a device that I had been in less than 3minutes before when it dropped. The long haul services were cut at around 12 with every thing else. I got a message from a path monitoring system at 00:04 local on Thursday and then a whole bunch of connection disconnects at 01:21 from logs.
The fiber network gremlins are in running rampant in central Alabama the last few days.
As I type this another engineer said cogent just went down again. #*^%*^#%!
Jason Grider
Network Engineer
Alabama Supercomputer Authority
Alabama Research and Education Network
800.338.8320
jgrider@asc.edu
________________________________
From: richey goldberg <richey.goldberg@gmail.com>
Sent: Friday, April 29, 2022 12:00:42 PM
To: Jason Grider <jgrider@asc.edu>; outages@outages.org <outages@outages.org>
Subject: Re: Cogent Fiber Cut Birmingham?
According to our guy who was on call at the time that time seems about right. I’m wondering what’s going on that they would have several cuts on the same ring at the same time that’s taken this long to get repaired. We did hear in the last update that they had engineers from other nearby markets on site as part of the restoration. I’m wondering if this was something that was malicious or just really bad timing.
-richey
From: Jason Grider <jgrider@asc.edu>
Date: Friday, April 29, 2022 at 12:08 PM
To: richey goldberg <richey.goldberg@gmail.com>, outages@outages.org <outages@outages.org>
Subject: Re: Cogent Fiber Cut Birmingham?
Yeah I got you, there are cut(s) in and around the Alabama Power HQ/6th Ave N in downtown that were affecting Cogent. We were still neighbored BGP with Cogent but they were not adverting any routes. Started at about midnight central on Wed night Thursday morning for us.
We do currently have routes in BGP from Cogent. Not sure if that was resolved yesterday or this morning for us.
Jason Grider
Network Engineer – ASA (AS 3464)
Alabama Supercomputer Authority
From: Outages <outages-bounces@outages.org> on behalf of richey goldberg via Outages <outages@outages.org>
Date: Friday, April 29, 2022 at 8:20 AM
To: outages@outages.org <outages@outages.org>
Subject: [outages] Cogent Fiber Cut Birmingham?
Does anyone know about a Cogent fiber cut in Birmingham, AL? We have a customer who’s been down for 30+ hours and Cogent has given us a run around. Their updates have been conflicting. We have someone asking for updates every 4-5 hours and every update is different from the last. Sometimes they tell us it’s fixed (when it’s not) then they tell us it’s being spliced, then they tell us a crew is in route. They are all just random responses. After 30+ hours I can’t imagine what they are doing that they have been on site splicing this long. They can’t tell us where it is other than it’s a a 3rd party vendor they are relying on.
Does anyone have some insight on what’s really going on?
-richey