Mailing List Archive

Yes there are impacts to services.

On Fri, Dec 25, 2020 at 2:16 PM Kurt Duncan via Outages <outages@outages.org>
wrote:

> Anyone else seeing phone or internet issues due to the AT&T building being
> damaged by the explosion in Nashville ?
>
> Thank God for the first responders working today who protect us.
> --
> Kurt Duncan
> _______________________________________________
> Outages mailing list
> Outages@outages.org
> https://puck.nether.net/mailman/listinfo/outages
>
--
Anthony Concepcion

Haven?t heard anything from my end on direct technical problems as a result of the explosion, but after any major event like this it?s highly likely that voice and data lines will be overloaded with traffic from nearby residents

Sent from Outlook<https://aka.ms/qtex0l> for iOS
________________________________
From: Outages <outages-bounces@outages.org> on behalf of Kurt Duncan via Outages <outages@outages.org>
Sent: Friday, December 25, 2020 2:09:15 PM
To: outages@outages.org <outages@outages.org>
Subject: [outages] Explosion in Nashville

Anyone else seeing phone or internet issues due to the AT&T building being damaged by the explosion in Nashville ?

Thank God for the first responders working today who protect us.
--
Kurt Duncan

Crossposting over to -discussion.

News reports indicate damage to the AT&T building is affecting cell coverage. But it wouldn’t surprise me if it was a/the target, given the timing which seems calculated to minimize human damage.

If you haven’t seen it, here’s footage from 154 2nd Ave. N.

https://www.youtube.com/watch?v=5nygTJeu9fU

In the video, the AT&T building is on the right edge on the other side of the street from the security camera.

----
Andy Ringsmuth
5609 Harding Drive
Lincoln, NE 68521-5831
(402) 304-0083
andy@andyring.com

“Better even die free, than to live slaves.” - Frederick Douglas, 1863

> On Dec 25, 2020, at 1:09 PM, Kurt Duncan via Outages <outages@outages.org> wrote:
>
> Anyone else seeing phone or internet issues due to the AT&T building being damaged by the explosion in Nashville ?
>
> Thank God for the first responders working today who protect us.
> --
> Kurt Duncan
> _______________________________________________
> Outages mailing list
> Outages@outages.org
> https://puck.nether.net/mailman/listinfo/outages

_______________________________________________
Outages mailing list
Outages@outages.org
https://puck.nether.net/mailman/listinfo/outages

I live in Nashville, local media is reporting issues with 911 service in
some areas, as well as problems with making/receiving calls and text
messages on the AT&T network. It's also affecting DirecTV and ATT U-verse

On Fri, Dec 25, 2020 at 1:22 PM William Peebles via Outages <
outages@outages.org> wrote:

> Haven’t heard anything from my end on direct technical problems as a
> result of the explosion, but after any major event like this it’s highly
> likely that voice and data lines will be overloaded with traffic from
> nearby residents
>
> Sent from Outlook <https://aka.ms/qtex0l> for iOS
> ------------------------------
> *From:* Outages <outages-bounces@outages.org> on behalf of Kurt Duncan
> via Outages <outages@outages.org>
> *Sent:* Friday, December 25, 2020 2:09:15 PM
> *To:* outages@outages.org <outages@outages.org>
> *Subject:* [outages] Explosion in Nashville
>
> Anyone else seeing phone or internet issues due to the AT&T building being
> damaged by the explosion in Nashville ?
>
> Thank God for the first responders working today who protect us.
> --
> Kurt Duncan
> _______________________________________________
> Outages mailing list
> Outages@outages.org
> https://puck.nether.net/mailman/listinfo/outages
>

Word we got from our fiber provider there is
that the entire building (ATT) is on battery backup as
the generators either did not come online or were
damaged in the blast. They told us to expect outages
as battery backups start to fail.

--
Larry Smith
lesmith@ecsis.net

On Fri December 25 2020 13:20, William Peebles via Outages wrote:
> Haven’t heard anything from my end on direct technical problems as a result
> of the explosion, but after any major event like this it’s highly likely
> that voice and data lines will be overloaded with traffic from nearby
> residents
>
> Sent from Outlook<https://aka.ms/qtex0l> for iOS
> ________________________________
> From: Outages <outages-bounces@outages.org> on behalf of Kurt Duncan via
> Outages <outages@outages.org> Sent: Friday, December 25, 2020 2:09:15 PM
> To: outages@outages.org <outages@outages.org>
> Subject: [outages] Explosion in Nashville
>
> Anyone else seeing phone or internet issues due to the AT&T building being
> damaged by the explosion in Nashville ?
>
> Thank God for the first responders working today who protect us.
> --
> Kurt Duncan
_______________________________________________
Outages mailing list
Outages@outages.org
https://puck.nether.net/mailman/listinfo/outages

We?ve seen a site we have on ATT in Nashville up and down all morning since the blast. It just went back down.

Confirming issues exist on ATT Dedicated Fiber.

T. Geoff Turner
Chief Executive Officer | Direct 971-245-5001
844.433.5247 | help@elevatetechgroup.com<mailto:help@elevatetechgroup.com> | Chat<https://remote.elevatecloud.co/api/start_session.ns?c2cjs=1&issue_menu=1>
[cid:b1872265-5d7e-4813-91de-2bb758f874d4.png]<https://elevatetechgroup.com/>
Respect || Simplicity || Empowerment || Commitment || Integrity

________________________________

[cid:843c9d1b-bec1-4c2c-83c2-87bd86851f64.png]<https://www.facebook.com/ElevateTechGroup/> [cid:adfb377b-6d48-416c-addb-ad0d94192877.png] <https://www.linkedin.com/company/elevatetechgroup> [cid:69a97524-88a1-4ccf-bc47-5f71b417acf8.png] [cid:a858ce4f-e687-4f1d-8924-2e116758259f.png] [cid:8c2c4e43-48bb-47a5-963e-0a869c155052.png] <https://landingpages.elevatecloud.co/CustomerFeedback?rating=4&source=emailsig&email=lesmith@ecsis.net;outages@outages.org&fromEmail=gturner@elevatetechgroup.com&fromFirstName=Thomas&fromLastName=Turner&messageSubject=Re:%20[outages]%20Explosion%20in%20Nashville> [cid:117e8f57-5384-4048-baf6-1c4e31d32c0d.png] <https://landingpages.elevatecloud.co/CustomerFeedback?rating=3&source=emailsig&email=lesmith@ecsis.net;outages@outages.org&fromEmail=gturner@elevatetechgroup.com&fromFirstName=Thomas&fromLastName=Turner&messageSubject=Re:%20[outages]%20Explosion%20in%20Nashville> [cid:4747ef10-b6ae-40da-8890-0ca57d0191d7.png] <https://landingpages.elevatecloud.co/CustomerFeedback?rating=2&source=emailsig&email=lesmith@ecsis.net;outages@outages.org&fromEmail=gturner@elevatetechgroup.com&fromFirstName=Thomas&fromLastName=Turner&messageSubject=Re:%20[outages]%20Explosion%20in%20Nashville> [cid:aa278838-4dfc-4326-a6f6-b826618157c1.png] <https://landingpages.elevatecloud.co/CustomerFeedback?rating=1&source=emailsig&email=lesmith@ecsis.net;outages@outages.org&fromEmail=gturner@elevatetechgroup.com&fromFirstName=Thomas&fromLastName=Turner&messageSubject=Re:%20[outages]%20Explosion%20in%20Nashville>
Come visit us! How are we doing?


________________________________
From: Outages <outages-bounces@outages.org> on behalf of Larry Smith via Outages <outages@outages.org>
Sent: Friday, December 25, 2020 11:28:21 AM
To: outages@outages.org <outages@outages.org>
Subject: Re: [outages] Explosion in Nashville

Word we got from our fiber provider there is
that the entire building (ATT) is on battery backup as
the generators either did not come online or were
damaged in the blast. They told us to expect outages
as battery backups start to fail.

--
Larry Smith
lesmith@ecsis.net

On Fri December 25 2020 13:20, William Peebles via Outages wrote:
> Haven?t heard anything from my end on direct technical problems as a result
> of the explosion, but after any major event like this it?s highly likely
> that voice and data lines will be overloaded with traffic from nearby
> residents
>
> Sent from Outlook<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2faka.ms%2fqtex0l&c=E,1,p1zhKBtyhstaN_j5_82nqcnmAhwRYpukdWGuXJbW2y52ZmzH1IcVkVVWGt7djDvuuEFEJq9AsEPU5BRpg982MPUObzMqIL7UP7DG4sUAGrA,&typo=1> for iOS
> ________________________________
> From: Outages <outages-bounces@outages.org> on behalf of Kurt Duncan via
> Outages <outages@outages.org> Sent: Friday, December 25, 2020 2:09:15 PM
> To: outages@outages.org <outages@outages.org>
> Subject: [outages] Explosion in Nashville
>
> Anyone else seeing phone or internet issues due to the AT&T building being
> damaged by the explosion in Nashville ?
>
> Thank God for the first responders working today who protect us.
> --
> Kurt Duncan
_______________________________________________
Outages mailing list
Outages@outages.org
https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fpuck.nether.net%2fmailman%2flistinfo%2foutages&c=E,1,PRNUJtFdMDnptZFnuBSa03fPRAKBpTLlKgrN81tc1s7qGd7DDdL9ezWeSxNeQAs5RjL5D6xgSwDUMKjWobjo0ACy1JTO9uPEodswUMwSJHzj8LOwfA,,&typo=1

Once upon a time, Kurt Duncan <kurt.duncan@callcenterusa.us> said:
> Anyone else seeing phone or internet issues due to the AT&T building being
> damaged by the explosion in Nashville ?

A friend in Nashville with AT&T cell says that voice and data are both
down (only able to send because connected to family member's mobile
hotspot on another carrier).
--
Chris Adams <cma@cmadams.net>
_______________________________________________
Outages mailing list
Outages@outages.org
https://puck.nether.net/mailman/listinfo/outages

----- Original Message -----
> From: "Kurt Duncan via Outages" <outages@outages.org>

> Anyone else seeing phone or internet issues due to the AT&T building being
> damaged by the explosion in Nashville ?

Let's please continue to confine replies here to specific information about
network impacts, and keep the speculation threads to -discuss; thanks to
whomever already started that thread. I would name them, but the caffeine
hasn't kicked in yet.

If you're in that building in Nash, we'll be particularly interested in
anything you're allowed to tell us in between bailing.

Cheers,
-- jr '<admin/>' a
--
Jay R. Ashworth Baylink jra@baylink.com
Designer The Things I Think RFC 2100
Ashworth & Associates http://www.bcp38.info 2000 Land Rover DII
St Petersburg FL USA BCP38: Ask For It By Name! +1 727 647 1274
_______________________________________________
Outages mailing list
Outages@outages.org
https://puck.nether.net/mailman/listinfo/outages

For those not from the area, that building is a major old school Bell (att)
CO. The second outage was from the gas mains being isolated by local
provider taking their gensets offline. They had been power isolated by the
blast from my understanding.

On Fri, Dec 25, 2020, 14:30 Chris Adams via Outages <outages@outages.org>
wrote:

> Once upon a time, Kurt Duncan <kurt.duncan@callcenterusa.us> said:
> > Anyone else seeing phone or internet issues due to the AT&T building
> being
> > damaged by the explosion in Nashville ?
>
> A friend in Nashville with AT&T cell says that voice and data are both
> down (only able to send because connected to family member's mobile
> hotspot on another carrier).
> --
> Chris Adams <cma@cmadams.net>
> _______________________________________________
> Outages mailing list
> Outages@outages.org
> https://puck.nether.net/mailman/listinfo/outages
>

Once upon a time, Kurt Duncan via Outages <outages@outages.org> said:
> Anyone else seeing phone or internet issues due to the AT&T building being
> damaged by the explosion in Nashville ?

The AT&T issues are also reaching into Alabama. Several North Alabama
911 centers are reporting issues - Muscle Shoals, Madison County, and
Marshall County are all saying they are having capacity issues with the
911 system, asking people to call alternate local numbers.

I suspect this may be some general AT&T capacity issues in the region,
as these areas are not connected directly to Nashville (Muscle Shoals
and Madison County are in the Huntsville, AL LATA, and Marshall County
is in the Birmingham, AL LATA).

https://whnt.com/news/major-att-cellular-outage-reported-in-southeast/
--
Chris Adams <cma@cmadams.net>
_______________________________________________
Outages mailing list
Outages@outages.org
https://puck.nether.net/mailman/listinfo/outages

From a NY Times story:

“AT&T is experiencing an outage in the area because gas lines have been shut off in the area, impacting the building’s backup natural gas generators.”

-Brian

> On Dec 25, 2020, at 1:35 PM, Larry Smith via Outages <outages@outages.org> wrote:
>
> ?Word we got from our fiber provider there is
> that the entire building (ATT) is on battery backup as
> the generators either did not come online or were
> damaged in the blast. They told us to expect outages
> as battery backups start to fail.
>
> --
> Larry Smith
> lesmith@ecsis.net
>
>> On Fri December 25 2020 13:20, William Peebles via Outages wrote:
>> Haven’t heard anything from my end on direct technical problems as a result
>> of the explosion, but after any major event like this it’s highly likely
>> that voice and data lines will be overloaded with traffic from nearby
>> residents
>>
>> Sent from Outlook<https://aka.ms/qtex0l> for iOS
>> ________________________________
>> From: Outages <outages-bounces@outages.org> on behalf of Kurt Duncan via
>> Outages <outages@outages.org> Sent: Friday, December 25, 2020 2:09:15 PM
>> To: outages@outages.org <outages@outages.org>
>> Subject: [outages] Explosion in Nashville
>>
>> Anyone else seeing phone or internet issues due to the AT&T building being
>> damaged by the explosion in Nashville ?
>>
>> Thank God for the first responders working today who protect us.
>> --
>> Kurt Duncan
> _______________________________________________
> Outages mailing list
> Outages@outages.org
> https://puck.nether.net/mailman/listinfo/outages

Why would you use natural gas for your backup power??? That just causes issues like this.

> On Dec 25, 2020, at 4:32 PM, Brian Knight via Outages <outages@outages.org> wrote:
>
_______________________________________________
Outages mailing list
Outages@outages.org
https://puck.nether.net/mailman/listinfo/outages

Once upon a time, Matt Hoppes <mattlists@rivervalleyinternet.net> said:
> Why would you use natural gas for your backup power??? That just causes issues like this.

For most common issues, natural gas is fine, and doesn't require fuel
delivery trucks. Some locations don't allow signifant on-w]site fuel
storage. There are always pros and cons...
--
Chris Adams <cma@cmadams.net>
_______________________________________________
Outages mailing list
Outages@outages.org
https://puck.nether.net/mailman/listinfo/outages

> On Dec 25, 2020, at 11:01 PM, Chris Adams via Outages <outages@outages.org> wrote:
>
> Once upon a time, Matt Hoppes <mattlists@rivervalleyinternet.net> said:
>> Why would you use natural gas for your backup power??? That just causes issues like this.
>
> For most common issues, natural gas is fine, and doesn't require fuel
> delivery trucks. Some locations don't allow signifant on-w]site fuel
> storage. There are always pros and cons...

Yep. I always used to do propane / CNG dual-mode. Often easier to get propane delivered than diesel, and you don’t deplete your stores for test-runs or unless the emergency is so bad that utility CNG gets shut off.

-Bill

Got this internally, sharing here:

Here is the update from AT&T:
>
> The front 1/3rd of their building is structuraly damaged. That includes
fiber entry.
> They have no power and there is major flooding on all floors including
the basement.
> The Fire Department is en route with pump trucks to help with the
flooding.
> They have Building Engineers and Electricians on the way to look into
the structure and getting power turned on.
> They have 2 850KW generators on the way and 2 1MW generators on standby.
> They have new A/C Chillers on the way (Existing Chiller plant was
destroyed)

On Fri, Dec 25, 2020 at 3:38 PM Jay R. Ashworth via Outages <
outages@outages.org> wrote:

> ----- Original Message -----
> > From: "Kurt Duncan via Outages" <outages@outages.org>
>
> > Anyone else seeing phone or internet issues due to the AT&T building
> being
> > damaged by the explosion in Nashville ?
>
> Let's please continue to confine replies here to specific information about
> network impacts, and keep the speculation threads to -discuss; thanks to
> whomever already started that thread. I would name them, but the caffeine
> hasn't kicked in yet.
>
> If you're in that building in Nash, we'll be particularly interested in
> anything you're allowed to tell us in between bailing.
>
> Cheers,
> -- jr '<admin/>' a
> --
> Jay R. Ashworth Baylink
> jra@baylink.com
> Designer The Things I Think RFC
> 2100
> Ashworth & Associates http://www.bcp38.info 2000 Land
> Rover DII
> St Petersburg FL USA BCP38: Ask For It By Name! +1 727 647
> 1274
> _______________________________________________
> Outages mailing list
> Outages@outages.org
> https://puck.nether.net/mailman/listinfo/outages
>

Latest update from AT&T:



AT&T Corporate Real Estate teams are now gathering on site after pausing
work last night due a fire that reignited on site around midnight. The Fire
Marshall has joined the AT&T team to provide official approval to proceed.
Last night, the teams were able to accomplish testing the busbar, placing
the temporary generators, completing power cabling, and opening all the
breakers on the individual floors. It is estimated that this round of power
related work is about 75% complete. The target is for energizing is 1200
CST. A partial analysis by the AT&T Structural Engineer initial found no
concrete or steel damage that caused major concern. Once team re-engage
this morning, they will complete their full analysis and ensure it is safe
for restoration crews to proceed. AT&T Mechanical Engineers have confirmed
there is roughly 3 feet of water in the basement with potential damage to
mechanical equipment in that area. It is also suspected that commercial
power may not be available for some time and as a result power will come
from AT&T and Tennessee Emergency Management Agency generators. Additional
technician support will be made available rotating over a 24 hour period.

Core CBB teams continue to work on the secondary plan to reroute around the
Nashville 2nd Ave location. They had some success this morning at 05:05 CT
when core connectivity was established for the Winchester, KY office.
Partial 911 connectivity has been restored in that specific area. Work will
continue on the CBB reroute plans in several locations around the Southeast
Region.

We have installed a SatCOLT positioned in the area of the CO NSVLTNMT
building and is up and taking traffic to aid in communication. The
secondary SatCOLT arrived and has been turned up to provide additional
coverage.

The next update will be issued at approximately 13:00 CT, unless there is a
significant change in status

On Fri, Dec 25, 2020, 7:49 PM Ross Tajvar <ross@tajvar.io> wrote:

> Got this internally, sharing here:
>
> Here is the update from AT&T:
> >
> > The front 1/3rd of their building is structuraly damaged. That includes
> fiber entry.
> > They have no power and there is major flooding on all floors including
> the basement.
> > The Fire Department is en route with pump trucks to help with the
> flooding.
> > They have Building Engineers and Electricians on the way to look into
> the structure and getting power turned on.
> > They have 2 850KW generators on the way and 2 1MW generators on standby.
> > They have new A/C Chillers on the way (Existing Chiller plant was
> destroyed)
>
> On Fri, Dec 25, 2020 at 3:38 PM Jay R. Ashworth via Outages <
> outages@outages.org> wrote:
>
>> ----- Original Message -----
>> > From: "Kurt Duncan via Outages" <outages@outages.org>
>>
>> > Anyone else seeing phone or internet issues due to the AT&T building
>> being
>> > damaged by the explosion in Nashville ?
>>
>> Let's please continue to confine replies here to specific information
>> about
>> network impacts, and keep the speculation threads to -discuss; thanks to
>> whomever already started that thread. I would name them, but the
>> caffeine
>> hasn't kicked in yet.
>>
>> If you're in that building in Nash, we'll be particularly interested in
>> anything you're allowed to tell us in between bailing.
>>
>> Cheers,
>> -- jr '<admin/>' a
>> --
>> Jay R. Ashworth Baylink
>> jra@baylink.com
>> Designer The Things I Think RFC
>> 2100
>> Ashworth & Associates http://www.bcp38.info 2000 Land
>> Rover DII
>> St Petersburg FL USA BCP38: Ask For It By Name! +1 727 647
>> 1274
>> _______________________________________________
>> Outages mailing list
>> Outages@outages.org
>> https://puck.nether.net/mailman/listinfo/outages
>>
>

May want to consider moving this discussion to something more private.



On Sat, Dec 26, 2020 at 9:30 AM Ross Tajvar via Outages <outages@outages.org>
wrote:

> Latest update from AT&T:
>
>
>
> AT&T Corporate Real Estate teams are now gathering on site after pausing
> work last night due a fire that reignited on site around midnight. The Fire
> Marshall has joined the AT&T team to provide official approval to proceed.
> Last night, the teams were able to accomplish testing the busbar, placing
> the temporary generators, completing power cabling, and opening all the
> breakers on the individual floors. It is estimated that this round of power
> related work is about 75% complete. The target is for energizing is 1200
> CST. A partial analysis by the AT&T Structural Engineer initial found no
> concrete or steel damage that caused major concern. Once team re-engage
> this morning, they will complete their full analysis and ensure it is safe
> for restoration crews to proceed. AT&T Mechanical Engineers have confirmed
> there is roughly 3 feet of water in the basement with potential damage to
> mechanical equipment in that area. It is also suspected that commercial
> power may not be available for some time and as a result power will come
> from AT&T and Tennessee Emergency Management Agency generators. Additional
> technician support will be made available rotating over a 24 hour period.
>
> Core CBB teams continue to work on the secondary plan to reroute around
> the Nashville 2nd Ave location. They had some success this morning at 05:05
> CT when core connectivity was established for the Winchester, KY office.
> Partial 911 connectivity has been restored in that specific area. Work will
> continue on the CBB reroute plans in several locations around the Southeast
> Region.
>
> We have installed a SatCOLT positioned in the area of the CO NSVLTNMT
> building and is up and taking traffic to aid in communication. The
> secondary SatCOLT arrived and has been turned up to provide additional
> coverage.
>
> The next update will be issued at approximately 13:00 CT, unless there is
> a significant change in status
>
> On Fri, Dec 25, 2020, 7:49 PM Ross Tajvar <ross@tajvar.io> wrote:
>
>> Got this internally, sharing here:
>>
>> Here is the update from AT&T:
>> >
>> > The front 1/3rd of their building is structuraly damaged. That
>> includes fiber entry.
>> > They have no power and there is major flooding on all floors including
>> the basement.
>> > The Fire Department is en route with pump trucks to help with the
>> flooding.
>> > They have Building Engineers and Electricians on the way to look into
>> the structure and getting power turned on.
>> > They have 2 850KW generators on the way and 2 1MW generators on standby.
>> > They have new A/C Chillers on the way (Existing Chiller plant was
>> destroyed)
>>
>> On Fri, Dec 25, 2020 at 3:38 PM Jay R. Ashworth via Outages <
>> outages@outages.org> wrote:
>>
>>> ----- Original Message -----
>>> > From: "Kurt Duncan via Outages" <outages@outages.org>
>>>
>>> > Anyone else seeing phone or internet issues due to the AT&T building
>>> being
>>> > damaged by the explosion in Nashville ?
>>>
>>> Let's please continue to confine replies here to specific information
>>> about
>>> network impacts, and keep the speculation threads to -discuss; thanks to
>>> whomever already started that thread. I would name them, but the
>>> caffeine
>>> hasn't kicked in yet.
>>>
>>> If you're in that building in Nash, we'll be particularly interested in
>>> anything you're allowed to tell us in between bailing.
>>>
>>> Cheers,
>>> -- jr '<admin/>' a
>>> --
>>> Jay R. Ashworth Baylink
>>> jra@baylink.com
>>> Designer The Things I Think
>>> RFC 2100
>>> Ashworth & Associates http://www.bcp38.info 2000 Land
>>> Rover DII
>>> St Petersburg FL USA BCP38: Ask For It By Name! +1 727
>>> 647 1274
>>> _______________________________________________
>>> Outages mailing list
>>> Outages@outages.org
>>> https://puck.nether.net/mailman/listinfo/outages
>>>
>> _______________________________________________
> Outages mailing list
> Outages@outages.org
> https://puck.nether.net/mailman/listinfo/outages
>

Sorry I don't follow. Isn't the point of this list to keep people informed?

On Sat, Dec 26, 2020, 1:31 PM Luis Diaz <noc@riotgames.com> wrote:

> May want to consider moving this discussion to something more private.
>
>
>
> On Sat, Dec 26, 2020 at 9:30 AM Ross Tajvar via Outages <
> outages@outages.org> wrote:
>
>> Latest update from AT&T:
>>
>>
>>
>> AT&T Corporate Real Estate teams are now gathering on site after pausing
>> work last night due a fire that reignited on site around midnight. The Fire
>> Marshall has joined the AT&T team to provide official approval to proceed.
>> Last night, the teams were able to accomplish testing the busbar, placing
>> the temporary generators, completing power cabling, and opening all the
>> breakers on the individual floors. It is estimated that this round of power
>> related work is about 75% complete. The target is for energizing is 1200
>> CST. A partial analysis by the AT&T Structural Engineer initial found no
>> concrete or steel damage that caused major concern. Once team re-engage
>> this morning, they will complete their full analysis and ensure it is safe
>> for restoration crews to proceed. AT&T Mechanical Engineers have confirmed
>> there is roughly 3 feet of water in the basement with potential damage to
>> mechanical equipment in that area. It is also suspected that commercial
>> power may not be available for some time and as a result power will come
>> from AT&T and Tennessee Emergency Management Agency generators. Additional
>> technician support will be made available rotating over a 24 hour period.
>>
>> Core CBB teams continue to work on the secondary plan to reroute around
>> the Nashville 2nd Ave location. They had some success this morning at 05:05
>> CT when core connectivity was established for the Winchester, KY office.
>> Partial 911 connectivity has been restored in that specific area. Work will
>> continue on the CBB reroute plans in several locations around the Southeast
>> Region.
>>
>> We have installed a SatCOLT positioned in the area of the CO NSVLTNMT
>> building and is up and taking traffic to aid in communication. The
>> secondary SatCOLT arrived and has been turned up to provide additional
>> coverage.
>>
>> The next update will be issued at approximately 13:00 CT, unless there is
>> a significant change in status
>>
>> On Fri, Dec 25, 2020, 7:49 PM Ross Tajvar <ross@tajvar.io> wrote:
>>
>>> Got this internally, sharing here:
>>>
>>> Here is the update from AT&T:
>>> >
>>> > The front 1/3rd of their building is structuraly damaged. That
>>> includes fiber entry.
>>> > They have no power and there is major flooding on all floors including
>>> the basement.
>>> > The Fire Department is en route with pump trucks to help with the
>>> flooding.
>>> > They have Building Engineers and Electricians on the way to look into
>>> the structure and getting power turned on.
>>> > They have 2 850KW generators on the way and 2 1MW generators on
>>> standby.
>>> > They have new A/C Chillers on the way (Existing Chiller plant was
>>> destroyed)
>>>
>>> On Fri, Dec 25, 2020 at 3:38 PM Jay R. Ashworth via Outages <
>>> outages@outages.org> wrote:
>>>
>>>> ----- Original Message -----
>>>> > From: "Kurt Duncan via Outages" <outages@outages.org>
>>>>
>>>> > Anyone else seeing phone or internet issues due to the AT&T building
>>>> being
>>>> > damaged by the explosion in Nashville ?
>>>>
>>>> Let's please continue to confine replies here to specific information
>>>> about
>>>> network impacts, and keep the speculation threads to -discuss; thanks to
>>>> whomever already started that thread. I would name them, but the
>>>> caffeine
>>>> hasn't kicked in yet.
>>>>
>>>> If you're in that building in Nash, we'll be particularly interested in
>>>> anything you're allowed to tell us in between bailing.
>>>>
>>>> Cheers,
>>>> -- jr '<admin/>' a
>>>> --
>>>> Jay R. Ashworth Baylink
>>>> jra@baylink.com
>>>> Designer The Things I Think
>>>> RFC 2100
>>>> Ashworth & Associates http://www.bcp38.info 2000 Land
>>>> Rover DII
>>>> St Petersburg FL USA BCP38: Ask For It By Name! +1 727
>>>> 647 1274
>>>> _______________________________________________
>>>> Outages mailing list
>>>> Outages@outages.org
>>>> https://puck.nether.net/mailman/listinfo/outages
>>>>
>>> _______________________________________________
>> Outages mailing list
>> Outages@outages.org
>> https://puck.nether.net/mailman/listinfo/outages
>>
>

This is turning into a mix of information. Its going to be that way for a
while until we get more stable, when it's just outage notification updates.

On Sat, Dec 26, 2020 at 1:38 PM Ross Tajvar via Outages <outages@outages.org>
wrote:

> Sorry I don't follow. Isn't the point of this list to keep people informed?
>
> On Sat, Dec 26, 2020, 1:31 PM Luis Diaz <noc@riotgames.com> wrote:
>
>> May want to consider moving this discussion to something more private.
>>
>>
>>
>> On Sat, Dec 26, 2020 at 9:30 AM Ross Tajvar via Outages <
>> outages@outages.org> wrote:
>>
>>> Latest update from AT&T:
>>>
>>>
>>>
>>> AT&T Corporate Real Estate teams are now gathering on site after pausing
>>> work last night due a fire that reignited on site around midnight. The Fire
>>> Marshall has joined the AT&T team to provide official approval to proceed.
>>> Last night, the teams were able to accomplish testing the busbar, placing
>>> the temporary generators, completing power cabling, and opening all the
>>> breakers on the individual floors. It is estimated that this round of power
>>> related work is about 75% complete. The target is for energizing is 1200
>>> CST. A partial analysis by the AT&T Structural Engineer initial found no
>>> concrete or steel damage that caused major concern. Once team re-engage
>>> this morning, they will complete their full analysis and ensure it is safe
>>> for restoration crews to proceed. AT&T Mechanical Engineers have confirmed
>>> there is roughly 3 feet of water in the basement with potential damage to
>>> mechanical equipment in that area. It is also suspected that commercial
>>> power may not be available for some time and as a result power will come
>>> from AT&T and Tennessee Emergency Management Agency generators. Additional
>>> technician support will be made available rotating over a 24 hour period.
>>>
>>> Core CBB teams continue to work on the secondary plan to reroute around
>>> the Nashville 2nd Ave location. They had some success this morning at 05:05
>>> CT when core connectivity was established for the Winchester, KY office.
>>> Partial 911 connectivity has been restored in that specific area. Work will
>>> continue on the CBB reroute plans in several locations around the Southeast
>>> Region.
>>>
>>> We have installed a SatCOLT positioned in the area of the CO NSVLTNMT
>>> building and is up and taking traffic to aid in communication. The
>>> secondary SatCOLT arrived and has been turned up to provide additional
>>> coverage.
>>>
>>> The next update will be issued at approximately 13:00 CT, unless there
>>> is a significant change in status
>>>
>>> On Fri, Dec 25, 2020, 7:49 PM Ross Tajvar <ross@tajvar.io> wrote:
>>>
>>>> Got this internally, sharing here:
>>>>
>>>> Here is the update from AT&T:
>>>> >
>>>> > The front 1/3rd of their building is structuraly damaged. That
>>>> includes fiber entry.
>>>> > They have no power and there is major flooding on all floors
>>>> including the basement.
>>>> > The Fire Department is en route with pump trucks to help with the
>>>> flooding.
>>>> > They have Building Engineers and Electricians on the way to look
>>>> into the structure and getting power turned on.
>>>> > They have 2 850KW generators on the way and 2 1MW generators on
>>>> standby.
>>>> > They have new A/C Chillers on the way (Existing Chiller plant was
>>>> destroyed)
>>>>
>>>> On Fri, Dec 25, 2020 at 3:38 PM Jay R. Ashworth via Outages <
>>>> outages@outages.org> wrote:
>>>>
>>>>> ----- Original Message -----
>>>>> > From: "Kurt Duncan via Outages" <outages@outages.org>
>>>>>
>>>>> > Anyone else seeing phone or internet issues due to the AT&T building
>>>>> being
>>>>> > damaged by the explosion in Nashville ?
>>>>>
>>>>> Let's please continue to confine replies here to specific information
>>>>> about
>>>>> network impacts, and keep the speculation threads to -discuss; thanks
>>>>> to
>>>>> whomever already started that thread. I would name them, but the
>>>>> caffeine
>>>>> hasn't kicked in yet.
>>>>>
>>>>> If you're in that building in Nash, we'll be particularly interested
>>>>> in
>>>>> anything you're allowed to tell us in between bailing.
>>>>>
>>>>> Cheers,
>>>>> -- jr '<admin/>' a
>>>>> --
>>>>> Jay R. Ashworth Baylink
>>>>> jra@baylink.com
>>>>> Designer The Things I Think
>>>>> RFC 2100
>>>>> Ashworth & Associates http://www.bcp38.info 2000 Land
>>>>> Rover DII
>>>>> St Petersburg FL USA BCP38: Ask For It By Name! +1 727
>>>>> 647 1274
>>>>> _______________________________________________
>>>>> Outages mailing list
>>>>> Outages@outages.org
>>>>> https://puck.nether.net/mailman/listinfo/outages
>>>>>
>>>> _______________________________________________
>>> Outages mailing list
>>> Outages@outages.org
>>> https://puck.nether.net/mailman/listinfo/outages
>>>
>> _______________________________________________
> Outages mailing list
> Outages@outages.org
> https://puck.nether.net/mailman/listinfo/outages
>


--
Sincerely,

Jason W Kuehl
Cell 920-419-8983
jason.w.kuehl@gmail.com

I appreciate the updates. I’m still getting outage alerts from Intrado due to the ongoing work.

John Bergoon
2513666986

On Dec 26, 2020, at 12:52 PM, Jason Kuehl via Outages <outages@outages.org> wrote:

?

CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you recognize the sender and know the content is safe.

This is turning into a mix of information. Its going to be that way for a while until we get more stable, when it's just outage notification updates.

On Sat, Dec 26, 2020 at 1:38 PM Ross Tajvar via Outages <outages@outages.org<mailto:outages@outages.org>> wrote:
Sorry I don't follow. Isn't the point of this list to keep people informed?

On Sat, Dec 26, 2020, 1:31 PM Luis Diaz <noc@riotgames.com<mailto:noc@riotgames.com>> wrote:
May want to consider moving this discussion to something more private.



On Sat, Dec 26, 2020 at 9:30 AM Ross Tajvar via Outages <outages@outages.org<mailto:outages@outages.org>> wrote:
Latest update from AT&T:



AT&T Corporate Real Estate teams are now gathering on site after pausing work last night due a fire that reignited on site around midnight. The Fire Marshall has joined the AT&T team to provide official approval to proceed. Last night, the teams were able to accomplish testing the busbar, placing the temporary generators, completing power cabling, and opening all the breakers on the individual floors. It is estimated that this round of power related work is about 75% complete. The target is for energizing is 1200 CST. A partial analysis by the AT&T Structural Engineer initial found no concrete or steel damage that caused major concern. Once team re-engage this morning, they will complete their full analysis and ensure it is safe for restoration crews to proceed. AT&T Mechanical Engineers have confirmed there is roughly 3 feet of water in the basement with potential damage to mechanical equipment in that area. It is also suspected that commercial power may not be available for some time and as a result power will come from AT&T and Tennessee Emergency Management Agency generators. Additional technician support will be made available rotating over a 24 hour period.

Core CBB teams continue to work on the secondary plan to reroute around the Nashville 2nd Ave location. They had some success this morning at 05:05 CT when core connectivity was established for the Winchester, KY office. Partial 911 connectivity has been restored in that specific area. Work will continue on the CBB reroute plans in several locations around the Southeast Region.

We have installed a SatCOLT positioned in the area of the CO NSVLTNMT building and is up and taking traffic to aid in communication. The secondary SatCOLT arrived and has been turned up to provide additional coverage.

The next update will be issued at approximately 13:00 CT, unless there is a significant change in status

On Fri, Dec 25, 2020, 7:49 PM Ross Tajvar <ross@tajvar.io<mailto:ross@tajvar.io>> wrote:
Got this internally, sharing here:

Here is the update from AT&T:
>
> The front 1/3rd of their building is structuraly damaged. That includes fiber entry.
> They have no power and there is major flooding on all floors including the basement.
> The Fire Department is en route with pump trucks to help with the flooding.
> They have Building Engineers and Electricians on the way to look into the structure and getting power turned on.
> They have 2 850KW generators on the way and 2 1MW generators on standby.
> They have new A/C Chillers on the way (Existing Chiller plant was destroyed)

On Fri, Dec 25, 2020 at 3:38 PM Jay R. Ashworth via Outages <outages@outages.org<mailto:outages@outages.org>> wrote:
----- Original Message -----
> From: "Kurt Duncan via Outages" <outages@outages.org<mailto:outages@outages.org>>

> Anyone else seeing phone or internet issues due to the AT&T building being
> damaged by the explosion in Nashville ?

Let's please continue to confine replies here to specific information about
network impacts, and keep the speculation threads to -discuss; thanks to
whomever already started that thread. I would name them, but the caffeine
hasn't kicked in yet.

If you're in that building in Nash, we'll be particularly interested in
anything you're allowed to tell us in between bailing.

Cheers,
-- jr '<admin/>' a
--
Jay R. Ashworth Baylink jra@baylink.com<mailto:jra@baylink.com>
Designer The Things I Think RFC 2100
Ashworth & Associates http://www.bcp38.info 2000 Land Rover DII
St Petersburg FL USA BCP38: Ask For It By Name! +1 727 647 1274
_______________________________________________
Outages mailing list
Outages@outages.org<mailto:Outages@outages.org>
https://puck.nether.net/mailman/listinfo/outages
_______________________________________________
Outages mailing list
Outages@outages.org<mailto:Outages@outages.org>
https://puck.nether.net/mailman/listinfo/outages
_______________________________________________
Outages mailing list
Outages@outages.org<mailto:Outages@outages.org>
https://puck.nether.net/mailman/listinfo/outages


--
Sincerely,

Jason W Kuehl
Cell 920-419-8983
jason.w.kuehl@gmail.com<mailto:jason.w.kuehl@gmail.com>
_______________________________________________
Outages mailing list
Outages@outages.org
https://puck.nether.net/mailman/listinfo/outages

Yes. Thanks! for the updates
________________________________
From: Outages <outages-bounces@outages.org> on behalf of Ross Tajvar via Outages <outages@outages.org>
Sent: Saturday, December 26, 2020 12:35:29 PM
To: Luis Diaz <noc@riotgames.com>
Cc: OUTAGES Mailing List <outages@outages.org>
Subject: Re: [outages] Explosion in Nashville


*** EXTERNAL SENDER ***



Sorry I don't follow. Isn't the point of this list to keep people informed?

On Sat, Dec 26, 2020, 1:31 PM Luis Diaz <noc@riotgames.com<mailto:noc@riotgames.com>> wrote:
May want to consider moving this discussion to something more private.



On Sat, Dec 26, 2020 at 9:30 AM Ross Tajvar via Outages <outages@outages.org<mailto:outages@outages.org>> wrote:
Latest update from AT&T:



AT&T Corporate Real Estate teams are now gathering on site after pausing work last night due a fire that reignited on site around midnight. The Fire Marshall has joined the AT&T team to provide official approval to proceed. Last night, the teams were able to accomplish testing the busbar, placing the temporary generators, completing power cabling, and opening all the breakers on the individual floors. It is estimated that this round of power related work is about 75% complete. The target is for energizing is 1200 CST. A partial analysis by the AT&T Structural Engineer initial found no concrete or steel damage that caused major concern. Once team re-engage this morning, they will complete their full analysis and ensure it is safe for restoration crews to proceed. AT&T Mechanical Engineers have confirmed there is roughly 3 feet of water in the basement with potential damage to mechanical equipment in that area. It is also suspected that commercial power may not be available for some time and as a result power will come from AT&T and Tennessee Emergency Management Agency generators. Additional technician support will be made available rotating over a 24 hour period.

Core CBB teams continue to work on the secondary plan to reroute around the Nashville 2nd Ave location. They had some success this morning at 05:05 CT when core connectivity was established for the Winchester, KY office. Partial 911 connectivity has been restored in that specific area. Work will continue on the CBB reroute plans in several locations around the Southeast Region.

We have installed a SatCOLT positioned in the area of the CO NSVLTNMT building and is up and taking traffic to aid in communication. The secondary SatCOLT arrived and has been turned up to provide additional coverage.

The next update will be issued at approximately 13:00 CT, unless there is a significant change in status

On Fri, Dec 25, 2020, 7:49 PM Ross Tajvar <ross@tajvar.io<mailto:ross@tajvar.io>> wrote:
Got this internally, sharing here:

Here is the update from AT&T:
>
> The front 1/3rd of their building is structuraly damaged. That includes fiber entry.
> They have no power and there is major flooding on all floors including the basement.
> The Fire Department is en route with pump trucks to help with the flooding.
> They have Building Engineers and Electricians on the way to look into the structure and getting power turned on.
> They have 2 850KW generators on the way and 2 1MW generators on standby.
> They have new A/C Chillers on the way (Existing Chiller plant was destroyed)

On Fri, Dec 25, 2020 at 3:38 PM Jay R. Ashworth via Outages <outages@outages.org<mailto:outages@outages.org>> wrote:
----- Original Message -----
> From: "Kurt Duncan via Outages" <outages@outages.org<mailto:outages@outages.org>>

> Anyone else seeing phone or internet issues due to the AT&T building being
> damaged by the explosion in Nashville ?

Let's please continue to confine replies here to specific information about
network impacts, and keep the speculation threads to -discuss; thanks to
whomever already started that thread. I would name them, but the caffeine
hasn't kicked in yet.

If you're in that building in Nash, we'll be particularly interested in
anything you're allowed to tell us in between bailing.

Cheers,
-- jr '<admin/>' a
--
Jay R. Ashworth Baylink jra@baylink.com<mailto:jra@baylink.com>
Designer The Things I Think RFC 2100
Ashworth & Associates http://www.bcp38.info 2000 Land Rover DII
St Petersburg FL USA BCP38: Ask For It By Name! +1 727 647 1274
_______________________________________________
Outages mailing list
Outages@outages.org<mailto:Outages@outages.org>
https://puck.nether.net/mailman/listinfo/outages
_______________________________________________
Outages mailing list
Outages@outages.org<mailto:Outages@outages.org>
https://puck.nether.net/mailman/listinfo/outages

It is. So unless that message that you reposted came to you with a flag line suggesting it should not be disseminated, you're welcome to post it here.

Cheers,
-- jr '<admin/>' a

On December 26, 2020 1:35:29 PM EST, Ross Tajvar via Outages <outages@outages.org> wrote:
>Sorry I don't follow. Isn't the point of this list to keep people
>informed?
>
>On Sat, Dec 26, 2020, 1:31 PM Luis Diaz <noc@riotgames.com> wrote:
>
>> May want to consider moving this discussion to something more
>private.
>>
>>
>>
>> On Sat, Dec 26, 2020 at 9:30 AM Ross Tajvar via Outages <
>> outages@outages.org> wrote:
>>
>>> Latest update from AT&T:
>>>
>>>
>>>
>>> AT&T Corporate Real Estate teams are now gathering on site after
>pausing
>>> work last night due a fire that reignited on site around midnight.
>The Fire
>>> Marshall has joined the AT&T team to provide official approval to
>proceed.
>>> Last night, the teams were able to accomplish testing the busbar,
>placing
>>> the temporary generators, completing power cabling, and opening all
>the
>>> breakers on the individual floors. It is estimated that this round
>of power
>>> related work is about 75% complete. The target is for energizing is
>1200
>>> CST. A partial analysis by the AT&T Structural Engineer initial
>found no
>>> concrete or steel damage that caused major concern. Once team
>re-engage
>>> this morning, they will complete their full analysis and ensure it
>is safe
>>> for restoration crews to proceed. AT&T Mechanical Engineers have
>confirmed
>>> there is roughly 3 feet of water in the basement with potential
>damage to
>>> mechanical equipment in that area. It is also suspected that
>commercial
>>> power may not be available for some time and as a result power will
>come
>>> from AT&T and Tennessee Emergency Management Agency generators.
>Additional
>>> technician support will be made available rotating over a 24 hour
>period.
>>>
>>> Core CBB teams continue to work on the secondary plan to reroute
>around
>>> the Nashville 2nd Ave location. They had some success this morning
>at 05:05
>>> CT when core connectivity was established for the Winchester, KY
>office.
>>> Partial 911 connectivity has been restored in that specific area.
>Work will
>>> continue on the CBB reroute plans in several locations around the
>Southeast
>>> Region.
>>>
>>> We have installed a SatCOLT positioned in the area of the CO
>NSVLTNMT
>>> building and is up and taking traffic to aid in communication. The
>>> secondary SatCOLT arrived and has been turned up to provide
>additional
>>> coverage.
>>>
>>> The next update will be issued at approximately 13:00 CT, unless
>there is
>>> a significant change in status
>>>
>>> On Fri, Dec 25, 2020, 7:49 PM Ross Tajvar <ross@tajvar.io> wrote:
>>>
>>>> Got this internally, sharing here:
>>>>
>>>> Here is the update from AT&T:
>>>> >
>>>> > The front 1/3rd of their building is structuraly damaged. That
>>>> includes fiber entry.
>>>> > They have no power and there is major flooding on all floors
>including
>>>> the basement.
>>>> > The Fire Department is en route with pump trucks to help with the
>>>> flooding.
>>>> > They have Building Engineers and Electricians on the way to look
>into
>>>> the structure and getting power turned on.
>>>> > They have 2 850KW generators on the way and 2 1MW generators on
>>>> standby.
>>>> > They have new A/C Chillers on the way (Existing Chiller plant was
>>>> destroyed)
>>>>
>>>> On Fri, Dec 25, 2020 at 3:38 PM Jay R. Ashworth via Outages <
>>>> outages@outages.org> wrote:
>>>>
>>>>> ----- Original Message -----
>>>>> > From: "Kurt Duncan via Outages" <outages@outages.org>
>>>>>
>>>>> > Anyone else seeing phone or internet issues due to the AT&T
>building
>>>>> being
>>>>> > damaged by the explosion in Nashville ?
>>>>>
>>>>> Let's please continue to confine replies here to specific
>information
>>>>> about
>>>>> network impacts, and keep the speculation threads to -discuss;
>thanks to
>>>>> whomever already started that thread. I would name them, but the
>>>>> caffeine
>>>>> hasn't kicked in yet.
>>>>>
>>>>> If you're in that building in Nash, we'll be particularly
>interested in
>>>>> anything you're allowed to tell us in between bailing.
>>>>>
>>>>> Cheers,
>>>>> -- jr '<admin/>' a
>>>>> --
>>>>> Jay R. Ashworth Baylink
>>>>> jra@baylink.com
>>>>> Designer The Things I Think
>>>>> RFC 2100
>>>>> Ashworth & Associates http://www.bcp38.info 2000
>Land
>>>>> Rover DII
>>>>> St Petersburg FL USA BCP38: Ask For It By Name! +1
>727
>>>>> 647 1274
>>>>> _______________________________________________
>>>>> Outages mailing list
>>>>> Outages@outages.org
>>>>> https://puck.nether.net/mailman/listinfo/outages
>>>>>
>>>> _______________________________________________
>>> Outages mailing list
>>> Outages@outages.org
>>> https://puck.nether.net/mailman/listinfo/outages
>>>
>>

--
Sent from my Android device with K-9 Mail. Please excuse my brevity.

Apologies if this has been already posted.


8pm update:

All - Teams have completed their walk-through with the Fire Marshal and
local authorities. Officials have given teams approval to begin restoring
power to the 2nd and 3rd floors. After the 2nd and 3rd floors are
energized, teams will begin working to restore power to the 1st, 4th and
5th floors. A fuel truck was brought in to refill the fuel tanks on the
generators and a Caterpilar technician is onsite getting the generators
ready to run. Teams will start one generator at a time. Once generators are
running, teams can begin energizing power plants and charging batteries.
Many batteries have reversed polarity and it is unclear at this time, if
those batteries can be recharged. AT&T Structural Engineer has found no
concrete or steel damage that would impair restoring power and service in
the building that caused major concern.

Core CBB teams continue to work on the secondary plan to reroute around the
Nashville 2nd Ave location. Service in and around the Lexington/Winchester,
KY has returned for 3G and 4G service when CBB service was rerouted around
the Nashville 2nd Ave location. 3G Data service is still degraded within
that specific area. Work will continue with the CBB reroute plans across
several locations around the Southeast Region including the Nashville, MTSO
(Brick Church Pike).

Two SATCOLTs are positioned on the East and West sides of the NSVLTNMT
Central Office and both are on-air and taking traffic to aid in
communication. There are currently 6 assets on air – the 2 at the CO plus 4
that are supporting FN Customer Requests. There are 4 assets on route to
their respective deployment locations- 2 in Alabama and 2 in TN. One asset
has arrived at its intended deployment location in TN and is in the process
of being turned up. Another 3 requests have been prioritized and assets
that just arrived at the staging location are being deployed to their
target locations in TN. Issues remain at various locations in GA, KY, & MS.




On Sat, Dec 26, 2020 at 6:08 PM Jay Ashworth via Outages <
outages@outages.org> wrote:

> It is. So unless that message that you reposted came to you with a flag
> line suggesting it should not be disseminated, you're welcome to post it
> here.
>
>
> Cheers,
> -- jr '<admin/>' a
>
> On December 26, 2020 1:35:29 PM EST, Ross Tajvar via Outages <
> outages@outages.org> wrote:
>>
>> Sorry I don't follow. Isn't the point of this list to keep people
>> informed?
>>
>> On Sat, Dec 26, 2020, 1:31 PM Luis Diaz <noc@riotgames.com> wrote:
>>
>>> May want to consider moving this discussion to something more private.
>>>
>>>
>>>
>>> On Sat, Dec 26, 2020 at 9:30 AM Ross Tajvar via Outages <
>>> outages@outages.org> wrote:
>>>
>>>> Latest update from AT&T:
>>>>
>>>>
>>>>
>>>> AT&T Corporate Real Estate teams are now gathering on site after
>>>> pausing work last night due a fire that reignited on site around midnight.
>>>> The Fire Marshall has joined the AT&T team to provide official approval to
>>>> proceed. Last night, the teams were able to accomplish testing the busbar,
>>>> placing the temporary generators, completing power cabling, and opening all
>>>> the breakers on the individual floors. It is estimated that this round of
>>>> power related work is about 75% complete. The target is for energizing is
>>>> 1200 CST. A partial analysis by the AT&T Structural Engineer initial found
>>>> no concrete or steel damage that caused major concern. Once team re-engage
>>>> this morning, they will complete their full analysis and ensure it is safe
>>>> for restoration crews to proceed. AT&T Mechanical Engineers have confirmed
>>>> there is roughly 3 feet of water in the basement with potential damage to
>>>> mechanical equipment in that area. It is also suspected that commercial
>>>> power may not be available for some time and as a result power will come
>>>> from AT&T and Tennessee Emergency Management Agency generators. Additional
>>>> technician support will be made available rotating over a 24 hour period.
>>>>
>>>> Core CBB teams continue to work on the secondary plan to reroute around
>>>> the Nashville 2nd Ave location. They had some success this morning at 05:05
>>>> CT when core connectivity was established for the Winchester, KY office.
>>>> Partial 911 connectivity has been restored in that specific area. Work will
>>>> continue on the CBB reroute plans in several locations around the Southeast
>>>> Region.
>>>>
>>>> We have installed a SatCOLT positioned in the area of the CO NSVLTNMT
>>>> building and is up and taking traffic to aid in communication. The
>>>> secondary SatCOLT arrived and has been turned up to provide additional
>>>> coverage.
>>>>
>>>> The next update will be issued at approximately 13:00 CT, unless there
>>>> is a significant change in status
>>>>
>>>> On Fri, Dec 25, 2020, 7:49 PM Ross Tajvar <ross@tajvar.io> wrote:
>>>>
>>>>> Got this internally, sharing here:
>>>>>
>>>>> Here is the update from AT&T:
>>>>> >
>>>>> > The front 1/3rd of their building is structuraly damaged. That
>>>>> includes fiber entry.
>>>>> > They have no power and there is major flooding on all floors
>>>>> including the basement.
>>>>> > The Fire Department is en route with pump trucks to help with the
>>>>> flooding.
>>>>> > They have Building Engineers and Electricians on the way to look
>>>>> into the structure and getting power turned on.
>>>>> > They have 2 850KW generators on the way and 2 1MW generators on
>>>>> standby.
>>>>> > They have new A/C Chillers on the way (Existing Chiller plant was
>>>>> destroyed)
>>>>>
>>>>> On Fri, Dec 25, 2020 at 3:38 PM Jay R. Ashworth via Outages <
>>>>> outages@outages.org> wrote:
>>>>>
>>>>>> ----- Original Message -----
>>>>>> > From: "Kurt Duncan via Outages" <outages@outages.org>
>>>>>>
>>>>>> > Anyone else seeing phone or internet issues due to the AT&T
>>>>>> building being
>>>>>> > damaged by the explosion in Nashville ?
>>>>>>
>>>>>> Let's please continue to confine replies here to specific information
>>>>>> about
>>>>>> network impacts, and keep the speculation threads to -discuss; thanks
>>>>>> to
>>>>>> whomever already started that thread. I would name them, but the
>>>>>> caffeine
>>>>>> hasn't kicked in yet.
>>>>>>
>>>>>> If you're in that building in Nash, we'll be particularly interested
>>>>>> in
>>>>>> anything you're allowed to tell us in between bailing.
>>>>>>
>>>>>> Cheers,
>>>>>> -- jr '<admin/>' a
>>>>>> --
>>>>>> Jay R. Ashworth Baylink
>>>>>> jra@baylink.com
>>>>>> Designer The Things I Think
>>>>>> RFC 2100
>>>>>> Ashworth & Associates http://www.bcp38.info 2000 Land
>>>>>> Rover DII
>>>>>> St Petersburg FL USA BCP38: Ask For It By Name! +1 727
>>>>>> 647 1274
>>>>>> _______________________________________________
>>>>>> Outages mailing list
>>>>>> Outages@outages.org
>>>>>> https://puck.nether.net/mailman/listinfo/outages
>>>>>>
>>>>> _______________________________________________
>>>> Outages mailing list
>>>> Outages@outages.org
>>>> https://puck.nether.net/mailman/listinfo/outages
>>>>
>>>
> --
> Sent from my Android device with K-9 Mail. Please excuse my brevity.
> _______________________________________________
> Outages mailing list
> Outages@outages.org
> https://puck.nether.net/mailman/listinfo/outages
>
--


*Walt Amason* | Director of Information Technology | Kelley Companies
*cell* (601) 382-5888 | *office* (601) 735-2541 | *website*
www.kelleycompanies.com

This is from 11pm. Some duplicate info here, but some other stuff as well:

Latest AT&T Update:



Within the Nashville Central Office, the 2nd, 3rd, 4th and 6th floors
currently have power to the rectifiers.


The 6th floor rectifiers are currently being energized to charge batteries.



Power has been applied to the Air Conditioning risers, and Gensets are
turned up. One handler is turned up on each of floors 2 and 3.



Pumping of the three feet of water in the basement has been completed;
however, the FBI has not yet released this portion of the building back to
AT&T.

It has been confirmed that the four utility service feeds are offline, and
the utility transformer bus duct experienced damage.

On Sat, Dec 26, 2020, 10:45 PM Walt Amason <wamason@kelleycompanies.com>
wrote:

> Apologies if this has been already posted.
>
>
> 8pm update:
>
> All - Teams have completed their walk-through with the Fire Marshal and
> local authorities. Officials have given teams approval to begin restoring
> power to the 2nd and 3rd floors. After the 2nd and 3rd floors are
> energized, teams will begin working to restore power to the 1st, 4th and
> 5th floors. A fuel truck was brought in to refill the fuel tanks on the
> generators and a Caterpilar technician is onsite getting the generators
> ready to run. Teams will start one generator at a time. Once generators are
> running, teams can begin energizing power plants and charging batteries.
> Many batteries have reversed polarity and it is unclear at this time, if
> those batteries can be recharged. AT&T Structural Engineer has found no
> concrete or steel damage that would impair restoring power and service in
> the building that caused major concern.
>
> Core CBB teams continue to work on the secondary plan to reroute around
> the Nashville 2nd Ave location. Service in and around the
> Lexington/Winchester, KY has returned for 3G and 4G service when CBB
> service was rerouted around the Nashville 2nd Ave location. 3G Data service
> is still degraded within that specific area. Work will continue with the
> CBB reroute plans across several locations around the Southeast Region
> including the Nashville, MTSO (Brick Church Pike).
>
> Two SATCOLTs are positioned on the East and West sides of the NSVLTNMT
> Central Office and both are on-air and taking traffic to aid in
> communication. There are currently 6 assets on air – the 2 at the CO plus 4
> that are supporting FN Customer Requests. There are 4 assets on route to
> their respective deployment locations- 2 in Alabama and 2 in TN. One asset
> has arrived at its intended deployment location in TN and is in the process
> of being turned up. Another 3 requests have been prioritized and assets
> that just arrived at the staging location are being deployed to their
> target locations in TN. Issues remain at various locations in GA, KY, & MS.
>
>
>
>
> On Sat, Dec 26, 2020 at 6:08 PM Jay Ashworth via Outages <
> outages@outages.org> wrote:
>
>> It is. So unless that message that you reposted came to you with a flag
>> line suggesting it should not be disseminated, you're welcome to post it
>> here.
>>
>>
>> Cheers,
>> -- jr '<admin/>' a
>>
>> On December 26, 2020 1:35:29 PM EST, Ross Tajvar via Outages <
>> outages@outages.org> wrote:
>>>
>>> Sorry I don't follow. Isn't the point of this list to keep people
>>> informed?
>>>
>>> On Sat, Dec 26, 2020, 1:31 PM Luis Diaz <noc@riotgames.com> wrote:
>>>
>>>> May want to consider moving this discussion to something more private.
>>>>
>>>>
>>>>
>>>> On Sat, Dec 26, 2020 at 9:30 AM Ross Tajvar via Outages <
>>>> outages@outages.org> wrote:
>>>>
>>>>> Latest update from AT&T:
>>>>>
>>>>>
>>>>>
>>>>> AT&T Corporate Real Estate teams are now gathering on site after
>>>>> pausing work last night due a fire that reignited on site around midnight.
>>>>> The Fire Marshall has joined the AT&T team to provide official approval to
>>>>> proceed. Last night, the teams were able to accomplish testing the busbar,
>>>>> placing the temporary generators, completing power cabling, and opening all
>>>>> the breakers on the individual floors. It is estimated that this round of
>>>>> power related work is about 75% complete. The target is for energizing is
>>>>> 1200 CST. A partial analysis by the AT&T Structural Engineer initial found
>>>>> no concrete or steel damage that caused major concern. Once team re-engage
>>>>> this morning, they will complete their full analysis and ensure it is safe
>>>>> for restoration crews to proceed. AT&T Mechanical Engineers have confirmed
>>>>> there is roughly 3 feet of water in the basement with potential damage to
>>>>> mechanical equipment in that area. It is also suspected that commercial
>>>>> power may not be available for some time and as a result power will come
>>>>> from AT&T and Tennessee Emergency Management Agency generators. Additional
>>>>> technician support will be made available rotating over a 24 hour period.
>>>>>
>>>>> Core CBB teams continue to work on the secondary plan to reroute
>>>>> around the Nashville 2nd Ave location. They had some success this morning
>>>>> at 05:05 CT when core connectivity was established for the Winchester, KY
>>>>> office. Partial 911 connectivity has been restored in that specific area.
>>>>> Work will continue on the CBB reroute plans in several locations around the
>>>>> Southeast Region.
>>>>>
>>>>> We have installed a SatCOLT positioned in the area of the CO NSVLTNMT
>>>>> building and is up and taking traffic to aid in communication. The
>>>>> secondary SatCOLT arrived and has been turned up to provide additional
>>>>> coverage.
>>>>>
>>>>> The next update will be issued at approximately 13:00 CT, unless there
>>>>> is a significant change in status
>>>>>
>>>>> On Fri, Dec 25, 2020, 7:49 PM Ross Tajvar <ross@tajvar.io> wrote:
>>>>>
>>>>>> Got this internally, sharing here:
>>>>>>
>>>>>> Here is the update from AT&T:
>>>>>> >
>>>>>> > The front 1/3rd of their building is structuraly damaged. That
>>>>>> includes fiber entry.
>>>>>> > They have no power and there is major flooding on all floors
>>>>>> including the basement.
>>>>>> > The Fire Department is en route with pump trucks to help with the
>>>>>> flooding.
>>>>>> > They have Building Engineers and Electricians on the way to look
>>>>>> into the structure and getting power turned on.
>>>>>> > They have 2 850KW generators on the way and 2 1MW generators on
>>>>>> standby.
>>>>>> > They have new A/C Chillers on the way (Existing Chiller plant was
>>>>>> destroyed)
>>>>>>
>>>>>> On Fri, Dec 25, 2020 at 3:38 PM Jay R. Ashworth via Outages <
>>>>>> outages@outages.org> wrote:
>>>>>>
>>>>>>> ----- Original Message -----
>>>>>>> > From: "Kurt Duncan via Outages" <outages@outages.org>
>>>>>>>
>>>>>>> > Anyone else seeing phone or internet issues due to the AT&T
>>>>>>> building being
>>>>>>> > damaged by the explosion in Nashville ?
>>>>>>>
>>>>>>> Let's please continue to confine replies here to specific
>>>>>>> information about
>>>>>>> network impacts, and keep the speculation threads to -discuss;
>>>>>>> thanks to
>>>>>>> whomever already started that thread. I would name them, but the
>>>>>>> caffeine
>>>>>>> hasn't kicked in yet.
>>>>>>>
>>>>>>> If you're in that building in Nash, we'll be particularly interested
>>>>>>> in
>>>>>>> anything you're allowed to tell us in between bailing.
>>>>>>>
>>>>>>> Cheers,
>>>>>>> -- jr '<admin/>' a
>>>>>>> --
>>>>>>> Jay R. Ashworth Baylink
>>>>>>> jra@baylink.com
>>>>>>> Designer The Things I Think
>>>>>>> RFC 2100
>>>>>>> Ashworth & Associates http://www.bcp38.info 2000
>>>>>>> Land Rover DII
>>>>>>> St Petersburg FL USA BCP38: Ask For It By Name! +1
>>>>>>> 727 647 1274
>>>>>>> _______________________________________________
>>>>>>> Outages mailing list
>>>>>>> Outages@outages.org
>>>>>>> https://puck.nether.net/mailman/listinfo/outages
>>>>>>>
>>>>>> _______________________________________________
>>>>> Outages mailing list
>>>>> Outages@outages.org
>>>>> https://puck.nether.net/mailman/listinfo/outages
>>>>>
>>>>
>> --
>> Sent from my Android device with K-9 Mail. Please excuse my brevity.
>> _______________________________________________
>> Outages mailing list
>> Outages@outages.org
>> https://puck.nether.net/mailman/listinfo/outages
>>
> --
>
>
> *Walt Amason* | Director of Information Technology | Kelley Companies
> *cell* (601) 382-5888 | *office* (601) 735-2541 | *website*
> www.kelleycompanies.com
>