Mailing List Archive

Same. We can't reach any server. You sure it's power vs network?
_______________________________________________
Outages mailing list
Outages@outages.org
https://puck.nether.net/mailman/listinfo/outages

On Mon, Jul 10, 2023, at 19:10, Alex wrote:
> Same. We can't reach any server. You sure it's power vs network?

We saw network loss around 1750 EDT. Things briefly came back, and I saw that my hardware had been hard booted around 1755 EDT.

Lost everything again around 1805 EDT.

-cjp
_______________________________________________
Outages mailing list
Outages@outages.org
https://puck.nether.net/mailman/listinfo/outages

There was a power issue.

On Mon, Jul 10, 2023, 7:18 PM Christopher J. Pilkington via Outages <
outages@outages.org> wrote:

> On Mon, Jul 10, 2023, at 19:10, Alex wrote:
> > Same. We can't reach any server. You sure it's power vs network?
>
> We saw network loss around 1750 EDT. Things briefly came back, and I saw
> that my hardware had been hard booted around 1755 EDT.
>
> Lost everything again around 1805 EDT.
>
> -cjp
> _______________________________________________
> Outages mailing list
> Outages@outages.org
> https://puck.nether.net/mailman/listinfo/outages
>

https://status.dedicated.com/incidents/44

> We've been informed that an electrical room experienced a fire, was put
out by retardant, and the datacenter is in emergency power off status at
the requirement of on-site fire fighters.

On Mon, Jul 10, 2023, 7:23 PM Ross Tajvar <ross@tajvar.io> wrote:

> There was a power issue.
>
> On Mon, Jul 10, 2023, 7:18 PM Christopher J. Pilkington via Outages <
> outages@outages.org> wrote:
>
>> On Mon, Jul 10, 2023, at 19:10, Alex wrote:
>> > Same. We can't reach any server. You sure it's power vs network?
>>
>> We saw network loss around 1750 EDT. Things briefly came back, and I saw
>> that my hardware had been hard booted around 1755 EDT.
>>
>> Lost everything again around 1805 EDT.
>>
>> -cjp
>> _______________________________________________
>> Outages mailing list
>> Outages@outages.org
>> https://puck.nether.net/mailman/listinfo/outages
>>
>

Confirmed, my coworker said he's seeing fire trucks there.

- Alex
_______________________________________________
Outages mailing list
Outages@outages.org
https://puck.nether.net/mailman/listinfo/outages

Hi guys. Has anyone received an update from them?
_______________________________________________
Outages mailing list
Outages@outages.org
https://puck.nether.net/mailman/listinfo/outages

Alex - looks like dedicated.com is still posting good updates.

Everyone else - a friend in the area sent me this picture:
https://imgur.com/a/QsxldPj
He also said it "smells like marshmallows", so...maybe someone is taking
the opportunity to make smores? :)

On Mon, Jul 10, 2023 at 8:26?PM Alex <mr.alexey@gmail.com> wrote:

> Hi guys. Has anyone received an update from them?
>

We received the same update from the datacenter
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Power remains off at our data center in Secaucus/EWR1 per the local fire
marshall.

We have had an electrical failure with one of our redundant UPS' that started to
smoke and then had a small fire in the UPS room. The fire department was
dispatched and the fire was extinguished quickly. The fire department
subsequently cut power to the entire data center and disabled our generators
while they and the utility verify electrical system. We have been working with
both the fire department and the utility to expedite this process.

We are currently waiting on the fire marshall and local utility to reenergize
the site. We are completely dependent upon their inspection and approval. We are
hoping to get an update that we can share in the next hour.

At the current time, the fire department is controlling access to the building
and we will not be able to let customers in.
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
_______________________________________________
Outages mailing list
Outages@outages.org
https://puck.nether.net/mailman/listinfo/outages

Looks like EWR1 will be down at least until the afternoon.

On Tue, Jul 11, 2023, at 08:52, Alex via Outages wrote:
> Looks like EWR1 will be down at least until the afternoon.

Latest update, 9am meet with fire marshal. Best case, customer loads energized 4-5 hours after.

-cjp
_______________________________________________
Outages mailing list
Outages@outages.org
https://puck.nether.net/mailman/listinfo/outages

ROFL we were making the same joke about gracing palms.

The update we received:
"Once we receive approval and utility is restored, we will turn up
critical systems. This will take approximately 5 hours. After the
critical systems are restored, we will be turning up the carriers and
then will start to turn the servers back on."
9am+5=2pm+$carriers_time_to_on, I'm betting on after 5pm counting the
time it will take the inspector to sigh off.

- Alex
_______________________________________________
Outages mailing list
Outages@outages.org
https://puck.nether.net/mailman/listinfo/outages

Utility power has been restored to the site!
The ETA for bringing up the critical infrastructure systems is
approximately 5 hours.
_______________________________________________
Outages mailing list
Outages@outages.org
https://puck.nether.net/mailman/listinfo/outages