Mailing List Archive

1 2  View All
Re: Never push the Big Red Button (New York City subway failure) [ In reply to ]
I once worked for a provider who had a company next door that ran a small datacenter of about a dozen or so racks. They had been sold and all of their infrastructure had been virtualized and moved to the new owner?s network. The last task of the local admin was to just get rid of everything. They didn?t care how just get it gone. So he came over and asked us to come take a look and we could have anything we wanted. I picked up a few servers for lab a bunch of racks and stuff.

While we were working I asked the guy ?So there is absolutely nothing that?s in production in here anymore?? He said ?Yep? so I asked ?Then if the power went off in here it wouldn?t be a big deal? and he said ?Not at all?. Then I asked ?Can I hit the red button?? He said ?Sure, I always wondered what happened?. I hit the button and with a loud booming sound the room went dead silent and then the UPS started beeping. It was at that moment everyone realized that you just don?t pull the button out to restart the room. It took us 20 minutes to figure out how to turn it all back on.

And with that when I got back to our office I made sure someone knew how to restart everything if we ever had to hit our red button.


-richey

From: NANOG <nanog-bounces+richey.goldberg=gmail.com@nanog.org> on behalf of Roy <r.engehausen@gmail.com>
Date: Thursday, September 16, 2021 at 12:41 AM
To: nanog <nanog@nanog.org>
Subject: Re: Never push the Big Red Button (New York City subway failure)
Miy story in the late 1970s I was working in a large computer facility
with both mainframes and mil-spec 400hz computers.
Management decided that the EPO should be tested. So we powered down
the disk and tapes. The electrician pressed
the EPO button and NOTHING. Everything kept running.

Turns out a wire had come loose and the fuse in the EPO circuit had blown.

Roy
Re: Never push the Big Red Button (New York City subway failure) [ In reply to ]
Sigh, people often mis-hear this when I say it, so I will try to say it
carefully.


If you have an Emergency Power Off (EPO), the electrical code (and
life-safety code) allows use of several alternative wiring methods. Some
people mistakenly believe the allowed alternatives are the rule, but they
are actually exceptions.

On the other hand....

If you do NOT have an Emergency Power Off (EPO), you are NOT allowed to
use the associated alternatives in the electrical code. Among the
alternatives NOT allowed without the EPO, is almost everything in
Article 645 - Information Technology Equipment rooms (or the equivalent
in international electrical codes). Most people, including licensed
electricians, believe it is the other way around.

Why does Article 645 exist? (and equivalents in other international
electrical codes)

IBM in the 1950s published construction specifications for Automatic Data
Processing rooms for its mainframe computers, which everyone else copied.
IBM wanted to use alternative wiring methods in ADP rooms for its
mainframe computers, so the Big Red Button was born. Also, mainframe
computers used to cost more than the building, so people (insurance
companies) didn't want the mainframe damaged during a fire.

It is possible to design a data center WITHOUT using those electrical
code exceptions, and WITHOUT a "Big Red Button."


You can check, because my data center ideas were copied by several
tech companies world-wide (you know who you are), and don't have Big Red
Buttons. All of those data centers also have water-based automatic fire
sprinklers. Both were very radical ideas at the time, which are now
commonly accepted.


In most cases, you'll need a fully licensed, Professional Engineer
specializing in Electrical Engineering to sign off on the final design.
A licensed electrician isn't enough. Nevertheless, it is possible to
build a safe, code-compliant data center WITHOUT a Big Red Button. The
design also seemed to be more reliable.


Let the misinterpretation begin ...
Re: Never push the Big Red Button (New York City subway failure) [ In reply to ]
> On Sep 17, 2021, at 8:59 AM, Sean Donelan <sean@donelan.com> wrote:
>
> It is possible to design a data center WITHOUT using those electrical code exceptions, and WITHOUT a "Big Red Button."
>
> You can check, because my data center ideas were copied by several tech companies world-wide (you know who you are), and don't have Big Red Buttons. All of those data centers also have water-based automatic fire sprinklers. Both were very radical ideas at the time, which are now commonly accepted.
>
> In most cases, you'll need a fully licensed, Professional Engineer specializing in Electrical Engineering to sign off on the final design. A licensed electrician isn't enough. Nevertheless, it is possible to build a safe, code-compliant data center WITHOUT a Big Red Button. The design also seemed to be more reliable.

What’s the gain in _not_ having one that makes it worth the sign-off and hassle? Just avoiding the possibility of accidental activation or something I’m not thinking of?

1 2  View All