On Fri, Sep 10, 2021 at 2:52 PM Christopher Morrow <morrowc.lists@gmail.com>
wrote:
>
>
> On Fri, Sep 10, 2021 at 1:49 PM Matthew Huff <mhuff@ox.com> wrote:
>
>> Reminds me of something that happened about 25 years ago when an
>> elementary school visited our data center of the insurance company where I
>> worked. One of our operators strategically positioned himself between the
>> kids and the mainframe, leaned back and hit it's EPO button.
>>
>>
> Or when your building engineering team cuts themselves a new key for the
> 'main breaker' for the facility... and tests it at 2pm on a tuesday.
> Or when that same team cuts a second key (gotta have 2 keys!) and tests
> that key on the same 'main breaker' ... at 2pm on the following tuesday.
>
> <quadruple face palm>
>
> not fakenews, a real story from a large building full of gov't employees
> and computers and all manner of 'critical infrastructure' for the agency
> occupying said building.
>
In the early 2000s a friend of mine worked for a company in NYC
that provided stock feeds to large banks and brokerages and similar. They'd
ship a (locked) cabinet full of stuff to their customers, complete with an
Ethernet cable stickin' out the back. Customer would plug this into their
network and, um, do whatever it is stock people do. There was some
horrendously expensive SLA attached, and so they outsourced support to one
of the managed services companies so that they could provide 24x7x2hour
response all over the country.
One day, one of their largest customers, a large bank, also in NYC is down.
This means that the brokerage arm is unable to do any trades, and so is,
um, annoyed. Support rushes over to the customer and "fix it". My friend
doesn't really get a good explanation of how it got fixed, but, meh, it's
working, so all good. A few weeks later, same thing - customer devices
disappear from monitoring, smart-hands/support rush over and fix it, no
useful RFO. This happens a few more times, and everyone is getting
increasingly annoyed.
Eventually my friend arranges it so that he gets paged at the same time as
the support provider. Pager goes off, friend jumps in a cab to the
customer. He arrives at the same time as the smart-hands person, who,
oddly, is clutching 1: an Ethernet face-plate and 2: a punch-down tool.
Somewhat mystified, my friend follows the support person to where the
cabinet is located. Because it's important and special, but not actually
bank owned, it cannot live in their data-center... and so it is located in
the corridor, just outside the server room.
Because of where the Ethernet cable comes out the back of the cabinet, and
where the wall jack is, there is basically no slack. When someone goes in
or out, especially if they are wheeling a cart or carrying a box of
equipment, they bang into the cabinet, which slowly rolls away -- ripping
the wall jack off the wall, and the cable out the back of the jack.
Support's "solution" to this has been to punch down the cable onto a new
wall jack, screw it back onto the wall, wheel the cabinet back into place,
and call it fixed.
My friend screwed down the cabinet feet, so it wasn't resting on the wheels
any more, replaced the 6ft Ethernet with a 15ft, and the issue never
recurred :-P
W
>
> Matthew Huff | Director of Technical Operations | OTA Management LLC
>>
>> Office: 914-460-4039
>> mhuff@ox.com | www.ox.com
>>
>> ...........................................................................................................................................
>>
>> -----Original Message-----
>> From: NANOG <nanog-bounces+mhuff=ox.com@nanog.org> On Behalf Of Sean
>> Donelan
>> Sent: Friday, September 10, 2021 12:38 PM
>> To: nanog@nanog.org
>> Subject: Never push the Big Red Button (New York City subway failure)
>>
>> NEW YORK CITY TRANSIT RAIL CONTROL CENTER POWER
>> OUTAGE ISSUE ON AUGUST 29, 2021
>> Key Findings
>> September 8, 2021
>>
>>
>>
>> https://www.governor.ny.gov/sites/default/files/2021-09/WSP_Key_Findings_Summary-for_release.pdf
>>
>> Key Findings
>> [...]
>>
>> 3. Based on the electrical equipment log readings and the manufacturer’s
>> official assessment, it was determined that the most likely cause of RCC
>> shutdown was the “Emergency Power Off” button being manually activated.
>>
>> Secondary Findings
>>
>> 1. The “Emergency Power Off” button did not have a protective cover at
>> the
>> time of the shutdown or the following WSP investigation.
>>
>> [...]
>> Mitigation Steps
>>
>> 1. Set up the electrical equipment Control and Communication systems
>> properly to stay active so that personnel can monitor RCC electrical
>> system operations.
>>
>> [...]
>>
>
--
The computing scientist’s main challenge is not to get confused by the
complexities of his own making.
-- E. W. Dijkstra