Mar 16, 1996, 12:47 PM
Post #6 of 10
(1901 views)
Permalink
Recently, Paul Vixie wrote...
>
> > P.S. what is the deal with the Santa Clara POP anyway? I've seen gas
> > main explosions, circuits cut by a blowtorch (gee? maybe they're related),
> > "missing" fiber, "lost" circuits, and other shenangans.
>
> The story may be growing with the telling, but I was told that a gas main
> ruptured, with or without a small explosion (depends on who you ask), and
> the fire marshall came in, smelled natural fumes all over, and ordered that
> power be cut to the surrounding buildings, presumably to prevent explosions
> due to sparks and such. WilTel ultimately brought in a portable generator
> and put this wonderful new spark generator out in the parking lot, closer
> to the natural gas fumes than anything inside the building would have been.
*sigh*
I'd kinda hoped the issue would die down, but since it won't, I guess
I'll actually speak up.
I was driving to work, running a bit late at around 11:10 am, and turned
west on Walsh Avenue from San Tomas. I pulled into the left turn lane
leading into the business park, and noticed the driveway was blocked
with orange cones and a backhoe digging. A woman in a dark, Pennsylvania
green lincoln towncar pulled up into the left turn lane behind me. When
the left turn arrow turned green, I made a u-turn on Walsh, and entered
the business park via a secondary driveway closer to San Tomas. I glanced
in my rear-view mirror, and saw that the woman in the green car was
stopped midway through the intersection, blocking oncoming traffic,
waiting for the construction crew to move the cones to let her into
the parking lot via that driveway; I guess she figured the earth
rotated around her, so why shouldn't everyone else change to fit?
So, as I pulled into the parking lot in front of work, I looked over
at the entrance, and saw that the construction workers were attempting
to be nice, and were moving the cones out of the way, and trying to
back up the backhoe so she could go past, and stop blocking the
traffic that was now getting very irate. :-) As the worker in
the backhoe was pulling the bucket out of the hole, he must have
hit the main, because there was a very loud "kawoosh" and a blast
of white condesation headed skyward. The construction crew started
running away from the hole, and I headed into work to alert my
coworkers to the problem.
One of my coworkers called PG&E while a second one called 911. PG&E
immediately said they were dispatching a crew to the site, and also
told us to begin shutting down any exterior equipment that was
likely to cause sparks. By this time, the smell of natural gas
was nearly overpowering outside, and was becoming more so
indoors. We began shutting down all the air conditioners on the
roof, and began prepping the site for evacuation. Within about
15 minutes, the fire dept. had arrived, as had PG&E, and began
roping off the area, and had us evacuate the building. We used
cellular phones to call back in, and changed the outgoing
messages for the NOC and tech support ACD queues to alert
customers that we had been evacuated, and could not currently
come to the phones. :-)
PG&E began working on isolating the section of pipe, but within
15 minutes of their arrival, a spark triggered a fireball that
roasted the dump truck, the backhoe, the tree nearby, the signs,
and pretty much everything else in the vicinity. The one good
offshoot of the fireball was that the pipe was actually burning
now, and sending flames ~75 feet in a nice vertical column; the
fire dept. immediately pulled hoses into place, to keep the fire
contained to the single jet, and the nice folks next door at
Wiltel rolled out their 75kVa generator out the BACK of the
building, which was approx 150 yards from the actual gas leak,
and fired it up, since there wasn't much danger of an explosion
now that there was a constant flame source burning off the gas
as it emerged.
Since we couldn't get to our cars, we walked to lunch, and
answered pages from various customers who noticed that
network connectivity was spotty or nonexistent to large
areas, and explained the situation. We got back to
work by about 2:30pm, the fire was still going, but PG&E
was working on clamping off the pipe physically a few
dozen yards upstream; once the pipe was crushed, and the
fire out, it only took about an hour to get clearance from
the incident commander to air out the building, get santa
clara electric to restore power, reset the PDU, reset the
UPS's, and by 3:30 full network connectivity through our
network was restored.
Hopefully this clarifies the situation from a firsthand
witness, and keeps people from blaming Wiltel for sparks
or any other such nonsense.
> > This time I'm
> > glad we followed WorldCom's (WilTel, whatevertheirnameisthisweek) site
> > recommendation and used DC power for our equipment. But this POP is
> > like working out of a telecommunications war zone.
>
> That's one of several good reasons why CIX has installed a router up in
> Digital's Palo Alto facility (I'd call it a NAP, since it has rack space
> for rent and GIGAswitches all over the place, but "NAP" seems to indicate
> congressional funding so I'll choose another word... "Interconnect/CoLo"
> (ICL)). In addition to having both AC and DC on UPS and a generator, Palo
> Alto has actual coffee houses for those late night adventures.
>
Matt Petach