Mailing List Archive

http://today.reuters.com/news/articleinvesting.aspx?type=bondsNews&storyID=2007-07-24T214600Z_01_N24247688_RTRIDST_0_UTILITIES-SANFRANCISCO-OUTAGES-UPDATE-1.XML

SAN FRANCISCO, July 24 (Reuters) - Sporadic power outages hit San
Francisco on Tuesday afternoon, affecting widespread areas including
the city's downtown financial district.

About 28,000 customers in San Francisco and an area south of the city
were affected, said Darlene Chiu, spokeswoman for Pacific Gas &
Electric Co., the main utility for the city. More than 20,000 of those
outages were in San Francisco, she said.

"We are investigating the cause of it," said Chiu, adding that the
utility is having on-and-off success in reconnecting customers by
backup circuits.

Traffic signals in much of downtown San Francisco were not working.
Bay Area Rapid Transit trains were moving, although some of the lights
in undergound stations were off, a BART spokesman told local radio.

The outages began about 2 p.m. PDT, Chiu said.

The utility is owned by PG&E Corp.




On 7/24/07, Rusty Hodge <rusty at hodge.com> wrote:
> My cabinet as well as some other customers I know over at 365Main
> (SF, CA) had a power interruption at about 1:50pm pacific today.
>
> Power is back up now.
> _______________________________________________
> Outages mailing list
> Outages at isotf.org
> http://isotf.org/mailman/listinfo/outages
>

There Is a line out the door to get access to 365main right now

On Jul 24, 2007, at 3:16 PM, "Adam Kujawski" <adamkuj at gmail.com> wrote:

> http://today.reuters.com/news/articleinvesting.aspx?type=bondsNews&storyID=2007-07-24T214600Z_01_N24247688_RTRIDST_0_UTILITIES-SANFRANCISCO-OUTAGES-UPDATE-1.XML
>
> SAN FRANCISCO, July 24 (Reuters) - Sporadic power outages hit San
> Francisco on Tuesday afternoon, affecting widespread areas including
> the city's downtown financial district.
>
> About 28,000 customers in San Francisco and an area south of the city
> were affected, said Darlene Chiu, spokeswoman for Pacific Gas &
> Electric Co., the main utility for the city. More than 20,000 of those
> outages were in San Francisco, she said.
>
> "We are investigating the cause of it," said Chiu, adding that the
> utility is having on-and-off success in reconnecting customers by
> backup circuits.
>
> Traffic signals in much of downtown San Francisco were not working.
> Bay Area Rapid Transit trains were moving, although some of the lights
> in undergound stations were off, a BART spokesman told local radio.
>
> The outages began about 2 p.m. PDT, Chiu said.
>
> The utility is owned by PG&E Corp.
>
>
>
>
> On 7/24/07, Rusty Hodge <rusty at hodge.com> wrote:
>> My cabinet as well as some other customers I know over at 365Main
>> (SF, CA) had a power interruption at about 1:50pm pacific today.
>>
>> Power is back up now.
>> _______________________________________________
>> Outages mailing list
>> Outages at isotf.org
>> http://isotf.org/mailman/listinfo/outages
>>
>

Gotta love it:

http://valleywag.com/tech/breakdowns/a-drunk-employee-kills-all-of-the-websites-you-care-about-282021.php

-Mike


On 7/24/07, Rusty Hodge <rusty at hodge.com> wrote:
> There Is a line out the door to get access to 365main right now
>
> On Jul 24, 2007, at 3:16 PM, "Adam Kujawski" <adamkuj at gmail.com> wrote:
>
> > http://today.reuters.com/news/articleinvesting.aspx?type=bondsNews&storyID=2007-07-24T214600Z_01_N24247688_RTRIDST_0_UTILITIES-SANFRANCISCO-OUTAGES-UPDATE-1.XML
> >
> > SAN FRANCISCO, July 24 (Reuters) - Sporadic power outages hit San
> > Francisco on Tuesday afternoon, affecting widespread areas including
> > the city's downtown financial district.
> >
> > About 28,000 customers in San Francisco and an area south of the city
> > were affected, said Darlene Chiu, spokeswoman for Pacific Gas &
> > Electric Co., the main utility for the city. More than 20,000 of those
> > outages were in San Francisco, she said.
> >
> > "We are investigating the cause of it," said Chiu, adding that the
> > utility is having on-and-off success in reconnecting customers by
> > backup circuits.
> >
> > Traffic signals in much of downtown San Francisco were not working.
> > Bay Area Rapid Transit trains were moving, although some of the lights
> > in undergound stations were off, a BART spokesman told local radio.
> >
> > The outages began about 2 p.m. PDT, Chiu said.
> >
> > The utility is owned by PG&E Corp.
> >
> >
> >
> >
> > On 7/24/07, Rusty Hodge <rusty at hodge.com> wrote:
> >> My cabinet as well as some other customers I know over at 365Main
> >> (SF, CA) had a power interruption at about 1:50pm pacific today.
> >>
> >> Power is back up now.
> >> _______________________________________________
> >> Outages mailing list
> >> Outages at isotf.org
> >> http://isotf.org/mailman/listinfo/outages
> >>
> >
> _______________________________________________
> Outages mailing list
> Outages at isotf.org
> http://isotf.org/mailman/listinfo/outages
>

Adam Kujawski wrote:
> http://today.reuters.com/news/articleinvesting.aspx?type=bondsNews&storyID=2007-07-24T214600Z_01_N24247688_RTRIDST_0_UTILITIES-SANFRANCISCO-OUTAGES-UPDATE-1.XML
>
> SAN FRANCISCO, July 24 (Reuters) - Sporadic power outages hit San
> Francisco on Tuesday afternoon, affecting widespread areas including
> the city's downtown financial district.
>
> About 28,000 customers in San Francisco and an area south of the city
> were affected, said Darlene Chiu, spokeswoman for Pacific Gas &
> Electric Co., the main utility for the city. More than 20,000 of those
> outages were in San Francisco, she said.
>
> "We are investigating the cause of it," said Chiu, adding that the
> utility is having on-and-off success in reconnecting customers by
> backup circuits.
>
> Traffic signals in much of downtown San Francisco were not working.
> Bay Area Rapid Transit trains were moving, although some of the lights
> in undergound stations were off, a BART spokesman told local radio.
>
> The outages began about 2 p.m. PDT, Chiu said.
>
> The utility is owned by PG&E Corp.
>

Poor excuse. Aren't they supposed to have generators/batteries? Does
anybody seriously accept "power outages" as an excuse your servers went
offline? I wouldn't. What are you actually paying them for if you do?

~Seth

Heh,

They have pretty anal security people there, so I tend to disbelieve,
but I guess anything is possible.

We lost power to all circuits in one of the colo-rooms there, but not
another, so it may just be more issues with their power distribution gear.

They dropped us and a bunch of other customers for several hours in a
different colo-room last year when a circuit breaker failed.

-Allen

Mike Lyon wrote:
> Gotta love it:
>
> http://valleywag.com/tech/breakdowns/a-drunk-employee-kills-all-of-the-websites-you-care-about-282021.php
>
> -Mike
>
>
> On 7/24/07, Rusty Hodge <rusty at hodge.com> wrote:
>
>> There Is a line out the door to get access to 365main right now
>>
>> On Jul 24, 2007, at 3:16 PM, "Adam Kujawski" <adamkuj at gmail.com> wrote:
>>
>>
>>> http://today.reuters.com/news/articleinvesting.aspx?type=bondsNews&storyID=2007-07-24T214600Z_01_N24247688_RTRIDST_0_UTILITIES-SANFRANCISCO-OUTAGES-UPDATE-1.XML
>>>
>>> SAN FRANCISCO, July 24 (Reuters) - Sporadic power outages hit San
>>> Francisco on Tuesday afternoon, affecting widespread areas including
>>> the city's downtown financial district.
>>>
>>> About 28,000 customers in San Francisco and an area south of the city
>>> were affected, said Darlene Chiu, spokeswoman for Pacific Gas &
>>> Electric Co., the main utility for the city. More than 20,000 of those
>>> outages were in San Francisco, she said.
>>>
>>> "We are investigating the cause of it," said Chiu, adding that the
>>> utility is having on-and-off success in reconnecting customers by
>>> backup circuits.
>>>
>>> Traffic signals in much of downtown San Francisco were not working.
>>> Bay Area Rapid Transit trains were moving, although some of the lights
>>> in undergound stations were off, a BART spokesman told local radio.
>>>
>>> The outages began about 2 p.m. PDT, Chiu said.
>>>
>>> The utility is owned by PG&E Corp.
>>>
>>>
>>>
>>>
>>> On 7/24/07, Rusty Hodge <rusty at hodge.com> wrote:
>>>
>>>> My cabinet as well as some other customers I know over at 365Main
>>>> (SF, CA) had a power interruption at about 1:50pm pacific today.
>>>>
>>>> Power is back up now.
>>>> _______________________________________________
>>>> Outages mailing list
>>>> Outages at isotf.org
>>>> http://isotf.org/mailman/listinfo/outages
>>>>
>>>>
>> _______________________________________________
>> Outages mailing list
>> Outages at isotf.org
>> http://isotf.org/mailman/listinfo/outages
>>
>>
> _______________________________________________
> Outages mailing list
> Outages at isotf.org
> http://isotf.org/mailman/listinfo/outages
>

> Poor excuse. Aren't they supposed to have generators/batteries? Does
> anybody seriously accept "power outages" as an excuse your servers
> went
> offline? I wouldn't. What are you actually paying them for if you do?

they do. my analysis of the events that led up to this failure:

http://somafm.com/blogs/rusty/2007/07/san-francisco-power-outage-
somafm.html

We (SomaFM.com) are in Colo 7.

Quoted:

"It was reported there were several brief outages in a row before
the power went out for good, so I bet the CPS (flywheel) systems weren't
fully back up to speed when the next sequential outage occurred. Since
several of these grid power interruption happened in a row, and were shorter
than the time required to trigger generator startup, the generators were not
automatically started, BUT the CPS didn't have time to get back up to full
capacity. By the 6th power glitch, there wasn't enough energy stored in the
flywheels to keep the system going long enough for the diesel generators to
start up and come to speed before switching over."

Ladies and Gentlemen, this is why you must program your power control
systems and/or auto transfer switches to be committed to transfer, even if
utility returns. This is almost ATS 101.

This same situation can occur in typical UPS systems -- several small hits
can cause a significant drain on batteries, but return quick enough to not
transfer a to-generator event.

You should:

a) ensure a generator start, no matter how short the outage duration
b) commit a transfer to generator even if utility has returned
c) stay on generator for a minimum of (some period of time) -- 15
minutes, imho.


> they do. my analysis of the events that led up to this failure:
>
>
http://somafm.com/blogs/rusty/2007/07/san-francisco-power-outage-somafm.html
>
> We (SomaFM.com) are in Colo 7.