Mailing List Archive

On Thu, Apr 09, 2009 at 08:43:47AM -0400, Joe Abley wrote:
> I am hearing from multiple people about connectivity problems in the bay
> area, and they all seem to have 200 Paul in common. ISC is reporting a
> fibre cut between 529 Bryant, Palo Alto and 200 Paul, SF. At least one
> Unitedlayer customer in 200 Paul seems to be off the air.
>
> Anybody know any specifics?

Only thing I've heard is how Abovenet has a "supposed" fibre cut
somewhere between San Francisco and Los Angeles, which is affecting a
large number of customers in the Bay Area. One of our gigE circuits
utilising that path is hard down (even now).

Could be related, then again might not be.

--
| Jeremy Chadwick jdc at parodius.com |
| Parodius Networking http://www.parodius.com/ |
| UNIX Systems Administrator Mountain View, CA, USA |
| Making life hard for others since 1977. PGP: 4BD6C0CB |

My co-worker tells me there's been an announcement of the below fibre
cut on 810AM radio this morning in the Bay, stating it's affecting AT&T,
Verizon, Verizon Wireless, and other carriers. The physical location
of the cut has not yet been disclosed.

So, the news/media is aware of the problem.

Good to know they all have redundant circuits running through alternate
locations and conduits. *shakes head*

--
| Jeremy Chadwick jdc at parodius.com |
| Parodius Networking http://www.parodius.com/ |
| UNIX Systems Administrator Mountain View, CA, USA |
| Making life hard for others since 1977. PGP: 4BD6C0CB |

On Thu, Apr 09, 2009 at 08:43:47AM -0400, Joe Abley wrote:
> Hey,
>
> I am hearing from multiple people about connectivity problems in the bay
> area, and they all seem to have 200 Paul in common. ISC is reporting a
> fibre cut between 529 Bryant, Palo Alto and 200 Paul, SF. At least one
> Unitedlayer customer in 200 Paul seems to be off the air.
>
> Anybody know any specifics?
>
>
> Joe
> _______________________________________________
> outages mailing list
> outages at outages.org
> https://puck.nether.net/mailman/listinfo/outages

http://twitter.com/BarrettLyon/status/1484862959

Jason Lewis wrote:
> http://twitter.com/BarrettLyon/status/1484862959

http://sandbox.bitgravity.com/blog/2009/04/09/destroy-the-internet-with-a-hacksaw/

> http://sandbox.bitgravity.com/blog/2009/04/09/destroy-the-internet-with-a-hacksaw/

"rather than going down into the manhole....."

Great... give the kids some new ideas. geeze.

-Jim P.

Yeah, I read that and my first thought was "just what we need, people
*explaining* how to cause problems".

I've never been a big fan of publicizing detailed fiber route maps
though, I guess this just gives me more reasons to make that argument...

-Bill


> >
http://sandbox.bitgravity.com/blog/2009/04/09/destroy-the-internet-with-
> a-hacksaw/
>
> "rather than going down into the manhole....."
>
> Great... give the kids some new ideas. geeze.
>
> -Jim P.
> _______________________________________________
> outages mailing list
> outages at outages.org
> https://puck.nether.net/mailman/listinfo/outages

Jim Popovitch wrote:
>> http://sandbox.bitgravity.com/blog/2009/04/09/destroy-the-internet-with-a-hacksaw/
>
> "rather than going down into the manhole....."
>
> Great... give the kids some new ideas. geeze.

Should have waited a bit--an Executive Order would have covered the
situation.

On Thu, Apr 9, 2009 at 3:51 PM, Bill Wichers <billw at waveform.net> wrote:
> Yeah, I read that and my first thought was "just what we need, people
> *explaining* how to cause problems".

Terrorists who want to damage the US and world economies by attacking
the Internet's soft underbelly no doubt prolly already figured it out.

On Thu, Apr 9, 2009 at 1:51 PM, Bill Wichers <billw at waveform.net> wrote:
> Yeah, I read that and my first thought was "just what we need, people
> *explaining* how to cause problems".
>
> I've never been a big fan of publicizing detailed fiber route maps
> though, I guess this just gives me more reasons to make that argument...

who needs detailed fiber maps when you can have the 1-800-DIG people
come out and spray paint the important routes for you?
people who want to destroy things will figure out how to destroy them.
hack saw, gasoline, or otherwise.

maybe we should lay so much fiber it wont matter what they firebomb (-:

Most likely yes, but I was thinking mostly of the home-grown
troublemakers that have always been around. Somehow telecom seems to be
mostly immune from them.

Since there really is no practical way to secure the outside plant (and
to the one that mentioned lockable manhole covers -- have a look at the
"key" to those sometime, there isn't very much security there), the only
thing we can do to keep it safe is to not make a big deal about where it
is or what can harm it.

-Bill

> On Thu, Apr 9, 2009 at 3:51 PM, Bill Wichers <billw at waveform.net>
wrote:
> > Yeah, I read that and my first thought was "just what we need,
people
> > *explaining* how to cause problems".
>
> Terrorists who want to damage the US and world economies by attacking
> the Internet's soft underbelly no doubt prolly already figured it out.
> _______________________________________________
> outages mailing list
> outages at outages.org
> https://puck.nether.net/mailman/listinfo/outages

On Thu, Apr 9, 2009 at 4:20 PM, Aaron Glenn <aaron.glenn at gmail.com> wrote:
>
> maybe we should lay so much fiber it wont matter what they firebomb (-:

We could always clawback the USF funds and require they be used
sensibly, as in your "lay so much fiber it wont matter."

/lurk mode off

I agree!

A determined person will get the information and wreck havoc regardless of how well it is hidden!

/lurk mode on



Thanks,
Jim

Please excuse the typos. Sent from my wireless handheld device.

----- Original Message -----
From: outages-bounces@outages.org <outages-bounces@outages.org>
To: outages at outages.org <outages at outages.org>
Sent: Thu Apr 09 17:20:56 2009
Subject: Re: [outages] fibre cut near 200 Paul, San Francisco

On Thu, Apr 9, 2009 at 1:51 PM, Bill Wichers <billw at waveform.net> wrote:
> Yeah, I read that and my first thought was "just what we need, people
> *explaining* how to cause problems".
>
> I've never been a big fan of publicizing detailed fiber route maps
> though, I guess this just gives me more reasons to make that argument...

who needs detailed fiber maps when you can have the 1-800-DIG people
come out and spray paint the important routes for you?
people who want to destroy things will figure out how to destroy them.
hack saw, gasoline, or otherwise.

maybe we should lay so much fiber it wont matter what they firebomb (-:
_______________________________________________
outages mailing list
outages at outages.org
https://puck.nether.net/mailman/listinfo/outages

> the only thing we can do to keep it safe is to not make a big deal about
where it is or what can harm it.

Agreed. Then we'll always know it's only the CWA members who are ever to
blame, like when they did this in my area in '04:

http://www.cwalocal4250.org/operatorservices/binarydata/Meet%20With%20Mediat
or.pdf

Of course, after that one, the police were investigating it as a "Terrorist
Activity" under the "Patriot Act", but no arrests were made in our area,
despite the eyewitness who practically saw it happen...

Any way we can petition the git with the blog entry to pull that map of his
page? Geez!

Kirk Coviello
Digital West Networks, Inc.
V: 805.781.WEST (9378)
Email: kirk at digitalwest.net

Bill Wichers wrote:
> Most likely yes, but I was thinking mostly of the home-grown
> troublemakers that have always been around. Somehow telecom seems to be
> mostly immune from them.
>
> Since there really is no practical way to secure the outside plant (and
> to the one that mentioned lockable manhole covers -- have a look at the
> "key" to those sometime, there isn't very much security there), the only
> thing we can do to keep it safe is to not make a big deal about where it
> is or what can harm it.

Some news reports suggest this may be and "inside" (so to speak) job.

I have no facts, beyond what I have seen here.

On Thu, 9 Apr 2009, Joe Abley wrote:

> I am hearing from multiple people about connectivity problems in the bay
> area, and they all seem to have 200 Paul in common. ISC is reporting a
> fibre cut between 529 Bryant, Palo Alto and 200 Paul, SF. At least one
> Unitedlayer customer in 200 Paul seems to be off the air.

We're showing spikes in latency out in LAX on Cogent, as well as a few
other interesting anomalies that are consistent with shifts in traffic in
a number of different locations. Has anyone noticed any network routing
issues outside of the immediately-effected area, as a result of this?

9 gi0-0-0.core01.lax05.atlas.cogentco.com (154.54.6.185)
65.447 ms 65.449 ms 65.434 ms
10 xo.lax05.atlas.cogentco.com (154.54.11.242)
127.708 ms 127.683 ms 127.680 ms



--
William R. Lorenz

On Thu, Apr 09, 2009 at 04:42:53PM -0500, Larry Sheldon wrote:
> Bill Wichers wrote:
>> Most likely yes, but I was thinking mostly of the home-grown
>> troublemakers that have always been around. Somehow telecom seems to be
>> mostly immune from them.
>>
>> Since there really is no practical way to secure the outside plant (and
>> to the one that mentioned lockable manhole covers -- have a look at the
>> "key" to those sometime, there isn't very much security there), the only
>> thing we can do to keep it safe is to not make a big deal about where it
>> is or what can harm it.
>
> Some news reports suggest this may be and "inside" (so to speak) job.
>
> I have no facts, beyond what I have seen here.

CWA claims no involvement:
http://www.msnbc.msn.com/id/30139776/

A CWA member states "only a nut would do this":
http://www.dslreports.com/shownews/ATT-Fiber-Cut-Impacts-Silicon-Valley-101827

Video of the cable cut itself indicates thieves may have been out
"cutting for copper". It's quite a clean cut (01:48 to 2:02):
http://cbs5.com/video/?id=48654 at kpix.dayport.com

The video may also help someone determine the exact location and
which of the two cuts it was.

--
| Jeremy Chadwick jdc at parodius.com |
| Parodius Networking http://www.parodius.com/ |
| UNIX Systems Administrator Mountain View, CA, USA |
| Making life hard for others since 1977. PGP: 4BD6C0CB |

As fashionable as it is to be paranoid, this does seem to fit the 'never
attribute to malice that which can be more easily explained by
stupidity' realm. IIRC, there have been a number of cases recently where
various fiber trunks (some of them aerial runs) have been cut by
miscreants looking for copper - they must have been so disappointed.....

Jeremy Chadwick wrote:
> Video of the cable cut itself indicates thieves may have been out
> "cutting for copper". It's quite a clean cut (01:48 to 2:02):
> http://cbs5.com/video/?id=48654 at kpix.dayport.com
>

--
Scanned by MailScanner

Jeremy Chadwick writes:
>Video of the cable cut itself indicates thieves may have been out
>"cutting for copper". It's quite a clean cut (01:48 to 2:02):
>http://cbs5.com/video/?id=48654 at kpix.dayport.com
>
>The video may also help someone determine the exact location and
>which of the two cuts it was.

Clearly not the San Carlos cut - seems to be very close to
W121.80542
N37.25718

Beginning of the ramp up from southbound Monterey Highway to
Westbound Blossom Hill Road.

There's a tree there in Google Earth - but you can see pieces of
cut down tree in the video, and the lightpole which is in the nearby
background up the ramp.


-george william herbert
gherbert at retro.com

I agree too, I could give my kids a box of tooth picks and take out
racks of switches and router by locking the fans, why not put the vaults
4-6 feet down under dirt?

-----Original Message-----
From: outages-bounces@outages.org [mailto:outages-bounces@outages.org]
On Behalf Of Jim McBurnett
Sent: Thursday, April 09, 2009 5:38 PM
To: aaron.glenn at gmail.com; outages at outages.org
Subject: Re: [outages] fibre cut near 200 Paul, San Francisco

/lurk mode off

I agree!

A determined person will get the information and wreck havoc regardless
of how well it is hidden!

/lurk mode on



Thanks,
Jim

Please excuse the typos. Sent from my wireless handheld device.

----- Original Message -----
From: outages-bounces@outages.org <outages-bounces@outages.org>
To: outages at outages.org <outages at outages.org>
Sent: Thu Apr 09 17:20:56 2009
Subject: Re: [outages] fibre cut near 200 Paul, San Francisco

On Thu, Apr 9, 2009 at 1:51 PM, Bill Wichers <billw at waveform.net> wrote:
> Yeah, I read that and my first thought was "just what we need, people
> *explaining* how to cause problems".
>
> I've never been a big fan of publicizing detailed fiber route maps
> though, I guess this just gives me more reasons to make that
argument...

who needs detailed fiber maps when you can have the 1-800-DIG people
come out and spray paint the important routes for you?
people who want to destroy things will figure out how to destroy them.
hack saw, gasoline, or otherwise.

maybe we should lay so much fiber it wont matter what they firebomb (-:
_______________________________________________
outages mailing list
outages at outages.org
https://puck.nether.net/mailman/listinfo/outages
_______________________________________________
outages mailing list
outages at outages.org
https://puck.nether.net/mailman/listinfo/outages

On Thu, Apr 9, 2009 at 17:26, Bill Wichers <billw at waveform.net> wrote:
> Most likely yes, but I was thinking mostly of the home-grown
> troublemakers that have always been around.

As was i, that and the slacker crowd looking for something to do on a
slow day. The issue wasn't whether security by obscurity is
effective, it was about giving ideas to those who don't need any
encouragement.

-Jim P.

Aaron Glenn wrote:
> On Thu, Apr 9, 2009 at 1:51 PM, Bill Wichers <billw at waveform.net> wrote:
>> Yeah, I read that and my first thought was "just what we need, people
>> *explaining* how to cause problems".
>>
>> I've never been a big fan of publicizing detailed fiber route maps
>> though, I guess this just gives me more reasons to make that argument...
>
> who needs detailed fiber maps when you can have the 1-800-DIG people
> come out and spray paint the important routes for you?
> people who want to destroy things will figure out how to destroy them.
> hack saw, gasoline, or otherwise.
>

Even easier: walk down the street and look for anything visibly marked
with bright orange. Or look in manholes marked as telcom for the same.
It's likely to be fiber.

~Seth

has anyone been able to pin point the cut?
There have been mentions of Redwood City, San Carlos, San Francisco.
Where exactly is the cut?



William R. Lorenz wrote:
> On Thu, 9 Apr 2009, Joe Abley wrote:
>
>> I am hearing from multiple people about connectivity problems in the
>> bay area, and they all seem to have 200 Paul in common. ISC is
>> reporting a fibre cut between 529 Bryant, Palo Alto and 200 Paul, SF.
>> At least one Unitedlayer customer in 200 Paul seems to be off the air.
>
> We're showing spikes in latency out in LAX on Cogent, as well as a few
> other interesting anomalies that are consistent with shifts in traffic in
> a number of different locations. Has anyone noticed any network routing
> issues outside of the immediately-effected area, as a result of this?
>
> 9 gi0-0-0.core01.lax05.atlas.cogentco.com (154.54.6.185)
> 65.447 ms 65.449 ms 65.434 ms
> 10 xo.lax05.atlas.cogentco.com (154.54.11.242)
> 127.708 ms 127.683 ms 127.680 ms
>
>
>
> --
> William R. Lorenz
>

On Thu, Apr 9, 2009 at 2:42 PM, Kirk Coviello <kcoviello at digitalwest.net> wrote:
>
> Any way we can petition the git with the blog entry to pull that map of his
> page? ?Geez!
>

for someone looking to damage fiber paths, it's painfully easy (but
never impresses dates) to walk/drive around and physically trace them
yourself. hiding maps is security theater

The reports seem to be settling on 2 distinct sites -- South
San Jose and San Carlos. The former had 4 cuts and latter
just 1.

-r

On Thu, Apr 09, 2009 at 06:27:53PM -0700, John Martinez wrote:
> has anyone been able to pin point the cut?
> There have been mentions of Redwood City, San Carlos, San Francisco.
> Where exactly is the cut?
>
>
>
> William R. Lorenz wrote:
> > On Thu, 9 Apr 2009, Joe Abley wrote:
> >
> >> I am hearing from multiple people about connectivity problems in the
> >> bay area, and they all seem to have 200 Paul in common. ISC is
> >> reporting a fibre cut between 529 Bryant, Palo Alto and 200 Paul, SF.
> >> At least one Unitedlayer customer in 200 Paul seems to be off the air.
> >
> > We're showing spikes in latency out in LAX on Cogent, as well as a few
> > other interesting anomalies that are consistent with shifts in traffic in
> > a number of different locations. Has anyone noticed any network routing
> > issues outside of the immediately-effected area, as a result of this?
> >
> > 9 gi0-0-0.core01.lax05.atlas.cogentco.com (154.54.6.185)
> > 65.447 ms 65.449 ms 65.434 ms
> > 10 xo.lax05.atlas.cogentco.com (154.54.11.242)
> > 127.708 ms 127.683 ms 127.680 ms
> >
> >
> >
> > --
> > William R. Lorenz
> >
>