Mailing List Archive

Have numerous customer sites down as well...have been told by VzB that there is a major fiber cut in their network.

We got this message from our account rep:

"We are have a catastrophic event (fiber cut) going in Riverdale, IL"

Not sure if it's related

On Thu, Aug 26, 2010 at 12:54 PM, notifications <notifications at mattfilip.com
> wrote:

> We currently have at least 5 sites down on Verizon Business MPLS in the
> Midwest. (Chicago, Milwaukee, Indy) We?ve been on hold with TAC for 30+
> minutes. Any info is appreciated.
>
> _______________________________________________
> Outages mailing list
> Outages at outages.org
> https://puck.nether.net/mailman/listinfo/outages
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://puck.nether.net/pipermail/outages/attachments/20100826/9073b303/attachment.html>

Yeah, this is it. Took us awhile to get info. Their IVR didn't report a
fiber cut, just that they were working on it.



From: Kenny Sallee [mailto:kenny.sallee@gmail.com]
Sent: Thursday, August 26, 2010 3:48 PM
To: notifications
Cc: outages at outages.org
Subject: Re: [outages] Verizon Business



We got this message from our account rep:



"We are have a catastrophic event (fiber cut) going in Riverdale, IL"



Not sure if it's related

On Thu, Aug 26, 2010 at 12:54 PM, notifications
<notifications at mattfilip.com> wrote:

We currently have at least 5 sites down on Verizon Business MPLS in the
Midwest. (Chicago, Milwaukee, Indy) We've been on hold with TAC for 30+
minutes. Any info is appreciated.


_______________________________________________
Outages mailing list
Outages at outages.org
https://puck.nether.net/mailman/listinfo/outages



No virus found in this incoming message.
Checked by AVG - www.avg.com
Version: 9.0.851 / Virus Database: 271.1.1/3095 - Release Date: 08/26/10
01:34:00

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://puck.nether.net/pipermail/outages/attachments/20100826/9b15aac9/attachment-0001.html>

Catastrophic event. Interesting language for a fibre cut?

~ jason
> We got this message from our account rep:
>
> "We are have a catastrophic event (fiber cut) going in Riverdale, IL"
>
> Not sure if it's related
>
> On Thu, Aug 26, 2010 at 12:54 PM, notifications <
notifications at mattfilip.com
>> wrote:
>
>> We currently have at least 5 sites down on Verizon Business MPLS in the
>> Midwest. (Chicago, Milwaukee, Indy) We?ve been on hold with TAC for 30+
>> minutes. Any info is appreciated.
>>
>> _______________________________________________
>> Outages mailing list
>> Outages at outages.org
>> https://puck.nether.net/mailman/listinfo/outages
>>
>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://puck.nether.net/pipermail/outages/attachments/20100826/6a328b28/attachment.html>

On Thu, Aug 26, 2010 at 2:10 PM, Jason L. Sparks <jlsparks at gmail.com> wrote:

> Catastrophic event. Interesting language for a fibre cut?
>
> ~ jason
>

Guess it depends on your perspective.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://puck.nether.net/pipermail/outages/attachments/20100826/63df71b6/attachment.html>

No kidding. Someone should ask that rep, if they have ever heard of MPLS
Fast Re-Route. Not that hard...

On Thu, Aug 26, 2010 at 2:10 PM, Jason L. Sparks <jlsparks at gmail.com> wrote:

> Catastrophic event. Interesting language for a fibre cut?
>
> ~ jason
> > We got this message from our account rep:
> >
> > "We are have a catastrophic event (fiber cut) going in Riverdale, IL"
> >
> > Not sure if it's related
> >
> > On Thu, Aug 26, 2010 at 12:54 PM, notifications <
> notifications at mattfilip.com
> >> wrote:
> >
> >> We currently have at least 5 sites down on Verizon Business MPLS in the
> >> Midwest. (Chicago, Milwaukee, Indy) We?ve been on hold with TAC for 30+
> >> minutes. Any info is appreciated.
> >>
> >> _______________________________________________
> >> Outages mailing list
> >> Outages at outages.org
> >> https://puck.nether.net/mailman/listinfo/outages
> >>
> >>
>
> _______________________________________________
> Outages mailing list
> Outages at outages.org
> https://puck.nether.net/mailman/listinfo/outages
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://puck.nether.net/pipermail/outages/attachments/20100826/fdd3c915/attachment.html>

On Aug 26, 2010, at 4:10 PM, Jason L. Sparks wrote:

> Catastrophic event. Interesting language for a fibre cut?
>
> ~ jason
>
> > We got this message from our account rep:
> >
> > "We are have a catastrophic event (fiber cut) going in Riverdale, IL"
> >
> > Not sure if it's related
> >
> > On Thu, Aug 26, 2010 at 12:54 PM, notifications <notifications at mattfilip.com
> >> wrote:
> >
> >> We currently have at least 5 sites down on Verizon Business MPLS in the
> >> Midwest. (Chicago, Milwaukee, Indy) We?ve been on hold with TAC for 30+
> >> minutes. Any info is appreciated.

Well, Riverdale does have about four very large railroad yards. So if the fiber cut is, for instance, hypothetically speaking, underneath a big burning pile of locomotives and tank cars carrying HazMat, I suppose that could be considered a "catastrophic event." Just sayin'....

----
Andy Ringsmuth
andy at newslink.com
News Link ? Manager Technology
1845 S. 11th St., Lincoln, NE 68502
(402) 475-6397 (402) 304-0083 cellular

On Thu, Aug 26, 2010 at 2:11 PM, Kenny Sallee <kenny.sallee at gmail.com>wrote:

>
> On Thu, Aug 26, 2010 at 2:10 PM, Jason L. Sparks <jlsparks at gmail.com>wrote:
>
>> Catastrophic event. Interesting language for a fibre cut?
>>
>
> Guess it depends on your perspective.
>


... and how many backup paths you have...

Scott.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://puck.nether.net/pipermail/outages/attachments/20100826/bdd8bd4b/attachment.html>

On Thu, Aug 26, 2010 at 02:11:26PM -0700, Kenny Sallee wrote:
> On Thu, Aug 26, 2010 at 2:10 PM, Jason L. Sparks
> <[1]jlsparks at gmail.com> wrote:
>
> Catastrophic event. Interesting language for a fibre cut?
>
> ~ jason
>
> Guess it depends on your perspective.

Sounds catastrophic to me, given that they lack physical redundancy in
their MPLS mesh. Customers should be asking them, post-mortem, why
failover paths weren't available.

--
| Jeremy Chadwick jdc at parodius.com |
| Parodius Networking http://www.parodius.com/ |
| UNIX Systems Administrator Mountain View, CA, USA |
| Making life hard for others since 1977. PGP: 4BD6C0CB |

On Aug 26, 2010, at 2:11 PM, Kenny Sallee wrote:

>
>
> On Thu, Aug 26, 2010 at 2:10 PM, Jason L. Sparks <jlsparks at gmail.com> wrote:
> Catastrophic event. Interesting language for a fibre cut?
>
> ~ jason
>
>
> Guess it depends on your perspective.
>
And whether you have a protect side, or, a second-side of the ring with even
more circuits on it (famous Worldcom stunt).

Owen

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://puck.nether.net/pipermail/outages/attachments/20100826/960dc641/attachment-0001.html>

Wait you mean have the network do something WITHOUT going through RCMAC?!
BLASPHEMY!

</sarcasm> :)

A little more seriously though, often times theres one of several issues
going on, either a 'collapsed loop' where the "redundant" paths are in the
same trench, or, there just is no redundant path in their network. Happens
far more often than any telco would want to admit to any customer.

If I were affected by this I'd push for a root cause and a documented fix,
along with credit heh.

--On Thursday, August 26, 2010 2:17 PM -0700 Robin Pimentel
<robin.pimentel at gmail.com> wrote:

> No kidding. ?Someone should ask that rep, if they have ever heard of
> MPLS Fast Re-Route. ?Not that hard...
>
>
> On Thu, Aug 26, 2010 at 2:10 PM, Jason L. Sparks <jlsparks at gmail.com>
> wrote:
>
>
> Catastrophic event. Interesting language for a fibre cut?
>
> ~ jason
>

Jeremy,

That's the catastrophic part.

~jason

On Thu, Aug 26, 2010 at 5:30 PM, Jeremy Chadwick
<outages at jdc.parodius.com>wrote:

> On Thu, Aug 26, 2010 at 02:11:26PM -0700, Kenny Sallee wrote:
> > On Thu, Aug 26, 2010 at 2:10 PM, Jason L. Sparks
> > <[1]jlsparks at gmail.com> wrote:
> >
> > Catastrophic event. Interesting language for a fibre cut?
> >
> > ~ jason
> >
> > Guess it depends on your perspective.
>
> Sounds catastrophic to me, given that they lack physical redundancy in
> their MPLS mesh. Customers should be asking them, post-mortem, why
> failover paths weren't available.
>
> --
> | Jeremy Chadwick jdc at parodius.com |
> | Parodius Networking http://www.parodius.com/ |
> | UNIX Systems Administrator Mountain View, CA, USA |
> | Making life hard for others since 1977. PGP: 4BD6C0CB |
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://puck.nether.net/pipermail/outages/attachments/20100826/3b5d3c77/attachment.html>

Catastrophic event sounds better, and makes customers less liked to be annoyed..

"We had a catastrophic event and that's why you network is down" sounds somewhat reasonable and you get the feeling that they are heroically struggling against terrorist and train wrecks and similar.

"Some clod with a spade sliced our cable in half and now we are screwed..." makes them sound incompetent...

Warren Kumari
------
Please excuse typing, etc -- This was sent from a device with a tiny keyboard.

On Aug 26, 2010, at 5:10 PM, "Jason L. Sparks" <jlsparks at gmail.com> wrote:

> Catastrophic event. Interesting language for a fibre cut?
>
> ~ jason
>
> > We got this message from our account rep:
> >
> > "We are have a catastrophic event (fiber cut) going in Riverdale, IL"
> >
> > Not sure if it's related
> >
> > On Thu, Aug 26, 2010 at 12:54 PM, notifications <notifications at mattfilip.com
> >> wrote:
> >
> >> We currently have at least 5 sites down on Verizon Business MPLS in the
> >> Midwest. (Chicago, Milwaukee, Indy) We?ve been on hold with TAC for 30+
> >> minutes. Any info is appreciated.
> >>
> >> _______________________________________________
> >> Outages mailing list
> >> Outages at outages.org
> >> https://puck.nether.net/mailman/listinfo/outages
> >>
> >>
> _______________________________________________
> Outages mailing list
> Outages at outages.org
> https://puck.nether.net/mailman/listinfo/outages
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://puck.nether.net/pipermail/outages/attachments/20100826/9e402dd2/attachment.html>

Fair point. I acknowledge I jumped the gun. Carry on.
~jason
> Catastrophic event sounds better, and makes customers less liked to be
annoyed..
>
> "We had a catastrophic event and that's why you network is down" sounds
somewhat reasonable and you get the feeling that they are heroically
struggling against terrorist and train wrecks and similar.
>
> "Some clod with a spade sliced our cable in half and now we are
screwed..." makes them sound incompetent...
>
> Warren Kumari
> ------
> Please excuse typing, etc -- This was sent from a device with a tiny
keyboard.
>
> On Aug 26, 2010, at 5:10 PM, "Jason L. Sparks" <jlsparks at gmail.com> wrote:
>
>> Catastrophic event. Interesting language for a fibre cut?
>>
>> ~ jason
>>
>> > We got this message from our account rep:
>> >
>> > "We are have a catastrophic event (fiber cut) going in Riverdale, IL"
>> >
>> > Not sure if it's related
>> >
>> > On Thu, Aug 26, 2010 at 12:54 PM, notifications <
notifications at mattfilip.com
>> >> wrote:
>> >
>> >> We currently have at least 5 sites down on Verizon Business MPLS in
the
>> >> Midwest. (Chicago, Milwaukee, Indy) We?ve been on hold with TAC for
30+
>> >> minutes. Any info is appreciated.
>> >>
>> >> _______________________________________________
>> >> Outages mailing list
>> >> Outages at outages.org
>> >> https://puck.nether.net/mailman/listinfo/outages
>> >>
>> >>
>> _______________________________________________
>> Outages mailing list
>> Outages at outages.org
>> https://puck.nether.net/mailman/listinfo/outages
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://puck.nether.net/pipermail/outages/attachments/20100826/d1ffde2f/attachment.html>