Mailing List Archive

Verizon west coast outage?
Anyone seeing packet loss going through .so-6-0-0.xt1.lax7.alter.net for hosts on Verizon but on the East Coast dying?

Have a prefix that isn't available if data comes into Verizon on the west coast only.


Joseph
Verizon west coast outage? [ In reply to ]
We just opened a ticket with Verizon and were informed of outages for Verizon on East Coast.

-----Original Message-----
From: outages-bounces@outages.org [mailto:outages-bounces@outages.org] On Behalf Of Joseph Jackson
Sent: Friday, October 02, 2009 3:46 PM
To: outages at outages.org
Subject: [outages] Verizon west coast outage?

Anyone seeing packet loss going through .so-6-0-0.xt1.lax7.alter.net for hosts on Verizon but on the East Coast dying?

Have a prefix that isn't available if data comes into Verizon on the west coast only.


Joseph


_______________________________________________
outages mailing list
outages at outages.org
https://puck.nether.net/mailman/listinfo/outages
Verizon west coast outage? [ In reply to ]
Lost east coast as well.

Issue started 3:13:58 PM, ended 3:50:11 PM EST.

Seamus

-----Original Message-----
From: outages-bounces@outages.org [mailto:outages-bounces@outages.org]
On Behalf Of Joseph Jackson
Sent: Friday, October 02, 2009 3:46 PM
To: outages at outages.org
Subject: [outages] Verizon west coast outage?

Anyone seeing packet loss going through .so-6-0-0.xt1.lax7.alter.net for
hosts on Verizon but on the East Coast dying?

Have a prefix that isn't available if data comes into Verizon on the
west coast only.


Joseph


_______________________________________________
outages mailing list
outages at outages.org
https://puck.nether.net/mailman/listinfo/outages
Verizon west coast outage? [ In reply to ]
Oh yeah.... Pretty bad. Any insider info? I am seeing issues at
0.ge-1-3-0.xl4.nyc4.alter.net [152.63.20.54]


On 10/2/09 3:45 PM, "Joseph Jackson" <jjackson at aninetworks.net> wrote:

> Anyone seeing packet loss going through .so-6-0-0.xt1.lax7.alter.net for hosts
> on Verizon but on the East Coast dying?
>
> Have a prefix that isn't available if data comes into Verizon on the west
> coast only.
>
>
> Joseph
>
>
> _______________________________________________
> outages mailing list
> outages at outages.org
> https://puck.nether.net/mailman/listinfo/outages
Verizon west coast outage? [ In reply to ]
Had a customer call in 10 minutes ago about inability to get VPN access to
one of our sites...their traceroute was also stuck at alter.net (in NYC, I
believe), they told me over the phone. But it came back 4 minutes, before I
could get my own traceroute in. When I was running my ping (before the
traceroute) I was seeing some pings of over 1 second.

Frank

-----Original Message-----
From: outages-bounces@outages.org [mailto:outages-bounces@outages.org] On
Behalf Of Joseph Jackson
Sent: Friday, October 02, 2009 2:46 PM
To: outages at outages.org
Subject: [outages] Verizon west coast outage?

Anyone seeing packet loss going through .so-6-0-0.xt1.lax7.alter.net for
hosts on Verizon but on the East Coast dying?

Have a prefix that isn't available if data comes into Verizon on the west
coast only.

Joseph

_______________________________________________
outages mailing list
outages at outages.org
https://puck.nether.net/mailman/listinfo/outages
Verizon west coast outage? [ In reply to ]
I can confirm that something is going on with Vz/MCI/UUnet/Alternet on
the west coast; there's a large number of route denouncements and
announcements being witnessed causing all kinds of (broken)
reconvergence with packets destined to the east coast as well as the
south.

I don't have any other information. BGPlay might come in handy.

--
| Jeremy Chadwick jdc at parodius.com |
| Parodius Networking http://www.parodius.com/ |
| UNIX Systems Administrator Mountain View, CA, USA |
| Making life hard for others since 1977. PGP: 4BD6C0CB |

On Fri, Oct 02, 2009 at 12:45:53PM -0700, Joseph Jackson wrote:
> Anyone seeing packet loss going through .so-6-0-0.xt1.lax7.alter.net for hosts on Verizon but on the East Coast dying?
>
> Have a prefix that isn't available if data comes into Verizon on the west coast only.
>
>
> Joseph
>
>
> _______________________________________________
> outages mailing list
> outages at outages.org
> https://puck.nether.net/mailman/listinfo/outages
Verizon west coast outage? [ In reply to ]
seeing similar on the east coast; somewhere between alter.net (vzb)
and level3 in nyc just before the chi hop.

at least that's what my traceroute is telling me.

On Oct 2, 2009, at 3:54 PM, Jake VanDewater wrote:

> We just opened a ticket with Verizon and were informed of outages
> for Verizon on East Coast.
>
> -----Original Message-----
> From: outages-bounces at outages.org [mailto:outages-
> bounces at outages.org] On Behalf Of Joseph Jackson
> Sent: Friday, October 02, 2009 3:46 PM
> To: outages at outages.org
> Subject: [outages] Verizon west coast outage?
>
> Anyone seeing packet loss going through .so-6-0-0.xt1.lax7.alter.net
> for hosts on Verizon but on the East Coast dying?
>
> Have a prefix that isn't available if data comes into Verizon on the
> west coast only.
>
>
> Joseph
>
>
> _______________________________________________
> outages mailing list
> outages at outages.org
> https://puck.nether.net/mailman/listinfo/outages
> _______________________________________________
> outages mailing list
> outages at outages.org
> https://puck.nether.net/mailman/listinfo/outages

--
Wes
http://claimid.com/wesyoung

-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 2444 bytes
Desc: not available
URL: <https://puck.nether.net/pipermail/outages/attachments/20091002/aec992da/attachment.bin>
Verizon west coast outage? [ In reply to ]
This also seemed to have effected Europe connecting to Verizon routes in
Denver and Sydney... and a Circuit we have on VZB in New York reaching
much of the internet...

Looks to be stable again now...

Lee


-----Original Message-----
From: outages-bounces@outages.org [mailto:outages-bounces@outages.org]
On Behalf Of Jake VanDewater
Sent: 02 October 2009 20:55
To: Joseph Jackson; outages at outages.org
Subject: Re: [outages] Verizon west coast outage?

We just opened a ticket with Verizon and were informed of outages for
Verizon on East Coast.

-----Original Message-----
From: outages-bounces@outages.org [mailto:outages-bounces@outages.org]
On Behalf Of Joseph Jackson
Sent: Friday, October 02, 2009 3:46 PM
To: outages at outages.org
Subject: [outages] Verizon west coast outage?

Anyone seeing packet loss going through .so-6-0-0.xt1.lax7.alter.net for
hosts on Verizon but on the East Coast dying?

Have a prefix that isn't available if data comes into Verizon on the
west coast only.


Joseph


_______________________________________________
outages mailing list
outages at outages.org
https://puck.nether.net/mailman/listinfo/outages
_______________________________________________
outages mailing list
outages at outages.org
https://puck.nether.net/mailman/listinfo/outages

______________________________________________________________________
This email has been scanned by the MessageLabs Email Security System.
For more information please visit http://www.messagelabs.com/email
______________________________________________________________________

______________________________________________________________________
This email has been scanned by the MessageLabs Email Security System.
For more information please visit http://www.messagelabs.com/email
______________________________________________________________________
Verizon west coast outage? [ In reply to ]
I'll add here that NANOG also has a couple threads going on about
this which appear quite relevant, but it's hard to tell what's broken
and why at this point.

http://www.merit.edu/mail.archives/nanog/msg00678.html

http://www.merit.edu/mail.archives/nanog/msg00676.html
http://www.merit.edu/mail.archives/nanog/msg00679.html

http://www.merit.edu/mail.archives/nanog/msg00677.html
http://www.merit.edu/mail.archives/nanog/msg00680.html
http://www.merit.edu/mail.archives/nanog/msg00681.html

And possibly related, but unsure given that this report is significantly
earlier than the others (AS7132 = AT&T).

http://www.merit.edu/mail.archives/nanog/msg00671.html

--
| Jeremy Chadwick jdc at parodius.com |
| Parodius Networking http://www.parodius.com/ |
| UNIX Systems Administrator Mountain View, CA, USA |
| Making life hard for others since 1977. PGP: 4BD6C0CB |

On Fri, Oct 02, 2009 at 03:00:05PM -0500, Frank Bulk - iName.com wrote:
> Had a customer call in 10 minutes ago about inability to get VPN access to
> one of our sites...their traceroute was also stuck at alter.net (in NYC, I
> believe), they told me over the phone. But it came back 4 minutes, before I
> could get my own traceroute in. When I was running my ping (before the
> traceroute) I was seeing some pings of over 1 second.
>
> Frank
>
> -----Original Message-----
> From: outages-bounces at outages.org [mailto:outages-bounces at outages.org] On
> Behalf Of Joseph Jackson
> Sent: Friday, October 02, 2009 2:46 PM
> To: outages at outages.org
> Subject: [outages] Verizon west coast outage?
>
> Anyone seeing packet loss going through .so-6-0-0.xt1.lax7.alter.net for
> hosts on Verizon but on the East Coast dying?
>
> Have a prefix that isn't available if data comes into Verizon on the west
> coast only.
>
> Joseph
>
> _______________________________________________
> outages mailing list
> outages at outages.org
> https://puck.nether.net/mailman/listinfo/outages
>
> _______________________________________________
> outages mailing list
> outages at outages.org
> https://puck.nether.net/mailman/listinfo/outages
Verizon west coast outage? [ In reply to ]
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

There appears to be a major outage on VZB Internet facing side of the
switch (east coast).

VZB west coast seems to be ok, at least for AS701 (uunet).


regards,
/virendra


Jeremy Chadwick wrote:
> I'll add here that NANOG also has a couple threads going on about
> this which appear quite relevant, but it's hard to tell what's broken
> and why at this point.
>
> http://www.merit.edu/mail.archives/nanog/msg00678.html
>
> http://www.merit.edu/mail.archives/nanog/msg00676.html
> http://www.merit.edu/mail.archives/nanog/msg00679.html
>
> http://www.merit.edu/mail.archives/nanog/msg00677.html
> http://www.merit.edu/mail.archives/nanog/msg00680.html
> http://www.merit.edu/mail.archives/nanog/msg00681.html
>
> And possibly related, but unsure given that this report is significantly
> earlier than the others (AS7132 = AT&T).
>
> http://www.merit.edu/mail.archives/nanog/msg00671.html
>
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.6 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iD8DBQFKxmZHpbZvCIJx1bcRAhRcAKCYlU1KBLJJ2X33k9yge1FKqGWLuQCgyqVx
Uh5QZp/5VNeVnkbFtcU8KoQ=
=pJuM
-----END PGP SIGNATURE-----
Verizon west coast outage? [ In reply to ]
I'll be very interested to know what the root cause of this was.

All the discussion indicates there is a single point of failure
somewhere within Verizon's network out east; route propagation and
reconvergence should have mitigated things, so the question is why it
didn't.

--
| Jeremy Chadwick jdc at parodius.com |
| Parodius Networking http://www.parodius.com/ |
| UNIX Systems Administrator Mountain View, CA, USA |
| Making life hard for others since 1977. PGP: 4BD6C0CB |

On Fri, Oct 02, 2009 at 09:12:47PM +0100, Lee Hetherington wrote:
> This also seemed to have effected Europe connecting to Verizon routes in
> Denver and Sydney... and a Circuit we have on VZB in New York reaching
> much of the internet...
>
> Looks to be stable again now...
>
> Lee
>
>
> -----Original Message-----
> From: outages-bounces at outages.org [mailto:outages-bounces at outages.org]
> On Behalf Of Jake VanDewater
> Sent: 02 October 2009 20:55
> To: Joseph Jackson; outages at outages.org
> Subject: Re: [outages] Verizon west coast outage?
>
> We just opened a ticket with Verizon and were informed of outages for
> Verizon on East Coast.
>
> -----Original Message-----
> From: outages-bounces at outages.org [mailto:outages-bounces at outages.org]
> On Behalf Of Joseph Jackson
> Sent: Friday, October 02, 2009 3:46 PM
> To: outages at outages.org
> Subject: [outages] Verizon west coast outage?
>
> Anyone seeing packet loss going through .so-6-0-0.xt1.lax7.alter.net for
> hosts on Verizon but on the East Coast dying?
>
> Have a prefix that isn't available if data comes into Verizon on the
> west coast only.
>
>
> Joseph
>
>
> _______________________________________________
> outages mailing list
> outages at outages.org
> https://puck.nether.net/mailman/listinfo/outages
> _______________________________________________
> outages mailing list
> outages at outages.org
> https://puck.nether.net/mailman/listinfo/outages
>
> ______________________________________________________________________
> This email has been scanned by the MessageLabs Email Security System.
> For more information please visit http://www.messagelabs.com/email
> ______________________________________________________________________
>
> ______________________________________________________________________
> This email has been scanned by the MessageLabs Email Security System.
> For more information please visit http://www.messagelabs.com/email
> ______________________________________________________________________
> _______________________________________________
> outages mailing list
> outages at outages.org
> https://puck.nether.net/mailman/listinfo/outages
Verizon west coast outage? [ In reply to ]
On Fri, 2009-10-02 at 12:45 -0700, Joseph Jackson wrote:
> Anyone seeing packet loss going through .so-6-0-0.xt1.lax7.alter.net for hosts on Verizon but on the East Coast dying?
>
> Have a prefix that isn't available if data comes into Verizon on the west coast only.
>

FYI, there was a power outage in downtown LA about that time. None of
my stuff dropped (carrier center, 600 hope), no diesel fumes from
generators, etc. (including one Wilshire)




>
> Joseph
>
>
> _______________________________________________
> outages mailing list
> outages at outages.org
> https://puck.nether.net/mailman/listinfo/outages
--
Christopher McCrory
"The guy that keeps the servers running"

chrismcc at pricegrabber.com
http://www.pricegrabber.com

Let's face it, there's no Hollow Earth, no robots, and
no 'mute rays.' And even if there were, waxed paper is
no defense. I tried it. Only tinfoil works.
Verizon west coast outage? [ In reply to ]
This was posted on VZ's website as an explanation for the outage
!
Original Link:
http://forums.verizon.com/t5/Verizon-at-Home/Northeast-Router-Failure/ba
-p/87001
!
"As many of you may know, a router failed in the Verizon network this
afternoon causing problems for some Verizon FiOS and High Speed Internet
users. Here's what happened.

The router in question is in New York City and handles traffic from the
Verizon network to the larger Internet. It is one of a number of such
routers in our network.

When routers have problems, they are designed to report that they are
sick. Internet traffic is rerouted to adjacent routers automatically and
sent around the trouble spot. In this case, that didn't happen. The
router went into a hung state and did not appear to the rest of the
network as though it was having problems.

That meant that some user traffic from the northeast continued to flow
to the stalled router, but couldn't be processed. Presto - an outage for
those users. Verizon FiOS, High Speed Internet and small business
customers were affected.

The problem occurred at approximately 3:15 EDT and was detected and
resolved :40 minutes later. Traffic began to flow normally and customers
returned to service at that point.

Currently, we're monitoring the router and the network watching for any
other problems."

!
Andre

-----Original Message-----
From: outages-bounces@outages.org [mailto:outages-bounces@outages.org]
On Behalf Of Seamus Hartmann
Sent: Friday, October 02, 2009 3:55 PM
To: outages at outages.org
Subject: Re: [outages] Verizon west coast outage?

Lost east coast as well.

Issue started 3:13:58 PM, ended 3:50:11 PM EST.

Seamus

-----Original Message-----
From: outages-bounces@outages.org [mailto:outages-bounces@outages.org]
On Behalf Of Joseph Jackson
Sent: Friday, October 02, 2009 3:46 PM
To: outages at outages.org
Subject: [outages] Verizon west coast outage?

Anyone seeing packet loss going through .so-6-0-0.xt1.lax7.alter.net for
hosts on Verizon but on the East Coast dying?

Have a prefix that isn't available if data comes into Verizon on the
west coast only.


Joseph


_______________________________________________
outages mailing list
outages at outages.org
https://puck.nether.net/mailman/listinfo/outages
_______________________________________________
outages mailing list
outages at outages.org
https://puck.nether.net/mailman/listinfo/outages
This message is intended only for the personal and confidential use of the recipients named above.
If the reader of this email is not the intended recipient, you have received this email in error and any review,
dissemination, distribution or copying is strictly prohibited. If you have received this email in error,
please notify the sender immediately by return email and permanently delete the copy you received.

This message is provided for informational purposes and should not be construed as a solicitation or offer
to buy or sell any securities or related financial instruments. Wolverine is not responsible for any
recommendation, solicitation, offer or agreement or any information about any transaction, customer account
or account activity that may be attached to or contained in this communication. Wolverine accepts no
liability for any content contained in the email, or any errors or omissions arising as a result of
email transmission. Any opinions contained in this email constitute the sender's best judgment at this
time and are subject to change without notice.
Verizon west coast outage? [ In reply to ]
On Fri, Oct 02, 2009 at 04:55:30PM -0500, Forestal, Andre Jr. wrote:
> When routers have problems, they are designed to report that they are
> sick. Internet traffic is rerouted to adjacent routers automatically and
> sent around the trouble spot. In this case, that didn't happen. The
> router went into a hung state and did not appear to the rest of the
> network as though it was having problems.

It appears BFD is a protocol foreign to Verizon. And why didn't hold
timers expire on associated iBGP peers? Questions, questions...

--
| Jeremy Chadwick jdc at parodius.com |
| Parodius Networking http://www.parodius.com/ |
| UNIX Systems Administrator Mountain View, CA, USA |
| Making life hard for others since 1977. PGP: 4BD6C0CB |
Verizon west coast outage? [ In reply to ]
And so the end of the story is... Epik has a more reliable network than Verizon. Ha!

Matt Kollander

Atlanta?|?Toronto?|?Los Angeles?|?Vancouver?|Denver|



-----Original Message-----
From: outages-bounces@outages.org [mailto:outages-bounces@outages.org] On Behalf Of Jeremy Chadwick
Sent: Friday, October 02, 2009 3:53 PM
To: outages at outages.org
Subject: Re: [outages] Verizon west coast outage?

On Fri, Oct 02, 2009 at 04:55:30PM -0500, Forestal, Andre Jr. wrote:
> When routers have problems, they are designed to report that they are
> sick. Internet traffic is rerouted to adjacent routers automatically and
> sent around the trouble spot. In this case, that didn't happen. The
> router went into a hung state and did not appear to the rest of the
> network as though it was having problems.

It appears BFD is a protocol foreign to Verizon. And why didn't hold
timers expire on associated iBGP peers? Questions, questions...

--
| Jeremy Chadwick jdc at parodius.com |
| Parodius Networking http://www.parodius.com/ |
| UNIX Systems Administrator Mountain View, CA, USA |
| Making life hard for others since 1977. PGP: 4BD6C0CB |

_______________________________________________
outages mailing list
outages at outages.org
https://puck.nether.net/mailman/listinfo/outages