Mailing List Archive

DC Region Verizon FiOS Outage Event 29-April 18:30
There was just a massive DC region Verizon FiOS backbone routing failure
starting at 6:27pm EST and running for ~11 minutes. All paths were
impacted but some were more loss prone than others.

Here's a loss/latency graph to Google's local anycasted 8.8.8.8 resolver:

[image: image.png]

vs a graph to Cogent:
[image: image.png]

vs one to Azure:
[image: image.png]

MTR graphs show major loss down every path that touches one of Verizon's "
ALTER.NET" named routers.

Really just sending this one in because the footprint was massive. I'm
used to smaller DC-region path failures, but this impacted nearly every
path for nearly every FiOS user in DC and the close-in Virginia and
Maryland suburbs. This was far worse than their usual routing hiccups.

- Cary
Re: DC Region Verizon FiOS Outage Event 29-April 18:30 [ In reply to ]
We noticed it as well (Northern VA). Latency spiked from normally 8ms
between Cox and FIOS to over 150ms, though not too much packet loss. There
was an initial loss of connections and packets on SSH, and my son reported
losing access to his online XBOX game.

My monitoring detected 100% packet loss for about 1 minute, 22:32:32 to
22:33:26 UTC, high RTT from 22:33:26 to 22:37:30, and recovery by 22:37:30
UTC. From my POV it lasted only about 5-6 minutes, though that was from Cox
into Verizon FIOS monitoring. Tests were done either every 30 seconds or 1
minute, +/- a few seconds.

It's too bad we likely won't hear the root cause or recovery story, I'm
sure it'd be fascinating.

Beckman

Cary wrote:

> There was just a massive DC region Verizon FiOS backbone routing failure
> starting at 6:27pm EST and running for ~11 minutes. All paths were
> impacted but some were more loss prone than others.

---------------------------------------------------------------------------
Peter Beckman Internet Guy
beckman@angryox.com https://www.angryox.com/
---------------------------------------------------------------------------
_______________________________________________
Outages mailing list
Outages@outages.org
https://puck.nether.net/mailman/listinfo/outages