Mailing List Archive

Website is down/not pingable. The phone lines for XO all say its a DNS
resolving issue.

A few of my DS-1 circuits are down, and a few have serious packet
loss. But some are up?!

-John

On Sep 16, 2010, at 11:26 AM, Bill Wichers wrote:

> Has anyone been able to find out what is going on with XO? We?re
> showing spotty connectivity all over their network, and I got
> disconnected from their PNA support number with a ?technical
> difficulties? message. I?m hoping maybe someone on here knows
> something?
>
> -Bill
> _______________________________________________
> Outages mailing list
> Outages at outages.org
> https://puck.nether.net/mailman/listinfo/outages

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://puck.nether.net/pipermail/outages/attachments/20100916/8fa7ff6e/attachment.html>

Wish I did.. I am also hearing it's a widespread issue affecting ATT as
well.

-Mike


On Thu, Sep 16, 2010 at 8:26 AM, Bill Wichers <billw at waveform.net> wrote:

> Has anyone been able to find out what is going on with XO? We?re showing
> spotty connectivity all over their network, and I got disconnected from
> their PNA support number with a ?technical difficulties? message. I?m hoping
> maybe someone on here knows something?
>
>
>
> -Bill
>
> _______________________________________________
> Outages mailing list
> Outages at outages.org
> https://puck.nether.net/mailman/listinfo/outages
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://puck.nether.net/pipermail/outages/attachments/20100916/18d05da6/attachment.html>

I am currently seeing a lot of issues with (currently) unknown carriers and
our VoIP calls. We use Bandwidth.com for outbound calls, and I know that XO
is one of their upstream providers. We are seeing dropped calls, garbled
audio. I have a ticket opened with them, but I don't think they have
identified the issue yet.





From: outages-bounces@outages.org [mailto:outages-bounces@outages.org] On
Behalf Of Bill Wichers
Sent: Thursday, September 16, 2010 11:27 AM
To: outages at outages.org
Subject: [outages] Major XO issues?



Has anyone been able to find out what is going on with XO? We're showing
spotty connectivity all over their network, and I got disconnected from
their PNA support number with a "technical difficulties" message. I'm hoping
maybe someone on here knows something?



-Bill

No virus found in this incoming message.
Checked by AVG - www.avg.com
Version: 8.5.445 / Virus Database: 271.1.1/3135 - Release Date: 09/16/10
06:34:00

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://puck.nether.net/pipermail/outages/attachments/20100916/80153b51/attachment.html>

We are experiencing intermittent issues with their voip orig and voip term products.


**************************
This email composed on my iPhone 4 whilst driving at 90 MPH with my knees.

Please excuse typo's and brevity due to accidents, cops and other mishaps.

Cheers,
Michael Self
PhoneFusion
954.343.5500 (home/office/cell/fax/SMS)


On Sep 16, 2010, at 11:57, "Bill Wichers" <billw at waveform.net<mailto:billw at waveform.net>> wrote:

Has anyone been able to find out what is going on with XO? We?re showing spotty connectivity all over their network, and I got disconnected from their PNA support number with a ?technical difficulties? message. I?m hoping maybe someone on here knows something?

-Bill
_______________________________________________
Outages mailing list
Outages at outages.org<mailto:Outages at outages.org>
https://puck.nether.net/mailman/listinfo/outages
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://puck.nether.net/pipermail/outages/attachments/20100916/1b5cb246/attachment.html>

I'd also be curious if anyone has any information.

Our data was completely down for almost an hour (returned at 1055 CDT)
. We're in the Dallas area (Plano). Things are working now (though
it seems as at least a few XO peers we do business with are still
routing through other upstreams).

No issues with Voice, other than what you mentioned, being
disconnected from the PNA number.

-Ed

On Thu, Sep 16, 2010 at 10:26 AM, Bill Wichers <billw at waveform.net> wrote:
> Has anyone been able to find out what is going on with XO? We?re showing
> spotty connectivity all over their network, and I got disconnected from
> their PNA support number with a ?technical difficulties? message. I?m hoping
> maybe someone on here knows something?
>
>
>
> ? -Bill

Seeing this here as well, of course XO in their usual fashion has no outage
notices posted on their site. We saw the issues starting at 10:56 AM EST,
and although they are better now we still have at least 1 customer in NY
complaining.



-Scott



From: outages-bounces@outages.org [mailto:outages-bounces@outages.org] On
Behalf Of Bill Wichers
Sent: Thursday, September 16, 2010 11:27 AM
To: outages at outages.org
Subject: [outages] Major XO issues?



Has anyone been able to find out what is going on with XO? We're showing
spotty connectivity all over their network, and I got disconnected from
their PNA support number with a "technical difficulties" message. I'm hoping
maybe someone on here knows something?



-Bill

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://puck.nether.net/pipermail/outages/attachments/20100916/3d97bc04/attachment-0001.html>

Repost from NANOG for everyone's benefit:

"XO Engineers are telling us that they are aware of packet loss across their
network and are looking into it. We're experiencing slow/degraded
connectivity out of St. Louis and Nashville but Atlanta and Dallas are
problem free."

William Collier-Byrd
will at collier-byrd.net


On Thu, Sep 16, 2010 at 11:26 AM, Bill Wichers <billw at waveform.net> wrote:

> Has anyone been able to find out what is going on with XO? We?re showing
> spotty connectivity all over their network, and I got disconnected from
> their PNA support number with a ?technical difficulties? message. I?m hoping
> maybe someone on here knows something?
>
>
>
> -Bill
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://puck.nether.net/pipermail/outages/attachments/20100916/b50907cb/attachment.html>

On 9/16/10 8:55 AM, John Von Essen wrote:
> Website is down/not pingable. The phone lines for XO all say its a DNS
> resolving issue.
>

I can get to the website (assuming xo.com) fine; I'm connected through
Sprint and Global Crossing.

~Seth

XO supposedly leaked a bunch of routes (2nd hand info), had at least one internal routing loop between NYC & DC (1st hand info). They had massive reachability problems across the USA (1st hand info).

They supposedly fixed the problem(s) starting at 12:14 EDT today, and reachability should return to normal (info from XO). But we are still seeing at least some locations having issues at 12:38 EDT.

--
TTFN,
patrick


On Sep 16, 2010, at 11:58 AM, Mike Lyon wrote:

> Wish I did.. I am also hearing it's a widespread issue affecting ATT as well.
>
> -Mike
>
>
> On Thu, Sep 16, 2010 at 8:26 AM, Bill Wichers <billw at waveform.net> wrote:
> Has anyone been able to find out what is going on with XO? We?re showing spotty connectivity all over their network, and I got disconnected from their PNA support number with a ?technical difficulties? message. I?m hoping maybe someone on here knows something?
>
>
> -Bill
>
>
> _______________________________________________
> Outages mailing list
> Outages at outages.org
> https://puck.nether.net/mailman/listinfo/outages
>
>
> _______________________________________________
> Outages mailing list
> Outages at outages.org
> https://puck.nether.net/mailman/listinfo/outages

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 09/16/2010 08:26 AM, Bill Wichers wrote:
> Has anyone been able to find out what is going on with XO? We're showing
> spotty connectivity all over their network, and I got disconnected from
> their PNA support number with a "technical difficulties" message. I'm
> hoping maybe someone on here knows something?
>
>
>
> -Bill
- ----------------------
Apparently the person I spoke to within XO said they were having
headless chicken syndrome and that they were in the midst of bringing up
sessions and that was the extend of the information I received.

Hopefully we'll see some sort of lesson learned from this.


regards,
/virendra
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.10 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iEYEARECAAYFAkySSEUACgkQpbZvCIJx1bcV6wCbBBx9IvdsNfwX1tberS7VAdIm
1zMAoJgEPksIGOKkRFN2ENSgIt7AElnc
=Hy8x
-----END PGP SIGNATURE-----

Both my XO and ATT peering are fine, I did have odd routing issues about an
hour ago though. I am in the SF Bay area.

Zaid


On 9/16/10 8:58 AM, "Mike Lyon" <mike.lyon at gmail.com> wrote:

> Wish I did.. I am also hearing it's a widespread issue affecting ATT as
> well.
>
> -Mike
>
>
> On Thu, Sep 16, 2010 at 8:26 AM, Bill Wichers <billw at waveform.net> wrote:
>
>> Has anyone been able to find out what is going on with XO? We?re showing
>> spotty connectivity all over their network, and I got disconnected from
>> their PNA support number with a ?technical difficulties? message. I?m hoping
>> maybe someone on here knows something?
>>
>>
>>
>> -Bill
>>
>> _______________________________________________
>> Outages mailing list
>> Outages at outages.org
>> https://puck.nether.net/mailman/listinfo/outages
>>
>>
> _______________________________________________
> Outages mailing list
> Outages at outages.org
> https://puck.nether.net/mailman/listinfo/outages

Our datacenter in Salt Lake City forwarded us this message from XO:



"I wanted to give notice in case you were having issues connecting to
various websites today. There is a problem with a peering router which
allows XO's network to connect to outside networks. Impact appears to
be pretty broad due to the volume of routes that egress/ingress that
peer. Our Engineering is working to correct. We don't have an
estimated time for resolution just yet, but this our top priority issue
right now and all levels of our repair organization are engaged."





Philip Holmes



From: outages-bounces@outages.org [mailto:outages-bounces@outages.org]
On Behalf Of Geoff Mina
Sent: Thursday, September 16, 2010 9:59 AM
To: 'Bill Wichers'; outages at outages.org
Subject: Re: [outages] Major XO issues?



I am currently seeing a lot of issues with (currently) unknown carriers
and our VoIP calls. We use Bandwidth.com for outbound calls, and I know
that XO is one of their upstream providers. We are seeing dropped
calls, garbled audio. I have a ticket opened with them, but I don't
think they have identified the issue yet.





From: outages-bounces@outages.org [mailto:outages-bounces@outages.org]
On Behalf Of Bill Wichers
Sent: Thursday, September 16, 2010 11:27 AM
To: outages at outages.org
Subject: [outages] Major XO issues?



Has anyone been able to find out what is going on with XO? We're showing
spotty connectivity all over their network, and I got disconnected from
their PNA support number with a "technical difficulties" message. I'm
hoping maybe someone on here knows something?



-Bill

No virus found in this incoming message.
Checked by AVG - www.avg.com
Version: 8.5.445 / Virus Database: 271.1.1/3135 - Release Date: 09/16/10
06:34:00

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://puck.nether.net/pipermail/outages/attachments/20100916/8b782817/attachment-0001.html>

Dead to the world via XO in Herndon, VA. Coincidentally (or not?) a few
miles from Ashburn.

Jason

On Thu, Sep 16, 2010 at 12:11 PM, Scott Berkman <scott at sberkman.net> wrote:

> Seeing this here as well, of course XO in their usual fashion has no
> outage notices posted on their site. We saw the issues starting at 10:56 AM
> EST, and although they are better now we still have at least 1 customer in
> NY complaining.
>
>
>
> -Scott
>
>
>
> *From:* outages-bounces at outages.org [mailto:outages-bounces at outages.org] *On
> Behalf Of *Bill Wichers
> *Sent:* Thursday, September 16, 2010 11:27 AM
> *To:* outages at outages.org
> *Subject:* [outages] Major XO issues?
>
>
>
> Has anyone been able to find out what is going on with XO? We?re showing
> spotty connectivity all over their network, and I got disconnected from
> their PNA support number with a ?technical difficulties? message. I?m hoping
> maybe someone on here knows something?
>
>
>
> -Bill
>
> _______________________________________________
> Outages mailing list
> Outages at outages.org
> https://puck.nether.net/mailman/listinfo/outages
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://puck.nether.net/pipermail/outages/attachments/20100916/41c41acc/attachment.html>

I've been told that there were "peering issues" and that there has not been an official "all clear" issued internally yet.

The thing that most concerned be about this issue is that when we figure out XO was having issues, we dropped our peering session with them but they appear to have continued to announce our routes -- at least in some areas. We haven't brought that up with them yet but will soon.

-Bill

-----Original Message-----
From: outages-bounces@outages.org [mailto:outages-bounces@outages.org] On Behalf Of Ed W
Sent: Thursday, September 16, 2010 12:10 PM
To: outages at outages.org
Subject: Re: [outages] Major XO issues?

I'd also be curious if anyone has any information.

Our data was completely down for almost an hour (returned at 1055 CDT)
. We're in the Dallas area (Plano). Things are working now (though
it seems as at least a few XO peers we do business with are still
routing through other upstreams).

No issues with Voice, other than what you mentioned, being
disconnected from the PNA number.

-Ed

On Thu, Sep 16, 2010 at 10:26 AM, Bill Wichers <billw at waveform.net> wrote:
> Has anyone been able to find out what is going on with XO? We're showing
> spotty connectivity all over their network, and I got disconnected from
> their PNA support number with a "technical difficulties" message. I'm hoping
> maybe someone on here knows something?
>
>
>
> ? -Bill

_______________________________________________
Outages mailing list
Outages at outages.org
https://puck.nether.net/mailman/listinfo/outages

I am on the West Coast and noticed a huge drop in our traffic this morning.
Starting to pick backup.



From: outages-bounces@outages.org [mailto:outages-bounces@outages.org] On
Behalf Of Scott Berkman
Sent: Thursday, September 16, 2010 9:11 AM
To: 'Bill Wichers'; outages at outages.org
Subject: Re: [outages] Major XO issues?



Seeing this here as well, of course XO in their usual fashion has no outage
notices posted on their site. We saw the issues starting at 10:56 AM EST,
and although they are better now we still have at least 1 customer in NY
complaining.



-Scott



From: outages-bounces@outages.org [mailto:outages-bounces@outages.org] On
Behalf Of Bill Wichers
Sent: Thursday, September 16, 2010 11:27 AM
To: outages at outages.org
Subject: [outages] Major XO issues?



Has anyone been able to find out what is going on with XO? We're showing
spotty connectivity all over their network, and I got disconnected from
their PNA support number with a "technical difficulties" message. I'm hoping
maybe someone on here knows something?



-Bill

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://puck.nether.net/pipermail/outages/attachments/20100916/ff0264ed/attachment-0001.html>

On 9/16/2010 12:38, Jacob wrote:
> I am on the West Coast and noticed a huge drop in our traffic this
> morning. Starting to pick backup.
>

I saw an abnormal increase in traffic since I do a lot of secondary MX
and DNS type services. Traffic is starting to go back to normal now.

~Seth