Mailing List Archive

----- Original Message -----
From: "Christopher Hunt" <dharmachris@gmail.com>

> This morning we're experiencing timeouts to and from a lot of sites.
I heard through the grapevine that XO instituted some new route limits
on their peers. Their site at http://businesscenter.xo.com is down,
their phone number 800-424-0583 is fast busy and their Premier
National Accounts 866-295-9696 reports "we are currently experiencing
technical difficulties and cannot take your call" then DISCONNECT.

> UPDATE
> One of my other vendors says that they have resolved it and it seems
to have improved. According to them, It was indeed a cap on the
number of routes accepted by XO from their peers.

Other reports here in the last hour suggest it was the opposite: XO suddenly
started *sending* a vastly increased number of presumably bogus routes to
their peers, many of whom tripped filter rules, and either shut the sessions
down, or limited them and picked the wrong routes to drop.

I expect it will be tomorrow before we assemble a coherent picture of what
the actual outage was. Anyone check Renesys' blog yet?

Cheers,
-- jra

Is there any chance that you guys can please try and look at the existing
threads on the list before posting a question that has already been posted?
If you have information to add that will help people you can reply to one of
the existing threads, but having 15 different threads on the same topic is
really annoying.

Thanks,

-Scott

-----Original Message-----
From: outages-bounces@outages.org [mailto:outages-bounces@outages.org] On
Behalf Of Christopher Hunt
Sent: Thursday, September 16, 2010 11:41 AM
To: outages at outages.org
Subject: [outages] XO snafu

This morning we're experiencing timeouts to and from a lot of sites.
I heard through the grapevine that XO instituted some new route limits
on their peers. Their site at http://businesscenter.xo.com is down,
their phone number 800-424-0583 is fast busy and their Premier
National Accounts 866-295-9696 reports "we are currently experiencing
technical difficulties and cannot take your call" then DISCONNECT.

UPDATE
One of my other vendors says that they have resolved it and it seems
to have improved. According to them, It was indeed a cap on the
number of routes accepted by XO from their peers.

Cheers,
Chris
_______________________________________________
Outages mailing list
Outages at outages.org
https://puck.nether.net/mailman/listinfo/outages

I agree and apologize. Apparently the list serve is lagging these
messages and I didn't receive most threads until just recently.

-Chris
On 9/16/2010 10:48 AM, Scott Berkman wrote:
> Is there any chance that you guys can please try and look at the existing
> threads on the list before posting a question that has already been posted?
> If you have information to add that will help people you can reply to one of
> the existing threads, but having 15 different threads on the same topic is
> really annoying.
>
> Thanks,
>
> -Scott
>
> -----Original Message-----
> From: outages-bounces at outages.org [mailto:outages-bounces at outages.org] On
> Behalf Of Christopher Hunt
> Sent: Thursday, September 16, 2010 11:41 AM
> To: outages at outages.org
> Subject: [outages] XO snafu
>
> This morning we're experiencing timeouts to and from a lot of sites.
> I heard through the grapevine that XO instituted some new route limits
> on their peers. Their site at http://businesscenter.xo.com is down,
> their phone number 800-424-0583 is fast busy and their Premier
> National Accounts 866-295-9696 reports "we are currently experiencing
> technical difficulties and cannot take your call" then DISCONNECT.
>
> UPDATE
> One of my other vendors says that they have resolved it and it seems
> to have improved. According to them, It was indeed a cap on the
> number of routes accepted by XO from their peers.
>
> Cheers,
> Chris
> _______________________________________________
> Outages mailing list
> Outages at outages.org
> https://puck.nether.net/mailman/listinfo/outages
>
>
> _______________________________________________
> Outages mailing list
> Outages at outages.org
> https://puck.nether.net/mailman/listinfo/outages

It appears the outages mailing list is running upto 30 min. behind from the time people post. People were most likely thinking they were the first ones to notice and start a thread. I don't know if this was caused by the volume of messages regarding the XO issue or if outages.org possibly was affected by xo. I have seen in the past the outages list run behind.

Dylan Ebner, Network Engineer
Consulting Radiologists, Ltd.
1221 Nicollet Mall, Minneapolis, MN 55403
ph. 612.573.2236 fax. 612.573.2250
dylan.ebner at crlmed.com
www.consultingradiologists.com

-----Original Message-----
From: outages-bounces@outages.org [mailto:outages-bounces@outages.org] On Behalf Of Scott Berkman
Sent: Thursday, September 16, 2010 12:49 PM
To: 'Christopher Hunt'; outages at outages.org
Subject: Re: [outages] XO snafu

Is there any chance that you guys can please try and look at the existing
threads on the list before posting a question that has already been posted?
If you have information to add that will help people you can reply to one of
the existing threads, but having 15 different threads on the same topic is
really annoying.

Thanks,

-Scott

-----Original Message-----
From: outages-bounces@outages.org [mailto:outages-bounces@outages.org] On
Behalf Of Christopher Hunt
Sent: Thursday, September 16, 2010 11:41 AM
To: outages at outages.org
Subject: [outages] XO snafu

This morning we're experiencing timeouts to and from a lot of sites.
I heard through the grapevine that XO instituted some new route limits
on their peers. Their site at http://businesscenter.xo.com is down,
their phone number 800-424-0583 is fast busy and their Premier
National Accounts 866-295-9696 reports "we are currently experiencing
technical difficulties and cannot take your call" then DISCONNECT.

UPDATE
One of my other vendors says that they have resolved it and it seems
to have improved. According to them, It was indeed a cap on the
number of routes accepted by XO from their peers.

Cheers,
Chris
_______________________________________________
Outages mailing list
Outages at outages.org
https://puck.nether.net/mailman/listinfo/outages


_______________________________________________
Outages mailing list
Outages at outages.org
https://puck.nether.net/mailman/listinfo/outages

I think perhaps it is people's mail servers that are behind, I've been
getting these in seemingly real time all along. Anyway, please excuse my
comments if anyone that started a new thread really did not know there were
already multiple.

-Scott

-----Original Message-----
From: Dylan Ebner [mailto:dylan.ebner@crlmed.com]
Sent: Thursday, September 16, 2010 2:05 PM
To: Scott Berkman; 'Christopher Hunt'; outages at outages.org
Subject: RE: [outages] XO snafu

It appears the outages mailing list is running upto 30 min. behind from the
time people post. People were most likely thinking they were the first ones
to notice and start a thread. I don't know if this was caused by the volume
of messages regarding the XO issue or if outages.org possibly was affected
by xo. I have seen in the past the outages list run behind.

Dylan Ebner, Network Engineer
Consulting Radiologists, Ltd.
1221 Nicollet Mall, Minneapolis, MN 55403
ph. 612.573.2236 fax. 612.573.2250
dylan.ebner at crlmed.com
www.consultingradiologists.com

-----Original Message-----
From: outages-bounces@outages.org [mailto:outages-bounces@outages.org] On
Behalf Of Scott Berkman
Sent: Thursday, September 16, 2010 12:49 PM
To: 'Christopher Hunt'; outages at outages.org
Subject: Re: [outages] XO snafu

Is there any chance that you guys can please try and look at the existing
threads on the list before posting a question that has already been posted?
If you have information to add that will help people you can reply to one of
the existing threads, but having 15 different threads on the same topic is
really annoying.

Thanks,

-Scott

-----Original Message-----
From: outages-bounces@outages.org [mailto:outages-bounces@outages.org] On
Behalf Of Christopher Hunt
Sent: Thursday, September 16, 2010 11:41 AM
To: outages at outages.org
Subject: [outages] XO snafu

This morning we're experiencing timeouts to and from a lot of sites.
I heard through the grapevine that XO instituted some new route limits
on their peers. Their site at http://businesscenter.xo.com is down,
their phone number 800-424-0583 is fast busy and their Premier
National Accounts 866-295-9696 reports "we are currently experiencing
technical difficulties and cannot take your call" then DISCONNECT.

UPDATE
One of my other vendors says that they have resolved it and it seems
to have improved. According to them, It was indeed a cap on the
number of routes accepted by XO from their peers.

Cheers,
Chris
_______________________________________________
Outages mailing list
Outages at outages.org
https://puck.nether.net/mailman/listinfo/outages


_______________________________________________
Outages mailing list
Outages at outages.org
https://puck.nether.net/mailman/listinfo/outages

Some people don't post regularly and were being held by milter-greylist.

I've now exempted @outages.org from greylist.

Please direct hate mail at me.

- Jared

On Sep 16, 2010, at 2:09 PM, Scott Berkman wrote:

> I think perhaps it is people's mail servers that are behind, I've been
> getting these in seemingly real time all along. Anyway, please excuse my
> comments if anyone that started a new thread really did not know there were
> already multiple.
>
> -Scott
>
> -----Original Message-----
> From: Dylan Ebner [mailto:dylan.ebner at crlmed.com]
> Sent: Thursday, September 16, 2010 2:05 PM
> To: Scott Berkman; 'Christopher Hunt'; outages at outages.org
> Subject: RE: [outages] XO snafu
>
> It appears the outages mailing list is running upto 30 min. behind from the
> time people post. People were most likely thinking they were the first ones
> to notice and start a thread. I don't know if this was caused by the volume
> of messages regarding the XO issue or if outages.org possibly was affected
> by xo. I have seen in the past the outages list run behind.
>
> Dylan Ebner, Network Engineer
> Consulting Radiologists, Ltd.
> 1221 Nicollet Mall, Minneapolis, MN 55403
> ph. 612.573.2236 fax. 612.573.2250
> dylan.ebner at crlmed.com
> www.consultingradiologists.com
>
> -----Original Message-----
> From: outages-bounces at outages.org [mailto:outages-bounces at outages.org] On
> Behalf Of Scott Berkman
> Sent: Thursday, September 16, 2010 12:49 PM
> To: 'Christopher Hunt'; outages at outages.org
> Subject: Re: [outages] XO snafu
>
> Is there any chance that you guys can please try and look at the existing
> threads on the list before posting a question that has already been posted?
> If you have information to add that will help people you can reply to one of
> the existing threads, but having 15 different threads on the same topic is
> really annoying.
>
> Thanks,
>
> -Scott
>
> -----Original Message-----
> From: outages-bounces at outages.org [mailto:outages-bounces at outages.org] On
> Behalf Of Christopher Hunt
> Sent: Thursday, September 16, 2010 11:41 AM
> To: outages at outages.org
> Subject: [outages] XO snafu
>
> This morning we're experiencing timeouts to and from a lot of sites.
> I heard through the grapevine that XO instituted some new route limits
> on their peers. Their site at http://businesscenter.xo.com is down,
> their phone number 800-424-0583 is fast busy and their Premier
> National Accounts 866-295-9696 reports "we are currently experiencing
> technical difficulties and cannot take your call" then DISCONNECT.
>
> UPDATE
> One of my other vendors says that they have resolved it and it seems
> to have improved. According to them, It was indeed a cap on the
> number of routes accepted by XO from their peers.
>
> Cheers,
> Chris
> _______________________________________________
> Outages mailing list
> Outages at outages.org
> https://puck.nether.net/mailman/listinfo/outages
>
>
> _______________________________________________
> Outages mailing list
> Outages at outages.org
> https://puck.nether.net/mailman/listinfo/outages
>
>
> _______________________________________________
> Outages mailing list
> Outages at outages.org
> https://puck.nether.net/mailman/listinfo/outages

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Hi,

On 09/16/2010 11:04 AM, Dylan Ebner wrote:
> It appears the outages mailing list is running upto 30 min. behind from the time people post. People were most likely thinking they were the first ones to notice and start a thread. I don't know if this was caused by the volume of messages regarding the XO issue or if outages.org possibly was affected by xo. I have seen in the past the outages list run behind.
>
> Dylan Ebner, Network Engineer
> Consulting Radiologists, Ltd.
> 1221 Nicollet Mall, Minneapolis, MN 55403
> ph. 612.573.2236 fax. 612.573.2250
> dylan.ebner at crlmed.com
> www.consultingradiologists.com
- ---------------------
You are correct in pointing this delay and Jared just addressed this
issue similar to what others have reported in the past.

Apparently the messages slowness was due to anti-spam greylist filtering
and this should be fixed starting now. Then again please let us know if
you notice it again.

Sorry about that.


regards,
/virendra
>
> -----Original Message-----
> From: outages-bounces at outages.org [mailto:outages-bounces at outages.org] On Behalf Of Scott Berkman
> Sent: Thursday, September 16, 2010 12:49 PM
> To: 'Christopher Hunt'; outages at outages.org
> Subject: Re: [outages] XO snafu
>
> Is there any chance that you guys can please try and look at the existing
> threads on the list before posting a question that has already been posted?
> If you have information to add that will help people you can reply to one of
> the existing threads, but having 15 different threads on the same topic is
> really annoying.
>
> Thanks,
>
> -Scott
>
> -----Original Message-----
> From: outages-bounces at outages.org [mailto:outages-bounces at outages.org] On
> Behalf Of Christopher Hunt
> Sent: Thursday, September 16, 2010 11:41 AM
> To: outages at outages.org
> Subject: [outages] XO snafu
>
> This morning we're experiencing timeouts to and from a lot of sites.
> I heard through the grapevine that XO instituted some new route limits
> on their peers. Their site at http://businesscenter.xo.com is down,
> their phone number 800-424-0583 is fast busy and their Premier
> National Accounts 866-295-9696 reports "we are currently experiencing
> technical difficulties and cannot take your call" then DISCONNECT.
>
> UPDATE
> One of my other vendors says that they have resolved it and it seems
> to have improved. According to them, It was indeed a cap on the
> number of routes accepted by XO from their peers.
>
> Cheers,
> Chris
> _______________________________________________
> Outages mailing list
> Outages at outages.org
> https://puck.nether.net/mailman/listinfo/outages
>
>
> _______________________________________________
> Outages mailing list
> Outages at outages.org
> https://puck.nether.net/mailman/listinfo/outages
>
> _______________________________________________
> Outages mailing list
> Outages at outages.org
> https://puck.nether.net/mailman/listinfo/outages

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.10 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iEYEARECAAYFAkySX3IACgkQpbZvCIJx1beSlQCePuqs7Y5mm9hc/In0VPKh4uhy
isYAnA+KL/2Pi3YYcswifRSzRKWqrP+Z
=qtmw
-----END PGP SIGNATURE-----

Interestingly-- some messages are coming in with 1 to 2 min lag..
(I checked some headers) and
I see some that are 30 min delayed....


Jim


-----Original Message-----
From: outages-bounces@outages.org [mailto:outages-bounces@outages.org] On Behalf Of Christopher Hunt
Sent: Thursday, September 16, 2010 1:52 PM
To: Scott Berkman
Cc: outages at outages.org; 'Christopher Hunt'
Subject: Re: [outages] XO snafu

I agree and apologize. Apparently the list serve is lagging these
messages and I didn't receive most threads until just recently.

-Chris
On 9/16/2010 10:48 AM, Scott Berkman wrote:
> Is there any chance that you guys can please try and look at the existing
> threads on the list before posting a question that has already been posted?
> If you have information to add that will help people you can reply to one of
> the existing threads, but having 15 different threads on the same topic is
> really annoying.
>
> Thanks,
>
> -Scott
>
> -----Original Message-----
> From: outages-bounces at outages.org [mailto:outages-bounces at outages.org] On
> Behalf Of Christopher Hunt
> Sent: Thursday, September 16, 2010 11:41 AM
> To: outages at outages.org
> Subject: [outages] XO snafu
>
> This morning we're experiencing timeouts to and from a lot of sites.
> I heard through the grapevine that XO instituted some new route limits
> on their peers. Their site at http://businesscenter.xo.com is down,
> their phone number 800-424-0583 is fast busy and their Premier
> National Accounts 866-295-9696 reports "we are currently experiencing
> technical difficulties and cannot take your call" then DISCONNECT.
>
> UPDATE
> One of my other vendors says that they have resolved it and it seems
> to have improved. According to them, It was indeed a cap on the
> number of routes accepted by XO from their peers.
>
> Cheers,
> Chris
> _______________________________________________
> Outages mailing list
> Outages at outages.org
> https://puck.nether.net/mailman/listinfo/outages
>
>
> _______________________________________________
> Outages mailing list
> Outages at outages.org
> https://puck.nether.net/mailman/listinfo/outages
_______________________________________________
Outages mailing list
Outages at outages.org
https://puck.nether.net/mailman/listinfo/outages

Anyone else notice a recurrence of XO issues today at about 11:58 GMT-4?

Traces during blip:

1 r1 (216.187.0.1) 0.720 ms 1.282 ms 1.542 ms
2 98.124.133.241 (98.124.133.241) 2.327 ms 15.476 ms 2.370 ms
3 te-1-1.bmf1.nyc1.gt-t.net (208.81.84.5) 2.830 ms 2.489 ms 2.135 ms
4 64.209.110.37 (64.209.110.37) 2.869 ms 2.464 ms 3.097 ms
5 * * *
6 * * *
7 * * *


1 r1 (216.187.0.1) 1.179 ms 0.708 ms 0.715 ms
2 98.124.133.241 (98.124.133.241) 2.131 ms 13.152 ms 2.367 ms
3 te-1-1.bmf1.nyc1.gt-t.net (208.81.84.5) 2.820 ms 2.482 ms 2.100 ms
4 64.209.110.37 (64.209.110.37) 3.013 ms 2.462 ms 3.001 ms
5 * * *
6 * * *
7 * * *


1 r1 (216.187.0.1) 1.714 ms 1.240 ms 1.073 ms
2 98.124.133.241 (98.124.133.241) 1.951 ms 13.566 ms 2.469 ms
3 te-1-1.bmf1.nyc1.gt-t.net (208.81.84.5) 2.802 ms 2.983 ms 2.066 ms
4 64.209.110.37 (64.209.110.37) 3.366 ms 2.215 ms 2.738 ms
5 * * *
6 * * *
7 * * *



Traces after blip:

1 r1 (216.187.0.1) 0.890 ms 0.949 ms 1.079 ms
2 98.124.133.241 (98.124.133.241) 1.450 ms 2.248 ms 2.811 ms
3 te-1-1.bmf1.nyc1.gt-t.net (208.81.84.5) 3.487 ms 2.725 ms 2.698 ms
4 64.209.110.37 (64.209.110.37) 2.240 ms 2.322 ms 2.507 ms
5 te2-1-0d0.cir1.nyc-ny.us.xo.net (206.111.13.53) 2.231 ms 3.890 ms 67.921 ms
6 207.88.15.22.ptr.us.xo.net (207.88.15.22) 2.400 ms 2.232 ms 2.903 ms
7 te5-0-0.cvr3.nyc-ny.us.xo.net (207.88.86.194) 3.656 ms 9.213 ms 3.198 ms
8 207.88.182.10.ptr.us.xo.net (207.88.182.10) 3.564 ms 2.559 ms 2.739 ms


1 r1 (216.187.0.1) 1.009 ms 1.061 ms 0.967 ms
2 98.124.133.241 (98.124.133.241) 2.260 ms 1.843 ms 1.736 ms
3 te-1-1.bmf1.nyc1.gt-t.net (208.81.84.5) 2.528 ms 3.096 ms 2.825 ms
4 64.209.110.37 (64.209.110.37) 2.544 ms 2.062 ms 2.708 ms
5 te2-1-0d0.cir1.nyc-ny.us.xo.net (206.111.13.53) 2.836 ms 69.511 ms 2.188 ms
6 207.88.15.22.ptr.us.xo.net (207.88.15.22) 2.375 ms 2.955 ms 3.108 ms
7 te5-0-0.cvr3.nyc-ny.us.xo.net (207.88.86.194) 2.458 ms 8.728 ms 3.686 ms
8 207.88.182.6.ptr.us.xo.net (207.88.182.6) 4.207 ms 4.449 ms 3.012 ms



1 r1 (216.187.0.1) 0.545 ms 0.966 ms 1.565 ms
2 98.124.133.241 (98.124.133.241) 2.129 ms 2.374 ms 2.036 ms
3 te-1-1.bmf1.nyc1.gt-t.net (208.81.84.5) 3.513 ms 2.732 ms 2.872 ms
4 64.209.110.37 (64.209.110.37) 2.323 ms 2.447 ms 2.425 ms
5 te2-1-0d0.cir1.nyc-ny.us.xo.net (206.111.13.53) 3.481 ms 68.279 ms 2.201 ms
6 207.88.15.22.ptr.us.xo.net (207.88.15.22) 2.476 ms 2.940 ms 3.108 ms
7 te5-0-0.cvr3.nyc-ny.us.xo.net (207.88.86.194) 2.304 ms 8.964 ms 3.195 ms
8 207.88.182.10.ptr.us.xo.net (207.88.182.10) 3.523 ms 2.656 ms 2.714 ms




On 9/16/2010 2:19 PM, Jim McBurnett wrote:
> Interestingly-- some messages are coming in with 1 to 2 min lag..
> (I checked some headers) and
> I see some that are 30 min delayed....
>
>
> Jim
>
>
> -----Original Message-----
> From: outages-bounces at outages.org [mailto:outages-bounces at outages.org] On Behalf Of Christopher Hunt
> Sent: Thursday, September 16, 2010 1:52 PM
> To: Scott Berkman
> Cc: outages at outages.org; 'Christopher Hunt'
> Subject: Re: [outages] XO snafu
>
> I agree and apologize. Apparently the list serve is lagging these
> messages and I didn't receive most threads until just recently.
>
> -Chris
> On 9/16/2010 10:48 AM, Scott Berkman wrote:
>> Is there any chance that you guys can please try and look at the existing
>> threads on the list before posting a question that has already been posted?
>> If you have information to add that will help people you can reply to one of
>> the existing threads, but having 15 different threads on the same topic is
>> really annoying.
>>
>> Thanks,
>>
>> -Scott
>>
>> -----Original Message-----
>> From: outages-bounces at outages.org [mailto:outages-bounces at outages.org] On
>> Behalf Of Christopher Hunt
>> Sent: Thursday, September 16, 2010 11:41 AM
>> To: outages at outages.org
>> Subject: [outages] XO snafu
>>
>> This morning we're experiencing timeouts to and from a lot of sites.
>> I heard through the grapevine that XO instituted some new route limits
>> on their peers. Their site at http://businesscenter.xo.com is down,
>> their phone number 800-424-0583 is fast busy and their Premier
>> National Accounts 866-295-9696 reports "we are currently experiencing
>> technical difficulties and cannot take your call" then DISCONNECT.
>>
>> UPDATE
>> One of my other vendors says that they have resolved it and it seems
>> to have improved. According to them, It was indeed a cap on the
>> number of routes accepted by XO from their peers.
>>
>> Cheers,
>> Chris
>> _______________________________________________
>> Outages mailing list
>> Outages at outages.org
>> https://puck.nether.net/mailman/listinfo/outages
>>
>>
>> _______________________________________________
>> Outages mailing list
>> Outages at outages.org
>> https://puck.nether.net/mailman/listinfo/outages
> _______________________________________________
> Outages mailing list
> Outages at outages.org
> https://puck.nether.net/mailman/listinfo/outages
>
> _______________________________________________
> Outages mailing list
> Outages at outages.org
> https://puck.nether.net/mailman/listinfo/outages
>
>