Mailing List Archive: peering in LA between XO and Savvis

We are seeing large (20-30%) packet loss in the forward and reverse path
from XO to Savvis in the Los Angeles peering point. This was happening
yesterday for about 2 hours, and has picked back up this morning. This
has made some sites hosted in Savvis unusable for us (like Netsuite).

Appropriate traces (some hops stripped to protect the innocent):

From Draper UT to Savvis SC4 datacenter

HOST: amds-mon Loss% Snt Last Avg Best Wrst StDev
1. stipped 0.0% 50 0.5 1.0 0.4 3.5 0.9
2. stripped 0.0% 50 29.1 10.7 0.2 38.6 13.9
3. stripped 0.0% 50 35.1 12.7 1.1 40.8 14.5
4. stripped 0.0% 50 43.1 21.8 2.9 48.9 15.8
5. p6-0-0d0.mar2.saltlake-ut.us 0.0% 50 40.0 25.2 3.1 155.9 25.8
6. p3-2-0d0.rar2.la-ca.us.xo.ne 0.0% 50 21.1 41.2 17.1 128.8 24.1
7. 207.88.12.157.ptr.us.xo.net 4.0% 50 26.6 34.9 17.8 60.9 15.7
8. 207.88.12.158.ptr.us.xo.net 0.0% 50 25.4 34.1 17.0 65.2 16.4
9. bpr2-so-6-1-0.losangeles.sav 30.0% 50 72.3 81.3 54.1 161.9 25.5
10. cr2-gig-0-7-5-2.losangeles.s 32.0% 50 86.1 80.3 57.0 107.8 17.0
11. cr2-pos0-0-0-0.sanfrancisco. 26.0% 50 76.7 90.9 65.5 116.8 16.3
12. er1-te-2-0-1.sanjose3equinix 44.0% 50 70.7 86.6 66.2 112.8 16.6
13. hr1-te-2-0-0.santaclarasc5.s 38.0% 50 70.8 86.2 66.6 116.5 16.8
14. 216.34.2.226 34.0% 50 92.1 174.0 66.8 823.9 203.2
15. stripped 36.0% 50 109.1 90.3 66.1 118.5 17.4
16. end-point 44.0% 50 107.1 89.7 66.8 115.6 18.9

From SC4 to same Draper UT network (reverse path, hop 16 is firewalled):

HOST: prod-monitor Loss% Snt Last Avg Best Wrst StDev

1. stripped 0.0% 50 0.4 0.4 0.3 0.5 0.1
2. stripped 0.0% 50 318.4 52.4 0.6 683.9 150.8
3. hr1-ge-7-47.santaclarasc5.sa 0.0% 50 0.9 3.3 0.6 128.2 18.0
4. hr1-te-1-0-1.santaclarasc4.s 0.0% 50 1.0 0.9 0.6 9.4 1.2
5. hr1-te-2-0-0.santaclarasc9.s 0.0% 50 1.1 1.8 0.6 28.6 4.1
6. pr1-ge-4-0-0.SanJoseEquinix. 0.0% 50 1.3 7.1 1.0 128.2 22.7
7. pr2-so-0-0-0.PaloAltoPaix.sa 0.0% 50 10.5 5.7 2.2 81.6 12.5
8. p2-2.IR1.PaloAlto-CA.us.xo.n 0.0% 50 2.7 2.3 2.0 3.1 0.2
9. 65.106.1.66.ptr.us.xo.net 0.0% 50 3.7 3.8 3.1 13.0 1.4
10. 65.106.1.65.ptr.us.xo.net 0.0% 50 15.7 6.1 3.0 35.9 7.8
11. p6-0-0.RAR1.LA-CA.us.xo.net 30.0% 50 56.4 57.8 49.9 91.1 8.7
12. p0-0-0d0.rar2.la-ca.us.xo.ne 34.0% 50 55.1 63.6 43.9 157.8 25.7
13. p1-0-0d0.mar2.saltlake-ut.us 42.0% 50 68.8 74.0 59.1 151.4 16.8
14. p15-0.chr1.saltlake-ut.us.xo 36.0% 50 72.8 68.2 60.6 76.3 3.6
15. stripped 36.0% 50 73.7 72.7 62.8 81.7 3.7
16. ??? 100.0 50 0.0 0.0 0.0 0.0 0.0

--Justin

After working with both vendors, it seems that a DDoS attack, aimed else
where, which transited XO's peering router in LA was the root cause of
this issue.

Seems to be cleared up now.

Justin Sharp wrote:
> We are seeing large (20-30%) packet loss in the forward and reverse
> path from XO to Savvis in the Los Angeles peering point. This was
> happening yesterday for about 2 hours, and has picked back up this
> morning. This has made some sites hosted in Savvis unusable for us
> (like Netsuite).
>
> Appropriate traces (some hops stripped to protect the innocent):
>
> From Draper UT to Savvis SC4 datacenter
>
> HOST: amds-mon Loss% Snt Last Avg Best Wrst
> StDev
> 1. stipped 0.0% 50 0.5 1.0 0.4 3.5 0.9
> 2. stripped 0.0% 50 29.1 10.7 0.2 38.6 13.9
> 3. stripped 0.0% 50 35.1 12.7 1.1 40.8 14.5
> 4. stripped 0.0% 50 43.1 21.8 2.9 48.9 15.8
> 5. p6-0-0d0.mar2.saltlake-ut.us 0.0% 50 40.0 25.2 3.1 155.9
> 25.8
> 6. p3-2-0d0.rar2.la-ca.us.xo.ne 0.0% 50 21.1 41.2 17.1 128.8
> 24.1
> 7. 207.88.12.157.ptr.us.xo.net 4.0% 50 26.6 34.9 17.8 60.9
> 15.7
> 8. 207.88.12.158.ptr.us.xo.net 0.0% 50 25.4 34.1 17.0 65.2
> 16.4
> 9. bpr2-so-6-1-0.losangeles.sav 30.0% 50 72.3 81.3 54.1 161.9
> 25.5
> 10. cr2-gig-0-7-5-2.losangeles.s 32.0% 50 86.1 80.3 57.0 107.8
> 17.0
> 11. cr2-pos0-0-0-0.sanfrancisco. 26.0% 50 76.7 90.9 65.5 116.8
> 16.3
> 12. er1-te-2-0-1.sanjose3equinix 44.0% 50 70.7 86.6 66.2 112.8
> 16.6
> 13. hr1-te-2-0-0.santaclarasc5.s 38.0% 50 70.8 86.2 66.6 116.5
> 16.8
> 14. 216.34.2.226 34.0% 50 92.1 174.0 66.8 823.9
> 203.2
> 15. stripped 36.0% 50 109.1 90.3 66.1 118.5 17.4
> 16. end-point 44.0% 50 107.1 89.7 66.8 115.6 18.9
>
>
>
>> From SC4 to same Draper UT network (reverse path, hop 16 is
>> firewalled):
>
> HOST: prod-monitor Loss% Snt Last Avg Best Wrst
> StDev
>
> 1. stripped 0.0% 50 0.4 0.4 0.3 0.5 0.1
> 2. stripped 0.0% 50 318.4 52.4 0.6 683.9 150.8
> 3. hr1-ge-7-47.santaclarasc5.sa 0.0% 50 0.9 3.3 0.6 128.2
> 18.0
> 4. hr1-te-1-0-1.santaclarasc4.s 0.0% 50 1.0 0.9 0.6
> 9.4 1.2
> 5. hr1-te-2-0-0.santaclarasc9.s 0.0% 50 1.1 1.8 0.6
> 28.6 4.1
> 6. pr1-ge-4-0-0.SanJoseEquinix. 0.0% 50 1.3 7.1 1.0 128.2
> 22.7
> 7. pr2-so-0-0-0.PaloAltoPaix.sa 0.0% 50 10.5 5.7 2.2 81.6
> 12.5
> 8. p2-2.IR1.PaloAlto-CA.us.xo.n 0.0% 50 2.7 2.3 2.0
> 3.1 0.2
> 9. 65.106.1.66.ptr.us.xo.net 0.0% 50 3.7 3.8 3.1
> 13.0 1.4
> 10. 65.106.1.65.ptr.us.xo.net 0.0% 50 15.7 6.1 3.0
> 35.9 7.8
> 11. p6-0-0.RAR1.LA-CA.us.xo.net 30.0% 50 56.4 57.8 49.9
> 91.1 8.7
> 12. p0-0-0d0.rar2.la-ca.us.xo.ne 34.0% 50 55.1 63.6 43.9 157.8
> 25.7
> 13. p1-0-0d0.mar2.saltlake-ut.us 42.0% 50 68.8 74.0 59.1 151.4
> 16.8
> 14. p15-0.chr1.saltlake-ut.us.xo 36.0% 50 72.8 68.2 60.6
> 76.3 3.6
> 15. stripped 36.0% 50 73.7 72.7 62.8 81.7 3.7
> 16. ??? 100.0 50 0.0 0.0 0.0
> 0.0 0.0
>
>
> --Justin
>
> _______________________________________________
> outages mailing list
> outages at outages.org
> https://puck.nether.net/mailman/listinfo/outages