Mailing List Archive: Twelve99 / AWS usw2 significant loss

Twelve99 / AWS usw2 significant loss

Jan 26, 2024, 12:19 AM

Post #1 of 7 (183 views)

Hi Folks,

Cross posting to outages and nanog.

Can anybody from Twelve99 / AWS investigate high loss and latency around SJO between Twelve99 and AWS?

Traces below:

88.99.88.67 to 216.147.3.209:

My traceroute [v0.94]
phil (10.88.10.10) -> 216.147.3.209 2024-01-26T08:16:16+0000
Keys: Help Display mode Restart statistics Order of fields quit
Packets Pings
Host Loss% Snt Last Avg Best Wrst StDev
1. 10.88.10.254 0.0% 176 0.2 0.1 0.1 0.3 0.1
2. fe-fw-01.lavin.me.uk 0.0% 176 0.4 0.4 0.2 0.7 0.1
3. 10.69.69.1 0.0% 176 0.5 0.4 0.2 0.7 0.1
4. static.225.8.4.46.clients.your-server.de 0.0% 176 0.8 1.0 0.5 8.2 1.0
5. core22.fsn1.hetzner.com 0.6% 176 0.8 2.0 0.7 39.0 4.6
6. static.213-239-254-234.clients.your-server.de 0.0% 176 3.2 3.8 3.1 36.5 3.3
7. nug-b1-link.ip.twelve99.net 0.0% 176 3.3 3.5 3.1 24.1 1.6
8. hbg-bb2-link.ip.twelve99.net 86.9% 175 18.9 18.9 18.7 19.2 0.1
9. ldn-bb2-link.ip.twelve99.net 92.0% 175 30.5 30.6 30.4 30.8 0.1
10. nyk-bb1-link.ip.twelve99.net 4.6% 175 99.5 99.5 99.3 100.1 0.2
11. sjo-b23-link.ip.twelve99.net 56.3% 175 296.8 306.0 289.7 315.0 5.5
12. amazon-ic-366608.ip.twelve99-cust.net 80.5% 175 510.0 513.5 500.7 539.7 8.4
13. (waiting for reply)
…...
26. 216.147.3.209 59.1% 150 534.2 534.4 513.5 549.7 8.1

44.236.47.236 to 178.63.26.145:

jcasc-rtc01 (10.96.54.240) -> hc-ping.com 2024-01-26T08:10:24+0000
Keys: Help Display mode Restart statistics Order of fields quit
Packets Pings
Host Loss% Snt Last Avg Best Wrst StDev
1. ip-10-96-50-153.us-west-2.compute.internal 0.0% 267 0.2 0.2 0.2 0.4 0.0
2. ec2-34-221-151-225.us-west-2.compute.amazonaw 0.0% 267 6.8 15.9 0.7 114.1 24.1
3. (waiting for reply)
4. (waiting for reply)
5. (waiting for reply)
6. 241.0.2.5 0.0% 267 0.4 0.4 0.3 1.2 0.1
7. 242.0.42.195 0.0% 267 1.2 1.7 0.5 23.9 3.0
8. 240.2.144.12 0.0% 267 5.2 5.2 5.2 27.4 1.4
9. amazon-ic-375418.ip.twelve99-cust.net 0.0% 267 5.3 5.2 5.2 6.1 0.1
10. port-b4-link.ip.twelve99.net 0.0% 267 5.3 5.5 4.9 35.5 3.0
11. port-b3-link.ip.twelve99.net 0.0% 267 5.8 5.9 5.6 11.8 0.5
12. palo-b24-link.ip.twelve99.net 4.9% 267 21.1 21.5 21.0 58.4 3.1
13. sjo-b23-link.ip.twelve99.net 0.0% 266 21.4 22.7 21.3 86.2 6.5
14. nyk-bb1-link.ip.twelve99.net 58.1% 266 432.7 422.7 407.2 438.5 6.5
15. ldn-bb2-link.ip.twelve99.net 98.1% 266 485.6 485.4 481.6 491.1 3.9
16. hbg-bb2-link.ip.twelve99.net 92.5% 266 504.1 499.8 489.8 510.1 5.9
17. nug-b1-link.ip.twelve99.net 55.5% 266 523.5 519.6 504.4 561.7 7.6
18. hetzner-ic-340780.ip.twelve99-cust.net 53.6% 266 524.4 519.2 506.0 545.5 6.9
19. core22.fsn1.hetzner.com 70.2% 266 521.7 519.2 498.5 531.7 6.6
20. static.213-239-254-150.clients.your-server.de 33.2% 266 382.4 375.4 364.9 396.5 4.1
21. static.145.26.63.178.clients.your-server.de 62.0% 266 529.9 518.4 506.9 531.3 6.1

Thanks

Re: Twelve99 / AWS usw2 significant loss [ In reply to ]

saku at ytti

Jan 26, 2024, 12:40 AM

Post #2 of 7 (183 views)

Permalink

On Fri, 26 Jan 2024 at 10:23, Phil Lavin via NANOG <nanog@nanog.org> wrote:

> 88.99.88.67 to 216.147.3.209:
> Host Loss% Snt Last Avg Best Wrst StDev
> 1. 10.88.10.254 0.0% 176 0.2 0.1 0.1 0.3 0.1
> 7. nug-b1-link.ip.twelve99.net 0.0% 176 3.3 3.5 3.1 24.1 1.6
> 8. hbg-bb2-link.ip.twelve99.net 86.9% 175 18.9 18.9 18.7 19.2 0.1
> 9. ldn-bb2-link.ip.twelve99.net 92.0% 175 30.5 30.6 30.4 30.8 0.1
> 10. nyk-bb1-link.ip.twelve99.net 4.6% 175 99.5 99.5 99.3 100.1 0.2
> 11. sjo-b23-link.ip.twelve99.net 56.3% 175 296.8 306.0 289.7 315.0 5.5
> 12. amazon-ic-366608.ip.twelve99-cust.net 80.5% 175 510.0 513.5 500.7 539.7 8.4

This implies the problem is not on this path, because #10 is not
experiencing it, possibly because it happens to return a packet via
another option, but certainly shows the problem didn't happen in this
direction yet at #10, but because #8 and #9 saw it, they already saw
it on the other direction.

> 44.236.47.236 to 178.63.26.145:
> Host Loss% Snt Last Avg Best Wrst StDev
> 1. ip-10-96-50-153.us-west-2.compute.internal 0.0% 267 0.2 0.2 0.2 0.4 0.0
> 11. port-b3-link.ip.twelve99.net 0.0% 267 5.8 5.9 5.6 11.8 0.5
> 12. palo-b24-link.ip.twelve99.net 4.9% 267 21.1 21.5 21.0 58.4 3.1
> 13. sjo-b23-link.ip.twelve99.net 0.0% 266 21.4 22.7 21.3 86.2 6.5
> 14. nyk-bb1-link.ip.twelve99.net 58.1% 266 432.7 422.7 407.2 438.5 6.5
> 15. ldn-bb2-link.ip.twelve99.net 98.1% 266 485.6 485.4 481.6 491.1 3.9
> 16. hbg-bb2-link.ip.twelve99.net 92.5% 266 504.1 499.8 489.8 510.1 5.9
> 17. nug-b1-link.ip.twelve99.net 55.5% 266 523.5 519.6 504.4 561.7 7.6
> 18. hetzner-ic-340780.ip.twelve99-cust.net 53.6% 266 524.4 519.2 506.0 545.5 6.9
> 19. core22.fsn1.hetzner.com 70.2% 266 521.7 519.2 498.5 531.7 6.6
> 20. static.213-239-254-150.clients.your-server.de 33.2% 266 382.4 375.4 364.9 396.5 4.1
> 21. static.145.26.63.178.clients.your-server.de 62.0% 266 529.9 518.4 506.9 531.3 6.1

This suggests the congestion point is from sjo to nyk, in 1299, not AWS at all.

You could try to fix SPORT/DPORT, and do several SPORT options, to see
if loss goes away with some, to determine if all LAG members are full
or just one.

At any rate, this seems business as usual, sometimes internet is very
lossy, you should contact your service provider, which I guess is AWS
here, so they can contact their service provider or 1299.

--
++ytti

Re: Twelve99 / AWS usw2 significant loss [ In reply to ]

nanog at nanog

Jan 26, 2024, 12:45 AM

Post #3 of 7 (183 views)

Permalink

Thanks, ytti. I have raised a case with AWS but I expect it to be as unproductive as usual. In this type of issue, there is often a friendly Engineer keeping an eye on NANOG list or IRC channel who can mitigate it. Hoping that to be the case this time, also.

> On 26 Jan 2024, at 08:40, Saku Ytti <saku@ytti.fi> wrote:
>
> On Fri, 26 Jan 2024 at 10:23, Phil Lavin via NANOG <nanog@nanog.org> wrote:
>
>
>> 88.99.88.67 to 216.147.3.209:
>> Host Loss% Snt Last Avg Best Wrst StDev
>> 1. 10.88.10.254 0.0% 176 0.2 0.1 0.1 0.3 0.1
>> 7. nug-b1-link.ip.twelve99.net 0.0% 176 3.3 3.5 3.1 24.1 1.6
>> 8. hbg-bb2-link.ip.twelve99.net 86.9% 175 18.9 18.9 18.7 19.2 0.1
>> 9. ldn-bb2-link.ip.twelve99.net 92.0% 175 30.5 30.6 30.4 30.8 0.1
>> 10. nyk-bb1-link.ip.twelve99.net 4.6% 175 99.5 99.5 99.3 100.1 0.2
>> 11. sjo-b23-link.ip.twelve99.net 56.3% 175 296.8 306.0 289.7 315.0 5.5
>> 12. amazon-ic-366608.ip.twelve99-cust.net 80.5% 175 510.0 513.5 500.7 539.7 8.4
>
> This implies the problem is not on this path, because #10 is not
> experiencing it, possibly because it happens to return a packet via
> another option, but certainly shows the problem didn't happen in this
> direction yet at #10, but because #8 and #9 saw it, they already saw
> it on the other direction.
>
>
>> 44.236.47.236 to 178.63.26.145:
>> Host Loss% Snt Last Avg Best Wrst StDev
>> 1. ip-10-96-50-153.us-west-2.compute.internal 0.0% 267 0.2 0.2 0.2 0.4 0.0
>> 11. port-b3-link.ip.twelve99.net 0.0% 267 5.8 5.9 5.6 11.8 0.5
>> 12. palo-b24-link.ip.twelve99.net 4.9% 267 21.1 21.5 21.0 58.4 3.1
>> 13. sjo-b23-link.ip.twelve99.net 0.0% 266 21.4 22.7 21.3 86.2 6.5
>> 14. nyk-bb1-link.ip.twelve99.net 58.1% 266 432.7 422.7 407.2 438.5 6.5
>> 15. ldn-bb2-link.ip.twelve99.net 98.1% 266 485.6 485.4 481.6 491.1 3.9
>> 16. hbg-bb2-link.ip.twelve99.net 92.5% 266 504.1 499.8 489.8 510.1 5.9
>> 17. nug-b1-link.ip.twelve99.net 55.5% 266 523.5 519.6 504.4 561.7 7.6
>> 18. hetzner-ic-340780.ip.twelve99-cust.net 53.6% 266 524.4 519.2 506.0 545.5 6.9
>> 19. core22.fsn1.hetzner.com 70.2% 266 521.7 519.2 498.5 531.7 6.6
>> 20. static.213-239-254-150.clients.your-server.de 33.2% 266 382.4 375.4 364.9 396.5 4.1
>> 21. static.145.26.63.178.clients.your-server.de 62.0% 266 529.9 518.4 506.9 531.3 6.1
>
> This suggests the congestion point is from sjo to nyk, in 1299, not AWS at all.
>
> You could try to fix SPORT/DPORT, and do several SPORT options, to see
> if loss goes away with some, to determine if all LAG members are full
> or just one.
>
>
> At any rate, this seems business as usual, sometimes internet is very
> lossy, you should contact your service provider, which I guess is AWS
> here, so they can contact their service provider or 1299.
>
> --
> ++ytti

Re: Twelve99 / AWS usw2 significant loss [ In reply to ]

nanog at nanog

Jan 26, 2024, 12:59 AM

Post #4 of 7 (183 views)

Permalink

I see the AWS->me path has just changed to Cogent. me->AWS path still appears to be Twelve99. Loss and latency has now subsided.

Thanks

> On 26 Jan 2024, at 08:19, Phil Lavin <phil.lavin@vonage.com> wrote:
>
> Hi Folks,
>
> Cross posting to outages and nanog.
>
> Can anybody from Twelve99 / AWS investigate high loss and latency around SJO between Twelve99 and AWS?
>
> Traces below:
>
> 88.99.88.67 to 216.147.3.209:
>
> My traceroute [v0.94]
> phil (10.88.10.10) -> 216.147.3.209 2024-01-26T08:16:16+0000
> Keys: Help Display mode Restart statistics Order of fields quit
> Packets Pings
> Host Loss% Snt Last Avg Best Wrst StDev
> 1. 10.88.10.254 0.0% 176 0.2 0.1 0.1 0.3 0.1
> 2. fe-fw-01.lavin.me.uk 0.0% 176 0.4 0.4 0.2 0.7 0.1
> 3. 10.69.69.1 0.0% 176 0.5 0.4 0.2 0.7 0.1
> 4. static.225.8.4.46.clients.your-server.de 0.0% 176 0.8 1.0 0.5 8.2 1.0
> 5. core22.fsn1.hetzner.com 0.6% 176 0.8 2.0 0.7 39.0 4.6
> 6. static.213-239-254-234.clients.your-server.de 0.0% 176 3.2 3.8 3.1 36.5 3.3
> 7. nug-b1-link.ip.twelve99.net 0.0% 176 3.3 3.5 3.1 24.1 1.6
> 8. hbg-bb2-link.ip.twelve99.net 86.9% 175 18.9 18.9 18.7 19.2 0.1
> 9. ldn-bb2-link.ip.twelve99.net 92.0% 175 30.5 30.6 30.4 30.8 0.1
> 10. nyk-bb1-link.ip.twelve99.net 4.6% 175 99.5 99.5 99.3 100.1 0.2
> 11. sjo-b23-link.ip.twelve99.net 56.3% 175 296.8 306.0 289.7 315.0 5.5
> 12. amazon-ic-366608.ip.twelve99-cust.net 80.5% 175 510.0 513.5 500.7 539.7 8.4
> 13. (waiting for reply)
> …...
> 26. 216.147.3.209 59.1% 150 534.2 534.4 513.5 549.7 8.1
>
> 44.236.47.236 to 178.63.26.145:
>
> jcasc-rtc01 (10.96.54.240) -> hc-ping.com 2024-01-26T08:10:24+0000
> Keys: Help Display mode Restart statistics Order of fields quit
> Packets Pings
> Host Loss% Snt Last Avg Best Wrst StDev
> 1. ip-10-96-50-153.us-west-2.compute.internal 0.0% 267 0.2 0.2 0.2 0.4 0.0
> 2. ec2-34-221-151-225.us-west-2.compute.amazonaw 0.0% 267 6.8 15.9 0.7 114.1 24.1
> 3. (waiting for reply)
> 4. (waiting for reply)
> 5. (waiting for reply)
> 6. 241.0.2.5 0.0% 267 0.4 0.4 0.3 1.2 0.1
> 7. 242.0.42.195 0.0% 267 1.2 1.7 0.5 23.9 3.0
> 8. 240.2.144.12 0.0% 267 5.2 5.2 5.2 27.4 1.4
> 9. amazon-ic-375418.ip.twelve99-cust.net 0.0% 267 5.3 5.2 5.2 6.1 0.1
> 10. port-b4-link.ip.twelve99.net 0.0% 267 5.3 5.5 4.9 35.5 3.0
> 11. port-b3-link.ip.twelve99.net 0.0% 267 5.8 5.9 5.6 11.8 0.5
> 12. palo-b24-link.ip.twelve99.net 4.9% 267 21.1 21.5 21.0 58.4 3.1
> 13. sjo-b23-link.ip.twelve99.net 0.0% 266 21.4 22.7 21.3 86.2 6.5
> 14. nyk-bb1-link.ip.twelve99.net 58.1% 266 432.7 422.7 407.2 438.5 6.5
> 15. ldn-bb2-link.ip.twelve99.net 98.1% 266 485.6 485.4 481.6 491.1 3.9
> 16. hbg-bb2-link.ip.twelve99.net 92.5% 266 504.1 499.8 489.8 510.1 5.9
> 17. nug-b1-link.ip.twelve99.net 55.5% 266 523.5 519.6 504.4 561.7 7.6
> 18. hetzner-ic-340780.ip.twelve99-cust.net 53.6% 266 524.4 519.2 506.0 545.5 6.9
> 19. core22.fsn1.hetzner.com 70.2% 266 521.7 519.2 498.5 531.7 6.6
> 20. static.213-239-254-150.clients.your-server.de 33.2% 266 382.4 375.4 364.9 396.5 4.1
> 21. static.145.26.63.178.clients.your-server.de 62.0% 266 529.9 518.4 506.9 531.3 6.1
>
>
> Thanks

Re: Twelve99 / AWS usw2 significant loss [ In reply to ]

nanog at nanog

Jan 26, 2024, 1:07 AM

Post #5 of 7 (181 views)

Permalink

Hi,

> Cross posting to outages and nanog.
>
> Can anybody from Twelve99 / AWS investigate high loss and latency around SJO between Twelve99 and AWS?
>
> Traces below:
>
> 88.99.88.67 to 216.147.3.209:
>

this seems not only AWS related - from AS50629 to AS36086 via AS1299 -
also high latency and loss since sjo:

9.|-- adm-bb2-link.ip.twelve99. 60.0% 10 10.1 10.3 10.1 10.5
0.2
10.|-- ldn-bb2-link.ip.twelve99. 90.0% 10 15.4 15.4 15.4 15.4
0.0
11.|-- nyk-bb1-link.ip.twelve99. 0.0% 10 84.2 84.2 83.9 84.8
0.3
12.|-- sjo-b23-link.ip.twelve99. 20.0% 10 296.5 297.0 294.9 298.3
1.1
13.|-- palo-b24-link.ip.twelve99 20.0% 10 485.5 487.2 481.9 491.2
3.2
14.|-- digitalrealty-ic-360311.i 70.0% 10 488.8 485.2 482.4 488.8
3.3
15.|-- irb-20-cr1-sfo1-dia.telxg 20.0% 10 510.1 508.9 502.5 516.9
5.0
16.|-- 208.65.109.190 60.0% 10 506.1 501.3 498.3 506.1
3.6
17.|-- ice5.somafm.com 60.0% 10 499.3 501.1 498.8 504.2
2.5

Regards
Bjoern

Re: [outages] Twelve99 / AWS usw2 significant loss [ In reply to ]

diosbejgli at gmail

Jan 26, 2024, 4:43 PM

Post #6 of 7 (171 views)

Permalink

Seems like the destination is in Hetzner, they could also raise it with
Twelve99 or prepend routes to use an alternate path.

On Fri, Jan 26, 2024 at 7:46?PM Phil Lavin via Outages <outages@outages.org>
wrote:

> Thanks, ytti. I have raised a case with AWS but I expect it to be as
> unproductive as usual. In this type of issue, there is often a friendly
> Engineer keeping an eye on NANOG list or IRC channel who can mitigate it.
> Hoping that to be the case this time, also.
>
>
> > On 26 Jan 2024, at 08:40, Saku Ytti <saku@ytti.fi> wrote:
> >
> > On Fri, 26 Jan 2024 at 10:23, Phil Lavin via NANOG <nanog@nanog.org>
> wrote:
> >
> >
> >> 88.99.88.67 to 216.147.3.209:
> >> Host Loss% Snt Last
> Avg Best Wrst StDev
> >> 1. 10.88.10.254 0.0% 176 0.2
> 0.1 0.1 0.3 0.1
> >> 7. nug-b1-link.ip.twelve99.net 0.0% 176 3.3
> 3.5 3.1 24.1 1.6
> >> 8. hbg-bb2-link.ip.twelve99.net 86.9% 175 18.9
> 18.9 18.7 19.2 0.1
> >> 9. ldn-bb2-link.ip.twelve99.net 92.0% 175 30.5
> 30.6 30.4 30.8 0.1
> >> 10. nyk-bb1-link.ip.twelve99.net 4.6% 175
> 99.5 99.5 99.3 100.1 0.2
> >> 11. sjo-b23-link.ip.twelve99.net 56.3% 175 296.8
> 306.0 289.7 315.0 5.5
> >> 12. amazon-ic-366608.ip.twelve99-cust.net 80.5% 175 510.0
> 513.5 500.7 539.7 8.4
> >
> > This implies the problem is not on this path, because #10 is not
> > experiencing it, possibly because it happens to return a packet via
> > another option, but certainly shows the problem didn't happen in this
> > direction yet at #10, but because #8 and #9 saw it, they already saw
> > it on the other direction.
> >
> >
> >> 44.236.47.236 to 178.63.26.145:
> >> Host Loss% Snt Last
> Avg Best Wrst StDev
> >> 1. ip-10-96-50-153.us-west-2.compute.internal 0.0% 267 0.2
> 0.2 0.2 0.4 0.0
> >> 11. port-b3-link.ip.twelve99.net 0.0% 267 5.8
> 5.9 5.6 11.8 0.5
> >> 12. palo-b24-link.ip.twelve99.net 4.9% 267 21.1
> 21.5 21.0 58.4 3.1
> >> 13. sjo-b23-link.ip.twelve99.net 0.0% 266 21.4
> 22.7 21.3 86.2 6.5
> >> 14. nyk-bb1-link.ip.twelve99.net 58.1% 266 432.7
> 422.7 407.2 438.5 6.5
> >> 15. ldn-bb2-link.ip.twelve99.net 98.1% 266 485.6
> 485.4 481.6 491.1 3.9
> >> 16. hbg-bb2-link.ip.twelve99.net 92.5% 266 504.1
> 499.8 489.8 510.1 5.9
> >> 17. nug-b1-link.ip.twelve99.net 55.5% 266 523.5
> 519.6 504.4 561.7 7.6
> >> 18. hetzner-ic-340780.ip.twelve99-cust.net 53.6% 266 524.4
> 519.2 506.0 545.5 6.9
> >> 19. core22.fsn1.hetzner.com 70.2% 266 521.7
> 519.2 498.5 531.7 6.6
> >> 20. static.213-239-254-150.clients.your-server.de 33.2% 266 382.4
> 375.4 364.9 396.5 4.1
> >> 21. static.145.26.63.178.clients.your-server.de 62.0% 266 529.9
> 518.4 506.9 531.3 6.1
> >
> > This suggests the congestion point is from sjo to nyk, in 1299, not AWS
> at all.
> >
> > You could try to fix SPORT/DPORT, and do several SPORT options, to see
> > if loss goes away with some, to determine if all LAG members are full
> > or just one.
> >
> >
> > At any rate, this seems business as usual, sometimes internet is very
> > lossy, you should contact your service provider, which I guess is AWS
> > here, so they can contact their service provider or 1299.
> >
> > --
> > ++ytti
>
> _______________________________________________
> Outages mailing list
> Outages@outages.org
> https://puck.nether.net/mailman/listinfo/outages
>

Re: [outages] Twelve99 / AWS usw2 significant loss [ In reply to ]

bryan at shout

Jan 27, 2024, 3:29 AM

Post #7 of 7 (169 views)

Permalink

Can we not talk about prepending again? ;)

On 1/27/24 01:43, Andras Toth wrote:
> Seems like the destination is in Hetzner, they could also raise it with
> Twelve99 or prepend routes to use an alternate path.
>
>
> On Fri, Jan 26, 2024 at 7:46?PM Phil Lavin via Outages
> <outages@outages.org <mailto:outages@outages.org>> wrote:
>
> Thanks, ytti. I have raised a case with AWS but I expect it to be as
> unproductive as usual. In this type of issue, there is often a
> friendly Engineer keeping an eye on NANOG list or IRC channel who
> can mitigate it. Hoping that to be the case this time, also.
>
>
> > On 26 Jan 2024, at 08:40, Saku Ytti <saku@ytti.fi
> <mailto:saku@ytti.fi>> wrote:
> >
> > On Fri, 26 Jan 2024 at 10:23, Phil Lavin via NANOG
> <nanog@nanog.org <mailto:nanog@nanog.org>> wrote:
> >
> >
> >> 88.99.88.67 to *MailScanner warning: numerical links are often
> malicious:* 216.147.3.209 <http://216.147.3.209>:
> >> Host Loss% Snt
> Last Avg Best Wrst StDev
> >> 1. 10.88.10.254 0.0% 176
> 0.2 0.1 0.1 0.3 0.1
> >> 7. nug-b1-link.ip.twelve99.net
> <http://nug-b1-link.ip.twelve99.net> 0.0%
> 176 3.3 3.5 3.1 24.1 1.6
> >> 8. hbg-bb2-link.ip.twelve99.net
> <http://hbg-bb2-link.ip.twelve99.net> 86.9%
> 175 18.9 18.9 18.7 19.2 0.1
> >> 9. ldn-bb2-link.ip.twelve99.net
> <http://ldn-bb2-link.ip.twelve99.net> 92.0%
> 175 30.5 30.6 30.4 30.8 0.1
> >> 10. nyk-bb1-link.ip.twelve99.net
> <http://nyk-bb1-link.ip.twelve99.net> 4.6%
> 175 99.5 99.5 99.3 100.1 0.2
> >> 11. sjo-b23-link.ip.twelve99.net
> <http://sjo-b23-link.ip.twelve99.net> 56.3%
> 175 296.8 306.0 289.7 315.0 5.5
> >> 12. amazon-ic-366608.ip.twelve99-cust.net
> <http://amazon-ic-366608.ip.twelve99-cust.net> 80.5%
> 175 510.0 513.5 500.7 539.7 8.4
> >
> > This implies the problem is not on this path, because #10 is not
> > experiencing it, possibly because it happens to return a packet via
> > another option, but certainly shows the problem didn't happen in this
> > direction yet at #10, but because #8 and #9 saw it, they already saw
> > it on the other direction.
> >
> >
> >> 44.236.47.236 to *MailScanner warning: numerical links are often
> malicious:* 178.63.26.145 <http://178.63.26.145>:
> >> Host Loss% Snt
> Last Avg Best Wrst StDev
> >> 1. ip-10-96-50-153.us-west-2.compute.internal 0.0% 267
> 0.2 0.2 0.2 0.4 0.0
> >> 11. port-b3-link.ip.twelve99.net
> <http://port-b3-link.ip.twelve99.net> 0.0% 267
> 5.8 5.9 5.6 11.8 0.5
> >> 12. palo-b24-link.ip.twelve99.net
> <http://palo-b24-link.ip.twelve99.net> 4.9% 267
> 21.1 21.5 21.0 58.4 3.1
> >> 13. sjo-b23-link.ip.twelve99.net
> <http://sjo-b23-link.ip.twelve99.net> 0.0% 266
> 21.4 22.7 21.3 86.2 6.5
> >> 14. nyk-bb1-link.ip.twelve99.net
> <http://nyk-bb1-link.ip.twelve99.net> 58.1% 266
> 432.7 422.7 407.2 438.5 6.5
> >> 15. ldn-bb2-link.ip.twelve99.net
> <http://ldn-bb2-link.ip.twelve99.net> 98.1% 266
> 485.6 485.4 481.6 491.1 3.9
> >> 16. hbg-bb2-link.ip.twelve99.net
> <http://hbg-bb2-link.ip.twelve99.net> 92.5% 266
> 504.1 499.8 489.8 510.1 5.9
> >> 17. nug-b1-link.ip.twelve99.net
> <http://nug-b1-link.ip.twelve99.net> 55.5% 266
> 523.5 519.6 504.4 561.7 7.6
> >> 18. hetzner-ic-340780.ip.twelve99-cust.net
> <http://hetzner-ic-340780.ip.twelve99-cust.net> 53.6% 266
> 524.4 519.2 506.0 545.5 6.9
> >> 19. core22.fsn1.hetzner.com <http://core22.fsn1.hetzner.com>
> 70.2% 266 521.7 519.2 498.5 531.7 6.6
> >> 20. static.213-239-254-150.clients.your-server.de
> <http://static.213-239-254-150.clients.your-server.de> 33.2% 266
> 382.4 375.4 364.9 396.5 4.1
> >> 21. static.145.26.63.178.clients.your-server.de
> <http://static.145.26.63.178.clients.your-server.de> 62.0% 266
> 529.9 518.4 506.9 531.3 6.1
> >
> > This suggests the congestion point is from sjo to nyk, in 1299,
> not AWS at all.
> >
> > You could try to fix SPORT/DPORT, and do several SPORT options,
> to see
> > if loss goes away with some, to determine if all LAG members are full
> > or just one.
> >
> >
> > At any rate, this seems business as usual, sometimes internet is very
> > lossy, you should contact your service provider, which I guess is AWS
> > here, so they can contact their service provider or 1299.
> >
> > --
> > ++ytti
>
> _______________________________________________
> Outages mailing list
> Outages@outages.org <mailto:Outages@outages.org>
> https://puck.nether.net/mailman/listinfo/outages
> <https://puck.nether.net/mailman/listinfo/outages>
>