Dear list,
There is major congestion for 8 - 10 hours every day on the GTT
network from Frankfurt, Germany towards Milan, Italy.
Before noon (localtime, that would be 1200~1300Z) things are looking
fine but in the afternoon, packet-loss and latency increase steadily
(peaking in the evening). We are talking packet-loss in excess of 10%
and ten times the expected latency:
lukas@htznr2:~$ mtr 62.10.128.1 -wrs1000 -c1000
Start: 2022-02-03T21:33:28+0100
HOST: htznr2 Loss% Snt Last
Avg Best Wrst StDev
1.|-- _gateway 0.0% 1000 3.1
4.9 2.3 18.0 1.3
2.|-- 10114.your-cloud.host 0.0% 1000 0.2
0.3 0.2 4.0 0.2
3.|-- ??? 100.0 1000 0.0
0.0 0.0 0.0 0.0
4.|-- 85-10-250-153.clients.your-server.de 0.0% 1000 1.3
4.8 0.8 98.6 11.8
5.|-- spine12.cloud1.nbg1.hetzner.com 0.0% 1000 1.1
4.8 0.8 66.8 11.4
6.|-- 213-239-239-121.clients.your-server.de 0.0% 1000 3.0
2.3 0.5 43.4 4.1
7.|-- core1.fra.hetzner.com 0.0% 1000 3.6
5.7 3.5 65.2 5.0
8.|-- ae18.cr6-fra2.ip4.gtt.net 0.0% 1000 4.8
5.8 3.6 116.7 6.7
9.|-- et-5-1-0.cr0-mil2.ip4.gtt.net 11.1% 1000 114.0
112.3 63.5 172.7 13.4
10.|-- tiscali.ip4.gtt.net 11.5% 1000 111.5
113.4 64.3 148.8 13.2
11.|-- static-94-32-126-81.clienti.tiscali.it 10.6% 1000 121.7
126.4 79.8 162.9 13.7
12.|-- static-94-32-135-162.clienti.tiscali.it 11.2% 1000 119.1
126.8 80.9 176.8 13.9
13.|-- static-94-32-135-182.clienti.tiscali.it 11.3% 1000 119.0
127.6 79.5 171.2 14.2
14.|-- static-213-205-54-210.clienti.tiscali.it 78.7% 1000 130.1
125.3 73.8 228.1 16.2
15.|-- ??? 100.0 1000 0.0
0.0 0.0 0.0 0.0
lukas@htznr2:~$
2022-02-01T18:10:07Z
https://pastebin.com/raw/MSvuSuR0
2022-02-02T17:39:44Z
https://pastebin.com/raw/zWD8GHGa
2022-02-03T20:33:28Z (as above)
https://pastebin.com/raw/DHWDyi1g
2022-02-04T13:11:51Z
https://pastebin.com/raw/qhrWZBQX
GTT NOC has been advised Feb 1st, 18:000Z, but apparently they always
verify network issues in the morning, which is also when those links
are uncongested, so at this point there is no confirmation from them.
I'm assuming the network teams are aware of it anyway, but we don't
know for sure. If you are using GTT for transit in Europe and traffic
to Italy matters to your customers, you probably want to reroute (for
example: reducing loc-pref based on GTT's BGP community 3257:53900 for
prefixes learned in Italy).
cheers,
lukas
_______________________________________________
Outages mailing list
Outages@outages.org
https://puck.nether.net/mailman/listinfo/outages
There is major congestion for 8 - 10 hours every day on the GTT
network from Frankfurt, Germany towards Milan, Italy.
Before noon (localtime, that would be 1200~1300Z) things are looking
fine but in the afternoon, packet-loss and latency increase steadily
(peaking in the evening). We are talking packet-loss in excess of 10%
and ten times the expected latency:
lukas@htznr2:~$ mtr 62.10.128.1 -wrs1000 -c1000
Start: 2022-02-03T21:33:28+0100
HOST: htznr2 Loss% Snt Last
Avg Best Wrst StDev
1.|-- _gateway 0.0% 1000 3.1
4.9 2.3 18.0 1.3
2.|-- 10114.your-cloud.host 0.0% 1000 0.2
0.3 0.2 4.0 0.2
3.|-- ??? 100.0 1000 0.0
0.0 0.0 0.0 0.0
4.|-- 85-10-250-153.clients.your-server.de 0.0% 1000 1.3
4.8 0.8 98.6 11.8
5.|-- spine12.cloud1.nbg1.hetzner.com 0.0% 1000 1.1
4.8 0.8 66.8 11.4
6.|-- 213-239-239-121.clients.your-server.de 0.0% 1000 3.0
2.3 0.5 43.4 4.1
7.|-- core1.fra.hetzner.com 0.0% 1000 3.6
5.7 3.5 65.2 5.0
8.|-- ae18.cr6-fra2.ip4.gtt.net 0.0% 1000 4.8
5.8 3.6 116.7 6.7
9.|-- et-5-1-0.cr0-mil2.ip4.gtt.net 11.1% 1000 114.0
112.3 63.5 172.7 13.4
10.|-- tiscali.ip4.gtt.net 11.5% 1000 111.5
113.4 64.3 148.8 13.2
11.|-- static-94-32-126-81.clienti.tiscali.it 10.6% 1000 121.7
126.4 79.8 162.9 13.7
12.|-- static-94-32-135-162.clienti.tiscali.it 11.2% 1000 119.1
126.8 80.9 176.8 13.9
13.|-- static-94-32-135-182.clienti.tiscali.it 11.3% 1000 119.0
127.6 79.5 171.2 14.2
14.|-- static-213-205-54-210.clienti.tiscali.it 78.7% 1000 130.1
125.3 73.8 228.1 16.2
15.|-- ??? 100.0 1000 0.0
0.0 0.0 0.0 0.0
lukas@htznr2:~$
2022-02-01T18:10:07Z
https://pastebin.com/raw/MSvuSuR0
2022-02-02T17:39:44Z
https://pastebin.com/raw/zWD8GHGa
2022-02-03T20:33:28Z (as above)
https://pastebin.com/raw/DHWDyi1g
2022-02-04T13:11:51Z
https://pastebin.com/raw/qhrWZBQX
GTT NOC has been advised Feb 1st, 18:000Z, but apparently they always
verify network issues in the morning, which is also when those links
are uncongested, so at this point there is no confirmation from them.
I'm assuming the network teams are aware of it anyway, but we don't
know for sure. If you are using GTT for transit in Europe and traffic
to Italy matters to your customers, you probably want to reroute (for
example: reducing loc-pref based on GTT's BGP community 3257:53900 for
prefixes learned in Italy).
cheers,
lukas
_______________________________________________
Outages mailing list
Outages@outages.org
https://puck.nether.net/mailman/listinfo/outages