Mailing List Archive: Newark Verizon Issues....

Many of my customers cant reach me from behind FioS. Traceroute shows a
problem spot in Newark in Verizon backbone. 256ms latency at the last hop,
then dies.

It several hops within Verizon, so it cant be an Abovenet peering issue.

If anyone from Verizon is listening out there, please look into this. Fios
support refuses to troubleshoot. I know at least 20 people who have called
in to complain, there are Verizon (DSL or Fios) through Jersey and
Delaware.

1 gw-238-225.quonix.net (208.72.238.225) 0.256 ms 0.268 ms 0.270 ms
2 ge-11-1-2.mpr3.phl2.us.above.net (209.249.122.165) 0.174 ms 0.195
ms 0.
166 ms
3 xe-2-3-0.cr1.lga5.us.above.net (64.125.31.34) 2.422 ms 2.357 ms
2.412 m
s
4 xe-0-1-0.er1.lga5.us.above.net (64.125.27.61) 2.222 ms 2.200 ms
2.228 m
s
5 0.ge-3-2-0.BR3.NYC4.ALTER.NET (204.255.168.25) 2.254 ms 2.297 ms
2.264
ms
6 0.ge-4-2-0.NY5030-BB-RTR2.verizon-gni.net (152.63.10.54) 2.573 ms
2.605 m
s 2.566 ms
7 P15-0-0.NWRKNJ-LCR-04.verizon-gni.net (130.81.29.195) 4.767 ms
4.701 ms
4.714 ms
8 P12-0-0.NWRKNJ-LCR-06.verizon-gni.net (130.81.27.7) 5.717 ms 5.348
ms 5
.262 ms
9 P14-0.NWRKNJ-LCR-08.verizon-gni.net (130.81.30.95) 5.265 ms 5.202
ms 5.
259 ms
10 * * *
11 * * *
12 * * *
13 * * *
14 * * *

On Thu, Sep 23, 2010 at 06:24:03PM +0000, john at quonix.net wrote:
> Many of my customers cant reach me from behind FioS. Traceroute shows a
> problem spot in Newark in Verizon backbone. 256ms latency at the last hop,
> then dies.
>
> It several hops within Verizon, so it cant be an Abovenet peering issue.
>
> If anyone from Verizon is listening out there, please look into this. Fios
> support refuses to troubleshoot. I know at least 20 people who have called
> in to complain, there are Verizon (DSL or Fios) through Jersey and
> Delaware.
>
> 1 gw-238-225.quonix.net (208.72.238.225) 0.256 ms 0.268 ms 0.270 ms
> 2 ge-11-1-2.mpr3.phl2.us.above.net (209.249.122.165) 0.174 ms 0.195
> ms 0.
> 166 ms
> 3 xe-2-3-0.cr1.lga5.us.above.net (64.125.31.34) 2.422 ms 2.357 ms
> 2.412 m
> s
> 4 xe-0-1-0.er1.lga5.us.above.net (64.125.27.61) 2.222 ms 2.200 ms
> 2.228 m
> s
> 5 0.ge-3-2-0.BR3.NYC4.ALTER.NET (204.255.168.25) 2.254 ms 2.297 ms
> 2.264
> ms
> 6 0.ge-4-2-0.NY5030-BB-RTR2.verizon-gni.net (152.63.10.54) 2.573 ms
> 2.605 m
> s 2.566 ms
> 7 P15-0-0.NWRKNJ-LCR-04.verizon-gni.net (130.81.29.195) 4.767 ms
> 4.701 ms
> 4.714 ms
> 8 P12-0-0.NWRKNJ-LCR-06.verizon-gni.net (130.81.27.7) 5.717 ms 5.348
> ms 5
> .262 ms
> 9 P14-0.NWRKNJ-LCR-08.verizon-gni.net (130.81.30.95) 5.265 ms 5.202
> ms 5.
> 259 ms
> 10 * * *
> 11 * * *
> 12 * * *
> 13 * * *
> 14 * * *

This isn't to say there's not a problem, but where is the "256ms latency
at the last hop" evidence? All that's shown here is 3 probes at hop #9
with RTTs of 5.265, 5.202, and 5.259ms. The copy-paste was done poorly
so the output is formatted badly.

You're going to need to provide traceroutes from both directions
(src->dst and dst->src), and should probably try UDP, TCP, and
ICMP-based traceroutes to rule out filtering at the endpoints. Some
traceroute implementations also offer a flag (on BSD it's -e, to be used
alongside -p) which causes UDP and TCP destination packets to *not*
increment their port number per probe, which can help if the destination
uses a firewall (and the user has control over it).

When dealing with Verizon, you should probably also indicate if all
forms of packets are being filtered (meaning provide actual packet
traces on both the source and destination ends showing something like
the results of an telnet x.x.x.x yyyy towards you, where yyyy should be
a port you're definitely listening on + have permitted on your firewall
and/or port forwarded). Basically you want to confirm if the remote end
receives a SYN+ACK response (from you) to their initial SYN.

Good luck.

--
| Jeremy Chadwick jdc at parodius.com |
| Parodius Networking http://www.parodius.com/ |
| UNIX Systems Administrator Mountain View, CA, USA |
| Making life hard for others since 1977. PGP: 4BD6C0CB |