Mailing List Archive

For a good long while (months) we have had similar issues with various Verizon destinations.
I observed it only happens when passing through certain geographic regions of the US. Other regions make it through without issue.

This is directly observable and repeatable using Cogent's Looking glass website.
Do an IPv4 Trace to 63.59.67.68 using their US-Minneapollis router. It dies.
Do an IPv4 Trace to 63.59.67.68 using their US-Los Angeles router. It reaches the destination.

I went through a handful of Cogent's looking glass locations and found some that work and some that don't and concluded there must be one or more Verizon routers in a certain set geographic area that are having the problems.

Ultimately the issue is not resolved for me but I was able to BGP TE the traffic around the problem areas to facilitate reachability to the impacted destinations. This is obviously a tenuous band-aid.


Long story short: please, please, please, someone at Verizon or someone who has the ear of someone at Verizon, please, please, please, look into this.
<cue holo recording R2D2 plays of Princess Leia asking for help>



-----Original Message-----
From: Richard Laager <rlaager@wiktel.com<mailto:Richard%20Laager%20%3crlaager@wiktel.com%3e>>
To: nanog@nanog.org<mailto:nanog@nanog.org>
Subject: Verizon Business Contact
Date: Thu, 08 Feb 2024 13:01:14 -0600

Can someone from Verizon Business please contact me?

It appears that your network is losing traffic from Verizon Wireless
(e.g. 63.59.39.232, 63.56.37.4, or 63.59.67.68) to me (AS33362, e.g. to
69.89.207.16). Note that 63.59.166.100 -> 69.89.207.16 was successfully
(around 2023-11-27).

This breaks email between us and it's been MONTHS of VZW getting nowhere.

Based on some traceroutes (on 2023-11-27 and again just now), the
working ones go through 140.222.234.223 (0.ae10.GW7.CHI13.ALTER.NET)
while the broken ones stop at 140.222.234.221 (0.ae9.GW7.CHI13.ALTER.NET).

On 2024-02-09 18:10, Justin Krejci wrote:
> For a good long while (months) we have had similar issues with various
> Verizon destinations.

Only Verizon *Wireless* destinations, or other Verizon *Business* things?

As of today, I'm told (via an upstream provider) that Verizon Business
says this is a Verizon Wireless issue.

--
Richard

For me it is some AS 6167 destinations.
WHOIS for that ASN says this is Verizon Business.


AS Number: 6167

Org Name: Verizon Business


I am not sure how I am supposed to accurately or authoritatively discern the differences in specific IP prefixes (or ASNs) as to whether they are are used in the Verizon Wireless, Verizon Business, Verizon XYZ, etc.
I am also not sure what the value would be understanding the difference as I have zero contacts at any Verizon entity: Wireless, Business, or any other.

I imagine at some level, there is a parent Verizon umbrella organization that is ultimately responsible for all underling organizations/divisions but I am not particularly interested in trying to pick apart the business silos of Verizon and then from there trying to chase down specific Verizon entity contacts to try and figure out who, might be the right contact to look into this. I have made efforts, prior to this NANOG thread even starting, to get this issue rectified but I have had zero luck so far getting any appropriate person at Verizon to take notice.

It kind of feels like trying to reach out to some company regarding a geolocation or IP-reputation type issue... just a lot of "Sorry, I don't know. try this other group that you already talked to" or simply "piss off" type responses. Both of which I have received in sizable quantities. Now that my brain is on that tangent, my favorite geolocation response was when I was told "your ISP needs to set the correct bits in the IP packets to designate the traffic as coming from the correct geography." I laughed and I cried at that one.



-----Original Message-----
From: Richard Laager <rlaager@wiktel.com<mailto:Richard%20Laager%20%3crlaager@wiktel.com%3e>>
To: Justin Krejci <JKrejci@usinternet.com<mailto:Justin%20Krejci%20%3cJKrejci@usinternet.com%3e>>
Cc: nanog@nanog.org <nanog@nanog.org<mailto:%22nanog@nanog.org%22%20%3cnanog@nanog.org%3e>>
Subject: Re: Verizon Business Contact
Date: Fri, 16 Feb 2024 20:41:04 -0600

On 2024-02-09 18:10, Justin Krejci wrote:

For a good long while (months) we have had similar issues with various Verizon destinations.

Only Verizon Wireless destinations, or other Verizon Business things?

As of today, I'm told (via an upstream provider) that Verizon Business says this is a Verizon Wireless issue.

Verizon Business is the fixed line business focused entity, formerly MCI and UUNET. Verizon Wireless is the wireless business entity.
Shane

On Feb 19, 2024, at 2:44?PM, Justin Krejci <JKrejci@usinternet.com> wrote:

? For me it is some AS 6167 destinations. WHOIS for that ASN says this is Verizon Business.
AS Number: 6167 Org Name: Verizon Business
I am not sure how I am supposed to accurately or authoritatively discern the differences in specific IP prefixes (or ASNs) as to whether they are are used in the Verizon Wireless, Verizon Business, Verizon XYZ, etc. I am also not sure what the value would be understanding the difference as I have zero contacts at any Verizon entity: Wireless, Business, or any other.
I imagine at some level, there is a parent Verizon umbrella organization that is ultimately responsible for all underling organizations/divisions but I am not particularly interested in trying to pick apart the business silos of Verizon and then from there trying to chase down specific Verizon entity contacts to try and figure out who, might be the right contact to look into this. I have made efforts, prior to this NANOG thread even starting, to get this issue rectified but I have had zero luck so far getting any appropriate person at Verizon to take notice.
It kind of feels like trying to reach out to some company regarding a geolocation or IP-reputation type issue... just a lot of "Sorry, I don't know. try this other group that you already talked to" or simply "piss off" type responses. Both of which I have received in sizable quantities. Now that my brain is on that tangent, my favorite geolocation response was when I was told "your ISP needs to set the correct bits in the IP packets to designate the traffic as coming from the correct geography." I laughed and I cried at that one.


-----Original Message----- From: Richard Laager <rlaager@wiktel.com> To: Justin Krejci <JKrejci@usinternet.com> Cc: nanog@nanog.org <nanog@nanog.org> Subject: Re: Verizon Business Contact Date: Fri, 16 Feb 2024 20:41:04 -0600
On 2024-02-09 18:10, Justin Krejci wrote:

For a good long while (months) we have had similar issues with various Verizon destinations.

Only Verizon Wireless destinations, or other Verizon Business things?


As of today, I'm told (via an upstream provider) that Verizon Business says this is a Verizon Wireless issue.

But then MCI is the one running fiber to all of the Verizon Wireless sites, so that doesn't help in de-muddying the waters.




-----
Mike Hammett
Intelligent Computing Solutions
http://www.ics-il.com

Midwest-IX
http://www.midwest-ix.com

----- Original Message -----

From: sronan@ronan-online.com
To: "Justin Krejci" <JKrejci@usinternet.com>
Cc: nanog@nanog.org
Sent: Monday, February 19, 2024 1:54:43 PM
Subject: Re: Verizon Business Contact



Verizon Business is the fixed line business focused entity, formerly MCI and UUNET. Verizon Wireless is the wireless business entity.


Shane



On Feb 19, 2024, at 2:44 PM, Justin Krejci <JKrejci@usinternet.com> wrote:




<blockquote>


For me it is some AS 6167 destinations.
WHOIS for that ASN says this is Verizon Business.

AS Number: 6167 Org Name: Verizon Business


I am not sure how I am supposed to accurately or authoritatively discern the differences in specific IP prefixes (or ASNs) as to whether they are are used in the Verizon Wireless, Verizon Business, Verizon XYZ, etc.
I am also not sure what the value would be understanding the difference as I have zero contacts at any Verizon entity: Wireless, Business, or any other.


I imagine at some level, there is a parent Verizon umbrella organization that is ultimately responsible for all underling organizations/divisions but I am not particularly interested in trying to pick apart the business silos of Verizon and then from there trying to chase down specific Verizon entity contacts to try and figure out who, might be the right contact to look into this. I have made efforts, prior to this NANOG thread even starting, to get this issue rectified but I have had zero luck so far getting any appropriate person at Verizon to take notice.


It kind of feels like trying to reach out to some company regarding a geolocation or IP-reputation type issue... just a lot of "Sorry, I don't know. try this other group that you already talked to" or simply "piss off" type responses. Both of which I have received in sizable quantities. Now that my brain is on that tangent, my favorite geolocation response was when I was told "your ISP needs to set the correct bits in the IP packets to designate the traffic as coming from the correct geography." I laughed and I cried at that one.






-----Original Message-----
From : Richard Laager < rlaager@wiktel.com >
To : Justin Krejci < JKrejci@usinternet.com >
Cc : nanog@nanog.org < nanog@nanog.org >
Subject : Re: Verizon Business Contact
Date : Fri, 16 Feb 2024 20:41:04 -0600


On 2024-02-09 18:10, Justin Krejci wrote:


<blockquote>

For a good long while (months) we have had similar issues with various Verizon destinations.
</blockquote>

Only Verizon Wireless destinations, or other Verizon Business things?

As of today, I'm told (via an upstream provider) that Verizon Business says this is a Verizon Wireless issue.



</blockquote>

No, Verizon Wireless has their own AS # and doesn’t actually use Verizon Business as their primary provider.
Shane

On Feb 19, 2024, at 2:58?PM, Mike Hammett <nanog@ics-il.net> wrote:

?But then MCI is the one running fiber to all of the Verizon Wireless sites, so that doesn't help in de-muddying the waters.



-----
Mike Hammett
Intelligent Computing Solutions
http://www.ics-il.com

Midwest-IX
http://www.midwest-ix.com

---

From: sronan@ronan-online.com
To: "Justin Krejci" <JKrejci@usinternet.com>
Cc: nanog@nanog.org
Sent: Monday, February 19, 2024 1:54:43 PM
Subject: Re: Verizon Business Contact

Verizon Business is the fixed line business focused entity, formerly MCI and UUNET. Verizon Wireless is the wireless business entity.
Shane

On Feb 19, 2024, at 2:44?PM, Justin Krejci <JKrejci@usinternet.com> wrote:

? For me it is some AS 6167 destinations. WHOIS for that ASN says this is Verizon Business.
AS Number: 6167 Org Name: Verizon Business
I am not sure how I am supposed to accurately or authoritatively discern the differences in specific IP prefixes (or ASNs) as to whether they are are used in the Verizon Wireless, Verizon Business, Verizon XYZ, etc. I am also not sure what the value would be understanding the difference as I have zero contacts at any Verizon entity: Wireless, Business, or any other.
I imagine at some level, there is a parent Verizon umbrella organization that is ultimately responsible for all underling organizations/divisions but I am not particularly interested in trying to pick apart the business silos of Verizon and then from there trying to chase down specific Verizon entity contacts to try and figure out who, might be the right contact to look into this. I have made efforts, prior to this NANOG thread even starting, to get this issue rectified but I have had zero luck so far getting any appropriate person at Verizon to take notice.
It kind of feels like trying to reach out to some company regarding a geolocation or IP-reputation type issue... just a lot of "Sorry, I don't know. try this other group that you already talked to" or simply "piss off" type responses. Both of which I have received in sizable quantities. Now that my brain is on that tangent, my favorite geolocation response was when I was told "your ISP needs to set the correct bits in the IP packets to designate the traffic as coming from the correct geography." I laughed and I cried at that one.


-----Original Message----- From: Richard Laager <rlaager@wiktel.com> To: Justin Krejci <JKrejci@usinternet.com> Cc: nanog@nanog.org <nanog@nanog.org> Subject: Re: Verizon Business Contact Date: Fri, 16 Feb 2024 20:41:04 -0600
On 2024-02-09 18:10, Justin Krejci wrote:

For a good long while (months) we have had similar issues with various Verizon destinations.

Only Verizon Wireless destinations, or other Verizon Business things?


As of today, I'm told (via an upstream provider) that Verizon Business says this is a Verizon Wireless issue.

I see the route originated by two different ASNs. I agree that when I
use the AS6167 path, it is broken (for the destinations where it is
broken; 63.59.166.100 was working despite using the AS6167 path).

BGP routing table entry for 63.59.0.0/16
Paths: 2 available
6939 701 22394
184.105.34.254 from 184.105.34.254 (216.218.253.228)
Origin IGP, metric 0, localpref 60, IGP metric 0, weight 0, tag 0
Received 21d19h ago, valid, external, best
Rx SAFI: Unicast
6461 701 6167
69.89.205.202 from 69.89.205.202 (69.89.205.202)
Origin IGP, metric 887, localpref 60, IGP metric 40, weight 0, tag 0
Received 4d03h ago, valid, internal
Community: 6461:5997
Rx SAFI: Unicast

Based on the names in WHOIS, I would say that both AS6167 and AS22394
are Verizon Wireless.

--
Richard

Based on the ASName of both AS, including CELLCO which is the actual name of the corporate entity known as Verizon Wireless, I would agree that both are in fact Verizon Wireless. The contacts are just corporate standard entities.

Shane

> On Feb 19, 2024, at 9:01?PM, Richard Laager <rlaager@wiktel.com> wrote:
>
> ?I see the route originated by two different ASNs. I agree that when I use the AS6167 path, it is broken (for the destinations where it is broken; 63.59.166.100 was working despite using the AS6167 path).
>
> BGP routing table entry for 63.59.0.0/16
> Paths: 2 available
> 6939 701 22394
> 184.105.34.254 from 184.105.34.254 (216.218.253.228)
> Origin IGP, metric 0, localpref 60, IGP metric 0, weight 0, tag 0
> Received 21d19h ago, valid, external, best
> Rx SAFI: Unicast
> 6461 701 6167
> 69.89.205.202 from 69.89.205.202 (69.89.205.202)
> Origin IGP, metric 887, localpref 60, IGP metric 40, weight 0, tag 0
> Received 4d03h ago, valid, internal
> Community: 6461:5997
> Rx SAFI: Unicast
>
> Based on the names in WHOIS, I would say that both AS6167 and AS22394 are Verizon Wireless.
>
> --
> Richard
>

To close the loop on this, Verizon Wireless reported to me that they
fixed the issue (whatever it was).

They further said that 63.56.37.4 was a typo; all IPs should have been
in 63.59.x.x.

I am able to reach the 63.59.0.0/16 IPs in question: 63.59.39.232 &
63.59.67.68.

Justin: Thanks for the detail that this was reproducible from Cogent's
looking glass. I think there's a good chance that contributed to them
being able to find it (i.e. having an easy way for them to test).

--
Richard