Mailing List Archive

VPN issues over Spectrum L3 boundaries
We have several offices over the Ohio and Pennsylvania area that are experiencing issues passing traffic over VPN tunnels (specifically, there is always a Spectrum >< Level 3 interconnect). It is a very strange issue. The VPN tunnel will actually establish, and if you source your ping from inside the internal network across the VPN tunnel to the destination, the traffic gets there and replies, but the replies never make it back to the original sending point.

Anyone else experiencing any similar issues like this?

Best Regards,
Josh

This email and its attachments may contain privileged and confidential information and/or protected health information (PHI) intended solely for the use of Netsmart Technologies and the recipient(s) named above. If you are not the recipient, or the employee or agent responsible for delivering this message to the intended recipient, you are hereby notified that any review, dissemination, distribution, printing or copying of this email message and/or any attachments is strictly prohibited. If you have received this transmission in error, please email compliance@NTST.com immediately and permanently delete this email and any attachments.
VPN issues over Spectrum L3 boundaries [ In reply to ]
Found a thread in the Spectrum forums talking about the issue finally - it was marked as resolved so I started a new one.

https://forums.timewarnercable.com/t5/Connectivity/Traffic-issues-at-66-109-7-162/m-p/164091#M53497

From: Outages <outages-bounces@outages.org> On Behalf Of Biddle, Josh via Outages
Sent: Sunday, October 13, 2019 12:00 PM
To: outages@outages.org
Subject: [outages] VPN issues over Spectrum L3 boundaries

We have several offices over the Ohio and Pennsylvania area that are experiencing issues passing traffic over VPN tunnels (specifically, there is always a Spectrum >< Level 3 interconnect). It is a very strange issue. The VPN tunnel will actually establish, and if you source your ping from inside the internal network across the VPN tunnel to the destination, the traffic gets there and replies, but the replies never make it back to the original sending point.

Anyone else experiencing any similar issues like this?

Best Regards,
Josh

This email and its attachments may contain privileged and confidential information and/or protected health information (PHI) intended solely for the use of Netsmart Technologies and the recipient(s) named above. If you are not the recipient, or the employee or agent responsible for delivering this message to the intended recipient, you are hereby notified that any review, dissemination, distribution, printing or copying of this email message and/or any attachments is strictly prohibited. If you have received this transmission in error, please email compliance@NTST.com<mailto:compliance@NTST.com> immediately and permanently delete this email and any attachments.
This email and its attachments may contain privileged and confidential information and/or protected health information (PHI) intended solely for the use of Netsmart Technologies and the recipient(s) named above. If you are not the recipient, or the employee or agent responsible for delivering this message to the intended recipient, you are hereby notified that any review, dissemination, distribution, printing or copying of this email message and/or any attachments is strictly prohibited. If you have received this transmission in error, please email compliance@NTST.com immediately and permanently delete this email and any attachments.
Re: VPN issues over Spectrum L3 boundaries [ In reply to ]
We had a similar issue last week that we chalked up to a Spectrum outage.

Because this was all new install we have not gone back and tested again yet but very similar to you - Multiple sites over Ohio. VPN would establish and one side would send traffic and it would be received on the other end. The other side would send traffic and it would not be received.

Justin

> On Oct 15, 2019, at 8:04 AM, Biddle, Josh via Outages <outages@outages.org> wrote:
>
> Found a thread in the Spectrum forums talking about the issue finally – it was marked as resolved so I started a new one.
>
> https://forums.timewarnercable.com/t5/Connectivity/Traffic-issues-at-66-109-7-162/m-p/164091#M53497 <https://forums.timewarnercable.com/t5/Connectivity/Traffic-issues-at-66-109-7-162/m-p/164091#M53497>
>
> From: Outages <outages-bounces@outages.org <mailto:outages-bounces@outages.org>> On Behalf Of Biddle, Josh via Outages
> Sent: Sunday, October 13, 2019 12:00 PM
> To: outages@outages.org <mailto:outages@outages.org>
> Subject: [outages] VPN issues over Spectrum L3 boundaries
>
> We have several offices over the Ohio and Pennsylvania area that are experiencing issues passing traffic over VPN tunnels (specifically, there is always a Spectrum >< Level 3 interconnect). It is a very strange issue. The VPN tunnel will actually establish, and if you source your ping from inside the internal network across the VPN tunnel to the destination, the traffic gets there and replies, but the replies never make it back to the original sending point.
>
> Anyone else experiencing any similar issues like this?
>
> Best Regards,
> Josh
>
> This email and its attachments may contain privileged and confidential information and/or protected health information (PHI) intended solely for the use of Netsmart Technologies and the recipient(s) named above. If you are not the recipient, or the employee or agent responsible for delivering this message to the intended recipient, you are hereby notified that any review, dissemination, distribution, printing or copying of this email message and/or any attachments is strictly prohibited. If you have received this transmission in error, please email compliance@NTST.com <mailto:compliance@NTST.com> immediately and permanently delete this email and any attachments.
> This email and its attachments may contain privileged and confidential information and/or protected health information (PHI) intended solely for the use of Netsmart Technologies and the recipient(s) named above. If you are not the recipient, or the employee or agent responsible for delivering this message to the intended recipient, you are hereby notified that any review, dissemination, distribution, printing or copying of this email message and/or any attachments is strictly prohibited. If you have received this transmission in error, please email compliance@NTST.com <mailto:compliance@NTST.com> immediately and permanently delete this email and any attachments. _______________________________________________
> Outages mailing list
> Outages@outages.org <mailto:Outages@outages.org>
> https://puck.nether.net/mailman/listinfo/outages <https://puck.nether.net/mailman/listinfo/outages>
Re: VPN issues over Spectrum L3 boundaries [ In reply to ]
See if you can get proof with traceroutes and post on the thread that I made to see if we can get some type of answer out of someone. My best guess at this time is that it is some business squabble at the transit provider where these two Internet providers interconnect.


From: Justin Oeder <justin.oeder@beyondhosting.net>
Sent: Tuesday, October 15, 2019 8:51 AM
To: Biddle, Josh <JBiddle@ntst.com>
Cc: outages@outages.org
Subject: Re: [outages] VPN issues over Spectrum L3 boundaries

We had a similar issue last week that we chalked up to a Spectrum outage.

Because this was all new install we have not gone back and tested again yet but very similar to you - Multiple sites over Ohio. VPN would establish and one side would send traffic and it would be received on the other end. The other side would send traffic and it would not be received.

Justin


On Oct 15, 2019, at 8:04 AM, Biddle, Josh via Outages <outages@outages.org<mailto:outages@outages.org>> wrote:

Found a thread in the Spectrum forums talking about the issue finally – it was marked as resolved so I started a new one.

https://forums.timewarnercable.com/t5/Connectivity/Traffic-issues-at-66-109-7-162/m-p/164091#M53497<https://urldefense.proofpoint.com/v2/url?u=https-3A__forums.timewarnercable.com_t5_Connectivity_Traffic-2Dissues-2Dat-2D66-2D109-2D7-2D162_m-2Dp_164091-23M53497&d=DwMFaQ&c=-7HNwxqfpkdcRXCW8HB54Q&r=svX1Si7sopSBMitBL3bFwQ&m=iQLepJM5vHmCTqpsc5_QOKPNiy5wXujKdcHvm7JWg0k&s=RUIXDoc0T_bmNFkrtsGtYl5C5cVvjOWGsqTFkuO39dE&e=>

From: Outages <outages-bounces@outages.org<mailto:outages-bounces@outages.org>> On Behalf Of Biddle, Josh via Outages
Sent: Sunday, October 13, 2019 12:00 PM
To: outages@outages.org<mailto:outages@outages.org>
Subject: [outages] VPN issues over Spectrum L3 boundaries

We have several offices over the Ohio and Pennsylvania area that are experiencing issues passing traffic over VPN tunnels (specifically, there is always a Spectrum >< Level 3 interconnect). It is a very strange issue. The VPN tunnel will actually establish, and if you source your ping from inside the internal network across the VPN tunnel to the destination, the traffic gets there and replies, but the replies never make it back to the original sending point.

Anyone else experiencing any similar issues like this?

Best Regards,
Josh

This email and its attachments may contain privileged and confidential information and/or protected health information (PHI) intended solely for the use of Netsmart Technologies and the recipient(s) named above. If you are not the recipient, or the employee or agent responsible for delivering this message to the intended recipient, you are hereby notified that any review, dissemination, distribution, printing or copying of this email message and/or any attachments is strictly prohibited. If you have received this transmission in error, please email compliance@NTST.com<mailto:compliance@NTST.com> immediately and permanently delete this email and any attachments.
This email and its attachments may contain privileged and confidential information and/or protected health information (PHI) intended solely for the use of Netsmart Technologies and the recipient(s) named above. If you are not the recipient, or the employee or agent responsible for delivering this message to the intended recipient, you are hereby notified that any review, dissemination, distribution, printing or copying of this email message and/or any attachments is strictly prohibited. If you have received this transmission in error, please email compliance@NTST.com<mailto:compliance@NTST.com> immediately and permanently delete this email and any attachments. _______________________________________________
Outages mailing list
Outages@outages.org<mailto:Outages@outages.org>
https://puck.nether.net/mailman/listinfo/outages<https://urldefense.proofpoint.com/v2/url?u=https-3A__puck.nether.net_mailman_listinfo_outages&d=DwMFaQ&c=-7HNwxqfpkdcRXCW8HB54Q&r=svX1Si7sopSBMitBL3bFwQ&m=iQLepJM5vHmCTqpsc5_QOKPNiy5wXujKdcHvm7JWg0k&s=2be7j5W6TaOMjgT-Wm0C5ThoYvwYYbN7BHQUklMyQdc&e=>

This email and its attachments may contain privileged and confidential information and/or protected health information (PHI) intended solely for the use of Netsmart Technologies and the recipient(s) named above. If you are not the recipient, or the employee or agent responsible for delivering this message to the intended recipient, you are hereby notified that any review, dissemination, distribution, printing or copying of this email message and/or any attachments is strictly prohibited. If you have received this transmission in error, please email compliance@NTST.com immediately and permanently delete this email and any attachments.
Re: VPN issues over Spectrum L3 boundaries [ In reply to ]
Sounds more like a technical issue such as a tunnel with lower MTU. So your
signalling works but the tunnel and data doest get established. I would
very much doubt a commercial problem caused someone to randomly implement
filters. You ought to be able to test it tho by seeing if the relevant
ports are open in either direction and if ping/no fragment works at the
maximum tunnel MTU.

Hth


On Wed, 16 Oct 2019, 00:33 Biddle, Josh via Outages, <outages@outages.org>
wrote:

> See if you can get proof with traceroutes and post on the thread that I
> made to see if we can get some type of answer out of someone. My best guess
> at this time is that it is some business squabble at the transit provider
> where these two Internet providers interconnect.
>
>
>
>
>
> *From:* Justin Oeder <justin.oeder@beyondhosting.net>
> *Sent:* Tuesday, October 15, 2019 8:51 AM
> *To:* Biddle, Josh <JBiddle@ntst.com>
> *Cc:* outages@outages.org
> *Subject:* Re: [outages] VPN issues over Spectrum L3 boundaries
>
>
>
> We had a similar issue last week that we chalked up to a Spectrum outage.
>
>
>
> Because this was all new install we have not gone back and tested again
> yet but very similar to you - Multiple sites over Ohio. VPN would
> establish and one side would send traffic and it would be received on the
> other end. The other side would send traffic and it would not be received.
>
>
>
> Justin
>
>
>
> On Oct 15, 2019, at 8:04 AM, Biddle, Josh via Outages <outages@outages.org>
> wrote:
>
>
>
> Found a thread in the Spectrum forums talking about the issue finally – it
> was marked as resolved so I started a new one.
>
>
>
>
> https://forums.timewarnercable.com/t5/Connectivity/Traffic-issues-at-66-109-7-162/m-p/164091#M53497
> <https://urldefense.proofpoint.com/v2/url?u=https-3A__forums.timewarnercable.com_t5_Connectivity_Traffic-2Dissues-2Dat-2D66-2D109-2D7-2D162_m-2Dp_164091-23M53497&d=DwMFaQ&c=-7HNwxqfpkdcRXCW8HB54Q&r=svX1Si7sopSBMitBL3bFwQ&m=iQLepJM5vHmCTqpsc5_QOKPNiy5wXujKdcHvm7JWg0k&s=RUIXDoc0T_bmNFkrtsGtYl5C5cVvjOWGsqTFkuO39dE&e=>
>
>
>
> *From:* Outages <outages-bounces@outages.org> *On Behalf Of *Biddle, Josh
> via Outages
> *Sent:* Sunday, October 13, 2019 12:00 PM
> *To:* outages@outages.org
> *Subject:* [outages] VPN issues over Spectrum L3 boundaries
>
>
>
> We have several offices over the Ohio and Pennsylvania area that are
> experiencing issues passing traffic over VPN tunnels (specifically, there
> is always a Spectrum >< Level 3 interconnect). It is a very strange issue.
> The VPN tunnel will actually establish, and if you source your ping from
> inside the internal network across the VPN tunnel to the destination, the
> traffic gets there and replies, but the replies never make it back to the
> original sending point.
>
>
>
> Anyone else experiencing any similar issues like this?
>
>
>
> Best Regards,
>
> *Josh*
>
>
>
> This email and its attachments may contain privileged and confidential
> information and/or protected health information (PHI) intended solely for
> the use of Netsmart Technologies and the recipient(s) named above. If you
> are not the recipient, or the employee or agent responsible for delivering
> this message to the intended recipient, you are hereby notified that any
> review, dissemination, distribution, printing or copying of this email
> message and/or any attachments is strictly prohibited. If you have received
> this transmission in error, please email compliance@NTST.com immediately
> and permanently delete this email and any attachments.
>
> This email and its attachments may contain privileged and confidential
> information and/or protected health information (PHI) intended solely for
> the use of Netsmart Technologies and the recipient(s) named above. If you
> are not the recipient, or the employee or agent responsible for delivering
> this message to the intended recipient, you are hereby notified that any
> review, dissemination, distribution, printing or copying of this email
> message and/or any attachments is strictly prohibited. If you have received
> this transmission in error, please email compliance@NTST.com immediately
> and permanently delete this email and any attachments.
> _______________________________________________
> Outages mailing list
> Outages@outages.org
> https://puck.nether.net/mailman/listinfo/outages
> <https://urldefense.proofpoint.com/v2/url?u=https-3A__puck.nether.net_mailman_listinfo_outages&d=DwMFaQ&c=-7HNwxqfpkdcRXCW8HB54Q&r=svX1Si7sopSBMitBL3bFwQ&m=iQLepJM5vHmCTqpsc5_QOKPNiy5wXujKdcHvm7JWg0k&s=2be7j5W6TaOMjgT-Wm0C5ThoYvwYYbN7BHQUklMyQdc&e=>
>
>
> This email and its attachments may contain privileged and confidential
> information and/or protected health information (PHI) intended solely for
> the use of Netsmart Technologies and the recipient(s) named above. If you
> are not the recipient, or the employee or agent responsible for delivering
> this message to the intended recipient, you are hereby notified that any
> review, dissemination, distribution, printing or copying of this email
> message and/or any attachments is strictly prohibited. If you have received
> this transmission in error, please email compliance@NTST.com immediately
> and permanently delete this email and any attachments.
> _______________________________________________
> Outages mailing list
> Outages@outages.org
> https://puck.nether.net/mailman/listinfo/outages
>
Re: VPN issues over Spectrum L3 boundaries [ In reply to ]
The original issue began 2-3 months ago at one client office. This past week it has spread to two clients over at least 6 different offices. Any other thoughts?

From: Stephen Wilcox <steve.wilcox@ixreach.com>
Sent: Tuesday, October 15, 2019 4:38 PM
To: Biddle, Josh <JBiddle@ntst.com>
Cc: Justin Oeder <justin.oeder@beyondhosting.net>; outages@outages.org
Subject: Re: [outages] VPN issues over Spectrum L3 boundaries

Sounds more like a technical issue such as a tunnel with lower MTU. So your signalling works but the tunnel and data doest get established. I would very much doubt a commercial problem caused someone to randomly implement filters. You ought to be able to test it tho by seeing if the relevant ports are open in either direction and if ping/no fragment works at the maximum tunnel MTU.

Hth

On Wed, 16 Oct 2019, 00:33 Biddle, Josh via Outages, <outages@outages.org<mailto:outages@outages.org>> wrote:
See if you can get proof with traceroutes and post on the thread that I made to see if we can get some type of answer out of someone. My best guess at this time is that it is some business squabble at the transit provider where these two Internet providers interconnect.


From: Justin Oeder <justin.oeder@beyondhosting.net<mailto:justin.oeder@beyondhosting.net>>
Sent: Tuesday, October 15, 2019 8:51 AM
To: Biddle, Josh <JBiddle@ntst.com<mailto:JBiddle@ntst.com>>
Cc: outages@outages.org<mailto:outages@outages.org>
Subject: Re: [outages] VPN issues over Spectrum L3 boundaries

We had a similar issue last week that we chalked up to a Spectrum outage.

Because this was all new install we have not gone back and tested again yet but very similar to you - Multiple sites over Ohio. VPN would establish and one side would send traffic and it would be received on the other end. The other side would send traffic and it would not be received.

Justin

On Oct 15, 2019, at 8:04 AM, Biddle, Josh via Outages <outages@outages.org<mailto:outages@outages.org>> wrote:

Found a thread in the Spectrum forums talking about the issue finally – it was marked as resolved so I started a new one.

https://forums.timewarnercable.com/t5/Connectivity/Traffic-issues-at-66-109-7-162/m-p/164091#M53497<https://urldefense.proofpoint.com/v2/url?u=https-3A__forums.timewarnercable.com_t5_Connectivity_Traffic-2Dissues-2Dat-2D66-2D109-2D7-2D162_m-2Dp_164091-23M53497&d=DwMFaQ&c=-7HNwxqfpkdcRXCW8HB54Q&r=svX1Si7sopSBMitBL3bFwQ&m=iQLepJM5vHmCTqpsc5_QOKPNiy5wXujKdcHvm7JWg0k&s=RUIXDoc0T_bmNFkrtsGtYl5C5cVvjOWGsqTFkuO39dE&e=>

From: Outages <outages-bounces@outages.org<mailto:outages-bounces@outages.org>> On Behalf Of Biddle, Josh via Outages
Sent: Sunday, October 13, 2019 12:00 PM
To: outages@outages.org<mailto:outages@outages.org>
Subject: [outages] VPN issues over Spectrum L3 boundaries

We have several offices over the Ohio and Pennsylvania area that are experiencing issues passing traffic over VPN tunnels (specifically, there is always a Spectrum >< Level 3 interconnect). It is a very strange issue. The VPN tunnel will actually establish, and if you source your ping from inside the internal network across the VPN tunnel to the destination, the traffic gets there and replies, but the replies never make it back to the original sending point.

Anyone else experiencing any similar issues like this?

Best Regards,
Josh

This email and its attachments may contain privileged and confidential information and/or protected health information (PHI) intended solely for the use of Netsmart Technologies and the recipient(s) named above. If you are not the recipient, or the employee or agent responsible for delivering this message to the intended recipient, you are hereby notified that any review, dissemination, distribution, printing or copying of this email message and/or any attachments is strictly prohibited. If you have received this transmission in error, please email compliance@NTST.com<mailto:compliance@NTST.com> immediately and permanently delete this email and any attachments.
This email and its attachments may contain privileged and confidential information and/or protected health information (PHI) intended solely for the use of Netsmart Technologies and the recipient(s) named above. If you are not the recipient, or the employee or agent responsible for delivering this message to the intended recipient, you are hereby notified that any review, dissemination, distribution, printing or copying of this email message and/or any attachments is strictly prohibited. If you have received this transmission in error, please email compliance@NTST.com<mailto:compliance@NTST.com> immediately and permanently delete this email and any attachments. _______________________________________________
Outages mailing list
Outages@outages.org<mailto:Outages@outages.org>
https://puck.nether.net/mailman/listinfo/outages<https://urldefense.proofpoint.com/v2/url?u=https-3A__puck.nether.net_mailman_listinfo_outages&d=DwMFaQ&c=-7HNwxqfpkdcRXCW8HB54Q&r=svX1Si7sopSBMitBL3bFwQ&m=iQLepJM5vHmCTqpsc5_QOKPNiy5wXujKdcHvm7JWg0k&s=2be7j5W6TaOMjgT-Wm0C5ThoYvwYYbN7BHQUklMyQdc&e=>

This email and its attachments may contain privileged and confidential information and/or protected health information (PHI) intended solely for the use of Netsmart Technologies and the recipient(s) named above. If you are not the recipient, or the employee or agent responsible for delivering this message to the intended recipient, you are hereby notified that any review, dissemination, distribution, printing or copying of this email message and/or any attachments is strictly prohibited. If you have received this transmission in error, please email compliance@NTST.com<mailto:compliance@NTST.com> immediately and permanently delete this email and any attachments.
_______________________________________________
Outages mailing list
Outages@outages.org<mailto:Outages@outages.org>
https://puck.nether.net/mailman/listinfo/outages<https://urldefense.proofpoint.com/v2/url?u=https-3A__puck.nether.net_mailman_listinfo_outages&d=DwMFaQ&c=-7HNwxqfpkdcRXCW8HB54Q&r=svX1Si7sopSBMitBL3bFwQ&m=p9pDGllAuH7XysxTdZgskh5NId7UuP2Nsa9hbEZh-rQ&s=6BXmu4K5MpMz8Qk1HOWepbDeauFt-Fq3YUQa4qOYHN0&e=>
This email and its attachments may contain privileged and confidential information and/or protected health information (PHI) intended solely for the use of Netsmart Technologies and the recipient(s) named above. If you are not the recipient, or the employee or agent responsible for delivering this message to the intended recipient, you are hereby notified that any review, dissemination, distribution, printing or copying of this email message and/or any attachments is strictly prohibited. If you have received this transmission in error, please email compliance@NTST.com immediately and permanently delete this email and any attachments.
Re: VPN issues over Spectrum L3 boundaries [ In reply to ]
A customer of mine has had the same issue with a TW-connected site in
Ohio and another in PA.? One VPN tunnel works fine, the other has
one-way ISAKMP traffic to the other head-end, which connects to Level3.?
A traceroute shows the failing path includes 66.109.7.162.? The failing
direction is from the PA/OH sites toward the L3 head end.? Full-size
pings work fine.? It's the UDP/500 that vanishes.? I.E., it has nothing
to do with MTU.

Both started having the issue around Aug 26.? Mysteriously, every week
to 10 days, the broken path will start working for a while. This is
usually shortly after midnight EDT; they go back down 1-3 hours later
and stay down.? The log entries for the two sites match within seconds.

The customer's contract is with Comcast Business so it's been difficult
to get to someone clueful about this symptom in TW.

-Marty

On 10/15/2019 8:04 AM, Biddle, Josh via Outages wrote:
>
> Found a thread in the Spectrum forums talking about the issue finally
> ? it was marked as resolved so I started a new one.
>
> https://forums.timewarnercable.com/t5/Connectivity/Traffic-issues-at-66-109-7-162/m-p/164091#M53497
>
> *From:* Outages <outages-bounces@outages.org> *On Behalf Of *Biddle,
> Josh via Outages
> *Sent:* Sunday, October 13, 2019 12:00 PM
> *To:* outages@outages.org
> *Subject:* [outages] VPN issues over Spectrum L3 boundaries
>
> We have several offices over the Ohio and Pennsylvania area that are
> experiencing issues passing traffic over VPN tunnels (specifically,
> there is always a Spectrum >< Level 3 interconnect). It is a very
> strange issue. The VPN tunnel will actually establish, and if you
> source your ping from inside the internal network across the VPN
> tunnel to the destination, the traffic gets there and replies, but the
> replies never make it back to the original sending point.
>
> Anyone else experiencing any similar issues like this?
>
> Best Regards,
>
> *Josh*
>
>
Re: VPN issues over Spectrum L3 boundaries [ In reply to ]
Our issues have recently magically resolved (last Thursday 10/17) due to L3 vanishing from the hop list. Our traffic now disappears into ntt.net and we are seeing two way IPsec traffic without any issues.

Marty, did your issues resolve?
[cid:image001.png@01D58815.06155610]

From: Marty Adkins <marty@martyadkins.com>
Sent: Friday, October 18, 2019 4:01 PM
To: Biddle, Josh <JBiddle@ntst.com>; outages@outages.org
Subject: Re: [outages] VPN issues over Spectrum L3 boundaries

A customer of mine has had the same issue with a TW-connected site in Ohio and another in PA. One VPN tunnel works fine, the other has one-way ISAKMP traffic to the other head-end, which connects to Level3. A traceroute shows the failing path includes 66.109.7.162. The failing direction is from the PA/OH sites toward the L3 head end. Full-size pings work fine. It's the UDP/500 that vanishes. I.E., it has nothing to do with MTU.

Both started having the issue around Aug 26. Mysteriously, every week to 10 days, the broken path will start working for a while. This is usually shortly after midnight EDT; they go back down 1-3 hours later and stay down. The log entries for the two sites match within seconds.

The customer's contract is with Comcast Business so it's been difficult to get to someone clueful about this symptom in TW.

-Marty

On 10/15/2019 8:04 AM, Biddle, Josh via Outages wrote:
Found a thread in the Spectrum forums talking about the issue finally - it was marked as resolved so I started a new one.

https://forums.timewarnercable.com/t5/Connectivity/Traffic-issues-at-66-109-7-162/m-p/164091#M53497<https://urldefense.proofpoint.com/v2/url?u=https-3A__forums.timewarnercable.com_t5_Connectivity_Traffic-2Dissues-2Dat-2D66-2D109-2D7-2D162_m-2Dp_164091-23M53497&d=DwMD-g&c=-7HNwxqfpkdcRXCW8HB54Q&r=svX1Si7sopSBMitBL3bFwQ&m=pXHvd5iI_J5DVYGMDKBSdUZl1iyilZUvR3oyvE3BJ0E&s=_90gCNY2Ln1XdqWtMNguWIRejhsdUxoLyvgTpo-R5jM&e=>

From: Outages <outages-bounces@outages.org><mailto:outages-bounces@outages.org> On Behalf Of Biddle, Josh via Outages
Sent: Sunday, October 13, 2019 12:00 PM
To: outages@outages.org<mailto:outages@outages.org>
Subject: [outages] VPN issues over Spectrum L3 boundaries

We have several offices over the Ohio and Pennsylvania area that are experiencing issues passing traffic over VPN tunnels (specifically, there is always a Spectrum >< Level 3 interconnect). It is a very strange issue. The VPN tunnel will actually establish, and if you source your ping from inside the internal network across the VPN tunnel to the destination, the traffic gets there and replies, but the replies never make it back to the original sending point.

Anyone else experiencing any similar issues like this?

Best Regards,
Josh


This email and its attachments may contain privileged and confidential information and/or protected health information (PHI) intended solely for the use of Netsmart Technologies and the recipient(s) named above. If you are not the recipient, or the employee or agent responsible for delivering this message to the intended recipient, you are hereby notified that any review, dissemination, distribution, printing or copying of this email message and/or any attachments is strictly prohibited. If you have received this transmission in error, please email compliance@NTST.com immediately and permanently delete this email and any attachments.
Re: VPN issues over Spectrum L3 boundaries [ In reply to ]
I had a recent issue that was similar to this. In that case it was a DDOS signature update on a specific vendors' DDOS scrubber at the host site that was the problem. The specific tunnel src/dst flow would be dropped, however all other traffic between endpoints was allowed. Because all the traffic for that specific flow hit a threshold above the rule and the IP's in question were flagged low enough in reputation to be fully inspected, it was blocked. After whitelisting the IP's in the DDOS solution, all VPN traffic worked fine.

Just a thought.
________________________________
From: Outages <outages-bounces@outages.org> on behalf of Biddle, Josh via Outages <outages@outages.org>
Sent: Monday, October 21, 2019 12:39 PM
To: Marty Adkins <marty@martyadkins.com>; outages@outages.org <outages@outages.org>
Cc: D L <route2null0@yahoo.com>; Cullis, Ben <BCullis@ntst.com>; Cochran, Brian <BCochran@ntst.com>
Subject: Re: [outages] VPN issues over Spectrum L3 boundaries


Our issues have recently magically resolved (last Thursday 10/17) due to L3 vanishing from the hop list. Our traffic now disappears into ntt.net and we are seeing two way IPsec traffic without any issues.



Marty, did your issues resolve?

[cid:image001.png@01D58815.06155610]



From: Marty Adkins <marty@martyadkins.com>
Sent: Friday, October 18, 2019 4:01 PM
To: Biddle, Josh <JBiddle@ntst.com>; outages@outages.org
Subject: Re: [outages] VPN issues over Spectrum L3 boundaries



A customer of mine has had the same issue with a TW-connected site in Ohio and another in PA. One VPN tunnel works fine, the other has one-way ISAKMP traffic to the other head-end, which connects to Level3. A traceroute shows the failing path includes 66.109.7.162. The failing direction is from the PA/OH sites toward the L3 head end. Full-size pings work fine. It's the UDP/500 that vanishes. I.E., it has nothing to do with MTU.

Both started having the issue around Aug 26. Mysteriously, every week to 10 days, the broken path will start working for a while. This is usually shortly after midnight EDT; they go back down 1-3 hours later and stay down. The log entries for the two sites match within seconds.

The customer's contract is with Comcast Business so it's been difficult to get to someone clueful about this symptom in TW.

-Marty

On 10/15/2019 8:04 AM, Biddle, Josh via Outages wrote:

Found a thread in the Spectrum forums talking about the issue finally ? it was marked as resolved so I started a new one.



https://forums.timewarnercable.com/t5/Connectivity/Traffic-issues-at-66-109-7-162/m-p/164091#M53497<https://urldefense.proofpoint.com/v2/url?u=https-3A__forums.timewarnercable.com_t5_Connectivity_Traffic-2Dissues-2Dat-2D66-2D109-2D7-2D162_m-2Dp_164091-23M53497&d=DwMD-g&c=-7HNwxqfpkdcRXCW8HB54Q&r=svX1Si7sopSBMitBL3bFwQ&m=pXHvd5iI_J5DVYGMDKBSdUZl1iyilZUvR3oyvE3BJ0E&s=_90gCNY2Ln1XdqWtMNguWIRejhsdUxoLyvgTpo-R5jM&e=>



From: Outages <outages-bounces@outages.org><mailto:outages-bounces@outages.org> On Behalf Of Biddle, Josh via Outages
Sent: Sunday, October 13, 2019 12:00 PM
To: outages@outages.org<mailto:outages@outages.org>
Subject: [outages] VPN issues over Spectrum L3 boundaries



We have several offices over the Ohio and Pennsylvania area that are experiencing issues passing traffic over VPN tunnels (specifically, there is always a Spectrum >< Level 3 interconnect). It is a very strange issue. The VPN tunnel will actually establish, and if you source your ping from inside the internal network across the VPN tunnel to the destination, the traffic gets there and replies, but the replies never make it back to the original sending point.



Anyone else experiencing any similar issues like this?



Best Regards,

Josh





This email and its attachments may contain privileged and confidential information and/or protected health information (PHI) intended solely for the use of Netsmart Technologies and the recipient(s) named above. If you are not the recipient, or the employee or agent responsible for delivering this message to the intended recipient, you are hereby notified that any review, dissemination, distribution, printing or copying of this email message and/or any attachments is strictly prohibited. If you have received this transmission in error, please email compliance@NTST.com immediately and permanently delete this email and any attachments.
Re: VPN issues over Spectrum L3 boundaries [ In reply to ]
No change for them because the one head-end is fed by L3, so there's no
way it can vanish from the path.? :)

On 10/21/2019 1:39 PM, Biddle, Josh wrote:
>
> Our issues have recently magically resolved (last Thursday 10/17) due
> to L3 vanishing from the hop list. Our traffic now disappears into
> ntt.net and we are seeing two way IPsec traffic without any issues.
>
> Marty, did your issues resolve?
>
>
> *From:* Marty Adkins <marty@martyadkins.com>
> *Sent:* Friday, October 18, 2019 4:01 PM
> *To:* Biddle, Josh <JBiddle@ntst.com>; outages@outages.org
> *Subject:* Re: [outages] VPN issues over Spectrum L3 boundaries
>
> A customer of mine has had the same issue with a TW-connected site in
> Ohio and another in PA.? One VPN tunnel works fine, the other has
> one-way ISAKMP traffic to the other head-end, which connects to
> Level3.? A traceroute shows the failing path includes 66.109.7.162.?
> The failing direction is from the PA/OH sites toward the L3 head end.
> Full-size pings work fine.? It's the UDP/500 that vanishes. I.E., it
> has nothing to do with MTU.
>
> Both started having the issue around Aug 26.? Mysteriously, every week
> to 10 days, the broken path will start working for a while.? This is
> usually shortly after midnight EDT; they go back down 1-3 hours later
> and stay down.? The log entries for the two sites match within seconds.
>
> The customer's contract is with Comcast Business so it's been
> difficult to get to someone clueful about this symptom in TW.
>
> -Marty
>
> On 10/15/2019 8:04 AM, Biddle, Josh via Outages wrote:
>
> Found a thread in the Spectrum forums talking about the issue
> finally ? it was marked as resolved so I started a new one.
>
> https://forums.timewarnercable.com/t5/Connectivity/Traffic-issues-at-66-109-7-162/m-p/164091#M53497
> <https://urldefense.proofpoint.com/v2/url?u=https-3A__forums.timewarnercable.com_t5_Connectivity_Traffic-2Dissues-2Dat-2D66-2D109-2D7-2D162_m-2Dp_164091-23M53497&d=DwMD-g&c=-7HNwxqfpkdcRXCW8HB54Q&r=svX1Si7sopSBMitBL3bFwQ&m=pXHvd5iI_J5DVYGMDKBSdUZl1iyilZUvR3oyvE3BJ0E&s=_90gCNY2Ln1XdqWtMNguWIRejhsdUxoLyvgTpo-R5jM&e=>
>
> *From:* Outages <outages-bounces@outages.org>
> <mailto:outages-bounces@outages.org> *On Behalf Of *Biddle, Josh
> via Outages
> *Sent:* Sunday, October 13, 2019 12:00 PM
> *To:* outages@outages.org <mailto:outages@outages.org>
> *Subject:* [outages] VPN issues over Spectrum L3 boundaries
>
> We have several offices over the Ohio and Pennsylvania area that
> are experiencing issues passing traffic over VPN tunnels
> (specifically, there is always a Spectrum >< Level 3
> interconnect). It is a very strange issue. The VPN tunnel will
> actually establish, and if you source your ping from inside the
> internal network across the VPN tunnel to the destination, the
> traffic gets there and replies, but the replies never make it back
> to the original sending point.
>
> Anyone else experiencing any similar issues like this?
>
> Best Regards,
>
> *Josh*
>
>
Re: VPN issues over Spectrum L3 boundaries [ In reply to ]
After getting the right TW/Spectrum folks involved, this was solved by a
routing change.? Traffic from OH and PA sites now traverse an L3 peering
point at Newark, and the NC site now goes via Atlanta.? The problematic
one-way path peered in D.C. and a TW team is still working to resolve
whether that's their issue or within L3.? Supposedly the routing changes
that were made on Oct 22 would affect other customers as well.

-Marty

On 10/21/2019 2:19 PM, Marty Adkins wrote:
> No change for them because the one head-end is fed by L3, so there's
> no way it can vanish from the path.? :)
>
> On 10/21/2019 1:39 PM, Biddle, Josh wrote:
>>
>> Our issues have recently magically resolved (last Thursday 10/17) due
>> to L3 vanishing from the hop list. Our traffic now disappears into
>> ntt.net and we are seeing two way IPsec traffic without any issues.
>>
>> Marty, did your issues resolve?
>>
>>
>>