Mailing List Archive: Level3 / Yipes

Level3 / Yipes

Aug 30, 2007, 9:34 PM

Post #1 of 4 (817 views)

I'm seeing some entertaining routes to sites on Yipes/Level3,
according to Yipes! support Level3 just lost an OC48 between NY &
Chicago.

(This is to a site on Yipes)
1 137 ms 134 ms 159 ms 5.sub-66-174-22.myvzw.com [66.174.22.5]
2 * * * Request timed out.
3 98 ms 131 ms 132 ms 113.sub-66-174-120.myvzw.com [66.174.120.113]
4 169 ms 130 ms 133 ms 65.sub-66-174-22.myvzw.com [66.174.22.65]
5 147 ms 155 ms 131 ms 98.sub-66-174-22.myvzw.com [66.174.22.98]
6 * * * Request timed out.
7 173 ms 133 ms 290 ms 64.200.80.25
8 426 ms 133 ms 76 ms nwrknj1wcx1-pos6-0.wcg.net [64.200.68.138]
9 111 ms 131 ms 131 ms nycmny2wcx3-pos11-1.wcg.net [64.200.240.233]
10 130 ms 157 ms 134 ms 64.200.249.154
11 134 ms 144 ms 139 ms 64.200.249.149
12 * 158 ms * 64.200.249.150
13 * 144 ms 162 ms 64.200.249.149
14 * 181 ms * 64.200.249.150
15 * * 163 ms 64.200.249.149
16 204 ms * 178 ms 64.200.249.150
17 * * 198 ms 64.200.249.149
18 * * * Request timed out.
19 * 194 ms 185 ms 64.200.249.149
20 * 334 ms * 64.200.249.150
21 * * * Request timed out.

-J

Level3 / Yipes [ In reply to ]

paul.f

Aug 30, 2007, 11:20 PM

Post #2 of 4 (813 views)

Permalink

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Primary Dates: 31 Aug 2007 00:00:00 - 31 Aug 2007 06:00:00 Eastern
Primary Dates GMT: 31 Aug 2007 04:00:00 - 31 Aug 2007 10:00:00
Location of Maintenance: East Coast

Summary Description of Maintenance: Upgrade - Code - Edge Router
Customer Impact:
Service Expected Impact Classification Duration Additional Notes
IP - Non-Redundant Outage SA Up to 30 minutes Summary
Description of Maintenance: Demand maintenance is needed to revert
router code to a previous version to prevent possible network outages.
This change should only take a few minutes while the router reloads, and
service impact will be minimal. Below is the list of services that will
be affected during the reload.

May be related?
Jayson Navitsky wrote:
> I'm seeing some entertaining routes to sites on Yipes/Level3,
> according to Yipes! support Level3 just lost an OC48 between NY &
> Chicago.
>
> (This is to a site on Yipes)
> 1 137 ms 134 ms 159 ms 5.sub-66-174-22.myvzw.com [66.174.22.5]
> 2 * * * Request timed out.
> 3 98 ms 131 ms 132 ms 113.sub-66-174-120.myvzw.com [66.174.120.113]
> 4 169 ms 130 ms 133 ms 65.sub-66-174-22.myvzw.com [66.174.22.65]
> 5 147 ms 155 ms 131 ms 98.sub-66-174-22.myvzw.com [66.174.22.98]
> 6 * * * Request timed out.
> 7 173 ms 133 ms 290 ms 64.200.80.25
> 8 426 ms 133 ms 76 ms nwrknj1wcx1-pos6-0.wcg.net [64.200.68.138]
> 9 111 ms 131 ms 131 ms nycmny2wcx3-pos11-1.wcg.net [64.200.240.233]
> 10 130 ms 157 ms 134 ms 64.200.249.154
> 11 134 ms 144 ms 139 ms 64.200.249.149
> 12 * 158 ms * 64.200.249.150
> 13 * 144 ms 162 ms 64.200.249.149
> 14 * 181 ms * 64.200.249.150
> 15 * * 163 ms 64.200.249.149
> 16 204 ms * 178 ms 64.200.249.150
> 17 * * 198 ms 64.200.249.149
> 18 * * * Request timed out.
> 19 * 194 ms 185 ms 64.200.249.149
> 20 * 334 ms * 64.200.249.150
> 21 * * * Request timed out.
>
>
> -J
> _______________________________________________
> Outages mailing list
> Outages at isotf.org
> http://isotf.org/mailman/listinfo/outages
>

- --
Paul Fleming
Network Operations
Hostdime.com Inc
Cell:407.468.4646
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.3 (MingW32)

iD8DBQFG17MXwlPOUqXUp3MRAixkAJ4kfwgMQ/4GRWzsin0VKAGlNn0UggCgz+9z
E2x6VSy19xA05iz7KShEvn0=
=4zGW
-----END PGP SIGNATURE-----

Level3 / Yipes [ In reply to ]

adamkuj

Aug 31, 2007, 7:32 AM

Post #3 of 4 (811 views)

Permalink

I've seen several Emergency Maintenance notifications from Level3 last
night, from all across their network -- apparently they were rebooting
a bunch (all?) of their routers:

"Demand maintenance is needed to revert router code to a previous
version to prevent possible network outages. This change should only
take a few minutes while the router reloads, and service impact will
be minimal."

From where I sit, I saw a bunch of route instability and loss of
reachability across their network for up to 30 minutes. When I finally
got ahold of Level3 around 12:30AM, they said they were just about to
start on the west coast, doing one router at a time.

-Adam

On 8/30/07, Jayson Navitsky <jnavitsky at gmail.com> wrote:
> I'm seeing some entertaining routes to sites on Yipes/Level3,
> according to Yipes! support Level3 just lost an OC48 between NY &
> Chicago.
>
> (This is to a site on Yipes)
> 1 137 ms 134 ms 159 ms 5.sub-66-174-22.myvzw.com [66.174.22.5]
> 2 * * * Request timed out.
> 3 98 ms 131 ms 132 ms 113.sub-66-174-120.myvzw.com [66.174.120.113]
> 4 169 ms 130 ms 133 ms 65.sub-66-174-22.myvzw.com [66.174.22.65]
> 5 147 ms 155 ms 131 ms 98.sub-66-174-22.myvzw.com [66.174.22.98]
> 6 * * * Request timed out.
> 7 173 ms 133 ms 290 ms 64.200.80.25
> 8 426 ms 133 ms 76 ms nwrknj1wcx1-pos6-0.wcg.net [64.200.68.138]
> 9 111 ms 131 ms 131 ms nycmny2wcx3-pos11-1.wcg.net [64.200.240.233]
> 10 130 ms 157 ms 134 ms 64.200.249.154
> 11 134 ms 144 ms 139 ms 64.200.249.149
> 12 * 158 ms * 64.200.249.150
> 13 * 144 ms 162 ms 64.200.249.149
> 14 * 181 ms * 64.200.249.150
> 15 * * 163 ms 64.200.249.149
> 16 204 ms * 178 ms 64.200.249.150
> 17 * * 198 ms 64.200.249.149
> 18 * * * Request timed out.
> 19 * 194 ms 185 ms 64.200.249.149
> 20 * 334 ms * 64.200.249.150
> 21 * * * Request timed out.
>
>
> -J
> _______________________________________________
> Outages mailing list
> Outages at isotf.org
> http://isotf.org/mailman/listinfo/outages
>

Level3 / Yipes [ In reply to ]

jnavitsky

Aug 31, 2007, 9:44 AM

Post #4 of 4 (811 views)

Permalink

Yeah I was seeing weird stuff, routing loops, path from NYC to Seattle
was via TX, etc. Maybe it was the annual Internet Cleaning...

-Jay

On 8/31/07, Adam Kujawski <adamkuj at gmail.com> wrote:
> I've seen several Emergency Maintenance notifications from Level3 last
> night, from all across their network -- apparently they were rebooting
> a bunch (all?) of their routers:
>
> "Demand maintenance is needed to revert router code to a previous
> version to prevent possible network outages. This change should only
> take a few minutes while the router reloads, and service impact will
> be minimal."
>
> From where I sit, I saw a bunch of route instability and loss of
> reachability across their network for up to 30 minutes. When I finally
> got ahold of Level3 around 12:30AM, they said they were just about to
> start on the west coast, doing one router at a time.
>
> -Adam
>
>
> On 8/30/07, Jayson Navitsky <jnavitsky at gmail.com> wrote:
> > I'm seeing some entertaining routes to sites on Yipes/Level3,
> > according to Yipes! support Level3 just lost an OC48 between NY &
> > Chicago.
> >
> > (This is to a site on Yipes)
> > 1 137 ms 134 ms 159 ms 5.sub-66-174-22.myvzw.com [66.174.22.5]
> > 2 * * * Request timed out.
> > 3 98 ms 131 ms 132 ms 113.sub-66-174-120.myvzw.com
> [66.174.120.113]
> > 4 169 ms 130 ms 133 ms 65.sub-66-174-22.myvzw.com [66.174.22.65]
> > 5 147 ms 155 ms 131 ms 98.sub-66-174-22.myvzw.com [66.174.22.98]
> > 6 * * * Request timed out.
> > 7 173 ms 133 ms 290 ms 64.200.80.25
> > 8 426 ms 133 ms 76 ms nwrknj1wcx1-pos6-0.wcg.net [64.200.68.138]
> > 9 111 ms 131 ms 131 ms nycmny2wcx3-pos11-1.wcg.net
> [64.200.240.233]
> > 10 130 ms 157 ms 134 ms 64.200.249.154
> > 11 134 ms 144 ms 139 ms 64.200.249.149
> > 12 * 158 ms * 64.200.249.150
> > 13 * 144 ms 162 ms 64.200.249.149
> > 14 * 181 ms * 64.200.249.150
> > 15 * * 163 ms 64.200.249.149
> > 16 204 ms * 178 ms 64.200.249.150
> > 17 * * 198 ms 64.200.249.149
> > 18 * * * Request timed out.
> > 19 * 194 ms 185 ms 64.200.249.149
> > 20 * 334 ms * 64.200.249.150
> > 21 * * * Request timed out.
> >
> >
> > -J
> > _______________________________________________
> > Outages mailing list
> > Outages at isotf.org
> > http://isotf.org/mailman/listinfo/outages
> >
>