Mailing List Archive

> > The folks at AMES and I cabled up the MAE-WEST facilities
> > there today and are waiting on MFS to enable the port so
> > that the two parts of MAE-WEST will talk to each other.
>
> Ummm - I've been peering with NSI for about a week now.
> It sure looks to me like the two parts of MAE-West are talking to each
> other just fine.

Actually, what Bill really meant was that the FDDI portion of MAE-West
is just now coming together. Yes, the ethernet port on the NetEdge has
been running for a week and is working fine. It's all MAE-West.

Jeff

Andrew Partan wrote:
> Ummm - I've been peering with NSI for about a week now.
> It sure looks to me like the two parts of MAE-West are talking to each
> other just fine.

Well, I don't know which halves of MAE-West Alternet and NSI are on, but I
noticed that Autralia (which was previously entirely unreachable unless
you had NSFNet routing) became reachable via both Alternet/MAE-West/NSI
and Sprint/FIX-West/NSI about a week ago.

--apb (Alan Barrett)

>
> > The folks at AMES and I cabled up the MAE-WEST facilities
> > there today and are waiting on MFS to enable the port so
> > that the two parts of MAE-WEST will talk to each other.
>
> Ummm - I've been peering with NSI for about a week now.
> It sure looks to me like the two parts of MAE-West are talking to each
> other just fine.
> --asp@uunet.uu.net (Andrew Partan)
>

NSI plugged directly into the netedge. ESnet, MCI, SPrint and the RA
are on another port.

--
--bill

Yeah, Most everyone is down.. I have received calls from NetCom, Agis,
MCI, BBN, and others asking we were down at Mae-West too.

Eric


_______________________________________________________
Eric D. Madison - Senior Network Engineer -
ACSI - Advanced Data Services - ATM/IP Backbone Group
24 Hour NMC/NOC (800)291-7889 Email: noc@acsi.net


On Fri, 11 Jul 1997, Golan Ben-Oni wrote:

> On Fri, 11 Jul 1997, Todd R. Stroup wrote:
>
> > So did Mae-West fall into the ocean out there or what? Does anybody know?
>
> Yes, reference master ticket #94946 re: Power Outage at Mae-West.
>
>

One of the benefits of being in the building is that if they don't
answer their phone... we can kick down the door. Our guys are down there
now.

According to our guys, it is a power problem... but they are getting
details...

I will post as soon as I know....


Rodney Joffe
Chief Technology Officer
Genuity Inc., a Bechtel company
http://www.genuity.net



> -----Original Message-----
> From: Todd R. Stroup [SMTP:tstroup@fibernet.net]
> Sent: Friday, July 11, 1997 7:49 AM
> To: nanog@merit.edu
> Subject: Mae-WEST
>
>
>
> So did Mae-West fall into the ocean out there or what? Does anybody
> know?
>
>
> Todd R. Stroup
> Fiber Network Solutions, Inc.

The MFS FDDI at MAE-West is currently down as of 7:30 PST. MFS
have dispatched a tech. who is supposed to be onsite within a short
while.


> So did Mae-West fall into the ocean out there or what? Does anybody
> know?
>
>
> Todd R. Stroup
> Fiber Network Solutions, Inc.

Last I heard, they rebooted the gigaswitch(es?) after the power came back
on, and the cards seemed to be loading their configs, but none were coming
active... Which seems to explain why I can't ping anything on the MAE
network even tho my FDDI is up :-)

Fun, fun, fun...

-Taner
--
D. Taner Halicioglu taner@isi.net
Programmer/Engineer/Sysadmin ISI / GlobalCenter
Voice: +1 408 543 0313 Fax: +1 408 541 9878
PGP Fingerprint: 65 0D 03 A8 26 21 6D B8 23 3A D6 67 23 6E C0 36

Matt,

Doesn't make for much of an exchange point when you are the only one
left standing, eh? :)

K

On Jul 11, 11:34am, Matthew Pearson wrote:
> Subject: MAE-WEST
> We are still up... for the time being at least.... heh.... I guess the guy
> wielding the chainsaw missed our power cables! Or was it a Ryder Truck?
>
> But 75% of our peers are down...
>
>-- End of excerpt from Matthew Pearson



--
| Kyle C. Bacon
| VP Operations
| kbacon@fibernet.net
| http://www.fibernet.net

connecting your world...

I was told that a contractor tripped the main breaker and the UPS's ran
down so that Gigaswitch-01 on the MFS side no longer had power.

Since the 4 OC-3c's that link MFS and Ames are on that switch, connectivity
has been lost until MFS can restore power. Folks on the Ames side
can still talk to each other-but that's about it.

-Lance-

Well power to my colo cabinet (the old BBN planet cabinet) in the B6 cage
just came back about an hour ago but we are still seeing a boat load of
neighbors down.

Todd R. Stroup
Fiber Network Solutions, Inc.

On Fri, 11 Jul 1997, Mike Leber wrote:

>
> On Fri, 11 Jul 1997, Todd R. Stroup wrote:
> > So did Mae-West fall into the ocean out there or what? Does anybody know?
>
> It appears that the Gigaswitches are still up. We can see Gridnet
> and IIJ at MAE-WEST. I would guess that power to the colo cages is down.
>
> Interesting. I remember seeing a cage or two with a customer provided
> 5000 watt UPS. Some people thought it was paranoia. Hmmm.
>
> Mike.
>
> +------------------- H U R R I C A N E - E L E C T R I C -------------------+
> | Mike Leber Direct Internet Connections Voice 408 282 1540 |
> | Hurricane Electric Web Hosting & Co-location Fax 408 971 3340 |
> | mleber@he.net http://www.he.net |
> +---------------------------------------------------------------------------+
>
>

I have a cage in the MFS Co-Lo and insisted on having a separate UPS from
the main.
While I never lost power today, my FDDI did go down.

This gets me thinking about the backup generator that is supposed to be
online.

Why didn't the generator kick-in before the UPS went down?

Perhaps the better question is if they have a functioning generator ...
Anyone from MFS listening?

-ps

At 09:15 AM 7/11/97 -0700, you wrote:
>
> I would guess that power to the colo cages is down.
>
>Mike.

Or, lets say, an automatic network monitor (ALA SNMP) that tells them
when UPS power is about to run down (before the lights go out) so they
can roll their generator into place.

-Deepak.,

On Fri, 11 Jul 1997, Philip Steffora wrote:

>
> I have a cage in the MFS Co-Lo and insisted on having a separate UPS from
> the main.
> While I never lost power today, my FDDI did go down.
>
> This gets me thinking about the backup generator that is supposed to be
> online.
>
> Why didn't the generator kick-in before the UPS went down?
>
> Perhaps the better question is if they have a functioning generator ...
> Anyone from MFS listening?
>
> -ps
>
> At 09:15 AM 7/11/97 -0700, you wrote:
> >
> > I would guess that power to the colo cages is down.
> >
> >Mike.
>
>
>

Yes, there's a generator. Unfortunately, it doesn't kick in when someone
trips or turns off the main breaker. (This is hearsay; I'm actually out
of the office and am getting my info second-hand.)
Steve

On Fri, 11 Jul 1997, Philip Steffora wrote:

>
> I have a cage in the MFS Co-Lo and insisted on having a separate UPS from
> the main.
> While I never lost power today, my FDDI did go down.
>
> This gets me thinking about the backup generator that is supposed to be
> online.
>
> Why didn't the generator kick-in before the UPS went down?
>
> Perhaps the better question is if they have a functioning generator ...
> Anyone from MFS listening?
>
> -ps
>
> At 09:15 AM 7/11/97 -0700, you wrote:
> >
> > I would guess that power to the colo cages is down.
> >
> >Mike.
>
>

The message I got was that a contractor tripped a breaker, the UPS was
run down and out of juice and the generator took a while to kick in.
The power outage took out Gigaswitch #1, effectivly dropping the whole
MAE. When power was restored the switch had a memory problem that
wouldn't allow the config to load. Memory fixed, config loaded, but now
there is still a problem with a switch port (#4, MFS said) that has kept
six of our peers down. A couple of suggestions/observations:

o Monitor the UPS so it doesn't run out of juice.
o Reconfigure the Gigaswitch topology so that one loss
doesn't take out all peers.
o MFS call handling at the customer service center was
good, considering the number of calls they must have
got. Shirley even took the time to call me directly
with an update. Cool. Maybe MFS should consider
a mail server that would send out periodic status
updates to subscribers?

----------------------------------

> I have a cage in the MFS Co-Lo and insisted on having a separate UPS
> from
> the main.
> While I never lost power today, my FDDI did go down.
>
> This gets me thinking about the backup generator that is supposed to
> be
> online.
>
> Why didn't the generator kick-in before the UPS went down?
>
> Perhaps the better question is if they have a functioning generator
> ...
> Anyone from MFS listening?
>
> -ps
>
> At 09:15 AM 7/11/97 -0700, you wrote:
> >
> > I would guess that power to the colo cages is down.
> >
> >Mike.
>

>Or, lets say, an automatic network monitor (ALA SNMP) that tells them
>when UPS power is about to run down (before the lights go out) so they
>can roll their generator into place.

Shades of the massive AT&T failure of a few years ago.

Lose external power, go to battery -- batteries run flat, cause nobody
is paying attention.

--
Eric Kozowski VP Internet Services
eric@structured.net Structured Network Systems, Inc.
(800)390-5945 Support A Verio Affiliate
(800)881-0962 Sales/Info http://www.structured.net/
PGP Key fingerprint = 0A E4 91 B8 BD CB 3E 95 42 12 04 4E 14 DF 86 76
'They that can give up essential liberty to obtain a little temporary
safety deserve neither liberty nor safety.' -- Benjamin Franklin 1759

>> This gets me thinking about the backup generator that is supposed to be
>> online.
>>
>> Why didn't the generator kick-in before the UPS went down?
>Or, lets say, an automatic network monitor (ALA SNMP) that tells them
>when UPS power is about to run down (before the lights go out) so they
>can roll their generator into place.

It is amazing how many connectivity failures are power related.

For an APC UPS, all it takes is a dedicated $500 sparcstation,
the $99 powerchute UPS monitoring software, and a $99 modem
attached to the sparc for sending out a text page every time
the UPS coughs.

Sorry for stating the obvious.

On Fri, 11 Jul 1997, Eric Kozowski wrote:

> Lose external power, go to battery -- batteries run flat, cause nobody
> is paying attention.

Getting MFS to pay attention to the Heat Levels at their Colo facilities
is another issue. In the not-so-distant past, we reported critical heat
levels at 55 Market, SJ. After having our ticket closed out on us, and
calling back several times, MFS finally sent someone over to have a look.

According to the information I got back, two of their Air Conditioning
units failed, which took them over a day to fix. This kinda thing does
wonders to our MTBFs :( ...

Ironically, our temperature device is now broken in our Cisco. Has anyone
else had similar problems with them? We now have a number of 7000 series
routers which report off-the-wall readings.

-Golan

Obviously, as a MFS customer that doesn't make me feel a whole bunch better
about the situation in San Jose.

-phil

==================================
Philip Steffora steffora@pa.dec.com
Operations Mgr. AltaVista - Digital Equip Corp
==================================

At 11:25 AM 7/11/97 -0700, Steve Feldman wrote:
>Yes, there's a generator. Unfortunately, it doesn't kick in when someone
>trips or turns off the main breaker. (This is hearsay; I'm actually out
>of the office and am getting my info second-hand.)
> Steve

At 03:40 PM 7/11/97 -0400, Golan Ben-Oni wrote:
>On Fri, 11 Jul 1997, Eric Kozowski wrote:
>
>> Lose external power, go to battery -- batteries run flat, cause nobody
>> is paying attention.
>
>Getting MFS to pay attention to the Heat Levels at their Colo facilities
>is another issue. In the not-so-distant past, we reported critical heat
>levels at 55 Market, SJ. After having our ticket closed out on us, and
>calling back several times, MFS finally sent someone over to have a look.
>
>According to the information I got back, two of their Air Conditioning
>units failed, which took them over a day to fix. This kinda thing does
>wonders to our MTBFs :( ...
>
>Ironically, our temperature device is now broken in our Cisco. Has anyone
>else had similar problems with them? We now have a number of 7000 series
>routers which report off-the-wall readings.
>
>-Golan
>
>
>

MFS is in the proccess of installing new cooling units on the roof. They
take up aprox. 1000sqft. So they are at least working on it.

You should have seen the air lift :)


James McKenzie
mcs@innetix.com
Innetix, Inc.
(408) 993-9477
http://www.innetix.com

> MFS is in the proccess of installing new cooling units on the roof. They
> take up aprox. 1000sqft. So they are at least working on it.
>
> You should have seen the air lift :)
>
>

Any idea what the capacity of the additional equipment?

Thanks.

-Deepak.

If I remember right, the 7000 series routers have the env sensors on
the arbiter. I don't remember it being on a particular card, so it's
not a simple swap :(

Barry

> At 03:40 PM 7/11/97 -0400, Golan Ben-Oni wrote:
> >On Fri, 11 Jul 1997, Eric Kozowski wrote:
> >Ironically, our temperature device is now broken in our Cisco. Has anyone
> >else had similar problems with them? We now have a number of 7000 series
> >routers which report off-the-wall readings.
> >
> >-Golan
> >
> >
> >
>
> MFS is in the proccess of installing new cooling units on the roof. They
> take up aprox. 1000sqft. So they are at least working on it.
>
> You should have seen the air lift :)
>
>
> James McKenzie
> mcs@innetix.com
> Innetix, Inc.
> (408) 993-9477
> http://www.innetix.com

From: Deepak Jain <deepak@jain.com>

Or, lets say, an automatic network monitor (ALA SNMP) that tells them
when UPS power is about to run down (before the lights go out) so they
can roll their generator into place.

Or, more likely, assuming that the story about a contractor popping an
upstream distribution breaker was correct, that the generator and
transfer switch were upstream from _that_ breaker (which would make it
quite unlikely that power would flow).

---Rob

According to nitrous.digex.net for its mae-west router:

card inlet hotpoint exhaust
RSP(4) 26C/78F 38C/100F 44C/111F

Seems a little toasty to me.


>Ironically, our temperature device is now broken in our Cisco. Has anyone
>else had similar problems with them? We now have a number of 7000 series
>routers which report off-the-wall readings.
>
>-Golan
>
>
>
---

"Don't go with a spineless ISP;
we have more backbone."

Alex Rubenstein -- alex@nac.net -- KC2BUO -- www.nac.net
net @ccess corporation, 201-983-0725 -- 201-983-0725