Mailing List Archive

At 09:08 AM 2024-01-15, Mike Hammett wrote:
>Let's say that hypothetically, a datacenter you're in had a cooling
>failure and escalated to an average of 120 degrees before
>mitigations started having an effect. What are normal QA procedures
>on your behalf? What is the facility likely to be doing?
>What should be expected in the aftermath?

One would hope they would have had disaster recovery plans to bring
in outside cold air, and have executed on it quickly, rather than
hoping the chillers got repaired.

All our owned facilities have large outside air intakes, automatic
dampers and air mixing chambers in case of mechanical cooling
failure, because cooling systems are often not designed to run well
in extreme cold. All of these can be manually run incase of controls
failure, but people tell me I'm a little obsessive over backup plans
for backup plans.

You will start to see premature failure of equipment over the coming
weeks/months/years.

Coincidentally, we have some gear in a data centre in the Chicago
area that is experiencing that sort of issue right now... :-(

Coincidence indeed.... ;-)




-----
Mike Hammett
Intelligent Computing Solutions

Midwest Internet Exchange

The Brothers WISP

----- Original Message -----

From: "Clayton Zekelman" <clayton@MNSi.Net>
To: "Mike Hammett" <nanog@ics-il.net>, "NANOG" <nanog@nanog.org>
Sent: Monday, January 15, 2024 8:23:37 AM
Subject: Re: "Hypothetical" Datacenter Overheating




At 09:08 AM 2024-01-15, Mike Hammett wrote:
>Let's say that hypothetically, a datacenter you're in had a cooling
>failure and escalated to an average of 120 degrees before
>mitigations started having an effect. What are normal QA procedures
>on your behalf? What is the facility likely to be doing?
>What should be expected in the aftermath?

One would hope they would have had disaster recovery plans to bring
in outside cold air, and have executed on it quickly, rather than
hoping the chillers got repaired.

All our owned facilities have large outside air intakes, automatic
dampers and air mixing chambers in case of mechanical cooling
failure, because cooling systems are often not designed to run well
in extreme cold. All of these can be manually run incase of controls
failure, but people tell me I'm a little obsessive over backup plans
for backup plans.

You will start to see premature failure of equipment over the coming
weeks/months/years.

Coincidentally, we have some gear in a data centre in the Chicago
area that is experiencing that sort of issue right now... :-(

On Mon, Jan 15, 2024 at 6:08?AM Mike Hammett <nanog@ics-il.net> wrote:
> Let's say that hypothetically, a datacenter you're in had a cooling failure
> and escalated to an average of 120 degrees before mitigations started
> having an effect. What should be expected in the aftermath?

Hi Mike,

A decade or so ago I maintained a computer room with a single air
conditioner because the boss wouldn't go for n+1. It failed in exactly
this manner several times. After the overheat was detected by the
monitoring system, it would be brought under control with a
combination of spot cooler and powering down to a minimal
configuration. But of course it takes time to get people there and set
up the mitigations, during which the heat continues to rise.

The main thing I noticed was a modest uptick in spinning drive
failures for the couple months that followed. If there was any other
consequence it was at a rate where I'd have had to be carefully
measuring before and after to detect it.

Regards,
Bill Herrin


--
William Herrin
bill@herrin.us
https://bill.herrin.us/

I think we're beyond "hypothetical" at this point, Mike ... ;)


On 1/15/24 15:49, Mike Hammett wrote:
> Coincidence indeed....   ;-)
>
>
>
> -----
> Mike Hammett
> Intelligent Computing Solutions <http://www.ics-il.com/>
> <https://www.facebook.com/ICSIL><https://plus.google.com/+IntelligentComputingSolutionsDeKalb><https://www.linkedin.com/company/intelligent-computing-solutions><https://twitter.com/ICSIL>
> Midwest Internet Exchange <http://www.midwest-ix.com/>
> <https://www.facebook.com/mdwestix><https://www.linkedin.com/company/midwest-internet-exchange><https://twitter.com/mdwestix>
> The Brothers WISP <http://www.thebrotherswisp.com/>
> <https://www.facebook.com/thebrotherswisp><https://www.youtube.com/channel/UCXSdfxQv7SpoRQYNyLwntZg>
> ------------------------------------------------------------------------
> *From: *"Clayton Zekelman" <clayton@MNSi.Net>
> *To: *"Mike Hammett" <nanog@ics-il.net>, "NANOG" <nanog@nanog.org>
> *Sent: *Monday, January 15, 2024 8:23:37 AM
> *Subject: *Re: "Hypothetical" Datacenter Overheating
>
>
>
>
> At 09:08 AM 2024-01-15, Mike Hammett wrote:
> >Let's say that hypothetically, a datacenter you're in had a cooling
> >failure and escalated to an average of 120 degrees before
> >mitigations started having an effect. What are normal QA procedures
> >on your behalf? What is the facility likely to be doing?
> >What  should be expected in the aftermath?
>
> One would hope they would have had disaster recovery plans to bring
> in outside cold air, and have executed on it quickly, rather than
> hoping the chillers got repaired.
>
> All our owned facilities have large outside air intakes, automatic
> dampers and air mixing chambers in case of mechanical cooling
> failure, because cooling systems are often not designed to run well
> in extreme cold.  All of these can be manually run incase of controls
> failure, but people tell me I'm a little obsessive over backup plans
> for backup plans.
>
> You will start to see premature failure of equipment over the coming
> weeks/months/years.
>
> Coincidentally, we have some gear in a data centre in the Chicago
> area that is experiencing that sort of issue right now... :-(
>
>
>
>
>

Our Zayo circuit just came up 30 minutes ago and it routes through 350 E
Cermak.  Chillers were all messed up.  No hypothetical there.  :-) It
was down for over 16 hours!

On 1/15/24 10:04 AM, Bryan Holloway wrote:
> I think we're beyond "hypothetical" at this point, Mike ... ;)
>
>
> On 1/15/24 15:49, Mike Hammett wrote:
>> Coincidence indeed....   ;-)
>>
>>
>>
>> -----
>> Mike Hammett
>> Intelligent Computing Solutions <http://www.ics-il.com/>
>> <https://www.facebook.com/ICSIL><https://plus.google.com/+IntelligentComputingSolutionsDeKalb><https://www.linkedin.com/company/intelligent-computing-solutions><https://twitter.com/ICSIL>
>>
>> Midwest Internet Exchange <http://www.midwest-ix.com/>
>> <https://www.facebook.com/mdwestix><https://www.linkedin.com/company/midwest-internet-exchange><https://twitter.com/mdwestix>
>>
>> The Brothers WISP <http://www.thebrotherswisp.com/>
>> <https://www.facebook.com/thebrotherswisp><https://www.youtube.com/channel/UCXSdfxQv7SpoRQYNyLwntZg>
>>
>> ------------------------------------------------------------------------
>> *From: *"Clayton Zekelman" <clayton@MNSi.Net>
>> *To: *"Mike Hammett" <nanog@ics-il.net>, "NANOG" <nanog@nanog.org>
>> *Sent: *Monday, January 15, 2024 8:23:37 AM
>> *Subject: *Re: "Hypothetical" Datacenter Overheating
>>
>>
>>
>>
>> At 09:08 AM 2024-01-15, Mike Hammett wrote:
>>  >Let's say that hypothetically, a datacenter you're in had a cooling
>>  >failure and escalated to an average of 120 degrees before
>>  >mitigations started having an effect. What are normal QA procedures
>>  >on your behalf? What is the facility likely to be doing?
>>  >What  should be expected in the aftermath?
>>
>> One would hope they would have had disaster recovery plans to bring
>> in outside cold air, and have executed on it quickly, rather than
>> hoping the chillers got repaired.
>>
>> All our owned facilities have large outside air intakes, automatic
>> dampers and air mixing chambers in case of mechanical cooling
>> failure, because cooling systems are often not designed to run well
>> in extreme cold.  All of these can be manually run incase of controls
>> failure, but people tell me I'm a little obsessive over backup plans
>> for backup plans.
>>
>> You will start to see premature failure of equipment over the coming
>> weeks/months/years.
>>
>> Coincidentally, we have some gear in a data centre in the Chicago
>> area that is experiencing that sort of issue right now... :-(
>>
>>
>>
>>
>>

I’m more interested in how you lose six chillers all at once.
Shane

On Jan 15, 2024, at 9:11?AM, Mike Hammett <nanog@ics-il.net> wrote:

?Let's say that hypothetically, a datacenter you're in had a cooling failure and escalated to an average of 120 degrees before mitigations started having an effect. What are normal QA procedures on your behalf? What is the facility likely to be doing? What should be expected in the aftermath?



-----
Mike Hammett
http://www.ics-il.com/"]Intelligent Computing Solutions
https://www.facebook.com/ICSIL"]https://plus.google.com/+IntelligentComputingSolutionsDeKalb"]https://www.linkedin.com/company/intelligent-computing-solutions"]https://twitter.com/ICSIL"]
http://www.midwest-ix.com/"]Midwest Internet Exchange
https://www.facebook.com/mdwestix"]https://www.linkedin.com/company/midwest-internet-exchange"]https://twitter.com/mdwestix"]
http://www.thebrotherswisp.com/"]The Brothers WISP
https://www.facebook.com/thebrotherswisp"]https://www.youtube.com/channel/UCXSdfxQv7SpoRQYNyLwntZg"]

and none in the other two facilities you operate in that same building had any failures.




-----
Mike Hammett
Intelligent Computing Solutions

Midwest Internet Exchange

The Brothers WISP

----- Original Message -----

From: sronan@ronan-online.com
To: "Mike Hammett" <nanog@ics-il.net>
Cc: "NANOG" <nanog@nanog.org>
Sent: Monday, January 15, 2024 9:14:49 AM
Subject: Re: "Hypothetical" Datacenter Overheating



I’m more interested in how you lose six chillers all at once.


Shane



On Jan 15, 2024, at 9:11 AM, Mike Hammett <nanog@ics-il.net> wrote:




<blockquote>


Let's say that hypothetically, a datacenter you're in had a cooling failure and escalated to an average of 120 degrees before mitigations started having an effect. What are normal QA procedures on your behalf? What is the facility likely to be doing? What should be expected in the aftermath?




-----
Mike Hammett
Intelligent Computing Solutions

Midwest Internet Exchange

The Brothers WISP


</blockquote>

Easy. Climate change. Lol!

-mel

On Jan 15, 2024, at 7:17 AM, sronan@ronan-online.com wrote:

?
I’m more interested in how you lose six chillers all at once.

Shane

On Jan 15, 2024, at 9:11?AM, Mike Hammett <nanog@ics-il.net> wrote:

?
Let's say that hypothetically, a datacenter you're in had a cooling failure and escalated to an average of 120 degrees before mitigations started having an effect. What are normal QA procedures on your behalf? What is the facility likely to be doing? What should be expected in the aftermath?



-----
Mike Hammett
Intelligent Computing Solutions<http://www.ics-il.com/>
[http://www.ics-il.com/images/fbicon.png]<https://www.facebook.com/ICSIL>[http://www.ics-il.com/images/googleicon.png]<https://plus.google.com/+IntelligentComputingSolutionsDeKalb>[http://www.ics-il.com/images/linkedinicon.png]<https://www.linkedin.com/company/intelligent-computing-solutions>[http://www.ics-il.com/images/twittericon.png]<https://twitter.com/ICSIL>
Midwest Internet Exchange<http://www.midwest-ix.com/>
[http://www.ics-il.com/images/fbicon.png]<https://www.facebook.com/mdwestix>[http://www.ics-il.com/images/linkedinicon.png]<https://www.linkedin.com/company/midwest-internet-exchange>[http://www.ics-il.com/images/twittericon.png]<https://twitter.com/mdwestix>
The Brothers WISP<http://www.thebrotherswisp.com/>
[http://www.ics-il.com/images/fbicon.png]<https://www.facebook.com/thebrotherswisp>[http://www.ics-il.com/images/youtubeicon.png]<https://www.youtube.com/channel/UCXSdfxQv7SpoRQYNyLwntZg>

Exactly. Perhaps they weren’t all online to begin with…

On Jan 15, 2024, at 10:18?AM, Mike Hammett <nanog@ics-il.net> wrote:

?and none in the other two facilities you operate in that same building had any failures.



-----
Mike Hammett
http://www.ics-il.com/"]Intelligent Computing Solutions
https://www.facebook.com/ICSIL"]https://plus.google.com/+IntelligentComputingSolutionsDeKalb"]https://www.linkedin.com/company/intelligent-computing-solutions"]https://twitter.com/ICSIL"]
http://www.midwest-ix.com/"]Midwest Internet Exchange
https://www.facebook.com/mdwestix"]https://www.linkedin.com/company/midwest-internet-exchange"]https://twitter.com/mdwestix"]
http://www.thebrotherswisp.com/"]The Brothers WISP
https://www.facebook.com/thebrotherswisp"]https://www.youtube.com/channel/UCXSdfxQv7SpoRQYNyLwntZg"]

---

From: sronan@ronan-online.com
To: "Mike Hammett" <nanog@ics-il.net>
Cc: "NANOG" <nanog@nanog.org>
Sent: Monday, January 15, 2024 9:14:49 AM
Subject: Re: "Hypothetical" Datacenter Overheating

I’m more interested in how you lose six chillers all at once.
Shane

On Jan 15, 2024, at 9:11?AM, Mike Hammett <nanog@ics-il.net> wrote:

?Let's say that hypothetically, a datacenter you're in had a cooling failure and escalated to an average of 120 degrees before mitigations started having an effect. What are normal QA procedures on your behalf? What is the facility likely to be doing? What should be expected in the aftermath?



-----
Mike Hammett
http://www.ics-il.com/"]Intelligent Computing Solutions
https://www.facebook.com/ICSIL"]https://plus.google.com/+IntelligentComputingSolutionsDeKalb"]https://www.linkedin.com/company/intelligent-computing-solutions"]https://twitter.com/ICSIL"]
http://www.midwest-ix.com/"]Midwest Internet Exchange
https://www.facebook.com/mdwestix"]https://www.linkedin.com/company/midwest-internet-exchange"]https://twitter.com/mdwestix"]
http://www.thebrotherswisp.com/"]The Brothers WISP
https://www.facebook.com/thebrotherswisp"]https://www.youtube.com/channel/UCXSdfxQv7SpoRQYNyLwntZg"]

On 1/15/24 07:21, Mel Beckman wrote:
> Easy. Climate change. Lol!

It was -8°F in Chicago yesterday.

>> On Jan 15, 2024, at 7:17 AM, sronan@ronan-online.com wrote:
>>
>> ?
>> I’m more interested in how you lose six chillers all at once.


--
Jay Hennigan - jay@west.net
Network Engineering - CCIE #7880
503 897-8550 - WB6RDV

On 1/15/24 10:37, Pennington, Scott wrote:
> yes but.... it has been -8 in Chicago plenty of times before this.
>  Very interested in root cause...

Absolutely. My point was that claiming "Global warming" isn't going to
fly as an excuse.

> ------------------------------------------------------------------------
> *From:* NANOG <nanog-bounces+scott.pennington=cinbell.com@nanog.org> on
> behalf of Jay Hennigan <jay@west.net>
> *Sent:* Monday, January 15, 2024 1:31 PM
> *To:* nanog@nanog.org <nanog@nanog.org>
> *Subject:* Re: "Hypothetical" Datacenter Overheating
> On 1/15/24 07:21, Mel Beckman wrote:
>> Easy. Climate change. Lol!
>
> It was -8°F in Chicago yesterday.
>
>>> On Jan 15, 2024, at 7:17 AM, sronan@ronan-online.com wrote:
>>>
>>> ?
>>> I’m more interested in how you lose six chillers all at once.

--
Jay Hennigan - jay@west.net
Network Engineering - CCIE #7880
503 897-8550 - WB6RDV

My sarcasm generator is clearly set incorrectly :)

-mel

> On Jan 15, 2024, at 10:33 AM, Jay Hennigan <jay@west.net> wrote:
>
> ?On 1/15/24 07:21, Mel Beckman wrote:
>> Easy. Climate change. Lol!
>
> It was -8°F in Chicago yesterday.
>
>>>> On Jan 15, 2024, at 7:17 AM, sronan@ronan-online.com wrote:
>>>
>>> ?
>>> I’m more interested in how you lose six chillers all at once.
>
>
> --
> Jay Hennigan - jay@west.net
> Network Engineering - CCIE #7880
> 503 897-8550 - WB6RDV
>

On Mon, Jan 15, 2024 at 4:10?PM Jay Hennigan <jay@west.net> wrote:

> On 1/15/24 10:37, Pennington, Scott wrote:
> > yes but.... it has been -8 in Chicago plenty of times before this.
> > Very interested in root cause...
>
> Absolutely. My point was that claiming "Global warming" isn't going to
> fly as an excuse.
>

+1

Is their design N+1?

https://www.equinix.com/data-centers/americas-colocation/united-states-colocation/chicago-data-centers/ch1

We're not smashing temp records in Chicago. At least it doesn't seem so
when you look across historical data:

https://www.weather.gov/lot/Chicago_Temperature_Records

HTH,

-M<

On Mon, Jan 15, 2024 at 7:14?AM <sronan@ronan-online.com> wrote:
> I’m more interested in how you lose six chillers all at once.

Extreme cold. If the transfer temperature is too low, they can reach a
state where the refrigerant liquifies too soon, damaging the
compressor.

Regards,
Bill Herrin


--
William Herrin
bill@herrin.us
https://bill.herrin.us/

On Mon, Jan 15, 2024 at 7:14?AM <sronan@ronan-online.com> wrote:
>> I’m more interested in how you lose six chillers all at once.
>Extreme cold. If the transfer temperature is too low, they can reach a
>state where the refrigerant liquifies too soon, damaging the
compressor.
>Regards,
>Bill Herrin

Our 70-ton Tranes here have kicked out on 'freeze warning' before; there's a strainer in the water loop at the evaporator that can clog, restricting flow enough to allow freezing to occur if the chiller is actively cooling.  It's so strange to have an overheating data center in subzero (F) temps.  The flow sensor in the water loop can sometimes get too cold and not register the flow as well.

________________________________

On Mon, 2024-01-15 at 08:08 -0600, Mike Hammett wrote:
> Let's say that hypothetically, a datacenter you're in had a cooling
> failure and escalated to an average of 120 degrees

Major double-take there for this non-US reader, until I realised you
just had to mean Fahrenheit.

Regards, K.

--
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Karl Auer (kauer@biplane.com.au)
http://www.biplane.com.au/kauer

Someone I talked to while on scene today said their area got to 130 and cooked two core routers.




-----
Mike Hammett
Intelligent Computing Solutions

Midwest Internet Exchange

The Brothers WISP

----- Original Message -----

From: "Mike Hammett" <nanog@ics-il.net>
To: "NANOG" <nanog@nanog.org>
Sent: Monday, January 15, 2024 8:08:25 AM
Subject: "Hypothetical" Datacenter Overheating


Let's say that hypothetically, a datacenter you're in had a cooling failure and escalated to an average of 120 degrees before mitigations started having an effect. What are normal QA procedures on your behalf? What is the facility likely to be doing? What should be expected in the aftermath?




-----
Mike Hammett
Intelligent Computing Solutions

Midwest Internet Exchange

The Brothers WISP

Something worth a thought is that as much as devices don't like being
too hot they also don't like to have their temperature change too
quickly. Parts can expand/shrink variably depending on their
composition.

A rule of thumb is a few degrees per hour change but YMMV, depends on
the equipment. Sometimes manufacturer's specs include this.

Throwing open the windows on a winter day to try to rapidly bring the
room down to a "normal" temperature may do more harm than good.

It might be worthwhile figuring out what is reasonable in advance with
buy-in rather than in a panic because, from personal experience,
someone will be screaming in your ear JUST OPEN ALL THE WINDOWS
WHADDYA STUPID?

On January 15, 2024 at 09:23 clayton@MNSi.Net (Clayton Zekelman) wrote:
>
>
>
> At 09:08 AM 2024-01-15, Mike Hammett wrote:
> >Let's say that hypothetically, a datacenter you're in had a cooling
> >failure and escalated to an average of 120 degrees before
> >mitigations started having an effect. What are normal QA procedures
> >on your behalf? What is the facility likely to be doing?
> >What should be expected in the aftermath?
>
> One would hope they would have had disaster recovery plans to bring
> in outside cold air, and have executed on it quickly, rather than
> hoping the chillers got repaired.
>
> All our owned facilities have large outside air intakes, automatic
> dampers and air mixing chambers in case of mechanical cooling
> failure, because cooling systems are often not designed to run well
> in extreme cold. All of these can be manually run incase of controls
> failure, but people tell me I'm a little obsessive over backup plans
> for backup plans.
>
> You will start to see premature failure of equipment over the coming
> weeks/months/years.
>
> Coincidentally, we have some gear in a data centre in the Chicago
> area that is experiencing that sort of issue right now... :-(
>
>
>

--
-Barry Shein

Software Tool & Die | bzs@TheWorld.com | http://www.TheWorld.com
Purveyors to the Trade | Voice: +1 617-STD-WRLD | 800-THE-WRLD
The World: Since 1989 | A Public Information Utility | *oo*

On Tue, 16 Jan 2024 at 08:51, <bzs@theworld.com> wrote:

> A rule of thumb is a few degrees per hour change but YMMV, depends on
> the equipment. Sometimes manufacturer's specs include this.

Is this common sense, or do you have reference to this, like paper
showing at what temperature change at what rate occurs what damage?

I regularly bring fine electronics, say iPhone, through significant
temperature gradients, as do most people who have to live in places
where inside and outside can be wildly different temperatures, with no
particular observable effect. iPhone does go into 'thermometer' mode,
when it overheats though.

Manufacturers, say Juniper and Cisco describe humidity, storage and
operating temperatures, but do not define temperature change rate.
Does NEBS have an opinion on this, or is this just a common case of
yours?

--
++ytti

Good thing there are no windows at this “hypothetical” location :)

> On Jan 16, 2024, at 1:51?AM, bzs@theworld.com wrote:
>
> ?
> Something worth a thought is that as much as devices don't like being
> too hot they also don't like to have their temperature change too
> quickly. Parts can expand/shrink variably depending on their
> composition.
>
> A rule of thumb is a few degrees per hour change but YMMV, depends on
> the equipment. Sometimes manufacturer's specs include this.
>
> Throwing open the windows on a winter day to try to rapidly bring the
> room down to a "normal" temperature may do more harm than good.
>
> It might be worthwhile figuring out what is reasonable in advance with
> buy-in rather than in a panic because, from personal experience,
> someone will be screaming in your ear JUST OPEN ALL THE WINDOWS
> WHADDYA STUPID?
>
>> On January 15, 2024 at 09:23 clayton@MNSi.Net (Clayton Zekelman) wrote:
>>
>>
>>
>> At 09:08 AM 2024-01-15, Mike Hammett wrote:
>>> Let's say that hypothetically, a datacenter you're in had a cooling
>>> failure and escalated to an average of 120 degrees before
>>> mitigations started having an effect. What are normal QA procedures
>>> on your behalf? What is the facility likely to be doing?
>>> What should be expected in the aftermath?
>>
>> One would hope they would have had disaster recovery plans to bring
>> in outside cold air, and have executed on it quickly, rather than
>> hoping the chillers got repaired.
>>
>> All our owned facilities have large outside air intakes, automatic
>> dampers and air mixing chambers in case of mechanical cooling
>> failure, because cooling systems are often not designed to run well
>> in extreme cold. All of these can be manually run incase of controls
>> failure, but people tell me I'm a little obsessive over backup plans
>> for backup plans.
>>
>> You will start to see premature failure of equipment over the coming
>> weeks/months/years.
>>
>> Coincidentally, we have some gear in a data centre in the Chicago
>> area that is experiencing that sort of issue right now... :-(
>>
>>
>>
>
> --
> -Barry Shein
>
> Software Tool & Die | bzs@TheWorld.com | http://www.TheWorld.com
> Purveyors to the Trade | Voice: +1 617-STD-WRLD | 800-THE-WRLD
> The World: Since 1989 | A Public Information Utility | *oo*

On Mon, Jan 15, 2024 at 11:08?PM Saku Ytti <saku@ytti.fi> wrote:
> On Tue, 16 Jan 2024 at 08:51, <bzs@theworld.com> wrote:
> > A rule of thumb is a few degrees per hour change but YMMV, depends on
> > the equipment. Sometimes manufacturer's specs include this.
>
> Is this common sense, or do you have reference to this, like paper
> showing at what temperature change at what rate occurs what damage?

It's uncommon sense.

You have a computer room humidified to 40% and you inject cold air
below the dew point. The surfaces in the room will get wet.

See also: https://en.wikipedia.org/wiki/Thermal_stress

Regards,
Bill Herrin

--
William Herrin
bill@herrin.us
https://bill.herrin.us/

On 1/15/24 23:11, Martin Hannigan wrote:
>
>
> On Mon, Jan 15, 2024 at 4:10?PM Jay Hennigan <jay@west.net
> <mailto:jay@west.net>> wrote:
>
> On 1/15/24 10:37, Pennington, Scott wrote:
> > yes but.... it has been -8 in Chicago plenty of times before this.
> >   Very interested in root cause...
>
> Absolutely. My point was that claiming "Global warming" isn't going to
> fly as an excuse.
>
>
> +1
>
> Is their design N+1?
>
> https://www.equinix.com/data-centers/americas-colocation/united-states-colocation/chicago-data-centers/ch1 <https://www.equinix.com/data-centers/americas-colocation/united-states-colocation/chicago-data-centers/ch1>
>
> We're not smashing temp records in Chicago. At least it doesn't seem so
> when you look across historical data:
>
> https://www.weather.gov/lot/Chicago_Temperature_Records
> <https://www.weather.gov/lot/Chicago_Temperature_Records>
>
> HTH,
>
> -M<

I was at that "hypothetical" location once when it was -17º F ...

No issues then ...

I really have to wonder how six chillers all failed at once.

On Tue, 16 Jan 2024 at 11:00, William Herrin <bill@herrin.us> wrote:

> You have a computer room humidified to 40% and you inject cold air
> below the dew point. The surfaces in the room will get wet.

I think humidity and condensation is well understood and indeed
documented but by NEBS and vendors as verboten.

I am more interested in temperature changes when not condensating and
causing water damage. Like we could theorise, some soldering will
expand/contract too fast, breaking or various other types of scenarios
one might guess without context, and indeed electronics often have to
experience large temperature gradients and appear to survive.
When you turn these things on, various parts rapidly heat from ambient
to 80-90c. So I have some doubts if this is actually a problem you
need to consider, in absence of condensation.

--
++ytti

On 16/01/2024 at 10:50:13?PM, Saku Ytti <saku@ytti.fi> wrote:

> On Tue, 16 Jan 2024 at 11:00, William Herrin <bill@herrin.us> wrote:
>
> You have a computer room humidified to 40% and you inject cold air
>
> below the dew point. The surfaces in the room will get wet.
>
>
> I think humidity and condensation is well understood and indeed
> documented but by NEBS and vendors as verboten.
>
> I am more interested in temperature changes when not condensating and
> causing water damage. Like we could theorise, some soldering will
> expand/contract too fast, breaking or various other types of scenarios
> one might guess without context, and indeed electronics often have to
> experience large temperature gradients and appear to survive.
> When you turn these things on, various parts rapidly heat from ambient
> to 80-90c. So I have some doubts if this is actually a problem you
> need to consider, in absence of condensation.
>

Here’s some manufacturer specs:

https://www.dell.com/support/manuals/en-nz/poweredge-r6515/per6515_ts_pub/environmental-specifications?guid=guid-debd273c-0dc8-40d8-abbc-be059a0ce59c&lang=en-us

3rd section, “Maximum temperature gradient”.

From memory, the management cards alarm when the gradient is exceeded, too.

--
Nathan Ward