Mailing List Archive

On 1/17/24 20:06, Tom Beecher wrote:
>
> If these chillers are connected to BACnet or similar network, then
> I wouldn't rule out the possibility of an attack.
>
>
> Don't insinuate something like this without evidence. Completely
> unreasonable and inappropriate.
>
I wasn't meaning to insinuate anything; it's as much of a reasonable
possibility as any other these days.

Perhaps I should have worded it differently: "if my small data centers'
chillers were connected to some building management network such as
BACnet and all of them went down concurrently I would be investigating
my building management network for signs of intrusion in addition to
checking other items, such as shared points of failure in things like
chilled water pumps, electrical supply, emergency shut-off circuits,
chiller/closed-loop configurations for various temperature, pressure,
and flow set points, etc."? Bit more wordy, but doesn't have the same
implication.? But I would think it unreasonable, if I were to find
myself in this situation in my own operations, to rule any possibility
out that can explain simultaneous shutdowns.

And this week we did have a chiller go out on freeze warning, but the DC
temp never made it quite up to 80F before the temperature raised back
into double digits and the chiller restarted.

On 1/15/24 10:14, sronan@ronan-online.com wrote:
> I’m more interested in how you lose six chillers all at once.
>
According to a post on a support forum for one of the clients in that
space: "We understand the issue is due to snow on the roof affecting the
cooling equipment."

Never overlook the simplest single points of failure.  Snow on cooling
tower fan blades....failed fan motors are possible or even likely at
that point.  Assuming the airflow won't be clogged; conceptually much
like the issue in having multiple providers for redundancy but they're
all in the same cable or conduit.

I'm actually referring to something like, I've not yet had a system
where they have made sense, I mostly deal with either places where I
have no say in the hvac or very small server rooms, but I've thought
these were an interesting concept since I first saw them years ago.

https://www.chiltrix.com/server-room-chiller.html

quoting from the page:

The Chiltrix SE Server Room Edition adds a "free cooling" option to CX34.

Server rooms need cooling all year, even when it is cold outside. If you
operate in a northern area with cold winters, this option is for you.

When outdoor temperatures drop below 38F, the CX34 glycol-water loop is
automatically extended through a special water-to-air heat exchanger to
harvest outdoor cold ambient conditions to pre-cool the glycol-water
loop so that the CX34 variable speed compressor can drop to a very slow
speed and consume less power. This can save about 50% off of it's
already low power consumption without lowering capacity.

At and below 28F, the CX34 chiller with Free Cooling SE add-on will turn
off the compressor entirely and still be able to maintain its rated
cooling capacity using only the variable speed pump and fan motors. At
this point, the CX34 achieves a COP of of >41 and EER of >141.

Enjoy the savings of 2 tons of cooling for less than 75 watts. The
colder it gets, the less water flow rate is needed, allowing the VSD
pump power draw to drop under 20 watts.

Depending on location, for some customers free cooling mode can be
active up to 3 months per year during the daytime and up to 5 months per
year at night.

On 1/17/2024 3:10 PM, Izaac wrote:
> On Wed, Jan 17, 2024 at 12:07:42AM -0500, Glenn McGurrin via NANOG wrote:
>> Free air cooling loops maybe? (Not direct free air cooling with air
>> exchange, the version with something much like an air handler outside with a
>> coil and an fan running cold outside air over the coil with the water/glycol
>> that would normally be the loop off of the chiller) the primary use of them
>> is cost savings by using less energy to cool when it's fairly cold out, but
>> it can also prevent low temperature issues on compressors by not running
>> them when it's cold. I'd expect it would not require the same sort of
>> facade changes as it could be on the roof and depending only need
>> water/glycol lines into the space, depending on cooling tower vs air cooled
>> and chiller location it could also potentially use the same piping (which I
>> think is the traditional use).
>
> You're looking for these: https://en.wikipedia.org/wiki/Thermal_wheel
>
> Basically, an aluminum honeycomb wheel. One half of its housing is an
> air duct "outside" while the other half is an air duct that's "inside."
> Cold outside air blows through the straws and cools the metal. Wheel
> rotates slowly. That straw is now "inside." Inside air blows through
> it and deposits heat onto the metal. Turn turn turn.
>
> A surprisingly effective way to lower heating/cooling costs. Basically
> "free," as you just need to turn it on the bearing. Do you get deposits
> in the comb? Yes, if you don't filter properly. Do you get
> condensation in the comb? Yeah. Treat it with desiccants.
>

----- Original Message -----
> From: "Tom Beecher" <beecher@beecher.cc>
> To: "Lamar Owen" <lowen@pari.edu>
> Cc: nanog@nanog.org
> Sent: Wednesday, January 17, 2024 8:06:07 PM
> Subject: Re: "Hypothetical" Datacenter Overheating

>> If these chillers are connected to BACnet or similar network, then I
>> wouldn't rule out the possibility of an attack.
>
> Don't insinuate something like this without evidence. Completely
> unreasonable and inappropriate.

WADR, horsecrap.

It's certainly one of many possible root causes which someone doing an
AAR on an event like this should be thinking about, and looking for in their
evaluation of the data they see.

He didn't *accuse* anyone, which would be out of bounds.

Cheers,
-- jra
--
Jay R. Ashworth Baylink jra@baylink.com
Designer The Things I Think RFC 2100
Ashworth & Associates http://www.bcp38.info 2000 Land Rover DII
St Petersburg FL USA BCP38: Ask For It By Name! +1 727 647 1274

>
> t's certainly one of many possible root causes which someone doing an
> AAR on an event like this should be thinking about, and looking for in
> their
> evaluation of the data they see.
>

And I'm sure they are and will.

By the time that post was made, the vendor had shared multiple updates
about what the actual cause seemed to be, which were very plausible. An
unaffiliated 3rd party stating 'maybe an attack!' when there has been no
observation or information shared that even remotely points to that simply
spreads FUD for no reason.

I respectfully disagree.



On Sun, Jan 21, 2024 at 1:22?AM Jay R. Ashworth <jra@baylink.com> wrote:

> ----- Original Message -----
> > From: "Tom Beecher" <beecher@beecher.cc>
> > To: "Lamar Owen" <lowen@pari.edu>
> > Cc: nanog@nanog.org
> > Sent: Wednesday, January 17, 2024 8:06:07 PM
> > Subject: Re: "Hypothetical" Datacenter Overheating
>
> >> If these chillers are connected to BACnet or similar network, then I
> >> wouldn't rule out the possibility of an attack.
> >
> > Don't insinuate something like this without evidence. Completely
> > unreasonable and inappropriate.
>
> WADR, horsecrap.
>
> It's certainly one of many possible root causes which someone doing an
> AAR on an event like this should be thinking about, and looking for in
> their
> evaluation of the data they see.
>
> He didn't *accuse* anyone, which would be out of bounds.
>
> Cheers,
> -- jra
> --
> Jay R. Ashworth Baylink
> jra@baylink.com
> Designer The Things I Think RFC
> 2100
> Ashworth & Associates http://www.bcp38.info 2000 Land
> Rover DII
> St Petersburg FL USA BCP38: Ask For It By Name! +1 727 647
> 1274
>

----- Original Message -----
> From: "Tom Beecher" <beecher@beecher.cc>

>> It's certainly one of many possible root causes which someone doing an
>> AAR on an event like this should be thinking about, and looking for in
>> their evaluation of the data they see.
>
> And I'm sure they are and will.
>
> By the time that post was made, the vendor had shared multiple updates
> about what the actual cause seemed to be, which were very plausible. An
> unaffiliated 3rd party stating 'maybe an attack!' when there has been no
> observation or information shared that even remotely points to that simply
> spreads FUD for no reason.

I didn't see any of them in the thread, which was the only thing I was paying
attention to, so those are fact not in evidence to *me*.

I didn't see an exclamation point in his comment, which seemed relatively
measured to me.

Cheers,
-- jra
--
Jay R. Ashworth Baylink jra@baylink.com
Designer The Things I Think RFC 2100
Ashworth & Associates http://www.bcp38.info 2000 Land Rover DII
St Petersburg FL USA BCP38: Ask For It By Name! +1 727 647 1274