Mailing List Archive

Alan Robertson wrote:

> Steve Beattie wrote:
> > Which begs me to ask: what is the security model
> > behind the authentication scheme? What sort of threats are you
> > attempting to prevent by using it?
>
> Members of the cluster are given significant privileges with respect to each
> other. In the current implementation, one cluster member can say to another
> "give up this resource" and they will. This is immediate denial of service, and
> opens the door wide for the hacker to masquerade as a cluster member.
>
> The authentication is used for two reasons:
>
> 1) To prevent joe-hacker from telling our cluster to do
> something we don't want it to do through the wire.
>
> 2) To detect packets corrupted by "normal" network problems.
> so that we don't try and act on them.
>
> We run on IP media, and on raw serial ports. The raw serial ports have no
> protection against dropped/mangled characters except for the authentication.
>
> Basically, we've raised the bar for an intruder to gain control of the cluster
> through the software we've written. And, we believe we've raised it a good bit
> above the highly-vulnerable "I trust everything" level it started with. Not
> only that, but the serial ports needed it for checksums :-)

I'm still a little vauge on the threat model that you're considering. Is the
authentication to prevent an outsider from attacking one of the nodes in the
cluster? To prevent someone from masquerading as one of the nodes in the cluster?
Or to prevent the attacker from leveraging a crack into one of the cluster nodes
into total control of the cluster? They're all important, but crypto applies to
each case in different ways ...

Crispin
-----
Crispin Cowan, CTO, WireX Communications, Inc. http://wirex.com
Free Hardened Linux Distribution: http://immunix.org

Crispin Cowan wrote:
>
> Alan Robertson wrote:
>
> > Steve Beattie wrote:
> > > Which begs me to ask: what is the security model
> > > behind the authentication scheme? What sort of threats are you
> > > attempting to prevent by using it?
> >
> > Members of the cluster are given significant privileges with respect to each
> > other. In the current implementation, one cluster member can say to another
> > "give up this resource" and they will. This is immediate denial of service, and
> > opens the door wide for the hacker to masquerade as a cluster member.
> >
> > The authentication is used for two reasons:
> >
> > 1) To prevent joe-hacker from telling our cluster to do
> > something we don't want it to do through the wire.
> >
> > 2) To detect packets corrupted by "normal" network problems.
> > so that we don't try and act on them.
> >
> > We run on IP media, and on raw serial ports. The raw serial ports have no
> > protection against dropped/mangled characters except for the authentication.
> >
> > Basically, we've raised the bar for an intruder to gain control of the cluster
> > through the software we've written. And, we believe we've raised it a good bit
> > above the highly-vulnerable "I trust everything" level it started with. Not
> > only that, but the serial ports needed it for checksums :-)
>
> I'm still a little vauge on the threat model that you're considering. Is the
> authentication to prevent an outsider from attacking one of the nodes in the
> cluster? To prevent someone from masquerading as one of the nodes in the cluster?
> Or to prevent the attacker from leveraging a crack into one of the cluster nodes
> into total control of the cluster? They're all important, but crypto applies to
> each case in different ways ...

The authentication was added to prevent an outside computer from being
accepted as a cluster member by the cluster itself, and thereby obtaining the
rights and privileges that go with that. It is believed that it will also
make it somewhat more difficult for an outside computer to disrupt
intracluster communications.

This should help from opening new holes that an attacker can use to leverage
into total control of the cluster.

It will not help keep an outside computer from masquerading as a cluster
member to clients, since they don't speak the heartbeat protocol, and aren't
privvy to the secret used to authenticate packets. It's all about
intracluster protocols, and keeping outsiders from joining the cluster from
the point of view of the cluster itself.

I haven't been worried about things I didn't create (like old holes).

I'm not a security expert. I was aware of some significant holes, so I asked
Neal McBurnett (a friend who is much more knowledgeable about security
matters) to help us keep from opening large new holes. I've CCed him on this
mail.

As a result, I still may not be answering your question.

Thanks for your patience,


-- Alan Robertson
alanr@bell-labs.com

Alan Robertson wrote:

> The authentication was added to prevent an outside computer from being
> accepted as a cluster member by the cluster itself, and thereby obtaining the
> rights and privileges that go with that. It is believed that it will also
> make it somewhat more difficult for an outside computer to disrupt
> intracluster communications.

Thanks! That does clarify the intent.


> I haven't been worried about things I didn't create (like old holes).

A very reasonable philosophy for a project who's main aim is not security. All too
many projects don't even do that :-(

So it's my understanding that the nodes keep sending each other "I know a secret, and
here's a hash of the secret" packets, and the "secret" in question is a key shared
among all the nodes. There's something interesting going on with the way you're using
crypto hashes, but I haven't thought about it very carefully.

Crispin
-----
Crispin Cowan, CTO, WireX Communications, Inc. http://wirex.com
Free Hardened Linux Distribution: http://immunix.org

Crispin Cowan wrote:
>
> Alan Robertson wrote:
>
> > The authentication was added to prevent an outside computer from being
> > accepted as a cluster member by the cluster itself, and thereby obtaining the
> > rights and privileges that go with that. It is believed that it will also
> > make it somewhat more difficult for an outside computer to disrupt
> > intracluster communications.
>
> Thanks! That does clarify the intent.
>
> > I haven't been worried about things I didn't create (like old holes).
>
> A very reasonable philosophy for a project who's main aim is not security. All too
> many projects don't even do that :-(

Linux boxes are like targets with big bull's-eye's on them. The current cluster
protocol is really trusting. A really bad combination if you don't have secure
communications. Any box worth protecting with HA is a box valuable enough to
crack.

> So it's my understanding that the nodes keep sending each other "I know a secret, and
> here's a hash of the secret" packets, and the "secret" in question is a key shared
> among all the nodes. There's something interesting going on with the way you're using
> crypto hashes, but I haven't thought about it very carefully.

The packets are mostly (but not exclusively) heartbeat packets. Each one of
them is signed with the crypto hash, so that we can tell that they are
authentic, in the manner you described. If you crank up the debug (send it
about 5 SIGUSR1's), you can see each packet in the logs, including the
authentication fields. SIGUSR2's lower the debug level.

A nice thing about what we're doing is that we allow several different methods
and different keys. The packets we send on the wire don't indicate directly
which hash method we're using (there's no fixed mapping between the numbers and
the methods), so it seems like it would make it a little more difficult to
crack, especially if you padded out the keys to all look the same in format.
Since this is open source code, you can easily add new ones, or mangle ones you
have for "security through obscurity".

The purpose of allowing several keys in the authkeys file is to make it possible
to smoothly switch to a new key in a continuously running system.

Assume you initially are authenticating on key 1:

You distribute out a new authkeys file to each machine which has key 1 and
a new key 2 both in it. The auth statement at the top still says
auth 1. Go to next step when this one is done on all nodes.

You can now distribute a new authkeys file which has the same keys in it,
but says "auth 2" at the top.
Go to next step when this one is done on all nodes.

Distribute a new authkeys file which has only key 2 in it.
The first key is now repudiated, and is no longer valid.

None of this disrupts the cluster at all. Modify the authkeys file, and send
heartbeat a SIGHUP. New authkeys are in now in effect.

Neal says this is somewhat similar to what NTP does. Myself, just think it's
pretty cool, and it's definitely needed :-)

We also got plain checksums (for physically secure networks) for "free" ;-)

Neal did the research to select the methods for security, freedom from patents
and export restrictions, and Mitja Sarp did all the hard work to make it work.
I acted as architect, and made minor changes in the way the pieces were glued
together. A pretty good team effort!

Special thanks to Neal and Mitja!

-- Alan Robertson
alanr@bell-labs.com

Alan Robertson wrote:

> A nice thing about what we're doing is that we allow several different methods
> and different keys. The packets we send on the wire don't indicate directly
> which hash method we're using (there's no fixed mapping between the numbers and
> the methods), so it seems like it would make it a little more difficult to
> crack, especially if you padded out the keys to all look the same in format.
> Since this is open source code, you can easily add new ones, or mangle ones you
> have for "security through obscurity".

This method is remarkably weak. If you have 4 crypto methods, that's effectively a 2-bit
extension to key length. The main strength of a protocol supporting multiple algorithms is
that if a crypto weakness in one of the hash or cipher algorithms, then administrators can
quickly select an alternate cipher, rather than having to wait for a patch to the
implementation.


> You distribute out a new authkeys file to each machine which has key 1 and
> a new key 2 both in it. The auth statement at the top still says
> auth 1. Go to next step when this one is done on all nodes.

Is there anything to prevent an attacker from sniffing such a packet to snarf the new key
2? This re-keying protocol is a subset of the general problem of key distribution, which
is never easy.

Crispin
-----
Crispin Cowan, CTO, WireX Communications, Inc. http://wirex.com
Free Hardened Linux Distribution: http://immunix.org

Crispin Cowan wrote:
>
> Alan Robertson wrote:
>
> > A nice thing about what we're doing is that we allow several different methods
> > and different keys. The packets we send on the wire don't indicate directly
> > which hash method we're using (there's no fixed mapping between the numbers and
> > the methods), so it seems like it would make it a little more difficult to
> > crack, especially if you padded out the keys to all look the same in format.
> > Since this is open source code, you can easily add new ones, or mangle ones you
> > have for "security through obscurity".
>
> This method is remarkably weak. If you have 4 crypto methods, that's effectively a 2-bit
> extension to key length. The main strength of a protocol supporting multiple algorithms is
> that if a crypto weakness in one of the hash or cipher algorithms, then administrators can
> quickly select an alternate cipher, rather than having to wait for a patch to the
> implementation.

It's certainly no weaker than one method. Also, in the case of a sophisticated
site, they don't necessarily know how many methods the site has at their
disposal. I didn't say this, but the main reason why we put more than one in
was to allow a site to make the implementation cost versus strength tradeoff
themselves, rather than us making it for them. Avoiding arguments is always a
good idea :-). Also, if someone needs a different method, they don't have to
fork the code to do it. It's simply a better design, which we needed it for
smooth key transitions anyway.

> > You distribute out a new authkeys file to each machine which has key 1 and
> > a new key 2 both in it. The auth statement at the top still says
> > auth 1. Go to next step when this one is done on all nodes.
>
> Is there anything to prevent an attacker from sniffing such a packet to snarf the new key
> 2? This re-keying protocol is a subset of the general problem of key distribution, which
> is never easy.

Rekeying isn't something we address yet. This is the first release with any
protection at all. Since we only handle small clusters (2 nodes) at this point,
it is not unreasonable to say "take this floppy...", and go from there. An
administrator could also use ssh, etc. As you said, it's never easy. Secure
key distribution tends to involve encryption, which is heavily regulated here in
the US. Mitja is outside the US, and he could do this... But, I couldn't
distribute it from here. Not the right place to start for a project which isn't
primarily a crypto project.

As we add nodes, and work with LVS clusters (with hundreds of nodes), this needs
to be solved. Applications are being accepted... ;-)

-- Alan Robertson
alanr@bell-labs.com