Hi!
I've been working on the heartbeat API code. It has come along nicely, and
everything appears to work at this point. I'm going to tie up some loose ends
and then put out a point release with it in a week or so.
Here's some sample output from the test program. I'll annotate it where it
seems interesting.
> alanr@kathyamy:~/linux-ha/heartbeat > ./api_test
> PID=13295
> Signing in with heartbeat
> Setting message filter mode
Ask heartbeat what nodes are in the cluster...
> Starting node walk
> Cluster node: kathyamy: status: active
Ask it what interfaces it can talk to kathyamy with:
> node kathyamy: intf: eth0 ifstatus: up
> node kathyamy: intf: /dev/ttyS0 ifstatus: dead
(note: not hooked up)
> Cluster node: amymarie.dr.lucent.com: status: active
Ditto for amymarie...
> node amymarie.dr.lucent.com: intf: eth0 ifstatus: up
> node amymarie.dr.lucent.com: intf: /dev/ttyS0 ifstatus: dead
> Setting message signal
Specify what signal I want to receive when a new message comes in.
> Waiting for messages...
I stopped heartbeat on amymarie...
> Status update: Node amymarie.dr.lucent.com now has status dead
> Link Status update: Link amymarie.dr.lucent.com/eth0 now has status dead
> Got a message of type [resource] from [kathyamy]
> Got a message of type [resource] from [kathyamy]
> Got a message of type [ip-request] from [kathyamy]
The first two messages are from callbacks. The last 3 were received as general
messages...
I restarted heartbeat on amymarie...
> Link Status update: Link amymarie.dr.lucent.com/eth0 now has status up
> Status update: Node amymarie.dr.lucent.com now has status up
> Status update: Node amymarie.dr.lucent.com now has status active
> Got a message of type [starting] from [amymarie.dr.lucent.com]
> Got a message of type [resource] from [kathyamy]
> Got a message of type [resource] from [amymarie.dr.lucent.com]
So, this all works pretty much like it should. All the various callbacks seem
to be working, etc.
The implementation got delayed because we decided that the original security
model was inadequate.
Here's a little explanation of how it works now, and what you can do with it...
The following description is for "casual" users. I'll explain more later...
There is a registration FIFO which is writable by group hb_api. When you want
to register, you create two FIFOs for communicating with heartbeat, and write
your registration request on the registration FIFO. The registration message
contains your PID. Heartbeat then verifies that your comm FIFOs are only
writable/readable by the UID that goes with the PID you claim to be. The casual
FIFO directory is verified for proper permissions and ownership at this time.
If this is not the case, the registration fails.
At this point, any requests that come from that FIFO are assumed to be going to
the PID you claim to be. At the very least, you know that they are coming from
a PID with the same UID as the one you claim to be (or root). This is
important, because you can cause heartbeat to send that process a signal. This
signal could be fatal to the process if it isn't prepared for it. However, if
you're the same uid as the process, then you could have sent that signal
directly, so even if you're masquerading as some other pid with the same uid,
you can't gain any new privs.
When you deregister, heartbeat removes your FIFOs (if you die unexpectedly).
Permissions and ownerships on the FIFOs are checked on every message.
Casual clients can monitor events, but cannot send messages to the cluster.
If you want to send messages to the cluster, you have to be a named client.
Named clients are similar, except that the FIFOs go in a different directory
which is NOT writable by the group hb_api. It must be only writable by root.
The FIFOs in it must be created by root in advance, and the permissions and
ownerships set according to the desires of the admin.
When a named client exits, no one removes the FIFOs.
The result of this is that the name is associated with a particular user id. If
the ownerships are administered consistently across the cluster, then all named
clients are guaranteed to have the same user id.
A named client can send messages to other named clients on other nodes in the
cluster which have registered with the same name. These messages are guaranteed
to be delivered to all registered clients on all machines in the cluster.
However, they are not guaranteed to be delivered in order, nor are they
guaranteed to be delivered simultaneously on every node.
....
The summary of the potentially bad properties of this security model are below:
You CAN register as another pid which has the same uid, and no one will be the
wiser.
You CAN cause heartbeat to send that pid signals whenever a message arrives.
[.however, you could have just sent it directly if this were the case]
You can attempt to register as a pid which has a different uid than yours if you
belong to the hb_api group.
The attempt will fail for casual clients. It can be made to succeed
for non-casual clients. However, you can't affect the processes because
you can't send any messages as them, nor can you receive any from them...
Certain services are only available to root (packet sniffing). I believe that
this is monitored correctly, and only root can sniff heartbeat packets using
this interface.
A few more details:
The uid of pid x is calculated by performing a stat on /proc/x, and looking at
the owner of that directory. This is assumed to be accurate.
The casual FIFO directory is sticky. Heartbeat is VERY particular about
permissions on the directories and FIFOs, and refuses to work if any unnecessary
privileges have been granted on them. As long as you belong to the appropriate
group, you can attach to the heartbeat API. You have to be a specific uid (and
also belong to the appropriate group) in order to use the named interface for a
given name.
-- Alan Robertson
alanr@suse.com
I've been working on the heartbeat API code. It has come along nicely, and
everything appears to work at this point. I'm going to tie up some loose ends
and then put out a point release with it in a week or so.
Here's some sample output from the test program. I'll annotate it where it
seems interesting.
> alanr@kathyamy:~/linux-ha/heartbeat > ./api_test
> PID=13295
> Signing in with heartbeat
> Setting message filter mode
Ask heartbeat what nodes are in the cluster...
> Starting node walk
> Cluster node: kathyamy: status: active
Ask it what interfaces it can talk to kathyamy with:
> node kathyamy: intf: eth0 ifstatus: up
> node kathyamy: intf: /dev/ttyS0 ifstatus: dead
(note: not hooked up)
> Cluster node: amymarie.dr.lucent.com: status: active
Ditto for amymarie...
> node amymarie.dr.lucent.com: intf: eth0 ifstatus: up
> node amymarie.dr.lucent.com: intf: /dev/ttyS0 ifstatus: dead
> Setting message signal
Specify what signal I want to receive when a new message comes in.
> Waiting for messages...
I stopped heartbeat on amymarie...
> Status update: Node amymarie.dr.lucent.com now has status dead
> Link Status update: Link amymarie.dr.lucent.com/eth0 now has status dead
> Got a message of type [resource] from [kathyamy]
> Got a message of type [resource] from [kathyamy]
> Got a message of type [ip-request] from [kathyamy]
The first two messages are from callbacks. The last 3 were received as general
messages...
I restarted heartbeat on amymarie...
> Link Status update: Link amymarie.dr.lucent.com/eth0 now has status up
> Status update: Node amymarie.dr.lucent.com now has status up
> Status update: Node amymarie.dr.lucent.com now has status active
> Got a message of type [starting] from [amymarie.dr.lucent.com]
> Got a message of type [resource] from [kathyamy]
> Got a message of type [resource] from [amymarie.dr.lucent.com]
So, this all works pretty much like it should. All the various callbacks seem
to be working, etc.
The implementation got delayed because we decided that the original security
model was inadequate.
Here's a little explanation of how it works now, and what you can do with it...
The following description is for "casual" users. I'll explain more later...
There is a registration FIFO which is writable by group hb_api. When you want
to register, you create two FIFOs for communicating with heartbeat, and write
your registration request on the registration FIFO. The registration message
contains your PID. Heartbeat then verifies that your comm FIFOs are only
writable/readable by the UID that goes with the PID you claim to be. The casual
FIFO directory is verified for proper permissions and ownership at this time.
If this is not the case, the registration fails.
At this point, any requests that come from that FIFO are assumed to be going to
the PID you claim to be. At the very least, you know that they are coming from
a PID with the same UID as the one you claim to be (or root). This is
important, because you can cause heartbeat to send that process a signal. This
signal could be fatal to the process if it isn't prepared for it. However, if
you're the same uid as the process, then you could have sent that signal
directly, so even if you're masquerading as some other pid with the same uid,
you can't gain any new privs.
When you deregister, heartbeat removes your FIFOs (if you die unexpectedly).
Permissions and ownerships on the FIFOs are checked on every message.
Casual clients can monitor events, but cannot send messages to the cluster.
If you want to send messages to the cluster, you have to be a named client.
Named clients are similar, except that the FIFOs go in a different directory
which is NOT writable by the group hb_api. It must be only writable by root.
The FIFOs in it must be created by root in advance, and the permissions and
ownerships set according to the desires of the admin.
When a named client exits, no one removes the FIFOs.
The result of this is that the name is associated with a particular user id. If
the ownerships are administered consistently across the cluster, then all named
clients are guaranteed to have the same user id.
A named client can send messages to other named clients on other nodes in the
cluster which have registered with the same name. These messages are guaranteed
to be delivered to all registered clients on all machines in the cluster.
However, they are not guaranteed to be delivered in order, nor are they
guaranteed to be delivered simultaneously on every node.
....
The summary of the potentially bad properties of this security model are below:
You CAN register as another pid which has the same uid, and no one will be the
wiser.
You CAN cause heartbeat to send that pid signals whenever a message arrives.
[.however, you could have just sent it directly if this were the case]
You can attempt to register as a pid which has a different uid than yours if you
belong to the hb_api group.
The attempt will fail for casual clients. It can be made to succeed
for non-casual clients. However, you can't affect the processes because
you can't send any messages as them, nor can you receive any from them...
Certain services are only available to root (packet sniffing). I believe that
this is monitored correctly, and only root can sniff heartbeat packets using
this interface.
A few more details:
The uid of pid x is calculated by performing a stat on /proc/x, and looking at
the owner of that directory. This is assumed to be accurate.
The casual FIFO directory is sticky. Heartbeat is VERY particular about
permissions on the directories and FIFOs, and refuses to work if any unnecessary
privileges have been granted on them. As long as you belong to the appropriate
group, you can attach to the heartbeat API. You have to be a specific uid (and
also belong to the appropriate group) in order to use the named interface for a
given name.
-- Alan Robertson
alanr@suse.com