Alan Robertson wrote:
>
> Lars Marowsky-Bree wrote:
> >
> > On 1999-12-24T14:57:11,
> > Alan Robertson <alanr@bell-labs.com> said:
> >
> > > You're right that this has been discussed on the list before. There is no
> > > known solution (except to use ntp and don't jump the clock).
> >
> > There is a known solution, which I have mentioned repeatedly. System uptime
> > (/proc/uptime) should be used as the local reference time, this would
> > eliminate the problem.
>
> The times(2) call is the "right" way to get system uptime.
>
> > > The problem is that you can't set an alarm(2) call based on uptime clock.
> > > The kernel keeps such a clock, but a user process can't do the equivalent of
> > > alarm(2) based on it. The fact that it was suggested that I do it doesn't
> > > mean that there's a way to do it from user land. Stephen thought you could
> > > do it from the kernel.
> >
> > alarm(2) is a relative time. You ask alarm(2) to go off 1, 2 seconds in the
> > future, and after the signal has been delivered, check the system uptime if it
> > really is time for us to wake up, otherwise sleep a little bit longer.
>
> Sending out heartbeats too often isn't a problem at all.
>
> > After all, alarm(2) is only guaranteing that _at least_ the specified time has
> > passed. This we can easily check.
> >
> > Did I miss something?
>
> In the presence of time jumps, it doesn't guarantee anything at all. It could
> be 3 days (or 3 years) before you get notified. If that happens, you've had a
> partitioned cluster for 3 days (or 3 years) before you notice.
>
> However, in a working cluster, I just realized that there *is* a way out. In a
> working cluster, we get messages every so often. When we get a message, I can
> check the lbolt time to see if it's time to send a new message, independently of
> whether we've gotten a SIGALRM recently or not. In fact, that's all I need to
> do.
> It will involve rewriting a little code, but it should be pretty easy.
>
> If I'm receiving messages from other cluster members, I can still be in control
> of getting alarms. I can also look at the time of day, and see if it's gone
> backwards. If it has, I can cancel the alarm, and issue a new one.
>
> If I'm not getting messages from other cluster members, it probably doesn't
> matter if my SIGALRM never comes :-) It has the possibility of deadlock if you
> jump the clocks forward on all the machines at once, assuming all their
> heartbeat times are synchronized. This is a huge improvement over what it does
> now.
>
> Of course, this is a pretty trivial extension over what Lars said (looked at in
> the context of the code). I'll fix it (probably tonight).
It's fixed. It was a tiny bit more complicated than described above, but still
easy.
-- Alan Robertson
alanr@bell-labs.com
>
> Lars Marowsky-Bree wrote:
> >
> > On 1999-12-24T14:57:11,
> > Alan Robertson <alanr@bell-labs.com> said:
> >
> > > You're right that this has been discussed on the list before. There is no
> > > known solution (except to use ntp and don't jump the clock).
> >
> > There is a known solution, which I have mentioned repeatedly. System uptime
> > (/proc/uptime) should be used as the local reference time, this would
> > eliminate the problem.
>
> The times(2) call is the "right" way to get system uptime.
>
> > > The problem is that you can't set an alarm(2) call based on uptime clock.
> > > The kernel keeps such a clock, but a user process can't do the equivalent of
> > > alarm(2) based on it. The fact that it was suggested that I do it doesn't
> > > mean that there's a way to do it from user land. Stephen thought you could
> > > do it from the kernel.
> >
> > alarm(2) is a relative time. You ask alarm(2) to go off 1, 2 seconds in the
> > future, and after the signal has been delivered, check the system uptime if it
> > really is time for us to wake up, otherwise sleep a little bit longer.
>
> Sending out heartbeats too often isn't a problem at all.
>
> > After all, alarm(2) is only guaranteing that _at least_ the specified time has
> > passed. This we can easily check.
> >
> > Did I miss something?
>
> In the presence of time jumps, it doesn't guarantee anything at all. It could
> be 3 days (or 3 years) before you get notified. If that happens, you've had a
> partitioned cluster for 3 days (or 3 years) before you notice.
>
> However, in a working cluster, I just realized that there *is* a way out. In a
> working cluster, we get messages every so often. When we get a message, I can
> check the lbolt time to see if it's time to send a new message, independently of
> whether we've gotten a SIGALRM recently or not. In fact, that's all I need to
> do.
> It will involve rewriting a little code, but it should be pretty easy.
>
> If I'm receiving messages from other cluster members, I can still be in control
> of getting alarms. I can also look at the time of day, and see if it's gone
> backwards. If it has, I can cancel the alarm, and issue a new one.
>
> If I'm not getting messages from other cluster members, it probably doesn't
> matter if my SIGALRM never comes :-) It has the possibility of deadlock if you
> jump the clocks forward on all the machines at once, assuming all their
> heartbeat times are synchronized. This is a huge improvement over what it does
> now.
>
> Of course, this is a pretty trivial extension over what Lars said (looked at in
> the context of the code). I'll fix it (probably tonight).
It's fixed. It was a tiny bit more complicated than described above, but still
easy.
-- Alan Robertson
alanr@bell-labs.com