Mailing List Archive

Philipp,

I am still waiting to know what you thing about recording the number of
block to transmit in order to be able to estimate the time needed for a
full resync.

Every minute, drdb count the number of block it have to transfer and backup
the previous value. We can keep track of three values which must be enough
to estimate the time needed to finish the resync.

The other solution is to save the time every XX block transfered. Perhaps
simpler and cleaner to implement.

We can then record those information number in /proc.

Thomas
--
Thomas Mangin (mailto:thomas.mangin@example.com)
System Administrator (mailto:systems@example.com)
Legend Internet Ltd. (http://www.legend.co.uk:/)
--
The urgent is done, the impossible is on the way, for miracles expect a
small delay


----- Original Message -----
From: Philipp Reisner <philipp@example.com>
To: Ching-Hsien Tsai <ching@example.com>
Cc: DRBD-List <drbd-devel@example.com>
Sent: Sunday, November 05, 2000 7:15 PM
Subject: [DRBD-dev] Re: Is it possible for mounting /dev/nb0 as root?


> * Ching-Hsien Tsai <ching@example.com> [001103 11:32]:
> > Hi:
> >
> > I am experimenting with drbd.
> >
> > Is it possible for mounting /dev/nb0 as root?
> >
> > I had looked up the site and found no relevent info.
> > http://www.complang.tuwien.ac.at/reisner/drbd/
> >
> > Thanks so much for your help. Appreciated.
> >
> > BEST Regards.
> >
> > Ching-Hsien Tsai
> > IDpoints Inc.
> >
>
> Hi,
>
> This is impossible, because only one of the machines can mount
> the FS on top of DRBD at a time.
> -- Maybe when we have support for GFS in DRBD, but this is for
> sure not soon :(
>
> -Philipp
> _______________________________________________
> DRBD-devel mailing list
> DRBD-devel@example.com
> http://lists.sourceforge.net/mailman/listinfo/drbd-devel
>

On Sun, Nov 05, 2000 at 08:15:59PM +0100, Philipp Reisner wrote:
>
> Hi,
>
> This is impossible, because only one of the machines can mount
> the FS on top of DRBD at a time.
> -- Maybe when we have support for GFS in DRBD, but this is for
> sure not soon :(

I have been thinking about some architectural changes to drbd to allow
this sort of thing, and to do a better job of helping users avoid
resyncing the wrong way or not knowing which is the correct volume etc.

One idea was to more strongly separate out the mirroring functionality
vs the block serving functionality. And to move some things, like
resyncing out to userland, since there is not any advantage to haveing
them in kernel space.

-dg

--
David Gould dg@example.com
SuSE, Inc., 580 2cd St. #210, Oakland, CA 94607 510.628.3380
"I personally think Unix is "superior" because on LSD it tastes
like Blue." -- jbarnett

On Tue, Nov 07, 2000 at 06:31:58PM -0800, David Gould wrote:
| I have been thinking about some architectural changes to drbd to allow
| this sort of thing, and to do a better job of helping users avoid
| resyncing the wrong way or not knowing which is the correct volume etc.
...

A quick hack in /etc/rc.d/init.d/drbd can give you some help:

---------------------------cut here--------------------------------
--- drbd Fri Oct 20 17:09:22 2000
+++ drbd2 Wed Nov 8 12:05:30 2000
@@ -80,9 +80,26 @@
RV=0 # this means there were errors in the current device
MINOR_COUNT=`echo $DEVICES | wc --words | sed -e s/\ //g`

+CRASH=$HA_DIR/test_crash
+CRASH_FILE=/var/run/drbd-node_$HOST
+
+
# See how we were called.
case "$1" in
start)
+ # Experimental stuff for testing crash...
+ if [ -e $CRASH ] # Are we looking for crashes?
+ then
+ echo "--> Test crash stuff active..."
+ if [ -e $CRASH_FILE ]
+ then
+ echo "Hey you! It seems like we've suffered a crash..."
+ echo "Do the right thing and then remove $CRASH_FILE"
+ exit 99
+ fi
+ touch $CRASH_FILE
+ fi
+
# use -s in modprobe/rmmod to keep errors in the logs
action "Loading DRBD module" /sbin/modprobe -s drbd \
minor_count=$MINOR_COUNT
@@ -137,6 +154,7 @@
stop)
action "Unloading DRBD module" /sbin/rmmod -s drbd
RETVAL=$?
+ [ -e $CRASH ] && rm $CRASH_FILE
;;
restart)
$0 stop
---------------------------cut here--------------------------------

Maybe it isn't the best place to put this code but... as a proof, it
works well. The idea is simple: Iif we're starting and there's a file called
/var/run/drbd-node_xxxx, where xxxx is the result of `hostname -s`, don't
start drbd 'cause we've crashed... call the admin and let him receive all
the guilt. :)

Luis

--
[ Luis Claudio R. Goncalves lclaudio@example.com ]
[. MSc coming soon -- Conectiva HA Team -- Gospel User -- Linuxer -- :) ]
[. Fault Tolerance - Real-Time - Distributed Systems - IECLB - IS 40:31 ]
[. LateNite Programmer -- Jesus Is The Solid Rock On Which I Stand -- ]

Hello,

I am not sure this crash code is a good idea. I can see two use of drbd and
none of them
needs such code.

1- DRBD and heartbeat

This is up to heartbeat to decide to start or stop drbd so crash recovery
must not be
handled by the drdb script

2- DRBD alone
Have a look to the new cvs code. Here I can see 3 crash case :

a - Slave crash.
The autonegotiate will bring the slave in slave mode and the sync will
continue

b - The master crash
The autonegotiate will bring the master in master mode.
The non transmitted data are lost but nothing can help .. use B/C mode + fs
sync mode !!

c - Both crashes
This is the only problem. If the both server are performing their side test
when they
are both slave they can try to both become master and succed !!

It is the only thing I am planning to change it before the next release. But
I didn't
figured yet the best way to do it as, as far as I know once the two nodes
went master
bringing one slave will NOT solve the problem. The connection is somehow
broken and
you have to unload and reload the module to solve the problem and I am not
even sure
this is enough to fix it !!

But even will the same hardware and software installed their is always a
boot time difference.
So I never saw that, however this is still possible.

I can not see any solution to properly solve this problem at a script level.
Only a "master mode locking" performed by drdb before a real setting of the
node would prevent this.

Suggestion are welcome ..

Thomas
--
Thomas Mangin (mailto:thomas.mangin@example.com)
System Administrator (mailto:systems@example.com)
Legend Internet Ltd. (http://www.legend.co.uk:/)
--
The urgent is done, the impossible is on the way, for miracles expect a
small delay


----- Original Message -----
From: Luis Claudio R.Goncalves <lclaudio@example.com>
To: David Gould <dg@example.com>
Cc: Philipp Reisner <philipp@example.com>; DRBD-List
<drbd-devel@example.com>
Sent: Wednesday, November 08, 2000 2:10 PM
Subject: Re: [DRBD-dev] Re: Is it possible for mounting /dev/nb0 as root?


> On Tue, Nov 07, 2000 at 06:31:58PM -0800, David Gould wrote:
> | I have been thinking about some architectural changes to drbd to allow
> | this sort of thing, and to do a better job of helping users avoid
> | resyncing the wrong way or not knowing which is the correct volume etc.
> ...
>
> A quick hack in /etc/rc.d/init.d/drbd can give you some help:
>
> ---------------------------cut here--------------------------------
> --- drbd Fri Oct 20 17:09:22 2000
> +++ drbd2 Wed Nov 8 12:05:30 2000
> @@ -80,9 +80,26 @@
> RV=0 # this means there were errors in the current device
> MINOR_COUNT=`echo $DEVICES | wc --words | sed -e s/\ file://g`
>
> +CRASH=$HA_DIR/test_crash
> +CRASH_FILE=/var/run/drbd-node_$HOST
> +
> +
> # See how we were called.
> case "$1" in
> start)
> + # Experimental stuff for testing crash...
> + if [ -e $CRASH ] # Are we looking for crashes?
> + then
> + echo "--> Test crash stuff active..."
> + if [ -e $CRASH_FILE ]
> + then
> + echo "Hey you! It seems like we've suffered a
crash..."
> + echo "Do the right thing and then remove
$CRASH_FILE"
> + exit 99
> + fi
> + touch $CRASH_FILE
> + fi
> +
> # use -s in modprobe/rmmod to keep errors in the logs
> action "Loading DRBD module" /sbin/modprobe -s drbd \
> minor_count=$MINOR_COUNT
> @@ -137,6 +154,7 @@
> stop)
> action "Unloading DRBD module" /sbin/rmmod -s drbd
> RETVAL=$?
> + [ -e $CRASH ] && rm $CRASH_FILE
> ;;
> restart)
> $0 stop
> ---------------------------cut here--------------------------------
>
> Maybe it isn't the best place to put this code but... as a proof, it
> works well. The idea is simple: Iif we're starting and there's a file
called
> /var/run/drbd-node_xxxx, where xxxx is the result of `hostname -s`, don't
> start drbd 'cause we've crashed... call the admin and let him receive all
> the guilt. :)
>
> Luis
>
> --
> [ Luis Claudio R. Goncalves lclaudio@example.com ]
> [. MSc coming soon -- Conectiva HA Team -- Gospel User -- Linuxer -- :) ]
> [. Fault Tolerance - Real-Time - Distributed Systems - IECLB - IS 40:31 ]
> [. LateNite Programmer -- Jesus Is The Solid Rock On Which I Stand -- ]
> _______________________________________________
> DRBD-devel mailing list
> DRBD-devel@example.com
> http://lists.sourceforge.net/mailman/listinfo/drbd-devel
>

Hello!

Excuse me for the delay...

On Wed, Nov 08, 2000 at 02:55:36PM -0000, Thomas Mangin wrote:
| Hello,
|
| I am not sure this crash code is a good idea. I can see two use of drbd and
| none of them
| needs such code.

As David Gould was talking about it, I thought that this quick hack
could be a good idea on the subject or a starting point. :)

| 1- DRBD and heartbeat
|
| This is up to heartbeat to decide to start or stop drbd so crash recovery
| must not be
| handled by the drdb script

And heartbeat doesn't do that well right now.
To be a bit more clear, the only case where we do have problems is the
"both nodes crashed!" one. That code only makes sense in that case. Its
purpose is to signal the admin that this node probably crashed and it may
not be a good idea turning it into primary.
I know that we don't want (and right now we can't) solve the multiple
crashes case. But we can advise the sysadmin that something happened...
As soon as the resource manager for heartbeat is done, this problem will
go away, we hope. But as it isn't done, we have to look for a workaround.

Luis :)
--
[ Luis Claudio R. Goncalves lclaudio@example.com ]
[. MSc coming soon -- Conectiva HA Team -- Gospel User -- Linuxer -- :) ]
[. Fault Tolerance - Real-Time - Distributed Systems - IECLB - IS 40:31 ]
[. LateNite Programmer -- Jesus Is The Solid Rock On Which I Stand -- ]

> | This is up to heartbeat to decide to start or stop drbd so crash
recovery
> | must not be
> | handled by the drdb script
>
> And heartbeat doesn't do that well right now.
> To be a bit more clear, the only case where we do have problems is the
> "both nodes crashed!" one. That code only makes sense in that case. Its
> purpose is to signal the admin that this node probably crashed and it may
> not be a good idea turning it into primary.
> I know that we don't want (and right now we can't) solve the multiple
> crashes case. But we can advise the sysadmin that something happened...
> As soon as the resource manager for heartbeat is done, this problem
will
> go away, we hope. But as it isn't done, we have to look for a workaround.

I don't think it is DRBD job to manage this situation.
1 - You are using heartbeat , go and kick ass on their mailing list (that I
am reading)
2 - You are not using heartbeat and you are using the ... what #$!%& name I
gave it ...
mode with autonegotation .. as long as you use a FS mouted with sync it must
be safe.
2 - You want your node to comme back like they was befor last reboot, once
again it must be
safe (when I will have fixed this "race condition").

Thomas

Hello! :)

On Mon, Nov 13, 2000 at 09:43:02PM -0000, Thomas Mangin, Personnal wrote:
| I don't think it is DRBD job to manage this situation.

My dirty, quick and polemical script was originated by this message:

On Tue, Nov 07, 2000 at 06:31:58PM -0800, David Gould wrote:
| I have been thinking about some architectural changes to drbd to allow
| this sort of thing, and to do a better job of helping users avoid
| resyncing the wrong way or not knowing which is the correct volume etc.

I agree with David, we have to help users to avoid problems that come from
crashes. But I also agree with you... this stuff don't need to be done by
drbd even if you're using drbd alone. Chating with Olivé we realized that
this code, or whatever the crash test will be, must be inside of datadisk.

| 1 - You are using heartbeat , go and kick ass on their mailing list (that I
| am reading)

The original script was written to heartbeat. I'm enhancing it to let
heartbeat know that the other node, that's starting right now, has crashed
before - or has stopped in a bad fashion.
Anyway, you can use drbd without heartbeat and, even in the future,
without a cluster manager.

| 2 - You are not using heartbeat and you are using the ... what #$!%& name I
| gave it ...
| mode with autonegotation .. as long as you use a FS mouted with sync it must
| be safe.

If both nodes crashed at the same time, yes, you right. But if that
classical problem where node A crashes, node B does the failover and
updates some data, node B crashes and node A returns, your statement
doesn't make sense. This is the case that we're trying to avoid.

| 2 - You want your node to comme back like they was befor last reboot, once
| again it must be
| safe (when I will have fixed this "race condition").

Hmmm? I can't get the point... could you please elaborate?
I think we're going in the right path (with this thread). Thanks Thomas for
letting me think a bit more about this subject.

Luis

PS: Sometimes my english skills betrays me... so, please, excuse me if I
sounded rude anytime.
--
[ Luis Claudio R. Goncalves lclaudio@example.com ]
[. MSc coming soon -- Conectiva HA Team -- Gospel User -- Linuxer -- :) ]
[. Fault Tolerance - Real-Time - Distributed Systems - IECLB - IS 40:31 ]
[. LateNite Programmer -- Jesus Is The Solid Rock On Which I Stand -- ]

> | 1 - You are using heartbeat , go and kick ass on their mailing list
(that I
> | am reading)
>
> The original script was written to heartbeat. I'm enhancing it to let
> heartbeat know that the other node, that's starting right now, has crashed
> before - or has stopped in a bad fashion.
> Anyway, you can use drbd without heartbeat and, even in the future,
> without a cluster manager.

Yes, the headers still report that ;*)

> | 2 - You are not using heartbeat and you are using the ... what #$!%&
name I
> | gave it ...
> | mode with autonegotation .. as long as you use a FS mouted with sync it
must
> | be safe.
>
> If both nodes crashed at the same time, yes, you right. But if that
> classical problem where node A crashes, node B does the failover and
> updates some data, node B crashes and node A returns, your statement
> doesn't make sense. This is the case that we're trying to avoid.

The autonegociation mode takes avantage of the new proc feature.
drdb report the state of the remote node. This is used to guess how to
setup the local one ..

> | 2 - You want your node to comme back like they was befor last reboot,
once
> | again it must be
> | safe (when I will have fixed this "race condition").
>
> Hmmm? I can't get the point... could you please elaborate?
> I think we're going in the right path (with this thread). Thanks Thomas
for
> letting me think a bit more about this subject.

When you set up a mode with drdbc (slave/master) this is saved.
If your server shutdown in this mode, at it next reboot it will check the
file to come back like it was before ..

> PS: Sometimes my english skills betrays me... so, please, excuse me if I
> sounded rude anytime.

IDEM ...

Thomas

On Tue, Nov 14, 2000 at 02:54:54PM -0000, Thomas Mangin wrote:
| > | 2 - You are not using heartbeat and you are using the ... what #$!%&
| name I
| > | gave it ...
| > | mode with autonegotation .. as long as you use a FS mouted with sync it
| must
| > | be safe.
| >
| > If both nodes crashed at the same time, yes, you right. But if that
| > classical problem where node A crashes, node B does the failover and
| > updates some data, node B crashes and node A returns, your statement
| > doesn't make sense. This is the case that we're trying to avoid.
|
| The autonegociation mode takes avantage of the new proc feature.
| drdb report the state of the remote node. This is used to guess how to
| setup the local one ..

But it doesn't help (unless I'm missing something) in the case stated
above, cause the other node (B, for example) is down when A comes back.
This is the case we're trying to avoid. When you are repaired after a crash
and no other node is avalable... It don't suffices just going back to the state
you were before (master or slave) because you may have missed some data and
thus, be inconsistent or at least outdated.
Sometimes having incomplete, out-of-date and out of sync, data may be
dangerous. It can be as bad as having your service unavailable.

| > | 2 - You want your node to comme back like they was befor last reboot,
| once
| > | again it must be
| > | safe (when I will have fixed this "race condition").
| >
| > Hmmm? I can't get the point... could you please elaborate?
| > I think we're going in the right path (with this thread). Thanks Thomas
| for
| > letting me think a bit more about this subject.
|
| When you set up a mode with drdbc (slave/master) this is saved.
| If your server shutdown in this mode, at it next reboot it will check the
| file to come back like it was before ..

Take a look at my last statement.
It seems to me that we both are trying to treat the same situation from
differents points of view. And it looks that we both are missing something
on the way... :)

Best regards,

Luis

--
[ Luis Claudio R. Goncalves lclaudio@example.com ]
[. MSc coming soon -- Conectiva HA Team -- Gospel User -- Linuxer -- :) ]
[. Fault Tolerance - Real-Time - Distributed Systems - IECLB - IS 40:31 ]
[. LateNite Programmer -- Jesus Is The Solid Rock On Which I Stand -- ]

Hi,

I added a safe mode in datadisk (see drbd.example for more information ;*)
like Luis wanted to see and applyed the patch received from Tony Willoughby.

Like usual I am not testing my code (but for once it is due to a lack of
time
and not resources ... sorry )

Thomas

> Oh, I included a test in the heartbeat stuff... something like "If the
> user created a file called /etc/ha.d/safe_operation then he wants to know
> when a crashed node comes back to life". I think it would be useful and
the
> user would be alerted only if he wants. :)

Can you tell me a little bit more about it to be able to use it in drbd

Thomas

> Oh, I included a test in the heartbeat stuff... something like "If the
> user created a file called /etc/ha.d/safe_operation then he wants to know
> when a crashed node comes back to life". I think it would be useful and
the
> user would be alerted only if he wants. :)

Can you tell me a little bit more about it to be able to use it in drbd

Thomas

> Oh, I included a test in the heartbeat stuff... something like "If the
> user created a file called /etc/ha.d/safe_operation then he wants to know
> when a crashed node comes back to life". I think it would be useful and
the
> user would be alerted only if he wants. :)

Can you tell me a little bit more about it to be able to use it in drbd

Thomas