Mailing List Archive: DRBD 0.5.7 timeout deadlock, somewhat of a resolution found though...

DRBD 0.5.7 timeout deadlock, somewhat of a resolution found though...

Sep 26, 2000, 6:51 PM

Post #1 of 6 (1812 views)

I ran the drbd benchmark, and got these results. Here are some hardware
specifications:

Node1
------
CPU: PII-350
RAM: 64MB
Disk: IDE 7200rpm Quantum Fireball 6.5GB
NIC: 3c905B Cyclone, set to full duplex.

Node2
------
CPU: Celeron 400
RAM: 64MB
Disk: IDE WDC AC26400R 6.5GB (Don't know the RPM's)
NIC: Intel i82558 based using eepro100 module, set to full duplex.

Both are sharing /dev/hda8, and I'm using ReiserFS primarily, but the
problem has been recreated with ext2.(reiserfs works beautifully btw, no
problems with resyncing at all).

I'm sending this to drbd-devel because Node2 has a very hard time on
sustained writes. It exhibits the same behavior that was reported with
0.5.6pre1. Deadlock, can switch virtual terminals, can't log in... TCP
responds with the SYNACK, but thats about it. This only happens on Node2,
writing 100MB or more using bonnie as a benchmark. Node1 had the same
problem, but I increased the rate to 10000(drbdsetup ...... -r 10000), and
changed the protocol from B to C. That seemed to alleviate the problem. In
fact, I've run bonnie -s 900 on that machine, and it happily mirrored
everything over to the other without complaining. However, bonnie -s 100
while Node2 is the primary, results in the timeout deadlock.

Anyways, I have another identical system to Node1, so these will probably
become my fault-tolerant SMB server for the backend of my NT based(no I
didn't make that decision) websites.

--------------- report --------------
DRBD Benchmark
Version: 0.5.7
SETSIZE = 100M

Node1:
Linux 2.2.17 i686
bogomips : 694.68
Disk write: 9.44 MB/sec (104857600 B / 00:10.595702)
Drbd unconnected: 9.43 MB/sec (104857600 B / 00:10.601198)

Node2:
Linux 2.2.17 i686
bogomips : 794.62
Disk write: 8.99 MB/sec (104857600 B / 00:11.120484)
Drbd unconnected: 8.69 MB/sec (104857600 B / 00:11.502283)

Network:
Bandwidth: 1.04 MB/sec (104857600 B / 01:36.113341)
Latency: round-trip min/avg/max = 0.1/0.1/0.2 ms

Drbd connected (writing on node1):
Protocol A: 5.62 MB/sec (104857600 B / 00:17.788790)
Protocol B: 6.77 MB/sec (104857600 B / 00:14.762859)
Protocol C: 6.07 MB/sec (104857600 B / 00:16.467106)

Drbd connected (writing on node2):
Protocol A: 8.48 MB/sec (104857600 B / 00:11.790881)
Protocol B: 7.96 MB/sec (104857600 B / 00:12.559939)
Protocol C: 8.36 MB/sec (104857600 B / 00:11.956605)
-------------------------------------

Re: DRBD 0.5.7 timeout deadlock, somewhat of a resolution found though... [ In reply to ]

lclaudio at example

Sep 27, 2000, 6:56 AM

Post #2 of 6 (1771 views)

Permalink

Hi!

> Node2
> ------
> CPU: Celeron 400
> RAM: 64MB
> Disk: IDE WDC AC26400R 6.5GB (Don't know the RPM's)
> NIC: Intel i82558 based using eepro100 module, set to full duplex.

What kernel and distro are you using? Some versions of the eepro100
module were broken under heavy load.

Luis

[ Luis Claudio R. Goncalves lclaudio@example.com ]
[. MSc coming soon -- Conectiva HA Team -- Gospel User -- Linuxer -- :) ]
[. Fault Tolerance - Real-Time - Distributed Systems - IECLB - IS 40:31 ]
[. LateNite Programmer -- Jesus Is The Solid Rock On Which I Stand -- ]

RE: DRBD 0.5.7 timeout deadlock, somewhat of a resolution found though... [ In reply to ]

cbyrum at example

Sep 27, 2000, 9:58 AM

Post #3 of 6 (1776 views)

Permalink

> -----Original Message-----
> From: Luis Claudio R. Goncalves [mailto:lclaudio@example.com]
> Sent: Wednesday, September 27, 2000 6:57 AM
>
>
> Hi!
>
> > Node2
> > ------
> > CPU: Celeron 400
> > RAM: 64MB
> > Disk: IDE WDC AC26400R 6.5GB (Don't know the RPM's)
> > NIC: Intel i82558 based using eepro100 module, set to full duplex.
>
> What kernel and distro are you using? Some versions of the eepro100
> module were broken under heavy load.
>

Kernel 2.2.17, Debian 2.2(Potato), with latest updates applied. eepro100 is
built in to the kernel(whoops, forgot to change it to a module).
<snip>

RE: DRBD 0.5.7 timeout deadlock, somewhat of a resolution found though... [ In reply to ]

cbyrum at example

Sep 27, 2000, 1:54 PM

Post #4 of 6 (1767 views)

Permalink

A little more info. After reading this, I realized that I wrote it while
tired and trying desperately to leave work. I'll fill in some of the details
below...

> -----Original Message-----
> From: drbd-devel-admin@example.com
> [mailto:drbd-devel-admin@example.com]On Behalf Of Clint Byrum
> Sent: Tuesday, September 26, 2000 6:52 PM
>
>
> I ran the drbd benchmark, and got these results. Here are some hardware
> specifications:
>
> Node1
> ------
> CPU: PII-350
> RAM: 64MB
> Disk: IDE 7200rpm Quantum Fireball 6.5GB
> NIC: 3c905B Cyclone, set to full duplex.
>
> Node2
> ------
> CPU: Celeron 400
> RAM: 64MB
> Disk: IDE WDC AC26400R 6.5GB (Don't know the RPM's)
> NIC: Intel i82558 based using eepro100 module, set to full duplex.
>
> Both are sharing /dev/hda8, and I'm using ReiserFS primarily, but the
> problem has been recreated with ext2.(reiserfs works beautifully btw, no
> problems with resyncing at all).
>
> I'm sending this to drbd-devel because Node2 has a very hard time on
> sustained writes. It exhibits the same behavior that was reported with
> 0.5.6pre1. Deadlock, can switch virtual terminals, can't log in... TCP
> responds with the SYNACK, but thats about it. This only happens on Node2,
> writing 100MB or more using bonnie as a benchmark. Node1 had the same
> problem, but I increased the rate to 10000(drbdsetup ...... -r 10000), and
> changed the protocol from B to C. That seemed to alleviate the problem. In
> fact, I've run bonnie -s 900 on that machine, and it happily mirrored
> everything over to the other without complaining. However, bonnie -s 100
> while Node2 is the primary, results in the timeout deadlock.
>

I've upped this to 50000... no change in throughput, and still deadlocks.

"Deadlock" is accompanied by this error message on the console:
drbd0: ack timeout detected (pc=32)!
drbd : timeout detected! (pid=2)
tcp_close: socket already locked!

This happens usually during the "Writing intelligently..." phase of bonnie.
While memory is generally low(mostly used up in buffers), there is always at
least 1 or 2 MB free when this happens.

ReiserFS seems to cause the link to die more quickly. Since my first try
with ext2(shame on me, I only tried it once), I've run through the full
gamut with it, trying different -r settings. It takes a lot more to deadlock
the Celeron with ext2. Namely, 500MB, instead of just 100-200 with ReiserFS.
I'm thinking this is just due to the added overhead of journal updates of
ReiserFS. While ReiserFS is not 100% necessary for this solution right
now(we're only wanting to keep 200-500 MB of dynamic data on the drbd
share), it will be(or ext3, or JFS, etc, haven't tried them yet) as we find
more and more things that change too often to be copied periodically.

Protocol B cannot make it through 100MB on ext2 or ReiserFS. Protocol A
works well, and makes it through 700MB without a problem using both ext2 and
ReiserFS. I stopped there because really, Protocol A isn't what I want. I
like the idea of writes being confirmed by the writing node in protocol C.

Again, the PII-350 has not deadlocked since setting -r >= 10000.

> Anyways, I have another identical system to Node1, so these will probably
> become my fault-tolerant SMB server for the backend of my NT based(no I
> didn't make that decision) websites.
>
<snip benchmarks>

In the benchmarks, I used ssh, so I imagine the bandwidth (about 1MB/sec) is
a little off, as this is a full-duplex 100MBit connection over a crossover
cable.

Re: DRBD 0.5.7 timeout deadlock, somewhat of a resolution found though... [ In reply to ]

wmcdonald at example

Sep 29, 2000, 5:02 AM

Post #5 of 6 (1766 views)

Permalink

This may be entirely unrelated but we had similar behaviour here and traced
it to running an smp kernel. We're using Supermicro SMP motherboards with
only one CPU, somehow install of linux we used (RH 6.2) noticed the SMP
capability and installed the smp kernel. Switching to the single processor
kernel sorted the problem out. Are either of these machines SMP capable?

Will.

----- Original Message -----
From: Clint Byrum <cbyrum@example.com>
To: Drbd-Devel@Lists. Sourceforge. Net <drbd-devel@example.com>
Sent: Wednesday, September 27, 2000 9:54 PM
Subject: RE: [DRBD-dev] DRBD 0.5.7 timeout deadlock, somewhat of a
resolution found though...

> A little more info. After reading this, I realized that I wrote it while
> tired and trying desperately to leave work. I'll fill in some of the
details
> below...
>
<snip>

RE: DRBD 0.5.7 timeout deadlock, somewhat of a resolution found though... [ In reply to ]

cbyrum at example

Sep 29, 2000, 12:04 PM

Post #6 of 6 (1777 views)

Permalink

> -----Original Message-----
> From: drbd-devel-admin@example.com
> [mailto:drbd-devel-admin@example.com]On Behalf Of Will Mc
> Donald
> Sent: Friday, September 29, 2000 5:03 AM
>
>
> This may be entirely unrelated but we had similar behaviour here
> and traced
> it to running an smp kernel. We're using Supermicro SMP motherboards with
> only one CPU, somehow install of linux we used (RH 6.2) noticed the SMP
> capability and installed the smp kernel. Switching to the single processor
> kernel sorted the problem out. Are either of these machines SMP capable?
>
> Will.
>

No, these are both single CPU boards and machines, and I compiled the kernel
myself with SMP off(just double checked).

On another note, I've tried running bonnie from a remote machine hooked up
the way the drbd share will be used. I mounted the drbd mount point via
smbfs on another linux machine connected to the drbd servers' primary
ethernet on a 100Mbit switch. Had no problems whatsoever, though, as
expected, write times were a lot slower than when writing directly.

> ----- Original Message -----
> From: Clint Byrum <cbyrum@example.com>
> To: Drbd-Devel@Lists. Sourceforge. Net <drbd-devel@example.com>
> Sent: Wednesday, September 27, 2000 9:54 PM
> Subject: RE: [DRBD-dev] DRBD 0.5.7 timeout deadlock, somewhat of a
> resolution found though...
>
>
> > A little more info. After reading this, I realized that I wrote it while
> > tired and trying desperately to leave work. I'll fill in some of the
> details
> > below...
> >
> <snip>
>
>
> _______________________________________________
> DRBD-devel mailing list
> DRBD-devel@example.com
> http://lists.sourceforge.net/mailman/listinfo/drbd-devel