Mailing List Archive

Additional serialization code [Jim Jagielski <jim@shado.jaguNET.com>] (fwd)
Jeez, Jim, pick an address! :)

Brian

---------- Forwarded message ----------
From: Jim Jagielski <jim@shado.jaguNET.com>
Subject: Additional serialization code
To: new-httpd@hyperreal.com
Date: Mon, 7 Aug 1995 19:21:45 -0400 (EDT)

As mentioned in a previous message, although the concept of serializing
accept() is _very_ code, for some systems and setups, the use of fcntl()
to perform the lock may not be the best idea, since it's implementation
of fcntl() locking (esp. under NFS) can take an appreciable amount of
time.

I've added another possible option: FLOCK_SERIALIZED_ACCEPT. Basically,
it simply uses flock() instead of fcntl(). Now, of course, if the server
and the log file are on different machines, then flock() won't work and
fcntl() is the way to go... I would guess that this is the vast minority
however. In any case, I saw some pretty major speed improvements moving
from FCNTL to FLOCK.

Here's the patch:

---------->8 cut here-------------------------

*** Ohttp_main.c Mon Aug 7 11:57:05 1995
--- http_main.c Mon Aug 7 13:01:02 1995
***************
*** 137,143 ****

int one_process = 0;

! #ifdef FCNTL_SERIALIZED_ACCEPT
struct flock lock_it = { F_WRLCK, 0, 0, 0 };
struct flock unlock_it = { F_UNLCK, 0, 0, 0 };

--- 137,143 ----

int one_process = 0;

! #if defined(FCNTL_SERIALIZED_ACCEPT)
struct flock lock_it = { F_WRLCK, 0, 0, 0 };
struct flock unlock_it = { F_UNLCK, 0, 0, 0 };

***************
*** 159,164 ****
--- 159,186 ----
void accept_mutex_off()
{
fcntl (fileno(server_conf->error_log), F_SETLKW, &unlock_it);
+ }
+
+ #elif defined(FLOCK_SERIALIZED_ACCEPT)
+
+ void accept_mutex_on()
+ {
+ int ret;
+
+ while ((ret = flock(fileno(server_conf->error_log),LOCK_EX)) < 0
+ && errno == EINTR)
+ continue;
+
+ if (ret < 0) {
+ log_error ("Unknown failure grabbing accept lock. Exiting!",
+ server_conf);
+ exit(1);
+ }
+ }
+
+ void accept_mutex_off()
+ {
+ flock (fileno(server_conf->error_log), LOCK_UN);
}
#else
/* Default --- no serialization. Other methods *could* go here,

---------->8 cut here-------------------------
--
Jim Jagielski << jim@jaguNET.com >> | "Stwike him Centuwian, vewy woughwy!"
** jaguNET Access Services ** | - Pontius Piwate
++ Email: info@jaguNET.com +++ Voice: 410-931-3157 ++
++ http://www.jaguNET.com/ +++ Data: 931-7060 ("guest") ++
Re: Additional serialization code [ In reply to ]
Re: Additional serialization code [Jim Jagielski <jim@shado.jaguNET.com>] (fwd) [ In reply to ]
On review of this code, it's got a bit of a problem --- locks reserved
by flock() are per-descriptor, *not* per-process (with fcntl, it's the
reverse). Since all server processes share a single descriptor with the
error log, what happens is that the first process to execute flock()
grabs the lock (against any other descriptor which may have been
obtained by a different open()) call, and all subsequent flock() calls
--- in the same process or a different one --- simply verify that the
lock is still held and return (which is very fast, but, unfortunately,
does nothing at all about mutual exclusion).

This *would* work on linux pre-1.3, in which flock had the same semantics
as fcntl locks (widely considered a bug), or using the scoreboard descriptor,
since each child process *does* have a distinct one of those.

rst

PS --- since this works for Jim anyway, I consider that confirmation that
no form of locking is actually required around the accept() in A/UX.
Re: Additional serialization code [ In reply to ]
From: Jim Jagielski <jim@jagunet.com>
Date: Tue, 8 Aug 1995 10:51:09 -0400 (EDT)
Precedence: bulk
Reply-To: new-httpd@hyperreal.com

Robert S. Thau writes:
>
> On review of this code, it's got a bit of a problem
>...
> obtained by a different open()) call, and all subsequent flock() calls
> --- in the same process or a different one --- simply verify that the
> lock is still held and return (which is very fast, but, unfortunately,
> does nothing at all about mutual exclusion).

If the lock is exclusive, then the next call to lock, if already locked,
should block until the lock is released.

Only if the processes have different locks --- the problem is that
since the processes are sharing the same file descriptor to the error
log, they share the same lock as well (either all of them have it, or
none of them do, so they can't exclude each other) --- flock() locks
behave like this; fcntl() locks are owned by processes and not by
descriptors, so they don't.

rst
Re: Additional serialization code [Jim Jagielski <jim@shado.jaguNET.com>] (fwd) [ In reply to ]
>On review of this code, it's got a bit of a problem --- locks reserved
>by flock() are per-descriptor, *not* per-process (with fcntl, it's the
>reverse). Since all server processes share a single descriptor with the
>error log, what happens is that the first process to execute flock()
>grabs the lock (against any other descriptor which may have been
>obtained by a different open()) call, and all subsequent flock() calls
>--- in the same process or a different one --- simply verify that the
>lock is still held and return (which is very fast, but, unfortunately,
>does nothing at all about mutual exclusion).
>
>This *would* work on linux pre-1.3, in which flock had the same semantics
>as fcntl locks (widely considered a bug), or using the scoreboard descriptor,
>since each child process *does* have a distinct one of those.

I had difficulty understanding what you meant, until I read this
extract from the Solaris flock manpage:

Locks are on files, not file descriptors. That is, file
descriptors duplicated through dup(2) or fork(2) do not
result in multiple instances of a lock, but rather multiple
references to a single lock. If a process holding a lock on
a file forks and the child explicitly unlocks the file, the
parent will lose its lock.

I think that the locking should be done on the scoreboard file rather
than the error log anyway, irrespective of which locking mechanism is
used. (Though for fcntl an fd to the scoreboard file would have to be
kept open.) This has two advantages:

Multiple httpd's can log to the same error_log (likely if you are
using BindAddress).

The scoreboard file should reside on a local non-NFS exported fs;
it's _just_ possible that an OS might recognise that in this case
it need not register the lock with the NFS lock daemon, and hence
be much faster at locking/unlocking. (I'll test this for Solaris)

David.
Re: Additional serialization code [Jim Jagielski <jim@shado.jaguNET.com>] (fwd) [ In reply to ]
I did some times on Solaris:
These are total times to lock and unlock a file that wasn't previously locked.

For any local filesystem: 0.18ms
For any remote filesystem: 17ms

By comparison, a memory based mutex lock takes 5 us 8-)

David.