Mailing List Archive

RE: [Lucene-dev] Re: FW: Lucene 1.2 and directory write permissio ns?
I second that having the lock location configurable would be a "nice to
have"... but this brings up the problem of the searcher process being
configured differently than the updating process, which will defeat the
usefulness of the lock.

I can imagine the sys-admin types wanting locks to go in a temp directory or
something like that, though.


Dave


David Snyder
Señor Software Engineer

NetGenics, Inc.
1717 E. 9th St., #1700
Cleveland, OH 44114
(216) 861-4007

-----Original Message-----
From: Dmitry Serebrennikov [mailto:dmitrys@earthlink.net]
Sent: Friday, October 05, 2001 3:24 PM
To: lucene-dev@jakarta.apache.org
Subject: [Lucene-dev] Re: FW: Lucene 1.2 and directory write permissions?


Just a though - as long as we are separating the locks, we should probably
make the location configurable and not require it to be a subdirectory of
the indexes. This will help in case the indexes come burned on a CD-ROM or
if the operating system does not support easy linking / permission magic.

Doug Cutting wrote:


Here's one vote for putting locks in a separate directory. Anyone dislike
that?

Doug

-----Original Message-----
From: Snyder, David [ mailto:dsnyder@netgenics.com
<mailto:dsnyder@netgenics.com> ]
Sent: Friday, October 05, 2001 11:23 AM
To: Doug Cutting
Subject: RE: Lucene 1.2 and directory write permissions?


The lock file synchronization is very handy for us as we do updates in a
separate process from searches... I was very pleased to see this in there!

I think splitting out the locks into a separate directory would solve our
problem... we have a reference set of data that we use for testing and our
sysadmin wants to make sure it doesn't accidentally get overwritten. It's
still possible (using links and other permission magic) but becomes more of
a maintenance headache. I definitely vote for the locks subdirectory idea.
It can be created when the index

es are initially loaded, we can go change
the permissions or maybe make it a link to a tmp directory or something, and
then our regular index files can be safe. Do you think this is something
very difficult to do? (I have yet to build Lucene myself, but I would love
to contribute... we are actually working on some XML based loaders that may
be of general interest)

Lucene has been great for us, but the way... we are indexing genetic data
(not the sequences themselves, but all the annotations and description stuff
that scientists tack on) and Lucene has been excellent... our indexes (we
use many with the multisearcher) are about 13 gigs now and Lucene has hardly
broken a sweat.

Thanks for your help,
Dave

David Snyder
Señor Software Engineer

NetGenics, Inc.
1717 E. 9th St., #1700
Cleveland, OH 44114
(216) 861-4007


-----Original Message-----
From: Doug Cutting [ mailto:DCutting@grandcentral.com
<mailto:DCutting@grandcentral.com> ]
Sent: Friday, October 05, 2001 12:24 PM
To: 'Snyder, David'
Subject: RE: Lucene 1.2 and directory write permissions?



From: Snyder, David [ mailto:dsnyder@netgenics.com
<mailto:dsnyder@netgenics.com> ]

I've been porting our application to use the 1.2 release
candidate 1 build
and now have a problem opening searchers on our existing
indexes. I get a
Permission Denied exception... our permissions are set up to
allow reading
of the directory and contained files during a search, but not writing.


Hmm. That is a problem. The reader now creates a lock file while it is
opening the index to keep a writer process from deleting files while they're
being opened. When opening an index the reader must first read the list of
files to open, then open them. If between reading the list and opening a
file that file were to disappear, then the open would fail. This was the
longstanding race condition that is fixed by the lock files. A writing
process will now wait for the reader to open all of the files before
updating things.

Perhaps we should instead write lock files in a subdirectory of the index
named "locks". You could make that directory read/write, but make the
parent read-only. Alternately, we could have an flag that turns off the use
of lock files, for those who know that there is no other process that is
potentially simultaneously updating the index. Which approach would folks
prefer?

D

oug