Mailing List Archive: Couldn't get lock

Couldn't get lock

jesse at erlbaum

Jun 18, 2007, 7:24 AM

Post #1 of 6 (2697 views)

Hi All --

We've been rolling out a Kinosearch-based site search engine, and
periodically get the following error:

Couldn't get lock using
'/tmp/kinosearch_lockdir/kinosearch-677654d74548827b32ec1abc555b92a7-com
mit.lock' at /path/to/lib/i686-linux/KinoSearch/Store/Lock.pm line 28

When I look in /tmp I see nothing. The directory "kinosearch_lockdir"
does not exist. What's going on here? Is there a way to set the lock
directory? Is there a way to tell Kino to create the directory if it is
missing?

This error causes the Search engine to periodically crash with an
Internal Server Error.

FYI, I don't know if it is important, but we build the index on one
machine and the front-end search runs on another. (The index directory
is NFS mounted, and UNIX user ids are in sync. The "/tmp" directory is
*not* shared between machines.)

Thanks,

-Jesse-

Jesse Erlbaum
The Erlbaum Group, LLC
817 Broadway, 10th floor
New York, NY 10003
212-684-6161 (office)
917-647-3059 (mobile)
212-684-6226 (fax)
jesse@erlbaum.net <blocked::mailto:jesse@erlbaum.net>

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://www.rectangular.com/pipermail/kinosearch/attachments/20070618/b9287fd8/attachment.htm

Couldn't get lock [ In reply to ]

marvin at rectangular

Jun 18, 2007, 9:06 AM

Post #2 of 6 (2611 views)

On Jun 18, 2007, at 7:12 AM, Jesse Erlbaum wrote:

> We've been rolling out a Kinosearch-based site search engine, and
> periodically get the following error:
>
> Couldn't get lock using '/tmp/kinosearch_lockdir/
> kinosearch-677654d74548827b32ec1abc555b92a7-commit.lock' at /path/
> to/lib/i686-linux/KinoSearch/Store/Lock.pm line 28
>
> When I look in /tmp I see nothing. The directory
> "kinosearch_lockdir" does not exist.

Presumably it exists sometimes -- otherwise, you'd never get a
Searcher. KS never removes that directory itself.

> What's going on here? Is there a way to set the lock directory?

There's a hack.

use KinoSearch::Store::FSInvIndex;
$KinoSearch::Store::FSInvIndex::LOCK_DIR = '/path/to/wherever';
mkdir('path/to/wherever') or die $!;

Do that before you create the Searcher.

This is for 0.1x. In the 0.20 branch, the locking mechanism has
changed. The lock dir is now the index dir itself, and Searchers
don't use locking by default. However, since your index is on an NFS
volume, you're going to need it.

> Is there a way to tell Kino to create the directory if it is missing?

That's happening at Class load time. Snoop the BEGIN block in
KinoSearch::Store::FSInvIndex.

> This error causes the Search engine to periodically crash with an
> Internal Server Error.
>
> FYI, I don't know if it is important, but we build the index on one
> machine and the front-end search runs on another. (The index
> directory is NFS mounted, and UNIX user ids are in sync. The "/
> tmp" directory is *not* shared between machines.)

NFS is going to require special care, but for other reasons. I
recommend that you upgrade to the 0.20 branch, and that you supply a
LockFactory to both InvIndexer and IndexReader. Please check out the
0.20 branch docs for those two classes here:

http://www.rectangular.com/kinosearch/docs/devel/

Note that 0.20 and 0.15 are not compatible either in terms of API or
file format.

Please write back to let us know how things go.

HTH,

Marvin Humphrey
Rectangular Research
http://www.rectangular.com/

Couldn't get lock [ In reply to ]

jesse at erlbaum

Jun 18, 2007, 9:18 AM

Post #3 of 6 (2617 views)

Hi Marvin --

> Presumably it exists sometimes -- otherwise, you'd never get a
> Searcher. KS never removes that directory itself.

Is it possible that tmpwatch deleted the directory? It is in tmp/,
which implies that maybe it won't exist for very long.

> Please write back to let us know how things go.

Will do.

-Jesse-

Jesse Erlbaum
The Erlbaum Group, LLC
817 Broadway, 10th floor
New York, NY 10003
212-684-6161 (office)
917-647-3059 (mobile)
212-684-6226 (fax)
jesse@erlbaum.net

Couldn't get lock [ In reply to ]

marvin at rectangular

Jun 18, 2007, 9:29 AM

Post #4 of 6 (2616 views)

On Jun 18, 2007, at 9:07 AM, Jesse Erlbaum wrote:

> Is it possible that tmpwatch deleted the directory?

Possibly. That would imply that you're using a persistence mechanism
a la FastCGI/mod_perl (so the classes don't get reloaded and the
mkdir logic doesn't get re-executed) and that there can be a long
enough time between creation of Searchers that /tmp/
kinosearch_lockdir doesn't get touched and tmpwatch sweeps it.

Sounds like a bug, but one that moving the lock dir in 0.20 should
have resolved.

Marvin Humphrey
Rectangular Research
http://www.rectangular.com/

Couldn't get lock [ In reply to ]

jesse at erlbaum

Jun 18, 2007, 9:37 AM

Post #5 of 6 (2608 views)

> Possibly. That would imply that you're using a persistence
> mechanism
> a la FastCGI/mod_perl (so the classes don't get reloaded and the
> mkdir logic doesn't get re-executed) and that there can be a long
> enough time between creation of Searchers that /tmp/
> kinosearch_lockdir doesn't get touched and tmpwatch sweeps it.

Yep -- running mod_perl.

> Sounds like a bug, but one that moving the lock dir in 0.20 should
> have resolved.

Is there some sort of primer on the API changes which require changes to
my code?

Thanks,

-Jesse-

Couldn't get lock [ In reply to ]

marvin at rectangular

Jun 18, 2007, 9:48 AM

Post #6 of 6 (2608 views)

On Jun 18, 2007, at 9:26 AM, Jesse Erlbaum wrote:

> Is there some sort of primer on the API changes which require
> changes to
> my code?

There isn't, but I've attempted to make the changes trigger
catastrophic failure rather than fail subtly.

The big change is that you'll need a Schema.

http://www.rectangular.com/kinosearch/docs/devel/KinoSearch/
Schema.html

Marvin Humphrey
Rectangular Research
http://www.rectangular.com/