Mailing List Archive

Locking with IndexWriter
Hello,

I posted this earlier in the Dev list as there is no answer there I posted
again hoping that someone might know here :D

I am using lucene in an EJB environment. I have a message driven bean that
subscribe to a queue that will consume the object that I want lucene to index.

Now the problem when I have many object in the queue, the Application Server
will instantiated multiple message driven bean object and runs the indexing
at the same time. When this happen I got this exception:

[IndexerMDB] Problem indexing email in indexer mdb
java.io.IOException: Index locked for write:
Lock@/usr/kernel/jakartalucene/install/write.lock
        at org.apache.lucene.index.IndexWriter.<init>(Unknown Source)
        at org.apache.lucene.index.IndexWriter.<init>(Unknown Source)
        at com.nuix.indexer.IndexerMDB.indexData(IndexerMDB.java:304)

Only the first message driven bean will successfully index the data.

I had a look on the code and really FSDirectory will just simply fail if it
can't create the new lock file! Is this done on purpose? Why can't
IndexWriter tries for a few times to create the lock file? Do I miss
something really big here?

I modified FSDirectory as follow so it works in our environment. Please
comment on this change and just slam me if this is blatantly against lucene
framework :D

Basically I changed it so it does not fail straight away, instead try and
wait for creating the new file.

RCS file:
/usr/local/cvsroot/nuix/kernel/jakartalucene/dist/src/java/org/apache/lucene/store/FSDirectory.java,v
retrieving revision 1.1.1.1
diff -u -u -r1.1.1.1 FSDirectory.java
--- FSDirectory.java    4 Apr 2002 01:14:11 -0000       1.1.1.1
+++ FSDirectory.java    22 May 2002 07:19:20 -0000
@@ -218,7 +218,19 @@
     return new Lock() {
        public boolean obtain() throws IOException {
           if (Constants.JAVA_1_1) return true;    // locks disabled in jdk
1.1
-          return lockFile.createNewFile();
+          boolean locked = false;
+          locked = lockFile.createNewFile();
+          for (int count = 0; count < 100 && !locked; count++) {
+              try {
+                  Thread.sleep(100);
+                  locked = lockFile.createNewFile();
+              } catch (InterruptedException e) {
+                  e.printStackTrace();
+                  return false;
+              }
+              
+          }
+          return locked;
        }
        public void release() {
           if (Constants.JAVA_1_1) return;         // locks disabled in jdk
1.1
--
Victor Hadianto

--
To unsubscribe, e-mail: <mailto:lucene-user-unsubscribe@jakarta.apache.org>
For additional commands, e-mail: <mailto:lucene-user-help@jakarta.apache.org>
Re: Locking with IndexWriter [ In reply to ]
On Thu, 23 May 2002 10:05, you wrote:
> > I modified FSDirectory as follow so it works in our environment. Please
> > comment on this change and just slam me if this is blatantly against
> > lucene framework :D
>
> I'm not too familiar with the source of Lucene. But I'm pondering
> your suggested solution. Wouldn't wait/notify semantics be a better
> solution? Have a writer specify the max amount of time it will wait
> before failing. When a writer relases a lock it should call notify();
> rather than notifyAll(); in order to minimize the amount of actively
> competing processes and polling.
>

That's an interesting suggestion, I will have a thought on that one. Thanks

> Alternatively, you could use your routine in your bean.
>

Hm unfortunately even using Lucene with EJB already skirting with danger :D.
With EJB aren't we not supposed to do our own threads management? The EJB
container suppose to do all that for us. Well so far we haven't found any
issues (yet). We are crossing our fingers.

regards,

--
Victor Hadianto

--
To unsubscribe, e-mail: <mailto:lucene-user-unsubscribe@jakarta.apache.org>
For additional commands, e-mail: <mailto:lucene-user-help@jakarta.apache.org>
Re: Locking with IndexWriter [ In reply to ]
> I modified FSDirectory as follow so it works in our environment. Please
> comment on this change and just slam me if this is blatantly against lucene
> framework :D

I'm not too familiar with the source of Lucene. But I'm pondering
your suggested solution. Wouldn't wait/notify semantics be a better
solution? Have a writer specify the max amount of time it will wait
before failing. When a writer relases a lock it should call notify();
rather than notifyAll(); in order to minimize the amount of actively
competing processes and polling.

Alternatively, you could use your routine in your bean.

BR,

Morten



--
To unsubscribe, e-mail: <mailto:lucene-user-unsubscribe@jakarta.apache.org>
For additional commands, e-mail: <mailto:lucene-user-help@jakarta.apache.org>
Re: Locking with IndexWriter [ In reply to ]
Is it possible to configure your app server to have just one message driven
bean instance in the pool? Obviously this is not a solution in general to
concurrent access to Lucene but would remove the need for multiple
IndexWriters in your particular case and give you the same overall throughput
- (hopefully the appserver would be looking after contention on the one bean
instance)

Cheers
Mark


--
To unsubscribe, e-mail: <mailto:lucene-user-unsubscribe@jakarta.apache.org>
For additional commands, e-mail: <mailto:lucene-user-help@jakarta.apache.org>
Re: Locking with IndexWriter [ In reply to ]
On Thu, 23 May 2002 18:35, you wrote:
> Is it possible to configure your app server to have just one message driven
> bean instance in the pool? Obviously this is not a solution in general to

There are more than just one type of message driven bean, I do not think
JBoss can set the pool size for individual MDB, or at least the last time I
check.


--
Victor Hadianto

--
To unsubscribe, e-mail: <mailto:lucene-user-unsubscribe@jakarta.apache.org>
For additional commands, e-mail: <mailto:lucene-user-help@jakarta.apache.org>