Mailing List Archive

IndexWriter and IndexReader open at the same time
Hello,

I have an application that gets many delete and write resquests at the
same time. to avoid opening and closing the IndexWriter and
IndexReader everytime one of them need to do a write operation, i keep
them both open and have a shared lock around them whenever i need to
use them for writing. everything seems to be working in order, but
i'm not sure if this is a safe thing to do. please let me know.

thank you,
lavafish
Re: IndexWriter and IndexReader open at the same time [ In reply to ]
If you have the Lucene book, look at Chapter 2 (page 59 under section
2.9 (Concurrency, thread-safety, and locking issues) in chapter 2
(Indexing)):

http://www.lucenebook.com/search?query=concurrency+rules

Also, look at Lucene's Bugzilla, where you'll find a contribution that
helps with concurrent IndexReader/IndexWriter usage.

Otis


--- Greg Love <lavafish@gmail.com> wrote:

> Hello,
>
> I have an application that gets many delete and write resquests at
> the
> same time. to avoid opening and closing the IndexWriter and
> IndexReader everytime one of them need to do a write operation, i
> keep
> them both open and have a shared lock around them whenever i need to
> use them for writing. everything seems to be working in order, but
> i'm not sure if this is a safe thing to do. please let me know.
>
> thank you,
> lavafish
Re: IndexWriter and IndexReader open at the same time [ In reply to ]
Thanx for the feedback Otis.

I did read that section of the book already. But i'm still not clear
on something... I know that the only way documents can be deleted
from the index is when i close IndexReader, what worries me is that
stuff gets cached in the ram directory when i do inserts. What I'm
not sure about is if those inserts go into the index at a random time,
or only when i try to insert something new or call close on
IndexWriter. If the inserted documents can go into the index on the
hard drive at any random time then what i'm doing is not safe.

In the TheServerSide case study of the book, page 375, they say that
they close the IndexWriter and even point out that they did before
openning the IndexReader and deleting. So that kinda makes me wonder
if i'm safe having an IndexReader with deletions and and IndexWriter
with inserts open at the same time (even though my code never does an
index modifying operation at the same time because they share a lock
in the my code).

thank you

On 8/8/05, Otis Gospodnetic <otis_gospodnetic@yahoo.com> wrote:
> If you have the Lucene book, look at Chapter 2 (page 59 under section
> 2.9 (Concurrency, thread-safety, and locking issues) in chapter 2
> (Indexing)):
>
> http://www.lucenebook.com/search?query=concurrency+rules
>
> Also, look at Lucene's Bugzilla, where you'll find a contribution that
> helps with concurrent IndexReader/IndexWriter usage.
>
> Otis
>
>
> --- Greg Love <lavafish@gmail.com> wrote:
>
> > Hello,
> >
> > I have an application that gets many delete and write resquests at
> > the
> > same time. to avoid opening and closing the IndexWriter and
> > IndexReader everytime one of them need to do a write operation, i
> > keep
> > them both open and have a shared lock around them whenever i need to
> > use them for writing. everything seems to be working in order, but
> > i'm not sure if this is a safe thing to do. please let me know.
> >
> > thank you,
> > lavafish
>
>
Re: IndexWriter and IndexReader open at the same time [ In reply to ]
Greg Love wrote:
> In the TheServerSide case study of the book, page 375, they say that
> they close the IndexWriter and even point out that they did before
> openning the IndexReader and deleting. So that kinda makes me wonder
> if i'm safe having an IndexReader with deletions and and IndexWriter
> with inserts open at the same time (even though my code never does an
> index modifying operation at the same time because they share a lock
> in the my code).

You should close the IndexReader you are using for deletions before
opening the IndexWriter you use for additions. For higher throughput,
queue additions and deletions and process them periodically as batches.
If you're concerned about an addition followed by a deletion of the
same document getting reversed in the queues, then simply check the
addition queue each time you queue a deletion, and remove any matching
additions.

Doug
Re: IndexWriter and IndexReader open at the same time [ In reply to ]
yeah, that was something i wanted to avoid doing but I guess i have no
choice. thanx for the help

On 8/9/05, Doug Cutting <cutting@apache.org> wrote:
> Greg Love wrote:
> > In the TheServerSide case study of the book, page 375, they say that
> > they close the IndexWriter and even point out that they did before
> > openning the IndexReader and deleting. So that kinda makes me wonder
> > if i'm safe having an IndexReader with deletions and and IndexWriter
> > with inserts open at the same time (even though my code never does an
> > index modifying operation at the same time because they share a lock
> > in the my code).
>
> You should close the IndexReader you are using for deletions before
> opening the IndexWriter you use for additions. For higher throughput,
> queue additions and deletions and process them periodically as batches.
> If you're concerned about an addition followed by a deletion of the
> same document getting reversed in the queues, then simply check the
> addition queue each time you queue a deletion, and remove any matching
> additions.
>
> Doug
>