Mailing List Archive

IndexReader Pool
I was going through the lucene-user posts on the web and I came accross
a posting by Scott Oshima
http://www.mail-archive.com/lucene-user@jakarta.apache.org/msg00693.html

witch is talking about creating a IndexReader pool to spead up the search
I've looked into that but I can't fiure out what to use for a DataSource
like in creating a pool for DB connections, is there an equivalant in the
lucene architecture or should one just take the initiative.

Nader S. Henein
Bayt.com , Dubai Internet City
Tel. +9714 3911900
Fax. +9714 3911915
GSM. +9715 05659557
www.bayt.com

--
To unsubscribe, e-mail: <mailto:lucene-user-unsubscribe@jakarta.apache.org>
For additional commands, e-mail: <mailto:lucene-user-help@jakarta.apache.org>
Re: IndexReader Pool [ In reply to ]
I don't think Lucene contains anything to help you create this pool.
However, if you look at Jakarta Commons project you will find a
subproject there that allows you to create pools of any kind of Java
object. You can probably use that to save yourself development and
debug time.

Otis


--- "Nader S. Henein" <nsh@bayt.net> wrote:
>
> I was going through the lucene-user posts on the web and I came
> accross
> a posting by Scott Oshima
>
http://www.mail-archive.com/lucene-user@jakarta.apache.org/msg00693.html
>
> witch is talking about creating a IndexReader pool to spead up the
> search
> I've looked into that but I can't fiure out what to use for a
> DataSource
> like in creating a pool for DB connections, is there an equivalant in
> the
> lucene architecture or should one just take the initiative.
>
> Nader S. Henein
> Bayt.com , Dubai Internet City
> Tel. +9714 3911900
> Fax. +9714 3911915
> GSM. +9715 05659557
> www.bayt.com
>
> --
> To unsubscribe, e-mail:
> <mailto:lucene-user-unsubscribe@jakarta.apache.org>
> For additional commands, e-mail:
> <mailto:lucene-user-help@jakarta.apache.org>
>


__________________________________________________
Do You Yahoo!?
Yahoo! - Official partner of 2002 FIFA World Cup
http://fifaworldcup.yahoo.com

--
To unsubscribe, e-mail: <mailto:lucene-user-unsubscribe@jakarta.apache.org>
For additional commands, e-mail: <mailto:lucene-user-help@jakarta.apache.org>
RE: IndexReader Pool [ In reply to ]
http://www.mail-archive.com/lucene-dev@jakarta.apache.org/msg00938.html

> -----Original Message-----
> From: Otis Gospodnetic [mailto:otis_gospodnetic@yahoo.com]
> Sent: Thursday, June 27, 2002 3:56 PM
> To: Lucene Users List; nsh@bayt.net
> Subject: Re: IndexReader Pool
>
>
> I don't think Lucene contains anything to help you create this pool.
> However, if you look at Jakarta Commons project you will find a
> subproject there that allows you to create pools of any kind of Java
> object. You can probably use that to save yourself development and
> debug time.
>
> Otis
>
>
> --- "Nader S. Henein" <nsh@bayt.net> wrote:
> >
> > I was going through the lucene-user posts on the web and I came
> > accross
> > a posting by Scott Oshima
> >
> http://www.mail-archive.com/lucene-user@jakarta.apache.org/msg
00693.html
>
> witch is talking about creating a IndexReader pool to spead up the
> search
> I've looked into that but I can't fiure out what to use for a
> DataSource
> like in creating a pool for DB connections, is there an equivalant in
> the
> lucene architecture or should one just take the initiative.
>
> Nader S. Henein
> Bayt.com , Dubai Internet City
> Tel. +9714 3911900
> Fax. +9714 3911915
> GSM. +9715 05659557
> www.bayt.com
>
> --
> To unsubscribe, e-mail:
> <mailto:lucene-user-unsubscribe@jakarta.apache.org>
> For additional commands, e-mail:
> <mailto:lucene-user-help@jakarta.apache.org>
>


__________________________________________________
Do You Yahoo!?
Yahoo! - Official partner of 2002 FIFA World Cup
http://fifaworldcup.yahoo.com

--
To unsubscribe, e-mail: <mailto:lucene-user-unsubscribe@jakarta.apache.org>
For additional commands, e-mail: <mailto:lucene-user-help@jakarta.apache.org>


--
To unsubscribe, e-mail: <mailto:lucene-user-unsubscribe@jakarta.apache.org>
For additional commands, e-mail: <mailto:lucene-user-help@jakarta.apache.org>
RE: IndexReader Pool [ In reply to ]
Hi Péter,

>
http://www.mail-archive.com/lucene-dev@jakarta.apache.org/msg00938.html

Something wrong with the code or i'm missing something. WRITER_PATHS is
always empty, i.e. there is no reuse, you always create a new object.


Regards,
Ilya


--
To unsubscribe, e-mail: <mailto:lucene-user-unsubscribe@jakarta.apache.org>
For additional commands, e-mail: <mailto:lucene-user-help@jakarta.apache.org>
RE: IndexReader Pool [ In reply to ]
You are correct. Actually, there have been a few bug fixes since that was
posted. Here's a diff to an updated version:

@@ -19,11 +19,21 @@
*/
public class IndexAccessControl
{
- public static final Analyzer LUCENE_ANALYZER = new LuceneAnalyzer();
+ private static Analyzer s_defaultAnalyzer;
private static final Map WRITER_PATHS = new HashMap(); // path ->
CheckoutI
nfo
private static final Map SEARCHER_PATHS = new HashMap(); // path ->
Searche
r
private static final Map OLD_SEARCHERS = new HashMap(); // Searcher ->
Chec
koutInfo

+ public static void setDefaultAnalyzer(Analyzer analyzer)
+ {
+ s_defaultAnalyzer = analyzer;
+ }
+
+ public static Analyzer getDefaultAnalyzer()
+ {
+ return s_defaultAnalyzer;
+ }
+
/** get for adding documents.
* blocks: readers until released
*/
@@ -47,7 +57,7 @@
{
try
{
- info.wait(); // wait for info to be released
+ synchronized (info) { info.wait(); } // wait
for in
fo to be released
}
catch (InterruptedException e)
{
@@ -61,9 +71,10 @@
{
boolean missing = !path.exists();
if (missing) path.mkdir();
- writer = new IndexWriter(path, LUCENE_ANALYZER,
/*create*/m
issing);
+ writer = new IndexWriter(path, s_defaultAnalyzer,
/*create*
/missing);
writer.mergeFactor = 2;
info = new CheckoutInfo(writer);
+ WRITER_PATHS.put(path, info);
}
}
while (writer == null);
@@ -88,7 +99,7 @@
{
WRITER_PATHS.remove(path);
writer = info.writer;
- info.notify(); // notify waiters to try again
+ synchronized (info) { info.notify(); } // notify
waiters to
try again
}
}
}
@@ -137,10 +148,12 @@
String sync = path.getAbsolutePath().intern();
synchronized (sync) // sync on specific index
{
+ boolean old = false;
CheckoutInfo info = (CheckoutInfo)SEARCHER_PATHS.get(path);
if (info == null || searcher != info.searcher) // this isn't
the in
fo we're looking for
{
info = (CheckoutInfo)OLD_SEARCHERS.get(searcher);
+ old = true;
}
if (info != null) // found a searcher
{
@@ -148,7 +161,7 @@
{
info.checkoutCount--;
}
- else // last reference to searcher
+ else if (old)// last reference to old searcher
{
info.searcher.close();
}
@@ -183,7 +196,7 @@
{
try
{
- info.wait(); // wait for info to be released
+ synchronized (info) { info.wait(); } // wait
for in
fo to be released
}
catch (InterruptedException e)
{
@@ -221,7 +234,7 @@
{
WRITER_PATHS.remove(path);
reader = info.reader;
- info.notify(); // notify waiters to try again
+ synchronized (info) { info.notify(); } // notify
waiters to
try again
}
}
}

Hope it helps,
Scott

> -----Original Message-----
> From: Ilya Khandamirov [mailto:ikh@startext.de]
> Sent: Friday, July 05, 2002 12:13 PM
> To: 'Lucene Users List'
> Subject: RE: IndexReader Pool
>
>
> Hi Péter,
>
> >
> http://www.mail-archive.com/lucene-dev@jakarta.apache.org/msg0
> 0938.html
>
> Something wrong with the code or i'm missing something.
> WRITER_PATHS is
> always empty, i.e. there is no reuse, you always create a new object.
>
>
> Regards,
> Ilya
>
>
> --
> To unsubscribe, e-mail:
> <mailto:lucene-user-unsubscribe@jakarta.apache.org>
> For additional commands, e-mail:
> <mailto:lucene-user-help@jakarta.apache.org>
>
RE: IndexReader Pool [ In reply to ]
>You are correct. Actually, there have been a few bug fixes since that
was posted.
>Here's a diff to an updated version:

Well, i do not see your actual version of this file, but it looks like
now you have two "synchronized" blocks:

synchronized ( sync )
...
synchronized ( info )

This may produce deadlocks in a multithreading environment. Have you
already solved this problem or i should take a closer look at it?


>Hope it helps,

Sure. Thank you.


>Scott

Regards,
Ilya



--
To unsubscribe, e-mail: <mailto:lucene-user-unsubscribe@jakarta.apache.org>
For additional commands, e-mail: <mailto:lucene-user-help@jakarta.apache.org>
RE: IndexReader Pool [ In reply to ]
Deadlocks could be created if the order in which locks are obtained is not
consistent. Note, though, that the locks are obtained in the same order
each time throughout. (BTW: The inner lock is merely needed because the
wait/notify calls need to own the monitor.)

Naturally, you are free to make any suggestions for improvement! :)

Scott

> -----Original Message-----
> From: Ilya Khandamirov [mailto:ikh@startext.de]
> Sent: Saturday, July 06, 2002 11:24 AM
> To: 'Lucene Users List'
> Subject: RE: IndexReader Pool
>
>
> >You are correct. Actually, there have been a few bug fixes
> since that
> was posted.
> >Here's a diff to an updated version:
>
> Well, i do not see your actual version of this file, but it looks like
> now you have two "synchronized" blocks:
>
> synchronized ( sync )
> ...
> synchronized ( info )
>
> This may produce deadlocks in a multithreading environment. Have you
> already solved this problem or i should take a closer look at it?
>
>
> >Hope it helps,
>
> Sure. Thank you.
>
>
> >Scott
>
> Regards,
> Ilya
>
>
>
> --
> To unsubscribe, e-mail:
> <mailto:lucene-user-unsubscribe@jakarta.apache.org>
> For additional commands, e-mail:
> <mailto:lucene-user-help@jakarta.apache.org>
>
RE: IndexReader Pool [ In reply to ]
Hi Scott,

I get deadlocks with the following test code:

File indDir = new File( "C:\\index" );
for ( int i = 0; i < 49; i++ )
{
IndexReader reader = IndexAccessControl.getReader( indDir );
Thread t = new Thread()
{
public void run()
{
File indDir = new File( "C:\\index" );
try
{
IndexWriter writer = IndexAccessControl.getWriter( indDir );
IndexAccessControl.releaseWriter( indDir, writer );
}
catch ( IOException e )
{
e.printStackTrace();
}
}
};
t.start();
System.out.println( i );
IndexAccessControl.releaseReader( indDir, reader );
}


To avoid that i decided to synchronize (and wait/notify) only on sync.
Since that it looks like it works correctly. What do you think?


Regards,
Ilya



-----Original Message-----
From: Scott Ganyo [mailto:scott.ganyo@eTapestry.com]
Sent: Montag, 8. Juli 2002 16:28
To: 'Lucene Users List'
Subject: RE: IndexReader Pool


Deadlocks could be created if the order in which locks are obtained is
not consistent. Note, though, that the locks are obtained in the same
order each time throughout. (BTW: The inner lock is merely needed
because the wait/notify calls need to own the monitor.)

Naturally, you are free to make any suggestions for improvement! :)

Scott


--
To unsubscribe, e-mail: <mailto:lucene-user-unsubscribe@jakarta.apache.org>
For additional commands, e-mail: <mailto:lucene-user-help@jakarta.apache.org>
RE: IndexReader Pool [ In reply to ]
hoops! you are right! i was on holiday last week but I can check the code on the weekend.

peter

> -----Original Message-----
> From: Ilya Khandamirov [mailto:ikh@startext.de]
> Sent: Monday, July 08, 2002 6:00 PM
> To: 'Lucene Users List'
> Subject: RE: IndexReader Pool
>
>
> Hi Scott,
>
> I get deadlocks with the following test code:
>
> File indDir = new File( "C:\\index" );
> for ( int i = 0; i < 49; i++ )
> {
> IndexReader reader = IndexAccessControl.getReader( indDir );
> Thread t = new Thread()
> {
> public void run()
> {
> File indDir = new File( "C:\\index" );
> try
> {
> IndexWriter writer = IndexAccessControl.getWriter( indDir );
> IndexAccessControl.releaseWriter( indDir, writer );
> }
> catch ( IOException e )
> {
> e.printStackTrace();
> }
> }
> };
> t.start();
> System.out.println( i );
> IndexAccessControl.releaseReader( indDir, reader );
> }
>
>
> To avoid that i decided to synchronize (and wait/notify) only on sync.
> Since that it looks like it works correctly. What do you think?
>
>
> Regards,
> Ilya
>
>
>
> -----Original Message-----
> From: Scott Ganyo [mailto:scott.ganyo@eTapestry.com]
> Sent: Montag, 8. Juli 2002 16:28
> To: 'Lucene Users List'
> Subject: RE: IndexReader Pool
>
>
> Deadlocks could be created if the order in which locks are obtained is
> not consistent. Note, though, that the locks are obtained in the same
> order each time throughout. (BTW: The inner lock is merely needed
> because the wait/notify calls need to own the monitor.)
>
> Naturally, you are free to make any suggestions for improvement! :)
>
> Scott
>
>
> --
> To unsubscribe, e-mail:
> <mailto:lucene-user-unsubscribe@jakarta.apache.org>
> For additional commands, e-mail:
> <mailto:lucene-user-help@jakarta.apache.org>
>
>

--
To unsubscribe, e-mail: <mailto:lucene-user-unsubscribe@jakarta.apache.org>
For additional commands, e-mail: <mailto:lucene-user-help@jakarta.apache.org>