Mailing List Archive

Winton Davies wrote:
> I've an index that should fit in RAM -- but I need it stored as a
> file. Is there anyway to read in the regular directory in the the
> RAMDirectory and then create an IndexSearcher from this ? It seems
> that there should be, but I can't find a method to construct a
> RAMDirectory from a regular Directory ?

Look at IndexWriter.addIndexes( Directory[] dirs )

You can store your RAMDirectory into a new FSDirectory and load a
FSDircetory into a new RAMDirectory.

Something like:

Directory loadIndex() {
Directory fs = FSDirectory.getDirectory( pathToIndex, false );
RAMDirectory ramdir = new RAMDircetory();
IndexWriter iw = new IndexWriter( ramdir, myAnalyzer, false );
iw.addIndexes( new Directory[]{fs} );
iw.close();
fs.close();
return ramdir;
}
void storeIndex( Directory ramdir ) {
Directory fs = FSDirectory.getDirectory( pathToIndex, true ); // create
a new Index here
IndexWriter iw = new IndexWriter( fs, myAnalyzer, true );
iw.addIndexes( new Directory[]{ramdir} );
iw.close();
fs.close();
}


HTH,
Gerhard

--
To unsubscribe, e-mail: <mailto:lucene-user-unsubscribe@jakarta.apache.org>
For additional commands, e-mail: <mailto:lucene-user-help@jakarta.apache.org>

Many thanks Gerhard!

Winton

Winton Davies
Lead Engineer, Overture (NSDQ: OVER)
1820 Gateway Drive, Suite 360
San Mateo, CA 94404
work: (650) 403-2259
cell: (650) 867-1598
http://www.overture.com/

--
To unsubscribe, e-mail: <mailto:lucene-user-unsubscribe@jakarta.apache.org>
For additional commands, e-mail: <mailto:lucene-user-help@jakarta.apache.org>

Hi again (sorry for the volume of question!)

I implemented this, with one change (the IndexWriter has to have TRUE
set at the end, as ramdiris a new directory).

It loaded into memory, but then when I tried a search I got this stack trace:

java.lang.ArrayIndexOutOfBoundsException
at java.lang.System.arraycopy(Native Method)
at org.apache.lucene.store.RAMInputStream.readInternal(Unknown Source)
at org.apache.lucene.store.InputStream.readBytes(Unknown Source)
at org.apache.lucene.index.SegmentReader.norms(Unknown Source)
at org.apache.lucene.index.SegmentReader.norms(Unknown Source)
at org.apache.lucene.search.TermQuery.scorer(Unknown Source)
at org.apache.lucene.search.Query.scorer(Unknown Source)
at org.apache.lucene.search.IndexSearcher.search(Unknown Source)
at org.apache.lucene.search.Hits.getMoreDocs(Unknown Source)
at org.apache.lucene.search.Hits.<init>(Unknown Source)
at org.apache.lucene.search.Searcher.search(Unknown Source)
at org.apache.lucene.search.Searcher.search(Unknown Source)
at
com.go2.idxsearch.godzilla.MiniSearchListings.processOneQueryStringArr
ay(MiniSearchListings.java:175)
at
com.go2.idxsearch.godzilla.FastGodzilla.callSearchResult(FastGodzilla.
java:55)
at
com.go2.idxsearch.godzilla.FastGodzilla.setSortType(FastGodzilla.java:
199)

Anyone have any ideas why it could fail ?

Cheers,
Winton


>Winton Davies wrote:
>> I've an index that should fit in RAM -- but I need it stored as a
>> file. Is there anyway to read in the regular directory in the the
>> RAMDirectory and then create an IndexSearcher from this ? It seems
>> that there should be, but I can't find a method to construct a
>> RAMDirectory from a regular Directory ?
>
>Look at IndexWriter.addIndexes( Directory[] dirs )
>
>You can store your RAMDirectory into a new FSDirectory and load a
>FSDircetory into a new RAMDirectory.
>
>Something like:
>
>Directory loadIndex() {
> Directory fs = FSDirectory.getDirectory( pathToIndex, false );
> RAMDirectory ramdir = new RAMDircetory();
> IndexWriter iw = new IndexWriter( ramdir, myAnalyzer, false );
> iw.addIndexes( new Directory[]{fs} );
> iw.close();
> fs.close();
> return ramdir;
>}
>void storeIndex( Directory ramdir ) {
> Directory fs = FSDirectory.getDirectory( pathToIndex, true ); // create
>a new Index here
> IndexWriter iw = new IndexWriter( fs, myAnalyzer, true );
> iw.addIndexes( new Directory[]{ramdir} );
> iw.close();
> fs.close();
>}
>
>
>HTH,
>Gerhard
>
>--
>To unsubscribe, e-mail: <mailto:lucene-user-unsubscribe@jakarta.apache.org>
>For additional commands, e-mail: <mailto:lucene-user-help@jakarta.apache.org>


Winton Davies
Lead Engineer, Overture (NSDQ: OVER)
1820 Gateway Drive, Suite 360
San Mateo, CA 94404
work: (650) 403-2259
cell: (650) 867-1598
http://www.overture.com/

--
To unsubscribe, e-mail: <mailto:lucene-user-unsubscribe@jakarta.apache.org>
For additional commands, e-mail: <mailto:lucene-user-help@jakarta.apache.org>

> From: Winton Davies [mailto:wdavies@overture.com]
>
> It loaded into memory, but then when I tried a search I got
> this stack trace:
>
> java.lang.ArrayIndexOutOfBoundsException
> at java.lang.System.arraycopy(Native Method)
> at
> org.apache.lucene.store.RAMInputStream.readInternal(Unknown Source)

What version of Lucene are you using? There was a bug in the last release
of RAMDirectory that is fixed in the nightly builds. If that does not solve
your problem, please submit a self-contained test case that reproduces it.

Doug

--
To unsubscribe, e-mail: <mailto:lucene-user-unsubscribe@jakarta.apache.org>
For additional commands, e-mail: <mailto:lucene-user-help@jakarta.apache.org>

Hi Doug,

Thanks! FYI -- I think I may have come up with a delicious
work-around -- that has a
bunch of nice properties:

Make an FS index, tar it up and then untar onto /tmp :)

Works a treat when you have 4GB and you are the only user :)

The really nice property is that you can now stop and start the search
engine without having a huge delay (maybe 20 minutes) while it loads up
the Index (in my case 1gb).

I got a speed up of around 100x or so - but some queries are still
killing my performance metrics. I'm trying to figure out what they are.

Any idea whether you think a RAMdir would be faster than the /tmpfs ?
I am not familiar with how the comparative disk/io operations would work
compared with what is set up inside a RAMdir

Winton


>> From: Winton Davies [mailto:wdavies@overture.com]
>>
>> It loaded into memory, but then when I tried a search I got
>> this stack trace:
>>
>> java.lang.ArrayIndexOutOfBoundsException
>> at java.lang.System.arraycopy(Native Method)
>> at
>> org.apache.lucene.store.RAMInputStream.readInternal(Unknown Source)
>
>What version of Lucene are you using? There was a bug in the last release
>of RAMDirectory that is fixed in the nightly builds. If that does not solve
>your problem, please submit a self-contained test case that reproduces it.
>
>Doug
>
>--
>To unsubscribe, e-mail: <mailto:lucene-user-unsubscribe@jakarta.apache.org>
>For additional commands, e-mail: <mailto:lucene-user-help@jakarta.apache.org>


Winton Davies
Lead Engineer, Overture (NSDQ: OVER)
1820 Gateway Drive, Suite 360
San Mateo, CA 94404
work: (650) 403-2259
cell: (650) 867-1598
http://www.overture.com/

--
To unsubscribe, e-mail: <mailto:lucene-user-unsubscribe@jakarta.apache.org>
For additional commands, e-mail: <mailto:lucene-user-help@jakarta.apache.org>