Mailing List Archive: Converting a FSDirectory (on disk index) to a RAMDirectory

Converting a FSDirectory (on disk index) to a RAMDirectory

Feb 21, 2002, 2:40 PM

Post #1 of 3 (522 views)

Inspired by the talk on slashdot recently about google using solid state
("ram")
disks I wanted to play around w/ Lucene runnning purely out of memory.

Could anyone glance at this and verify that this code is correct.
Goal is to convert an existing, on-disk, index to a RAMDirectory, which
presumably is purely in memory.

If the code is correct I'd suggest someone w/ CVS powers adding it to
the
source base - maybe a static method in RAMDirectory itself.

public static RAMDirectory convert( String path)
throws IOException
{
final RAMDirectory ram = new RAMDirectory();
final Directory d = FSDirectory.getDirectory( path, false);
final String[] ar = d.list();
for ( int i = 0; i< ar.length; i++)
{
// make place on ram disk
OutputStream os = ram.createFile( ar[ i]);
// read current file
InputStream is = d.openFile( ar[ i]);
// and copy to ram disk
int len = (int) is.length();
byte[] buf = new byte[ len];
is.readBytes( buf, 0, len);
os.writeBytes( buf, len);
// graceful cleanup
is.close();
os.close();
}
return ram;
}

--
To unsubscribe, e-mail: <mailto:lucene-dev-unsubscribe@jakarta.apache.org>
For additional commands, e-mail: <mailto:lucene-dev-help@jakarta.apache.org>

RE: Converting a FSDirectory (on disk index) to a RAMDirectory [ In reply to ]

DCutting at grandcentral

Feb 21, 2002, 2:33 PM

Post #2 of 3 (496 views)

Permalink

> From: Spencer, Dave [mailto:dave@lumos.com]
>
> Could anyone glance at this and verify that this code is correct.
> Goal is to convert an existing, on-disk, index to a
> RAMDirectory, which presumably is purely in memory.

It looks right to me. Did you test it? Did it work?

> If the code is correct I'd suggest someone w/ CVS powers adding it to
> the source base - maybe a static method in RAMDirectory itself.

How about a RAMDirectory constructor? Since only generic Directory methods
are required it could just be:

public RAMDirectory(Directory dirToCopy) { ... }

and, as conveniences:

public RAMDirectory(File f) { this(new FSDirectory(f)); }
public RAMDirectory(String s) { this(new FSDirectory(s)); }

Doug

--
To unsubscribe, e-mail: <mailto:lucene-dev-unsubscribe@jakarta.apache.org>
For additional commands, e-mail: <mailto:lucene-dev-help@jakarta.apache.org>

RE: Converting a FSDirectory (on disk index) to a RAMDirectory [ In reply to ]

dave at lumos

Feb 21, 2002, 6:43 PM

Post #3 of 3 (505 views)

Permalink

Have tested it and yes it works.
I'm trying to prove that
(a) the jvm eats up more memory and [just to "prove" that it's
putting the index in ram]
(b) the queries are at least as fast as without using it, esp
when compared to
a, um, "hot" query (e.g. do same query 2x in a row, 1st
query presumably
lets lucene cache info, 2nd query is then "hot" and of
interest - if caching
works well could be same speed as RAMDirectory, would
never expect this
to be faster).
Am working w/ a 10MB index of around 15k "docs" [index formed from mysql
database].
Will try to report back on the above benchmark/proof - trying to quickly
reply
to Doug w/ a "yes it does work" in answer to his question below.

How to expose this code fragment: I'm in agreement w/ the ctr's below.
The "requirement" to satisfy would just be that it's easy to convert a
on-disk index into a RAMDirectory. The use case would be a search server
in which
you wanted to eat RAM to have fast responses.

-----Original Message-----
From: Doug Cutting [mailto:DCutting@grandcentral.com]
Sent: Thursday, February 21, 2002 1:33 PM
To: 'Lucene Developers List'
Subject: RE: Converting a FSDirectory (on disk index) to a RAMDirectory

> From: Spencer, Dave [mailto:dave@lumos.com]
>
> Could anyone glance at this and verify that this code is correct.
> Goal is to convert an existing, on-disk, index to a
> RAMDirectory, which presumably is purely in memory.

It looks right to me. Did you test it? Did it work?

> If the code is correct I'd suggest someone w/ CVS powers adding it to
> the source base - maybe a static method in RAMDirectory itself.

How about a RAMDirectory constructor? Since only generic Directory
methods
are required it could just be:

public RAMDirectory(Directory dirToCopy) { ... }

and, as conveniences:

public RAMDirectory(File f) { this(new FSDirectory(f)); }
public RAMDirectory(String s) { this(new FSDirectory(s)); }

Doug

--
To unsubscribe, e-mail:
<mailto:lucene-dev-unsubscribe@jakarta.apache.org>
For additional commands, e-mail:
<mailto:lucene-dev-help@jakarta.apache.org>

--
To unsubscribe, e-mail: <mailto:lucene-dev-unsubscribe@jakarta.apache.org>
For additional commands, e-mail: <mailto:lucene-dev-help@jakarta.apache.org>