Mailing List Archive: Build index using RAMDirectory "out of memory" errors

Build index using RAMDirectory "out of memory" errors

Feb 25, 2002, 12:15 PM

Post #1 of 4 (1062 views)

I have been using Lucene for 3 weeks and it rules.

The indexing process can be slow. So I searched the mailgroup archives
and found example code using RAMDirectory to improve indexing speed.
The example code I found was indexing 100,000 files at a time to the
RAMDirectory before writing to disk.

I tried indexing 10,000 files at a time to the RAMDirectory before writing
to disk. This drastically improved indexing times but sometimes I get
"out of memory" errors. I am indexing text files and adding 9 fields from
an Oracle database.

Environment:
Solaris 2.8 with 1G of ram and 2G of swap
Java 1.3.1
Lucene 1.2-rc4

Any ideas for eliminating the "out of memory" errors ?

--
To unsubscribe, e-mail: <mailto:lucene-user-unsubscribe@jakarta.apache.org>
For additional commands, e-mail: <mailto:lucene-user-help@jakarta.apache.org>

Re: Build index using RAMDirectory "out of memory" errors [ In reply to ]

wdavies at cs

Feb 25, 2002, 12:21 PM

Post #2 of 4 (1038 views)

Permalink

java -Xmx1000m ....

Sorry if you already tried resizing your heap. Actually with 1.3.1
you could go up above a gig, but really swapping aint gonna help much.

Winton

>I have been using Lucene for 3 weeks and it rules.
>
>The indexing process can be slow. So I searched the mailgroup archives
>and found example code using RAMDirectory to improve indexing speed.
>The example code I found was indexing 100,000 files at a time to the
>RAMDirectory before writing to disk.
>
>I tried indexing 10,000 files at a time to the RAMDirectory before writing
>to disk. This drastically improved indexing times but sometimes I get
>"out of memory" errors. I am indexing text files and adding 9 fields from
>an Oracle database.
>
>Environment:
>Solaris 2.8 with 1G of ram and 2G of swap
>Java 1.3.1
>Lucene 1.2-rc4
>
>Any ideas for eliminating the "out of memory" errors ?
>
>
>
>
>--
>To unsubscribe, e-mail: <mailto:lucene-user-unsubscribe@jakarta.apache.org>
>For additional commands, e-mail: <mailto:lucene-user-help@jakarta.apache.org>

--

Winton Davies
Lead Engineer, Overture (NSDQ: OVER)
1820 Gateway Drive, Suite 360
San Mateo, CA 94404
work: (650) 403-2259
cell: (650) 867-1598
http://www.overture.com/

--
To unsubscribe, e-mail: <mailto:lucene-user-unsubscribe@jakarta.apache.org>
For additional commands, e-mail: <mailto:lucene-user-help@jakarta.apache.org>

Re: Build index using RAMDirectory "out of memory" errors [ In reply to ]

ian.lea at blackwell

Feb 25, 2002, 2:53 PM

Post #3 of 4 (1041 views)

Permalink

Have you tried different values for IndexWriter.mergeFactor?
Setting it to 1000 gave me a 10* speed improvement on some
large index some time ago. Not RAMDirectory though.
Your mileage may vary.

--
Ian.

Kurt Vaag wrote:
>
> I have been using Lucene for 3 weeks and it rules.
>
> The indexing process can be slow. So I searched the mailgroup archives
> and found example code using RAMDirectory to improve indexing speed.
> The example code I found was indexing 100,000 files at a time to the
> RAMDirectory before writing to disk.
>
> I tried indexing 10,000 files at a time to the RAMDirectory before writing
> to disk. This drastically improved indexing times but sometimes I get
> "out of memory" errors. I am indexing text files and adding 9 fields from
> an Oracle database.
>
> Environment:
> Solaris 2.8 with 1G of ram and 2G of swap
> Java 1.3.1
> Lucene 1.2-rc4
>
> Any ideas for eliminating the "out of memory" errors ?

--
To unsubscribe, e-mail: <mailto:lucene-user-unsubscribe@jakarta.apache.org>
For additional commands, e-mail: <mailto:lucene-user-help@jakarta.apache.org>

RE: Build index using RAMDirectory "out of memory" errors [ In reply to ]

kgv at knowledgeworkers

Feb 25, 2002, 2:54 PM

Post #4 of 4 (1043 views)

Permalink

Thanks Winton,

Thats what it was. I just assumed java would take all the 1G that
it needed. Didn't realize the default was 64M. Also thanks for not
saying RTFM (which I had done but didn't know what TF to do with the
-Xmx option).

-Kurt

-----Original Message-----
From: Winton Davies [mailto:wdavies@cs.stanford.edu]
Sent: Monday, February 25, 2002 12:22 PM
To: Lucene Users List
Subject: Re: Build index using RAMDirectory "out of memory" errors

java -Xmx1000m ....

Sorry if you already tried resizing your heap. Actually with 1.3.1
you could go up above a gig, but really swapping aint gonna help much.

Winton

>I have been using Lucene for 3 weeks and it rules.
>
>The indexing process can be slow. So I searched the mailgroup archives
>and found example code using RAMDirectory to improve indexing speed.
>The example code I found was indexing 100,000 files at a time to the
>RAMDirectory before writing to disk.
>
>I tried indexing 10,000 files at a time to the RAMDirectory before writing
>to disk. This drastically improved indexing times but sometimes I get
>"out of memory" errors. I am indexing text files and adding 9 fields from
>an Oracle database.
>
>Environment:
>Solaris 2.8 with 1G of ram and 2G of swap
>Java 1.3.1
>Lucene 1.2-rc4
>
>Any ideas for eliminating the "out of memory" errors ?
>
>
>
>
>--
>To unsubscribe, e-mail:
<mailto:lucene-user-unsubscribe@jakarta.apache.org>
>For additional commands, e-mail:
<mailto:lucene-user-help@jakarta.apache.org>

--

Winton Davies
Lead Engineer, Overture (NSDQ: OVER)
1820 Gateway Drive, Suite 360
San Mateo, CA 94404
work: (650) 403-2259
cell: (650) 867-1598
http://www.overture.com/

--
To unsubscribe, e-mail:
<mailto:lucene-user-unsubscribe@jakarta.apache.org>
For additional commands, e-mail:
<mailto:lucene-user-help@jakarta.apache.org>

--
To unsubscribe, e-mail: <mailto:lucene-user-unsubscribe@jakarta.apache.org>
For additional commands, e-mail: <mailto:lucene-user-help@jakarta.apache.org>