Mailing List Archive: large index size

On Jul 25, 2008, at 7:25 PM, hao chen wrote:

> After indexing some html files (4.7G), I got a _1.cfs file that is
> 8.4G. Is this normal?

Probably. There's the index files used for lookup/scoring, the stored
fields returned when retrieving hits, and the data used by the
highlighter, which basically duplicates what's in the index files.

If you don't care about highlighting/excerpting and you just want to
fetch titles, set "stored" and "vectorized" to 0 for everything but
the "title" field and you'll cut down significantly on disk usage.

> I only modified the directory of the sample invindex.plx file for my
> indexing

I strongly recommend using a real HTML parser rather than the cheesy
regex tag stripper in the sample app. It's only there because it's
easy to grok at a glance.

Marvin Humphrey
Rectangular Research
http://www.rectangular.com/

_______________________________________________
KinoSearch mailing list
KinoSearch@rectangular.com
http://www.rectangular.com/mailman/listinfo/kinosearch