Mailing List Archive: optimize broken?

Hi All,

I'm doing some parallel indexing by firing off batches using gearman,
then once a task completes using ->add_invindex() to merge the batch to
a single index. After all batches are merged I optimize the index. This
works pretty reliably with around 27m records.

However, in the latest trunk of KinoSearch, ->finish(optimize => 1) no
longer seems to work. It returns almost immediately and none of the
segments are collapsed. The index is searchable but it's pretty slow.

The test suite runs successfully. Any ideas?

Cheers
~mark

p.s. I was pleased to see indexing speed significantly improved in the
latest trunk. It's now taking perhaps half the time of the
KinoSearch-0.20_051 cpan release and getting close to speeds I'm used to
seeing with Solr.

_______________________________________________
KinoSearch mailing list
KinoSearch@rectangular.com
http://www.rectangular.com/mailman/listinfo/kinosearch

On Aug 14, 2008, at 10:48 AM, Mark Holland wrote:

> However, in the latest trunk of KinoSearch, ->finish(optimize => 1)
> no longer seems to work. It returns almost immediately and none of
> the segments are collapsed. The index is searchable but it's pretty
> slow.
>
> The test suite runs successfully. Any ideas?

Sorry, but I've been unable to duplicate the problem so far. I've
now added a test verifying that optimization is working at least under
normal circumstances (commits 3730-3731, affecting t/213-
segment_merging.t). I'll keep looking, but if you can give me any
other hints, I'd appreciate it.

> p.s. I was pleased to see indexing speed significantly improved in
> the latest trunk. It's now taking perhaps half the time of the
> KinoSearch-0.20_051 cpan release and getting close to speeds I'm
> used to seeing with Solr.

Groovy. A lot has happened since 0.20_051.

Marvin Humphrey
Rectangular Research
http://www.rectangular.com/

_______________________________________________
KinoSearch mailing list
KinoSearch@rectangular.com
http://www.rectangular.com/mailman/listinfo/kinosearch