Hi folks,
I was running the HNSW benchmark today and found some weird results. Want
to share it here and see whether people have any ideas.
The set up is:
the 384 dimension vector that's available in luceneutil, 100k documents.
And lucene main branch.
max_conn=64, fanout=0, beam_width=250
I first tried with the default setting where we use a 1994MB writer buffer,
so with 100k documents, there will be no merge happening and I will have 1
segment at the end.
This gives me 0.755 recall and 101113ms index building time.
Then I tried with 50MB writer buffer and then forcemerge at the last, and
with 100k documents, I'll get several segments (the final index is around
300MB so I guess 5 or 6) before merge, and then merge them into 1 at last.
This gives me 0.692 recall but it took only 81562ms (including 34394ms
doing the merge) to index.
I have also tried disabling the initialize from graph feature (such that
when we merge we always rebuild the whole graph), or change the random
seed, but still get the similar result.
I'm wondering:
1. Why recall drops that much in the later setup?
2. Why index time is way better? I think we still need to rebuild the whole
graph, or maybe it's just because we're using more off-heap memory (and
less heap) when merge (do we?)?
Best
Patrick
I was running the HNSW benchmark today and found some weird results. Want
to share it here and see whether people have any ideas.
The set up is:
the 384 dimension vector that's available in luceneutil, 100k documents.
And lucene main branch.
max_conn=64, fanout=0, beam_width=250
I first tried with the default setting where we use a 1994MB writer buffer,
so with 100k documents, there will be no merge happening and I will have 1
segment at the end.
This gives me 0.755 recall and 101113ms index building time.
Then I tried with 50MB writer buffer and then forcemerge at the last, and
with 100k documents, I'll get several segments (the final index is around
300MB so I guess 5 or 6) before merge, and then merge them into 1 at last.
This gives me 0.692 recall but it took only 81562ms (including 34394ms
doing the merge) to index.
I have also tried disabling the initialize from graph feature (such that
when we merge we always rebuild the whole graph), or change the random
seed, but still get the similar result.
I'm wondering:
1. Why recall drops that much in the later setup?
2. Why index time is way better? I think we still need to rebuild the whole
graph, or maybe it's just because we're using more off-heap memory (and
less heap) when merge (do we?)?
Best
Patrick