Mailing List Archive

It makes sense to me. I don't have the full picture, but I did just
implement merging for vector format, and that at least, could be done
fully concurrent with other formats. I expect the same is true of
DocValues, Terms, etc. I'm not sure about the different kinds of
DocValues - they might want to be done together?

On Mon, Jan 25, 2021 at 5:45 AM Dawid Weiss <dawid.weiss@gmail.com> wrote:
>
>
> Hey everyone,
>
> I'm trying to cut the total wall-time of indexing for some fairly large document collections on machines with a high CPU count (> 32 indexing threads). So far my observations are:
>
> 1) I resigned from using the concurrent merge scheduler in favor of "same thread" merging. This means the indexing thread that encounters a merge just does it. The CMS is designed to favor concurrent searches over indexing and it really didn't do anything I needed - in fact, I had to disable most things it offers. I/O throttling and thread stalling are not really practical on fast I/O in the absence of concurrent searches - you can literally just use as many merge threads as needed to saturate the I/O.
>
> 2) It is quite frequent that everything is churning nicely until the last few merges combine huge smaller segments and form a "long-tail" where most cores are just idle... Here comes my question - can we execute the individual "parts" involved in segment merging (the logic inside SegmentMerger) in separate threads? On the surface it looks like these steps can be done independently (even if they're executed sequentially at the moment) but perhaps I'm missing something?
>
> I'd like to ask before I try to tinker with it. Thanks for any feedback.
>
> Dawid

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org

I suppose we should add a CallerRunsMergeScheduler (a new superclass of
SerialMergeScheduler)? Or make this aspect of SMS configurable. We might
use a semaphore to control how many callers can merge at once (1 == SMS of
today, larger for expanded). It might be debatable if it is then "serial"
or not.

I do think it'd be possible to merge parts of a segment at once! That'd be
a cool feature to add.

~ David Smiley
Apache Lucene/Solr Search Developer
http://www.linkedin.com/in/davidwsmiley


On Mon, Jan 25, 2021 at 11:05 AM Michael Sokolov <msokolov@gmail.com> wrote:

> It makes sense to me. I don't have the full picture, but I did just
> implement merging for vector format, and that at least, could be done
> fully concurrent with other formats. I expect the same is true of
> DocValues, Terms, etc. I'm not sure about the different kinds of
> DocValues - they might want to be done together?
>
> On Mon, Jan 25, 2021 at 5:45 AM Dawid Weiss <dawid.weiss@gmail.com> wrote:
> >
> >
> > Hey everyone,
> >
> > I'm trying to cut the total wall-time of indexing for some fairly large
> document collections on machines with a high CPU count (> 32 indexing
> threads). So far my observations are:
> >
> > 1) I resigned from using the concurrent merge scheduler in favor of
> "same thread" merging. This means the indexing thread that encounters a
> merge just does it. The CMS is designed to favor concurrent searches over
> indexing and it really didn't do anything I needed - in fact, I had to
> disable most things it offers. I/O throttling and thread stalling are not
> really practical on fast I/O in the absence of concurrent searches - you
> can literally just use as many merge threads as needed to saturate the I/O.
> >
> > 2) It is quite frequent that everything is churning nicely until the
> last few merges combine huge smaller segments and form a "long-tail" where
> most cores are just idle... Here comes my question - can we execute the
> individual "parts" involved in segment merging (the logic inside
> SegmentMerger) in separate threads? On the surface it looks like these
> steps can be done independently (even if they're executed sequentially at
> the moment) but perhaps I'm missing something?
> >
> > I'd like to ask before I try to tinker with it. Thanks for any feedback.
> >
> > Dawid
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: dev-help@lucene.apache.org
>
>

At least in theory, since the segmentWriteState is shared among these
phases, there could be dependencies, but it seems as if it ought to be
limited to making sure that the FieldInfos are written last? This is
pure speculation, I haven't dug deeply in the code. However, it would
be necessary to have some kind of synchronization on updates to that
state if these were to be run concurrently. If we do this, should we
also handle the various steps in IndexingChain.flush concurrently? I
guess the mechanism fort providing threads to do so might be
different. At least in this case, there do seem to be *some*
dependencies, like between norms and terms?

On Mon, Jan 25, 2021 at 1:58 PM David Smiley <dsmiley@apache.org> wrote:
>
> I suppose we should add a CallerRunsMergeScheduler (a new superclass of SerialMergeScheduler)? Or make this aspect of SMS configurable. We might use a semaphore to control how many callers can merge at once (1 == SMS of today, larger for expanded). It might be debatable if it is then "serial" or not.
>
> I do think it'd be possible to merge parts of a segment at once! That'd be a cool feature to add.
>
> ~ David Smiley
> Apache Lucene/Solr Search Developer
> http://www.linkedin.com/in/davidwsmiley
>
>
> On Mon, Jan 25, 2021 at 11:05 AM Michael Sokolov <msokolov@gmail.com> wrote:
>>
>> It makes sense to me. I don't have the full picture, but I did just
>> implement merging for vector format, and that at least, could be done
>> fully concurrent with other formats. I expect the same is true of
>> DocValues, Terms, etc. I'm not sure about the different kinds of
>> DocValues - they might want to be done together?
>>
>> On Mon, Jan 25, 2021 at 5:45 AM Dawid Weiss <dawid.weiss@gmail.com> wrote:
>> >
>> >
>> > Hey everyone,
>> >
>> > I'm trying to cut the total wall-time of indexing for some fairly large document collections on machines with a high CPU count (> 32 indexing threads). So far my observations are:
>> >
>> > 1) I resigned from using the concurrent merge scheduler in favor of "same thread" merging. This means the indexing thread that encounters a merge just does it. The CMS is designed to favor concurrent searches over indexing and it really didn't do anything I needed - in fact, I had to disable most things it offers. I/O throttling and thread stalling are not really practical on fast I/O in the absence of concurrent searches - you can literally just use as many merge threads as needed to saturate the I/O.
>> >
>> > 2) It is quite frequent that everything is churning nicely until the last few merges combine huge smaller segments and form a "long-tail" where most cores are just idle... Here comes my question - can we execute the individual "parts" involved in segment merging (the logic inside SegmentMerger) in separate threads? On the surface it looks like these steps can be done independently (even if they're executed sequentially at the moment) but perhaps I'm missing something?
>> >
>> > I'd like to ask before I try to tinker with it. Thanks for any feedback.
>> >
>> > Dawid
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: dev-help@lucene.apache.org
>>

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org

Thanks for early feedback.

I freely admit I never had to touch codecs so I'm not sure what ordering
dependencies need to be respected. But it's certainly something I'd like to
look into since that "last" segment merge can now take ~10 minutes on
mostly idle CPU (64 cores, remember...) and I/O. Worth a shot to improve
this.

Dawid

On Mon, Jan 25, 2021 at 10:39 PM Michael Sokolov <msokolov@gmail.com> wrote:

> At least in theory, since the segmentWriteState is shared among these
> phases, there could be dependencies, but it seems as if it ought to be
> limited to making sure that the FieldInfos are written last? This is
> pure speculation, I haven't dug deeply in the code. However, it would
> be necessary to have some kind of synchronization on updates to that
> state if these were to be run concurrently. If we do this, should we
> also handle the various steps in IndexingChain.flush concurrently? I
> guess the mechanism fort providing threads to do so might be
> different. At least in this case, there do seem to be *some*
> dependencies, like between norms and terms?
>
> On Mon, Jan 25, 2021 at 1:58 PM David Smiley <dsmiley@apache.org> wrote:
> >
> > I suppose we should add a CallerRunsMergeScheduler (a new superclass of
> SerialMergeScheduler)? Or make this aspect of SMS configurable. We might
> use a semaphore to control how many callers can merge at once (1 == SMS of
> today, larger for expanded). It might be debatable if it is then "serial"
> or not.
> >
> > I do think it'd be possible to merge parts of a segment at once! That'd
> be a cool feature to add.
> >
> > ~ David Smiley
> > Apache Lucene/Solr Search Developer
> > http://www.linkedin.com/in/davidwsmiley
> >
> >
> > On Mon, Jan 25, 2021 at 11:05 AM Michael Sokolov <msokolov@gmail.com>
> wrote:
> >>
> >> It makes sense to me. I don't have the full picture, but I did just
> >> implement merging for vector format, and that at least, could be done
> >> fully concurrent with other formats. I expect the same is true of
> >> DocValues, Terms, etc. I'm not sure about the different kinds of
> >> DocValues - they might want to be done together?
> >>
> >> On Mon, Jan 25, 2021 at 5:45 AM Dawid Weiss <dawid.weiss@gmail.com>
> wrote:
> >> >
> >> >
> >> > Hey everyone,
> >> >
> >> > I'm trying to cut the total wall-time of indexing for some fairly
> large document collections on machines with a high CPU count (> 32 indexing
> threads). So far my observations are:
> >> >
> >> > 1) I resigned from using the concurrent merge scheduler in favor of
> "same thread" merging. This means the indexing thread that encounters a
> merge just does it. The CMS is designed to favor concurrent searches over
> indexing and it really didn't do anything I needed - in fact, I had to
> disable most things it offers. I/O throttling and thread stalling are not
> really practical on fast I/O in the absence of concurrent searches - you
> can literally just use as many merge threads as needed to saturate the I/O.
> >> >
> >> > 2) It is quite frequent that everything is churning nicely until the
> last few merges combine huge smaller segments and form a "long-tail" where
> most cores are just idle... Here comes my question - can we execute the
> individual "parts" involved in segment merging (the logic inside
> SegmentMerger) in separate threads? On the surface it looks like these
> steps can be done independently (even if they're executed sequentially at
> the moment) but perhaps I'm missing something?
> >> >
> >> > I'd like to ask before I try to tinker with it. Thanks for any
> feedback.
> >> >
> >> > Dawid
> >>
> >> ---------------------------------------------------------------------
> >> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
> >> For additional commands, e-mail: dev-help@lucene.apache.org
> >>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: dev-help@lucene.apache.org
>
>

Parallelizing segment merges would be nice. When indexing a dataset into a
single segment, it is not rare that the final merge down to 1 segment take
longer than indexing just because merging can only use one thread. It's
frustrating to wait for this merge to finish with only one busy core. :-)

Apart from the dependency of postings on norms in order to write impacts that
Mike S mentioned, I can't think of a reason why segment parts can't be
merged in parallel.

For full text collections, I believe that the bottleneck is usually
terms+postings so it might not save much time. Maybe we could also
parallelize on a per-field basis by writing to temporary files and then
copying the raw data to the target segment part. For instance for the
Wikipedia dataset we use for nightly benchmarks, maybe the inverted indexes
for 'title' and 'body' could be merged in parallel this way.


Le lun. 25 janv. 2021 à 22:46, Dawid Weiss <dawid.weiss@gmail.com> a écrit :

>
> Thanks for early feedback.
>
> I freely admit I never had to touch codecs so I'm not sure what ordering
> dependencies need to be respected. But it's certainly something I'd like to
> look into since that "last" segment merge can now take ~10 minutes on
> mostly idle CPU (64 cores, remember...) and I/O. Worth a shot to improve
> this.
>
> Dawid
>
> On Mon, Jan 25, 2021 at 10:39 PM Michael Sokolov <msokolov@gmail.com>
> wrote:
>
>> At least in theory, since the segmentWriteState is shared among these
>> phases, there could be dependencies, but it seems as if it ought to be
>> limited to making sure that the FieldInfos are written last? This is
>> pure speculation, I haven't dug deeply in the code. However, it would
>> be necessary to have some kind of synchronization on updates to that
>> state if these were to be run concurrently. If we do this, should we
>> also handle the various steps in IndexingChain.flush concurrently? I
>> guess the mechanism fort providing threads to do so might be
>> different. At least in this case, there do seem to be *some*
>> dependencies, like between norms and terms?
>>
>> On Mon, Jan 25, 2021 at 1:58 PM David Smiley <dsmiley@apache.org> wrote:
>> >
>> > I suppose we should add a CallerRunsMergeScheduler (a new superclass of
>> SerialMergeScheduler)? Or make this aspect of SMS configurable. We might
>> use a semaphore to control how many callers can merge at once (1 == SMS of
>> today, larger for expanded). It might be debatable if it is then "serial"
>> or not.
>> >
>> > I do think it'd be possible to merge parts of a segment at once!
>> That'd be a cool feature to add.
>> >
>> > ~ David Smiley
>> > Apache Lucene/Solr Search Developer
>> > http://www.linkedin.com/in/davidwsmiley
>> >
>> >
>> > On Mon, Jan 25, 2021 at 11:05 AM Michael Sokolov <msokolov@gmail.com>
>> wrote:
>> >>
>> >> It makes sense to me. I don't have the full picture, but I did just
>> >> implement merging for vector format, and that at least, could be done
>> >> fully concurrent with other formats. I expect the same is true of
>> >> DocValues, Terms, etc. I'm not sure about the different kinds of
>> >> DocValues - they might want to be done together?
>> >>
>> >> On Mon, Jan 25, 2021 at 5:45 AM Dawid Weiss <dawid.weiss@gmail.com>
>> wrote:
>> >> >
>> >> >
>> >> > Hey everyone,
>> >> >
>> >> > I'm trying to cut the total wall-time of indexing for some fairly
>> large document collections on machines with a high CPU count (> 32 indexing
>> threads). So far my observations are:
>> >> >
>> >> > 1) I resigned from using the concurrent merge scheduler in favor of
>> "same thread" merging. This means the indexing thread that encounters a
>> merge just does it. The CMS is designed to favor concurrent searches over
>> indexing and it really didn't do anything I needed - in fact, I had to
>> disable most things it offers. I/O throttling and thread stalling are not
>> really practical on fast I/O in the absence of concurrent searches - you
>> can literally just use as many merge threads as needed to saturate the I/O.
>> >> >
>> >> > 2) It is quite frequent that everything is churning nicely until the
>> last few merges combine huge smaller segments and form a "long-tail" where
>> most cores are just idle... Here comes my question - can we execute the
>> individual "parts" involved in segment merging (the logic inside
>> SegmentMerger) in separate threads? On the surface it looks like these
>> steps can be done independently (even if they're executed sequentially at
>> the moment) but perhaps I'm missing something?
>> >> >
>> >> > I'd like to ask before I try to tinker with it. Thanks for any
>> feedback.
>> >> >
>> >> > Dawid
>> >>
>> >> ---------------------------------------------------------------------
>> >> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
>> >> For additional commands, e-mail: dev-help@lucene.apache.org
>> >>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: dev-help@lucene.apache.org
>>
>>

Yep, I agree. The granularity of this parallelisation can be made finer
with time, once the basic blocks are in place... but you did kill my
initial enthusiasm with the hint at what is the dominant factor in segment
writing. I grepped through the logs of a recent run - and it's almost all
postings and vectors at the top of the cost. These are reverse-sorted by
merge contribution time:

2021-01-19 18:11:52,551 TRACE l4g.lucene.infostream: IW: 16140 msec to
write postings and finish vectors [qt:worker-2:T61]
2021-01-19 18:11:51,902 TRACE l4g.lucene.infostream: IW: 16126 msec to
write postings and finish vectors [qt:worker-2:T12]
2021-01-19 18:11:54,748 TRACE l4g.lucene.infostream: IW: 15758 msec to
write postings and finish vectors [qt:worker-2:T23]
2021-01-19 18:11:54,073 TRACE l4g.lucene.infostream: IW: 15713 msec to
write postings and finish vectors [qt:worker-2:T8]
2021-01-19 18:11:55,012 TRACE l4g.lucene.infostream: IW: 15636 msec to
write postings and finish vectors [qt:worker-2:T64]
...
[the first not-postings related]
2021-01-19 18:08:43,697 TRACE l4g.lucene.infostream: IW: 529 msec to finish
stored fields [qt:worker-2:T8]
2

Darn. That splitting by field sounds all more attractive now!

D.


On Tue, Jan 26, 2021 at 2:29 PM Adrien Grand <jpountz@gmail.com> wrote:

> Parallelizing segment merges would be nice. When indexing a dataset into a
> single segment, it is not rare that the final merge down to 1 segment take
> longer than indexing just because merging can only use one thread. It's
> frustrating to wait for this merge to finish with only one busy core. :-)
>
> Apart from the dependency of postings on norms in order to write impacts that
> Mike S mentioned, I can't think of a reason why segment parts can't be
> merged in parallel.
>
> For full text collections, I believe that the bottleneck is usually
> terms+postings so it might not save much time. Maybe we could also
> parallelize on a per-field basis by writing to temporary files and then
> copying the raw data to the target segment part. For instance for the
> Wikipedia dataset we use for nightly benchmarks, maybe the inverted indexes
> for 'title' and 'body' could be merged in parallel this way.
>
>
> Le lun. 25 janv. 2021 à 22:46, Dawid Weiss <dawid.weiss@gmail.com> a
> écrit :
>
>>
>> Thanks for early feedback.
>>
>> I freely admit I never had to touch codecs so I'm not sure what ordering
>> dependencies need to be respected. But it's certainly something I'd like to
>> look into since that "last" segment merge can now take ~10 minutes on
>> mostly idle CPU (64 cores, remember...) and I/O. Worth a shot to improve
>> this.
>>
>> Dawid
>>
>> On Mon, Jan 25, 2021 at 10:39 PM Michael Sokolov <msokolov@gmail.com>
>> wrote:
>>
>>> At least in theory, since the segmentWriteState is shared among these
>>> phases, there could be dependencies, but it seems as if it ought to be
>>> limited to making sure that the FieldInfos are written last? This is
>>> pure speculation, I haven't dug deeply in the code. However, it would
>>> be necessary to have some kind of synchronization on updates to that
>>> state if these were to be run concurrently. If we do this, should we
>>> also handle the various steps in IndexingChain.flush concurrently? I
>>> guess the mechanism fort providing threads to do so might be
>>> different. At least in this case, there do seem to be *some*
>>> dependencies, like between norms and terms?
>>>
>>> On Mon, Jan 25, 2021 at 1:58 PM David Smiley <dsmiley@apache.org> wrote:
>>> >
>>> > I suppose we should add a CallerRunsMergeScheduler (a new superclass
>>> of SerialMergeScheduler)? Or make this aspect of SMS configurable. We
>>> might use a semaphore to control how many callers can merge at once (1 ==
>>> SMS of today, larger for expanded). It might be debatable if it is then
>>> "serial" or not.
>>> >
>>> > I do think it'd be possible to merge parts of a segment at once!
>>> That'd be a cool feature to add.
>>> >
>>> > ~ David Smiley
>>> > Apache Lucene/Solr Search Developer
>>> > http://www.linkedin.com/in/davidwsmiley
>>> >
>>> >
>>> > On Mon, Jan 25, 2021 at 11:05 AM Michael Sokolov <msokolov@gmail.com>
>>> wrote:
>>> >>
>>> >> It makes sense to me. I don't have the full picture, but I did just
>>> >> implement merging for vector format, and that at least, could be done
>>> >> fully concurrent with other formats. I expect the same is true of
>>> >> DocValues, Terms, etc. I'm not sure about the different kinds of
>>> >> DocValues - they might want to be done together?
>>> >>
>>> >> On Mon, Jan 25, 2021 at 5:45 AM Dawid Weiss <dawid.weiss@gmail.com>
>>> wrote:
>>> >> >
>>> >> >
>>> >> > Hey everyone,
>>> >> >
>>> >> > I'm trying to cut the total wall-time of indexing for some fairly
>>> large document collections on machines with a high CPU count (> 32 indexing
>>> threads). So far my observations are:
>>> >> >
>>> >> > 1) I resigned from using the concurrent merge scheduler in favor of
>>> "same thread" merging. This means the indexing thread that encounters a
>>> merge just does it. The CMS is designed to favor concurrent searches over
>>> indexing and it really didn't do anything I needed - in fact, I had to
>>> disable most things it offers. I/O throttling and thread stalling are not
>>> really practical on fast I/O in the absence of concurrent searches - you
>>> can literally just use as many merge threads as needed to saturate the I/O.
>>> >> >
>>> >> > 2) It is quite frequent that everything is churning nicely until
>>> the last few merges combine huge smaller segments and form a "long-tail"
>>> where most cores are just idle... Here comes my question - can we execute
>>> the individual "parts" involved in segment merging (the logic inside
>>> SegmentMerger) in separate threads? On the surface it looks like these
>>> steps can be done independently (even if they're executed sequentially at
>>> the moment) but perhaps I'm missing something?
>>> >> >
>>> >> > I'd like to ask before I try to tinker with it. Thanks for any
>>> feedback.
>>> >> >
>>> >> > Dawid
>>> >>
>>> >> ---------------------------------------------------------------------
>>> >> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
>>> >> For additional commands, e-mail: dev-help@lucene.apache.org
>>> >>
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
>>> For additional commands, e-mail: dev-help@lucene.apache.org
>>>
>>>

On Tue, Jan 26, 2021 at 8:29 AM Adrien Grand <jpountz@gmail.com> wrote:
> For full text collections, I believe that the bottleneck is usually terms+postings so it might not save much time. Maybe we could also parallelize on a per-field basis by writing to temporary files and then copying the raw data to the target segment part. For instance for the Wikipedia dataset we use for nightly benchmarks, maybe the inverted indexes for 'title' and 'body' could be merged in parallel this way.
>
>

if you want to experiment with something like that, you can hackishly
simulate it today to quickly see the overhead, correct? its a small
hack to PerFieldPostingsFormat to force it to emit "files-per-field"
and then CFS will combine it all together.

but doing it explicitly and then making our own internal "compound"
seems kinda risky, wouldn't all the offsets be wrong without further
file changes (e.g. per-field "start offset" where all the postings for
that field begin) ?

And this does nothing to solve dawid's problem of slow vectors, if you
have vectors on that's always gonna be the bottleneck and those are
per-doc.

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org

+1 to make a single merge concurrent! It is horribly frustrating to watch
that last merge running on a single core :) I have lost many hours of my
life to this frustration.

I do think we need to explore concurrency within terms/postings across
fields in one segment to really see gains in the common case where merge
time is dominated by postings.

Mike McCandless

http://blog.mikemccandless.com


On Tue, Jan 26, 2021 at 9:09 AM Robert Muir <rcmuir@gmail.com> wrote:

> On Tue, Jan 26, 2021 at 8:29 AM Adrien Grand <jpountz@gmail.com> wrote:
> > For full text collections, I believe that the bottleneck is usually
> terms+postings so it might not save much time. Maybe we could also
> parallelize on a per-field basis by writing to temporary files and then
> copying the raw data to the target segment part. For instance for the
> Wikipedia dataset we use for nightly benchmarks, maybe the inverted indexes
> for 'title' and 'body' could be merged in parallel this way.
> >
> >
>
> if you want to experiment with something like that, you can hackishly
> simulate it today to quickly see the overhead, correct? its a small
> hack to PerFieldPostingsFormat to force it to emit "files-per-field"
> and then CFS will combine it all together.
>
> but doing it explicitly and then making our own internal "compound"
> seems kinda risky, wouldn't all the offsets be wrong without further
> file changes (e.g. per-field "start offset" where all the postings for
> that field begin) ?
>
> And this does nothing to solve dawid's problem of slow vectors, if you
> have vectors on that's always gonna be the bottleneck and those are
> per-doc.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: dev-help@lucene.apache.org
>
>

Maybe don't get discouraged too quickly. By default merging stored fields
is super fast because Lucene can copy compressed data directly, but if
there are deletes or index sorting is enabled this optimization is not
applicable anymore and I wouldn't be surprised if stored fields started
taking non negligible time.

Le mar. 26 janv. 2021 à 14:48, Dawid Weiss <dawid.weiss@gmail.com> a écrit :

>
> Yep, I agree. The granularity of this parallelisation can be made finer
> with time, once the basic blocks are in place... but you did kill my
> initial enthusiasm with the hint at what is the dominant factor in segment
> writing. I grepped through the logs of a recent run - and it's almost all
> postings and vectors at the top of the cost. These are reverse-sorted by
> merge contribution time:
>
> 2021-01-19 18:11:52,551 TRACE l4g.lucene.infostream: IW: 16140 msec to
> write postings and finish vectors [qt:worker-2:T61]
> 2021-01-19 18:11:51,902 TRACE l4g.lucene.infostream: IW: 16126 msec to
> write postings and finish vectors [qt:worker-2:T12]
> 2021-01-19 18:11:54,748 TRACE l4g.lucene.infostream: IW: 15758 msec to
> write postings and finish vectors [qt:worker-2:T23]
> 2021-01-19 18:11:54,073 TRACE l4g.lucene.infostream: IW: 15713 msec to
> write postings and finish vectors [qt:worker-2:T8]
> 2021-01-19 18:11:55,012 TRACE l4g.lucene.infostream: IW: 15636 msec to
> write postings and finish vectors [qt:worker-2:T64]
> ...
> [the first not-postings related]
> 2021-01-19 18:08:43,697 TRACE l4g.lucene.infostream: IW: 529 msec to
> finish stored fields [qt:worker-2:T8]
> 2
>
> Darn. That splitting by field sounds all more attractive now!
>
> D.
>
>
> On Tue, Jan 26, 2021 at 2:29 PM Adrien Grand <jpountz@gmail.com> wrote:
>
>> Parallelizing segment merges would be nice. When indexing a dataset into
>> a single segment, it is not rare that the final merge down to 1 segment
>> take longer than indexing just because merging can only use one thread.
>> It's frustrating to wait for this merge to finish with only one busy core.
>> :-)
>>
>> Apart from the dependency of postings on norms in order to write impacts that
>> Mike S mentioned, I can't think of a reason why segment parts can't be
>> merged in parallel.
>>
>> For full text collections, I believe that the bottleneck is usually
>> terms+postings so it might not save much time. Maybe we could also
>> parallelize on a per-field basis by writing to temporary files and then
>> copying the raw data to the target segment part. For instance for the
>> Wikipedia dataset we use for nightly benchmarks, maybe the inverted indexes
>> for 'title' and 'body' could be merged in parallel this way.
>>
>>
>> Le lun. 25 janv. 2021 à 22:46, Dawid Weiss <dawid.weiss@gmail.com> a
>> écrit :
>>
>>>
>>> Thanks for early feedback.
>>>
>>> I freely admit I never had to touch codecs so I'm not sure what ordering
>>> dependencies need to be respected. But it's certainly something I'd like to
>>> look into since that "last" segment merge can now take ~10 minutes on
>>> mostly idle CPU (64 cores, remember...) and I/O. Worth a shot to improve
>>> this.
>>>
>>> Dawid
>>>
>>> On Mon, Jan 25, 2021 at 10:39 PM Michael Sokolov <msokolov@gmail.com>
>>> wrote:
>>>
>>>> At least in theory, since the segmentWriteState is shared among these
>>>> phases, there could be dependencies, but it seems as if it ought to be
>>>> limited to making sure that the FieldInfos are written last? This is
>>>> pure speculation, I haven't dug deeply in the code. However, it would
>>>> be necessary to have some kind of synchronization on updates to that
>>>> state if these were to be run concurrently. If we do this, should we
>>>> also handle the various steps in IndexingChain.flush concurrently? I
>>>> guess the mechanism fort providing threads to do so might be
>>>> different. At least in this case, there do seem to be *some*
>>>> dependencies, like between norms and terms?
>>>>
>>>> On Mon, Jan 25, 2021 at 1:58 PM David Smiley <dsmiley@apache.org>
>>>> wrote:
>>>> >
>>>> > I suppose we should add a CallerRunsMergeScheduler (a new superclass
>>>> of SerialMergeScheduler)? Or make this aspect of SMS configurable. We
>>>> might use a semaphore to control how many callers can merge at once (1 ==
>>>> SMS of today, larger for expanded). It might be debatable if it is then
>>>> "serial" or not.
>>>> >
>>>> > I do think it'd be possible to merge parts of a segment at once!
>>>> That'd be a cool feature to add.
>>>> >
>>>> > ~ David Smiley
>>>> > Apache Lucene/Solr Search Developer
>>>> > http://www.linkedin.com/in/davidwsmiley
>>>> >
>>>> >
>>>> > On Mon, Jan 25, 2021 at 11:05 AM Michael Sokolov <msokolov@gmail.com>
>>>> wrote:
>>>> >>
>>>> >> It makes sense to me. I don't have the full picture, but I did just
>>>> >> implement merging for vector format, and that at least, could be done
>>>> >> fully concurrent with other formats. I expect the same is true of
>>>> >> DocValues, Terms, etc. I'm not sure about the different kinds of
>>>> >> DocValues - they might want to be done together?
>>>> >>
>>>> >> On Mon, Jan 25, 2021 at 5:45 AM Dawid Weiss <dawid.weiss@gmail.com>
>>>> wrote:
>>>> >> >
>>>> >> >
>>>> >> > Hey everyone,
>>>> >> >
>>>> >> > I'm trying to cut the total wall-time of indexing for some fairly
>>>> large document collections on machines with a high CPU count (> 32 indexing
>>>> threads). So far my observations are:
>>>> >> >
>>>> >> > 1) I resigned from using the concurrent merge scheduler in favor
>>>> of "same thread" merging. This means the indexing thread that encounters a
>>>> merge just does it. The CMS is designed to favor concurrent searches over
>>>> indexing and it really didn't do anything I needed - in fact, I had to
>>>> disable most things it offers. I/O throttling and thread stalling are not
>>>> really practical on fast I/O in the absence of concurrent searches - you
>>>> can literally just use as many merge threads as needed to saturate the I/O.
>>>> >> >
>>>> >> > 2) It is quite frequent that everything is churning nicely until
>>>> the last few merges combine huge smaller segments and form a "long-tail"
>>>> where most cores are just idle... Here comes my question - can we execute
>>>> the individual "parts" involved in segment merging (the logic inside
>>>> SegmentMerger) in separate threads? On the surface it looks like these
>>>> steps can be done independently (even if they're executed sequentially at
>>>> the moment) but perhaps I'm missing something?
>>>> >> >
>>>> >> > I'd like to ask before I try to tinker with it. Thanks for any
>>>> feedback.
>>>> >> >
>>>> >> > Dawid
>>>> >>
>>>> >> ---------------------------------------------------------------------
>>>> >> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
>>>> >> For additional commands, e-mail: dev-help@lucene.apache.org
>>>> >>
>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
>>>> For additional commands, e-mail: dev-help@lucene.apache.org
>>>>
>>>>

> +1 to make a single merge concurrent! It is horribly frustrating to watch that last merge running on a single core :) I have lost many hours of my life to this frustration.

> Yeah... it is, isn't it? Especially on new machines where you have super-fast SSDs, countless cores, etc... That last merge consumes so few resources that the computer feels practically idle... it's hard to explain to people using our software (who invested in hardware) why we're basically doing nothing... :)

> I do think we need to explore concurrency within terms/postings across fields in one segment to really see gains in the common case where merge time is dominated by postings.

Yeah, probably.

> if you want to experiment with something like that, you can hackishly simulate it today to quickly see the overhead, correct? its a small hack to PerFieldPostingsFormat to force it to emit "files-per-field" and then CFS will combine it all together.

Good idea, Robert. I'll try this.

> By default merging stored fields is super fast because Lucene can copy compressed data directly, but if there are deletes or index sorting is enabled this optimization is not applicable anymore and I wouldn't be surprised if stored fields started taking non negligible time.

In this case these segments are essentially made from scratch but with
lots and lots of term vectors and postings... But the more parallel
stages we can introduce, the better.

I have some other stuff on my plate before I can dive deep into this
but I eventually will. Thanks for the pointers, everyone - helpful!

D.

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org

Oh I found this long ago (well, ~2 years) issue exploring this:
https://issues.apache.org/jira/browse/LUCENE-8580

Mike McCandless

http://blog.mikemccandless.com


On Tue, Jan 26, 2021 at 3:38 PM Dawid Weiss <dawid.weiss@gmail.com> wrote:

> > +1 to make a single merge concurrent! It is horribly frustrating to
> watch that last merge running on a single core :) I have lost many hours
> of my life to this frustration.
>
> > Yeah... it is, isn't it? Especially on new machines where you have
> super-fast SSDs, countless cores, etc... That last merge consumes so few
> resources that the computer feels practically idle... it's hard to explain
> to people using our software (who invested in hardware) why we're basically
> doing nothing... :)
>
> > I do think we need to explore concurrency within terms/postings across
> fields in one segment to really see gains in the common case where merge
> time is dominated by postings.
>
> Yeah, probably.
>
> > if you want to experiment with something like that, you can hackishly
> simulate it today to quickly see the overhead, correct? its a small hack to
> PerFieldPostingsFormat to force it to emit "files-per-field" and then CFS
> will combine it all together.
>
> Good idea, Robert. I'll try this.
>
> > By default merging stored fields is super fast because Lucene can copy
> compressed data directly, but if there are deletes or index sorting is
> enabled this optimization is not applicable anymore and I wouldn't be
> surprised if stored fields started taking non negligible time.
>
> In this case these segments are essentially made from scratch but with
> lots and lots of term vectors and postings... But the more parallel
> stages we can introduce, the better.
>
> I have some other stuff on my plate before I can dive deep into this
> but I eventually will. Thanks for the pointers, everyone - helpful!
>
> D.
>

LOL and it was Dawid :-) Having amnesia Dawid?
I think I've re-explored my own ideas before too.

~ David Smiley
Apache Lucene/Solr Search Developer
http://www.linkedin.com/in/davidwsmiley


On Tue, Jan 26, 2021 at 5:39 PM Michael McCandless <
lucene@mikemccandless.com> wrote:

> Oh I found this long ago (well, ~2 years) issue exploring this:
> https://issues.apache.org/jira/browse/LUCENE-8580
>
> Mike McCandless
>
> http://blog.mikemccandless.com
>
>
> On Tue, Jan 26, 2021 at 3:38 PM Dawid Weiss <dawid.weiss@gmail.com> wrote:
>
>> > +1 to make a single merge concurrent! It is horribly frustrating to
>> watch that last merge running on a single core :) I have lost many hours
>> of my life to this frustration.
>>
>> > Yeah... it is, isn't it? Especially on new machines where you have
>> super-fast SSDs, countless cores, etc... That last merge consumes so few
>> resources that the computer feels practically idle... it's hard to explain
>> to people using our software (who invested in hardware) why we're basically
>> doing nothing... :)
>>
>> > I do think we need to explore concurrency within terms/postings across
>> fields in one segment to really see gains in the common case where merge
>> time is dominated by postings.
>>
>> Yeah, probably.
>>
>> > if you want to experiment with something like that, you can hackishly
>> simulate it today to quickly see the overhead, correct? its a small hack to
>> PerFieldPostingsFormat to force it to emit "files-per-field" and then CFS
>> will combine it all together.
>>
>> Good idea, Robert. I'll try this.
>>
>> > By default merging stored fields is super fast because Lucene can copy
>> compressed data directly, but if there are deletes or index sorting is
>> enabled this optimization is not applicable anymore and I wouldn't be
>> surprised if stored fields started taking non negligible time.
>>
>> In this case these segments are essentially made from scratch but with
>> lots and lots of term vectors and postings... But the more parallel
>> stages we can introduce, the better.
>>
>> I have some other stuff on my plate before I can dive deep into this
>> but I eventually will. Thanks for the pointers, everyone - helpful!
>>
>> D.
>>
>

Darn... I swear sometimes, when I try hard enough, I can hear my brain
cells giving up to atrophy... Sigh.


D.

On Wed, Jan 27, 2021 at 4:44 AM David Smiley <dsmiley@apache.org> wrote:
>
> LOL and it was Dawid :-) Having amnesia Dawid?
> I think I've re-explored my own ideas before too.
>
> ~ David Smiley
> Apache Lucene/Solr Search Developer
> http://www.linkedin.com/in/davidwsmiley
>
>
> On Tue, Jan 26, 2021 at 5:39 PM Michael McCandless <lucene@mikemccandless.com> wrote:
>>
>> Oh I found this long ago (well, ~2 years) issue exploring this: https://issues.apache.org/jira/browse/LUCENE-8580
>>
>> Mike McCandless
>>
>> http://blog.mikemccandless.com
>>
>>
>> On Tue, Jan 26, 2021 at 3:38 PM Dawid Weiss <dawid.weiss@gmail.com> wrote:
>>>
>>> > +1 to make a single merge concurrent! It is horribly frustrating to watch that last merge running on a single core :) I have lost many hours of my life to this frustration.
>>>
>>> > Yeah... it is, isn't it? Especially on new machines where you have super-fast SSDs, countless cores, etc... That last merge consumes so few resources that the computer feels practically idle... it's hard to explain to people using our software (who invested in hardware) why we're basically doing nothing... :)
>>>
>>> > I do think we need to explore concurrency within terms/postings across fields in one segment to really see gains in the common case where merge time is dominated by postings.
>>>
>>> Yeah, probably.
>>>
>>> > if you want to experiment with something like that, you can hackishly simulate it today to quickly see the overhead, correct? its a small hack to PerFieldPostingsFormat to force it to emit "files-per-field" and then CFS will combine it all together.
>>>
>>> Good idea, Robert. I'll try this.
>>>
>>> > By default merging stored fields is super fast because Lucene can copy compressed data directly, but if there are deletes or index sorting is enabled this optimization is not applicable anymore and I wouldn't be surprised if stored fields started taking non negligible time.
>>>
>>> In this case these segments are essentially made from scratch but with
>>> lots and lots of term vectors and postings... But the more parallel
>>> stages we can introduce, the better.
>>>
>>> I have some other stuff on my plate before I can dive deep into this
>>> but I eventually will. Thanks for the pointers, everyone - helpful!
>>>
>>> D.

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org

I thought I remembered the discussion, searched for the issue in jira, but
could not find. Probably Mike used his souped up search?

On Wed, Jan 27, 2021, 3:07 AM Dawid Weiss <dawid.weiss@gmail.com> wrote:

> Darn... I swear sometimes, when I try hard enough, I can hear my brain
> cells giving up to atrophy... Sigh.
>
>
> D.
>
> On Wed, Jan 27, 2021 at 4:44 AM David Smiley <dsmiley@apache.org> wrote:
> >
> > LOL and it was Dawid :-) Having amnesia Dawid?
> > I think I've re-explored my own ideas before too.
> >
> > ~ David Smiley
> > Apache Lucene/Solr Search Developer
> > http://www.linkedin.com/in/davidwsmiley
> >
> >
> > On Tue, Jan 26, 2021 at 5:39 PM Michael McCandless <
> lucene@mikemccandless.com> wrote:
> >>
> >> Oh I found this long ago (well, ~2 years) issue exploring this:
> https://issues.apache.org/jira/browse/LUCENE-8580
> >>
> >> Mike McCandless
> >>
> >> http://blog.mikemccandless.com
> >>
> >>
> >> On Tue, Jan 26, 2021 at 3:38 PM Dawid Weiss <dawid.weiss@gmail.com>
> wrote:
> >>>
> >>> > +1 to make a single merge concurrent! It is horribly frustrating to
> watch that last merge running on a single core :) I have lost many hours
> of my life to this frustration.
> >>>
> >>> > Yeah... it is, isn't it? Especially on new machines where you have
> super-fast SSDs, countless cores, etc... That last merge consumes so few
> resources that the computer feels practically idle... it's hard to explain
> to people using our software (who invested in hardware) why we're basically
> doing nothing... :)
> >>>
> >>> > I do think we need to explore concurrency within terms/postings
> across fields in one segment to really see gains in the common case where
> merge time is dominated by postings.
> >>>
> >>> Yeah, probably.
> >>>
> >>> > if you want to experiment with something like that, you can
> hackishly simulate it today to quickly see the overhead, correct? its a
> small hack to PerFieldPostingsFormat to force it to emit "files-per-field"
> and then CFS will combine it all together.
> >>>
> >>> Good idea, Robert. I'll try this.
> >>>
> >>> > By default merging stored fields is super fast because Lucene can
> copy compressed data directly, but if there are deletes or index sorting is
> enabled this optimization is not applicable anymore and I wouldn't be
> surprised if stored fields started taking non negligible time.
> >>>
> >>> In this case these segments are essentially made from scratch but with
> >>> lots and lots of term vectors and postings... But the more parallel
> >>> stages we can introduce, the better.
> >>>
> >>> I have some other stuff on my plate before I can dive deep into this
> >>> but I eventually will. Thanks for the pointers, everyone - helpful!
> >>>
> >>> D.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: dev-help@lucene.apache.org
>
>

LOL Mike did use http://jirasearch.mikemccandless.com, our dog food Lucene
search application demonstrating many of Lucene's features (
http://blog.mikemccandless.com/2016/10/jiraseseach-20-dog-food-using-lucene-to.html),
but it was NOT easy to find!

I think I had one lonely brain cell still insisting we had indeed talked
about this somewhat recently :)

Mike McCandless

http://blog.mikemccandless.com


On Wed, Jan 27, 2021 at 6:43 AM Michael Sokolov <msokolov@gmail.com> wrote:

> I thought I remembered the discussion, searched for the issue in jira, but
> could not find. Probably Mike used his souped up search?
>
> On Wed, Jan 27, 2021, 3:07 AM Dawid Weiss <dawid.weiss@gmail.com> wrote:
>
>> Darn... I swear sometimes, when I try hard enough, I can hear my brain
>> cells giving up to atrophy... Sigh.
>>
>>
>> D.
>>
>> On Wed, Jan 27, 2021 at 4:44 AM David Smiley <dsmiley@apache.org> wrote:
>> >
>> > LOL and it was Dawid :-) Having amnesia Dawid?
>> > I think I've re-explored my own ideas before too.
>> >
>> > ~ David Smiley
>> > Apache Lucene/Solr Search Developer
>> > http://www.linkedin.com/in/davidwsmiley
>> >
>> >
>> > On Tue, Jan 26, 2021 at 5:39 PM Michael McCandless <
>> lucene@mikemccandless.com> wrote:
>> >>
>> >> Oh I found this long ago (well, ~2 years) issue exploring this:
>> https://issues.apache.org/jira/browse/LUCENE-8580
>> >>
>> >> Mike McCandless
>> >>
>> >> http://blog.mikemccandless.com
>> >>
>> >>
>> >> On Tue, Jan 26, 2021 at 3:38 PM Dawid Weiss <dawid.weiss@gmail.com>
>> wrote:
>> >>>
>> >>> > +1 to make a single merge concurrent! It is horribly frustrating
>> to watch that last merge running on a single core :) I have lost many
>> hours of my life to this frustration.
>> >>>
>> >>> > Yeah... it is, isn't it? Especially on new machines where you have
>> super-fast SSDs, countless cores, etc... That last merge consumes so few
>> resources that the computer feels practically idle... it's hard to explain
>> to people using our software (who invested in hardware) why we're basically
>> doing nothing... :)
>> >>>
>> >>> > I do think we need to explore concurrency within terms/postings
>> across fields in one segment to really see gains in the common case where
>> merge time is dominated by postings.
>> >>>
>> >>> Yeah, probably.
>> >>>
>> >>> > if you want to experiment with something like that, you can
>> hackishly simulate it today to quickly see the overhead, correct? its a
>> small hack to PerFieldPostingsFormat to force it to emit "files-per-field"
>> and then CFS will combine it all together.
>> >>>
>> >>> Good idea, Robert. I'll try this.
>> >>>
>> >>> > By default merging stored fields is super fast because Lucene can
>> copy compressed data directly, but if there are deletes or index sorting is
>> enabled this optimization is not applicable anymore and I wouldn't be
>> surprised if stored fields started taking non negligible time.
>> >>>
>> >>> In this case these segments are essentially made from scratch but with
>> >>> lots and lots of term vectors and postings... But the more parallel
>> >>> stages we can introduce, the better.
>> >>>
>> >>> I have some other stuff on my plate before I can dive deep into this
>> >>> but I eventually will. Thanks for the pointers, everyone - helpful!
>> >>>
>> >>> D.
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: dev-help@lucene.apache.org
>>
>>