Mailing List Archive

Fwd: High concurrency scenarios IndexReader‘s registerParentReader method takes a lot of CPU time
>
> Hi, all
> In our Elastic Search benchmark, found out that IndexReader‘s registerParentReader method takes a lot of CPU time
> Environment:
> ES Version: 7.5.1
> Lucene Version: 8.3.0
>
> Index:
> Shard Num 1
> Replicator’s Num:1
> Segment Count: 538
> Size:49.5GB
>
> Benchmark result: qps:5200,pct99 600ms
> CPU Util:93%
>
> Use flame graph find the hot method IndexReader‘s registerParentReader. dive into code find out that Have a race condition that on the synchronizedSet Because of only one
> Shard and one shard have too many segments, this intensify competition
>
>
>
>
> To verify this assumption,execute force merge on this index?after force merge:
> QPS: more then 5600
> PCT99: 60ms
> CPU Util: 11%
> The performance improved by a factor of 10x
>
> So we can use ConcurrentWeakKeyHashmap instead of synchronizedSet to reduce lock competition?
>
>
>
>
>
Re: High concurrency scenarios IndexReader‘s registerParentReader method takes a lot of CPU time [ In reply to ]
Hi,

Next time please reply/send these technical questions to
java-user@lucene.apache.org or dev@lucene.apache.org to get more audience.

538 segments seems huge. What is your configuration to get this number of
segments?

Le jeu. 16 avr. 2020 à 16:28, ??? <zhanghailei@bytedance.com> a écrit :

>
>
>
> Hi, all
> In our Elastic Search benchmark, found out that IndexReader‘s
> registerParentReader method takes a lot of CPU time
> Environment:
> ES Version: 7.5.1
> Lucene Version: 8.3.0
>
> Index:
> Shard Num 1
> Replicator’s Num:1
> Segment Count: 538
> Size:49.5GB
>
> Benchmark result: qps:5200,pct99 600ms
> CPU Util:93%
>
> Use flame graph find the hot method IndexReader‘s registerParentReader.
> dive into code find out that Have a race condition that on the
> synchronizedSet Because of only one
> Shard and one shard have too many segments, this intensify competition
>
>
>
> To verify this assumption,execute force merge on this index?after force
> merge:
> QPS: more then 5600
> PCT99: 60ms
> CPU Util: 11%
> The performance improved by a factor of 10x
>
> So we can use ConcurrentWeakKeyHashmap instead of synchronizedSet to
> reduce lock competition?
>
>
>
>
>
>
>