Mailing List Archive

Re: [Phishing Risk] [External] Re: High concurrency scenarios IndexReader‘s registerParentReader method takes a lot of CPU time
Using elastic search index stats apihttps://www.elastic.co/guide/en/elasticsearch/reference/current/indices-stats.html#indices-stats <https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-stats.html#indices-stats>

> 2020?4?17? ??11:49?Bruno Roustant <bruno.roustant@gmail.com> ???
>
> Hi,
>
> Next time please reply/send these technical questions to java-user@lucene.apache.org <mailto:java-user@lucene.apache.org> or dev@lucene.apache.org <mailto:dev@lucene.apache.org> to get more audience.
>
> 538 segments seems huge. What is your configuration to get this number of segments?
>
> Le jeu. 16 avr. 2020 à 16:28, ??? <zhanghailei@bytedance.com <mailto:zhanghailei@bytedance.com>> a écrit :
>
>
>>
>> Hi, all
>> In our Elastic Search benchmark, found out that IndexReader‘s registerParentReader method takes a lot of CPU time
>> Environment:
>> ES Version: 7.5.1
>> Lucene Version: 8.3.0
>>
>> Index:
>> Shard Num 1
>> Replicator’s Num:1
>> Segment Count: 538
>> Size:49.5GB
>>
>> Benchmark result: qps:5200,pct99 600ms
>> CPU Util:93%
>>
>> Use flame graph find the hot method IndexReader‘s registerParentReader. dive into code find out that Have a race condition that on the synchronizedSet Because of only one
>> Shard and one shard have too many segments, this intensify competition
>>
>>
>>
>>
>> To verify this assumption,execute force merge on this index?after force merge:
>> QPS: more then 5600
>> PCT99: 60ms
>> CPU Util: 11%
>> The performance improved by a factor of 10x
>>
>> So we can use ConcurrentWeakKeyHashmap instead of synchronizedSet to reduce lock competition?
>>
>>
>>
>>
>>
>