Mailing List Archive

Sorry, PR link: https://github.com/apache/lucene/pull/12865

-Chris

On Saturday, December 2, 2023, Uwe Schindler <uwe@thetaphi.de> wrote:

> Hi Chris,
>
> I can't find the PR.
>
>
>
> I am interested, because I wrote the original ParallelReader tests.
>
> IMHO the parallel readers are so sensitive to random changes, the test
> setup should not use any indexwriter randomization at all.
>
> ParallelReader is also seldomly used, maybe we should remove support at
> some point. I don't know anybody using it, because it is very complicated
> to maintain consistent indexes. It only works with stable merge policies.
>
> Uwe
>

>
> Am 2. Dezember 2023 09:34:46 MEZ schrieb Chris Hegarty
> <christopher.hegarty@elastic.co.INVALID>:
>
>> Hi,
>>
>> I noticed this failure locally, and opened a PR for it yesterday. It is a
>> test issues, and indeed related to the recent merge policy test
>> randomization change.
>>
>> -Chris
>>
>> On Saturday, December 2, 2023, Patrick Zhai <zhai7631@gmail.com> wrote:
>>
>>> Seems it's because this MockRandomMergePolicy change
>>> <https://github.com/apache/lucene/blob/main/lucene/test-framework/src/java/org/apache/lucene/tests/index/MockRandomMergePolicy.java#L242> recently
>>> makes ParallelLeafReader unhappy - it's reading two parallel segments from
>>> 2 dir and this MP makes one of the segments' documents order reversed.
>>>
>>> But should be just test util issue and not affecting release.
>>>
>>> Adrien do you want to take a look? I'm not sure what's the best way to
>>> fix it, adding an index sort for that test seems a bit overkill?
>>>
>>> Patrick
>>>
>>> On Fri, Dec 1, 2023 at 2:06?PM Michael McCandless <
>>> lucene@mikemccandless.com> wrote:
>>>
>>>> Hmm this reproduces for me, and looks new/unique. Could it be related
>>>> to recent 9.9.0 changes / release blocker?
>>>>
>>>> Mike
>>>>
>>>> On Fri, Dec 1, 2023 at 3:33?PM Policeman Jenkins Server <
>>>> jenkins@thetaphi.de> wrote:
>>>>
>>>>> Build: https://jenkins.thetaphi.de/job/Lucene-9.x-Linux/14204/
>>>>> Java: 64bit/hotspot/jdk-11.0.21 -XX:+UseCompressedOops
>>>>> -XX:+UseParallelGC
>>>>>
>>>>> 1 tests failed.
>>>>> FAILED: org.apache.lucene.index.TestParallelLeafReader.testQueries
>>>>>
>>>>> Error Message:
>>>>> org.junit.ComparisonFailure: expected:<v[2]> but was:<v[1]>
>>>>>
>>>>> Stack Trace:
>>>>> org.junit.ComparisonFailure: expected:<v[2]> but was:<v[1]>
>>>>> at __randomizedtesting.SeedInfo.s
>>>>> eed([6CA57EA3FB50CA0D:302BB278E1397FA3]:0)
>>>>> at org.junit.Assert.assertEquals(Assert.java:117)
>>>>> at org.junit.Assert.assertEquals(Assert.java:146)
>>>>> at org.apache.lucene.index.TestPa
>>>>> rallelLeafReader.queryTest(TestParallelLeafReader.java:263)
>>>>> at org.apache.lucene.index.TestPa
>>>>> rallelLeafReader.testQueries(TestParallelLeafReader.java:48)
>>>>> at java.base/jdk.internal.reflect
>>>>> .NativeMethodAccessorImpl.invoke0(Native Method)
>>>>> at java.base/jdk.internal.reflect
>>>>> .NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>>>>> at java.base/jdk.internal.reflect
>>>>> .DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccesso
>>>>> rImpl.java:43)
>>>>> at java.base/java.lang.reflect.Method.invoke(Method.java:566)
>>>>> at com.carrotsearch.randomizedtes
>>>>> ting.RandomizedRunner.invoke(RandomizedRunner.java:1758)
>>>>> at com.carrotsearch.randomizedtesting.RandomizedRunner$8.
>>>>> evaluate(RandomizedRunner.java:946)
>>>>> at com.carrotsearch.randomizedtesting.RandomizedRunner$9.
>>>>> evaluate(RandomizedRunner.java:982)
>>>>> at com.carrotsearch.randomizedtesting.RandomizedRunner$10.
>>>>> evaluate(RandomizedRunner.java:996)
>>>>> at org.apache.lucene.tests.util.T
>>>>> estRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardown
>>>>> Chained.java:48)
>>>>> at org.apache.lucene.tests.util.A
>>>>> bstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
>>>>> at org.apache.lucene.tests.util.T
>>>>> estRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:45)
>>>>> at org.apache.lucene.tests.util.T
>>>>> estRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfter
>>>>> MaxFailures.java:60)
>>>>> at org.apache.lucene.tests.util.T
>>>>> estRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
>>>>> at org.junit.rules.RunRules.evaluate(RunRules.java:20)
>>>>> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.
>>>>> evaluate(StatementAdapter.java:36)
>>>>> at com.carrotsearch.randomizedtes
>>>>> ting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:390)
>>>>> at com.carrotsearch.randomizedtes
>>>>> ting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:843)
>>>>> at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.
>>>>> evaluate(ThreadLeakControl.java:490)
>>>>> at com.carrotsearch.randomizedtes
>>>>> ting.RandomizedRunner.runSingleTest(RandomizedRunner.java:955)
>>>>> at com.carrotsearch.randomizedtesting.RandomizedRunner$5.
>>>>> evaluate(RandomizedRunner.java:840)
>>>>> at com.carrotsearch.randomizedtesting.RandomizedRunner$6.
>>>>> evaluate(RandomizedRunner.java:891)
>>>>> at com.carrotsearch.randomizedtesting.RandomizedRunner$7.
>>>>> evaluate(RandomizedRunner.java:902)
>>>>> at org.apache.lucene.tests.util.A
>>>>> bstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
>>>>> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.
>>>>> evaluate(StatementAdapter.java:36)
>>>>> at org.apache.lucene.tests.util.T
>>>>> estRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:38)
>>>>> at com.carrotsearch.randomizedtes
>>>>> ting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(No
>>>>> ShadowingOrOverridesOnMethodsRule.java:40)
>>>>> at com.carrotsearch.randomizedtes
>>>>> ting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(No
>>>>> ShadowingOrOverridesOnMethodsRule.java:40)
>>>>> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.
>>>>> evaluate(StatementAdapter.java:36)
>>>>> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.
>>>>> evaluate(StatementAdapter.java:36)
>>>>> at org.apache.lucene.tests.util.T
>>>>> estRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequi
>>>>> red.java:53)
>>>>> at org.apache.lucene.tests.util.A
>>>>> bstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
>>>>> at org.apache.lucene.tests.util.T
>>>>> estRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
>>>>> at org.apache.lucene.tests.util.T
>>>>> estRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfter
>>>>> MaxFailures.java:60)
>>>>> at org.apache.lucene.tests.util.T
>>>>> estRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:47)
>>>>> at org.junit.rules.RunRules.evaluate(RunRules.java:20)
>>>>> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.
>>>>> evaluate(StatementAdapter.java:36)
>>>>> at com.carrotsearch.randomizedtes
>>>>> ting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:390)
>>>>> at com.carrotsearch.randomizedtes
>>>>> ting.ThreadLeakControl.lambda$forkTimeoutingTask$0(ThreadLea
>>>>> kControl.java:850)
>>>>> at java.base/java.lang.Thread.run(Thread.java:829)
>>>>>
>>>>> ---------------------------------------------------------------------
>>>>> To unsubscribe, e-mail: builds-unsubscribe@lucene.apache.org
>>>>> For additional commands, e-mail: builds-help@lucene.apache.org
>>>>
>>>> --
> Uwe Schindler
> Achterdiek 19, 28357 Bremen
> <https://www.google.com/maps/search/Achterdiek+19,+28357+Bremen?entry=gmail&source=g>
> https://www.thetaphi.de
>

Found the PR. Somehow the mailinglist didn't get it.

Am 2. Dezember 2023 09:58:45 MEZ schrieb Uwe Schindler <uwe@thetaphi.de>:
>Hi Chris,
>
>I can't find the PR.
>
>I am interested, because I wrote the original ParallelReader tests.
>
>IMHO the parallel readers are so sensitive to random changes, the test setup should not use any indexwriter randomization at all.
>
>ParallelReader is also seldomly used, maybe we should remove support at some point. I don't know anybody using it, because it is very complicated to maintain consistent indexes. It only works with stable merge policies.
>
>Uwe
>
>Am 2. Dezember 2023 09:34:46 MEZ schrieb Chris Hegarty <christopher.hegarty@elastic.co.INVALID>:
>>Hi,
>>
>>I noticed this failure locally, and opened a PR for it yesterday. It is a
>>test issues, and indeed related to the recent merge policy test
>>randomization change.
>>
>>-Chris
>>
>>On Saturday, December 2, 2023, Patrick Zhai <zhai7631@gmail.com> wrote:
>>
>>> Seems it's because this MockRandomMergePolicy change
>>> <https://github.com/apache/lucene/blob/main/lucene/test-framework/src/java/org/apache/lucene/tests/index/MockRandomMergePolicy.java#L242> recently
>>> makes ParallelLeafReader unhappy - it's reading two parallel segments from
>>> 2 dir and this MP makes one of the segments' documents order reversed.
>>>
>>> But should be just test util issue and not affecting release.
>>>
>>> Adrien do you want to take a look? I'm not sure what's the best way to fix
>>> it, adding an index sort for that test seems a bit overkill?
>>>
>>> Patrick
>>>
>>> On Fri, Dec 1, 2023 at 2:06?PM Michael McCandless <
>>> lucene@mikemccandless.com> wrote:
>>>
>>>> Hmm this reproduces for me, and looks new/unique. Could it be related to
>>>> recent 9.9.0 changes / release blocker?
>>>>
>>>> Mike
>>>>
>>>> On Fri, Dec 1, 2023 at 3:33?PM Policeman Jenkins Server <
>>>> jenkins@thetaphi.de> wrote:
>>>>
>>>>> Build: https://jenkins.thetaphi.de/job/Lucene-9.x-Linux/14204/
>>>>> Java: 64bit/hotspot/jdk-11.0.21 -XX:+UseCompressedOops -XX:+UseParallelGC
>>>>>
>>>>> 1 tests failed.
>>>>> FAILED: org.apache.lucene.index.TestParallelLeafReader.testQueries
>>>>>
>>>>> Error Message:
>>>>> org.junit.ComparisonFailure: expected:<v[2]> but was:<v[1]>
>>>>>
>>>>> Stack Trace:
>>>>> org.junit.ComparisonFailure: expected:<v[2]> but was:<v[1]>
>>>>> at __randomizedtesting.SeedInfo.seed([6CA57EA3FB50CA0D:
>>>>> 302BB278E1397FA3]:0)
>>>>> at org.junit.Assert.assertEquals(Assert.java:117)
>>>>> at org.junit.Assert.assertEquals(Assert.java:146)
>>>>> at org.apache.lucene.index.TestParallelLeafReader.queryTest(
>>>>> TestParallelLeafReader.java:263)
>>>>> at org.apache.lucene.index.TestParallelLeafReader.testQueries(
>>>>> TestParallelLeafReader.java:48)
>>>>> at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native
>>>>> Method)
>>>>> at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.
>>>>> invoke(NativeMethodAccessorImpl.java:62)
>>>>> at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.
>>>>> invoke(DelegatingMethodAccessorImpl.java:43)
>>>>> at java.base/java.lang.reflect.Method.invoke(Method.java:566)
>>>>> at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(
>>>>> RandomizedRunner.java:1758)
>>>>> at com.carrotsearch.randomizedtesting.
>>>>> RandomizedRunner$8.evaluate(RandomizedRunner.java:946)
>>>>> at com.carrotsearch.randomizedtesting.
>>>>> RandomizedRunner$9.evaluate(RandomizedRunner.java:982)
>>>>> at com.carrotsearch.randomizedtesting.
>>>>> RandomizedRunner$10.evaluate(RandomizedRunner.java:996)
>>>>> at org.apache.lucene.tests.util.TestRuleSetupTeardownChained$
>>>>> 1.evaluate(TestRuleSetupTeardownChained.java:48)
>>>>> at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.
>>>>> evaluate(AbstractBeforeAfterRule.java:43)
>>>>> at org.apache.lucene.tests.util.TestRuleThreadAndTestName$1.
>>>>> evaluate(TestRuleThreadAndTestName.java:45)
>>>>> at org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures
>>>>> $1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
>>>>> at org.apache.lucene.tests.util.TestRuleMarkFailure$1.
>>>>> evaluate(TestRuleMarkFailure.java:44)
>>>>> at org.junit.rules.RunRules.evaluate(RunRules.java:20)
>>>>> at com.carrotsearch.randomizedtesting.rules.
>>>>> StatementAdapter.evaluate(StatementAdapter.java:36)
>>>>> at com.carrotsearch.randomizedtesting.ThreadLeakControl$
>>>>> StatementRunner.run(ThreadLeakControl.java:390)
>>>>> at com.carrotsearch.randomizedtesting.ThreadLeakControl.
>>>>> forkTimeoutingTask(ThreadLeakControl.java:843)
>>>>> at com.carrotsearch.randomizedtesting.
>>>>> ThreadLeakControl$3.evaluate(ThreadLeakControl.java:490)
>>>>> at com.carrotsearch.randomizedtesting.RandomizedRunner.
>>>>> runSingleTest(RandomizedRunner.java:955)
>>>>> at com.carrotsearch.randomizedtesting.
>>>>> RandomizedRunner$5.evaluate(RandomizedRunner.java:840)
>>>>> at com.carrotsearch.randomizedtesting.
>>>>> RandomizedRunner$6.evaluate(RandomizedRunner.java:891)
>>>>> at com.carrotsearch.randomizedtesting.
>>>>> RandomizedRunner$7.evaluate(RandomizedRunner.java:902)
>>>>> at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.
>>>>> evaluate(AbstractBeforeAfterRule.java:43)
>>>>> at com.carrotsearch.randomizedtesting.rules.
>>>>> StatementAdapter.evaluate(StatementAdapter.java:36)
>>>>> at org.apache.lucene.tests.util.TestRuleStoreClassName$1.
>>>>> evaluate(TestRuleStoreClassName.java:38)
>>>>> at com.carrotsearch.randomizedtesting.rules.
>>>>> NoShadowingOrOverridesOnMethodsRule$1.evaluate(
>>>>> NoShadowingOrOverridesOnMethodsRule.java:40)
>>>>> at com.carrotsearch.randomizedtesting.rules.
>>>>> NoShadowingOrOverridesOnMethodsRule$1.evaluate(
>>>>> NoShadowingOrOverridesOnMethodsRule.java:40)
>>>>> at com.carrotsearch.randomizedtesting.rules.
>>>>> StatementAdapter.evaluate(StatementAdapter.java:36)
>>>>> at com.carrotsearch.randomizedtesting.rules.
>>>>> StatementAdapter.evaluate(StatementAdapter.java:36)
>>>>> at org.apache.lucene.tests.util.TestRuleAssertionsRequired$1.
>>>>> evaluate(TestRuleAssertionsRequired.java:53)
>>>>> at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.
>>>>> evaluate(AbstractBeforeAfterRule.java:43)
>>>>> at org.apache.lucene.tests.util.TestRuleMarkFailure$1.
>>>>> evaluate(TestRuleMarkFailure.java:44)
>>>>> at org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures
>>>>> $1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
>>>>> at org.apache.lucene.tests.util.TestRuleIgnoreTestSuites$1.
>>>>> evaluate(TestRuleIgnoreTestSuites.java:47)
>>>>> at org.junit.rules.RunRules.evaluate(RunRules.java:20)
>>>>> at com.carrotsearch.randomizedtesting.rules.
>>>>> StatementAdapter.evaluate(StatementAdapter.java:36)
>>>>> at com.carrotsearch.randomizedtesting.ThreadLeakControl$
>>>>> StatementRunner.run(ThreadLeakControl.java:390)
>>>>> at com.carrotsearch.randomizedtesting.ThreadLeakControl.lambda$
>>>>> forkTimeoutingTask$0(ThreadLeakControl.java:850)
>>>>> at java.base/java.lang.Thread.run(Thread.java:829)
>>>>>
>>>>> ---------------------------------------------------------------------
>>>>> To unsubscribe, e-mail: builds-unsubscribe@lucene.apache.org
>>>>> For additional commands, e-mail: builds-help@lucene.apache.org
>>>>
>>>>
>
>--
>Uwe Schindler
>Achterdiek 19, 28357 Bremen
>https://www.thetaphi.de
--
Uwe Schindler
Achterdiek 19, 28357 Bremen
https://www.thetaphi.de

> ParallelReader is also seldomly used, maybe we should remove support at
> some point. I don't know anybody using it, because it is very complicated
> to maintain consistent indexes. It only works with stable merge policies.
>

You do know somebody - you know me. We're using it extensively - the
scenario is for storing data derived from the main document in a separate
index, merging this data dynamically. The data can then be reindexed/
modified independently. Yes, we do use stable merge policies.

Dawid

Ha. Cool! Nice to meet.

I suggested to use this reader to some customers, but they were using Solr or Elasticsearch and it's not easy to implement it there. And they didn't want to pay the expensive Uwe. ????

How do you handle deletes. Because the main issue with those readers is that you can't update documents without also updating the main reader (although it's a fake update).

If this is used, have you thought of a SynchronizedMergePolicy that just applies the same merges in the secondary index?

Uwe

Am 2. Dezember 2023 10:27:20 MEZ schrieb Dawid Weiss <dawid.weiss@gmail.com>:
>> ParallelReader is also seldomly used, maybe we should remove support at
>> some point. I don't know anybody using it, because it is very complicated
>> to maintain consistent indexes. It only works with stable merge policies.
>>
>
>You do know somebody - you know me. We're using it extensively - the
>scenario is for storing data derived from the main document in a separate
>index, merging this data dynamically. The data can then be reindexed/
>modified independently. Yes, we do use stable merge policies.
>
>Dawid

--
Uwe Schindler
Achterdiek 19, 28357 Bremen
https://www.thetaphi.de

Hi Uwe,

Apologies for the late reply.

I suggested to use this reader to some customers, but they were using Solr
> or Elasticsearch and it's not easy to implement it there. And they didn't
> want to pay the expensive Uwe.
>
????
>

Oh, they should know better - Mr Uwe is worth his price tag for technical
merits but also his eloquence and samurai-sword-sharp code smell
observation skills!


> How do you handle deletes. Because the main issue with those readers is
> that you can't update documents without also updating the main reader
> (although it's a fake update).
>

It's not a general search/ document storage system - we allow updates but
they're not real time and reindexing creates both new segments and
second-tier segments. It gets even more complicated because there are
multiple living Lucene commits (custom deletion policy). Long story but
works just fine!


> If this is used, have you thought of a SynchronizedMergePolicy that just
> applies the same merges in the secondary index?
>

I don't think it's applicable to us (as above), but it's an interesting
idea!

D.

>