Mailing List Archive

oh! I overlooked tests.dups -- but it doesn't seem to be doing what I
expected. EG I tried

./gradlew -p lucene/core test --tests TestByteVectorSimilarityQuery
-Ptests.dups=1000 -Ptests.multiplier=3

and it completes very quickly reporting having run only 13 tests

On Tue, Apr 2, 2024 at 4:14?PM Michael Sokolov <msokolov@gmail.com> wrote:
>
> Is there a convenient way to run a test multiple times with different
> seeds? Do I need to write my own script? I feel like I used to be able
> to do this in IntelliJ, but that option seems to have vanished, and I
> don't see any such option in gradle testOpts either. I tried
> -tests.iter but that seems to run the same test multiple times with
> the same seed,

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org

>
> ./gradlew -p lucene/core test --tests TestByteVectorSimilarityQuery
> -Ptests.dups=1000 -Ptests.multiplier=3
>
> and it completes very quickly reporting having run only 13 tests
>

The task is called 'beast', not 'test', Mike.

D.

I think you could try this:

./gradlew -p lucene/core beast -Ptests.dups=10 --tests
TestByteVectorSimilarityQuery

I confirmed it uses a different seed (long value) for each run by printing
the seed here
<https://github.com/apache/lucene/blob/main/gradle/testing/beasting.gradle#L62-L66>
in beasting.gradle
<https://github.com/apache/lucene/blob/main/gradle/testing/beasting.gradle>.

- Shubham

On Wed, Apr 3, 2024 at 1:49?AM Michael Sokolov <msokolov@gmail.com> wrote:

> oh! I overlooked tests.dups -- but it doesn't seem to be doing what I
> expected. EG I tried
>
> ./gradlew -p lucene/core test --tests TestByteVectorSimilarityQuery
> -Ptests.dups=1000 -Ptests.multiplier=3
>
> and it completes very quickly reporting having run only 13 tests
>
> On Tue, Apr 2, 2024 at 4:14?PM Michael Sokolov <msokolov@gmail.com> wrote:
> >
> > Is there a convenient way to run a test multiple times with different
> > seeds? Do I need to write my own script? I feel like I used to be able
> > to do this in IntelliJ, but that option seems to have vanished, and I
> > don't see any such option in gradle testOpts either. I tried
> > -tests.iter but that seems to run the same test multiple times with
> > the same seed,
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: dev-help@lucene.apache.org
>
>

This section of the help file for testing explains the difference between
'beast', 'test' and various reiteration methods -

https://github.com/apache/lucene/blob/main/help/tests.txt#L89-L123

In *most* cases, tests.iters will be just as good as beasting (and much
faster). The only difference is when you want class-level settings to be
randomized differently (static initializers, for example).

D.

On Tue, Apr 2, 2024 at 10:54?PM Shubham Chaudhary <shubhmasand@gmail.com>
wrote:

> I think you could try this:
>
> ./gradlew -p lucene/core beast -Ptests.dups=10 --tests
> TestByteVectorSimilarityQuery
>
> I confirmed it uses a different seed (long value) for each run by printing
> the seed here
> <https://github.com/apache/lucene/blob/main/gradle/testing/beasting.gradle#L62-L66>
> in beasting.gradle
> <https://github.com/apache/lucene/blob/main/gradle/testing/beasting.gradle>
> .
>
> - Shubham
>
> On Wed, Apr 3, 2024 at 1:49?AM Michael Sokolov <msokolov@gmail.com> wrote:
>
>> oh! I overlooked tests.dups -- but it doesn't seem to be doing what I
>> expected. EG I tried
>>
>> ./gradlew -p lucene/core test --tests TestByteVectorSimilarityQuery
>> -Ptests.dups=1000 -Ptests.multiplier=3
>>
>> and it completes very quickly reporting having run only 13 tests
>>
>> On Tue, Apr 2, 2024 at 4:14?PM Michael Sokolov <msokolov@gmail.com>
>> wrote:
>> >
>> > Is there a convenient way to run a test multiple times with different
>> > seeds? Do I need to write my own script? I feel like I used to be able
>> > to do this in IntelliJ, but that option seems to have vanished, and I
>> > don't see any such option in gradle testOpts either. I tried
>> > -tests.iter but that seems to run the same test multiple times with
>> > the same seed,
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: dev-help@lucene.apache.org
>>
>>

Thank you! Now I just need to understand why the test failure is no longer
reproducing lol. Also it's mildly confusing that when you specify
tests.iters it prints a single test seed if it is actually going to use
many different ones? Anyway I will read more docs I am probably still
confusing beast and test?

On Tue, Apr 2, 2024, 6:27?PM Dawid Weiss <dawid.weiss@gmail.com> wrote:

>
> This section of the help file for testing explains the difference between
> 'beast', 'test' and various reiteration methods -
>
> https://github.com/apache/lucene/blob/main/help/tests.txt#L89-L123
>
> In *most* cases, tests.iters will be just as good as beasting (and much
> faster). The only difference is when you want class-level settings to be
> randomized differently (static initializers, for example).
>
> D.
>
> On Tue, Apr 2, 2024 at 10:54?PM Shubham Chaudhary <shubhmasand@gmail.com>
> wrote:
>
>> I think you could try this:
>>
>> ./gradlew -p lucene/core beast -Ptests.dups=10 --tests
>> TestByteVectorSimilarityQuery
>>
>> I confirmed it uses a different seed (long value) for each run by
>> printing the seed here
>> <https://github.com/apache/lucene/blob/main/gradle/testing/beasting.gradle#L62-L66>
>> in beasting.gradle
>> <https://github.com/apache/lucene/blob/main/gradle/testing/beasting.gradle>
>> .
>>
>> - Shubham
>>
>> On Wed, Apr 3, 2024 at 1:49?AM Michael Sokolov <msokolov@gmail.com>
>> wrote:
>>
>>> oh! I overlooked tests.dups -- but it doesn't seem to be doing what I
>>> expected. EG I tried
>>>
>>> ./gradlew -p lucene/core test --tests TestByteVectorSimilarityQuery
>>> -Ptests.dups=1000 -Ptests.multiplier=3
>>>
>>> and it completes very quickly reporting having run only 13 tests
>>>
>>> On Tue, Apr 2, 2024 at 4:14?PM Michael Sokolov <msokolov@gmail.com>
>>> wrote:
>>> >
>>> > Is there a convenient way to run a test multiple times with different
>>> > seeds? Do I need to write my own script? I feel like I used to be able
>>> > to do this in IntelliJ, but that option seems to have vanished, and I
>>> > don't see any such option in gradle testOpts either. I tried
>>> > -tests.iter but that seems to run the same test multiple times with
>>> > the same seed,
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
>>> For additional commands, e-mail: dev-help@lucene.apache.org
>>>
>>>

> Now I just need to understand why the test failure is no longer
> reproducing lol.
>

This is indeed the hard part!


> Also it's mildly confusing that when you specify tests.iters it prints a
> single test seed if it is actually going to use many different ones?
>

It prints a single seed because it starts from that seed (the static
initialization, that is). But each test has its own starting point derived
from the main seed and the test name (if I recall right). So when you pass
tests.iters=100 and run a single test, any random call in that test
(excluding static hooks and static blocks) should be different on each
iteration. You can try it by adding:

assumeTrue("", RandomizedTest.randomBoolean());

For example (I added it to TestSearch.java):

./gradlew -p lucene/core test --tests TestSearch -Ptests.iters=100
...
:lucene:core:test (SUCCESS): 100 test(s), 52 skipped

if you modify this to assertTrue, you'll get to see the hierarchical seed
chain for each test that failed - note the first part is constant, the
second is derived for each test iteration:

./gradlew -p lucene/core test --tests TestSearch -Ptests.iters=100
-Ptests.seed=deadbeef

and two example failures:

- org.apache.lucene.TestSearch.testSearch
{seed=[DEADBEEF:3EDB7869EFFD5034]} (:lucene:core)
Test output:
/Users/dweiss/work/lucene/lucene/core/build/test-results/test/outputs/OUTPUT-org.apache.lucene.TestSearch.txt
Reproduce with: gradlew :lucene:core:test --tests
"org.apache.lucene.TestSearch.testSearch
{seed=[DEADBEEF:3EDB7869EFFD5034]}" -Ptests.jvms=4
"-Ptests.jvmargs=-XX:TieredStopAtLevel=1 -XX:+UseParallelGC
-XX:ActiveProcessorCount=1" -Ptests.seed=deadbeef -Ptests.iters=100
-Ptests.gui=false -Ptests.file.encoding=UTF-8 -Ptests.vectorsize=512

- org.apache.lucene.TestSearch.testSearch
{seed=[DEADBEEF:F44F3D10E8B98D27]} (:lucene:core)
Test output:
/Users/dweiss/work/lucene/lucene/core/build/test-results/test/outputs/OUTPUT-org.apache.lucene.TestSearch.txt
Reproduce with: gradlew :lucene:core:test --tests
"org.apache.lucene.TestSearch.testSearch
{seed=[DEADBEEF:F44F3D10E8B98D27]}" -Ptests.jvms=4
"-Ptests.jvmargs=-XX:TieredStopAtLevel=1 -XX:+UseParallelGC
-XX:ActiveProcessorCount=1" -Ptests.seed=deadbeef -Ptests.iters=100
-Ptests.gui=false -Ptests.file.encoding=UTF-8 -Ptests.vectorsize=512

If you'd like to repeat tests with *the same* starting seed for each test,
you need to pass the full chain, including the second part of the seed. For
example, this will fail 100 times (and not approximately 50% of the times):

./gradlew -p lucene/core test --tests TestSearch -Ptests.iters=100
-Ptests.seed=deadbeef:F44F3D10E8B98D27

It may seem a bit complicated but it really isn't... I hope! And for 99%
of tests, you'd probably rerun with the first part of the seed and it'd be
sufficient to locate the problem.

The 'beast' task is a bit different because it physically re-launches the
test infrastructure so if you don't fix the initial seed, each started JVM
will have a different "starting" seed for static initializers and hooks.
This may matter for locale randomization, jvm issues or static initializers
that rely on randomness. But most isolated test methods should only rely on
their starting seed (not the "global starting seed").

Dawid

Thanks for the explanation. It makes sense that we start with a given
seed and then each iteration is different because it re-uses the same
Random instance (or whatever static state?) without re-initialization?

On Wed, Apr 3, 2024 at 6:09?PM Dawid Weiss <dawid.weiss@gmail.com> wrote:
>
>
>> Now I just need to understand why the test failure is no longer reproducing lol.
>
>
> This is indeed the hard part!
>
>>
>> Also it's mildly confusing that when you specify tests.iters it prints a single test seed if it is actually going to use many different ones?
>
>
> It prints a single seed because it starts from that seed (the static initialization, that is). But each test has its own starting point derived from the main seed and the test name (if I recall right). So when you pass tests.iters=100 and run a single test, any random call in that test (excluding static hooks and static blocks) should be different on each iteration. You can try it by adding:
>
> assumeTrue("", RandomizedTest.randomBoolean());
>
> For example (I added it to TestSearch.java):
>
> ./gradlew -p lucene/core test --tests TestSearch -Ptests.iters=100
> ...
> :lucene:core:test (SUCCESS): 100 test(s), 52 skipped
>
> if you modify this to assertTrue, you'll get to see the hierarchical seed chain for each test that failed - note the first part is constant, the second is derived for each test iteration:
>
> ./gradlew -p lucene/core test --tests TestSearch -Ptests.iters=100 -Ptests.seed=deadbeef
>
> and two example failures:
>
> - org.apache.lucene.TestSearch.testSearch {seed=[DEADBEEF:3EDB7869EFFD5034]} (:lucene:core)
> Test output: /Users/dweiss/work/lucene/lucene/core/build/test-results/test/outputs/OUTPUT-org.apache.lucene.TestSearch.txt
> Reproduce with: gradlew :lucene:core:test --tests "org.apache.lucene.TestSearch.testSearch {seed=[DEADBEEF:3EDB7869EFFD5034]}" -Ptests.jvms=4 "-Ptests.jvmargs=-XX:TieredStopAtLevel=1 -XX:+UseParallelGC -XX:ActiveProcessorCount=1" -Ptests.seed=deadbeef -Ptests.iters=100 -Ptests.gui=false -Ptests.file.encoding=UTF-8 -Ptests.vectorsize=512
>
> - org.apache.lucene.TestSearch.testSearch {seed=[DEADBEEF:F44F3D10E8B98D27]} (:lucene:core)
> Test output: /Users/dweiss/work/lucene/lucene/core/build/test-results/test/outputs/OUTPUT-org.apache.lucene.TestSearch.txt
> Reproduce with: gradlew :lucene:core:test --tests "org.apache.lucene.TestSearch.testSearch {seed=[DEADBEEF:F44F3D10E8B98D27]}" -Ptests.jvms=4 "-Ptests.jvmargs=-XX:TieredStopAtLevel=1 -XX:+UseParallelGC -XX:ActiveProcessorCount=1" -Ptests.seed=deadbeef -Ptests.iters=100 -Ptests.gui=false -Ptests.file.encoding=UTF-8 -Ptests.vectorsize=512
>
> If you'd like to repeat tests with *the same* starting seed for each test, you need to pass the full chain, including the second part of the seed. For example, this will fail 100 times (and not approximately 50% of the times):
>
> ./gradlew -p lucene/core test --tests TestSearch -Ptests.iters=100 -Ptests.seed=deadbeef:F44F3D10E8B98D27
>
> It may seem a bit complicated but it really isn't... I hope! And for 99% of tests, you'd probably rerun with the first part of the seed and it'd be sufficient to locate the problem.
>
> The 'beast' task is a bit different because it physically re-launches the test infrastructure so if you don't fix the initial seed, each started JVM will have a different "starting" seed for static initializers and hooks. This may matter for locale randomization, jvm issues or static initializers that rely on randomness. But most isolated test methods should only rely on their starting seed (not the "global starting seed").
>
> Dawid

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org

> Thanks for the explanation. It makes sense that we start with a given
> seed and then each iteration is different because it re-uses the same
> Random instance (or whatever static state?) without re-initialization?
>

It doesn't reuse the same random instance - it's not that simple - it
hierarchically forks the random seed between tests, classes, reiterations
etc., and then initializes the context's random so that runs are
predictable and repeatable even if you run a subset of tests for the given
main seed. Random instances are also not reused between threads in an
attempt to make failures reproducible (this, of course, is not fool-proof).

If you care to look, there are more documented examples with somewhat
increasing complexity here:
https://github.com/randomizedtesting/randomizedtesting/tree/master/examples/maven/src/main/java/com/carrotsearch/examples/randomizedrunner

In particular, these talk about hierarchical seed propagation:

https://github.com/randomizedtesting/randomizedtesting/blob/master/examples/maven/src/main/java/com/carrotsearch/examples/randomizedrunner/Test004MoreRandomness.java
https://github.com/randomizedtesting/randomizedtesting/blob/master/examples/maven/src/main/java/com/carrotsearch/examples/randomizedrunner/Test006RepeatingTests.java

D.