Mailing List Archive

Hmm, that test is taking a very long time on my laptop (running
Ubuntu) with JDK11, so doesn't seem to be a Windows or JDK17 issue

./gradlew :lucene:core:test --tests
"org.apache.lucene.index.TestIndexWriterMergePolicy.testStressUpdateSameDocumentWithMergeOnGetReader"
-Dtests.nightly=true -Dtests.seed=B67ECC7381FE35B

after 12m44s I killed it


On Sat, Nov 27, 2021 at 11:59 AM Michael Sokolov <msokolov@gmail.com> wrote:
>
> I see that we periodically get various test failures from this test.
> Has anybody been tracking this more carefully than me, and if so, do
> you remember if it's always on Windows where these too-many-open-files
> (and sometimes out of disk space) errors show up? I wonder if we
> should reduce the cardinality of these test settings:
>
> AtomicInteger iters = new AtomicInteger(100 +
> random().nextInt(TEST_NIGHTLY ? 5000 : 1000));
> AtomicInteger numFullFlushes =
> new AtomicInteger(10 + random().nextInt(TEST_NIGHTLY ? 500 : 100));
>
> On Sat, Nov 27, 2021 at 5:54 AM Policeman Jenkins Server
> <jenkins@thetaphi.de> wrote:
> >
> > Build: https://jenkins.thetaphi.de/job/Lucene-jdk17panama-Windows/701/
> > Java: 64bit/jdk-17 -XX:+UseCompressedOops -XX:+UseG1GC
> >
> > 2 tests failed.
> > FAILED: org.apache.lucene.index.TestIndexWriterMergePolicy.classMethod
> >
> > Error Message:
> > java.lang.AssertionError: The test or suite printed 10871 bytes to stdout and stderr, even though the limit was set to 8192 bytes. Increase the limit with @Limit, ignore it completely with @SuppressSysoutChecks or run with -Dtests.verbose=true
> >
> > Stack Trace:
> > java.lang.AssertionError: The test or suite printed 10871 bytes to stdout and stderr, even though the limit was set to 8192 bytes. Increase the limit with @Limit, ignore it completely with @SuppressSysoutChecks or run with -Dtests.verbose=true
> > at __randomizedtesting.SeedInfo.seed([B67ECC7381FE35B]:0)
> > at org.apache.lucene.util.TestRuleLimitSysouts.afterIfSuccessful(TestRuleLimitSysouts.java:283)
> > at com.carrotsearch.randomizedtesting.rules.TestRuleAdapter$1.afterIfSuccessful(TestRuleAdapter.java:36)
> > at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:37)
> > at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
> > at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
> > at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
> > at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
> > at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:47)
> > at org.junit.rules.RunRules.evaluate(RunRules.java:20)
> > at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
> > at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:370)
> > at com.carrotsearch.randomizedtesting.ThreadLeakControl.lambda$forkTimeoutingTask$0(ThreadLeakControl.java:826)
> > at java.base/java.lang.Thread.run(Thread.java:833)
> >
> >
> > FAILED: org.apache.lucene.index.TestIndexWriterMergePolicy.testStressUpdateSameDocumentWithMergeOnGetReader
> >
> > Error Message:
> > com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=869, name=Thread-739, state=RUNNABLE, group=TGRP-TestIndexWriterMergePolicy]
> >
> > Stack Trace:
> > com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=869, name=Thread-739, state=RUNNABLE, group=TGRP-TestIndexWriterMergePolicy]
> > Caused by: java.lang.AssertionError: java.nio.file.FileSystemException: C:\Users\jenkins\workspace\Lucene-jdk17panama-Windows\lucene\core\build\tmp\tests-tmp\lucene.index.TestIndexWriterMergePolicy_B67ECC7381FE35B-001\index-MMapDirectory-015\_na_Lucene90FieldsIndex-doc_ids_1ak.tmp: Too many open files
> > at __randomizedtesting.SeedInfo.seed([B67ECC7381FE35B]:0)
> > at org.apache.lucene.index.TestIndexWriterMergePolicy.lambda$stressUpdateSameDocumentWithMergeOnX$4(TestIndexWriterMergePolicy.java:752)
> > at java.base/java.lang.Thread.run(Thread.java:833)
> > Caused by: java.nio.file.FileSystemException: C:\Users\jenkins\workspace\Lucene-jdk17panama-Windows\lucene\core\build\tmp\tests-tmp\lucene.index.TestIndexWriterMergePolicy_B67ECC7381FE35B-001\index-MMapDirectory-015\_na_Lucene90FieldsIndex-doc_ids_1ak.tmp: Too many open files
> > at org.apache.lucene.mockfile.HandleLimitFS.onOpen(HandleLimitFS.java:45)
> > at org.apache.lucene.mockfile.HandleTrackingFS.callOpenHook(HandleTrackingFS.java:82)
> > at org.apache.lucene.mockfile.HandleTrackingFS.newOutputStream(HandleTrackingFS.java:163)
> > at org.apache.lucene.mockfile.FilterFileSystemProvider.newOutputStream(FilterFileSystemProvider.java:194)
> > at java.base/java.nio.file.Files.newOutputStream(Files.java:228)
> > at org.apache.lucene.store.FSDirectory$FSIndexOutput.<init>(FSDirectory.java:392)
> > at org.apache.lucene.store.FSDirectory.createTempOutput(FSDirectory.java:234)
> > at org.apache.lucene.store.MockDirectoryWrapper.createTempOutput(MockDirectoryWrapper.java:745)
> > at org.apache.lucene.store.FilterDirectory.createTempOutput(FilterDirectory.java:81)
> > at org.apache.lucene.store.TrackingDirectoryWrapper.createTempOutput(TrackingDirectoryWrapper.java:49)
> > at org.apache.lucene.codecs.lucene90.compressing.FieldsIndexWriter.<init>(FieldsIndexWriter.java:83)
> > at org.apache.lucene.codecs.lucene90.compressing.Lucene90CompressingStoredFieldsWriter.<init>(Lucene90CompressingStoredFieldsWriter.java:148)
> > at org.apache.lucene.codecs.lucene90.compressing.Lucene90CompressingStoredFieldsFormat.fieldsWriter(Lucene90CompressingStoredFieldsFormat.java:140)
> > at org.apache.lucene.codecs.lucene90.Lucene90StoredFieldsFormat.fieldsWriter(Lucene90StoredFieldsFormat.java:154)
> > at org.apache.lucene.codecs.asserting.AssertingStoredFieldsFormat.fieldsWriter(AssertingStoredFieldsFormat.java:48)
> > at org.apache.lucene.index.StoredFieldsConsumer.initStoredFieldsWriter(StoredFieldsConsumer.java:49)
> > at org.apache.lucene.index.StoredFieldsConsumer.startDocument(StoredFieldsConsumer.java:56)
> > at org.apache.lucene.index.IndexingChain.startStoredFields(IndexingChain.java:556)
> > at org.apache.lucene.index.IndexingChain.processDocument(IndexingChain.java:587)
> > at org.apache.lucene.index.DocumentsWriterPerThread.updateDocuments(DocumentsWriterPerThread.java:239)
> > at org.apache.lucene.index.DocumentsWriter.updateDocuments(DocumentsWriter.java:432)
> > at org.apache.lucene.index.IndexWriter.updateDocuments(IndexWriter.java:1530)
> > at org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1815)
> > at org.apache.lucene.index.RandomIndexWriter.updateDocument(RandomIndexWriter.java:305)
> > at org.apache.lucene.index.TestIndexWriterMergePolicy.lambda$stressUpdateSameDocumentWithMergeOnX$4(TestIndexWriterMergePolicy.java:746)
> > ... 1 more
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: builds-unsubscribe@lucene.apache.org
> > For additional commands, e-mail: builds-help@lucene.apache.org

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org

I tried reducing 500 full flushes to 250 and 5000 iters to 2500, and
then the test would fairly reliably finish in 1 minute or so, although
I did once see

* Exception: java.lang.IllegalArgumentException thrown
from the UncaughtExceptionHandler in thread "Lucene Merge Thread #102"

I added an UncaughtExceptionHandler with printStackTrace to threads
created by ConcurrentMergeScheduler but was never able to reproduce
that one

with the larger limits, this test generally seems to fail with OOM, or
else takes so long that I kill it, so I think we should reduce the
limits; then perhaps we might see some more interesting failure modes

On Sat, Nov 27, 2021 at 1:30 PM Michael Sokolov <msokolov@gmail.com> wrote:
>
> Hmm, that test is taking a very long time on my laptop (running
> Ubuntu) with JDK11, so doesn't seem to be a Windows or JDK17 issue
>
> ./gradlew :lucene:core:test --tests
> "org.apache.lucene.index.TestIndexWriterMergePolicy.testStressUpdateSameDocumentWithMergeOnGetReader"
> -Dtests.nightly=true -Dtests.seed=B67ECC7381FE35B
>
> after 12m44s I killed it
>
>
> On Sat, Nov 27, 2021 at 11:59 AM Michael Sokolov <msokolov@gmail.com> wrote:
> >
> > I see that we periodically get various test failures from this test.
> > Has anybody been tracking this more carefully than me, and if so, do
> > you remember if it's always on Windows where these too-many-open-files
> > (and sometimes out of disk space) errors show up? I wonder if we
> > should reduce the cardinality of these test settings:
> >
> > AtomicInteger iters = new AtomicInteger(100 +
> > random().nextInt(TEST_NIGHTLY ? 5000 : 1000));
> > AtomicInteger numFullFlushes =
> > new AtomicInteger(10 + random().nextInt(TEST_NIGHTLY ? 500 : 100));
> >
> > On Sat, Nov 27, 2021 at 5:54 AM Policeman Jenkins Server
> > <jenkins@thetaphi.de> wrote:
> > >
> > > Build: https://jenkins.thetaphi.de/job/Lucene-jdk17panama-Windows/701/
> > > Java: 64bit/jdk-17 -XX:+UseCompressedOops -XX:+UseG1GC
> > >
> > > 2 tests failed.
> > > FAILED: org.apache.lucene.index.TestIndexWriterMergePolicy.classMethod
> > >
> > > Error Message:
> > > java.lang.AssertionError: The test or suite printed 10871 bytes to stdout and stderr, even though the limit was set to 8192 bytes. Increase the limit with @Limit, ignore it completely with @SuppressSysoutChecks or run with -Dtests.verbose=true
> > >
> > > Stack Trace:
> > > java.lang.AssertionError: The test or suite printed 10871 bytes to stdout and stderr, even though the limit was set to 8192 bytes. Increase the limit with @Limit, ignore it completely with @SuppressSysoutChecks or run with -Dtests.verbose=true
> > > at __randomizedtesting.SeedInfo.seed([B67ECC7381FE35B]:0)
> > > at org.apache.lucene.util.TestRuleLimitSysouts.afterIfSuccessful(TestRuleLimitSysouts.java:283)
> > > at com.carrotsearch.randomizedtesting.rules.TestRuleAdapter$1.afterIfSuccessful(TestRuleAdapter.java:36)
> > > at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:37)
> > > at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
> > > at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
> > > at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
> > > at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
> > > at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:47)
> > > at org.junit.rules.RunRules.evaluate(RunRules.java:20)
> > > at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
> > > at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:370)
> > > at com.carrotsearch.randomizedtesting.ThreadLeakControl.lambda$forkTimeoutingTask$0(ThreadLeakControl.java:826)
> > > at java.base/java.lang.Thread.run(Thread.java:833)
> > >
> > >
> > > FAILED: org.apache.lucene.index.TestIndexWriterMergePolicy.testStressUpdateSameDocumentWithMergeOnGetReader
> > >
> > > Error Message:
> > > com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=869, name=Thread-739, state=RUNNABLE, group=TGRP-TestIndexWriterMergePolicy]
> > >
> > > Stack Trace:
> > > com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=869, name=Thread-739, state=RUNNABLE, group=TGRP-TestIndexWriterMergePolicy]
> > > Caused by: java.lang.AssertionError: java.nio.file.FileSystemException: C:\Users\jenkins\workspace\Lucene-jdk17panama-Windows\lucene\core\build\tmp\tests-tmp\lucene.index.TestIndexWriterMergePolicy_B67ECC7381FE35B-001\index-MMapDirectory-015\_na_Lucene90FieldsIndex-doc_ids_1ak.tmp: Too many open files
> > > at __randomizedtesting.SeedInfo.seed([B67ECC7381FE35B]:0)
> > > at org.apache.lucene.index.TestIndexWriterMergePolicy.lambda$stressUpdateSameDocumentWithMergeOnX$4(TestIndexWriterMergePolicy.java:752)
> > > at java.base/java.lang.Thread.run(Thread.java:833)
> > > Caused by: java.nio.file.FileSystemException: C:\Users\jenkins\workspace\Lucene-jdk17panama-Windows\lucene\core\build\tmp\tests-tmp\lucene.index.TestIndexWriterMergePolicy_B67ECC7381FE35B-001\index-MMapDirectory-015\_na_Lucene90FieldsIndex-doc_ids_1ak.tmp: Too many open files
> > > at org.apache.lucene.mockfile.HandleLimitFS.onOpen(HandleLimitFS.java:45)
> > > at org.apache.lucene.mockfile.HandleTrackingFS.callOpenHook(HandleTrackingFS.java:82)
> > > at org.apache.lucene.mockfile.HandleTrackingFS.newOutputStream(HandleTrackingFS.java:163)
> > > at org.apache.lucene.mockfile.FilterFileSystemProvider.newOutputStream(FilterFileSystemProvider.java:194)
> > > at java.base/java.nio.file.Files.newOutputStream(Files.java:228)
> > > at org.apache.lucene.store.FSDirectory$FSIndexOutput.<init>(FSDirectory.java:392)
> > > at org.apache.lucene.store.FSDirectory.createTempOutput(FSDirectory.java:234)
> > > at org.apache.lucene.store.MockDirectoryWrapper.createTempOutput(MockDirectoryWrapper.java:745)
> > > at org.apache.lucene.store.FilterDirectory.createTempOutput(FilterDirectory.java:81)
> > > at org.apache.lucene.store.TrackingDirectoryWrapper.createTempOutput(TrackingDirectoryWrapper.java:49)
> > > at org.apache.lucene.codecs.lucene90.compressing.FieldsIndexWriter.<init>(FieldsIndexWriter.java:83)
> > > at org.apache.lucene.codecs.lucene90.compressing.Lucene90CompressingStoredFieldsWriter.<init>(Lucene90CompressingStoredFieldsWriter.java:148)
> > > at org.apache.lucene.codecs.lucene90.compressing.Lucene90CompressingStoredFieldsFormat.fieldsWriter(Lucene90CompressingStoredFieldsFormat.java:140)
> > > at org.apache.lucene.codecs.lucene90.Lucene90StoredFieldsFormat.fieldsWriter(Lucene90StoredFieldsFormat.java:154)
> > > at org.apache.lucene.codecs.asserting.AssertingStoredFieldsFormat.fieldsWriter(AssertingStoredFieldsFormat.java:48)
> > > at org.apache.lucene.index.StoredFieldsConsumer.initStoredFieldsWriter(StoredFieldsConsumer.java:49)
> > > at org.apache.lucene.index.StoredFieldsConsumer.startDocument(StoredFieldsConsumer.java:56)
> > > at org.apache.lucene.index.IndexingChain.startStoredFields(IndexingChain.java:556)
> > > at org.apache.lucene.index.IndexingChain.processDocument(IndexingChain.java:587)
> > > at org.apache.lucene.index.DocumentsWriterPerThread.updateDocuments(DocumentsWriterPerThread.java:239)
> > > at org.apache.lucene.index.DocumentsWriter.updateDocuments(DocumentsWriter.java:432)
> > > at org.apache.lucene.index.IndexWriter.updateDocuments(IndexWriter.java:1530)
> > > at org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1815)
> > > at org.apache.lucene.index.RandomIndexWriter.updateDocument(RandomIndexWriter.java:305)
> > > at org.apache.lucene.index.TestIndexWriterMergePolicy.lambda$stressUpdateSameDocumentWithMergeOnX$4(TestIndexWriterMergePolicy.java:746)
> > > ... 1 more
> > >
> > > ---------------------------------------------------------------------
> > > To unsubscribe, e-mail: builds-unsubscribe@lucene.apache.org
> > > For additional commands, e-mail: builds-help@lucene.apache.org

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org

It's this issue, Mike:
https://issues.apache.org/jira/browse/LUCENE-10088

On Sat, Nov 27, 2021 at 8:11 PM Michael Sokolov <msokolov@gmail.com> wrote:

> I tried reducing 500 full flushes to 250 and 5000 iters to 2500, and
> then the test would fairly reliably finish in 1 minute or so, although
> I did once see
>
> * Exception: java.lang.IllegalArgumentException thrown
> from the UncaughtExceptionHandler in thread "Lucene Merge Thread #102"
>
> I added an UncaughtExceptionHandler with printStackTrace to threads
> created by ConcurrentMergeScheduler but was never able to reproduce
> that one
>
> with the larger limits, this test generally seems to fail with OOM, or
> else takes so long that I kill it, so I think we should reduce the
> limits; then perhaps we might see some more interesting failure modes
>
> On Sat, Nov 27, 2021 at 1:30 PM Michael Sokolov <msokolov@gmail.com>
> wrote:
> >
> > Hmm, that test is taking a very long time on my laptop (running
> > Ubuntu) with JDK11, so doesn't seem to be a Windows or JDK17 issue
> >
> > ./gradlew :lucene:core:test --tests
> >
> "org.apache.lucene.index.TestIndexWriterMergePolicy.testStressUpdateSameDocumentWithMergeOnGetReader"
> > -Dtests.nightly=true -Dtests.seed=B67ECC7381FE35B
> >
> > after 12m44s I killed it
> >
> >
> > On Sat, Nov 27, 2021 at 11:59 AM Michael Sokolov <msokolov@gmail.com>
> wrote:
> > >
> > > I see that we periodically get various test failures from this test.
> > > Has anybody been tracking this more carefully than me, and if so, do
> > > you remember if it's always on Windows where these too-many-open-files
> > > (and sometimes out of disk space) errors show up? I wonder if we
> > > should reduce the cardinality of these test settings:
> > >
> > > AtomicInteger iters = new AtomicInteger(100 +
> > > random().nextInt(TEST_NIGHTLY ? 5000 : 1000));
> > > AtomicInteger numFullFlushes =
> > > new AtomicInteger(10 + random().nextInt(TEST_NIGHTLY ? 500
> : 100));
> > >
> > > On Sat, Nov 27, 2021 at 5:54 AM Policeman Jenkins Server
> > > <jenkins@thetaphi.de> wrote:
> > > >
> > > > Build:
> https://jenkins.thetaphi.de/job/Lucene-jdk17panama-Windows/701/
> > > > Java: 64bit/jdk-17 -XX:+UseCompressedOops -XX:+UseG1GC
> > > >
> > > > 2 tests failed.
> > > > FAILED:
> org.apache.lucene.index.TestIndexWriterMergePolicy.classMethod
> > > >
> > > > Error Message:
> > > > java.lang.AssertionError: The test or suite printed 10871 bytes to
> stdout and stderr, even though the limit was set to 8192 bytes. Increase
> the limit with @Limit, ignore it completely with @SuppressSysoutChecks or
> run with -Dtests.verbose=true
> > > >
> > > > Stack Trace:
> > > > java.lang.AssertionError: The test or suite printed 10871 bytes to
> stdout and stderr, even though the limit was set to 8192 bytes. Increase
> the limit with @Limit, ignore it completely with @SuppressSysoutChecks or
> run with -Dtests.verbose=true
> > > > at __randomizedtesting.SeedInfo.seed([B67ECC7381FE35B]:0)
> > > > at
> org.apache.lucene.util.TestRuleLimitSysouts.afterIfSuccessful(TestRuleLimitSysouts.java:283)
> > > > at
> com.carrotsearch.randomizedtesting.rules.TestRuleAdapter$1.afterIfSuccessful(TestRuleAdapter.java:36)
> > > > at
> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:37)
> > > > at
> org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
> > > > at
> org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
> > > > at
> org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
> > > > at
> org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
> > > > at
> org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:47)
> > > > at org.junit.rules.RunRules.evaluate(RunRules.java:20)
> > > > at
> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
> > > > at
> com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:370)
> > > > at
> com.carrotsearch.randomizedtesting.ThreadLeakControl.lambda$forkTimeoutingTask$0(ThreadLeakControl.java:826)
> > > > at java.base/java.lang.Thread.run(Thread.java:833)
> > > >
> > > >
> > > > FAILED:
> org.apache.lucene.index.TestIndexWriterMergePolicy.testStressUpdateSameDocumentWithMergeOnGetReader
> > > >
> > > > Error Message:
> > > > com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured
> an uncaught exception in thread: Thread[id=869, name=Thread-739,
> state=RUNNABLE, group=TGRP-TestIndexWriterMergePolicy]
> > > >
> > > > Stack Trace:
> > > > com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured
> an uncaught exception in thread: Thread[id=869, name=Thread-739,
> state=RUNNABLE, group=TGRP-TestIndexWriterMergePolicy]
> > > > Caused by: java.lang.AssertionError:
> java.nio.file.FileSystemException:
> C:\Users\jenkins\workspace\Lucene-jdk17panama-Windows\lucene\core\build\tmp\tests-tmp\lucene.index.TestIndexWriterMergePolicy_B67ECC7381FE35B-001\index-MMapDirectory-015\_na_Lucene90FieldsIndex-doc_ids_1ak.tmp:
> Too many open files
> > > > at __randomizedtesting.SeedInfo.seed([B67ECC7381FE35B]:0)
> > > > at
> org.apache.lucene.index.TestIndexWriterMergePolicy.lambda$stressUpdateSameDocumentWithMergeOnX$4(TestIndexWriterMergePolicy.java:752)
> > > > at java.base/java.lang.Thread.run(Thread.java:833)
> > > > Caused by: java.nio.file.FileSystemException:
> C:\Users\jenkins\workspace\Lucene-jdk17panama-Windows\lucene\core\build\tmp\tests-tmp\lucene.index.TestIndexWriterMergePolicy_B67ECC7381FE35B-001\index-MMapDirectory-015\_na_Lucene90FieldsIndex-doc_ids_1ak.tmp:
> Too many open files
> > > > at
> org.apache.lucene.mockfile.HandleLimitFS.onOpen(HandleLimitFS.java:45)
> > > > at
> org.apache.lucene.mockfile.HandleTrackingFS.callOpenHook(HandleTrackingFS.java:82)
> > > > at
> org.apache.lucene.mockfile.HandleTrackingFS.newOutputStream(HandleTrackingFS.java:163)
> > > > at
> org.apache.lucene.mockfile.FilterFileSystemProvider.newOutputStream(FilterFileSystemProvider.java:194)
> > > > at
> java.base/java.nio.file.Files.newOutputStream(Files.java:228)
> > > > at
> org.apache.lucene.store.FSDirectory$FSIndexOutput.<init>(FSDirectory.java:392)
> > > > at
> org.apache.lucene.store.FSDirectory.createTempOutput(FSDirectory.java:234)
> > > > at
> org.apache.lucene.store.MockDirectoryWrapper.createTempOutput(MockDirectoryWrapper.java:745)
> > > > at
> org.apache.lucene.store.FilterDirectory.createTempOutput(FilterDirectory.java:81)
> > > > at
> org.apache.lucene.store.TrackingDirectoryWrapper.createTempOutput(TrackingDirectoryWrapper.java:49)
> > > > at
> org.apache.lucene.codecs.lucene90.compressing.FieldsIndexWriter.<init>(FieldsIndexWriter.java:83)
> > > > at
> org.apache.lucene.codecs.lucene90.compressing.Lucene90CompressingStoredFieldsWriter.<init>(Lucene90CompressingStoredFieldsWriter.java:148)
> > > > at
> org.apache.lucene.codecs.lucene90.compressing.Lucene90CompressingStoredFieldsFormat.fieldsWriter(Lucene90CompressingStoredFieldsFormat.java:140)
> > > > at
> org.apache.lucene.codecs.lucene90.Lucene90StoredFieldsFormat.fieldsWriter(Lucene90StoredFieldsFormat.java:154)
> > > > at
> org.apache.lucene.codecs.asserting.AssertingStoredFieldsFormat.fieldsWriter(AssertingStoredFieldsFormat.java:48)
> > > > at
> org.apache.lucene.index.StoredFieldsConsumer.initStoredFieldsWriter(StoredFieldsConsumer.java:49)
> > > > at
> org.apache.lucene.index.StoredFieldsConsumer.startDocument(StoredFieldsConsumer.java:56)
> > > > at
> org.apache.lucene.index.IndexingChain.startStoredFields(IndexingChain.java:556)
> > > > at
> org.apache.lucene.index.IndexingChain.processDocument(IndexingChain.java:587)
> > > > at
> org.apache.lucene.index.DocumentsWriterPerThread.updateDocuments(DocumentsWriterPerThread.java:239)
> > > > at
> org.apache.lucene.index.DocumentsWriter.updateDocuments(DocumentsWriter.java:432)
> > > > at
> org.apache.lucene.index.IndexWriter.updateDocuments(IndexWriter.java:1530)
> > > > at
> org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1815)
> > > > at
> org.apache.lucene.index.RandomIndexWriter.updateDocument(RandomIndexWriter.java:305)
> > > > at
> org.apache.lucene.index.TestIndexWriterMergePolicy.lambda$stressUpdateSameDocumentWithMergeOnX$4(TestIndexWriterMergePolicy.java:746)
> > > > ... 1 more
> > > >
> > > > ---------------------------------------------------------------------
> > > > To unsubscribe, e-mail: builds-unsubscribe@lucene.apache.org
> > > > For additional commands, e-mail: builds-help@lucene.apache.org
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: builds-unsubscribe@lucene.apache.org
> For additional commands, e-mail: builds-help@lucene.apache.org
>
>

Oh! Thank you; after reading it I see folks have progressed to a root
cause and a proposed solution.

On Sat, Nov 27, 2021 at 4:24 PM Dawid Weiss <dawid.weiss@gmail.com> wrote:
>
>
> It's this issue, Mike:
> https://issues.apache.org/jira/browse/LUCENE-10088
>
> On Sat, Nov 27, 2021 at 8:11 PM Michael Sokolov <msokolov@gmail.com> wrote:
>>
>> I tried reducing 500 full flushes to 250 and 5000 iters to 2500, and
>> then the test would fairly reliably finish in 1 minute or so, although
>> I did once see
>>
>> * Exception: java.lang.IllegalArgumentException thrown
>> from the UncaughtExceptionHandler in thread "Lucene Merge Thread #102"
>>
>> I added an UncaughtExceptionHandler with printStackTrace to threads
>> created by ConcurrentMergeScheduler but was never able to reproduce
>> that one
>>
>> with the larger limits, this test generally seems to fail with OOM, or
>> else takes so long that I kill it, so I think we should reduce the
>> limits; then perhaps we might see some more interesting failure modes
>>
>> On Sat, Nov 27, 2021 at 1:30 PM Michael Sokolov <msokolov@gmail.com> wrote:
>> >
>> > Hmm, that test is taking a very long time on my laptop (running
>> > Ubuntu) with JDK11, so doesn't seem to be a Windows or JDK17 issue
>> >
>> > ./gradlew :lucene:core:test --tests
>> > "org.apache.lucene.index.TestIndexWriterMergePolicy.testStressUpdateSameDocumentWithMergeOnGetReader"
>> > -Dtests.nightly=true -Dtests.seed=B67ECC7381FE35B
>> >
>> > after 12m44s I killed it
>> >
>> >
>> > On Sat, Nov 27, 2021 at 11:59 AM Michael Sokolov <msokolov@gmail.com> wrote:
>> > >
>> > > I see that we periodically get various test failures from this test.
>> > > Has anybody been tracking this more carefully than me, and if so, do
>> > > you remember if it's always on Windows where these too-many-open-files
>> > > (and sometimes out of disk space) errors show up? I wonder if we
>> > > should reduce the cardinality of these test settings:
>> > >
>> > > AtomicInteger iters = new AtomicInteger(100 +
>> > > random().nextInt(TEST_NIGHTLY ? 5000 : 1000));
>> > > AtomicInteger numFullFlushes =
>> > > new AtomicInteger(10 + random().nextInt(TEST_NIGHTLY ? 500 : 100));
>> > >
>> > > On Sat, Nov 27, 2021 at 5:54 AM Policeman Jenkins Server
>> > > <jenkins@thetaphi.de> wrote:
>> > > >
>> > > > Build: https://jenkins.thetaphi.de/job/Lucene-jdk17panama-Windows/701/
>> > > > Java: 64bit/jdk-17 -XX:+UseCompressedOops -XX:+UseG1GC
>> > > >
>> > > > 2 tests failed.
>> > > > FAILED: org.apache.lucene.index.TestIndexWriterMergePolicy.classMethod
>> > > >
>> > > > Error Message:
>> > > > java.lang.AssertionError: The test or suite printed 10871 bytes to stdout and stderr, even though the limit was set to 8192 bytes. Increase the limit with @Limit, ignore it completely with @SuppressSysoutChecks or run with -Dtests.verbose=true
>> > > >
>> > > > Stack Trace:
>> > > > java.lang.AssertionError: The test or suite printed 10871 bytes to stdout and stderr, even though the limit was set to 8192 bytes. Increase the limit with @Limit, ignore it completely with @SuppressSysoutChecks or run with -Dtests.verbose=true
>> > > > at __randomizedtesting.SeedInfo.seed([B67ECC7381FE35B]:0)
>> > > > at org.apache.lucene.util.TestRuleLimitSysouts.afterIfSuccessful(TestRuleLimitSysouts.java:283)
>> > > > at com.carrotsearch.randomizedtesting.rules.TestRuleAdapter$1.afterIfSuccessful(TestRuleAdapter.java:36)
>> > > > at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:37)
>> > > > at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
>> > > > at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
>> > > > at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
>> > > > at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
>> > > > at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:47)
>> > > > at org.junit.rules.RunRules.evaluate(RunRules.java:20)
>> > > > at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>> > > > at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:370)
>> > > > at com.carrotsearch.randomizedtesting.ThreadLeakControl.lambda$forkTimeoutingTask$0(ThreadLeakControl.java:826)
>> > > > at java.base/java.lang.Thread.run(Thread.java:833)
>> > > >
>> > > >
>> > > > FAILED: org.apache.lucene.index.TestIndexWriterMergePolicy.testStressUpdateSameDocumentWithMergeOnGetReader
>> > > >
>> > > > Error Message:
>> > > > com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=869, name=Thread-739, state=RUNNABLE, group=TGRP-TestIndexWriterMergePolicy]
>> > > >
>> > > > Stack Trace:
>> > > > com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=869, name=Thread-739, state=RUNNABLE, group=TGRP-TestIndexWriterMergePolicy]
>> > > > Caused by: java.lang.AssertionError: java.nio.file.FileSystemException: C:\Users\jenkins\workspace\Lucene-jdk17panama-Windows\lucene\core\build\tmp\tests-tmp\lucene.index.TestIndexWriterMergePolicy_B67ECC7381FE35B-001\index-MMapDirectory-015\_na_Lucene90FieldsIndex-doc_ids_1ak.tmp: Too many open files
>> > > > at __randomizedtesting.SeedInfo.seed([B67ECC7381FE35B]:0)
>> > > > at org.apache.lucene.index.TestIndexWriterMergePolicy.lambda$stressUpdateSameDocumentWithMergeOnX$4(TestIndexWriterMergePolicy.java:752)
>> > > > at java.base/java.lang.Thread.run(Thread.java:833)
>> > > > Caused by: java.nio.file.FileSystemException: C:\Users\jenkins\workspace\Lucene-jdk17panama-Windows\lucene\core\build\tmp\tests-tmp\lucene.index.TestIndexWriterMergePolicy_B67ECC7381FE35B-001\index-MMapDirectory-015\_na_Lucene90FieldsIndex-doc_ids_1ak.tmp: Too many open files
>> > > > at org.apache.lucene.mockfile.HandleLimitFS.onOpen(HandleLimitFS.java:45)
>> > > > at org.apache.lucene.mockfile.HandleTrackingFS.callOpenHook(HandleTrackingFS.java:82)
>> > > > at org.apache.lucene.mockfile.HandleTrackingFS.newOutputStream(HandleTrackingFS.java:163)
>> > > > at org.apache.lucene.mockfile.FilterFileSystemProvider.newOutputStream(FilterFileSystemProvider.java:194)
>> > > > at java.base/java.nio.file.Files.newOutputStream(Files.java:228)
>> > > > at org.apache.lucene.store.FSDirectory$FSIndexOutput.<init>(FSDirectory.java:392)
>> > > > at org.apache.lucene.store.FSDirectory.createTempOutput(FSDirectory.java:234)
>> > > > at org.apache.lucene.store.MockDirectoryWrapper.createTempOutput(MockDirectoryWrapper.java:745)
>> > > > at org.apache.lucene.store.FilterDirectory.createTempOutput(FilterDirectory.java:81)
>> > > > at org.apache.lucene.store.TrackingDirectoryWrapper.createTempOutput(TrackingDirectoryWrapper.java:49)
>> > > > at org.apache.lucene.codecs.lucene90.compressing.FieldsIndexWriter.<init>(FieldsIndexWriter.java:83)
>> > > > at org.apache.lucene.codecs.lucene90.compressing.Lucene90CompressingStoredFieldsWriter.<init>(Lucene90CompressingStoredFieldsWriter.java:148)
>> > > > at org.apache.lucene.codecs.lucene90.compressing.Lucene90CompressingStoredFieldsFormat.fieldsWriter(Lucene90CompressingStoredFieldsFormat.java:140)
>> > > > at org.apache.lucene.codecs.lucene90.Lucene90StoredFieldsFormat.fieldsWriter(Lucene90StoredFieldsFormat.java:154)
>> > > > at org.apache.lucene.codecs.asserting.AssertingStoredFieldsFormat.fieldsWriter(AssertingStoredFieldsFormat.java:48)
>> > > > at org.apache.lucene.index.StoredFieldsConsumer.initStoredFieldsWriter(StoredFieldsConsumer.java:49)
>> > > > at org.apache.lucene.index.StoredFieldsConsumer.startDocument(StoredFieldsConsumer.java:56)
>> > > > at org.apache.lucene.index.IndexingChain.startStoredFields(IndexingChain.java:556)
>> > > > at org.apache.lucene.index.IndexingChain.processDocument(IndexingChain.java:587)
>> > > > at org.apache.lucene.index.DocumentsWriterPerThread.updateDocuments(DocumentsWriterPerThread.java:239)
>> > > > at org.apache.lucene.index.DocumentsWriter.updateDocuments(DocumentsWriter.java:432)
>> > > > at org.apache.lucene.index.IndexWriter.updateDocuments(IndexWriter.java:1530)
>> > > > at org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1815)
>> > > > at org.apache.lucene.index.RandomIndexWriter.updateDocument(RandomIndexWriter.java:305)
>> > > > at org.apache.lucene.index.TestIndexWriterMergePolicy.lambda$stressUpdateSameDocumentWithMergeOnX$4(TestIndexWriterMergePolicy.java:746)
>> > > > ... 1 more
>> > > >
>> > > > ---------------------------------------------------------------------
>> > > > To unsubscribe, e-mail: builds-unsubscribe@lucene.apache.org
>> > > > For additional commands, e-mail: builds-help@lucene.apache.org
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: builds-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: builds-help@lucene.apache.org
>>

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org