Mailing List Archive: Re: [JENKINS] Lucene-MMAPv2-Windows (64bit/hotspot/jdk-21-rc) - Build # 801

Re: [JENKINS] Lucene-MMAPv2-Windows (64bit/hotspot/jdk-21-rc) - Build # 801 - Still Unstable!

Sep 18, 2023, 9:13 AM

Post #1 of 5 (118 views)

Hi,

this issue is a real one. The problem is: The default ForkJoin thread
pool runs all tasks with zero permissions if a security manager is
present. As the MMap Jenkins enforces usage of MMapDirectory for all
tests (it passes -Dtests.directory=MMapDirectory), all disk IO fails.

This will be a big issue for Elasticsearch/Opensearch/Solr if we use the
default thread pool. If this is a test only issue, we should fix it:

* use non-FS-based directory
* use our own thread pool

If this issue is in 9.8 branch we have to fix it!

Uwe

Am 18.09.2023 um 17:59 schrieb Policeman Jenkins Server:
> Build:https://jenkins.thetaphi.de/job/Lucene-MMAPv2-Windows/801/
> Java: 64bit/hotspot/jdk-21-rc -XX:-UseCompressedOops -XX:+UseG1GC
>
> 1 tests failed.
> FAILED: org.apache.lucene.misc.index.TestBPIndexReorderer.testSingleTermWithForkJoinPool
>
> Error Message:
> java.security.AccessControlException: access denied ("java.io.FilePermission" "C:\Users\jenkins\workspace\Lucene-MMAPv2-Windows\lucene\misc\build\tmp\tests-tmp\lucene.misc.index.TestBPIndexReorderer_4B02FABB1F62D832-001\index-MMapDirectory-003\forward-index_sort_5.tmp" "write")
>
> Stack Trace:
> java.security.AccessControlException: access denied ("java.io.FilePermission" "C:\Users\jenkins\workspace\Lucene-MMAPv2-Windows\lucene\misc\build\tmp\tests-tmp\lucene.misc.index.TestBPIndexReorderer_4B02FABB1F62D832-001\index-MMapDirectory-003\forward-index_sort_5.tmp" "write")
> at __randomizedtesting.SeedInfo.seed([4B02FABB1F62D832:77694EDC9D6E8956]:0)
> at java.base/java.security.AccessControlContext.checkPermission(AccessControlContext.java:488)
> at java.base/java.security.AccessController.checkPermission(AccessController.java:1071)
> at java.base/java.lang.SecurityManager.checkPermission(SecurityManager.java:411)
> at java.base/java.lang.SecurityManager.checkWrite(SecurityManager.java:833)
> at java.base/sun.nio.fs.WindowsChannelFactory.open(WindowsChannelFactory.java:302)
> at java.base/sun.nio.fs.WindowsChannelFactory.newFileChannel(WindowsChannelFactory.java:168)
> at java.base/sun.nio.fs.WindowsFileSystemProvider.newByteChannel(WindowsFileSystemProvider.java:229)
> at java.base/java.nio.file.spi.FileSystemProvider.newOutputStream(FileSystemProvider.java:482)
> atorg.apache.lucene.test_framework@10.0.0-SNAPSHOT/org.apache.lucene.tests.mockfile.FilterFileSystemProvider.newOutputStream(FilterFileSystemProvider.java:198)
> atorg.apache.lucene.test_framework@10.0.0-SNAPSHOT/org.apache.lucene.tests.mockfile.FilterFileSystemProvider.newOutputStream(FilterFileSystemProvider.java:198)
> atorg.apache.lucene.test_framework@10.0.0-SNAPSHOT/org.apache.lucene.tests.mockfile.HandleTrackingFS.newOutputStream(HandleTrackingFS.java:132)
> atorg.apache.lucene.test_framework@10.0.0-SNAPSHOT/org.apache.lucene.tests.mockfile.HandleTrackingFS.newOutputStream(HandleTrackingFS.java:132)
> atorg.apache.lucene.test_framework@10.0.0-SNAPSHOT/org.apache.lucene.tests.mockfile.FilterFileSystemProvider.newOutputStream(FilterFileSystemProvider.java:198)
> at java.base/java.nio.file.Files.newOutputStream(Files.java:227)
> atorg.apache.lucene.core@10.0.0-SNAPSHOT/org.apache.lucene.store.FSDirectory$FSIndexOutput.<init>(FSDirectory.java:394)
> atorg.apache.lucene.core@10.0.0-SNAPSHOT/org.apache.lucene.store.FSDirectory.createTempOutput(FSDirectory.java:234)
> atorg.apache.lucene.test_framework@10.0.0-SNAPSHOT/org.apache.lucene.tests.store.MockDirectoryWrapper.createTempOutput(MockDirectoryWrapper.java:752)
> atorg.apache.lucene.core@10.0.0-SNAPSHOT/org.apache.lucene.store.TrackingDirectoryWrapper.createTempOutput(TrackingDirectoryWrapper.java:49)
> atorg.apache.lucene.core@10.0.0-SNAPSHOT/org.apache.lucene.store.TrackingDirectoryWrapper.createTempOutput(TrackingDirectoryWrapper.java:49)
> atorg.apache.lucene.core@10.0.0-SNAPSHOT/org.apache.lucene.util.OfflineSorter$SortPartitionTask.call(OfflineSorter.java:623)
> atorg.apache.lucene.core@10.0.0-SNAPSHOT/org.apache.lucene.util.OfflineSorter$SortPartitionTask.call(OfflineSorter.java:610)
> at java.base/java.util.concurrent.ForkJoinTask$AdaptedCallable.exec(ForkJoinTask.java:1456)
> at java.base/java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:387)
> at java.base/java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(ForkJoinPool.java:1312)
> at java.base/java.util.concurrent.ForkJoinPool.scan(ForkJoinPool.java:1843)
> at java.base/java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1808)
> at java.base/java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:188)
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail:builds-unsubscribe@lucene.apache.org
> For additional commands, e-mail:builds-help@lucene.apache.org

--
Uwe Schindler
Achterdiek 19, D-28357 Bremen
https://www.thetaphi.de
eMail:uwe@thetaphi.de

Re: [JENKINS] Lucene-MMAPv2-Windows (64bit/hotspot/jdk-21-rc) - Build # 801 - Still Unstable! [ In reply to ]

jpountz at gmail

Sep 18, 2023, 9:18 AM

Post #2 of 5 (118 views)

Permalink

Thanks Uwe for digging. The fork-join pool is optional, I will change
the test to use a ByteBuffersDirectory.

On Mon, Sep 18, 2023 at 6:15?PM Uwe Schindler <uwe@thetaphi.de> wrote:
>
> Hi,
>
> this issue is a real one. The problem is: The default ForkJoin thread pool runs all tasks with zero permissions if a security manager is present. As the MMap Jenkins enforces usage of MMapDirectory for all tests (it passes -Dtests.directory=MMapDirectory), all disk IO fails.
>
> This will be a big issue for Elasticsearch/Opensearch/Solr if we use the default thread pool. If this is a test only issue, we should fix it:
>
> use non-FS-based directory
> use our own thread pool
>
> If this issue is in 9.8 branch we have to fix it!
>
> Uwe
>
> Am 18.09.2023 um 17:59 schrieb Policeman Jenkins Server:
>
> Build: https://jenkins.thetaphi.de/job/Lucene-MMAPv2-Windows/801/
> Java: 64bit/hotspot/jdk-21-rc -XX:-UseCompressedOops -XX:+UseG1GC
>
> 1 tests failed.
> FAILED: org.apache.lucene.misc.index.TestBPIndexReorderer.testSingleTermWithForkJoinPool
>
> Error Message:
> java.security.AccessControlException: access denied ("java.io.FilePermission" "C:\Users\jenkins\workspace\Lucene-MMAPv2-Windows\lucene\misc\build\tmp\tests-tmp\lucene.misc.index.TestBPIndexReorderer_4B02FABB1F62D832-001\index-MMapDirectory-003\forward-index_sort_5.tmp" "write")
>
> Stack Trace:
> java.security.AccessControlException: access denied ("java.io.FilePermission" "C:\Users\jenkins\workspace\Lucene-MMAPv2-Windows\lucene\misc\build\tmp\tests-tmp\lucene.misc.index.TestBPIndexReorderer_4B02FABB1F62D832-001\index-MMapDirectory-003\forward-index_sort_5.tmp" "write")
> at __randomizedtesting.SeedInfo.seed([4B02FABB1F62D832:77694EDC9D6E8956]:0)
> at java.base/java.security.AccessControlContext.checkPermission(AccessControlContext.java:488)
> at java.base/java.security.AccessController.checkPermission(AccessController.java:1071)
> at java.base/java.lang.SecurityManager.checkPermission(SecurityManager.java:411)
> at java.base/java.lang.SecurityManager.checkWrite(SecurityManager.java:833)
> at java.base/sun.nio.fs.WindowsChannelFactory.open(WindowsChannelFactory.java:302)
> at java.base/sun.nio.fs.WindowsChannelFactory.newFileChannel(WindowsChannelFactory.java:168)
> at java.base/sun.nio.fs.WindowsFileSystemProvider.newByteChannel(WindowsFileSystemProvider.java:229)
> at java.base/java.nio.file.spi.FileSystemProvider.newOutputStream(FileSystemProvider.java:482)
> at org.apache.lucene.test_framework@10.0.0-SNAPSHOT/org.apache.lucene.tests.mockfile.FilterFileSystemProvider.newOutputStream(FilterFileSystemProvider.java:198)
> at org.apache.lucene.test_framework@10.0.0-SNAPSHOT/org.apache.lucene.tests.mockfile.FilterFileSystemProvider.newOutputStream(FilterFileSystemProvider.java:198)
> at org.apache.lucene.test_framework@10.0.0-SNAPSHOT/org.apache.lucene.tests.mockfile.HandleTrackingFS.newOutputStream(HandleTrackingFS.java:132)
> at org.apache.lucene.test_framework@10.0.0-SNAPSHOT/org.apache.lucene.tests.mockfile.HandleTrackingFS.newOutputStream(HandleTrackingFS.java:132)
> at org.apache.lucene.test_framework@10.0.0-SNAPSHOT/org.apache.lucene.tests.mockfile.FilterFileSystemProvider.newOutputStream(FilterFileSystemProvider.java:198)
> at java.base/java.nio.file.Files.newOutputStream(Files.java:227)
> at org.apache.lucene.core@10.0.0-SNAPSHOT/org.apache.lucene.store.FSDirectory$FSIndexOutput.<init>(FSDirectory.java:394)
> at org.apache.lucene.core@10.0.0-SNAPSHOT/org.apache.lucene.store.FSDirectory.createTempOutput(FSDirectory.java:234)
> at org.apache.lucene.test_framework@10.0.0-SNAPSHOT/org.apache.lucene.tests.store.MockDirectoryWrapper.createTempOutput(MockDirectoryWrapper.java:752)
> at org.apache.lucene.core@10.0.0-SNAPSHOT/org.apache.lucene.store.TrackingDirectoryWrapper.createTempOutput(TrackingDirectoryWrapper.java:49)
> at org.apache.lucene.core@10.0.0-SNAPSHOT/org.apache.lucene.store.TrackingDirectoryWrapper.createTempOutput(TrackingDirectoryWrapper.java:49)
> at org.apache.lucene.core@10.0.0-SNAPSHOT/org.apache.lucene.util.OfflineSorter$SortPartitionTask.call(OfflineSorter.java:623)
> at org.apache.lucene.core@10.0.0-SNAPSHOT/org.apache.lucene.util.OfflineSorter$SortPartitionTask.call(OfflineSorter.java:610)
> at java.base/java.util.concurrent.ForkJoinTask$AdaptedCallable.exec(ForkJoinTask.java:1456)
> at java.base/java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:387)
> at java.base/java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(ForkJoinPool.java:1312)
> at java.base/java.util.concurrent.ForkJoinPool.scan(ForkJoinPool.java:1843)
> at java.base/java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1808)
> at java.base/java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:188)
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: builds-unsubscribe@lucene.apache.org
> For additional commands, e-mail: builds-help@lucene.apache.org
>
> --
> Uwe Schindler
> Achterdiek 19, D-28357 Bremen
> https://www.thetaphi.de
> eMail: uwe@thetaphi.de

--
Adrien

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org

Re: [JENKINS] Lucene-MMAPv2-Windows (64bit/hotspot/jdk-21-rc) - Build # 801 - Still Unstable! [ In reply to ]

uwe at thetaphi

Sep 18, 2023, 9:24 AM

Post #3 of 5 (118 views)

Permalink

It may still be a good idea to show an example how to pass a
ForkJoinPool to the sorter that does not limit permissions (just
examples for the educated reader).

Uwe

Am 18.09.2023 um 18:18 schrieb Adrien Grand:
> Thanks Uwe for digging. The fork-join pool is optional, I will change
> the test to use a ByteBuffersDirectory.
>
> On Mon, Sep 18, 2023 at 6:15?PM Uwe Schindler <uwe@thetaphi.de> wrote:
>> Hi,
>>
>> this issue is a real one. The problem is: The default ForkJoin thread pool runs all tasks with zero permissions if a security manager is present. As the MMap Jenkins enforces usage of MMapDirectory for all tests (it passes -Dtests.directory=MMapDirectory), all disk IO fails.
>>
>> This will be a big issue for Elasticsearch/Opensearch/Solr if we use the default thread pool. If this is a test only issue, we should fix it:
>>
>> use non-FS-based directory
>> use our own thread pool
>>
>> If this issue is in 9.8 branch we have to fix it!
>>
>> Uwe
>>
>> Am 18.09.2023 um 17:59 schrieb Policeman Jenkins Server:
>>
>> Build: https://jenkins.thetaphi.de/job/Lucene-MMAPv2-Windows/801/
>> Java: 64bit/hotspot/jdk-21-rc -XX:-UseCompressedOops -XX:+UseG1GC
>>
>> 1 tests failed.
>> FAILED: org.apache.lucene.misc.index.TestBPIndexReorderer.testSingleTermWithForkJoinPool
>>
>> Error Message:
>> java.security.AccessControlException: access denied ("java.io.FilePermission" "C:\Users\jenkins\workspace\Lucene-MMAPv2-Windows\lucene\misc\build\tmp\tests-tmp\lucene.misc.index.TestBPIndexReorderer_4B02FABB1F62D832-001\index-MMapDirectory-003\forward-index_sort_5.tmp" "write")
>>
>> Stack Trace:
>> java.security.AccessControlException: access denied ("java.io.FilePermission" "C:\Users\jenkins\workspace\Lucene-MMAPv2-Windows\lucene\misc\build\tmp\tests-tmp\lucene.misc.index.TestBPIndexReorderer_4B02FABB1F62D832-001\index-MMapDirectory-003\forward-index_sort_5.tmp" "write")
>> at __randomizedtesting.SeedInfo.seed([4B02FABB1F62D832:77694EDC9D6E8956]:0)
>> at java.base/java.security.AccessControlContext.checkPermission(AccessControlContext.java:488)
>> at java.base/java.security.AccessController.checkPermission(AccessController.java:1071)
>> at java.base/java.lang.SecurityManager.checkPermission(SecurityManager.java:411)
>> at java.base/java.lang.SecurityManager.checkWrite(SecurityManager.java:833)
>> at java.base/sun.nio.fs.WindowsChannelFactory.open(WindowsChannelFactory.java:302)
>> at java.base/sun.nio.fs.WindowsChannelFactory.newFileChannel(WindowsChannelFactory.java:168)
>> at java.base/sun.nio.fs.WindowsFileSystemProvider.newByteChannel(WindowsFileSystemProvider.java:229)
>> at java.base/java.nio.file.spi.FileSystemProvider.newOutputStream(FileSystemProvider.java:482)
>> at org.apache.lucene.test_framework@10.0.0-SNAPSHOT/org.apache.lucene.tests.mockfile.FilterFileSystemProvider.newOutputStream(FilterFileSystemProvider.java:198)
>> at org.apache.lucene.test_framework@10.0.0-SNAPSHOT/org.apache.lucene.tests.mockfile.FilterFileSystemProvider.newOutputStream(FilterFileSystemProvider.java:198)
>> at org.apache.lucene.test_framework@10.0.0-SNAPSHOT/org.apache.lucene.tests.mockfile.HandleTrackingFS.newOutputStream(HandleTrackingFS.java:132)
>> at org.apache.lucene.test_framework@10.0.0-SNAPSHOT/org.apache.lucene.tests.mockfile.HandleTrackingFS.newOutputStream(HandleTrackingFS.java:132)
>> at org.apache.lucene.test_framework@10.0.0-SNAPSHOT/org.apache.lucene.tests.mockfile.FilterFileSystemProvider.newOutputStream(FilterFileSystemProvider.java:198)
>> at java.base/java.nio.file.Files.newOutputStream(Files.java:227)
>> at org.apache.lucene.core@10.0.0-SNAPSHOT/org.apache.lucene.store.FSDirectory$FSIndexOutput.<init>(FSDirectory.java:394)
>> at org.apache.lucene.core@10.0.0-SNAPSHOT/org.apache.lucene.store.FSDirectory.createTempOutput(FSDirectory.java:234)
>> at org.apache.lucene.test_framework@10.0.0-SNAPSHOT/org.apache.lucene.tests.store.MockDirectoryWrapper.createTempOutput(MockDirectoryWrapper.java:752)
>> at org.apache.lucene.core@10.0.0-SNAPSHOT/org.apache.lucene.store.TrackingDirectoryWrapper.createTempOutput(TrackingDirectoryWrapper.java:49)
>> at org.apache.lucene.core@10.0.0-SNAPSHOT/org.apache.lucene.store.TrackingDirectoryWrapper.createTempOutput(TrackingDirectoryWrapper.java:49)
>> at org.apache.lucene.core@10.0.0-SNAPSHOT/org.apache.lucene.util.OfflineSorter$SortPartitionTask.call(OfflineSorter.java:623)
>> at org.apache.lucene.core@10.0.0-SNAPSHOT/org.apache.lucene.util.OfflineSorter$SortPartitionTask.call(OfflineSorter.java:610)
>> at java.base/java.util.concurrent.ForkJoinTask$AdaptedCallable.exec(ForkJoinTask.java:1456)
>> at java.base/java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:387)
>> at java.base/java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(ForkJoinPool.java:1312)
>> at java.base/java.util.concurrent.ForkJoinPool.scan(ForkJoinPool.java:1843)
>> at java.base/java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1808)
>> at java.base/java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:188)
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: builds-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: builds-help@lucene.apache.org
>>
>> --
>> Uwe Schindler
>> Achterdiek 19, D-28357 Bremen
>> https://www.thetaphi.de
>> eMail: uwe@thetaphi.de
>
>
--
Uwe Schindler
Achterdiek 19, D-28357 Bremen
https://www.thetaphi.de
eMail: uwe@thetaphi.de

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org

Re: [JENKINS] Lucene-MMAPv2-Windows (64bit/hotspot/jdk-21-rc) - Build # 801 - Still Unstable! [ In reply to ]

zhai7631 at gmail

Sep 18, 2023, 9:31 AM

Post #4 of 5 (118 views)

Permalink

Thanks Uwe and Adrien, I haven't cut the branch, will wait for this fix then

On Mon, Sep 18, 2023 at 9:28?AM Adrien Grand <jpountz@gmail.com> wrote:

> Thanks Uwe for digging. The fork-join pool is optional, I will change
> the test to use a ByteBuffersDirectory.
>
> On Mon, Sep 18, 2023 at 6:15?PM Uwe Schindler <uwe@thetaphi.de> wrote:
> >
> > Hi,
> >
> > this issue is a real one. The problem is: The default ForkJoin thread
> pool runs all tasks with zero permissions if a security manager is present.
> As the MMap Jenkins enforces usage of MMapDirectory for all tests (it
> passes -Dtests.directory=MMapDirectory), all disk IO fails.
> >
> > This will be a big issue for Elasticsearch/Opensearch/Solr if we use the
> default thread pool. If this is a test only issue, we should fix it:
> >
> > use non-FS-based directory
> > use our own thread pool
> >
> > If this issue is in 9.8 branch we have to fix it!
> >
> > Uwe
> >
> > Am 18.09.2023 um 17:59 schrieb Policeman Jenkins Server:
> >
> > Build: https://jenkins.thetaphi.de/job/Lucene-MMAPv2-Windows/801/
> > Java: 64bit/hotspot/jdk-21-rc -XX:-UseCompressedOops -XX:+UseG1GC
> >
> > 1 tests failed.
> > FAILED:
> org.apache.lucene.misc.index.TestBPIndexReorderer.testSingleTermWithForkJoinPool
> >
> > Error Message:
> > java.security.AccessControlException: access denied
> ("java.io.FilePermission"
> "C:\Users\jenkins\workspace\Lucene-MMAPv2-Windows\lucene\misc\build\tmp\tests-tmp\lucene.misc.index.TestBPIndexReorderer_4B02FABB1F62D832-001\index-MMapDirectory-003\forward-index_sort_5.tmp"
> "write")
> >
> > Stack Trace:
> > java.security.AccessControlException: access denied
> ("java.io.FilePermission"
> "C:\Users\jenkins\workspace\Lucene-MMAPv2-Windows\lucene\misc\build\tmp\tests-tmp\lucene.misc.index.TestBPIndexReorderer_4B02FABB1F62D832-001\index-MMapDirectory-003\forward-index_sort_5.tmp"
> "write")
> > at
> __randomizedtesting.SeedInfo.seed([4B02FABB1F62D832:77694EDC9D6E8956]:0)
> > at
> java.base/java.security.AccessControlContext.checkPermission(AccessControlContext.java:488)
> > at
> java.base/java.security.AccessController.checkPermission(AccessController.java:1071)
> > at
> java.base/java.lang.SecurityManager.checkPermission(SecurityManager.java:411)
> > at
> java.base/java.lang.SecurityManager.checkWrite(SecurityManager.java:833)
> > at
> java.base/sun.nio.fs.WindowsChannelFactory.open(WindowsChannelFactory.java:302)
> > at
> java.base/sun.nio.fs.WindowsChannelFactory.newFileChannel(WindowsChannelFactory.java:168)
> > at
> java.base/sun.nio.fs.WindowsFileSystemProvider.newByteChannel(WindowsFileSystemProvider.java:229)
> > at
> java.base/java.nio.file.spi.FileSystemProvider.newOutputStream(FileSystemProvider.java:482)
> > at org.apache.lucene.test_framework@10.0.0-SNAPSHOT
> /org.apache.lucene.tests.mockfile.FilterFileSystemProvider.newOutputStream(FilterFileSystemProvider.java:198)
> > at org.apache.lucene.test_framework@10.0.0-SNAPSHOT
> /org.apache.lucene.tests.mockfile.FilterFileSystemProvider.newOutputStream(FilterFileSystemProvider.java:198)
> > at org.apache.lucene.test_framework@10.0.0-SNAPSHOT
> /org.apache.lucene.tests.mockfile.HandleTrackingFS.newOutputStream(HandleTrackingFS.java:132)
> > at org.apache.lucene.test_framework@10.0.0-SNAPSHOT
> /org.apache.lucene.tests.mockfile.HandleTrackingFS.newOutputStream(HandleTrackingFS.java:132)
> > at org.apache.lucene.test_framework@10.0.0-SNAPSHOT
> /org.apache.lucene.tests.mockfile.FilterFileSystemProvider.newOutputStream(FilterFileSystemProvider.java:198)
> > at java.base/java.nio.file.Files.newOutputStream(Files.java:227)
> > at org.apache.lucene.core@10.0.0-SNAPSHOT
> /org.apache.lucene.store.FSDirectory$FSIndexOutput.<init>(FSDirectory.java:394)
> > at org.apache.lucene.core@10.0.0-SNAPSHOT
> /org.apache.lucene.store.FSDirectory.createTempOutput(FSDirectory.java:234)
> > at org.apache.lucene.test_framework@10.0.0-SNAPSHOT
> /org.apache.lucene.tests.store.MockDirectoryWrapper.createTempOutput(MockDirectoryWrapper.java:752)
> > at org.apache.lucene.core@10.0.0-SNAPSHOT
> /org.apache.lucene.store.TrackingDirectoryWrapper.createTempOutput(TrackingDirectoryWrapper.java:49)
> > at org.apache.lucene.core@10.0.0-SNAPSHOT
> /org.apache.lucene.store.TrackingDirectoryWrapper.createTempOutput(TrackingDirectoryWrapper.java:49)
> > at org.apache.lucene.core@10.0.0-SNAPSHOT
> /org.apache.lucene.util.OfflineSorter$SortPartitionTask.call(OfflineSorter.java:623)
> > at org.apache.lucene.core@10.0.0-SNAPSHOT
> /org.apache.lucene.util.OfflineSorter$SortPartitionTask.call(OfflineSorter.java:610)
> > at
> java.base/java.util.concurrent.ForkJoinTask$AdaptedCallable.exec(ForkJoinTask.java:1456)
> > at
> java.base/java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:387)
> > at
> java.base/java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(ForkJoinPool.java:1312)
> > at
> java.base/java.util.concurrent.ForkJoinPool.scan(ForkJoinPool.java:1843)
> > at
> java.base/java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1808)
> > at
> java.base/java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:188)
> >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: builds-unsubscribe@lucene.apache.org
> > For additional commands, e-mail: builds-help@lucene.apache.org
> >
> > --
> > Uwe Schindler
> > Achterdiek 19, D-28357 Bremen
> > https://www.thetaphi.de
> > eMail: uwe@thetaphi.de
>
>
>
> --
> Adrien
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: dev-help@lucene.apache.org
>
>

Re: [JENKINS] Lucene-MMAPv2-Windows (64bit/hotspot/jdk-21-rc) - Build # 801 - Still Unstable! [ In reply to ]

jpountz at gmail

Sep 18, 2023, 9:35 AM

Post #5 of 5 (118 views)

Permalink

OK I just did that too, see https://github.com/apache/lucene/pull/12568.

On Mon, Sep 18, 2023 at 6:32?PM Uwe Schindler <uwe@thetaphi.de> wrote:
>
> It may still be a good idea to show an example how to pass a
> ForkJoinPool to the sorter that does not limit permissions (just
> examples for the educated reader).
>
> Uwe
>
> Am 18.09.2023 um 18:18 schrieb Adrien Grand:
> > Thanks Uwe for digging. The fork-join pool is optional, I will change
> > the test to use a ByteBuffersDirectory.
> >
> > On Mon, Sep 18, 2023 at 6:15?PM Uwe Schindler <uwe@thetaphi.de> wrote:
> >> Hi,
> >>
> >> this issue is a real one. The problem is: The default ForkJoin thread pool runs all tasks with zero permissions if a security manager is present. As the MMap Jenkins enforces usage of MMapDirectory for all tests (it passes -Dtests.directory=MMapDirectory), all disk IO fails.
> >>
> >> This will be a big issue for Elasticsearch/Opensearch/Solr if we use the default thread pool. If this is a test only issue, we should fix it:
> >>
> >> use non-FS-based directory
> >> use our own thread pool
> >>
> >> If this issue is in 9.8 branch we have to fix it!
> >>
> >> Uwe
> >>
> >> Am 18.09.2023 um 17:59 schrieb Policeman Jenkins Server:
> >>
> >> Build: https://jenkins.thetaphi.de/job/Lucene-MMAPv2-Windows/801/
> >> Java: 64bit/hotspot/jdk-21-rc -XX:-UseCompressedOops -XX:+UseG1GC
> >>
> >> 1 tests failed.
> >> FAILED: org.apache.lucene.misc.index.TestBPIndexReorderer.testSingleTermWithForkJoinPool
> >>
> >> Error Message:
> >> java.security.AccessControlException: access denied ("java.io.FilePermission" "C:\Users\jenkins\workspace\Lucene-MMAPv2-Windows\lucene\misc\build\tmp\tests-tmp\lucene.misc.index.TestBPIndexReorderer_4B02FABB1F62D832-001\index-MMapDirectory-003\forward-index_sort_5.tmp" "write")
> >>
> >> Stack Trace:
> >> java.security.AccessControlException: access denied ("java.io.FilePermission" "C:\Users\jenkins\workspace\Lucene-MMAPv2-Windows\lucene\misc\build\tmp\tests-tmp\lucene.misc.index.TestBPIndexReorderer_4B02FABB1F62D832-001\index-MMapDirectory-003\forward-index_sort_5.tmp" "write")
> >> at __randomizedtesting.SeedInfo.seed([4B02FABB1F62D832:77694EDC9D6E8956]:0)
> >> at java.base/java.security.AccessControlContext.checkPermission(AccessControlContext.java:488)
> >> at java.base/java.security.AccessController.checkPermission(AccessController.java:1071)
> >> at java.base/java.lang.SecurityManager.checkPermission(SecurityManager.java:411)
> >> at java.base/java.lang.SecurityManager.checkWrite(SecurityManager.java:833)
> >> at java.base/sun.nio.fs.WindowsChannelFactory.open(WindowsChannelFactory.java:302)
> >> at java.base/sun.nio.fs.WindowsChannelFactory.newFileChannel(WindowsChannelFactory.java:168)
> >> at java.base/sun.nio.fs.WindowsFileSystemProvider.newByteChannel(WindowsFileSystemProvider.java:229)
> >> at java.base/java.nio.file.spi.FileSystemProvider.newOutputStream(FileSystemProvider.java:482)
> >> at org.apache.lucene.test_framework@10.0.0-SNAPSHOT/org.apache.lucene.tests.mockfile.FilterFileSystemProvider.newOutputStream(FilterFileSystemProvider.java:198)
> >> at org.apache.lucene.test_framework@10.0.0-SNAPSHOT/org.apache.lucene.tests.mockfile.FilterFileSystemProvider.newOutputStream(FilterFileSystemProvider.java:198)
> >> at org.apache.lucene.test_framework@10.0.0-SNAPSHOT/org.apache.lucene.tests.mockfile.HandleTrackingFS.newOutputStream(HandleTrackingFS.java:132)
> >> at org.apache.lucene.test_framework@10.0.0-SNAPSHOT/org.apache.lucene.tests.mockfile.HandleTrackingFS.newOutputStream(HandleTrackingFS.java:132)
> >> at org.apache.lucene.test_framework@10.0.0-SNAPSHOT/org.apache.lucene.tests.mockfile.FilterFileSystemProvider.newOutputStream(FilterFileSystemProvider.java:198)
> >> at java.base/java.nio.file.Files.newOutputStream(Files.java:227)
> >> at org.apache.lucene.core@10.0.0-SNAPSHOT/org.apache.lucene.store.FSDirectory$FSIndexOutput.<init>(FSDirectory.java:394)
> >> at org.apache.lucene.core@10.0.0-SNAPSHOT/org.apache.lucene.store.FSDirectory.createTempOutput(FSDirectory.java:234)
> >> at org.apache.lucene.test_framework@10.0.0-SNAPSHOT/org.apache.lucene.tests.store.MockDirectoryWrapper.createTempOutput(MockDirectoryWrapper.java:752)
> >> at org.apache.lucene.core@10.0.0-SNAPSHOT/org.apache.lucene.store.TrackingDirectoryWrapper.createTempOutput(TrackingDirectoryWrapper.java:49)
> >> at org.apache.lucene.core@10.0.0-SNAPSHOT/org.apache.lucene.store.TrackingDirectoryWrapper.createTempOutput(TrackingDirectoryWrapper.java:49)
> >> at org.apache.lucene.core@10.0.0-SNAPSHOT/org.apache.lucene.util.OfflineSorter$SortPartitionTask.call(OfflineSorter.java:623)
> >> at org.apache.lucene.core@10.0.0-SNAPSHOT/org.apache.lucene.util.OfflineSorter$SortPartitionTask.call(OfflineSorter.java:610)
> >> at java.base/java.util.concurrent.ForkJoinTask$AdaptedCallable.exec(ForkJoinTask.java:1456)
> >> at java.base/java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:387)
> >> at java.base/java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(ForkJoinPool.java:1312)
> >> at java.base/java.util.concurrent.ForkJoinPool.scan(ForkJoinPool.java:1843)
> >> at java.base/java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1808)
> >> at java.base/java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:188)
> >>
> >>
> >> ---------------------------------------------------------------------
> >> To unsubscribe, e-mail: builds-unsubscribe@lucene.apache.org
> >> For additional commands, e-mail: builds-help@lucene.apache.org
> >>
> >> --
> >> Uwe Schindler
> >> Achterdiek 19, D-28357 Bremen
> >> https://www.thetaphi.de
> >> eMail: uwe@thetaphi.de
> >
> >
> --
> Uwe Schindler
> Achterdiek 19, D-28357 Bremen
> https://www.thetaphi.de
> eMail: uwe@thetaphi.de
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: dev-help@lucene.apache.org
>

--
Adrien

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org