Mailing List Archive

Recurring index corruption
We are experimenting with Elastic Search deployed in Azure Container Instances (Debian + OpenJDK). The ES indexes are stored into an Azure file share mounted via SMB (3.0). The Elastic Search cluster is made up of 4 nodes, each one have a separate file share to store the indices.

This configuration has been influenced by some ACIs limitations, specifically:

we cannot set the max_map_count value as we do not have access to the underlying host (https://www.elastic.co/guide/en/elasticsearch/reference/current/vm-max-map-count.html). Unfortunately, this is required to run an ES cluster, therefore we were forced to use NIOF
ACI’s storage is ephemera, therefore we had to map volumes to persist the indexes. ACIs only allow volume mappings using Azure File Shares, which only works with NFS or SMB.

We are experiencing recurring index corruption, specifically a "read past EOF" exception. I asked on the Elastic Search forum but the answer I got was a bit generic and not really helpful other than confirming that, from ES point of view, ES should work on an SMB share as long as it behaves as a local drive. As the underlying exception relates to an issue with a Lucene index, I was wondering if you could help out? Specifically, can Lucene work on SMB? I can only find sparse information on this configuration and, while NFS seems a no-no, for SMB is not that clear. Below is the exception we are getting.

java.io.IOException: read past EOF: NIOFSIndexInput(path="/bitnami/elasticsearch/data/indices/mS2bUbLtSeG0FSAMuKX7JQ/0/index/_ldsn_1.fnm") buffer: java.nio.HeapByteBuffer[pos=0 lim=1024 cap=1024] chunkLen: 1024 end: 2331: NIOFSIndexInput(path="/bitnami/elasticsearch/data/indices/mS2bUbLtSeG0FSAMuKX7JQ/0/index/_ldsn_1.fnm")
at org.apache.lucene.store.NIOFSDirectory$NIOFSIndexInput.readInternal(NIOFSDirectory.java:200) ~[lucene-core-9.3.0.jar:?]
at org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.java:291) ~[lucene-core-9.3.0.jar:?]
at org.apache.lucene.store.BufferedIndexInput.readByte(BufferedIndexInput.java:55) ~[lucene-core-9.3.0.jar:?]
at org.apache.lucene.store.BufferedChecksumIndexInput.readByte(BufferedChecksumIndexInput.java:39) ~[lucene-core-9.3.0.jar:?]
at org.apache.lucene.codecs.CodecUtil.readBEInt(CodecUtil.java:667) ~[lucene-core-9.3.0.jar:?]
at org.apache.lucene.codecs.CodecUtil.checkHeader(CodecUtil.java:184) ~[lucene-core-9.3.0.jar:?]
at org.apache.lucene.codecs.CodecUtil.checkIndexHeader(CodecUtil.java:253) ~[lucene-core-9.3.0.jar:?]
at org.apache.lucene.codecs.lucene90.Lucene90FieldInfosFormat.read(Lucene90FieldInfosFormat.java:128) ~[lucene-core-9.3.0.jar:?]
at org.apache.lucene.index.SegmentReader.initFieldInfos(SegmentReader.java:205) ~[lucene-core-9.3.0.jar:?]
at org.apache.lucene.index.SegmentReader.<init>(SegmentReader.java:156) ~[lucene-core-9.3.0.jar:?]
at org.apache.lucene.index.ReadersAndUpdates.createNewReaderWithLatestLiveDocs(ReadersAndUpdates.java:738) ~[lucene-core-9.3.0.jar:?]
at org.apache.lucene.index.ReadersAndUpdates.swapNewReaderWithLatestLiveDocs(ReadersAndUpdates.java:754) ~[lucene-core-9.3.0.jar:?]
at org.apache.lucene.index.ReadersAndUpdates.writeFieldUpdates(ReadersAndUpdates.java:678) ~[lucene-core-9.3.0.jar:?]
at org.apache.lucene.index.ReaderPool.writeAllDocValuesUpdates(ReaderPool.java:251) ~[lucene-core-9.3.0.jar:?]
at org.apache.lucene.index.IndexWriter.writeReaderPool(IndexWriter.java:3743) ~[lucene-core-9.3.0.jar:?]
at org.apache.lucene.index.IndexWriter.getReader(IndexWriter.java:591) ~[lucene-core-9.3.0.jar:?]
at org.apache.lucene.index.StandardDirectoryReader.doOpenFromWriter(StandardDirectoryReader.java:381) ~[lucene-core-9.3.0.jar:?]
at org.apache.lucene.index.StandardDirectoryReader.doOpenIfChanged(StandardDirectoryReader.java:355) ~[lucene-core-9.3.0.jar:?]
at org.apache.lucene.index.StandardDirectoryReader.doOpenIfChanged(StandardDirectoryReader.java:345) ~[lucene-core-9.3.0.jar:?]
at org.apache.lucene.index.FilterDirectoryReader.doOpenIfChanged(FilterDirectoryReader.java:112) ~[lucene-core-9.3.0.jar:?]
at org.apache.lucene.index.DirectoryReader.openIfChanged(DirectoryReader.java:170) ~[lucene-core-9.3.0.jar:?]
at org.elasticsearch.index.engine.ElasticsearchReaderManager.refreshIfNeeded(ElasticsearchReaderManager.java:48) ~[elasticsearch-8.4.1.jar:?]
at org.elasticsearch.index.engine.ElasticsearchReaderManager.refreshIfNeeded(ElasticsearchReaderManager.java:27) ~[elasticsearch-8.4.1.jar:?]
at org.apache.lucene.search.ReferenceManager.doMaybeRefresh(ReferenceManager.java:167) ~[lucene-core-9.3.0.jar:?]
at org.apache.lucene.search.ReferenceManager.maybeRefreshBlocking(ReferenceManager.java:240) ~[lucene-core-9.3.0.jar:?]
at org.elasticsearch.index.engine.InternalEngine$ExternalReaderManager.refreshIfNeeded(InternalEngine.java:355) ~[elasticsearch-8.4.1.jar:?]
at org.elasticsearch.index.engine.InternalEngine$ExternalReaderManager.refreshIfNeeded(InternalEngine.java:335) ~[elasticsearch-8.4.1.jar:?]
at org.apache.lucene.search.ReferenceManager.doMaybeRefresh(ReferenceManager.java:167) ~[lucene-core-9.3.0.jar:?]
Many thanks.

Seb
Re: Recurring index corruption [ In reply to ]
Hi,

in general you can still use MMapDirectory. There is no requirement to
set vm.max_map_count for smaller clusters. The information in Elastics
documentation is not mandatory and misleading.

If you use newest version of Elasticsearch with Java 19 and you use
`--enable-preview` in you jvm.properties file, you don't even need to
change that setting even with larger clusters.

Uwe

Am 02.01.2023 um 11:18 schrieb S S:
> We are experimenting with Elastic Search deployed in Azure Container Instances (Debian + OpenJDK). The ES indexes are stored into an Azure file share mounted via SMB (3.0). The Elastic Search cluster is made up of 4 nodes, each one have a separate file share to store the indices.
>
> This configuration has been influenced by some ACIs limitations, specifically:
>
> we cannot set the max_map_count value as we do not have access to the underlying host (https://www.elastic.co/guide/en/elasticsearch/reference/current/vm-max-map-count.html). Unfortunately, this is required to run an ES cluster, therefore we were forced to use NIOF
> ACI’s storage is ephemera, therefore we had to map volumes to persist the indexes. ACIs only allow volume mappings using Azure File Shares, which only works with NFS or SMB.
>
> We are experiencing recurring index corruption, specifically a "read past EOF" exception. I asked on the Elastic Search forum but the answer I got was a bit generic and not really helpful other than confirming that, from ES point of view, ES should work on an SMB share as long as it behaves as a local drive. As the underlying exception relates to an issue with a Lucene index, I was wondering if you could help out? Specifically, can Lucene work on SMB? I can only find sparse information on this configuration and, while NFS seems a no-no, for SMB is not that clear. Below is the exception we are getting.
>
> java.io.IOException: read past EOF: NIOFSIndexInput(path="/bitnami/elasticsearch/data/indices/mS2bUbLtSeG0FSAMuKX7JQ/0/index/_ldsn_1.fnm") buffer: java.nio.HeapByteBuffer[pos=0 lim=1024 cap=1024] chunkLen: 1024 end: 2331: NIOFSIndexInput(path="/bitnami/elasticsearch/data/indices/mS2bUbLtSeG0FSAMuKX7JQ/0/index/_ldsn_1.fnm")
> at org.apache.lucene.store.NIOFSDirectory$NIOFSIndexInput.readInternal(NIOFSDirectory.java:200) ~[lucene-core-9.3.0.jar:?]
> at org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.java:291) ~[lucene-core-9.3.0.jar:?]
> at org.apache.lucene.store.BufferedIndexInput.readByte(BufferedIndexInput.java:55) ~[lucene-core-9.3.0.jar:?]
> at org.apache.lucene.store.BufferedChecksumIndexInput.readByte(BufferedChecksumIndexInput.java:39) ~[lucene-core-9.3.0.jar:?]
> at org.apache.lucene.codecs.CodecUtil.readBEInt(CodecUtil.java:667) ~[lucene-core-9.3.0.jar:?]
> at org.apache.lucene.codecs.CodecUtil.checkHeader(CodecUtil.java:184) ~[lucene-core-9.3.0.jar:?]
> at org.apache.lucene.codecs.CodecUtil.checkIndexHeader(CodecUtil.java:253) ~[lucene-core-9.3.0.jar:?]
> at org.apache.lucene.codecs.lucene90.Lucene90FieldInfosFormat.read(Lucene90FieldInfosFormat.java:128) ~[lucene-core-9.3.0.jar:?]
> at org.apache.lucene.index.SegmentReader.initFieldInfos(SegmentReader.java:205) ~[lucene-core-9.3.0.jar:?]
> at org.apache.lucene.index.SegmentReader.<init>(SegmentReader.java:156) ~[lucene-core-9.3.0.jar:?]
> at org.apache.lucene.index.ReadersAndUpdates.createNewReaderWithLatestLiveDocs(ReadersAndUpdates.java:738) ~[lucene-core-9.3.0.jar:?]
> at org.apache.lucene.index.ReadersAndUpdates.swapNewReaderWithLatestLiveDocs(ReadersAndUpdates.java:754) ~[lucene-core-9.3.0.jar:?]
> at org.apache.lucene.index.ReadersAndUpdates.writeFieldUpdates(ReadersAndUpdates.java:678) ~[lucene-core-9.3.0.jar:?]
> at org.apache.lucene.index.ReaderPool.writeAllDocValuesUpdates(ReaderPool.java:251) ~[lucene-core-9.3.0.jar:?]
> at org.apache.lucene.index.IndexWriter.writeReaderPool(IndexWriter.java:3743) ~[lucene-core-9.3.0.jar:?]
> at org.apache.lucene.index.IndexWriter.getReader(IndexWriter.java:591) ~[lucene-core-9.3.0.jar:?]
> at org.apache.lucene.index.StandardDirectoryReader.doOpenFromWriter(StandardDirectoryReader.java:381) ~[lucene-core-9.3.0.jar:?]
> at org.apache.lucene.index.StandardDirectoryReader.doOpenIfChanged(StandardDirectoryReader.java:355) ~[lucene-core-9.3.0.jar:?]
> at org.apache.lucene.index.StandardDirectoryReader.doOpenIfChanged(StandardDirectoryReader.java:345) ~[lucene-core-9.3.0.jar:?]
> at org.apache.lucene.index.FilterDirectoryReader.doOpenIfChanged(FilterDirectoryReader.java:112) ~[lucene-core-9.3.0.jar:?]
> at org.apache.lucene.index.DirectoryReader.openIfChanged(DirectoryReader.java:170) ~[lucene-core-9.3.0.jar:?]
> at org.elasticsearch.index.engine.ElasticsearchReaderManager.refreshIfNeeded(ElasticsearchReaderManager.java:48) ~[elasticsearch-8.4.1.jar:?]
> at org.elasticsearch.index.engine.ElasticsearchReaderManager.refreshIfNeeded(ElasticsearchReaderManager.java:27) ~[elasticsearch-8.4.1.jar:?]
> at org.apache.lucene.search.ReferenceManager.doMaybeRefresh(ReferenceManager.java:167) ~[lucene-core-9.3.0.jar:?]
> at org.apache.lucene.search.ReferenceManager.maybeRefreshBlocking(ReferenceManager.java:240) ~[lucene-core-9.3.0.jar:?]
> at org.elasticsearch.index.engine.InternalEngine$ExternalReaderManager.refreshIfNeeded(InternalEngine.java:355) ~[elasticsearch-8.4.1.jar:?]
> at org.elasticsearch.index.engine.InternalEngine$ExternalReaderManager.refreshIfNeeded(InternalEngine.java:335) ~[elasticsearch-8.4.1.jar:?]
> at org.apache.lucene.search.ReferenceManager.doMaybeRefresh(ReferenceManager.java:167) ~[lucene-core-9.3.0.jar:?]
> Many thanks.
>
> Seb

--
Uwe Schindler
Achterdiek 19, D-28357 Bremen
https://www.thetaphi.de
eMail: uwe@thetaphi.de


---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Recurring index corruption [ In reply to ]
Hi Uwe,

This sounds very interesting, I will try this configuration this afternoon and I’ll let you know what happens.

Many thanks for the suggestion! :)

Seb



> On 2 Jan 2023, at 11:48, Uwe Schindler <uwe@thetaphi.de> wrote:
>
> Hi,
>
> in general you can still use MMapDirectory. There is no requirement to set vm.max_map_count for smaller clusters. The information in Elastics documentation is not mandatory and misleading.
>
> If you use newest version of Elasticsearch with Java 19 and you use `--enable-preview` in you jvm.properties file, you don't even need to change that setting even with larger clusters.
>
> Uwe
>
> Am 02.01.2023 um 11:18 schrieb S S:
>> We are experimenting with Elastic Search deployed in Azure Container Instances (Debian + OpenJDK). The ES indexes are stored into an Azure file share mounted via SMB (3.0). The Elastic Search cluster is made up of 4 nodes, each one have a separate file share to store the indices.
>>
>> This configuration has been influenced by some ACIs limitations, specifically:
>>
>> we cannot set the max_map_count value as we do not have access to the underlying host (https://www.elastic.co/guide/en/elasticsearch/reference/current/vm-max-map-count.html). Unfortunately, this is required to run an ES cluster, therefore we were forced to use NIOF
>> ACI’s storage is ephemera, therefore we had to map volumes to persist the indexes. ACIs only allow volume mappings using Azure File Shares, which only works with NFS or SMB.
>>
>> We are experiencing recurring index corruption, specifically a "read past EOF" exception. I asked on the Elastic Search forum but the answer I got was a bit generic and not really helpful other than confirming that, from ES point of view, ES should work on an SMB share as long as it behaves as a local drive. As the underlying exception relates to an issue with a Lucene index, I was wondering if you could help out? Specifically, can Lucene work on SMB? I can only find sparse information on this configuration and, while NFS seems a no-no, for SMB is not that clear. Below is the exception we are getting.
>>
>> java.io.IOException: read past EOF: NIOFSIndexInput(path="/bitnami/elasticsearch/data/indices/mS2bUbLtSeG0FSAMuKX7JQ/0/index/_ldsn_1.fnm") buffer: java.nio.HeapByteBuffer[pos=0 lim=1024 cap=1024] chunkLen: 1024 end: 2331: NIOFSIndexInput(path="/bitnami/elasticsearch/data/indices/mS2bUbLtSeG0FSAMuKX7JQ/0/index/_ldsn_1.fnm")
>> at org.apache.lucene.store.NIOFSDirectory$NIOFSIndexInput.readInternal(NIOFSDirectory.java:200) ~[lucene-core-9.3.0.jar:?]
>> at org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.java:291) ~[lucene-core-9.3.0.jar:?]
>> at org.apache.lucene.store.BufferedIndexInput.readByte(BufferedIndexInput.java:55) ~[lucene-core-9.3.0.jar:?]
>> at org.apache.lucene.store.BufferedChecksumIndexInput.readByte(BufferedChecksumIndexInput.java:39) ~[lucene-core-9.3.0.jar:?]
>> at org.apache.lucene.codecs.CodecUtil.readBEInt(CodecUtil.java:667) ~[lucene-core-9.3.0.jar:?]
>> at org.apache.lucene.codecs.CodecUtil.checkHeader(CodecUtil.java:184) ~[lucene-core-9.3.0.jar:?]
>> at org.apache.lucene.codecs.CodecUtil.checkIndexHeader(CodecUtil.java:253) ~[lucene-core-9.3.0.jar:?]
>> at org.apache.lucene.codecs.lucene90.Lucene90FieldInfosFormat.read(Lucene90FieldInfosFormat.java:128) ~[lucene-core-9.3.0.jar:?]
>> at org.apache.lucene.index.SegmentReader.initFieldInfos(SegmentReader.java:205) ~[lucene-core-9.3.0.jar:?]
>> at org.apache.lucene.index.SegmentReader.<init>(SegmentReader.java:156) ~[lucene-core-9.3.0.jar:?]
>> at org.apache.lucene.index.ReadersAndUpdates.createNewReaderWithLatestLiveDocs(ReadersAndUpdates.java:738) ~[lucene-core-9.3.0.jar:?]
>> at org.apache.lucene.index.ReadersAndUpdates.swapNewReaderWithLatestLiveDocs(ReadersAndUpdates.java:754) ~[lucene-core-9.3.0.jar:?]
>> at org.apache.lucene.index.ReadersAndUpdates.writeFieldUpdates(ReadersAndUpdates.java:678) ~[lucene-core-9.3.0.jar:?]
>> at org.apache.lucene.index.ReaderPool.writeAllDocValuesUpdates(ReaderPool.java:251) ~[lucene-core-9.3.0.jar:?]
>> at org.apache.lucene.index.IndexWriter.writeReaderPool(IndexWriter.java:3743) ~[lucene-core-9.3.0.jar:?]
>> at org.apache.lucene.index.IndexWriter.getReader(IndexWriter.java:591) ~[lucene-core-9.3.0.jar:?]
>> at org.apache.lucene.index.StandardDirectoryReader.doOpenFromWriter(StandardDirectoryReader.java:381) ~[lucene-core-9.3.0.jar:?]
>> at org.apache.lucene.index.StandardDirectoryReader.doOpenIfChanged(StandardDirectoryReader.java:355) ~[lucene-core-9.3.0.jar:?]
>> at org.apache.lucene.index.StandardDirectoryReader.doOpenIfChanged(StandardDirectoryReader.java:345) ~[lucene-core-9.3.0.jar:?]
>> at org.apache.lucene.index.FilterDirectoryReader.doOpenIfChanged(FilterDirectoryReader.java:112) ~[lucene-core-9.3.0.jar:?]
>> at org.apache.lucene.index.DirectoryReader.openIfChanged(DirectoryReader.java:170) ~[lucene-core-9.3.0.jar:?]
>> at org.elasticsearch.index.engine.ElasticsearchReaderManager.refreshIfNeeded(ElasticsearchReaderManager.java:48) ~[elasticsearch-8.4.1.jar:?]
>> at org.elasticsearch.index.engine.ElasticsearchReaderManager.refreshIfNeeded(ElasticsearchReaderManager.java:27) ~[elasticsearch-8.4.1.jar:?]
>> at org.apache.lucene.search.ReferenceManager.doMaybeRefresh(ReferenceManager.java:167) ~[lucene-core-9.3.0.jar:?]
>> at org.apache.lucene.search.ReferenceManager.maybeRefreshBlocking(ReferenceManager.java:240) ~[lucene-core-9.3.0.jar:?]
>> at org.elasticsearch.index.engine.InternalEngine$ExternalReaderManager.refreshIfNeeded(InternalEngine.java:355) ~[elasticsearch-8.4.1.jar:?]
>> at org.elasticsearch.index.engine.InternalEngine$ExternalReaderManager.refreshIfNeeded(InternalEngine.java:335) ~[elasticsearch-8.4.1.jar:?]
>> at org.apache.lucene.search.ReferenceManager.doMaybeRefresh(ReferenceManager.java:167) ~[lucene-core-9.3.0.jar:?]
>> Many thanks.
>>
>> Seb
>
> --
> Uwe Schindler
> Achterdiek 19, D-28357 Bremen
> https://www.thetaphi.de
> eMail: uwe@thetaphi.de
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>


---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Recurring index corruption [ In reply to ]
Your files are getting truncated. Nothing lucene can do.

If this is really the only way you can store data in this azure cloud,
and this is how they treat it, then run away... don't just walk... to
a different cloud.

On Mon, Jan 2, 2023 at 5:19 AM S S <sebastiano1972@yahoo.it.invalid> wrote:
>
> We are experimenting with Elastic Search deployed in Azure Container Instances (Debian + OpenJDK). The ES indexes are stored into an Azure file share mounted via SMB (3.0). The Elastic Search cluster is made up of 4 nodes, each one have a separate file share to store the indices.
>
> This configuration has been influenced by some ACIs limitations, specifically:
>
> we cannot set the max_map_count value as we do not have access to the underlying host (https://www.elastic.co/guide/en/elasticsearch/reference/current/vm-max-map-count.html). Unfortunately, this is required to run an ES cluster, therefore we were forced to use NIOF
> ACI’s storage is ephemera, therefore we had to map volumes to persist the indexes. ACIs only allow volume mappings using Azure File Shares, which only works with NFS or SMB.
>
> We are experiencing recurring index corruption, specifically a "read past EOF" exception. I asked on the Elastic Search forum but the answer I got was a bit generic and not really helpful other than confirming that, from ES point of view, ES should work on an SMB share as long as it behaves as a local drive. As the underlying exception relates to an issue with a Lucene index, I was wondering if you could help out? Specifically, can Lucene work on SMB? I can only find sparse information on this configuration and, while NFS seems a no-no, for SMB is not that clear. Below is the exception we are getting.
>
> java.io.IOException: read past EOF: NIOFSIndexInput(path="/bitnami/elasticsearch/data/indices/mS2bUbLtSeG0FSAMuKX7JQ/0/index/_ldsn_1.fnm") buffer: java.nio.HeapByteBuffer[pos=0 lim=1024 cap=1024] chunkLen: 1024 end: 2331: NIOFSIndexInput(path="/bitnami/elasticsearch/data/indices/mS2bUbLtSeG0FSAMuKX7JQ/0/index/_ldsn_1.fnm")
> at org.apache.lucene.store.NIOFSDirectory$NIOFSIndexInput.readInternal(NIOFSDirectory.java:200) ~[lucene-core-9.3.0.jar:?]
> at org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.java:291) ~[lucene-core-9.3.0.jar:?]
> at org.apache.lucene.store.BufferedIndexInput.readByte(BufferedIndexInput.java:55) ~[lucene-core-9.3.0.jar:?]
> at org.apache.lucene.store.BufferedChecksumIndexInput.readByte(BufferedChecksumIndexInput.java:39) ~[lucene-core-9.3.0.jar:?]
> at org.apache.lucene.codecs.CodecUtil.readBEInt(CodecUtil.java:667) ~[lucene-core-9.3.0.jar:?]
> at org.apache.lucene.codecs.CodecUtil.checkHeader(CodecUtil.java:184) ~[lucene-core-9.3.0.jar:?]
> at org.apache.lucene.codecs.CodecUtil.checkIndexHeader(CodecUtil.java:253) ~[lucene-core-9.3.0.jar:?]
> at org.apache.lucene.codecs.lucene90.Lucene90FieldInfosFormat.read(Lucene90FieldInfosFormat.java:128) ~[lucene-core-9.3.0.jar:?]
> at org.apache.lucene.index.SegmentReader.initFieldInfos(SegmentReader.java:205) ~[lucene-core-9.3.0.jar:?]
> at org.apache.lucene.index.SegmentReader.<init>(SegmentReader.java:156) ~[lucene-core-9.3.0.jar:?]
> at org.apache.lucene.index.ReadersAndUpdates.createNewReaderWithLatestLiveDocs(ReadersAndUpdates.java:738) ~[lucene-core-9.3.0.jar:?]
> at org.apache.lucene.index.ReadersAndUpdates.swapNewReaderWithLatestLiveDocs(ReadersAndUpdates.java:754) ~[lucene-core-9.3.0.jar:?]
> at org.apache.lucene.index.ReadersAndUpdates.writeFieldUpdates(ReadersAndUpdates.java:678) ~[lucene-core-9.3.0.jar:?]
> at org.apache.lucene.index.ReaderPool.writeAllDocValuesUpdates(ReaderPool.java:251) ~[lucene-core-9.3.0.jar:?]
> at org.apache.lucene.index.IndexWriter.writeReaderPool(IndexWriter.java:3743) ~[lucene-core-9.3.0.jar:?]
> at org.apache.lucene.index.IndexWriter.getReader(IndexWriter.java:591) ~[lucene-core-9.3.0.jar:?]
> at org.apache.lucene.index.StandardDirectoryReader.doOpenFromWriter(StandardDirectoryReader.java:381) ~[lucene-core-9.3.0.jar:?]
> at org.apache.lucene.index.StandardDirectoryReader.doOpenIfChanged(StandardDirectoryReader.java:355) ~[lucene-core-9.3.0.jar:?]
> at org.apache.lucene.index.StandardDirectoryReader.doOpenIfChanged(StandardDirectoryReader.java:345) ~[lucene-core-9.3.0.jar:?]
> at org.apache.lucene.index.FilterDirectoryReader.doOpenIfChanged(FilterDirectoryReader.java:112) ~[lucene-core-9.3.0.jar:?]
> at org.apache.lucene.index.DirectoryReader.openIfChanged(DirectoryReader.java:170) ~[lucene-core-9.3.0.jar:?]
> at org.elasticsearch.index.engine.ElasticsearchReaderManager.refreshIfNeeded(ElasticsearchReaderManager.java:48) ~[elasticsearch-8.4.1.jar:?]
> at org.elasticsearch.index.engine.ElasticsearchReaderManager.refreshIfNeeded(ElasticsearchReaderManager.java:27) ~[elasticsearch-8.4.1.jar:?]
> at org.apache.lucene.search.ReferenceManager.doMaybeRefresh(ReferenceManager.java:167) ~[lucene-core-9.3.0.jar:?]
> at org.apache.lucene.search.ReferenceManager.maybeRefreshBlocking(ReferenceManager.java:240) ~[lucene-core-9.3.0.jar:?]
> at org.elasticsearch.index.engine.InternalEngine$ExternalReaderManager.refreshIfNeeded(InternalEngine.java:355) ~[elasticsearch-8.4.1.jar:?]
> at org.elasticsearch.index.engine.InternalEngine$ExternalReaderManager.refreshIfNeeded(InternalEngine.java:335) ~[elasticsearch-8.4.1.jar:?]
> at org.apache.lucene.search.ReferenceManager.doMaybeRefresh(ReferenceManager.java:167) ~[lucene-core-9.3.0.jar:?]
> Many thanks.
>
> Seb

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Recurring index corruption [ In reply to ]
Hi Uwe,

Sorry for the late reply but upgrading the docker image to use OpenJDK was easier said that done.

I am not a Java developer/expert so, sorry for the stupid question but, how do I specify the --enable-preview flag? ES has got a quite complex way to start so I cannot specify the flag on the command line. You suggested to use a jvm.properties file but I cannot find anything useful about it when google-ing. Where should it be placed? And what should I write in it?

I can see ES recognising OpenJDK 19 while bootstrapping and suggesting to enable preview, but it does not suggest how, and I cannot find anything on the ES website.

Many thanks.

Seb

> On 2 Jan 2023, at 11:48, Uwe Schindler <uwe@thetaphi.de> wrote:
>
> Hi,
>
> in general you can still use MMapDirectory. There is no requirement to set vm.max_map_count for smaller clusters. The information in Elastics documentation is not mandatory and misleading.
>
> If you use newest version of Elasticsearch with Java 19 and you use `--enable-preview` in you jvm.properties file, you don't even need to change that setting even with larger clusters.
>
> Uwe
>
> Am 02.01.2023 um 11:18 schrieb S S:
>> We are experimenting with Elastic Search deployed in Azure Container Instances (Debian + OpenJDK). The ES indexes are stored into an Azure file share mounted via SMB (3.0). The Elastic Search cluster is made up of 4 nodes, each one have a separate file share to store the indices.
>>
>> This configuration has been influenced by some ACIs limitations, specifically:
>>
>> we cannot set the max_map_count value as we do not have access to the underlying host (https://www.elastic.co/guide/en/elasticsearch/reference/current/vm-max-map-count.html). Unfortunately, this is required to run an ES cluster, therefore we were forced to use NIOF
>> ACI’s storage is ephemera, therefore we had to map volumes to persist the indexes. ACIs only allow volume mappings using Azure File Shares, which only works with NFS or SMB.
>>
>> We are experiencing recurring index corruption, specifically a "read past EOF" exception. I asked on the Elastic Search forum but the answer I got was a bit generic and not really helpful other than confirming that, from ES point of view, ES should work on an SMB share as long as it behaves as a local drive. As the underlying exception relates to an issue with a Lucene index, I was wondering if you could help out? Specifically, can Lucene work on SMB? I can only find sparse information on this configuration and, while NFS seems a no-no, for SMB is not that clear. Below is the exception we are getting.
>>
>> java.io.IOException: read past EOF: NIOFSIndexInput(path="/bitnami/elasticsearch/data/indices/mS2bUbLtSeG0FSAMuKX7JQ/0/index/_ldsn_1.fnm") buffer: java.nio.HeapByteBuffer[pos=0 lim=1024 cap=1024] chunkLen: 1024 end: 2331: NIOFSIndexInput(path="/bitnami/elasticsearch/data/indices/mS2bUbLtSeG0FSAMuKX7JQ/0/index/_ldsn_1.fnm")
>> at org.apache.lucene.store.NIOFSDirectory$NIOFSIndexInput.readInternal(NIOFSDirectory.java:200) ~[lucene-core-9.3.0.jar:?]
>> at org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.java:291) ~[lucene-core-9.3.0.jar:?]
>> at org.apache.lucene.store.BufferedIndexInput.readByte(BufferedIndexInput.java:55) ~[lucene-core-9.3.0.jar:?]
>> at org.apache.lucene.store.BufferedChecksumIndexInput.readByte(BufferedChecksumIndexInput.java:39) ~[lucene-core-9.3.0.jar:?]
>> at org.apache.lucene.codecs.CodecUtil.readBEInt(CodecUtil.java:667) ~[lucene-core-9.3.0.jar:?]
>> at org.apache.lucene.codecs.CodecUtil.checkHeader(CodecUtil.java:184) ~[lucene-core-9.3.0.jar:?]
>> at org.apache.lucene.codecs.CodecUtil.checkIndexHeader(CodecUtil.java:253) ~[lucene-core-9.3.0.jar:?]
>> at org.apache.lucene.codecs.lucene90.Lucene90FieldInfosFormat.read(Lucene90FieldInfosFormat.java:128) ~[lucene-core-9.3.0.jar:?]
>> at org.apache.lucene.index.SegmentReader.initFieldInfos(SegmentReader.java:205) ~[lucene-core-9.3.0.jar:?]
>> at org.apache.lucene.index.SegmentReader.<init>(SegmentReader.java:156) ~[lucene-core-9.3.0.jar:?]
>> at org.apache.lucene.index.ReadersAndUpdates.createNewReaderWithLatestLiveDocs(ReadersAndUpdates.java:738) ~[lucene-core-9.3.0.jar:?]
>> at org.apache.lucene.index.ReadersAndUpdates.swapNewReaderWithLatestLiveDocs(ReadersAndUpdates.java:754) ~[lucene-core-9.3.0.jar:?]
>> at org.apache.lucene.index.ReadersAndUpdates.writeFieldUpdates(ReadersAndUpdates.java:678) ~[lucene-core-9.3.0.jar:?]
>> at org.apache.lucene.index.ReaderPool.writeAllDocValuesUpdates(ReaderPool.java:251) ~[lucene-core-9.3.0.jar:?]
>> at org.apache.lucene.index.IndexWriter.writeReaderPool(IndexWriter.java:3743) ~[lucene-core-9.3.0.jar:?]
>> at org.apache.lucene.index.IndexWriter.getReader(IndexWriter.java:591) ~[lucene-core-9.3.0.jar:?]
>> at org.apache.lucene.index.StandardDirectoryReader.doOpenFromWriter(StandardDirectoryReader.java:381) ~[lucene-core-9.3.0.jar:?]
>> at org.apache.lucene.index.StandardDirectoryReader.doOpenIfChanged(StandardDirectoryReader.java:355) ~[lucene-core-9.3.0.jar:?]
>> at org.apache.lucene.index.StandardDirectoryReader.doOpenIfChanged(StandardDirectoryReader.java:345) ~[lucene-core-9.3.0.jar:?]
>> at org.apache.lucene.index.FilterDirectoryReader.doOpenIfChanged(FilterDirectoryReader.java:112) ~[lucene-core-9.3.0.jar:?]
>> at org.apache.lucene.index.DirectoryReader.openIfChanged(DirectoryReader.java:170) ~[lucene-core-9.3.0.jar:?]
>> at org.elasticsearch.index.engine.ElasticsearchReaderManager.refreshIfNeeded(ElasticsearchReaderManager.java:48) ~[elasticsearch-8.4.1.jar:?]
>> at org.elasticsearch.index.engine.ElasticsearchReaderManager.refreshIfNeeded(ElasticsearchReaderManager.java:27) ~[elasticsearch-8.4.1.jar:?]
>> at org.apache.lucene.search.ReferenceManager.doMaybeRefresh(ReferenceManager.java:167) ~[lucene-core-9.3.0.jar:?]
>> at org.apache.lucene.search.ReferenceManager.maybeRefreshBlocking(ReferenceManager.java:240) ~[lucene-core-9.3.0.jar:?]
>> at org.elasticsearch.index.engine.InternalEngine$ExternalReaderManager.refreshIfNeeded(InternalEngine.java:355) ~[elasticsearch-8.4.1.jar:?]
>> at org.elasticsearch.index.engine.InternalEngine$ExternalReaderManager.refreshIfNeeded(InternalEngine.java:335) ~[elasticsearch-8.4.1.jar:?]
>> at org.apache.lucene.search.ReferenceManager.doMaybeRefresh(ReferenceManager.java:167) ~[lucene-core-9.3.0.jar:?]
>> Many thanks.
>>
>> Seb
>
> --
> Uwe Schindler
> Achterdiek 19, D-28357 Bremen
> https://www.thetaphi.de
> eMail: uwe@thetaphi.de
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>


---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Recurring index corruption [ In reply to ]
Hi,

in recent versions it works like that:

https://www.elastic.co/guide/en/elasticsearch/reference/current/advanced-configuration.html#set-jvm-options

So in folder jvm.options.d/ add a new file (like "preview.conf") and put
"19:--enable-preview" into it. It is basically the same like modifying
heap size.

But in general, you can simply use MMapDirectory, the max-map-count
setting is only relevant in *huge* (huge means hundreds of huge indexes
per node). In that case Java 19's preview features would be recommended.

Uwe

Am 02.01.2023 um 17:41 schrieb S S:
> Hi Uwe,
>
> Sorry for the late reply but upgrading the docker image to use OpenJDK was easier said that done.
>
> I am not a Java developer/expert so, sorry for the stupid question but, how do I specify the --enable-preview flag? ES has got a quite complex way to start so I cannot specify the flag on the command line. You suggested to use a jvm.properties file but I cannot find anything useful about it when google-ing. Where should it be placed? And what should I write in it?
>
> I can see ES recognising OpenJDK 19 while bootstrapping and suggesting to enable preview, but it does not suggest how, and I cannot find anything on the ES website.
>
> Many thanks.
>
> Seb
>
>> On 2 Jan 2023, at 11:48, Uwe Schindler <uwe@thetaphi.de> wrote:
>>
>> Hi,
>>
>> in general you can still use MMapDirectory. There is no requirement to set vm.max_map_count for smaller clusters. The information in Elastics documentation is not mandatory and misleading.
>>
>> If you use newest version of Elasticsearch with Java 19 and you use `--enable-preview` in you jvm.properties file, you don't even need to change that setting even with larger clusters.
>>
>> Uwe
>>
>> Am 02.01.2023 um 11:18 schrieb S S:
>>> We are experimenting with Elastic Search deployed in Azure Container Instances (Debian + OpenJDK). The ES indexes are stored into an Azure file share mounted via SMB (3.0). The Elastic Search cluster is made up of 4 nodes, each one have a separate file share to store the indices.
>>>
>>> This configuration has been influenced by some ACIs limitations, specifically:
>>>
>>> we cannot set the max_map_count value as we do not have access to the underlying host (https://www.elastic.co/guide/en/elasticsearch/reference/current/vm-max-map-count.html). Unfortunately, this is required to run an ES cluster, therefore we were forced to use NIOF
>>> ACI’s storage is ephemera, therefore we had to map volumes to persist the indexes. ACIs only allow volume mappings using Azure File Shares, which only works with NFS or SMB.
>>>
>>> We are experiencing recurring index corruption, specifically a "read past EOF" exception. I asked on the Elastic Search forum but the answer I got was a bit generic and not really helpful other than confirming that, from ES point of view, ES should work on an SMB share as long as it behaves as a local drive. As the underlying exception relates to an issue with a Lucene index, I was wondering if you could help out? Specifically, can Lucene work on SMB? I can only find sparse information on this configuration and, while NFS seems a no-no, for SMB is not that clear. Below is the exception we are getting.
>>>
>>> java.io.IOException: read past EOF: NIOFSIndexInput(path="/bitnami/elasticsearch/data/indices/mS2bUbLtSeG0FSAMuKX7JQ/0/index/_ldsn_1.fnm") buffer: java.nio.HeapByteBuffer[pos=0 lim=1024 cap=1024] chunkLen: 1024 end: 2331: NIOFSIndexInput(path="/bitnami/elasticsearch/data/indices/mS2bUbLtSeG0FSAMuKX7JQ/0/index/_ldsn_1.fnm")
>>> at org.apache.lucene.store.NIOFSDirectory$NIOFSIndexInput.readInternal(NIOFSDirectory.java:200) ~[lucene-core-9.3.0.jar:?]
>>> at org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.java:291) ~[lucene-core-9.3.0.jar:?]
>>> at org.apache.lucene.store.BufferedIndexInput.readByte(BufferedIndexInput.java:55) ~[lucene-core-9.3.0.jar:?]
>>> at org.apache.lucene.store.BufferedChecksumIndexInput.readByte(BufferedChecksumIndexInput.java:39) ~[lucene-core-9.3.0.jar:?]
>>> at org.apache.lucene.codecs.CodecUtil.readBEInt(CodecUtil.java:667) ~[lucene-core-9.3.0.jar:?]
>>> at org.apache.lucene.codecs.CodecUtil.checkHeader(CodecUtil.java:184) ~[lucene-core-9.3.0.jar:?]
>>> at org.apache.lucene.codecs.CodecUtil.checkIndexHeader(CodecUtil.java:253) ~[lucene-core-9.3.0.jar:?]
>>> at org.apache.lucene.codecs.lucene90.Lucene90FieldInfosFormat.read(Lucene90FieldInfosFormat.java:128) ~[lucene-core-9.3.0.jar:?]
>>> at org.apache.lucene.index.SegmentReader.initFieldInfos(SegmentReader.java:205) ~[lucene-core-9.3.0.jar:?]
>>> at org.apache.lucene.index.SegmentReader.<init>(SegmentReader.java:156) ~[lucene-core-9.3.0.jar:?]
>>> at org.apache.lucene.index.ReadersAndUpdates.createNewReaderWithLatestLiveDocs(ReadersAndUpdates.java:738) ~[lucene-core-9.3.0.jar:?]
>>> at org.apache.lucene.index.ReadersAndUpdates.swapNewReaderWithLatestLiveDocs(ReadersAndUpdates.java:754) ~[lucene-core-9.3.0.jar:?]
>>> at org.apache.lucene.index.ReadersAndUpdates.writeFieldUpdates(ReadersAndUpdates.java:678) ~[lucene-core-9.3.0.jar:?]
>>> at org.apache.lucene.index.ReaderPool.writeAllDocValuesUpdates(ReaderPool.java:251) ~[lucene-core-9.3.0.jar:?]
>>> at org.apache.lucene.index.IndexWriter.writeReaderPool(IndexWriter.java:3743) ~[lucene-core-9.3.0.jar:?]
>>> at org.apache.lucene.index.IndexWriter.getReader(IndexWriter.java:591) ~[lucene-core-9.3.0.jar:?]
>>> at org.apache.lucene.index.StandardDirectoryReader.doOpenFromWriter(StandardDirectoryReader.java:381) ~[lucene-core-9.3.0.jar:?]
>>> at org.apache.lucene.index.StandardDirectoryReader.doOpenIfChanged(StandardDirectoryReader.java:355) ~[lucene-core-9.3.0.jar:?]
>>> at org.apache.lucene.index.StandardDirectoryReader.doOpenIfChanged(StandardDirectoryReader.java:345) ~[lucene-core-9.3.0.jar:?]
>>> at org.apache.lucene.index.FilterDirectoryReader.doOpenIfChanged(FilterDirectoryReader.java:112) ~[lucene-core-9.3.0.jar:?]
>>> at org.apache.lucene.index.DirectoryReader.openIfChanged(DirectoryReader.java:170) ~[lucene-core-9.3.0.jar:?]
>>> at org.elasticsearch.index.engine.ElasticsearchReaderManager.refreshIfNeeded(ElasticsearchReaderManager.java:48) ~[elasticsearch-8.4.1.jar:?]
>>> at org.elasticsearch.index.engine.ElasticsearchReaderManager.refreshIfNeeded(ElasticsearchReaderManager.java:27) ~[elasticsearch-8.4.1.jar:?]
>>> at org.apache.lucene.search.ReferenceManager.doMaybeRefresh(ReferenceManager.java:167) ~[lucene-core-9.3.0.jar:?]
>>> at org.apache.lucene.search.ReferenceManager.maybeRefreshBlocking(ReferenceManager.java:240) ~[lucene-core-9.3.0.jar:?]
>>> at org.elasticsearch.index.engine.InternalEngine$ExternalReaderManager.refreshIfNeeded(InternalEngine.java:355) ~[elasticsearch-8.4.1.jar:?]
>>> at org.elasticsearch.index.engine.InternalEngine$ExternalReaderManager.refreshIfNeeded(InternalEngine.java:335) ~[elasticsearch-8.4.1.jar:?]
>>> at org.apache.lucene.search.ReferenceManager.doMaybeRefresh(ReferenceManager.java:167) ~[lucene-core-9.3.0.jar:?]
>>> Many thanks.
>>>
>>> Seb
>> --
>> Uwe Schindler
>> Achterdiek 19, D-28357 Bremen
>> https://www.thetaphi.de
>> eMail: uwe@thetaphi.de
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
--
Uwe Schindler
Achterdiek 19, D-28357 Bremen
https://www.thetaphi.de
eMail: uwe@thetaphi.de


---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Recurring index corruption [ In reply to ]
Thank you Uwe, this is great! I am rebuilding the cluster using MMapDirectory and no enable-preview, as you suggested. Let’s see what happens.

Cheers,

Seb

> On 2 Jan 2023, at 17:51, Uwe Schindler <uwe@thetaphi.de> wrote:
>
> Hi,
>
> in recent versions it works like that:
>
> https://www.elastic.co/guide/en/elasticsearch/reference/current/advanced-configuration.html#set-jvm-options
>
> So in folder jvm.options.d/ add a new file (like "preview.conf") and put "19:--enable-preview" into it. It is basically the same like modifying heap size.
>
> But in general, you can simply use MMapDirectory, the max-map-count setting is only relevant in *huge* (huge means hundreds of huge indexes per node). In that case Java 19's preview features would be recommended.
>
> Uwe
>
> Am 02.01.2023 um 17:41 schrieb S S:
>> Hi Uwe,
>>
>> Sorry for the late reply but upgrading the docker image to use OpenJDK was easier said that done.
>>
>> I am not a Java developer/expert so, sorry for the stupid question but, how do I specify the --enable-preview flag? ES has got a quite complex way to start so I cannot specify the flag on the command line. You suggested to use a jvm.properties file but I cannot find anything useful about it when google-ing. Where should it be placed? And what should I write in it?
>>
>> I can see ES recognising OpenJDK 19 while bootstrapping and suggesting to enable preview, but it does not suggest how, and I cannot find anything on the ES website.
>>
>> Many thanks.
>>
>> Seb
>>
>>> On 2 Jan 2023, at 11:48, Uwe Schindler <uwe@thetaphi.de> wrote:
>>>
>>> Hi,
>>>
>>> in general you can still use MMapDirectory. There is no requirement to set vm.max_map_count for smaller clusters. The information in Elastics documentation is not mandatory and misleading.
>>>
>>> If you use newest version of Elasticsearch with Java 19 and you use `--enable-preview` in you jvm.properties file, you don't even need to change that setting even with larger clusters.
>>>
>>> Uwe
>>>
>>> Am 02.01.2023 um 11:18 schrieb S S:
>>>> We are experimenting with Elastic Search deployed in Azure Container Instances (Debian + OpenJDK). The ES indexes are stored into an Azure file share mounted via SMB (3.0). The Elastic Search cluster is made up of 4 nodes, each one have a separate file share to store the indices.
>>>>
>>>> This configuration has been influenced by some ACIs limitations, specifically:
>>>>
>>>> we cannot set the max_map_count value as we do not have access to the underlying host (https://www.elastic.co/guide/en/elasticsearch/reference/current/vm-max-map-count.html). Unfortunately, this is required to run an ES cluster, therefore we were forced to use NIOF
>>>> ACI’s storage is ephemera, therefore we had to map volumes to persist the indexes. ACIs only allow volume mappings using Azure File Shares, which only works with NFS or SMB.
>>>>
>>>> We are experiencing recurring index corruption, specifically a "read past EOF" exception. I asked on the Elastic Search forum but the answer I got was a bit generic and not really helpful other than confirming that, from ES point of view, ES should work on an SMB share as long as it behaves as a local drive. As the underlying exception relates to an issue with a Lucene index, I was wondering if you could help out? Specifically, can Lucene work on SMB? I can only find sparse information on this configuration and, while NFS seems a no-no, for SMB is not that clear. Below is the exception we are getting.
>>>>
>>>> java.io.IOException: read past EOF: NIOFSIndexInput(path="/bitnami/elasticsearch/data/indices/mS2bUbLtSeG0FSAMuKX7JQ/0/index/_ldsn_1.fnm") buffer: java.nio.HeapByteBuffer[pos=0 lim=1024 cap=1024] chunkLen: 1024 end: 2331: NIOFSIndexInput(path="/bitnami/elasticsearch/data/indices/mS2bUbLtSeG0FSAMuKX7JQ/0/index/_ldsn_1.fnm")
>>>> at org.apache.lucene.store.NIOFSDirectory$NIOFSIndexInput.readInternal(NIOFSDirectory.java:200) ~[lucene-core-9.3.0.jar:?]
>>>> at org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.java:291) ~[lucene-core-9.3.0.jar:?]
>>>> at org.apache.lucene.store.BufferedIndexInput.readByte(BufferedIndexInput.java:55) ~[lucene-core-9.3.0.jar:?]
>>>> at org.apache.lucene.store.BufferedChecksumIndexInput.readByte(BufferedChecksumIndexInput.java:39) ~[lucene-core-9.3.0.jar:?]
>>>> at org.apache.lucene.codecs.CodecUtil.readBEInt(CodecUtil.java:667) ~[lucene-core-9.3.0.jar:?]
>>>> at org.apache.lucene.codecs.CodecUtil.checkHeader(CodecUtil.java:184) ~[lucene-core-9.3.0.jar:?]
>>>> at org.apache.lucene.codecs.CodecUtil.checkIndexHeader(CodecUtil.java:253) ~[lucene-core-9.3.0.jar:?]
>>>> at org.apache.lucene.codecs.lucene90.Lucene90FieldInfosFormat.read(Lucene90FieldInfosFormat.java:128) ~[lucene-core-9.3.0.jar:?]
>>>> at org.apache.lucene.index.SegmentReader.initFieldInfos(SegmentReader.java:205) ~[lucene-core-9.3.0.jar:?]
>>>> at org.apache.lucene.index.SegmentReader.<init>(SegmentReader.java:156) ~[lucene-core-9.3.0.jar:?]
>>>> at org.apache.lucene.index.ReadersAndUpdates.createNewReaderWithLatestLiveDocs(ReadersAndUpdates.java:738) ~[lucene-core-9.3.0.jar:?]
>>>> at org.apache.lucene.index.ReadersAndUpdates.swapNewReaderWithLatestLiveDocs(ReadersAndUpdates.java:754) ~[lucene-core-9.3.0.jar:?]
>>>> at org.apache.lucene.index.ReadersAndUpdates.writeFieldUpdates(ReadersAndUpdates.java:678) ~[lucene-core-9.3.0.jar:?]
>>>> at org.apache.lucene.index.ReaderPool.writeAllDocValuesUpdates(ReaderPool.java:251) ~[lucene-core-9.3.0.jar:?]
>>>> at org.apache.lucene.index.IndexWriter.writeReaderPool(IndexWriter.java:3743) ~[lucene-core-9.3.0.jar:?]
>>>> at org.apache.lucene.index.IndexWriter.getReader(IndexWriter.java:591) ~[lucene-core-9.3.0.jar:?]
>>>> at org.apache.lucene.index.StandardDirectoryReader.doOpenFromWriter(StandardDirectoryReader.java:381) ~[lucene-core-9.3.0.jar:?]
>>>> at org.apache.lucene.index.StandardDirectoryReader.doOpenIfChanged(StandardDirectoryReader.java:355) ~[lucene-core-9.3.0.jar:?]
>>>> at org.apache.lucene.index.StandardDirectoryReader.doOpenIfChanged(StandardDirectoryReader.java:345) ~[lucene-core-9.3.0.jar:?]
>>>> at org.apache.lucene.index.FilterDirectoryReader.doOpenIfChanged(FilterDirectoryReader.java:112) ~[lucene-core-9.3.0.jar:?]
>>>> at org.apache.lucene.index.DirectoryReader.openIfChanged(DirectoryReader.java:170) ~[lucene-core-9.3.0.jar:?]
>>>> at org.elasticsearch.index.engine.ElasticsearchReaderManager.refreshIfNeeded(ElasticsearchReaderManager.java:48) ~[elasticsearch-8.4.1.jar:?]
>>>> at org.elasticsearch.index.engine.ElasticsearchReaderManager.refreshIfNeeded(ElasticsearchReaderManager.java:27) ~[elasticsearch-8.4.1.jar:?]
>>>> at org.apache.lucene.search.ReferenceManager.doMaybeRefresh(ReferenceManager.java:167) ~[lucene-core-9.3.0.jar:?]
>>>> at org.apache.lucene.search.ReferenceManager.maybeRefreshBlocking(ReferenceManager.java:240) ~[lucene-core-9.3.0.jar:?]
>>>> at org.elasticsearch.index.engine.InternalEngine$ExternalReaderManager.refreshIfNeeded(InternalEngine.java:355) ~[elasticsearch-8.4.1.jar:?]
>>>> at org.elasticsearch.index.engine.InternalEngine$ExternalReaderManager.refreshIfNeeded(InternalEngine.java:335) ~[elasticsearch-8.4.1.jar:?]
>>>> at org.apache.lucene.search.ReferenceManager.doMaybeRefresh(ReferenceManager.java:167) ~[lucene-core-9.3.0.jar:?]
>>>> Many thanks.
>>>>
>>>> Seb
>>> --
>>> Uwe Schindler
>>> Achterdiek 19, D-28357 Bremen
>>> https://www.thetaphi.de
>>> eMail: uwe@thetaphi.de
>>>
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>
> --
> Uwe Schindler
> Achterdiek 19, D-28357 Bremen
> https://www.thetaphi.de
> eMail: uwe@thetaphi.de
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>


---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Recurring index corruption [ In reply to ]
Hmm, I’m afraid I hit a roadblock:

2023-01-02T17:01:31,157][INFO ][o.e.b.BootstrapChecks ] [fs-sdlc-elasticsearch-001] bound or publishing to a non-loopback address, enforcing bootstrap checks
bootstrap check failure [1] of [1]: max virtual memory areas vm.max_map_count [65530] is too low, increase to at least [262144]

This is the error I got when I decided to go for NIOFS.

Maybe there is a way to tell ES to ignore this?

Seb


> On 2 Jan 2023, at 17:51, Uwe Schindler <uwe@thetaphi.de> wrote:
>
> Hi,
>
> in recent versions it works like that:
>
> https://www.elastic.co/guide/en/elasticsearch/reference/current/advanced-configuration.html#set-jvm-options
>
> So in folder jvm.options.d/ add a new file (like "preview.conf") and put "19:--enable-preview" into it. It is basically the same like modifying heap size.
>
> But in general, you can simply use MMapDirectory, the max-map-count setting is only relevant in *huge* (huge means hundreds of huge indexes per node). In that case Java 19's preview features would be recommended.
>
> Uwe
>
> Am 02.01.2023 um 17:41 schrieb S S:
>> Hi Uwe,
>>
>> Sorry for the late reply but upgrading the docker image to use OpenJDK was easier said that done.
>>
>> I am not a Java developer/expert so, sorry for the stupid question but, how do I specify the --enable-preview flag? ES has got a quite complex way to start so I cannot specify the flag on the command line. You suggested to use a jvm.properties file but I cannot find anything useful about it when google-ing. Where should it be placed? And what should I write in it?
>>
>> I can see ES recognising OpenJDK 19 while bootstrapping and suggesting to enable preview, but it does not suggest how, and I cannot find anything on the ES website.
>>
>> Many thanks.
>>
>> Seb
>>
>>> On 2 Jan 2023, at 11:48, Uwe Schindler <uwe@thetaphi.de> wrote:
>>>
>>> Hi,
>>>
>>> in general you can still use MMapDirectory. There is no requirement to set vm.max_map_count for smaller clusters. The information in Elastics documentation is not mandatory and misleading.
>>>
>>> If you use newest version of Elasticsearch with Java 19 and you use `--enable-preview` in you jvm.properties file, you don't even need to change that setting even with larger clusters.
>>>
>>> Uwe
>>>
>>> Am 02.01.2023 um 11:18 schrieb S S:
>>>> We are experimenting with Elastic Search deployed in Azure Container Instances (Debian + OpenJDK). The ES indexes are stored into an Azure file share mounted via SMB (3.0). The Elastic Search cluster is made up of 4 nodes, each one have a separate file share to store the indices.
>>>>
>>>> This configuration has been influenced by some ACIs limitations, specifically:
>>>>
>>>> we cannot set the max_map_count value as we do not have access to the underlying host (https://www.elastic.co/guide/en/elasticsearch/reference/current/vm-max-map-count.html). Unfortunately, this is required to run an ES cluster, therefore we were forced to use NIOF
>>>> ACI’s storage is ephemera, therefore we had to map volumes to persist the indexes. ACIs only allow volume mappings using Azure File Shares, which only works with NFS or SMB.
>>>>
>>>> We are experiencing recurring index corruption, specifically a "read past EOF" exception. I asked on the Elastic Search forum but the answer I got was a bit generic and not really helpful other than confirming that, from ES point of view, ES should work on an SMB share as long as it behaves as a local drive. As the underlying exception relates to an issue with a Lucene index, I was wondering if you could help out? Specifically, can Lucene work on SMB? I can only find sparse information on this configuration and, while NFS seems a no-no, for SMB is not that clear. Below is the exception we are getting.
>>>>
>>>> java.io.IOException: read past EOF: NIOFSIndexInput(path="/bitnami/elasticsearch/data/indices/mS2bUbLtSeG0FSAMuKX7JQ/0/index/_ldsn_1.fnm") buffer: java.nio.HeapByteBuffer[pos=0 lim=1024 cap=1024] chunkLen: 1024 end: 2331: NIOFSIndexInput(path="/bitnami/elasticsearch/data/indices/mS2bUbLtSeG0FSAMuKX7JQ/0/index/_ldsn_1.fnm")
>>>> at org.apache.lucene.store.NIOFSDirectory$NIOFSIndexInput.readInternal(NIOFSDirectory.java:200) ~[lucene-core-9.3.0.jar:?]
>>>> at org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.java:291) ~[lucene-core-9.3.0.jar:?]
>>>> at org.apache.lucene.store.BufferedIndexInput.readByte(BufferedIndexInput.java:55) ~[lucene-core-9.3.0.jar:?]
>>>> at org.apache.lucene.store.BufferedChecksumIndexInput.readByte(BufferedChecksumIndexInput.java:39) ~[lucene-core-9.3.0.jar:?]
>>>> at org.apache.lucene.codecs.CodecUtil.readBEInt(CodecUtil.java:667) ~[lucene-core-9.3.0.jar:?]
>>>> at org.apache.lucene.codecs.CodecUtil.checkHeader(CodecUtil.java:184) ~[lucene-core-9.3.0.jar:?]
>>>> at org.apache.lucene.codecs.CodecUtil.checkIndexHeader(CodecUtil.java:253) ~[lucene-core-9.3.0.jar:?]
>>>> at org.apache.lucene.codecs.lucene90.Lucene90FieldInfosFormat.read(Lucene90FieldInfosFormat.java:128) ~[lucene-core-9.3.0.jar:?]
>>>> at org.apache.lucene.index.SegmentReader.initFieldInfos(SegmentReader.java:205) ~[lucene-core-9.3.0.jar:?]
>>>> at org.apache.lucene.index.SegmentReader.<init>(SegmentReader.java:156) ~[lucene-core-9.3.0.jar:?]
>>>> at org.apache.lucene.index.ReadersAndUpdates.createNewReaderWithLatestLiveDocs(ReadersAndUpdates.java:738) ~[lucene-core-9.3.0.jar:?]
>>>> at org.apache.lucene.index.ReadersAndUpdates.swapNewReaderWithLatestLiveDocs(ReadersAndUpdates.java:754) ~[lucene-core-9.3.0.jar:?]
>>>> at org.apache.lucene.index.ReadersAndUpdates.writeFieldUpdates(ReadersAndUpdates.java:678) ~[lucene-core-9.3.0.jar:?]
>>>> at org.apache.lucene.index.ReaderPool.writeAllDocValuesUpdates(ReaderPool.java:251) ~[lucene-core-9.3.0.jar:?]
>>>> at org.apache.lucene.index.IndexWriter.writeReaderPool(IndexWriter.java:3743) ~[lucene-core-9.3.0.jar:?]
>>>> at org.apache.lucene.index.IndexWriter.getReader(IndexWriter.java:591) ~[lucene-core-9.3.0.jar:?]
>>>> at org.apache.lucene.index.StandardDirectoryReader.doOpenFromWriter(StandardDirectoryReader.java:381) ~[lucene-core-9.3.0.jar:?]
>>>> at org.apache.lucene.index.StandardDirectoryReader.doOpenIfChanged(StandardDirectoryReader.java:355) ~[lucene-core-9.3.0.jar:?]
>>>> at org.apache.lucene.index.StandardDirectoryReader.doOpenIfChanged(StandardDirectoryReader.java:345) ~[lucene-core-9.3.0.jar:?]
>>>> at org.apache.lucene.index.FilterDirectoryReader.doOpenIfChanged(FilterDirectoryReader.java:112) ~[lucene-core-9.3.0.jar:?]
>>>> at org.apache.lucene.index.DirectoryReader.openIfChanged(DirectoryReader.java:170) ~[lucene-core-9.3.0.jar:?]
>>>> at org.elasticsearch.index.engine.ElasticsearchReaderManager.refreshIfNeeded(ElasticsearchReaderManager.java:48) ~[elasticsearch-8.4.1.jar:?]
>>>> at org.elasticsearch.index.engine.ElasticsearchReaderManager.refreshIfNeeded(ElasticsearchReaderManager.java:27) ~[elasticsearch-8.4.1.jar:?]
>>>> at org.apache.lucene.search.ReferenceManager.doMaybeRefresh(ReferenceManager.java:167) ~[lucene-core-9.3.0.jar:?]
>>>> at org.apache.lucene.search.ReferenceManager.maybeRefreshBlocking(ReferenceManager.java:240) ~[lucene-core-9.3.0.jar:?]
>>>> at org.elasticsearch.index.engine.InternalEngine$ExternalReaderManager.refreshIfNeeded(InternalEngine.java:355) ~[elasticsearch-8.4.1.jar:?]
>>>> at org.elasticsearch.index.engine.InternalEngine$ExternalReaderManager.refreshIfNeeded(InternalEngine.java:335) ~[elasticsearch-8.4.1.jar:?]
>>>> at org.apache.lucene.search.ReferenceManager.doMaybeRefresh(ReferenceManager.java:167) ~[lucene-core-9.3.0.jar:?]
>>>> Many thanks.
>>>>
>>>> Seb
>>> --
>>> Uwe Schindler
>>> Achterdiek 19, D-28357 Bremen
>>> https://www.thetaphi.de
>>> eMail: uwe@thetaphi.de
>>>
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>
> --
> Uwe Schindler
> Achterdiek 19, D-28357 Bremen
> https://www.thetaphi.de
> eMail: uwe@thetaphi.de
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
Re: Recurring index corruption [ In reply to ]
I also tried enabling preview but no joy, same error :(

It looks like it is not possible to start a multinode ES cluster without setting vm.max_map_count. I also googled it and this check cannot be disabled.

I guess MMapDirectory is not an option for ES on ACIs, unless you have something else I can try?

Many thanks,

Seb

> On 2 Jan 2023, at 17:55, S S <sebastiano1972@yahoo.it.INVALID> wrote:
>
> Thank you Uwe, this is great! I am rebuilding the cluster using MMapDirectory and no enable-preview, as you suggested. Let’s see what happens.
>
> Cheers,
>
> Seb
>
>> On 2 Jan 2023, at 17:51, Uwe Schindler <uwe@thetaphi.de> wrote:
>>
>> Hi,
>>
>> in recent versions it works like that:
>>
>> https://www.elastic.co/guide/en/elasticsearch/reference/current/advanced-configuration.html#set-jvm-options
>>
>> So in folder jvm.options.d/ add a new file (like "preview.conf") and put "19:--enable-preview" into it. It is basically the same like modifying heap size.
>>
>> But in general, you can simply use MMapDirectory, the max-map-count setting is only relevant in *huge* (huge means hundreds of huge indexes per node). In that case Java 19's preview features would be recommended.
>>
>> Uwe
>>
>> Am 02.01.2023 um 17:41 schrieb S S:
>>> Hi Uwe,
>>>
>>> Sorry for the late reply but upgrading the docker image to use OpenJDK was easier said that done.
>>>
>>> I am not a Java developer/expert so, sorry for the stupid question but, how do I specify the --enable-preview flag? ES has got a quite complex way to start so I cannot specify the flag on the command line. You suggested to use a jvm.properties file but I cannot find anything useful about it when google-ing. Where should it be placed? And what should I write in it?
>>>
>>> I can see ES recognising OpenJDK 19 while bootstrapping and suggesting to enable preview, but it does not suggest how, and I cannot find anything on the ES website.
>>>
>>> Many thanks.
>>>
>>> Seb
>>>
>>>> On 2 Jan 2023, at 11:48, Uwe Schindler <uwe@thetaphi.de> wrote:
>>>>
>>>> Hi,
>>>>
>>>> in general you can still use MMapDirectory. There is no requirement to set vm.max_map_count for smaller clusters. The information in Elastics documentation is not mandatory and misleading.
>>>>
>>>> If you use newest version of Elasticsearch with Java 19 and you use `--enable-preview` in you jvm.properties file, you don't even need to change that setting even with larger clusters.
>>>>
>>>> Uwe
>>>>
>>>> Am 02.01.2023 um 11:18 schrieb S S:
>>>>> We are experimenting with Elastic Search deployed in Azure Container Instances (Debian + OpenJDK). The ES indexes are stored into an Azure file share mounted via SMB (3.0). The Elastic Search cluster is made up of 4 nodes, each one have a separate file share to store the indices.
>>>>>
>>>>> This configuration has been influenced by some ACIs limitations, specifically:
>>>>>
>>>>> we cannot set the max_map_count value as we do not have access to the underlying host (https://www.elastic.co/guide/en/elasticsearch/reference/current/vm-max-map-count.html). Unfortunately, this is required to run an ES cluster, therefore we were forced to use NIOF
>>>>> ACI’s storage is ephemera, therefore we had to map volumes to persist the indexes. ACIs only allow volume mappings using Azure File Shares, which only works with NFS or SMB.
>>>>>
>>>>> We are experiencing recurring index corruption, specifically a "read past EOF" exception. I asked on the Elastic Search forum but the answer I got was a bit generic and not really helpful other than confirming that, from ES point of view, ES should work on an SMB share as long as it behaves as a local drive. As the underlying exception relates to an issue with a Lucene index, I was wondering if you could help out? Specifically, can Lucene work on SMB? I can only find sparse information on this configuration and, while NFS seems a no-no, for SMB is not that clear. Below is the exception we are getting.
>>>>>
>>>>> java.io.IOException: read past EOF: NIOFSIndexInput(path="/bitnami/elasticsearch/data/indices/mS2bUbLtSeG0FSAMuKX7JQ/0/index/_ldsn_1.fnm") buffer: java.nio.HeapByteBuffer[pos=0 lim=1024 cap=1024] chunkLen: 1024 end: 2331: NIOFSIndexInput(path="/bitnami/elasticsearch/data/indices/mS2bUbLtSeG0FSAMuKX7JQ/0/index/_ldsn_1.fnm")
>>>>> at org.apache.lucene.store.NIOFSDirectory$NIOFSIndexInput.readInternal(NIOFSDirectory.java:200) ~[lucene-core-9.3.0.jar:?]
>>>>> at org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.java:291) ~[lucene-core-9.3.0.jar:?]
>>>>> at org.apache.lucene.store.BufferedIndexInput.readByte(BufferedIndexInput.java:55) ~[lucene-core-9.3.0.jar:?]
>>>>> at org.apache.lucene.store.BufferedChecksumIndexInput.readByte(BufferedChecksumIndexInput.java:39) ~[lucene-core-9.3.0.jar:?]
>>>>> at org.apache.lucene.codecs.CodecUtil.readBEInt(CodecUtil.java:667) ~[lucene-core-9.3.0.jar:?]
>>>>> at org.apache.lucene.codecs.CodecUtil.checkHeader(CodecUtil.java:184) ~[lucene-core-9.3.0.jar:?]
>>>>> at org.apache.lucene.codecs.CodecUtil.checkIndexHeader(CodecUtil.java:253) ~[lucene-core-9.3.0.jar:?]
>>>>> at org.apache.lucene.codecs.lucene90.Lucene90FieldInfosFormat.read(Lucene90FieldInfosFormat.java:128) ~[lucene-core-9.3.0.jar:?]
>>>>> at org.apache.lucene.index.SegmentReader.initFieldInfos(SegmentReader.java:205) ~[lucene-core-9.3.0.jar:?]
>>>>> at org.apache.lucene.index.SegmentReader.<init>(SegmentReader.java:156) ~[lucene-core-9.3.0.jar:?]
>>>>> at org.apache.lucene.index.ReadersAndUpdates.createNewReaderWithLatestLiveDocs(ReadersAndUpdates.java:738) ~[lucene-core-9.3.0.jar:?]
>>>>> at org.apache.lucene.index.ReadersAndUpdates.swapNewReaderWithLatestLiveDocs(ReadersAndUpdates.java:754) ~[lucene-core-9.3.0.jar:?]
>>>>> at org.apache.lucene.index.ReadersAndUpdates.writeFieldUpdates(ReadersAndUpdates.java:678) ~[lucene-core-9.3.0.jar:?]
>>>>> at org.apache.lucene.index.ReaderPool.writeAllDocValuesUpdates(ReaderPool.java:251) ~[lucene-core-9.3.0.jar:?]
>>>>> at org.apache.lucene.index.IndexWriter.writeReaderPool(IndexWriter.java:3743) ~[lucene-core-9.3.0.jar:?]
>>>>> at org.apache.lucene.index.IndexWriter.getReader(IndexWriter.java:591) ~[lucene-core-9.3.0.jar:?]
>>>>> at org.apache.lucene.index.StandardDirectoryReader.doOpenFromWriter(StandardDirectoryReader.java:381) ~[lucene-core-9.3.0.jar:?]
>>>>> at org.apache.lucene.index.StandardDirectoryReader.doOpenIfChanged(StandardDirectoryReader.java:355) ~[lucene-core-9.3.0.jar:?]
>>>>> at org.apache.lucene.index.StandardDirectoryReader.doOpenIfChanged(StandardDirectoryReader.java:345) ~[lucene-core-9.3.0.jar:?]
>>>>> at org.apache.lucene.index.FilterDirectoryReader.doOpenIfChanged(FilterDirectoryReader.java:112) ~[lucene-core-9.3.0.jar:?]
>>>>> at org.apache.lucene.index.DirectoryReader.openIfChanged(DirectoryReader.java:170) ~[lucene-core-9.3.0.jar:?]
>>>>> at org.elasticsearch.index.engine.ElasticsearchReaderManager.refreshIfNeeded(ElasticsearchReaderManager.java:48) ~[elasticsearch-8.4.1.jar:?]
>>>>> at org.elasticsearch.index.engine.ElasticsearchReaderManager.refreshIfNeeded(ElasticsearchReaderManager.java:27) ~[elasticsearch-8.4.1.jar:?]
>>>>> at org.apache.lucene.search.ReferenceManager.doMaybeRefresh(ReferenceManager.java:167) ~[lucene-core-9.3.0.jar:?]
>>>>> at org.apache.lucene.search.ReferenceManager.maybeRefreshBlocking(ReferenceManager.java:240) ~[lucene-core-9.3.0.jar:?]
>>>>> at org.elasticsearch.index.engine.InternalEngine$ExternalReaderManager.refreshIfNeeded(InternalEngine.java:355) ~[elasticsearch-8.4.1.jar:?]
>>>>> at org.elasticsearch.index.engine.InternalEngine$ExternalReaderManager.refreshIfNeeded(InternalEngine.java:335) ~[elasticsearch-8.4.1.jar:?]
>>>>> at org.apache.lucene.search.ReferenceManager.doMaybeRefresh(ReferenceManager.java:167) ~[lucene-core-9.3.0.jar:?]
>>>>> Many thanks.
>>>>>
>>>>> Seb
>>>> --
>>>> Uwe Schindler
>>>> Achterdiek 19, D-28357 Bremen
>>>> https://www.thetaphi.de
>>>> eMail: uwe@thetaphi.de
>>>>
>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>>>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>>>
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>>
>> --
>> Uwe Schindler
>> Achterdiek 19, D-28357 Bremen
>> https://www.thetaphi.de
>> eMail: uwe@thetaphi.de
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>


---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Recurring index corruption [ In reply to ]
Hi,

Please open a bug report at ES. The setting vm.max_map_count is not
needed and should not be changed unless really needed, because it uses
kernel resources.

This has to do with their support (they try to tell people to overshard
and to prevent support requests they ask to raise this seeting). The
default value on Linux is 65530. This would allow you to memory map
65530 chunks of 1 GiB (thats the limitation without preview enabled in
the Java 19 JVM). A shard in Lucene has about 70 files, most of them
with filesizes < 1 Gib and a few >1 GiB (but < 5 GiB) chunks (so lets
assume 90 mappings needed for a shard), so you could have approx 728
shards per node. Sorry raising this setting is not needed!

Uwe

Am 02.01.2023 um 18:24 schrieb S S:
> I also tried enabling preview but no joy, same error :(
>
> It looks like it is not possible to start a multinode ES cluster without setting vm.max_map_count. I also googled it and this check cannot be disabled.
>
> I guess MMapDirectory is not an option for ES on ACIs, unless you have something else I can try?
>
> Many thanks,
>
> Seb
>
>> On 2 Jan 2023, at 17:55, S S <sebastiano1972@yahoo.it.INVALID> wrote:
>>
>> Thank you Uwe, this is great! I am rebuilding the cluster using MMapDirectory and no enable-preview, as you suggested. Let’s see what happens.
>>
>> Cheers,
>>
>> Seb
>>
>>> On 2 Jan 2023, at 17:51, Uwe Schindler <uwe@thetaphi.de> wrote:
>>>
>>> Hi,
>>>
>>> in recent versions it works like that:
>>>
>>> https://www.elastic.co/guide/en/elasticsearch/reference/current/advanced-configuration.html#set-jvm-options
>>>
>>> So in folder jvm.options.d/ add a new file (like "preview.conf") and put "19:--enable-preview" into it. It is basically the same like modifying heap size.
>>>
>>> But in general, you can simply use MMapDirectory, the max-map-count setting is only relevant in *huge* (huge means hundreds of huge indexes per node). In that case Java 19's preview features would be recommended.
>>>
>>> Uwe
>>>
>>> Am 02.01.2023 um 17:41 schrieb S S:
>>>> Hi Uwe,
>>>>
>>>> Sorry for the late reply but upgrading the docker image to use OpenJDK was easier said that done.
>>>>
>>>> I am not a Java developer/expert so, sorry for the stupid question but, how do I specify the --enable-preview flag? ES has got a quite complex way to start so I cannot specify the flag on the command line. You suggested to use a jvm.properties file but I cannot find anything useful about it when google-ing. Where should it be placed? And what should I write in it?
>>>>
>>>> I can see ES recognising OpenJDK 19 while bootstrapping and suggesting to enable preview, but it does not suggest how, and I cannot find anything on the ES website.
>>>>
>>>> Many thanks.
>>>>
>>>> Seb
>>>>
>>>>> On 2 Jan 2023, at 11:48, Uwe Schindler <uwe@thetaphi.de> wrote:
>>>>>
>>>>> Hi,
>>>>>
>>>>> in general you can still use MMapDirectory. There is no requirement to set vm.max_map_count for smaller clusters. The information in Elastics documentation is not mandatory and misleading.
>>>>>
>>>>> If you use newest version of Elasticsearch with Java 19 and you use `--enable-preview` in you jvm.properties file, you don't even need to change that setting even with larger clusters.
>>>>>
>>>>> Uwe
>>>>>
>>>>> Am 02.01.2023 um 11:18 schrieb S S:
>>>>>> We are experimenting with Elastic Search deployed in Azure Container Instances (Debian + OpenJDK). The ES indexes are stored into an Azure file share mounted via SMB (3.0). The Elastic Search cluster is made up of 4 nodes, each one have a separate file share to store the indices.
>>>>>>
>>>>>> This configuration has been influenced by some ACIs limitations, specifically:
>>>>>>
>>>>>> we cannot set the max_map_count value as we do not have access to the underlying host (https://www.elastic.co/guide/en/elasticsearch/reference/current/vm-max-map-count.html). Unfortunately, this is required to run an ES cluster, therefore we were forced to use NIOF
>>>>>> ACI’s storage is ephemera, therefore we had to map volumes to persist the indexes. ACIs only allow volume mappings using Azure File Shares, which only works with NFS or SMB.
>>>>>>
>>>>>> We are experiencing recurring index corruption, specifically a "read past EOF" exception. I asked on the Elastic Search forum but the answer I got was a bit generic and not really helpful other than confirming that, from ES point of view, ES should work on an SMB share as long as it behaves as a local drive. As the underlying exception relates to an issue with a Lucene index, I was wondering if you could help out? Specifically, can Lucene work on SMB? I can only find sparse information on this configuration and, while NFS seems a no-no, for SMB is not that clear. Below is the exception we are getting.
>>>>>>
>>>>>> java.io.IOException: read past EOF: NIOFSIndexInput(path="/bitnami/elasticsearch/data/indices/mS2bUbLtSeG0FSAMuKX7JQ/0/index/_ldsn_1.fnm") buffer: java.nio.HeapByteBuffer[pos=0 lim=1024 cap=1024] chunkLen: 1024 end: 2331: NIOFSIndexInput(path="/bitnami/elasticsearch/data/indices/mS2bUbLtSeG0FSAMuKX7JQ/0/index/_ldsn_1.fnm")
>>>>>> at org.apache.lucene.store.NIOFSDirectory$NIOFSIndexInput.readInternal(NIOFSDirectory.java:200) ~[lucene-core-9.3.0.jar:?]
>>>>>> at org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.java:291) ~[lucene-core-9.3.0.jar:?]
>>>>>> at org.apache.lucene.store.BufferedIndexInput.readByte(BufferedIndexInput.java:55) ~[lucene-core-9.3.0.jar:?]
>>>>>> at org.apache.lucene.store.BufferedChecksumIndexInput.readByte(BufferedChecksumIndexInput.java:39) ~[lucene-core-9.3.0.jar:?]
>>>>>> at org.apache.lucene.codecs.CodecUtil.readBEInt(CodecUtil.java:667) ~[lucene-core-9.3.0.jar:?]
>>>>>> at org.apache.lucene.codecs.CodecUtil.checkHeader(CodecUtil.java:184) ~[lucene-core-9.3.0.jar:?]
>>>>>> at org.apache.lucene.codecs.CodecUtil.checkIndexHeader(CodecUtil.java:253) ~[lucene-core-9.3.0.jar:?]
>>>>>> at org.apache.lucene.codecs.lucene90.Lucene90FieldInfosFormat.read(Lucene90FieldInfosFormat.java:128) ~[lucene-core-9.3.0.jar:?]
>>>>>> at org.apache.lucene.index.SegmentReader.initFieldInfos(SegmentReader.java:205) ~[lucene-core-9.3.0.jar:?]
>>>>>> at org.apache.lucene.index.SegmentReader.<init>(SegmentReader.java:156) ~[lucene-core-9.3.0.jar:?]
>>>>>> at org.apache.lucene.index.ReadersAndUpdates.createNewReaderWithLatestLiveDocs(ReadersAndUpdates.java:738) ~[lucene-core-9.3.0.jar:?]
>>>>>> at org.apache.lucene.index.ReadersAndUpdates.swapNewReaderWithLatestLiveDocs(ReadersAndUpdates.java:754) ~[lucene-core-9.3.0.jar:?]
>>>>>> at org.apache.lucene.index.ReadersAndUpdates.writeFieldUpdates(ReadersAndUpdates.java:678) ~[lucene-core-9.3.0.jar:?]
>>>>>> at org.apache.lucene.index.ReaderPool.writeAllDocValuesUpdates(ReaderPool.java:251) ~[lucene-core-9.3.0.jar:?]
>>>>>> at org.apache.lucene.index.IndexWriter.writeReaderPool(IndexWriter.java:3743) ~[lucene-core-9.3.0.jar:?]
>>>>>> at org.apache.lucene.index.IndexWriter.getReader(IndexWriter.java:591) ~[lucene-core-9.3.0.jar:?]
>>>>>> at org.apache.lucene.index.StandardDirectoryReader.doOpenFromWriter(StandardDirectoryReader.java:381) ~[lucene-core-9.3.0.jar:?]
>>>>>> at org.apache.lucene.index.StandardDirectoryReader.doOpenIfChanged(StandardDirectoryReader.java:355) ~[lucene-core-9.3.0.jar:?]
>>>>>> at org.apache.lucene.index.StandardDirectoryReader.doOpenIfChanged(StandardDirectoryReader.java:345) ~[lucene-core-9.3.0.jar:?]
>>>>>> at org.apache.lucene.index.FilterDirectoryReader.doOpenIfChanged(FilterDirectoryReader.java:112) ~[lucene-core-9.3.0.jar:?]
>>>>>> at org.apache.lucene.index.DirectoryReader.openIfChanged(DirectoryReader.java:170) ~[lucene-core-9.3.0.jar:?]
>>>>>> at org.elasticsearch.index.engine.ElasticsearchReaderManager.refreshIfNeeded(ElasticsearchReaderManager.java:48) ~[elasticsearch-8.4.1.jar:?]
>>>>>> at org.elasticsearch.index.engine.ElasticsearchReaderManager.refreshIfNeeded(ElasticsearchReaderManager.java:27) ~[elasticsearch-8.4.1.jar:?]
>>>>>> at org.apache.lucene.search.ReferenceManager.doMaybeRefresh(ReferenceManager.java:167) ~[lucene-core-9.3.0.jar:?]
>>>>>> at org.apache.lucene.search.ReferenceManager.maybeRefreshBlocking(ReferenceManager.java:240) ~[lucene-core-9.3.0.jar:?]
>>>>>> at org.elasticsearch.index.engine.InternalEngine$ExternalReaderManager.refreshIfNeeded(InternalEngine.java:355) ~[elasticsearch-8.4.1.jar:?]
>>>>>> at org.elasticsearch.index.engine.InternalEngine$ExternalReaderManager.refreshIfNeeded(InternalEngine.java:335) ~[elasticsearch-8.4.1.jar:?]
>>>>>> at org.apache.lucene.search.ReferenceManager.doMaybeRefresh(ReferenceManager.java:167) ~[lucene-core-9.3.0.jar:?]
>>>>>> Many thanks.
>>>>>>
>>>>>> Seb
>>>>> --
>>>>> Uwe Schindler
>>>>> Achterdiek 19, D-28357 Bremen
>>>>> https://www.thetaphi.de
>>>>> eMail: uwe@thetaphi.de
>>>>>
>>>>>
>>>>> ---------------------------------------------------------------------
>>>>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>>>>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>>>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>>>
>>> --
>>> Uwe Schindler
>>> Achterdiek 19, D-28357 Bremen
>>> https://www.thetaphi.de
>>> eMail: uwe@thetaphi.de
>>>
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
--
Uwe Schindler
Achterdiek 19, D-28357 Bremen
https://www.thetaphi.de
eMail: uwe@thetaphi.de


---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Recurring index corruption [ In reply to ]
Hi Uwe,

I will report the bug to ES, as you suggested.

Do you recon using Mmap would have an effect to the index corruption when using SMB? I have to report back to my manager in few days to decide wether to carry on with ACIs or find another hosting solution. It is unfortunate there seems to be problems with this solution. Microsoft seems not interested in extending the volume mapping options for ACIs and K8 is overkilling for our use case.

Thank you for your help so far, you have been very kind :)

Cheers,

Seb

> On 2 Jan 2023, at 19:09, Uwe Schindler <uwe@thetaphi.de> wrote:
>
> Hi,
>
> Please open a bug report at ES. The setting vm.max_map_count is not needed and should not be changed unless really needed, because it uses kernel resources.
>
> This has to do with their support (they try to tell people to overshard and to prevent support requests they ask to raise this seeting). The default value on Linux is 65530. This would allow you to memory map 65530 chunks of 1 GiB (thats the limitation without preview enabled in the Java 19 JVM). A shard in Lucene has about 70 files, most of them with filesizes < 1 Gib and a few >1 GiB (but < 5 GiB) chunks (so lets assume 90 mappings needed for a shard), so you could have approx 728 shards per node. Sorry raising this setting is not needed!
>
> Uwe
>
> Am 02.01.2023 um 18:24 schrieb S S:
>> I also tried enabling preview but no joy, same error :(
>>
>> It looks like it is not possible to start a multinode ES cluster without setting vm.max_map_count. I also googled it and this check cannot be disabled.
>>
>> I guess MMapDirectory is not an option for ES on ACIs, unless you have something else I can try?
>>
>> Many thanks,
>>
>> Seb
>>
>>> On 2 Jan 2023, at 17:55, S S <sebastiano1972@yahoo.it.INVALID> wrote:
>>>
>>> Thank you Uwe, this is great! I am rebuilding the cluster using MMapDirectory and no enable-preview, as you suggested. Let’s see what happens.
>>>
>>> Cheers,
>>>
>>> Seb
>>>
>>>> On 2 Jan 2023, at 17:51, Uwe Schindler <uwe@thetaphi.de> wrote:
>>>>
>>>> Hi,
>>>>
>>>> in recent versions it works like that:
>>>>
>>>> https://www.elastic.co/guide/en/elasticsearch/reference/current/advanced-configuration.html#set-jvm-options
>>>>
>>>> So in folder jvm.options.d/ add a new file (like "preview.conf") and put "19:--enable-preview" into it. It is basically the same like modifying heap size.
>>>>
>>>> But in general, you can simply use MMapDirectory, the max-map-count setting is only relevant in *huge* (huge means hundreds of huge indexes per node). In that case Java 19's preview features would be recommended.
>>>>
>>>> Uwe
>>>>
>>>> Am 02.01.2023 um 17:41 schrieb S S:
>>>>> Hi Uwe,
>>>>>
>>>>> Sorry for the late reply but upgrading the docker image to use OpenJDK was easier said that done.
>>>>>
>>>>> I am not a Java developer/expert so, sorry for the stupid question but, how do I specify the --enable-preview flag? ES has got a quite complex way to start so I cannot specify the flag on the command line. You suggested to use a jvm.properties file but I cannot find anything useful about it when google-ing. Where should it be placed? And what should I write in it?
>>>>>
>>>>> I can see ES recognising OpenJDK 19 while bootstrapping and suggesting to enable preview, but it does not suggest how, and I cannot find anything on the ES website.
>>>>>
>>>>> Many thanks.
>>>>>
>>>>> Seb
>>>>>
>>>>>> On 2 Jan 2023, at 11:48, Uwe Schindler <uwe@thetaphi.de> wrote:
>>>>>>
>>>>>> Hi,
>>>>>>
>>>>>> in general you can still use MMapDirectory. There is no requirement to set vm.max_map_count for smaller clusters. The information in Elastics documentation is not mandatory and misleading.
>>>>>>
>>>>>> If you use newest version of Elasticsearch with Java 19 and you use `--enable-preview` in you jvm.properties file, you don't even need to change that setting even with larger clusters.
>>>>>>
>>>>>> Uwe
>>>>>>
>>>>>> Am 02.01.2023 um 11:18 schrieb S S:
>>>>>>> We are experimenting with Elastic Search deployed in Azure Container Instances (Debian + OpenJDK). The ES indexes are stored into an Azure file share mounted via SMB (3.0). The Elastic Search cluster is made up of 4 nodes, each one have a separate file share to store the indices.
>>>>>>>
>>>>>>> This configuration has been influenced by some ACIs limitations, specifically:
>>>>>>>
>>>>>>> we cannot set the max_map_count value as we do not have access to the underlying host (https://www.elastic.co/guide/en/elasticsearch/reference/current/vm-max-map-count.html). Unfortunately, this is required to run an ES cluster, therefore we were forced to use NIOF
>>>>>>> ACI’s storage is ephemera, therefore we had to map volumes to persist the indexes. ACIs only allow volume mappings using Azure File Shares, which only works with NFS or SMB.
>>>>>>>
>>>>>>> We are experiencing recurring index corruption, specifically a "read past EOF" exception. I asked on the Elastic Search forum but the answer I got was a bit generic and not really helpful other than confirming that, from ES point of view, ES should work on an SMB share as long as it behaves as a local drive. As the underlying exception relates to an issue with a Lucene index, I was wondering if you could help out? Specifically, can Lucene work on SMB? I can only find sparse information on this configuration and, while NFS seems a no-no, for SMB is not that clear. Below is the exception we are getting.
>>>>>>>
>>>>>>> java.io.IOException: read past EOF: NIOFSIndexInput(path="/bitnami/elasticsearch/data/indices/mS2bUbLtSeG0FSAMuKX7JQ/0/index/_ldsn_1.fnm") buffer: java.nio.HeapByteBuffer[pos=0 lim=1024 cap=1024] chunkLen: 1024 end: 2331: NIOFSIndexInput(path="/bitnami/elasticsearch/data/indices/mS2bUbLtSeG0FSAMuKX7JQ/0/index/_ldsn_1.fnm")
>>>>>>> at org.apache.lucene.store.NIOFSDirectory$NIOFSIndexInput.readInternal(NIOFSDirectory.java:200) ~[lucene-core-9.3.0.jar:?]
>>>>>>> at org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.java:291) ~[lucene-core-9.3.0.jar:?]
>>>>>>> at org.apache.lucene.store.BufferedIndexInput.readByte(BufferedIndexInput.java:55) ~[lucene-core-9.3.0.jar:?]
>>>>>>> at org.apache.lucene.store.BufferedChecksumIndexInput.readByte(BufferedChecksumIndexInput.java:39) ~[lucene-core-9.3.0.jar:?]
>>>>>>> at org.apache.lucene.codecs.CodecUtil.readBEInt(CodecUtil.java:667) ~[lucene-core-9.3.0.jar:?]
>>>>>>> at org.apache.lucene.codecs.CodecUtil.checkHeader(CodecUtil.java:184) ~[lucene-core-9.3.0.jar:?]
>>>>>>> at org.apache.lucene.codecs.CodecUtil.checkIndexHeader(CodecUtil.java:253) ~[lucene-core-9.3.0.jar:?]
>>>>>>> at org.apache.lucene.codecs.lucene90.Lucene90FieldInfosFormat.read(Lucene90FieldInfosFormat.java:128) ~[lucene-core-9.3.0.jar:?]
>>>>>>> at org.apache.lucene.index.SegmentReader.initFieldInfos(SegmentReader.java:205) ~[lucene-core-9.3.0.jar:?]
>>>>>>> at org.apache.lucene.index.SegmentReader.<init>(SegmentReader.java:156) ~[lucene-core-9.3.0.jar:?]
>>>>>>> at org.apache.lucene.index.ReadersAndUpdates.createNewReaderWithLatestLiveDocs(ReadersAndUpdates.java:738) ~[lucene-core-9.3.0.jar:?]
>>>>>>> at org.apache.lucene.index.ReadersAndUpdates.swapNewReaderWithLatestLiveDocs(ReadersAndUpdates.java:754) ~[lucene-core-9.3.0.jar:?]
>>>>>>> at org.apache.lucene.index.ReadersAndUpdates.writeFieldUpdates(ReadersAndUpdates.java:678) ~[lucene-core-9.3.0.jar:?]
>>>>>>> at org.apache.lucene.index.ReaderPool.writeAllDocValuesUpdates(ReaderPool.java:251) ~[lucene-core-9.3.0.jar:?]
>>>>>>> at org.apache.lucene.index.IndexWriter.writeReaderPool(IndexWriter.java:3743) ~[lucene-core-9.3.0.jar:?]
>>>>>>> at org.apache.lucene.index.IndexWriter.getReader(IndexWriter.java:591) ~[lucene-core-9.3.0.jar:?]
>>>>>>> at org.apache.lucene.index.StandardDirectoryReader.doOpenFromWriter(StandardDirectoryReader.java:381) ~[lucene-core-9.3.0.jar:?]
>>>>>>> at org.apache.lucene.index.StandardDirectoryReader.doOpenIfChanged(StandardDirectoryReader.java:355) ~[lucene-core-9.3.0.jar:?]
>>>>>>> at org.apache.lucene.index.StandardDirectoryReader.doOpenIfChanged(StandardDirectoryReader.java:345) ~[lucene-core-9.3.0.jar:?]
>>>>>>> at org.apache.lucene.index.FilterDirectoryReader.doOpenIfChanged(FilterDirectoryReader.java:112) ~[lucene-core-9.3.0.jar:?]
>>>>>>> at org.apache.lucene.index.DirectoryReader.openIfChanged(DirectoryReader.java:170) ~[lucene-core-9.3.0.jar:?]
>>>>>>> at org.elasticsearch.index.engine.ElasticsearchReaderManager.refreshIfNeeded(ElasticsearchReaderManager.java:48) ~[elasticsearch-8.4.1.jar:?]
>>>>>>> at org.elasticsearch.index.engine.ElasticsearchReaderManager.refreshIfNeeded(ElasticsearchReaderManager.java:27) ~[elasticsearch-8.4.1.jar:?]
>>>>>>> at org.apache.lucene.search.ReferenceManager.doMaybeRefresh(ReferenceManager.java:167) ~[lucene-core-9.3.0.jar:?]
>>>>>>> at org.apache.lucene.search.ReferenceManager.maybeRefreshBlocking(ReferenceManager.java:240) ~[lucene-core-9.3.0.jar:?]
>>>>>>> at org.elasticsearch.index.engine.InternalEngine$ExternalReaderManager.refreshIfNeeded(InternalEngine.java:355) ~[elasticsearch-8.4.1.jar:?]
>>>>>>> at org.elasticsearch.index.engine.InternalEngine$ExternalReaderManager.refreshIfNeeded(InternalEngine.java:335) ~[elasticsearch-8.4.1.jar:?]
>>>>>>> at org.apache.lucene.search.ReferenceManager.doMaybeRefresh(ReferenceManager.java:167) ~[lucene-core-9.3.0.jar:?]
>>>>>>> Many thanks.
>>>>>>>
>>>>>>> Seb
>>>>>> --
>>>>>> Uwe Schindler
>>>>>> Achterdiek 19, D-28357 Bremen
>>>>>> https://www.thetaphi.de
>>>>>> eMail: uwe@thetaphi.de
>>>>>>
>>>>>>
>>>>>> ---------------------------------------------------------------------
>>>>>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>>>>>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>>>>>
>>>>> ---------------------------------------------------------------------
>>>>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>>>>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>>>>
>>>> --
>>>> Uwe Schindler
>>>> Achterdiek 19, D-28357 Bremen
>>>> https://www.thetaphi.de
>>>> eMail: uwe@thetaphi.de
>>>>
>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>>>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>>>
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>
> --
> Uwe Schindler
> Achterdiek 19, D-28357 Bremen
> https://www.thetaphi.de
> eMail: uwe@thetaphi.de
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>


---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Recurring index corruption [ In reply to ]
Hi,

Yeah, sorry for leading the issue into the wrong direction. I was just
stumbling on the exception message and because we do not spend much time
in improving/supporting the use of NIOFSDirectory, I may have moved this
mailing list thread into the wrong direction.

I don't think the directory implementation will change much when using
Samba. Recent versions of the CIFS specification (the spec behind Samba)
and the implementation in Windows Server are fine. But I doubt that
recent implementations in Linux kernel are correct and fully support the
spec.

The corrumption issue you see is just detected by NIOFSDirectory (which
I stumbled upon), but with MMapDirectory it will happen in the same way
as the problem is caused by fsync not working correctly and files
appearing in wrong order on network devices (this is very important for
Lucene). When writing and fsyncing index files, Lucene uses the same
code (sequential writes to segment files, then writing commit file and
finally atomic renaming it to its final location - a 2 phase commit).
This must be fully supported and implemented by the file system also
with making changes visible in the correct order, which is not the case
for NFS (due to stale caches), but *should* work with the recent CIFS
spec, but Samba may not have implemented it correctly in Linux kernel.

This is the reason for the recommendation to not use network file
systems. You're example just confirmed this recommendation.

Uwe

Am 02.01.2023 um 19:24 schrieb S S:
> Hi Uwe,
>
> I will report the bug to ES, as you suggested.
>
> Do you recon using Mmap would have an effect to the index corruption when using SMB? I have to report back to my manager in few days to decide wether to carry on with ACIs or find another hosting solution. It is unfortunate there seems to be problems with this solution. Microsoft seems not interested in extending the volume mapping options for ACIs and K8 is overkilling for our use case.
>
> Thank you for your help so far, you have been very kind :)
>
> Cheers,
>
> Seb
>
>> On 2 Jan 2023, at 19:09, Uwe Schindler <uwe@thetaphi.de> wrote:
>>
>> Hi,
>>
>> Please open a bug report at ES. The setting vm.max_map_count is not needed and should not be changed unless really needed, because it uses kernel resources.
>>
>> This has to do with their support (they try to tell people to overshard and to prevent support requests they ask to raise this seeting). The default value on Linux is 65530. This would allow you to memory map 65530 chunks of 1 GiB (thats the limitation without preview enabled in the Java 19 JVM). A shard in Lucene has about 70 files, most of them with filesizes < 1 Gib and a few >1 GiB (but < 5 GiB) chunks (so lets assume 90 mappings needed for a shard), so you could have approx 728 shards per node. Sorry raising this setting is not needed!
>>
>> Uwe
>>
>> Am 02.01.2023 um 18:24 schrieb S S:
>>> I also tried enabling preview but no joy, same error :(
>>>
>>> It looks like it is not possible to start a multinode ES cluster without setting vm.max_map_count. I also googled it and this check cannot be disabled.
>>>
>>> I guess MMapDirectory is not an option for ES on ACIs, unless you have something else I can try?
>>>
>>> Many thanks,
>>>
>>> Seb
>>>
>>>> On 2 Jan 2023, at 17:55, S S <sebastiano1972@yahoo.it.INVALID> wrote:
>>>>
>>>> Thank you Uwe, this is great! I am rebuilding the cluster using MMapDirectory and no enable-preview, as you suggested. Let’s see what happens.
>>>>
>>>> Cheers,
>>>>
>>>> Seb
>>>>
>>>>> On 2 Jan 2023, at 17:51, Uwe Schindler <uwe@thetaphi.de> wrote:
>>>>>
>>>>> Hi,
>>>>>
>>>>> in recent versions it works like that:
>>>>>
>>>>> https://www.elastic.co/guide/en/elasticsearch/reference/current/advanced-configuration.html#set-jvm-options
>>>>>
>>>>> So in folder jvm.options.d/ add a new file (like "preview.conf") and put "19:--enable-preview" into it. It is basically the same like modifying heap size.
>>>>>
>>>>> But in general, you can simply use MMapDirectory, the max-map-count setting is only relevant in *huge* (huge means hundreds of huge indexes per node). In that case Java 19's preview features would be recommended.
>>>>>
>>>>> Uwe
>>>>>
>>>>> Am 02.01.2023 um 17:41 schrieb S S:
>>>>>> Hi Uwe,
>>>>>>
>>>>>> Sorry for the late reply but upgrading the docker image to use OpenJDK was easier said that done.
>>>>>>
>>>>>> I am not a Java developer/expert so, sorry for the stupid question but, how do I specify the --enable-preview flag? ES has got a quite complex way to start so I cannot specify the flag on the command line. You suggested to use a jvm.properties file but I cannot find anything useful about it when google-ing. Where should it be placed? And what should I write in it?
>>>>>>
>>>>>> I can see ES recognising OpenJDK 19 while bootstrapping and suggesting to enable preview, but it does not suggest how, and I cannot find anything on the ES website.
>>>>>>
>>>>>> Many thanks.
>>>>>>
>>>>>> Seb
>>>>>>
>>>>>>> On 2 Jan 2023, at 11:48, Uwe Schindler <uwe@thetaphi.de> wrote:
>>>>>>>
>>>>>>> Hi,
>>>>>>>
>>>>>>> in general you can still use MMapDirectory. There is no requirement to set vm.max_map_count for smaller clusters. The information in Elastics documentation is not mandatory and misleading.
>>>>>>>
>>>>>>> If you use newest version of Elasticsearch with Java 19 and you use `--enable-preview` in you jvm.properties file, you don't even need to change that setting even with larger clusters.
>>>>>>>
>>>>>>> Uwe
>>>>>>>
>>>>>>> Am 02.01.2023 um 11:18 schrieb S S:
>>>>>>>> We are experimenting with Elastic Search deployed in Azure Container Instances (Debian + OpenJDK). The ES indexes are stored into an Azure file share mounted via SMB (3.0). The Elastic Search cluster is made up of 4 nodes, each one have a separate file share to store the indices.
>>>>>>>>
>>>>>>>> This configuration has been influenced by some ACIs limitations, specifically:
>>>>>>>>
>>>>>>>> we cannot set the max_map_count value as we do not have access to the underlying host (https://www.elastic.co/guide/en/elasticsearch/reference/current/vm-max-map-count.html). Unfortunately, this is required to run an ES cluster, therefore we were forced to use NIOF
>>>>>>>> ACI’s storage is ephemera, therefore we had to map volumes to persist the indexes. ACIs only allow volume mappings using Azure File Shares, which only works with NFS or SMB.
>>>>>>>>
>>>>>>>> We are experiencing recurring index corruption, specifically a "read past EOF" exception. I asked on the Elastic Search forum but the answer I got was a bit generic and not really helpful other than confirming that, from ES point of view, ES should work on an SMB share as long as it behaves as a local drive. As the underlying exception relates to an issue with a Lucene index, I was wondering if you could help out? Specifically, can Lucene work on SMB? I can only find sparse information on this configuration and, while NFS seems a no-no, for SMB is not that clear. Below is the exception we are getting.
>>>>>>>>
>>>>>>>> java.io.IOException: read past EOF: NIOFSIndexInput(path="/bitnami/elasticsearch/data/indices/mS2bUbLtSeG0FSAMuKX7JQ/0/index/_ldsn_1.fnm") buffer: java.nio.HeapByteBuffer[pos=0 lim=1024 cap=1024] chunkLen: 1024 end: 2331: NIOFSIndexInput(path="/bitnami/elasticsearch/data/indices/mS2bUbLtSeG0FSAMuKX7JQ/0/index/_ldsn_1.fnm")
>>>>>>>> at org.apache.lucene.store.NIOFSDirectory$NIOFSIndexInput.readInternal(NIOFSDirectory.java:200) ~[lucene-core-9.3.0.jar:?]
>>>>>>>> at org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.java:291) ~[lucene-core-9.3.0.jar:?]
>>>>>>>> at org.apache.lucene.store.BufferedIndexInput.readByte(BufferedIndexInput.java:55) ~[lucene-core-9.3.0.jar:?]
>>>>>>>> at org.apache.lucene.store.BufferedChecksumIndexInput.readByte(BufferedChecksumIndexInput.java:39) ~[lucene-core-9.3.0.jar:?]
>>>>>>>> at org.apache.lucene.codecs.CodecUtil.readBEInt(CodecUtil.java:667) ~[lucene-core-9.3.0.jar:?]
>>>>>>>> at org.apache.lucene.codecs.CodecUtil.checkHeader(CodecUtil.java:184) ~[lucene-core-9.3.0.jar:?]
>>>>>>>> at org.apache.lucene.codecs.CodecUtil.checkIndexHeader(CodecUtil.java:253) ~[lucene-core-9.3.0.jar:?]
>>>>>>>> at org.apache.lucene.codecs.lucene90.Lucene90FieldInfosFormat.read(Lucene90FieldInfosFormat.java:128) ~[lucene-core-9.3.0.jar:?]
>>>>>>>> at org.apache.lucene.index.SegmentReader.initFieldInfos(SegmentReader.java:205) ~[lucene-core-9.3.0.jar:?]
>>>>>>>> at org.apache.lucene.index.SegmentReader.<init>(SegmentReader.java:156) ~[lucene-core-9.3.0.jar:?]
>>>>>>>> at org.apache.lucene.index.ReadersAndUpdates.createNewReaderWithLatestLiveDocs(ReadersAndUpdates.java:738) ~[lucene-core-9.3.0.jar:?]
>>>>>>>> at org.apache.lucene.index.ReadersAndUpdates.swapNewReaderWithLatestLiveDocs(ReadersAndUpdates.java:754) ~[lucene-core-9.3.0.jar:?]
>>>>>>>> at org.apache.lucene.index.ReadersAndUpdates.writeFieldUpdates(ReadersAndUpdates.java:678) ~[lucene-core-9.3.0.jar:?]
>>>>>>>> at org.apache.lucene.index.ReaderPool.writeAllDocValuesUpdates(ReaderPool.java:251) ~[lucene-core-9.3.0.jar:?]
>>>>>>>> at org.apache.lucene.index.IndexWriter.writeReaderPool(IndexWriter.java:3743) ~[lucene-core-9.3.0.jar:?]
>>>>>>>> at org.apache.lucene.index.IndexWriter.getReader(IndexWriter.java:591) ~[lucene-core-9.3.0.jar:?]
>>>>>>>> at org.apache.lucene.index.StandardDirectoryReader.doOpenFromWriter(StandardDirectoryReader.java:381) ~[lucene-core-9.3.0.jar:?]
>>>>>>>> at org.apache.lucene.index.StandardDirectoryReader.doOpenIfChanged(StandardDirectoryReader.java:355) ~[lucene-core-9.3.0.jar:?]
>>>>>>>> at org.apache.lucene.index.StandardDirectoryReader.doOpenIfChanged(StandardDirectoryReader.java:345) ~[lucene-core-9.3.0.jar:?]
>>>>>>>> at org.apache.lucene.index.FilterDirectoryReader.doOpenIfChanged(FilterDirectoryReader.java:112) ~[lucene-core-9.3.0.jar:?]
>>>>>>>> at org.apache.lucene.index.DirectoryReader.openIfChanged(DirectoryReader.java:170) ~[lucene-core-9.3.0.jar:?]
>>>>>>>> at org.elasticsearch.index.engine.ElasticsearchReaderManager.refreshIfNeeded(ElasticsearchReaderManager.java:48) ~[elasticsearch-8.4.1.jar:?]
>>>>>>>> at org.elasticsearch.index.engine.ElasticsearchReaderManager.refreshIfNeeded(ElasticsearchReaderManager.java:27) ~[elasticsearch-8.4.1.jar:?]
>>>>>>>> at org.apache.lucene.search.ReferenceManager.doMaybeRefresh(ReferenceManager.java:167) ~[lucene-core-9.3.0.jar:?]
>>>>>>>> at org.apache.lucene.search.ReferenceManager.maybeRefreshBlocking(ReferenceManager.java:240) ~[lucene-core-9.3.0.jar:?]
>>>>>>>> at org.elasticsearch.index.engine.InternalEngine$ExternalReaderManager.refreshIfNeeded(InternalEngine.java:355) ~[elasticsearch-8.4.1.jar:?]
>>>>>>>> at org.elasticsearch.index.engine.InternalEngine$ExternalReaderManager.refreshIfNeeded(InternalEngine.java:335) ~[elasticsearch-8.4.1.jar:?]
>>>>>>>> at org.apache.lucene.search.ReferenceManager.doMaybeRefresh(ReferenceManager.java:167) ~[lucene-core-9.3.0.jar:?]
>>>>>>>> Many thanks.
>>>>>>>>
>>>>>>>> Seb
>>>>>>> --
>>>>>>> Uwe Schindler
>>>>>>> Achterdiek 19, D-28357 Bremen
>>>>>>> https://www.thetaphi.de
>>>>>>> eMail: uwe@thetaphi.de
>>>>>>>
>>>>>>>
>>>>>>> ---------------------------------------------------------------------
>>>>>>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>>>>>>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>>>>>>
>>>>>> ---------------------------------------------------------------------
>>>>>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>>>>>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>>>>>
>>>>> --
>>>>> Uwe Schindler
>>>>> Achterdiek 19, D-28357 Bremen
>>>>> https://www.thetaphi.de
>>>>> eMail: uwe@thetaphi.de
>>>>>
>>>>>
>>>>> ---------------------------------------------------------------------
>>>>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>>>>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>>>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>>
>> --
>> Uwe Schindler
>> Achterdiek 19, D-28357 Bremen
>> https://www.thetaphi.de
>> eMail: uwe@thetaphi.de
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
--
Uwe Schindler
Achterdiek 19, D-28357 Bremen
https://www.thetaphi.de
eMail: uwe@thetaphi.de


---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Recurring index corruption [ In reply to ]
Thank you Uwe,

This is the answer I was looking for. A bit unfortunate this is happening but, it is what it is. At least, when Google will index this discussion, this will be available and informative for anyone attempting using the same solution.

Many thanks for your help and for your explanation.

Best regards,

Seb

> On 2 Jan 2023, at 20:02, Uwe Schindler <uwe@thetaphi.de> wrote:
>
> Hi,
>
> Yeah, sorry for leading the issue into the wrong direction. I was just stumbling on the exception message and because we do not spend much time in improving/supporting the use of NIOFSDirectory, I may have moved this mailing list thread into the wrong direction.
>
> I don't think the directory implementation will change much when using Samba. Recent versions of the CIFS specification (the spec behind Samba) and the implementation in Windows Server are fine. But I doubt that recent implementations in Linux kernel are correct and fully support the spec.
>
> The corrumption issue you see is just detected by NIOFSDirectory (which I stumbled upon), but with MMapDirectory it will happen in the same way as the problem is caused by fsync not working correctly and files appearing in wrong order on network devices (this is very important for Lucene). When writing and fsyncing index files, Lucene uses the same code (sequential writes to segment files, then writing commit file and finally atomic renaming it to its final location - a 2 phase commit). This must be fully supported and implemented by the file system also with making changes visible in the correct order, which is not the case for NFS (due to stale caches), but *should* work with the recent CIFS spec, but Samba may not have implemented it correctly in Linux kernel.
>
> This is the reason for the recommendation to not use network file systems. You're example just confirmed this recommendation.
>
> Uwe
>
> Am 02.01.2023 um 19:24 schrieb S S:
>> Hi Uwe,
>>
>> I will report the bug to ES, as you suggested.
>>
>> Do you recon using Mmap would have an effect to the index corruption when using SMB? I have to report back to my manager in few days to decide wether to carry on with ACIs or find another hosting solution. It is unfortunate there seems to be problems with this solution. Microsoft seems not interested in extending the volume mapping options for ACIs and K8 is overkilling for our use case.
>>
>> Thank you for your help so far, you have been very kind :)
>>
>> Cheers,
>>
>> Seb
>>
>>> On 2 Jan 2023, at 19:09, Uwe Schindler <uwe@thetaphi.de> wrote:
>>>
>>> Hi,
>>>
>>> Please open a bug report at ES. The setting vm.max_map_count is not needed and should not be changed unless really needed, because it uses kernel resources.
>>>
>>> This has to do with their support (they try to tell people to overshard and to prevent support requests they ask to raise this seeting). The default value on Linux is 65530. This would allow you to memory map 65530 chunks of 1 GiB (thats the limitation without preview enabled in the Java 19 JVM). A shard in Lucene has about 70 files, most of them with filesizes < 1 Gib and a few >1 GiB (but < 5 GiB) chunks (so lets assume 90 mappings needed for a shard), so you could have approx 728 shards per node. Sorry raising this setting is not needed!
>>>
>>> Uwe
>>>
>>> Am 02.01.2023 um 18:24 schrieb S S:
>>>> I also tried enabling preview but no joy, same error :(
>>>>
>>>> It looks like it is not possible to start a multinode ES cluster without setting vm.max_map_count. I also googled it and this check cannot be disabled.
>>>>
>>>> I guess MMapDirectory is not an option for ES on ACIs, unless you have something else I can try?
>>>>
>>>> Many thanks,
>>>>
>>>> Seb
>>>>
>>>>> On 2 Jan 2023, at 17:55, S S <sebastiano1972@yahoo.it.INVALID> wrote:
>>>>>
>>>>> Thank you Uwe, this is great! I am rebuilding the cluster using MMapDirectory and no enable-preview, as you suggested. Let’s see what happens.
>>>>>
>>>>> Cheers,
>>>>>
>>>>> Seb
>>>>>
>>>>>> On 2 Jan 2023, at 17:51, Uwe Schindler <uwe@thetaphi.de> wrote:
>>>>>>
>>>>>> Hi,
>>>>>>
>>>>>> in recent versions it works like that:
>>>>>>
>>>>>> https://www.elastic.co/guide/en/elasticsearch/reference/current/advanced-configuration.html#set-jvm-options
>>>>>>
>>>>>> So in folder jvm.options.d/ add a new file (like "preview.conf") and put "19:--enable-preview" into it. It is basically the same like modifying heap size.
>>>>>>
>>>>>> But in general, you can simply use MMapDirectory, the max-map-count setting is only relevant in *huge* (huge means hundreds of huge indexes per node). In that case Java 19's preview features would be recommended.
>>>>>>
>>>>>> Uwe
>>>>>>
>>>>>> Am 02.01.2023 um 17:41 schrieb S S:
>>>>>>> Hi Uwe,
>>>>>>>
>>>>>>> Sorry for the late reply but upgrading the docker image to use OpenJDK was easier said that done.
>>>>>>>
>>>>>>> I am not a Java developer/expert so, sorry for the stupid question but, how do I specify the --enable-preview flag? ES has got a quite complex way to start so I cannot specify the flag on the command line. You suggested to use a jvm.properties file but I cannot find anything useful about it when google-ing. Where should it be placed? And what should I write in it?
>>>>>>>
>>>>>>> I can see ES recognising OpenJDK 19 while bootstrapping and suggesting to enable preview, but it does not suggest how, and I cannot find anything on the ES website.
>>>>>>>
>>>>>>> Many thanks.
>>>>>>>
>>>>>>> Seb
>>>>>>>
>>>>>>>> On 2 Jan 2023, at 11:48, Uwe Schindler <uwe@thetaphi.de> wrote:
>>>>>>>>
>>>>>>>> Hi,
>>>>>>>>
>>>>>>>> in general you can still use MMapDirectory. There is no requirement to set vm.max_map_count for smaller clusters. The information in Elastics documentation is not mandatory and misleading.
>>>>>>>>
>>>>>>>> If you use newest version of Elasticsearch with Java 19 and you use `--enable-preview` in you jvm.properties file, you don't even need to change that setting even with larger clusters.
>>>>>>>>
>>>>>>>> Uwe
>>>>>>>>
>>>>>>>> Am 02.01.2023 um 11:18 schrieb S S:
>>>>>>>>> We are experimenting with Elastic Search deployed in Azure Container Instances (Debian + OpenJDK). The ES indexes are stored into an Azure file share mounted via SMB (3.0). The Elastic Search cluster is made up of 4 nodes, each one have a separate file share to store the indices.
>>>>>>>>>
>>>>>>>>> This configuration has been influenced by some ACIs limitations, specifically:
>>>>>>>>>
>>>>>>>>> we cannot set the max_map_count value as we do not have access to the underlying host (https://www.elastic.co/guide/en/elasticsearch/reference/current/vm-max-map-count.html). Unfortunately, this is required to run an ES cluster, therefore we were forced to use NIOF
>>>>>>>>> ACI’s storage is ephemera, therefore we had to map volumes to persist the indexes. ACIs only allow volume mappings using Azure File Shares, which only works with NFS or SMB.
>>>>>>>>>
>>>>>>>>> We are experiencing recurring index corruption, specifically a "read past EOF" exception. I asked on the Elastic Search forum but the answer I got was a bit generic and not really helpful other than confirming that, from ES point of view, ES should work on an SMB share as long as it behaves as a local drive. As the underlying exception relates to an issue with a Lucene index, I was wondering if you could help out? Specifically, can Lucene work on SMB? I can only find sparse information on this configuration and, while NFS seems a no-no, for SMB is not that clear. Below is the exception we are getting.
>>>>>>>>>
>>>>>>>>> java.io.IOException: read past EOF: NIOFSIndexInput(path="/bitnami/elasticsearch/data/indices/mS2bUbLtSeG0FSAMuKX7JQ/0/index/_ldsn_1.fnm") buffer: java.nio.HeapByteBuffer[pos=0 lim=1024 cap=1024] chunkLen: 1024 end: 2331: NIOFSIndexInput(path="/bitnami/elasticsearch/data/indices/mS2bUbLtSeG0FSAMuKX7JQ/0/index/_ldsn_1.fnm")
>>>>>>>>> at org.apache.lucene.store.NIOFSDirectory$NIOFSIndexInput.readInternal(NIOFSDirectory.java:200) ~[lucene-core-9.3.0.jar:?]
>>>>>>>>> at org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.java:291) ~[lucene-core-9.3.0.jar:?]
>>>>>>>>> at org.apache.lucene.store.BufferedIndexInput.readByte(BufferedIndexInput.java:55) ~[lucene-core-9.3.0.jar:?]
>>>>>>>>> at org.apache.lucene.store.BufferedChecksumIndexInput.readByte(BufferedChecksumIndexInput.java:39) ~[lucene-core-9.3.0.jar:?]
>>>>>>>>> at org.apache.lucene.codecs.CodecUtil.readBEInt(CodecUtil.java:667) ~[lucene-core-9.3.0.jar:?]
>>>>>>>>> at org.apache.lucene.codecs.CodecUtil.checkHeader(CodecUtil.java:184) ~[lucene-core-9.3.0.jar:?]
>>>>>>>>> at org.apache.lucene.codecs.CodecUtil.checkIndexHeader(CodecUtil.java:253) ~[lucene-core-9.3.0.jar:?]
>>>>>>>>> at org.apache.lucene.codecs.lucene90.Lucene90FieldInfosFormat.read(Lucene90FieldInfosFormat.java:128) ~[lucene-core-9.3.0.jar:?]
>>>>>>>>> at org.apache.lucene.index.SegmentReader.initFieldInfos(SegmentReader.java:205) ~[lucene-core-9.3.0.jar:?]
>>>>>>>>> at org.apache.lucene.index.SegmentReader.<init>(SegmentReader.java:156) ~[lucene-core-9.3.0.jar:?]
>>>>>>>>> at org.apache.lucene.index.ReadersAndUpdates.createNewReaderWithLatestLiveDocs(ReadersAndUpdates.java:738) ~[lucene-core-9.3.0.jar:?]
>>>>>>>>> at org.apache.lucene.index.ReadersAndUpdates.swapNewReaderWithLatestLiveDocs(ReadersAndUpdates.java:754) ~[lucene-core-9.3.0.jar:?]
>>>>>>>>> at org.apache.lucene.index.ReadersAndUpdates.writeFieldUpdates(ReadersAndUpdates.java:678) ~[lucene-core-9.3.0.jar:?]
>>>>>>>>> at org.apache.lucene.index.ReaderPool.writeAllDocValuesUpdates(ReaderPool.java:251) ~[lucene-core-9.3.0.jar:?]
>>>>>>>>> at org.apache.lucene.index.IndexWriter.writeReaderPool(IndexWriter.java:3743) ~[lucene-core-9.3.0.jar:?]
>>>>>>>>> at org.apache.lucene.index.IndexWriter.getReader(IndexWriter.java:591) ~[lucene-core-9.3.0.jar:?]
>>>>>>>>> at org.apache.lucene.index.StandardDirectoryReader.doOpenFromWriter(StandardDirectoryReader.java:381) ~[lucene-core-9.3.0.jar:?]
>>>>>>>>> at org.apache.lucene.index.StandardDirectoryReader.doOpenIfChanged(StandardDirectoryReader.java:355) ~[lucene-core-9.3.0.jar:?]
>>>>>>>>> at org.apache.lucene.index.StandardDirectoryReader.doOpenIfChanged(StandardDirectoryReader.java:345) ~[lucene-core-9.3.0.jar:?]
>>>>>>>>> at org.apache.lucene.index.FilterDirectoryReader.doOpenIfChanged(FilterDirectoryReader.java:112) ~[lucene-core-9.3.0.jar:?]
>>>>>>>>> at org.apache.lucene.index.DirectoryReader.openIfChanged(DirectoryReader.java:170) ~[lucene-core-9.3.0.jar:?]
>>>>>>>>> at org.elasticsearch.index.engine.ElasticsearchReaderManager.refreshIfNeeded(ElasticsearchReaderManager.java:48) ~[elasticsearch-8.4.1.jar:?]
>>>>>>>>> at org.elasticsearch.index.engine.ElasticsearchReaderManager.refreshIfNeeded(ElasticsearchReaderManager.java:27) ~[elasticsearch-8.4.1.jar:?]
>>>>>>>>> at org.apache.lucene.search.ReferenceManager.doMaybeRefresh(ReferenceManager.java:167) ~[lucene-core-9.3.0.jar:?]
>>>>>>>>> at org.apache.lucene.search.ReferenceManager.maybeRefreshBlocking(ReferenceManager.java:240) ~[lucene-core-9.3.0.jar:?]
>>>>>>>>> at org.elasticsearch.index.engine.InternalEngine$ExternalReaderManager.refreshIfNeeded(InternalEngine.java:355) ~[elasticsearch-8.4.1.jar:?]
>>>>>>>>> at org.elasticsearch.index.engine.InternalEngine$ExternalReaderManager.refreshIfNeeded(InternalEngine.java:335) ~[elasticsearch-8.4.1.jar:?]
>>>>>>>>> at org.apache.lucene.search.ReferenceManager.doMaybeRefresh(ReferenceManager.java:167) ~[lucene-core-9.3.0.jar:?]
>>>>>>>>> Many thanks.
>>>>>>>>>
>>>>>>>>> Seb
>>>>>>>> --
>>>>>>>> Uwe Schindler
>>>>>>>> Achterdiek 19, D-28357 Bremen
>>>>>>>> https://www.thetaphi.de
>>>>>>>> eMail: uwe@thetaphi.de
>>>>>>>>
>>>>>>>>
>>>>>>>> ---------------------------------------------------------------------
>>>>>>>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>>>>>>>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>>>>>>>
>>>>>>> ---------------------------------------------------------------------
>>>>>>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>>>>>>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>>>>>>
>>>>>> --
>>>>>> Uwe Schindler
>>>>>> Achterdiek 19, D-28357 Bremen
>>>>>> https://www.thetaphi.de
>>>>>> eMail: uwe@thetaphi.de
>>>>>>
>>>>>>
>>>>>> ---------------------------------------------------------------------
>>>>>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>>>>>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>>>>>
>>>>> ---------------------------------------------------------------------
>>>>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>>>>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>>>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>>>
>>> --
>>> Uwe Schindler
>>> Achterdiek 19, D-28357 Bremen
>>> https://www.thetaphi.de
>>> eMail: uwe@thetaphi.de
>>>
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>
> --
> Uwe Schindler
> Achterdiek 19, D-28357 Bremen
> https://www.thetaphi.de
> eMail: uwe@thetaphi.de
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>


---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org