Mailing List Archive

Lucene indexes corrupted : Missing .si file
Hello Fellas,

I am in dire need of your expertise. We have a java process that saves its
logs in Lucene indexes. We re-start this process every midnight. While
starting, it initializes IndexWriter. We recently had an incident in our
production environment when we tried to restart our java process and it
failed to start, giving the error:

Unexpected file read error while reading index.
> (resource=BufferedChecksumIndexInput(MMapIndexInput(path="/home/maps/appli/expl/lst/sch/Application/prod/segments_2lpu")))
> java.lang.RuntimeException: Error attempting to initialize index
> /home/maps/appli/expl/lst/sch/Application/prod:
> Unexpected file read error while reading index.
> (resource=BufferedChecksumIndexInput(MMapIndexInput(path="/home/maps/appli/expl/lst/sch/Application/prod/segments_2lpu")))
> at
> ...
> ...
> Caused by: org.apache.lucene.index.CorruptIndexException: Unexpected file
> read error while reading index.
>
> (resource=BufferedChecksumIndexInput(MMapIndexInput(path="/home/maps/appli/expl/lst/sch/Application/prod/segments_2lpu")))
> at org.apache.lucene.index.SegmentInfos.readCommit(SegmentInfos.java:291)
> ...
> ...
> Caused by: java.nio.file.NoSuchFileException:
> /home/maps/appli/expl/lst/sch/Application/prod/_dko.si at
> sun.nio.fs.UnixException.translateToIOException(UnixException.java:86)
> ~[?:1.8.0_282] at
> sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
> ~[?:1.8.0_282] at
> sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107)
> ~[?:1.8.0_282] at
> sun.nio.fs.UnixFileSystemProvider.newFileChannel(UnixFileSystemProvider.java:177)
> ~
> [?:1.8.0_282] at java.nio.channels.FileChannel.open(FileChannel.java:287)
> ~[?:1.8.0_282] at
> java.nio.channels.FileChannel.open(FileChannel.java:335) ~[?:1.8.0_282] at
> org.apache.lucene.store.MMapDirectory.openInput(MMapDirectory.java:238)
> ~[lucene-core-
> 8.5.1.jar:8.5.1 edb9fc409398f2c3446883f9f80595c884d245d0 - ivera -
> 2020-04-08 08:55:42] at
> org.apache.lucene.store.Directory.openChecksumInput(Directory.java:157)
> ~[lucene-core-
> 8.5.1.jar:8.5.1 edb9fc409398f2c3446883f9f80595c884d245d0 - ivera -
> 2020-04-08 08:55:42] at
> org.apache.lucene.codecs.lucene70.Lucene70SegmentInfoFormat.read(Lucene70SegmentInfoFormat.java:9
> ... 4 more


We tried to run the CheckIndex command to repair the index, but got the
following error while doing so:

$ java -cp lucene-core-8.5.1.jar -ea:org.apache.lucene...
> org.apache.lucene.index.CheckIndex prod/ -exorcise
>
> Opening index @ prod/
>
> ERROR: could not read any segments file in directory
> org.apache.lucene.index.CorruptIndexException: Unexpected file read error
> while reading index.
> (resource=BufferedChecksumIndexInput(MMapIndexInput(path="/home/maps/appli/expl/lst/sch/Application/prod/segments_2lpu")))
> at
> org.apache.lucene.index.SegmentInfos.readCommit(SegmentInfos.java:291)
> at org.apache.lucene.index.CheckIndex.checkIndex(CheckIndex.java:524)
> at org.apache.lucene.index.CheckIndex.doCheck(CheckIndex.java:2973)
> at org.apache.lucene.index.CheckIndex.doMain(CheckIndex.java:2871)
> at org.apache.lucene.index.CheckIndex.main(CheckIndex.java:2797)
> Caused by: java.nio.file.NoSuchFileException:
> /home/maps/appli/expl/lst/sch/Application/prod/_ogc.si
> at
> sun.nio.fs.UnixException.translateToIOException(UnixException.java:86)
> at
> sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
> at
> sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107)
> at
> sun.nio.fs.UnixFileSystemProvider.newFileChannel(UnixFileSystemProvider.java:177)
> at java.nio.channels.FileChannel.open(FileChannel.java:287)
> at java.nio.channels.FileChannel.open(FileChannel.java:335)
> at
> org.apache.lucene.store.MMapDirectory.openInput(MMapDirectory.java:238)
> at
> org.apache.lucene.store.Directory.openChecksumInput(Directory.java:157)
> at
> org.apache.lucene.codecs.lucene70.Lucene70SegmentInfoFormat.read(Lucene70SegmentInfoFormat.java:91)
> at
> org.apache.lucene.index.SegmentInfos.readCommit(SegmentInfos.java:353)
> at
> org.apache.lucene.index.SegmentInfos.readCommit(SegmentInfos.java:289)
> ... 4 more


For storage of our Lucene indexes, we're using an XFS volume in a SAN
(Storage Area Network) (NetApp, full flash, redundant FO connectivity).

*OS*: RHEL7

*Lucene version*: 8.5.1

We're not sure how the Lucene indexes got corrupted. Can some expert give
us any lead about why this could have happened? In what circumstances can
.si files get missing from indexes? We need to know this information so
that we can prevent any further occurrence of this issue in our production
environment.

Thanks so much for your help!

Bhavit