Mailing List Archive

Uwe, it looks a little crazy:

we've got asserts here that "index" is in bounds and certainly not -1
right before the method call!
https://github.com/apache/lucene/blob/main/lucene/core/src/java/org/apache/lucene/codecs/lucene90/Lucene90DocValuesProducer.java#L1123-L1125

On Wed, Dec 8, 2021 at 12:33 PM Uwe Schindler <uwe@thetaphi.de> wrote:
>
> I updated the dependencies of Solr's main branch to Lucene 9.0
>
> On Mac and Linux, the following error occurs sometimes:
> https://jenkins.thetaphi.de/job/Solr-main-Linux/2070/testReport/junit/org.ap
> ache.solr.request/SimpleFacetsTest/testRangeFacetFilterVsDocValuesRandom/
> https://jenkins.thetaphi.de/job/Solr-main-MacOSX/517/testReport/junit/org.ap
> ache.solr/TestRandomDVFaceting/testRandomFaceting/
>
> "ArrayIndexOutOfBoundsException: Index -1 out of bounds for length 1" at
> org.apache.lucene.util.packed.DirectMonotonicReader.get(DirectMonotonicReade
> r.java:161)
>
> This looks like a bug (or is Solr not using something with docvalues
> correctly, but an AIOOBE should not happen)!
>
> When searching through the failure mails, it looks like this was already
> there with the Lucene preview from summer 2021.
>
> Uwe
>
> -----
> Uwe Schindler
> Achterdiek 19, D-28357 Bremen
> https://www.thetaphi.de
> eMail: uwe@thetaphi.de
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: dev-help@lucene.apache.org
>

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org

I checked out the commit hash and re-ran the test with the same seed,
it doesn't fail. I only ran the test twice, once with
TieredStopAtLevel=1 and once without that.

Given that the stacktrace looks "impossible", i suspect this might be
an openjdk issue? I assume this test usually reproduces failures for
given random seeds?

On Wed, Dec 8, 2021 at 12:44 PM Robert Muir <rcmuir@gmail.com> wrote:
>
> Uwe, it looks a little crazy:
>
> we've got asserts here that "index" is in bounds and certainly not -1
> right before the method call!
> https://github.com/apache/lucene/blob/main/lucene/core/src/java/org/apache/lucene/codecs/lucene90/Lucene90DocValuesProducer.java#L1123-L1125
>
> On Wed, Dec 8, 2021 at 12:33 PM Uwe Schindler <uwe@thetaphi.de> wrote:
> >
> > I updated the dependencies of Solr's main branch to Lucene 9.0
> >
> > On Mac and Linux, the following error occurs sometimes:
> > https://jenkins.thetaphi.de/job/Solr-main-Linux/2070/testReport/junit/org.ap
> > ache.solr.request/SimpleFacetsTest/testRangeFacetFilterVsDocValuesRandom/
> > https://jenkins.thetaphi.de/job/Solr-main-MacOSX/517/testReport/junit/org.ap
> > ache.solr/TestRandomDVFaceting/testRandomFaceting/
> >
> > "ArrayIndexOutOfBoundsException: Index -1 out of bounds for length 1" at
> > org.apache.lucene.util.packed.DirectMonotonicReader.get(DirectMonotonicReade
> > r.java:161)
> >
> > This looks like a bug (or is Solr not using something with docvalues
> > correctly, but an AIOOBE should not happen)!
> >
> > When searching through the failure mails, it looks like this was already
> > there with the Lucene preview from summer 2021.
> >
> > Uwe
> >
> > -----
> > Uwe Schindler
> > Achterdiek 19, D-28357 Bremen
> > https://www.thetaphi.de
> > eMail: uwe@thetaphi.de
> >
> >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
> > For additional commands, e-mail: dev-help@lucene.apache.org
> >

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org

I also generally tried to "beast" this
SimpleFacetsTest.testRangeFacetFilterVsDocValuesRandom without any
seed at all (JDK 17/linux), but I didn't manage to provoke any
failures.

On Wed, Dec 8, 2021 at 1:08 PM Robert Muir <rcmuir@gmail.com> wrote:
>
> I checked out the commit hash and re-ran the test with the same seed,
> it doesn't fail. I only ran the test twice, once with
> TieredStopAtLevel=1 and once without that.
>
> Given that the stacktrace looks "impossible", i suspect this might be
> an openjdk issue? I assume this test usually reproduces failures for
> given random seeds?
>
> On Wed, Dec 8, 2021 at 12:44 PM Robert Muir <rcmuir@gmail.com> wrote:
> >
> > Uwe, it looks a little crazy:
> >
> > we've got asserts here that "index" is in bounds and certainly not -1
> > right before the method call!
> > https://github.com/apache/lucene/blob/main/lucene/core/src/java/org/apache/lucene/codecs/lucene90/Lucene90DocValuesProducer.java#L1123-L1125
> >
> > On Wed, Dec 8, 2021 at 12:33 PM Uwe Schindler <uwe@thetaphi.de> wrote:
> > >
> > > I updated the dependencies of Solr's main branch to Lucene 9.0
> > >
> > > On Mac and Linux, the following error occurs sometimes:
> > > https://jenkins.thetaphi.de/job/Solr-main-Linux/2070/testReport/junit/org.ap
> > > ache.solr.request/SimpleFacetsTest/testRangeFacetFilterVsDocValuesRandom/
> > > https://jenkins.thetaphi.de/job/Solr-main-MacOSX/517/testReport/junit/org.ap
> > > ache.solr/TestRandomDVFaceting/testRandomFaceting/
> > >
> > > "ArrayIndexOutOfBoundsException: Index -1 out of bounds for length 1" at
> > > org.apache.lucene.util.packed.DirectMonotonicReader.get(DirectMonotonicReade
> > > r.java:161)
> > >
> > > This looks like a bug (or is Solr not using something with docvalues
> > > correctly, but an AIOOBE should not happen)!
> > >
> > > When searching through the failure mails, it looks like this was already
> > > there with the Lucene preview from summer 2021.
> > >
> > > Uwe
> > >
> > > -----
> > > Uwe Schindler
> > > Achterdiek 19, D-28357 Bremen
> > > https://www.thetaphi.de
> > > eMail: uwe@thetaphi.de
> > >
> > >
> > >
> > > ---------------------------------------------------------------------
> > > To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
> > > For additional commands, e-mail: dev-help@lucene.apache.org
> > >

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org

I was just able to reproduce this on my local macOS machine with

gradlew test --tests TestRandomDVFaceting.testRandomFaceting
-Dtests.seed=3B93BA61C91F26D4 -Dtests.slow=true -Dtests.locale=uz-Latn
-Dtests.timezone=America/Santa_Isabel -Dtests.asserts=true
-Dtests.file.encoding=UTF-8

on commit 0287458f836

with openjdk version "11.0.13"

On Wed, Dec 8, 2021 at 12:30 PM Robert Muir <rcmuir@gmail.com> wrote:

> I also generally tried to "beast" this
> SimpleFacetsTest.testRangeFacetFilterVsDocValuesRandom without any
> seed at all (JDK 17/linux), but I didn't manage to provoke any
> failures.
>
> On Wed, Dec 8, 2021 at 1:08 PM Robert Muir <rcmuir@gmail.com> wrote:
> >
> > I checked out the commit hash and re-ran the test with the same seed,
> > it doesn't fail. I only ran the test twice, once with
> > TieredStopAtLevel=1 and once without that.
> >
> > Given that the stacktrace looks "impossible", i suspect this might be
> > an openjdk issue? I assume this test usually reproduces failures for
> > given random seeds?
> >
> > On Wed, Dec 8, 2021 at 12:44 PM Robert Muir <rcmuir@gmail.com> wrote:
> > >
> > > Uwe, it looks a little crazy:
> > >
> > > we've got asserts here that "index" is in bounds and certainly not -1
> > > right before the method call!
> > >
> https://github.com/apache/lucene/blob/main/lucene/core/src/java/org/apache/lucene/codecs/lucene90/Lucene90DocValuesProducer.java#L1123-L1125
> > >
> > > On Wed, Dec 8, 2021 at 12:33 PM Uwe Schindler <uwe@thetaphi.de> wrote:
> > > >
> > > > I updated the dependencies of Solr's main branch to Lucene 9.0
> > > >
> > > > On Mac and Linux, the following error occurs sometimes:
> > > >
> https://jenkins.thetaphi.de/job/Solr-main-Linux/2070/testReport/junit/org.ap
> > > >
> ache.solr.request/SimpleFacetsTest/testRangeFacetFilterVsDocValuesRandom/
> > > >
> https://jenkins.thetaphi.de/job/Solr-main-MacOSX/517/testReport/junit/org.ap
> > > > ache.solr/TestRandomDVFaceting/testRandomFaceting/
> > > >
> > > > "ArrayIndexOutOfBoundsException: Index -1 out of bounds for length
> 1" at
> > > >
> org.apache.lucene.util.packed.DirectMonotonicReader.get(DirectMonotonicReade
> > > > r.java:161)
> > > >
> > > > This looks like a bug (or is Solr not using something with docvalues
> > > > correctly, but an AIOOBE should not happen)!
> > > >
> > > > When searching through the failure mails, it looks like this was
> already
> > > > there with the Lucene preview from summer 2021.
> > > >
> > > > Uwe
> > > >
> > > > -----
> > > > Uwe Schindler
> > > > Achterdiek 19, D-28357 Bremen
> > > > https://www.thetaphi.de
> > > > eMail: uwe@thetaphi.de
> > > >
> > > >
> > > >
> > > > ---------------------------------------------------------------------
> > > > To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
> > > > For additional commands, e-mail: dev-help@lucene.apache.org
> > > >
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: dev-help@lucene.apache.org
>
>

Sorry, meant to add that the reproduction is before the update to lucene
9.0.0 release. But I'm also able to reproduce it after the upgrade, like
Uwe noted.

On Wed, Dec 8, 2021 at 12:53 PM Mike Drob <mdrob@mdrob.com> wrote:

> I was just able to reproduce this on my local macOS machine with
>
> gradlew test --tests TestRandomDVFaceting.testRandomFaceting
> -Dtests.seed=3B93BA61C91F26D4 -Dtests.slow=true -Dtests.locale=uz-Latn
> -Dtests.timezone=America/Santa_Isabel -Dtests.asserts=true
> -Dtests.file.encoding=UTF-8
>
> on commit 0287458f836
>
> with openjdk version "11.0.13"
>
> On Wed, Dec 8, 2021 at 12:30 PM Robert Muir <rcmuir@gmail.com> wrote:
>
>> I also generally tried to "beast" this
>> SimpleFacetsTest.testRangeFacetFilterVsDocValuesRandom without any
>> seed at all (JDK 17/linux), but I didn't manage to provoke any
>> failures.
>>
>> On Wed, Dec 8, 2021 at 1:08 PM Robert Muir <rcmuir@gmail.com> wrote:
>> >
>> > I checked out the commit hash and re-ran the test with the same seed,
>> > it doesn't fail. I only ran the test twice, once with
>> > TieredStopAtLevel=1 and once without that.
>> >
>> > Given that the stacktrace looks "impossible", i suspect this might be
>> > an openjdk issue? I assume this test usually reproduces failures for
>> > given random seeds?
>> >
>> > On Wed, Dec 8, 2021 at 12:44 PM Robert Muir <rcmuir@gmail.com> wrote:
>> > >
>> > > Uwe, it looks a little crazy:
>> > >
>> > > we've got asserts here that "index" is in bounds and certainly not -1
>> > > right before the method call!
>> > >
>> https://github.com/apache/lucene/blob/main/lucene/core/src/java/org/apache/lucene/codecs/lucene90/Lucene90DocValuesProducer.java#L1123-L1125
>> > >
>> > > On Wed, Dec 8, 2021 at 12:33 PM Uwe Schindler <uwe@thetaphi.de>
>> wrote:
>> > > >
>> > > > I updated the dependencies of Solr's main branch to Lucene 9.0
>> > > >
>> > > > On Mac and Linux, the following error occurs sometimes:
>> > > >
>> https://jenkins.thetaphi.de/job/Solr-main-Linux/2070/testReport/junit/org.ap
>> > > >
>> ache.solr.request/SimpleFacetsTest/testRangeFacetFilterVsDocValuesRandom/
>> > > >
>> https://jenkins.thetaphi.de/job/Solr-main-MacOSX/517/testReport/junit/org.ap
>> > > > ache.solr/TestRandomDVFaceting/testRandomFaceting/
>> > > >
>> > > > "ArrayIndexOutOfBoundsException: Index -1 out of bounds for length
>> 1" at
>> > > >
>> org.apache.lucene.util.packed.DirectMonotonicReader.get(DirectMonotonicReade
>> > > > r.java:161)
>> > > >
>> > > > This looks like a bug (or is Solr not using something with docvalues
>> > > > correctly, but an AIOOBE should not happen)!
>> > > >
>> > > > When searching through the failure mails, it looks like this was
>> already
>> > > > there with the Lucene preview from summer 2021.
>> > > >
>> > > > Uwe
>> > > >
>> > > > -----
>> > > > Uwe Schindler
>> > > > Achterdiek 19, D-28357 Bremen
>> > > > https://www.thetaphi.de
>> > > > eMail: uwe@thetaphi.de
>> > > >
>> > > >
>> > > >
>> > > >
>> ---------------------------------------------------------------------
>> > > > To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
>> > > > For additional commands, e-mail: dev-help@lucene.apache.org
>> > > >
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: dev-help@lucene.apache.org
>>
>>

I was able to reproduce the problems with JDK 11.0.13 and also JDK 17:

export TEST_JVM_ARGS=-XX:+UseCompressedOops -XX:+UseG1GC
gradlew :solr:core:test -Ptests.iters=1000 --tests TestRandomDVFaceting.testRandomFaceting -Dtests.seed=3B93BA61C91F26D4 -Dtests.slow=true -Dtests.locale=uz-Latn -Dtests.timezone=America/Santa_Isabel -Dtests.asserts=true -Dtests.file.encoding=UTF-8

This is important to do:
* use -Ptests.iters=1000 to run the same test over and over in same JVM
* use the above TEST_JVM_ARGS to not run it with tiered compilation turned off.

It looks like a JVM issue in all JVMs after 11. So this is a serious bug. In Lucene tests we havent seen this!

Uwe

-----
Uwe Schindler
Achterdiek 19, D-28357 Bremen
https://www.thetaphi.de
eMail: uwe@thetaphi.de

> -----Original Message-----
> From: Robert Muir <rcmuir@gmail.com>
> Sent: Wednesday, December 8, 2021 6:44 PM
> To: dev@lucene.apache.org
> Cc: dev@solr.apache.org
> Subject: Re: Strange error in Solr with Lucene 9.0.0
>
> Uwe, it looks a little crazy:
>
> we've got asserts here that "index" is in bounds and certainly not -1
> right before the method call!
> https://github.com/apache/lucene/blob/main/lucene/core/src/java/org/apache
> /lucene/codecs/lucene90/Lucene90DocValuesProducer.java#L1123-L1125
>
> On Wed, Dec 8, 2021 at 12:33 PM Uwe Schindler <uwe@thetaphi.de> wrote:
> >
> > I updated the dependencies of Solr's main branch to Lucene 9.0
> >
> > On Mac and Linux, the following error occurs sometimes:
> > https://jenkins.thetaphi.de/job/Solr-main-Linux/2070/testReport/junit/org.ap
> >
> ache.solr.request/SimpleFacetsTest/testRangeFacetFilterVsDocValuesRandom/
> > https://jenkins.thetaphi.de/job/Solr-main-
> MacOSX/517/testReport/junit/org.ap
> > ache.solr/TestRandomDVFaceting/testRandomFaceting/
> >
> > "ArrayIndexOutOfBoundsException: Index -1 out of bounds for length 1" at
> >
> org.apache.lucene.util.packed.DirectMonotonicReader.get(DirectMonotonicRea
> de
> > r.java:161)
> >
> > This looks like a bug (or is Solr not using something with docvalues
> > correctly, but an AIOOBE should not happen)!
> >
> > When searching through the failure mails, it looks like this was already
> > there with the Lucene preview from summer 2021.
> >
> > Uwe
> >
> > -----
> > Uwe Schindler
> > Achterdiek 19, D-28357 Bremen
> > https://www.thetaphi.de
> > eMail: uwe@thetaphi.de
> >
> >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
> > For additional commands, e-mail: dev-help@lucene.apache.org
> >
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@solr.apache.org
> For additional commands, e-mail: dev-help@solr.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org

I'm able to reproduce this TestRandomDVFaceting one with Uwe's
instructions. Just had no luck with the SimpleFacets test i was trying
earlier.

On Wed, Dec 8, 2021 at 2:07 PM Uwe Schindler <uwe@thetaphi.de> wrote:
>
> I was able to reproduce the problems with JDK 11.0.13 and also JDK 17:
>
> export TEST_JVM_ARGS=-XX:+UseCompressedOops -XX:+UseG1GC
> gradlew :solr:core:test -Ptests.iters=1000 --tests TestRandomDVFaceting.testRandomFaceting -Dtests.seed=3B93BA61C91F26D4 -Dtests.slow=true -Dtests.locale=uz-Latn -Dtests.timezone=America/Santa_Isabel -Dtests.asserts=true -Dtests.file.encoding=UTF-8
>
> This is important to do:
> * use -Ptests.iters=1000 to run the same test over and over in same JVM
> * use the above TEST_JVM_ARGS to not run it with tiered compilation turned off.
>
> It looks like a JVM issue in all JVMs after 11. So this is a serious bug. In Lucene tests we havent seen this!
>
> Uwe
>
> -----
> Uwe Schindler
> Achterdiek 19, D-28357 Bremen
> https://www.thetaphi.de
> eMail: uwe@thetaphi.de
>
> > -----Original Message-----
> > From: Robert Muir <rcmuir@gmail.com>
> > Sent: Wednesday, December 8, 2021 6:44 PM
> > To: dev@lucene.apache.org
> > Cc: dev@solr.apache.org
> > Subject: Re: Strange error in Solr with Lucene 9.0.0
> >
> > Uwe, it looks a little crazy:
> >
> > we've got asserts here that "index" is in bounds and certainly not -1
> > right before the method call!
> > https://github.com/apache/lucene/blob/main/lucene/core/src/java/org/apache
> > /lucene/codecs/lucene90/Lucene90DocValuesProducer.java#L1123-L1125
> >
> > On Wed, Dec 8, 2021 at 12:33 PM Uwe Schindler <uwe@thetaphi.de> wrote:
> > >
> > > I updated the dependencies of Solr's main branch to Lucene 9.0
> > >
> > > On Mac and Linux, the following error occurs sometimes:
> > > https://jenkins.thetaphi.de/job/Solr-main-Linux/2070/testReport/junit/org.ap
> > >
> > ache.solr.request/SimpleFacetsTest/testRangeFacetFilterVsDocValuesRandom/
> > > https://jenkins.thetaphi.de/job/Solr-main-
> > MacOSX/517/testReport/junit/org.ap
> > > ache.solr/TestRandomDVFaceting/testRandomFaceting/
> > >
> > > "ArrayIndexOutOfBoundsException: Index -1 out of bounds for length 1" at
> > >
> > org.apache.lucene.util.packed.DirectMonotonicReader.get(DirectMonotonicRea
> > de
> > > r.java:161)
> > >
> > > This looks like a bug (or is Solr not using something with docvalues
> > > correctly, but an AIOOBE should not happen)!
> > >
> > > When searching through the failure mails, it looks like this was already
> > > there with the Lucene preview from summer 2021.
> > >
> > > Uwe
> > >
> > > -----
> > > Uwe Schindler
> > > Achterdiek 19, D-28357 Bremen
> > > https://www.thetaphi.de
> > > eMail: uwe@thetaphi.de
> > >
> > >
> > >
> > > ---------------------------------------------------------------------
> > > To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
> > > For additional commands, e-mail: dev-help@lucene.apache.org
> > >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: dev-unsubscribe@solr.apache.org
> > For additional commands, e-mail: dev-help@solr.apache.org
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: dev-help@lucene.apache.org
>

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org