Mailing List Archive

Java 17 and Lucene
Hello,

We are using Lucene 8 and planning to upgrade from Java 11 to Java 17. We
are curious:

- How lucene is testing against java versions. Are there correctness and
performance tests using java 17?
- Additionally, besides Java 17, how are new Java releases tested?
- Are there any other orgs using Java 17 with Lucene?
- Any other considerations we should be aware of?


Best,
Kevin Rosendahl
Re: Java 17 and Lucene [ In reply to ]
We test different releases on different platforms (e.g. Linux, Windows, Mac).
We also test EA (Early Access) releases of openjdk versions during the
development process.
This finds bugs before they get released.

More information about versions/EA testing: https://jenkins.thetaphi.de/

On Mon, Oct 18, 2021 at 5:33 PM Kevin Rosendahl
<kevindrosendahl@gmail.com> wrote:
>
> Hello,
>
> We are using Lucene 8 and planning to upgrade from Java 11 to Java 17. We
> are curious:
>
> - How lucene is testing against java versions. Are there correctness and
> performance tests using java 17?
> - Additionally, besides Java 17, how are new Java releases tested?
> - Are there any other orgs using Java 17 with Lucene?
> - Any other considerations we should be aware of?
>
>
> Best,
> Kevin Rosendahl

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Java 17 and Lucene [ In reply to ]
Also, I try to semi-aggressively upgrade Lucene's nightly benchmarks to new
JDK releases and leave an annotation on the nightly charts:
https://home.apache.org/~mikemccand/lucenebench/

I just now upgraded to JDK 17 and kicked off a new benchmark run ... in a
few hours it should show the new data points and then I'll try to remember
to annotate it tomorrow.

So let's see whether nightly benchmarks uncover any performance changes
from JDK17 :)

Mike McCandless

http://blog.mikemccandless.com


On Mon, Oct 18, 2021 at 5:36 PM Robert Muir <rcmuir@gmail.com> wrote:

> We test different releases on different platforms (e.g. Linux, Windows,
> Mac).
> We also test EA (Early Access) releases of openjdk versions during the
> development process.
> This finds bugs before they get released.
>
> More information about versions/EA testing: https://jenkins.thetaphi.de/
>
> On Mon, Oct 18, 2021 at 5:33 PM Kevin Rosendahl
> <kevindrosendahl@gmail.com> wrote:
> >
> > Hello,
> >
> > We are using Lucene 8 and planning to upgrade from Java 11 to Java 17. We
> > are curious:
> >
> > - How lucene is testing against java versions. Are there correctness
> and
> > performance tests using java 17?
> > - Additionally, besides Java 17, how are new Java releases tested?
> > - Are there any other orgs using Java 17 with Lucene?
> > - Any other considerations we should be aware of?
> >
> >
> > Best,
> > Kevin Rosendahl
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>
Re: Java 17 and Lucene [ In reply to ]
Hey,

Our team at Amazon Product Search recently ran our internal benchmarks with
JDK 17.
We saw a ~5% increase in throughput and are in the process of
experimenting/enabling it in production.
We also plan to test the new Corretto Generational Shenandoah GC.

On a side note, the Lucene codebase still uses the deprecated (as of
JDK17) AccessController
in the RamUsageEstimator class.
We suppressed the warning for now (based on recommendations
<http://mail-archives.apache.org/mod_mbox/db-derby-dev/202106.mbox/%3CJIRA.13369440.1617476525000.615331.1623951480059@Atlassian.JIRA%3E>
from the Apache Derby mailing list).

-
Gautam Worah.


On Mon, Oct 18, 2021 at 3:02 PM Michael McCandless <
lucene@mikemccandless.com> wrote:

> Also, I try to semi-aggressively upgrade Lucene's nightly benchmarks to new
> JDK releases and leave an annotation on the nightly charts:
> https://home.apache.org/~mikemccand/lucenebench/
>
> I just now upgraded to JDK 17 and kicked off a new benchmark run ... in a
> few hours it should show the new data points and then I'll try to remember
> to annotate it tomorrow.
>
> So let's see whether nightly benchmarks uncover any performance changes
> from JDK17 :)
>
> Mike McCandless
>
> http://blog.mikemccandless.com
>
>
> On Mon, Oct 18, 2021 at 5:36 PM Robert Muir <rcmuir@gmail.com> wrote:
>
> > We test different releases on different platforms (e.g. Linux, Windows,
> > Mac).
> > We also test EA (Early Access) releases of openjdk versions during the
> > development process.
> > This finds bugs before they get released.
> >
> > More information about versions/EA testing: https://jenkins.thetaphi.de/
> >
> > On Mon, Oct 18, 2021 at 5:33 PM Kevin Rosendahl
> > <kevindrosendahl@gmail.com> wrote:
> > >
> > > Hello,
> > >
> > > We are using Lucene 8 and planning to upgrade from Java 11 to Java 17.
> We
> > > are curious:
> > >
> > > - How lucene is testing against java versions. Are there correctness
> > and
> > > performance tests using java 17?
> > > - Additionally, besides Java 17, how are new Java releases
> tested?
> > > - Are there any other orgs using Java 17 with Lucene?
> > > - Any other considerations we should be aware of?
> > >
> > >
> > > Best,
> > > Kevin Rosendahl
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> > For additional commands, e-mail: java-user-help@lucene.apache.org
> >
> >
>
RE: Java 17 and Lucene [ In reply to ]
Hi,

> Hey,
>
> Our team at Amazon Product Search recently ran our internal benchmarks with
> JDK 17.
> We saw a ~5% increase in throughput and are in the process of
> experimenting/enabling it in production.
> We also plan to test the new Corretto Generational Shenandoah GC.

I would a bit careful: On our Jenkins server running with AMD Ryzen CPU it happens quite often that JDK 16, JDK 17 and JDK 18 hang during tests and stay unkillable (only a hard kill with" kill -9"). Previous Java versions don't hang. It happens not all the time (about 1/4th of all builds) and due to the fact that the JVM is unresponsible it is not possible to get a stack trace with "jstack". If you know a way to get the stack trace, I'd happy to get help.

Once I figured out what makes it hang, I will open issues in OpenJDK (I am OpenJDK member/editor). I have now many stuck JVMs running to analyze on the server, so you're invited to help! At the moment, I have no time to take care, so any help is useful.

> On a side note, the Lucene codebase still uses the deprecated (as of
> JDK17) AccessController
> in the RamUsageEstimator class.
> We suppressed the warning for now (based on recommendations
> <http://mail-archives.apache.org/mod_mbox/db-derby-
> dev/202106.mbox/%3CJIRA.13369440.1617476525000.615331.16239514800
> 59@Atlassian.JIRA%3E>
> from the Apache Derby mailing list).

This should not be an issue, because we compile Lucene with javac parameter "--release 11", so it won't show any warning that you need to suppress. Looks like your build system at Amazon is not the original one by Lucene's Gradle, which shows no warnings at all.

Uwe

> Gautam Worah.
>
>
> On Mon, Oct 18, 2021 at 3:02 PM Michael McCandless <
> lucene@mikemccandless.com> wrote:
>
> > Also, I try to semi-aggressively upgrade Lucene's nightly benchmarks to new
> > JDK releases and leave an annotation on the nightly charts:
> > https://home.apache.org/~mikemccand/lucenebench/
> >
> > I just now upgraded to JDK 17 and kicked off a new benchmark run ... in a
> > few hours it should show the new data points and then I'll try to remember
> > to annotate it tomorrow.
> >
> > So let's see whether nightly benchmarks uncover any performance changes
> > from JDK17 :)
> >
> > Mike McCandless
> >
> > http://blog.mikemccandless.com
> >
> >
> > On Mon, Oct 18, 2021 at 5:36 PM Robert Muir <rcmuir@gmail.com> wrote:
> >
> > > We test different releases on different platforms (e.g. Linux, Windows,
> > > Mac).
> > > We also test EA (Early Access) releases of openjdk versions during the
> > > development process.
> > > This finds bugs before they get released.
> > >
> > > More information about versions/EA testing: https://jenkins.thetaphi.de/
> > >
> > > On Mon, Oct 18, 2021 at 5:33 PM Kevin Rosendahl
> > > <kevindrosendahl@gmail.com> wrote:
> > > >
> > > > Hello,
> > > >
> > > > We are using Lucene 8 and planning to upgrade from Java 11 to Java 17.
> > We
> > > > are curious:
> > > >
> > > > - How lucene is testing against java versions. Are there correctness
> > > and
> > > > performance tests using java 17?
> > > > - Additionally, besides Java 17, how are new Java releases
> > tested?
> > > > - Are there any other orgs using Java 17 with Lucene?
> > > > - Any other considerations we should be aware of?
> > > >
> > > >
> > > > Best,
> > > > Kevin Rosendahl
> > >
> > > ---------------------------------------------------------------------
> > > To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> > > For additional commands, e-mail: java-user-help@lucene.apache.org
> > >
> > >
> >


---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
RE: Java 17 and Lucene [ In reply to ]
Hi,

> > On a side note, the Lucene codebase still uses the deprecated (as of
> > JDK17) AccessController
> > in the RamUsageEstimator class.
> > We suppressed the warning for now (based on recommendations
> > <http://mail-archives.apache.org/mod_mbox/db-derby-
> >
> dev/202106.mbox/%3CJIRA.13369440.1617476525000.615331.16239514800
> > 59@Atlassian.JIRA%3E>
> > from the Apache Derby mailing list).
>
> This should not be an issue, because we compile Lucene with javac parameter
> "--release 11", so it won't show any warning that you need to suppress. Looks
> like your build system at Amazon is not the original one by Lucene's Gradle,
> which shows no warnings at all.

Do you get this error in lucene:core:ecjLintMain and not during compile? Then this is https://issues.apache.org/jira/browse/LUCENE-10185, solved already. This problem did not happen in Lucene builds because we run Gradle with JDK 11 (JAVA_HOME) and only compile with JDK 17 (passed as RUNTIME_JAVA_HOME).

Uwe

> Uwe
>
> > Gautam Worah.
> >
> >
> > On Mon, Oct 18, 2021 at 3:02 PM Michael McCandless <
> > lucene@mikemccandless.com> wrote:
> >
> > > Also, I try to semi-aggressively upgrade Lucene's nightly benchmarks to new
> > > JDK releases and leave an annotation on the nightly charts:
> > > https://home.apache.org/~mikemccand/lucenebench/
> > >
> > > I just now upgraded to JDK 17 and kicked off a new benchmark run ... in a
> > > few hours it should show the new data points and then I'll try to remember
> > > to annotate it tomorrow.
> > >
> > > So let's see whether nightly benchmarks uncover any performance changes
> > > from JDK17 :)
> > >
> > > Mike McCandless
> > >
> > > http://blog.mikemccandless.com
> > >
> > >
> > > On Mon, Oct 18, 2021 at 5:36 PM Robert Muir <rcmuir@gmail.com> wrote:
> > >
> > > > We test different releases on different platforms (e.g. Linux, Windows,
> > > > Mac).
> > > > We also test EA (Early Access) releases of openjdk versions during the
> > > > development process.
> > > > This finds bugs before they get released.
> > > >
> > > > More information about versions/EA testing: https://jenkins.thetaphi.de/
> > > >
> > > > On Mon, Oct 18, 2021 at 5:33 PM Kevin Rosendahl
> > > > <kevindrosendahl@gmail.com> wrote:
> > > > >
> > > > > Hello,
> > > > >
> > > > > We are using Lucene 8 and planning to upgrade from Java 11 to Java 17.
> > > We
> > > > > are curious:
> > > > >
> > > > > - How lucene is testing against java versions. Are there correctness
> > > > and
> > > > > performance tests using java 17?
> > > > > - Additionally, besides Java 17, how are new Java releases
> > > tested?
> > > > > - Are there any other orgs using Java 17 with Lucene?
> > > > > - Any other considerations we should be aware of?
> > > > >
> > > > >
> > > > > Best,
> > > > > Kevin Rosendahl
> > > >
> > > > ---------------------------------------------------------------------
> > > > To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> > > > For additional commands, e-mail: java-user-help@lucene.apache.org
> > > >
> > > >
> > >
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Java 17 and Lucene [ In reply to ]
> I would a bit careful: On our Jenkins server running with AMD Ryzen CPU it happens quite often that JDK 16, JDK 17 and JDK 18 hang during tests and stay unkillable (only a hard kill with" kill -9"). Previous Java versions don't hang. It happens not all the time (about 1/4th of all builds) and due to the fact that the JVM is unresponsible it is not possible to get a stack trace with "jstack". If you know a way to get the stack trace, I'd happy to get help.

ooh that sounds scary. I suppose one could maybe get core dumps using
the right signal and debug that way? Oh wait you said only 9 works,
darn! How about attaching using gdb? Do we maintain GC logs for these
Jenkins builds? Maybe something suspicious would show up there.

By the way the JDK is absolutely "responsible" in this situation! Not
responsive maybe ...

On Tue, Oct 19, 2021 at 4:46 AM Uwe Schindler <uwe@thetaphi.de> wrote:
>
> Hi,
>
> > Hey,
> >
> > Our team at Amazon Product Search recently ran our internal benchmarks with
> > JDK 17.
> > We saw a ~5% increase in throughput and are in the process of
> > experimenting/enabling it in production.
> > We also plan to test the new Corretto Generational Shenandoah GC.
>
> I would a bit careful: On our Jenkins server running with AMD Ryzen CPU it happens quite often that JDK 16, JDK 17 and JDK 18 hang during tests and stay unkillable (only a hard kill with" kill -9"). Previous Java versions don't hang. It happens not all the time (about 1/4th of all builds) and due to the fact that the JVM is unresponsible it is not possible to get a stack trace with "jstack". If you know a way to get the stack trace, I'd happy to get help.
>
> Once I figured out what makes it hang, I will open issues in OpenJDK (I am OpenJDK member/editor). I have now many stuck JVMs running to analyze on the server, so you're invited to help! At the moment, I have no time to take care, so any help is useful.
>
> > On a side note, the Lucene codebase still uses the deprecated (as of
> > JDK17) AccessController
> > in the RamUsageEstimator class.
> > We suppressed the warning for now (based on recommendations
> > <http://mail-archives.apache.org/mod_mbox/db-derby-
> > dev/202106.mbox/%3CJIRA.13369440.1617476525000.615331.16239514800
> > 59@Atlassian.JIRA%3E>
> > from the Apache Derby mailing list).
>
> This should not be an issue, because we compile Lucene with javac parameter "--release 11", so it won't show any warning that you need to suppress. Looks like your build system at Amazon is not the original one by Lucene's Gradle, which shows no warnings at all.
>
> Uwe
>
> > Gautam Worah.
> >
> >
> > On Mon, Oct 18, 2021 at 3:02 PM Michael McCandless <
> > lucene@mikemccandless.com> wrote:
> >
> > > Also, I try to semi-aggressively upgrade Lucene's nightly benchmarks to new
> > > JDK releases and leave an annotation on the nightly charts:
> > > https://home.apache.org/~mikemccand/lucenebench/
> > >
> > > I just now upgraded to JDK 17 and kicked off a new benchmark run ... in a
> > > few hours it should show the new data points and then I'll try to remember
> > > to annotate it tomorrow.
> > >
> > > So let's see whether nightly benchmarks uncover any performance changes
> > > from JDK17 :)
> > >
> > > Mike McCandless
> > >
> > > http://blog.mikemccandless.com
> > >
> > >
> > > On Mon, Oct 18, 2021 at 5:36 PM Robert Muir <rcmuir@gmail.com> wrote:
> > >
> > > > We test different releases on different platforms (e.g. Linux, Windows,
> > > > Mac).
> > > > We also test EA (Early Access) releases of openjdk versions during the
> > > > development process.
> > > > This finds bugs before they get released.
> > > >
> > > > More information about versions/EA testing: https://jenkins.thetaphi.de/
> > > >
> > > > On Mon, Oct 18, 2021 at 5:33 PM Kevin Rosendahl
> > > > <kevindrosendahl@gmail.com> wrote:
> > > > >
> > > > > Hello,
> > > > >
> > > > > We are using Lucene 8 and planning to upgrade from Java 11 to Java 17.
> > > We
> > > > > are curious:
> > > > >
> > > > > - How lucene is testing against java versions. Are there correctness
> > > > and
> > > > > performance tests using java 17?
> > > > > - Additionally, besides Java 17, how are new Java releases
> > > tested?
> > > > > - Are there any other orgs using Java 17 with Lucene?
> > > > > - Any other considerations we should be aware of?
> > > > >
> > > > >
> > > > > Best,
> > > > > Kevin Rosendahl
> > > >
> > > > ---------------------------------------------------------------------
> > > > To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> > > > For additional commands, e-mail: java-user-help@lucene.apache.org
> > > >
> > > >
> > >
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Java 17 and Lucene [ In reply to ]
Thank you all for the information, it's very useful. Seems like it's best
to hold off on upgrading for now, but great to know that different JDK
versions are at least being exercised in CI.

I'm wondering, is there a better way to assess the production readiness of
a Lucene/JDK combination than just emailing the user group, or is this our
best bet in the future as well?

Thanks again!
Kevin

On Tue, Oct 19, 2021 at 5:07 AM Michael Sokolov <msokolov@gmail.com> wrote:

> > I would a bit careful: On our Jenkins server running with AMD Ryzen CPU
> it happens quite often that JDK 16, JDK 17 and JDK 18 hang during tests and
> stay unkillable (only a hard kill with" kill -9"). Previous Java versions
> don't hang. It happens not all the time (about 1/4th of all builds) and due
> to the fact that the JVM is unresponsible it is not possible to get a stack
> trace with "jstack". If you know a way to get the stack trace, I'd happy to
> get help.
>
> ooh that sounds scary. I suppose one could maybe get core dumps using
> the right signal and debug that way? Oh wait you said only 9 works,
> darn! How about attaching using gdb? Do we maintain GC logs for these
> Jenkins builds? Maybe something suspicious would show up there.
>
> By the way the JDK is absolutely "responsible" in this situation! Not
> responsive maybe ...
>
> On Tue, Oct 19, 2021 at 4:46 AM Uwe Schindler <uwe@thetaphi.de> wrote:
> >
> > Hi,
> >
> > > Hey,
> > >
> > > Our team at Amazon Product Search recently ran our internal benchmarks
> with
> > > JDK 17.
> > > We saw a ~5% increase in throughput and are in the process of
> > > experimenting/enabling it in production.
> > > We also plan to test the new Corretto Generational Shenandoah GC.
> >
> > I would a bit careful: On our Jenkins server running with AMD Ryzen CPU
> it happens quite often that JDK 16, JDK 17 and JDK 18 hang during tests and
> stay unkillable (only a hard kill with" kill -9"). Previous Java versions
> don't hang. It happens not all the time (about 1/4th of all builds) and due
> to the fact that the JVM is unresponsible it is not possible to get a stack
> trace with "jstack". If you know a way to get the stack trace, I'd happy to
> get help.
> >
> > Once I figured out what makes it hang, I will open issues in OpenJDK (I
> am OpenJDK member/editor). I have now many stuck JVMs running to analyze on
> the server, so you're invited to help! At the moment, I have no time to
> take care, so any help is useful.
> >
> > > On a side note, the Lucene codebase still uses the deprecated (as of
> > > JDK17) AccessController
> > > in the RamUsageEstimator class.
> > > We suppressed the warning for now (based on recommendations
> > > <http://mail-archives.apache.org/mod_mbox/db-derby-
> > > dev/202106.mbox/%3CJIRA.13369440.1617476525000.615331.16239514800
> > > 59@Atlassian.JIRA%3E>
> > > from the Apache Derby mailing list).
> >
> > This should not be an issue, because we compile Lucene with javac
> parameter "--release 11", so it won't show any warning that you need to
> suppress. Looks like your build system at Amazon is not the original one by
> Lucene's Gradle, which shows no warnings at all.
> >
> > Uwe
> >
> > > Gautam Worah.
> > >
> > >
> > > On Mon, Oct 18, 2021 at 3:02 PM Michael McCandless <
> > > lucene@mikemccandless.com> wrote:
> > >
> > > > Also, I try to semi-aggressively upgrade Lucene's nightly benchmarks
> to new
> > > > JDK releases and leave an annotation on the nightly charts:
> > > > https://home.apache.org/~mikemccand/lucenebench/
> > > >
> > > > I just now upgraded to JDK 17 and kicked off a new benchmark run ...
> in a
> > > > few hours it should show the new data points and then I'll try to
> remember
> > > > to annotate it tomorrow.
> > > >
> > > > So let's see whether nightly benchmarks uncover any performance
> changes
> > > > from JDK17 :)
> > > >
> > > > Mike McCandless
> > > >
> > > > http://blog.mikemccandless.com
> > > >
> > > >
> > > > On Mon, Oct 18, 2021 at 5:36 PM Robert Muir <rcmuir@gmail.com>
> wrote:
> > > >
> > > > > We test different releases on different platforms (e.g. Linux,
> Windows,
> > > > > Mac).
> > > > > We also test EA (Early Access) releases of openjdk versions during
> the
> > > > > development process.
> > > > > This finds bugs before they get released.
> > > > >
> > > > > More information about versions/EA testing:
> https://jenkins.thetaphi.de/
> > > > >
> > > > > On Mon, Oct 18, 2021 at 5:33 PM Kevin Rosendahl
> > > > > <kevindrosendahl@gmail.com> wrote:
> > > > > >
> > > > > > Hello,
> > > > > >
> > > > > > We are using Lucene 8 and planning to upgrade from Java 11 to
> Java 17.
> > > > We
> > > > > > are curious:
> > > > > >
> > > > > > - How lucene is testing against java versions. Are there
> correctness
> > > > > and
> > > > > > performance tests using java 17?
> > > > > > - Additionally, besides Java 17, how are new Java releases
> > > > tested?
> > > > > > - Are there any other orgs using Java 17 with Lucene?
> > > > > > - Any other considerations we should be aware of?
> > > > > >
> > > > > >
> > > > > > Best,
> > > > > > Kevin Rosendahl
> > > > >
> > > > >
> ---------------------------------------------------------------------
> > > > > To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> > > > > For additional commands, e-mail: java-user-help@lucene.apache.org
> > > > >
> > > > >
> > > >
> >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> > For additional commands, e-mail: java-user-help@lucene.apache.org
> >
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>
Re: Java 17 and Lucene [ In reply to ]
Thanks for the note of caution Uwe.

> On our Jenkins server running with AMD Ryzen CPU it happens quite often
that JDK 16, JDK 17 and JDK 18 hang during tests and stay unkillable (only
a hard kill with" kill -9")

Scary stuff.
I'll try to reproduce the hang first and then try to get the JVM logs. I'll
respond back here if I find something useful.

> Do you get this error in lucene:core:ecjLintMain and not during compile?
Then this is https://issues.apache.org/jira/browse/LUCENE-10185, solved
already.

Ahh. I should've been clearer with my comment. The error we see is because
we have forked the class and have modified it a bit.
I just assumed that the upstream Lucene package would've also gotten errors
on the JDK17 build because it was untouched.

-
Gautam Worah.


On Tue, Oct 19, 2021 at 5:07 AM Michael Sokolov <msokolov@gmail.com> wrote:

> > I would a bit careful: On our Jenkins server running with AMD Ryzen CPU
> it happens quite often that JDK 16, JDK 17 and JDK 18 hang during tests and
> stay unkillable (only a hard kill with" kill -9"). Previous Java versions
> don't hang. It happens not all the time (about 1/4th of all builds) and due
> to the fact that the JVM is unresponsible it is not possible to get a stack
> trace with "jstack". If you know a way to get the stack trace, I'd happy to
> get help.
>
> ooh that sounds scary. I suppose one could maybe get core dumps using
> the right signal and debug that way? Oh wait you said only 9 works,
> darn! How about attaching using gdb? Do we maintain GC logs for these
> Jenkins builds? Maybe something suspicious would show up there.
>
> By the way the JDK is absolutely "responsible" in this situation! Not
> responsive maybe ...
>
> On Tue, Oct 19, 2021 at 4:46 AM Uwe Schindler <uwe@thetaphi.de> wrote:
> >
> > Hi,
> >
> > > Hey,
> > >
> > > Our team at Amazon Product Search recently ran our internal benchmarks
> with
> > > JDK 17.
> > > We saw a ~5% increase in throughput and are in the process of
> > > experimenting/enabling it in production.
> > > We also plan to test the new Corretto Generational Shenandoah GC.
> >
> > I would a bit careful: On our Jenkins server running with AMD Ryzen CPU
> it happens quite often that JDK 16, JDK 17 and JDK 18 hang during tests and
> stay unkillable (only a hard kill with" kill -9"). Previous Java versions
> don't hang. It happens not all the time (about 1/4th of all builds) and due
> to the fact that the JVM is unresponsible it is not possible to get a stack
> trace with "jstack". If you know a way to get the stack trace, I'd happy to
> get help.
> >
> > Once I figured out what makes it hang, I will open issues in OpenJDK (I
> am OpenJDK member/editor). I have now many stuck JVMs running to analyze on
> the server, so you're invited to help! At the moment, I have no time to
> take care, so any help is useful.
> >
> > > On a side note, the Lucene codebase still uses the deprecated (as of
> > > JDK17) AccessController
> > > in the RamUsageEstimator class.
> > > We suppressed the warning for now (based on recommendations
> > > <http://mail-archives.apache.org/mod_mbox/db-derby-
> > > dev/202106.mbox/%3CJIRA.13369440.1617476525000.615331.16239514800
> > > 59@Atlassian.JIRA%3E>
> > > from the Apache Derby mailing list).
> >
> > This should not be an issue, because we compile Lucene with javac
> parameter "--release 11", so it won't show any warning that you need to
> suppress. Looks like your build system at Amazon is not the original one by
> Lucene's Gradle, which shows no warnings at all.
> >
> > Uwe
> >
> > > Gautam Worah.
> > >
> > >
> > > On Mon, Oct 18, 2021 at 3:02 PM Michael McCandless <
> > > lucene@mikemccandless.com> wrote:
> > >
> > > > Also, I try to semi-aggressively upgrade Lucene's nightly benchmarks
> to new
> > > > JDK releases and leave an annotation on the nightly charts:
> > > > https://home.apache.org/~mikemccand/lucenebench/
> > > >
> > > > I just now upgraded to JDK 17 and kicked off a new benchmark run ...
> in a
> > > > few hours it should show the new data points and then I'll try to
> remember
> > > > to annotate it tomorrow.
> > > >
> > > > So let's see whether nightly benchmarks uncover any performance
> changes
> > > > from JDK17 :)
> > > >
> > > > Mike McCandless
> > > >
> > > > http://blog.mikemccandless.com
> > > >
> > > >
> > > > On Mon, Oct 18, 2021 at 5:36 PM Robert Muir <rcmuir@gmail.com>
> wrote:
> > > >
> > > > > We test different releases on different platforms (e.g. Linux,
> Windows,
> > > > > Mac).
> > > > > We also test EA (Early Access) releases of openjdk versions during
> the
> > > > > development process.
> > > > > This finds bugs before they get released.
> > > > >
> > > > > More information about versions/EA testing:
> https://jenkins.thetaphi.de/
> > > > >
> > > > > On Mon, Oct 18, 2021 at 5:33 PM Kevin Rosendahl
> > > > > <kevindrosendahl@gmail.com> wrote:
> > > > > >
> > > > > > Hello,
> > > > > >
> > > > > > We are using Lucene 8 and planning to upgrade from Java 11 to
> Java 17.
> > > > We
> > > > > > are curious:
> > > > > >
> > > > > > - How lucene is testing against java versions. Are there
> correctness
> > > > > and
> > > > > > performance tests using java 17?
> > > > > > - Additionally, besides Java 17, how are new Java releases
> > > > tested?
> > > > > > - Are there any other orgs using Java 17 with Lucene?
> > > > > > - Any other considerations we should be aware of?
> > > > > >
> > > > > >
> > > > > > Best,
> > > > > > Kevin Rosendahl
> > > > >
> > > > >
> ---------------------------------------------------------------------
> > > > > To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> > > > > For additional commands, e-mail: java-user-help@lucene.apache.org
> > > > >
> > > > >
> > > >
> >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> > For additional commands, e-mail: java-user-help@lucene.apache.org
> >
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>
Re: Java 17 and Lucene [ In reply to ]
Nightly benchmarks managed to succeed (once, so far) on JDK 17:
https://home.apache.org/~mikemccand/lucenebench/

No obvious performance changes on quick look.

Mike McCandless

http://blog.mikemccandless.com


On Tue, Oct 19, 2021 at 8:42 PM Gautam Worah <worah.gautam@gmail.com> wrote:

> Thanks for the note of caution Uwe.
>
> > On our Jenkins server running with AMD Ryzen CPU it happens quite often
> that JDK 16, JDK 17 and JDK 18 hang during tests and stay unkillable (only
> a hard kill with" kill -9")
>
> Scary stuff.
> I'll try to reproduce the hang first and then try to get the JVM logs. I'll
> respond back here if I find something useful.
>
> > Do you get this error in lucene:core:ecjLintMain and not during compile?
> Then this is https://issues.apache.org/jira/browse/LUCENE-10185, solved
> already.
>
> Ahh. I should've been clearer with my comment. The error we see is because
> we have forked the class and have modified it a bit.
> I just assumed that the upstream Lucene package would've also gotten errors
> on the JDK17 build because it was untouched.
>
> -
> Gautam Worah.
>
>
> On Tue, Oct 19, 2021 at 5:07 AM Michael Sokolov <msokolov@gmail.com>
> wrote:
>
> > > I would a bit careful: On our Jenkins server running with AMD Ryzen CPU
> > it happens quite often that JDK 16, JDK 17 and JDK 18 hang during tests
> and
> > stay unkillable (only a hard kill with" kill -9"). Previous Java versions
> > don't hang. It happens not all the time (about 1/4th of all builds) and
> due
> > to the fact that the JVM is unresponsible it is not possible to get a
> stack
> > trace with "jstack". If you know a way to get the stack trace, I'd happy
> to
> > get help.
> >
> > ooh that sounds scary. I suppose one could maybe get core dumps using
> > the right signal and debug that way? Oh wait you said only 9 works,
> > darn! How about attaching using gdb? Do we maintain GC logs for these
> > Jenkins builds? Maybe something suspicious would show up there.
> >
> > By the way the JDK is absolutely "responsible" in this situation! Not
> > responsive maybe ...
> >
> > On Tue, Oct 19, 2021 at 4:46 AM Uwe Schindler <uwe@thetaphi.de> wrote:
> > >
> > > Hi,
> > >
> > > > Hey,
> > > >
> > > > Our team at Amazon Product Search recently ran our internal
> benchmarks
> > with
> > > > JDK 17.
> > > > We saw a ~5% increase in throughput and are in the process of
> > > > experimenting/enabling it in production.
> > > > We also plan to test the new Corretto Generational Shenandoah GC.
> > >
> > > I would a bit careful: On our Jenkins server running with AMD Ryzen CPU
> > it happens quite often that JDK 16, JDK 17 and JDK 18 hang during tests
> and
> > stay unkillable (only a hard kill with" kill -9"). Previous Java versions
> > don't hang. It happens not all the time (about 1/4th of all builds) and
> due
> > to the fact that the JVM is unresponsible it is not possible to get a
> stack
> > trace with "jstack". If you know a way to get the stack trace, I'd happy
> to
> > get help.
> > >
> > > Once I figured out what makes it hang, I will open issues in OpenJDK (I
> > am OpenJDK member/editor). I have now many stuck JVMs running to analyze
> on
> > the server, so you're invited to help! At the moment, I have no time to
> > take care, so any help is useful.
> > >
> > > > On a side note, the Lucene codebase still uses the deprecated (as of
> > > > JDK17) AccessController
> > > > in the RamUsageEstimator class.
> > > > We suppressed the warning for now (based on recommendations
> > > > <http://mail-archives.apache.org/mod_mbox/db-derby-
> > > > dev/202106.mbox/%3CJIRA.13369440.1617476525000.615331.16239514800
> > > > 59@Atlassian.JIRA%3E>
> > > > from the Apache Derby mailing list).
> > >
> > > This should not be an issue, because we compile Lucene with javac
> > parameter "--release 11", so it won't show any warning that you need to
> > suppress. Looks like your build system at Amazon is not the original one
> by
> > Lucene's Gradle, which shows no warnings at all.
> > >
> > > Uwe
> > >
> > > > Gautam Worah.
> > > >
> > > >
> > > > On Mon, Oct 18, 2021 at 3:02 PM Michael McCandless <
> > > > lucene@mikemccandless.com> wrote:
> > > >
> > > > > Also, I try to semi-aggressively upgrade Lucene's nightly
> benchmarks
> > to new
> > > > > JDK releases and leave an annotation on the nightly charts:
> > > > > https://home.apache.org/~mikemccand/lucenebench/
> > > > >
> > > > > I just now upgraded to JDK 17 and kicked off a new benchmark run
> ...
> > in a
> > > > > few hours it should show the new data points and then I'll try to
> > remember
> > > > > to annotate it tomorrow.
> > > > >
> > > > > So let's see whether nightly benchmarks uncover any performance
> > changes
> > > > > from JDK17 :)
> > > > >
> > > > > Mike McCandless
> > > > >
> > > > > http://blog.mikemccandless.com
> > > > >
> > > > >
> > > > > On Mon, Oct 18, 2021 at 5:36 PM Robert Muir <rcmuir@gmail.com>
> > wrote:
> > > > >
> > > > > > We test different releases on different platforms (e.g. Linux,
> > Windows,
> > > > > > Mac).
> > > > > > We also test EA (Early Access) releases of openjdk versions
> during
> > the
> > > > > > development process.
> > > > > > This finds bugs before they get released.
> > > > > >
> > > > > > More information about versions/EA testing:
> > https://jenkins.thetaphi.de/
> > > > > >
> > > > > > On Mon, Oct 18, 2021 at 5:33 PM Kevin Rosendahl
> > > > > > <kevindrosendahl@gmail.com> wrote:
> > > > > > >
> > > > > > > Hello,
> > > > > > >
> > > > > > > We are using Lucene 8 and planning to upgrade from Java 11 to
> > Java 17.
> > > > > We
> > > > > > > are curious:
> > > > > > >
> > > > > > > - How lucene is testing against java versions. Are there
> > correctness
> > > > > > and
> > > > > > > performance tests using java 17?
> > > > > > > - Additionally, besides Java 17, how are new Java
> releases
> > > > > tested?
> > > > > > > - Are there any other orgs using Java 17 with Lucene?
> > > > > > > - Any other considerations we should be aware of?
> > > > > > >
> > > > > > >
> > > > > > > Best,
> > > > > > > Kevin Rosendahl
> > > > > >
> > > > > >
> > ---------------------------------------------------------------------
> > > > > > To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> > > > > > For additional commands, e-mail:
> java-user-help@lucene.apache.org
> > > > > >
> > > > > >
> > > > >
> > >
> > >
> > > ---------------------------------------------------------------------
> > > To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> > > For additional commands, e-mail: java-user-help@lucene.apache.org
> > >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> > For additional commands, e-mail: java-user-help@lucene.apache.org
> >
> >
>
Re: Java 17 and Lucene [ In reply to ]
The "System Requirements" page for each release lists the JDK it was
built with and tested most extensively with; eg
https://lucene.apache.org/core/8_10_1/SYSTEM_REQUIREMENTS.html (JDK8
there; 9.0 will target JDK11)

That is pretty conservative but safe. Generally speaking we are always
testing cutting-edge JDKs, so you can be pretty confident about say
JDK11, but it's best to run your own tests, of course.

On Tue, Oct 19, 2021 at 8:19 PM Kevin Rosendahl
<kevindrosendahl@gmail.com> wrote:
>
> Thank you all for the information, it's very useful. Seems like it's best
> to hold off on upgrading for now, but great to know that different JDK
> versions are at least being exercised in CI.
>
> I'm wondering, is there a better way to assess the production readiness of
> a Lucene/JDK combination than just emailing the user group, or is this our
> best bet in the future as well?
>
> Thanks again!
> Kevin
>
> On Tue, Oct 19, 2021 at 5:07 AM Michael Sokolov <msokolov@gmail.com> wrote:
>
> > > I would a bit careful: On our Jenkins server running with AMD Ryzen CPU
> > it happens quite often that JDK 16, JDK 17 and JDK 18 hang during tests and
> > stay unkillable (only a hard kill with" kill -9"). Previous Java versions
> > don't hang. It happens not all the time (about 1/4th of all builds) and due
> > to the fact that the JVM is unresponsible it is not possible to get a stack
> > trace with "jstack". If you know a way to get the stack trace, I'd happy to
> > get help.
> >
> > ooh that sounds scary. I suppose one could maybe get core dumps using
> > the right signal and debug that way? Oh wait you said only 9 works,
> > darn! How about attaching using gdb? Do we maintain GC logs for these
> > Jenkins builds? Maybe something suspicious would show up there.
> >
> > By the way the JDK is absolutely "responsible" in this situation! Not
> > responsive maybe ...
> >
> > On Tue, Oct 19, 2021 at 4:46 AM Uwe Schindler <uwe@thetaphi.de> wrote:
> > >
> > > Hi,
> > >
> > > > Hey,
> > > >
> > > > Our team at Amazon Product Search recently ran our internal benchmarks
> > with
> > > > JDK 17.
> > > > We saw a ~5% increase in throughput and are in the process of
> > > > experimenting/enabling it in production.
> > > > We also plan to test the new Corretto Generational Shenandoah GC.
> > >
> > > I would a bit careful: On our Jenkins server running with AMD Ryzen CPU
> > it happens quite often that JDK 16, JDK 17 and JDK 18 hang during tests and
> > stay unkillable (only a hard kill with" kill -9"). Previous Java versions
> > don't hang. It happens not all the time (about 1/4th of all builds) and due
> > to the fact that the JVM is unresponsible it is not possible to get a stack
> > trace with "jstack". If you know a way to get the stack trace, I'd happy to
> > get help.
> > >
> > > Once I figured out what makes it hang, I will open issues in OpenJDK (I
> > am OpenJDK member/editor). I have now many stuck JVMs running to analyze on
> > the server, so you're invited to help! At the moment, I have no time to
> > take care, so any help is useful.
> > >
> > > > On a side note, the Lucene codebase still uses the deprecated (as of
> > > > JDK17) AccessController
> > > > in the RamUsageEstimator class.
> > > > We suppressed the warning for now (based on recommendations
> > > > <http://mail-archives.apache.org/mod_mbox/db-derby-
> > > > dev/202106.mbox/%3CJIRA.13369440.1617476525000.615331.16239514800
> > > > 59@Atlassian.JIRA%3E>
> > > > from the Apache Derby mailing list).
> > >
> > > This should not be an issue, because we compile Lucene with javac
> > parameter "--release 11", so it won't show any warning that you need to
> > suppress. Looks like your build system at Amazon is not the original one by
> > Lucene's Gradle, which shows no warnings at all.
> > >
> > > Uwe
> > >
> > > > Gautam Worah.
> > > >
> > > >
> > > > On Mon, Oct 18, 2021 at 3:02 PM Michael McCandless <
> > > > lucene@mikemccandless.com> wrote:
> > > >
> > > > > Also, I try to semi-aggressively upgrade Lucene's nightly benchmarks
> > to new
> > > > > JDK releases and leave an annotation on the nightly charts:
> > > > > https://home.apache.org/~mikemccand/lucenebench/
> > > > >
> > > > > I just now upgraded to JDK 17 and kicked off a new benchmark run ...
> > in a
> > > > > few hours it should show the new data points and then I'll try to
> > remember
> > > > > to annotate it tomorrow.
> > > > >
> > > > > So let's see whether nightly benchmarks uncover any performance
> > changes
> > > > > from JDK17 :)
> > > > >
> > > > > Mike McCandless
> > > > >
> > > > > http://blog.mikemccandless.com
> > > > >
> > > > >
> > > > > On Mon, Oct 18, 2021 at 5:36 PM Robert Muir <rcmuir@gmail.com>
> > wrote:
> > > > >
> > > > > > We test different releases on different platforms (e.g. Linux,
> > Windows,
> > > > > > Mac).
> > > > > > We also test EA (Early Access) releases of openjdk versions during
> > the
> > > > > > development process.
> > > > > > This finds bugs before they get released.
> > > > > >
> > > > > > More information about versions/EA testing:
> > https://jenkins.thetaphi.de/
> > > > > >
> > > > > > On Mon, Oct 18, 2021 at 5:33 PM Kevin Rosendahl
> > > > > > <kevindrosendahl@gmail.com> wrote:
> > > > > > >
> > > > > > > Hello,
> > > > > > >
> > > > > > > We are using Lucene 8 and planning to upgrade from Java 11 to
> > Java 17.
> > > > > We
> > > > > > > are curious:
> > > > > > >
> > > > > > > - How lucene is testing against java versions. Are there
> > correctness
> > > > > > and
> > > > > > > performance tests using java 17?
> > > > > > > - Additionally, besides Java 17, how are new Java releases
> > > > > tested?
> > > > > > > - Are there any other orgs using Java 17 with Lucene?
> > > > > > > - Any other considerations we should be aware of?
> > > > > > >
> > > > > > >
> > > > > > > Best,
> > > > > > > Kevin Rosendahl
> > > > > >
> > > > > >
> > ---------------------------------------------------------------------
> > > > > > To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> > > > > > For additional commands, e-mail: java-user-help@lucene.apache.org
> > > > > >
> > > > > >
> > > > >
> > >
> > >
> > > ---------------------------------------------------------------------
> > > To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> > > For additional commands, e-mail: java-user-help@lucene.apache.org
> > >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> > For additional commands, e-mail: java-user-help@lucene.apache.org
> >
> >

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Java 17 and Lucene [ In reply to ]
Michael,

Is this recommended "-XX:+UseZGC options to enable ZGC." as it claims very
low pauses.

For "*DY* (2021-10-19 08:14:33): Upgrade to JDK17+35" execution for "Indexing
throughput <https://home.apache.org/~mikemccand/lucenebench/indexing.html>"
is ZGC used for the "Indexing throughput
<https://home.apache.org/~mikemccand/lucenebench/indexing.html>" test?


On Wed, Oct 20, 2021 at 8:27 AM Michael McCandless <
lucene@mikemccandless.com> wrote:

> Nightly benchmarks managed to succeed (once, so far) on JDK 17:
> https://home.apache.org/~mikemccand/lucenebench/
>
> No obvious performance changes on quick look.
>
> Mike McCandless
>
> http://blog.mikemccandless.com
>
>
> On Tue, Oct 19, 2021 at 8:42 PM Gautam Worah <worah.gautam@gmail.com>
> wrote:
>
> > Thanks for the note of caution Uwe.
> >
> > > On our Jenkins server running with AMD Ryzen CPU it happens quite often
> > that JDK 16, JDK 17 and JDK 18 hang during tests and stay unkillable
> (only
> > a hard kill with" kill -9")
> >
> > Scary stuff.
> > I'll try to reproduce the hang first and then try to get the JVM logs.
> I'll
> > respond back here if I find something useful.
> >
> > > Do you get this error in lucene:core:ecjLintMain and not during
> compile?
> > Then this is https://issues.apache.org/jira/browse/LUCENE-10185, solved
> > already.
> >
> > Ahh. I should've been clearer with my comment. The error we see is
> because
> > we have forked the class and have modified it a bit.
> > I just assumed that the upstream Lucene package would've also gotten
> errors
> > on the JDK17 build because it was untouched.
> >
> > -
> > Gautam Worah.
> >
> >
> > On Tue, Oct 19, 2021 at 5:07 AM Michael Sokolov <msokolov@gmail.com>
> > wrote:
> >
> > > > I would a bit careful: On our Jenkins server running with AMD Ryzen
> CPU
> > > it happens quite often that JDK 16, JDK 17 and JDK 18 hang during tests
> > and
> > > stay unkillable (only a hard kill with" kill -9"). Previous Java
> versions
> > > don't hang. It happens not all the time (about 1/4th of all builds) and
> > due
> > > to the fact that the JVM is unresponsible it is not possible to get a
> > stack
> > > trace with "jstack". If you know a way to get the stack trace, I'd
> happy
> > to
> > > get help.
> > >
> > > ooh that sounds scary. I suppose one could maybe get core dumps using
> > > the right signal and debug that way? Oh wait you said only 9 works,
> > > darn! How about attaching using gdb? Do we maintain GC logs for these
> > > Jenkins builds? Maybe something suspicious would show up there.
> > >
> > > By the way the JDK is absolutely "responsible" in this situation! Not
> > > responsive maybe ...
> > >
> > > On Tue, Oct 19, 2021 at 4:46 AM Uwe Schindler <uwe@thetaphi.de> wrote:
> > > >
> > > > Hi,
> > > >
> > > > > Hey,
> > > > >
> > > > > Our team at Amazon Product Search recently ran our internal
> > benchmarks
> > > with
> > > > > JDK 17.
> > > > > We saw a ~5% increase in throughput and are in the process of
> > > > > experimenting/enabling it in production.
> > > > > We also plan to test the new Corretto Generational Shenandoah GC.
> > > >
> > > > I would a bit careful: On our Jenkins server running with AMD Ryzen
> CPU
> > > it happens quite often that JDK 16, JDK 17 and JDK 18 hang during tests
> > and
> > > stay unkillable (only a hard kill with" kill -9"). Previous Java
> versions
> > > don't hang. It happens not all the time (about 1/4th of all builds) and
> > due
> > > to the fact that the JVM is unresponsible it is not possible to get a
> > stack
> > > trace with "jstack". If you know a way to get the stack trace, I'd
> happy
> > to
> > > get help.
> > > >
> > > > Once I figured out what makes it hang, I will open issues in OpenJDK
> (I
> > > am OpenJDK member/editor). I have now many stuck JVMs running to
> analyze
> > on
> > > the server, so you're invited to help! At the moment, I have no time to
> > > take care, so any help is useful.
> > > >
> > > > > On a side note, the Lucene codebase still uses the deprecated (as
> of
> > > > > JDK17) AccessController
> > > > > in the RamUsageEstimator class.
> > > > > We suppressed the warning for now (based on recommendations
> > > > > <http://mail-archives.apache.org/mod_mbox/db-derby-
> > > > > dev/202106.mbox/%3CJIRA.13369440.1617476525000.615331.16239514800
> > > > > 59@Atlassian.JIRA%3E>
> > > > > from the Apache Derby mailing list).
> > > >
> > > > This should not be an issue, because we compile Lucene with javac
> > > parameter "--release 11", so it won't show any warning that you need to
> > > suppress. Looks like your build system at Amazon is not the original
> one
> > by
> > > Lucene's Gradle, which shows no warnings at all.
> > > >
> > > > Uwe
> > > >
> > > > > Gautam Worah.
> > > > >
> > > > >
> > > > > On Mon, Oct 18, 2021 at 3:02 PM Michael McCandless <
> > > > > lucene@mikemccandless.com> wrote:
> > > > >
> > > > > > Also, I try to semi-aggressively upgrade Lucene's nightly
> > benchmarks
> > > to new
> > > > > > JDK releases and leave an annotation on the nightly charts:
> > > > > > https://home.apache.org/~mikemccand/lucenebench/
> > > > > >
> > > > > > I just now upgraded to JDK 17 and kicked off a new benchmark run
> > ...
> > > in a
> > > > > > few hours it should show the new data points and then I'll try to
> > > remember
> > > > > > to annotate it tomorrow.
> > > > > >
> > > > > > So let's see whether nightly benchmarks uncover any performance
> > > changes
> > > > > > from JDK17 :)
> > > > > >
> > > > > > Mike McCandless
> > > > > >
> > > > > > http://blog.mikemccandless.com
> > > > > >
> > > > > >
> > > > > > On Mon, Oct 18, 2021 at 5:36 PM Robert Muir <rcmuir@gmail.com>
> > > wrote:
> > > > > >
> > > > > > > We test different releases on different platforms (e.g. Linux,
> > > Windows,
> > > > > > > Mac).
> > > > > > > We also test EA (Early Access) releases of openjdk versions
> > during
> > > the
> > > > > > > development process.
> > > > > > > This finds bugs before they get released.
> > > > > > >
> > > > > > > More information about versions/EA testing:
> > > https://jenkins.thetaphi.de/
> > > > > > >
> > > > > > > On Mon, Oct 18, 2021 at 5:33 PM Kevin Rosendahl
> > > > > > > <kevindrosendahl@gmail.com> wrote:
> > > > > > > >
> > > > > > > > Hello,
> > > > > > > >
> > > > > > > > We are using Lucene 8 and planning to upgrade from Java 11 to
> > > Java 17.
> > > > > > We
> > > > > > > > are curious:
> > > > > > > >
> > > > > > > > - How lucene is testing against java versions. Are there
> > > correctness
> > > > > > > and
> > > > > > > > performance tests using java 17?
> > > > > > > > - Additionally, besides Java 17, how are new Java
> > releases
> > > > > > tested?
> > > > > > > > - Are there any other orgs using Java 17 with Lucene?
> > > > > > > > - Any other considerations we should be aware of?
> > > > > > > >
> > > > > > > >
> > > > > > > > Best,
> > > > > > > > Kevin Rosendahl
> > > > > > >
> > > > > > >
> > > ---------------------------------------------------------------------
> > > > > > > To unsubscribe, e-mail:
> java-user-unsubscribe@lucene.apache.org
> > > > > > > For additional commands, e-mail:
> > java-user-help@lucene.apache.org
> > > > > > >
> > > > > > >
> > > > > >
> > > >
> > > >
> > > > ---------------------------------------------------------------------
> > > > To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> > > > For additional commands, e-mail: java-user-help@lucene.apache.org
> > > >
> > >
> > > ---------------------------------------------------------------------
> > > To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> > > For additional commands, e-mail: java-user-help@lucene.apache.org
> > >
> > >
> >
>
RE: Java 17 and Lucene [ In reply to ]
Hi,

> Is this recommended "-XX:+UseZGC options to enable ZGC." as it claims very
> low pauses.

You may have seen my prvious post that JDK 16, 17 and 18 have hangs on our build server. All of those hanging builds have one thing in common: They are running with ZGC. So my answer in short: Don’t use ZGC, which is anyways not a good idea with Lucene. It reduces pauses, but on the other hand reduces throughput by >10%. So IMHO, better use G1GC and have higher throughput. With G1GC the average pauses are limited, too. But I would say, with common workloads it is better to have 10% faster queries and maybe have some of them wait 200 ms because of a pause!? If you have multiple replicas just distribute your queries and the pause will be not really visible to many people. And: Why is 200 ms response time bad if it happens seldom?

In addition: Lucene does not apply pressure to garbage collector, so use low heap space and use docvalues and other off-heap features of Lucene. Anybody running Lucene/Solr/Elasticsearch with huge heap space does something wrong!

Uwe

> For "*DY* (2021-10-19 08:14:33): Upgrade to JDK17+35" execution for
> "Indexing
> throughput
> <https://home.apache.org/~mikemccand/lucenebench/indexing.html>"
> is ZGC used for the "Indexing throughput
> <https://home.apache.org/~mikemccand/lucenebench/indexing.html>" test?
>
>
> On Wed, Oct 20, 2021 at 8:27 AM Michael McCandless <
> lucene@mikemccandless.com> wrote:
>
> > Nightly benchmarks managed to succeed (once, so far) on JDK 17:
> > https://home.apache.org/~mikemccand/lucenebench/
> >
> > No obvious performance changes on quick look.
> >
> > Mike McCandless
> >
> > http://blog.mikemccandless.com
> >
> >
> > On Tue, Oct 19, 2021 at 8:42 PM Gautam Worah
> <worah.gautam@gmail.com>
> > wrote:
> >
> > > Thanks for the note of caution Uwe.
> > >
> > > > On our Jenkins server running with AMD Ryzen CPU it happens quite often
> > > that JDK 16, JDK 17 and JDK 18 hang during tests and stay unkillable
> > (only
> > > a hard kill with" kill -9")
> > >
> > > Scary stuff.
> > > I'll try to reproduce the hang first and then try to get the JVM logs.
> > I'll
> > > respond back here if I find something useful.
> > >
> > > > Do you get this error in lucene:core:ecjLintMain and not during
> > compile?
> > > Then this is https://issues.apache.org/jira/browse/LUCENE-10185, solved
> > > already.
> > >
> > > Ahh. I should've been clearer with my comment. The error we see is
> > because
> > > we have forked the class and have modified it a bit.
> > > I just assumed that the upstream Lucene package would've also gotten
> > errors
> > > on the JDK17 build because it was untouched.
> > >
> > > -
> > > Gautam Worah.
> > >
> > >
> > > On Tue, Oct 19, 2021 at 5:07 AM Michael Sokolov <msokolov@gmail.com>
> > > wrote:
> > >
> > > > > I would a bit careful: On our Jenkins server running with AMD Ryzen
> > CPU
> > > > it happens quite often that JDK 16, JDK 17 and JDK 18 hang during tests
> > > and
> > > > stay unkillable (only a hard kill with" kill -9"). Previous Java
> > versions
> > > > don't hang. It happens not all the time (about 1/4th of all builds) and
> > > due
> > > > to the fact that the JVM is unresponsible it is not possible to get a
> > > stack
> > > > trace with "jstack". If you know a way to get the stack trace, I'd
> > happy
> > > to
> > > > get help.
> > > >
> > > > ooh that sounds scary. I suppose one could maybe get core dumps using
> > > > the right signal and debug that way? Oh wait you said only 9 works,
> > > > darn! How about attaching using gdb? Do we maintain GC logs for these
> > > > Jenkins builds? Maybe something suspicious would show up there.
> > > >
> > > > By the way the JDK is absolutely "responsible" in this situation! Not
> > > > responsive maybe ...
> > > >
> > > > On Tue, Oct 19, 2021 at 4:46 AM Uwe Schindler <uwe@thetaphi.de>
> wrote:
> > > > >
> > > > > Hi,
> > > > >
> > > > > > Hey,
> > > > > >
> > > > > > Our team at Amazon Product Search recently ran our internal
> > > benchmarks
> > > > with
> > > > > > JDK 17.
> > > > > > We saw a ~5% increase in throughput and are in the process of
> > > > > > experimenting/enabling it in production.
> > > > > > We also plan to test the new Corretto Generational Shenandoah GC.
> > > > >
> > > > > I would a bit careful: On our Jenkins server running with AMD Ryzen
> > CPU
> > > > it happens quite often that JDK 16, JDK 17 and JDK 18 hang during tests
> > > and
> > > > stay unkillable (only a hard kill with" kill -9"). Previous Java
> > versions
> > > > don't hang. It happens not all the time (about 1/4th of all builds) and
> > > due
> > > > to the fact that the JVM is unresponsible it is not possible to get a
> > > stack
> > > > trace with "jstack". If you know a way to get the stack trace, I'd
> > happy
> > > to
> > > > get help.
> > > > >
> > > > > Once I figured out what makes it hang, I will open issues in OpenJDK
> > (I
> > > > am OpenJDK member/editor). I have now many stuck JVMs running to
> > analyze
> > > on
> > > > the server, so you're invited to help! At the moment, I have no time to
> > > > take care, so any help is useful.
> > > > >
> > > > > > On a side note, the Lucene codebase still uses the deprecated (as
> > of
> > > > > > JDK17) AccessController
> > > > > > in the RamUsageEstimator class.
> > > > > > We suppressed the warning for now (based on recommendations
> > > > > > <http://mail-archives.apache.org/mod_mbox/db-derby-
> > > > > >
> dev/202106.mbox/%3CJIRA.13369440.1617476525000.615331.16239514800
> > > > > > 59@Atlassian.JIRA%3E>
> > > > > > from the Apache Derby mailing list).
> > > > >
> > > > > This should not be an issue, because we compile Lucene with javac
> > > > parameter "--release 11", so it won't show any warning that you need to
> > > > suppress. Looks like your build system at Amazon is not the original
> > one
> > > by
> > > > Lucene's Gradle, which shows no warnings at all.
> > > > >
> > > > > Uwe
> > > > >
> > > > > > Gautam Worah.
> > > > > >
> > > > > >
> > > > > > On Mon, Oct 18, 2021 at 3:02 PM Michael McCandless <
> > > > > > lucene@mikemccandless.com> wrote:
> > > > > >
> > > > > > > Also, I try to semi-aggressively upgrade Lucene's nightly
> > > benchmarks
> > > > to new
> > > > > > > JDK releases and leave an annotation on the nightly charts:
> > > > > > > https://home.apache.org/~mikemccand/lucenebench/
> > > > > > >
> > > > > > > I just now upgraded to JDK 17 and kicked off a new benchmark run
> > > ...
> > > > in a
> > > > > > > few hours it should show the new data points and then I'll try to
> > > > remember
> > > > > > > to annotate it tomorrow.
> > > > > > >
> > > > > > > So let's see whether nightly benchmarks uncover any performance
> > > > changes
> > > > > > > from JDK17 :)
> > > > > > >
> > > > > > > Mike McCandless
> > > > > > >
> > > > > > > http://blog.mikemccandless.com
> > > > > > >
> > > > > > >
> > > > > > > On Mon, Oct 18, 2021 at 5:36 PM Robert Muir <rcmuir@gmail.com>
> > > > wrote:
> > > > > > >
> > > > > > > > We test different releases on different platforms (e.g. Linux,
> > > > Windows,
> > > > > > > > Mac).
> > > > > > > > We also test EA (Early Access) releases of openjdk versions
> > > during
> > > > the
> > > > > > > > development process.
> > > > > > > > This finds bugs before they get released.
> > > > > > > >
> > > > > > > > More information about versions/EA testing:
> > > > https://jenkins.thetaphi.de/
> > > > > > > >
> > > > > > > > On Mon, Oct 18, 2021 at 5:33 PM Kevin Rosendahl
> > > > > > > > <kevindrosendahl@gmail.com> wrote:
> > > > > > > > >
> > > > > > > > > Hello,
> > > > > > > > >
> > > > > > > > > We are using Lucene 8 and planning to upgrade from Java 11 to
> > > > Java 17.
> > > > > > > We
> > > > > > > > > are curious:
> > > > > > > > >
> > > > > > > > > - How lucene is testing against java versions. Are there
> > > > correctness
> > > > > > > > and
> > > > > > > > > performance tests using java 17?
> > > > > > > > > - Additionally, besides Java 17, how are new Java
> > > releases
> > > > > > > tested?
> > > > > > > > > - Are there any other orgs using Java 17 with Lucene?
> > > > > > > > > - Any other considerations we should be aware of?
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > Best,
> > > > > > > > > Kevin Rosendahl
> > > > > > > >
> > > > > > > >
> > > > ---------------------------------------------------------------------
> > > > > > > > To unsubscribe, e-mail:
> > java-user-unsubscribe@lucene.apache.org
> > > > > > > > For additional commands, e-mail:
> > > java-user-help@lucene.apache.org
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > >
> > > > >
> > > > > ---------------------------------------------------------------------
> > > > > To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> > > > > For additional commands, e-mail: java-user-help@lucene.apache.org
> > > > >
> > > >
> > > > ---------------------------------------------------------------------
> > > > To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> > > > For additional commands, e-mail: java-user-help@lucene.apache.org
> > > >
> > > >
> > >
> >


---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Java 17 and Lucene [ In reply to ]
Uwe, thanks for pointing out that ZGC is associated with all the
pauses you've observed. I'm feeling more confident now (since we are
generally using G1GC anyway, although sometimes experimenting with
other things). Indeed GC pauses have been much less of a problem since
we started using G1 to the point we don't worry about them much now. I
will say that with most of these so-called "pauseless" collectors it
seems they can really only achieve their promise by having a lot of
spare heap available, so if 200ms pauses are unacceptable this could
be a reason to run with a larger heap.

On Tue, Oct 26, 2021 at 1:14 PM Uwe Schindler <uwe@thetaphi.de> wrote:
>
> Hi,
>
> > Is this recommended "-XX:+UseZGC options to enable ZGC." as it claims very
> > low pauses.
>
> You may have seen my prvious post that JDK 16, 17 and 18 have hangs on our build server. All of those hanging builds have one thing in common: They are running with ZGC. So my answer in short: Don’t use ZGC, which is anyways not a good idea with Lucene. It reduces pauses, but on the other hand reduces throughput by >10%. So IMHO, better use G1GC and have higher throughput. With G1GC the average pauses are limited, too. But I would say, with common workloads it is better to have 10% faster queries and maybe have some of them wait 200 ms because of a pause!? If you have multiple replicas just distribute your queries and the pause will be not really visible to many people. And: Why is 200 ms response time bad if it happens seldom?
>
> In addition: Lucene does not apply pressure to garbage collector, so use low heap space and use docvalues and other off-heap features of Lucene. Anybody running Lucene/Solr/Elasticsearch with huge heap space does something wrong!
>
> Uwe
>
> > For "*DY* (2021-10-19 08:14:33): Upgrade to JDK17+35" execution for
> > "Indexing
> > throughput
> > <https://home.apache.org/~mikemccand/lucenebench/indexing.html>"
> > is ZGC used for the "Indexing throughput
> > <https://home.apache.org/~mikemccand/lucenebench/indexing.html>" test?
> >
> >
> > On Wed, Oct 20, 2021 at 8:27 AM Michael McCandless <
> > lucene@mikemccandless.com> wrote:
> >
> > > Nightly benchmarks managed to succeed (once, so far) on JDK 17:
> > > https://home.apache.org/~mikemccand/lucenebench/
> > >
> > > No obvious performance changes on quick look.
> > >
> > > Mike McCandless
> > >
> > > http://blog.mikemccandless.com
> > >
> > >
> > > On Tue, Oct 19, 2021 at 8:42 PM Gautam Worah
> > <worah.gautam@gmail.com>
> > > wrote:
> > >
> > > > Thanks for the note of caution Uwe.
> > > >
> > > > > On our Jenkins server running with AMD Ryzen CPU it happens quite often
> > > > that JDK 16, JDK 17 and JDK 18 hang during tests and stay unkillable
> > > (only
> > > > a hard kill with" kill -9")
> > > >
> > > > Scary stuff.
> > > > I'll try to reproduce the hang first and then try to get the JVM logs.
> > > I'll
> > > > respond back here if I find something useful.
> > > >
> > > > > Do you get this error in lucene:core:ecjLintMain and not during
> > > compile?
> > > > Then this is https://issues.apache.org/jira/browse/LUCENE-10185, solved
> > > > already.
> > > >
> > > > Ahh. I should've been clearer with my comment. The error we see is
> > > because
> > > > we have forked the class and have modified it a bit.
> > > > I just assumed that the upstream Lucene package would've also gotten
> > > errors
> > > > on the JDK17 build because it was untouched.
> > > >
> > > > -
> > > > Gautam Worah.
> > > >
> > > >
> > > > On Tue, Oct 19, 2021 at 5:07 AM Michael Sokolov <msokolov@gmail.com>
> > > > wrote:
> > > >
> > > > > > I would a bit careful: On our Jenkins server running with AMD Ryzen
> > > CPU
> > > > > it happens quite often that JDK 16, JDK 17 and JDK 18 hang during tests
> > > > and
> > > > > stay unkillable (only a hard kill with" kill -9"). Previous Java
> > > versions
> > > > > don't hang. It happens not all the time (about 1/4th of all builds) and
> > > > due
> > > > > to the fact that the JVM is unresponsible it is not possible to get a
> > > > stack
> > > > > trace with "jstack". If you know a way to get the stack trace, I'd
> > > happy
> > > > to
> > > > > get help.
> > > > >
> > > > > ooh that sounds scary. I suppose one could maybe get core dumps using
> > > > > the right signal and debug that way? Oh wait you said only 9 works,
> > > > > darn! How about attaching using gdb? Do we maintain GC logs for these
> > > > > Jenkins builds? Maybe something suspicious would show up there.
> > > > >
> > > > > By the way the JDK is absolutely "responsible" in this situation! Not
> > > > > responsive maybe ...
> > > > >
> > > > > On Tue, Oct 19, 2021 at 4:46 AM Uwe Schindler <uwe@thetaphi.de>
> > wrote:
> > > > > >
> > > > > > Hi,
> > > > > >
> > > > > > > Hey,
> > > > > > >
> > > > > > > Our team at Amazon Product Search recently ran our internal
> > > > benchmarks
> > > > > with
> > > > > > > JDK 17.
> > > > > > > We saw a ~5% increase in throughput and are in the process of
> > > > > > > experimenting/enabling it in production.
> > > > > > > We also plan to test the new Corretto Generational Shenandoah GC.
> > > > > >
> > > > > > I would a bit careful: On our Jenkins server running with AMD Ryzen
> > > CPU
> > > > > it happens quite often that JDK 16, JDK 17 and JDK 18 hang during tests
> > > > and
> > > > > stay unkillable (only a hard kill with" kill -9"). Previous Java
> > > versions
> > > > > don't hang. It happens not all the time (about 1/4th of all builds) and
> > > > due
> > > > > to the fact that the JVM is unresponsible it is not possible to get a
> > > > stack
> > > > > trace with "jstack". If you know a way to get the stack trace, I'd
> > > happy
> > > > to
> > > > > get help.
> > > > > >
> > > > > > Once I figured out what makes it hang, I will open issues in OpenJDK
> > > (I
> > > > > am OpenJDK member/editor). I have now many stuck JVMs running to
> > > analyze
> > > > on
> > > > > the server, so you're invited to help! At the moment, I have no time to
> > > > > take care, so any help is useful.
> > > > > >
> > > > > > > On a side note, the Lucene codebase still uses the deprecated (as
> > > of
> > > > > > > JDK17) AccessController
> > > > > > > in the RamUsageEstimator class.
> > > > > > > We suppressed the warning for now (based on recommendations
> > > > > > > <http://mail-archives.apache.org/mod_mbox/db-derby-
> > > > > > >
> > dev/202106.mbox/%3CJIRA.13369440.1617476525000.615331.16239514800
> > > > > > > 59@Atlassian.JIRA%3E>
> > > > > > > from the Apache Derby mailing list).
> > > > > >
> > > > > > This should not be an issue, because we compile Lucene with javac
> > > > > parameter "--release 11", so it won't show any warning that you need to
> > > > > suppress. Looks like your build system at Amazon is not the original
> > > one
> > > > by
> > > > > Lucene's Gradle, which shows no warnings at all.
> > > > > >
> > > > > > Uwe
> > > > > >
> > > > > > > Gautam Worah.
> > > > > > >
> > > > > > >
> > > > > > > On Mon, Oct 18, 2021 at 3:02 PM Michael McCandless <
> > > > > > > lucene@mikemccandless.com> wrote:
> > > > > > >
> > > > > > > > Also, I try to semi-aggressively upgrade Lucene's nightly
> > > > benchmarks
> > > > > to new
> > > > > > > > JDK releases and leave an annotation on the nightly charts:
> > > > > > > > https://home.apache.org/~mikemccand/lucenebench/
> > > > > > > >
> > > > > > > > I just now upgraded to JDK 17 and kicked off a new benchmark run
> > > > ...
> > > > > in a
> > > > > > > > few hours it should show the new data points and then I'll try to
> > > > > remember
> > > > > > > > to annotate it tomorrow.
> > > > > > > >
> > > > > > > > So let's see whether nightly benchmarks uncover any performance
> > > > > changes
> > > > > > > > from JDK17 :)
> > > > > > > >
> > > > > > > > Mike McCandless
> > > > > > > >
> > > > > > > > http://blog.mikemccandless.com
> > > > > > > >
> > > > > > > >
> > > > > > > > On Mon, Oct 18, 2021 at 5:36 PM Robert Muir <rcmuir@gmail.com>
> > > > > wrote:
> > > > > > > >
> > > > > > > > > We test different releases on different platforms (e.g. Linux,
> > > > > Windows,
> > > > > > > > > Mac).
> > > > > > > > > We also test EA (Early Access) releases of openjdk versions
> > > > during
> > > > > the
> > > > > > > > > development process.
> > > > > > > > > This finds bugs before they get released.
> > > > > > > > >
> > > > > > > > > More information about versions/EA testing:
> > > > > https://jenkins.thetaphi.de/
> > > > > > > > >
> > > > > > > > > On Mon, Oct 18, 2021 at 5:33 PM Kevin Rosendahl
> > > > > > > > > <kevindrosendahl@gmail.com> wrote:
> > > > > > > > > >
> > > > > > > > > > Hello,
> > > > > > > > > >
> > > > > > > > > > We are using Lucene 8 and planning to upgrade from Java 11 to
> > > > > Java 17.
> > > > > > > > We
> > > > > > > > > > are curious:
> > > > > > > > > >
> > > > > > > > > > - How lucene is testing against java versions. Are there
> > > > > correctness
> > > > > > > > > and
> > > > > > > > > > performance tests using java 17?
> > > > > > > > > > - Additionally, besides Java 17, how are new Java
> > > > releases
> > > > > > > > tested?
> > > > > > > > > > - Are there any other orgs using Java 17 with Lucene?
> > > > > > > > > > - Any other considerations we should be aware of?
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > Best,
> > > > > > > > > > Kevin Rosendahl
> > > > > > > > >
> > > > > > > > >
> > > > > ---------------------------------------------------------------------
> > > > > > > > > To unsubscribe, e-mail:
> > > java-user-unsubscribe@lucene.apache.org
> > > > > > > > > For additional commands, e-mail:
> > > > java-user-help@lucene.apache.org
> > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > >
> > > > > >
> > > > > > ---------------------------------------------------------------------
> > > > > > To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> > > > > > For additional commands, e-mail: java-user-help@lucene.apache.org
> > > > > >
> > > > >
> > > > > ---------------------------------------------------------------------
> > > > > To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> > > > > For additional commands, e-mail: java-user-help@lucene.apache.org
> > > > >
> > > > >
> > > >
> > >
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Java 17 and Lucene [ In reply to ]
Or just keep your small heap and tweak the 200ms target to be a lower
value. It is just a JVM parameter, and it is dumb as rocks:

That 200ms constant doesn't consider how many cores you have, or how
small your heap is.

On Tue, Oct 26, 2021 at 11:42 PM Michael Sokolov <msokolov@gmail.com> wrote:
>
> Uwe, thanks for pointing out that ZGC is associated with all the
> pauses you've observed. I'm feeling more confident now (since we are
> generally using G1GC anyway, although sometimes experimenting with
> other things). Indeed GC pauses have been much less of a problem since
> we started using G1 to the point we don't worry about them much now. I
> will say that with most of these so-called "pauseless" collectors it
> seems they can really only achieve their promise by having a lot of
> spare heap available, so if 200ms pauses are unacceptable this could
> be a reason to run with a larger heap.
>
> On Tue, Oct 26, 2021 at 1:14 PM Uwe Schindler <uwe@thetaphi.de> wrote:
> >
> > Hi,
> >
> > > Is this recommended "-XX:+UseZGC options to enable ZGC." as it claims very
> > > low pauses.
> >
> > You may have seen my prvious post that JDK 16, 17 and 18 have hangs on our build server. All of those hanging builds have one thing in common: They are running with ZGC. So my answer in short: Don’t use ZGC, which is anyways not a good idea with Lucene. It reduces pauses, but on the other hand reduces throughput by >10%. So IMHO, better use G1GC and have higher throughput. With G1GC the average pauses are limited, too. But I would say, with common workloads it is better to have 10% faster queries and maybe have some of them wait 200 ms because of a pause!? If you have multiple replicas just distribute your queries and the pause will be not really visible to many people. And: Why is 200 ms response time bad if it happens seldom?
> >
> > In addition: Lucene does not apply pressure to garbage collector, so use low heap space and use docvalues and other off-heap features of Lucene. Anybody running Lucene/Solr/Elasticsearch with huge heap space does something wrong!
> >
> > Uwe
> >
> > > For "*DY* (2021-10-19 08:14:33): Upgrade to JDK17+35" execution for
> > > "Indexing
> > > throughput
> > > <https://home.apache.org/~mikemccand/lucenebench/indexing.html>"
> > > is ZGC used for the "Indexing throughput
> > > <https://home.apache.org/~mikemccand/lucenebench/indexing.html>" test?
> > >
> > >
> > > On Wed, Oct 20, 2021 at 8:27 AM Michael McCandless <
> > > lucene@mikemccandless.com> wrote:
> > >
> > > > Nightly benchmarks managed to succeed (once, so far) on JDK 17:
> > > > https://home.apache.org/~mikemccand/lucenebench/
> > > >
> > > > No obvious performance changes on quick look.
> > > >
> > > > Mike McCandless
> > > >
> > > > http://blog.mikemccandless.com
> > > >
> > > >
> > > > On Tue, Oct 19, 2021 at 8:42 PM Gautam Worah
> > > <worah.gautam@gmail.com>
> > > > wrote:
> > > >
> > > > > Thanks for the note of caution Uwe.
> > > > >
> > > > > > On our Jenkins server running with AMD Ryzen CPU it happens quite often
> > > > > that JDK 16, JDK 17 and JDK 18 hang during tests and stay unkillable
> > > > (only
> > > > > a hard kill with" kill -9")
> > > > >
> > > > > Scary stuff.
> > > > > I'll try to reproduce the hang first and then try to get the JVM logs.
> > > > I'll
> > > > > respond back here if I find something useful.
> > > > >
> > > > > > Do you get this error in lucene:core:ecjLintMain and not during
> > > > compile?
> > > > > Then this is https://issues.apache.org/jira/browse/LUCENE-10185, solved
> > > > > already.
> > > > >
> > > > > Ahh. I should've been clearer with my comment. The error we see is
> > > > because
> > > > > we have forked the class and have modified it a bit.
> > > > > I just assumed that the upstream Lucene package would've also gotten
> > > > errors
> > > > > on the JDK17 build because it was untouched.
> > > > >
> > > > > -
> > > > > Gautam Worah.
> > > > >
> > > > >
> > > > > On Tue, Oct 19, 2021 at 5:07 AM Michael Sokolov <msokolov@gmail.com>
> > > > > wrote:
> > > > >
> > > > > > > I would a bit careful: On our Jenkins server running with AMD Ryzen
> > > > CPU
> > > > > > it happens quite often that JDK 16, JDK 17 and JDK 18 hang during tests
> > > > > and
> > > > > > stay unkillable (only a hard kill with" kill -9"). Previous Java
> > > > versions
> > > > > > don't hang. It happens not all the time (about 1/4th of all builds) and
> > > > > due
> > > > > > to the fact that the JVM is unresponsible it is not possible to get a
> > > > > stack
> > > > > > trace with "jstack". If you know a way to get the stack trace, I'd
> > > > happy
> > > > > to
> > > > > > get help.
> > > > > >
> > > > > > ooh that sounds scary. I suppose one could maybe get core dumps using
> > > > > > the right signal and debug that way? Oh wait you said only 9 works,
> > > > > > darn! How about attaching using gdb? Do we maintain GC logs for these
> > > > > > Jenkins builds? Maybe something suspicious would show up there.
> > > > > >
> > > > > > By the way the JDK is absolutely "responsible" in this situation! Not
> > > > > > responsive maybe ...
> > > > > >
> > > > > > On Tue, Oct 19, 2021 at 4:46 AM Uwe Schindler <uwe@thetaphi.de>
> > > wrote:
> > > > > > >
> > > > > > > Hi,
> > > > > > >
> > > > > > > > Hey,
> > > > > > > >
> > > > > > > > Our team at Amazon Product Search recently ran our internal
> > > > > benchmarks
> > > > > > with
> > > > > > > > JDK 17.
> > > > > > > > We saw a ~5% increase in throughput and are in the process of
> > > > > > > > experimenting/enabling it in production.
> > > > > > > > We also plan to test the new Corretto Generational Shenandoah GC.
> > > > > > >
> > > > > > > I would a bit careful: On our Jenkins server running with AMD Ryzen
> > > > CPU
> > > > > > it happens quite often that JDK 16, JDK 17 and JDK 18 hang during tests
> > > > > and
> > > > > > stay unkillable (only a hard kill with" kill -9"). Previous Java
> > > > versions
> > > > > > don't hang. It happens not all the time (about 1/4th of all builds) and
> > > > > due
> > > > > > to the fact that the JVM is unresponsible it is not possible to get a
> > > > > stack
> > > > > > trace with "jstack". If you know a way to get the stack trace, I'd
> > > > happy
> > > > > to
> > > > > > get help.
> > > > > > >
> > > > > > > Once I figured out what makes it hang, I will open issues in OpenJDK
> > > > (I
> > > > > > am OpenJDK member/editor). I have now many stuck JVMs running to
> > > > analyze
> > > > > on
> > > > > > the server, so you're invited to help! At the moment, I have no time to
> > > > > > take care, so any help is useful.
> > > > > > >
> > > > > > > > On a side note, the Lucene codebase still uses the deprecated (as
> > > > of
> > > > > > > > JDK17) AccessController
> > > > > > > > in the RamUsageEstimator class.
> > > > > > > > We suppressed the warning for now (based on recommendations
> > > > > > > > <http://mail-archives.apache.org/mod_mbox/db-derby-
> > > > > > > >
> > > dev/202106.mbox/%3CJIRA.13369440.1617476525000.615331.16239514800
> > > > > > > > 59@Atlassian.JIRA%3E>
> > > > > > > > from the Apache Derby mailing list).
> > > > > > >
> > > > > > > This should not be an issue, because we compile Lucene with javac
> > > > > > parameter "--release 11", so it won't show any warning that you need to
> > > > > > suppress. Looks like your build system at Amazon is not the original
> > > > one
> > > > > by
> > > > > > Lucene's Gradle, which shows no warnings at all.
> > > > > > >
> > > > > > > Uwe
> > > > > > >
> > > > > > > > Gautam Worah.
> > > > > > > >
> > > > > > > >
> > > > > > > > On Mon, Oct 18, 2021 at 3:02 PM Michael McCandless <
> > > > > > > > lucene@mikemccandless.com> wrote:
> > > > > > > >
> > > > > > > > > Also, I try to semi-aggressively upgrade Lucene's nightly
> > > > > benchmarks
> > > > > > to new
> > > > > > > > > JDK releases and leave an annotation on the nightly charts:
> > > > > > > > > https://home.apache.org/~mikemccand/lucenebench/
> > > > > > > > >
> > > > > > > > > I just now upgraded to JDK 17 and kicked off a new benchmark run
> > > > > ...
> > > > > > in a
> > > > > > > > > few hours it should show the new data points and then I'll try to
> > > > > > remember
> > > > > > > > > to annotate it tomorrow.
> > > > > > > > >
> > > > > > > > > So let's see whether nightly benchmarks uncover any performance
> > > > > > changes
> > > > > > > > > from JDK17 :)
> > > > > > > > >
> > > > > > > > > Mike McCandless
> > > > > > > > >
> > > > > > > > > http://blog.mikemccandless.com
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > On Mon, Oct 18, 2021 at 5:36 PM Robert Muir <rcmuir@gmail.com>
> > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > We test different releases on different platforms (e.g. Linux,
> > > > > > Windows,
> > > > > > > > > > Mac).
> > > > > > > > > > We also test EA (Early Access) releases of openjdk versions
> > > > > during
> > > > > > the
> > > > > > > > > > development process.
> > > > > > > > > > This finds bugs before they get released.
> > > > > > > > > >
> > > > > > > > > > More information about versions/EA testing:
> > > > > > https://jenkins.thetaphi.de/
> > > > > > > > > >
> > > > > > > > > > On Mon, Oct 18, 2021 at 5:33 PM Kevin Rosendahl
> > > > > > > > > > <kevindrosendahl@gmail.com> wrote:
> > > > > > > > > > >
> > > > > > > > > > > Hello,
> > > > > > > > > > >
> > > > > > > > > > > We are using Lucene 8 and planning to upgrade from Java 11 to
> > > > > > Java 17.
> > > > > > > > > We
> > > > > > > > > > > are curious:
> > > > > > > > > > >
> > > > > > > > > > > - How lucene is testing against java versions. Are there
> > > > > > correctness
> > > > > > > > > > and
> > > > > > > > > > > performance tests using java 17?
> > > > > > > > > > > - Additionally, besides Java 17, how are new Java
> > > > > releases
> > > > > > > > > tested?
> > > > > > > > > > > - Are there any other orgs using Java 17 with Lucene?
> > > > > > > > > > > - Any other considerations we should be aware of?
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > Best,
> > > > > > > > > > > Kevin Rosendahl
> > > > > > > > > >
> > > > > > > > > >
> > > > > > ---------------------------------------------------------------------
> > > > > > > > > > To unsubscribe, e-mail:
> > > > java-user-unsubscribe@lucene.apache.org
> > > > > > > > > > For additional commands, e-mail:
> > > > > java-user-help@lucene.apache.org
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > ---------------------------------------------------------------------
> > > > > > > To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> > > > > > > For additional commands, e-mail: java-user-help@lucene.apache.org
> > > > > > >
> > > > > >
> > > > > > ---------------------------------------------------------------------
> > > > > > To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> > > > > > For additional commands, e-mail: java-user-help@lucene.apache.org
> > > > > >
> > > > > >
> > > > >
> > > >
> >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> > For additional commands, e-mail: java-user-help@lucene.apache.org
> >
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Java 17 and Lucene [ In reply to ]
Just circling back here with some of our (Product Search at Amazon)
findings on running Lucene with JDK 17 in production (to hopefully motivate
other teams to also upgrade their JDKs). We have not had any stability
issues or any hanging machines so far.

When we switched to JDK17 from JDK15, our production saw a 10-13.5%
decrease in p99.9 Search time, 11-13% decrease in CPU time.
This increase in performance was mostly driven by just the JDK17 runtime
(even with an older bytecode).
In general, we found that new generation GCs with low pauses caused a
10-25% drop in our throughput. This matches what Uwe described in an
earlier email.


Regards,
Gautam Worah.


On Wed, Oct 27, 2021 at 1:13 AM Robert Muir <rcmuir@gmail.com> wrote:

> Or just keep your small heap and tweak the 200ms target to be a lower
> value. It is just a JVM parameter, and it is dumb as rocks:
>
> That 200ms constant doesn't consider how many cores you have, or how
> small your heap is.
>
> On Tue, Oct 26, 2021 at 11:42 PM Michael Sokolov <msokolov@gmail.com>
> wrote:
> >
> > Uwe, thanks for pointing out that ZGC is associated with all the
> > pauses you've observed. I'm feeling more confident now (since we are
> > generally using G1GC anyway, although sometimes experimenting with
> > other things). Indeed GC pauses have been much less of a problem since
> > we started using G1 to the point we don't worry about them much now. I
> > will say that with most of these so-called "pauseless" collectors it
> > seems they can really only achieve their promise by having a lot of
> > spare heap available, so if 200ms pauses are unacceptable this could
> > be a reason to run with a larger heap.
> >
> > On Tue, Oct 26, 2021 at 1:14 PM Uwe Schindler <uwe@thetaphi.de> wrote:
> > >
> > > Hi,
> > >
> > > > Is this recommended "-XX:+UseZGC options to enable ZGC." as it
> claims very
> > > > low pauses.
> > >
> > > You may have seen my prvious post that JDK 16, 17 and 18 have hangs on
> our build server. All of those hanging builds have one thing in common:
> They are running with ZGC. So my answer in short: Don’t use ZGC, which is
> anyways not a good idea with Lucene. It reduces pauses, but on the other
> hand reduces throughput by >10%. So IMHO, better use G1GC and have higher
> throughput. With G1GC the average pauses are limited, too. But I would say,
> with common workloads it is better to have 10% faster queries and maybe
> have some of them wait 200 ms because of a pause!? If you have multiple
> replicas just distribute your queries and the pause will be not really
> visible to many people. And: Why is 200 ms response time bad if it happens
> seldom?
> > >
> > > In addition: Lucene does not apply pressure to garbage collector, so
> use low heap space and use docvalues and other off-heap features of Lucene.
> Anybody running Lucene/Solr/Elasticsearch with huge heap space does
> something wrong!
> > >
> > > Uwe
> > >
> > > > For "*DY* (2021-10-19 08:14:33): Upgrade to JDK17+35" execution for
> > > > "Indexing
> > > > throughput
> > > > <https://home.apache.org/~mikemccand/lucenebench/indexing.html>"
> > > > is ZGC used for the "Indexing throughput
> > > > <https://home.apache.org/~mikemccand/lucenebench/indexing.html>"
> test?
> > > >
> > > >
> > > > On Wed, Oct 20, 2021 at 8:27 AM Michael McCandless <
> > > > lucene@mikemccandless.com> wrote:
> > > >
> > > > > Nightly benchmarks managed to succeed (once, so far) on JDK 17:
> > > > > https://home.apache.org/~mikemccand/lucenebench/
> > > > >
> > > > > No obvious performance changes on quick look.
> > > > >
> > > > > Mike McCandless
> > > > >
> > > > > http://blog.mikemccandless.com
> > > > >
> > > > >
> > > > > On Tue, Oct 19, 2021 at 8:42 PM Gautam Worah
> > > > <worah.gautam@gmail.com>
> > > > > wrote:
> > > > >
> > > > > > Thanks for the note of caution Uwe.
> > > > > >
> > > > > > > On our Jenkins server running with AMD Ryzen CPU it happens
> quite often
> > > > > > that JDK 16, JDK 17 and JDK 18 hang during tests and stay
> unkillable
> > > > > (only
> > > > > > a hard kill with" kill -9")
> > > > > >
> > > > > > Scary stuff.
> > > > > > I'll try to reproduce the hang first and then try to get the JVM
> logs.
> > > > > I'll
> > > > > > respond back here if I find something useful.
> > > > > >
> > > > > > > Do you get this error in lucene:core:ecjLintMain and not during
> > > > > compile?
> > > > > > Then this is https://issues.apache.org/jira/browse/LUCENE-10185,
> solved
> > > > > > already.
> > > > > >
> > > > > > Ahh. I should've been clearer with my comment. The error we see
> is
> > > > > because
> > > > > > we have forked the class and have modified it a bit.
> > > > > > I just assumed that the upstream Lucene package would've also
> gotten
> > > > > errors
> > > > > > on the JDK17 build because it was untouched.
> > > > > >
> > > > > > -
> > > > > > Gautam Worah.
> > > > > >
> > > > > >
> > > > > > On Tue, Oct 19, 2021 at 5:07 AM Michael Sokolov <
> msokolov@gmail.com>
> > > > > > wrote:
> > > > > >
> > > > > > > > I would a bit careful: On our Jenkins server running with
> AMD Ryzen
> > > > > CPU
> > > > > > > it happens quite often that JDK 16, JDK 17 and JDK 18 hang
> during tests
> > > > > > and
> > > > > > > stay unkillable (only a hard kill with" kill -9"). Previous
> Java
> > > > > versions
> > > > > > > don't hang. It happens not all the time (about 1/4th of all
> builds) and
> > > > > > due
> > > > > > > to the fact that the JVM is unresponsible it is not possible
> to get a
> > > > > > stack
> > > > > > > trace with "jstack". If you know a way to get the stack trace,
> I'd
> > > > > happy
> > > > > > to
> > > > > > > get help.
> > > > > > >
> > > > > > > ooh that sounds scary. I suppose one could maybe get core
> dumps using
> > > > > > > the right signal and debug that way? Oh wait you said only 9
> works,
> > > > > > > darn! How about attaching using gdb? Do we maintain GC logs
> for these
> > > > > > > Jenkins builds? Maybe something suspicious would show up there.
> > > > > > >
> > > > > > > By the way the JDK is absolutely "responsible" in this
> situation! Not
> > > > > > > responsive maybe ...
> > > > > > >
> > > > > > > On Tue, Oct 19, 2021 at 4:46 AM Uwe Schindler <uwe@thetaphi.de
> >
> > > > wrote:
> > > > > > > >
> > > > > > > > Hi,
> > > > > > > >
> > > > > > > > > Hey,
> > > > > > > > >
> > > > > > > > > Our team at Amazon Product Search recently ran our internal
> > > > > > benchmarks
> > > > > > > with
> > > > > > > > > JDK 17.
> > > > > > > > > We saw a ~5% increase in throughput and are in the process
> of
> > > > > > > > > experimenting/enabling it in production.
> > > > > > > > > We also plan to test the new Corretto Generational
> Shenandoah GC.
> > > > > > > >
> > > > > > > > I would a bit careful: On our Jenkins server running with
> AMD Ryzen
> > > > > CPU
> > > > > > > it happens quite often that JDK 16, JDK 17 and JDK 18 hang
> during tests
> > > > > > and
> > > > > > > stay unkillable (only a hard kill with" kill -9"). Previous
> Java
> > > > > versions
> > > > > > > don't hang. It happens not all the time (about 1/4th of all
> builds) and
> > > > > > due
> > > > > > > to the fact that the JVM is unresponsible it is not possible
> to get a
> > > > > > stack
> > > > > > > trace with "jstack". If you know a way to get the stack trace,
> I'd
> > > > > happy
> > > > > > to
> > > > > > > get help.
> > > > > > > >
> > > > > > > > Once I figured out what makes it hang, I will open issues in
> OpenJDK
> > > > > (I
> > > > > > > am OpenJDK member/editor). I have now many stuck JVMs running
> to
> > > > > analyze
> > > > > > on
> > > > > > > the server, so you're invited to help! At the moment, I have
> no time to
> > > > > > > take care, so any help is useful.
> > > > > > > >
> > > > > > > > > On a side note, the Lucene codebase still uses the
> deprecated (as
> > > > > of
> > > > > > > > > JDK17) AccessController
> > > > > > > > > in the RamUsageEstimator class.
> > > > > > > > > We suppressed the warning for now (based on recommendations
> > > > > > > > > <http://mail-archives.apache.org/mod_mbox/db-derby-
> > > > > > > > >
> > > > dev/202106.mbox/%3CJIRA.13369440.1617476525000.615331.16239514800
> > > > > > > > > 59@Atlassian.JIRA%3E>
> > > > > > > > > from the Apache Derby mailing list).
> > > > > > > >
> > > > > > > > This should not be an issue, because we compile Lucene with
> javac
> > > > > > > parameter "--release 11", so it won't show any warning that
> you need to
> > > > > > > suppress. Looks like your build system at Amazon is not the
> original
> > > > > one
> > > > > > by
> > > > > > > Lucene's Gradle, which shows no warnings at all.
> > > > > > > >
> > > > > > > > Uwe
> > > > > > > >
> > > > > > > > > Gautam Worah.
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > On Mon, Oct 18, 2021 at 3:02 PM Michael McCandless <
> > > > > > > > > lucene@mikemccandless.com> wrote:
> > > > > > > > >
> > > > > > > > > > Also, I try to semi-aggressively upgrade Lucene's nightly
> > > > > > benchmarks
> > > > > > > to new
> > > > > > > > > > JDK releases and leave an annotation on the nightly
> charts:
> > > > > > > > > > https://home.apache.org/~mikemccand/lucenebench/
> > > > > > > > > >
> > > > > > > > > > I just now upgraded to JDK 17 and kicked off a new
> benchmark run
> > > > > > ...
> > > > > > > in a
> > > > > > > > > > few hours it should show the new data points and then
> I'll try to
> > > > > > > remember
> > > > > > > > > > to annotate it tomorrow.
> > > > > > > > > >
> > > > > > > > > > So let's see whether nightly benchmarks uncover any
> performance
> > > > > > > changes
> > > > > > > > > > from JDK17 :)
> > > > > > > > > >
> > > > > > > > > > Mike McCandless
> > > > > > > > > >
> > > > > > > > > > http://blog.mikemccandless.com
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > On Mon, Oct 18, 2021 at 5:36 PM Robert Muir <
> rcmuir@gmail.com>
> > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > We test different releases on different platforms
> (e.g. Linux,
> > > > > > > Windows,
> > > > > > > > > > > Mac).
> > > > > > > > > > > We also test EA (Early Access) releases of openjdk
> versions
> > > > > > during
> > > > > > > the
> > > > > > > > > > > development process.
> > > > > > > > > > > This finds bugs before they get released.
> > > > > > > > > > >
> > > > > > > > > > > More information about versions/EA testing:
> > > > > > > https://jenkins.thetaphi.de/
> > > > > > > > > > >
> > > > > > > > > > > On Mon, Oct 18, 2021 at 5:33 PM Kevin Rosendahl
> > > > > > > > > > > <kevindrosendahl@gmail.com> wrote:
> > > > > > > > > > > >
> > > > > > > > > > > > Hello,
> > > > > > > > > > > >
> > > > > > > > > > > > We are using Lucene 8 and planning to upgrade from
> Java 11 to
> > > > > > > Java 17.
> > > > > > > > > > We
> > > > > > > > > > > > are curious:
> > > > > > > > > > > >
> > > > > > > > > > > > - How lucene is testing against java versions.
> Are there
> > > > > > > correctness
> > > > > > > > > > > and
> > > > > > > > > > > > performance tests using java 17?
> > > > > > > > > > > > - Additionally, besides Java 17, how are new
> Java
> > > > > > releases
> > > > > > > > > > tested?
> > > > > > > > > > > > - Are there any other orgs using Java 17 with
> Lucene?
> > > > > > > > > > > > - Any other considerations we should be aware of?
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > Best,
> > > > > > > > > > > > Kevin Rosendahl
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > >
> ---------------------------------------------------------------------
> > > > > > > > > > > To unsubscribe, e-mail:
> > > > > java-user-unsubscribe@lucene.apache.org
> > > > > > > > > > > For additional commands, e-mail:
> > > > > > java-user-help@lucene.apache.org
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> ---------------------------------------------------------------------
> > > > > > > > To unsubscribe, e-mail:
> java-user-unsubscribe@lucene.apache.org
> > > > > > > > For additional commands, e-mail:
> java-user-help@lucene.apache.org
> > > > > > > >
> > > > > > >
> > > > > > >
> ---------------------------------------------------------------------
> > > > > > > To unsubscribe, e-mail:
> java-user-unsubscribe@lucene.apache.org
> > > > > > > For additional commands, e-mail:
> java-user-help@lucene.apache.org
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > >
> > >
> > > ---------------------------------------------------------------------
> > > To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> > > For additional commands, e-mail: java-user-help@lucene.apache.org
> > >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> > For additional commands, e-mail: java-user-help@lucene.apache.org
> >
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>