Mailing List Archive

Hi,

I had another idea: Why not release main as 10.0.0 *NOW* and create
branch_10x (with Java 17) minimum, stop working on 9.x, and move main
branch to 21?

I would be happy to remove the MmapByteBuffer directory in Java 18.

Unfortunately in Java 21 we still need a hack top compile the
MemorySegment classes because of the preview flag. And for the incubator
we also need the APIJAR files. But we can do this then without MR-JAR
unless we need a new version for Java 22, 23 of vectors. My idea would
be to patch in the api JAR during compile of "main" sourceset classes.

Uwe

Am 03.11.2023 um 13:20 schrieb Chris Hegarty:
> Hi,
>
> I would like to start the discussion and gather feedback on bumping the
> minimum Java version requirement to 21.
>
> I have no particular timeline in mind, but these kinda bumps often
> require dependency updates [*], small code refactorings, etc, and can
> take some time to plan and execute. It's best to at least have a plan
> for when, rather than if! Any bump would of course be limited to the
> _main_ branch, and therefore targeting a major Lucene release (no
> changes to branches targeting minor patch releases).
>
> I'm sure subscribers to this list are already familiar with the various
> goodies that have been added between Java 17 and 21, so I'll not
> enumerate them here, but rather callout just two particular benefits
> that I think are significant to the Lucene project.
>
> 1) Put a lower bound on the number of memory segment mmap and Panama
> Vector similarity implementations that we need to carry. This not only
> reduces maintenance cost, but avoids additional consideration and
> experimentation for performance improvements.
>
> 2) Support for half float, Float::float16ToFloat and Float::floatToFloat16,
> which will likely be beneficial in several places.
>
> More concretely, and somewhat orthogonal to the discussion of when, I
> would like to create a meta-issue capturing the prerequisites to a
> version bump.
>
> Your thoughts, comments, and feedback are very much welcome.
>
> -Chris.
>
> [*] we need at least an ECJ JDT dependency update, that supports
> Java 21, https://www.eclipse.org/lists/eclipse-dev/msg12203.html
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: dev-help@lucene.apache.org
>
--
Uwe Schindler
Achterdiek 19, D-28357 Bremen
https://www.thetaphi.de
eMail: uwe@thetaphi.de


---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org

Hi Uwe,

Thanks for your reply, comments inline.

> On 3 Nov 2023, at 13:11, Uwe Schindler <uwe@thetaphi.de> wrote:
>
> Hi,
>
> I had another idea: Why not release main as 10.0.0 *NOW* and create branch_10x (with Java 17) minimum, stop working on 9.x, and move main branch to 21?

I see now that 9.x has a minimum Java version of 11, and that _main_ has a minimum version of 17. I previously overlooked this ( I thought that 9.x was on 17, but it is not ). Ok, so your idea is actually quite inline with how things have happened in the past.

For ease of reference, here are the dates of the last 4 major releases.
9.0.0 Dec 2021
8.0.0 Mar 2019
7.0.0 Sep 2017
6.0.0 Apr 2016

If we release 10.0.0 now (with a minimum of 17) that drops the need to support Java 11 (since work in 9.x will mostly stop). I’m ok with this, and we get the benefits of dropping < Java 17. But can we be more ambitious in our approach here?

I’ll defer to others about what is in _main_ to justify a major release or not - the driver for a release should be more than just the minimum Java version.

Alternatively, what if we were to not release 10.0.0 for another while, say 3 - 6 months, and at the same time bump it to Java 21. In the meantime we can keep the 9.x updates coming. My motivation for suggesting this is that it appears that major Lucene versions seem to be around every 2 years or so, and if we release 10 with Java 17, the we’ll still be reluctant to use Java APIs and features between 17 and 21 for the next, likely, 2 years. An alternative to that is to release Lucene 11.0.0 sometime before the 2 year mark.

> I would be happy to remove the MmapByteBuffer directory in Java 18.

We can only do this when we move to a minimum Java > 17, so in your proposal that would be in _main_ some time post the fork for branch_10x. That seems ok.

> Unfortunately in Java 21 we still need a hack top compile the MemorySegment classes because of the preview flag. And for the incubator we also need the APIJAR files. But we can do this then without MR-JAR unless we need a new version for Java 22, 23 of vectors. My idea would be to patch in the api JAR during compile of "main" sourceset classes.

Yeah, regardless of the minimum version bump some work is needed here :-( Where possible we should try to minimise it, but I agree we’ll likely need updates for the vector stuff in 22+.

-Chris.

> Uwe
>
> Am 03.11.2023 um 13:20 schrieb Chris Hegarty:
>> Hi,
>>
>> I would like to start the discussion and gather feedback on bumping the
>> minimum Java version requirement to 21.
>>
>> I have no particular timeline in mind, but these kinda bumps often
>> require dependency updates [*], small code refactorings, etc, and can
>> take some time to plan and execute. It's best to at least have a plan
>> for when, rather than if! Any bump would of course be limited to the
>> _main_ branch, and therefore targeting a major Lucene release (no
>> changes to branches targeting minor patch releases).
>>
>> I'm sure subscribers to this list are already familiar with the various
>> goodies that have been added between Java 17 and 21, so I'll not
>> enumerate them here, but rather callout just two particular benefits
>> that I think are significant to the Lucene project.
>>
>> 1) Put a lower bound on the number of memory segment mmap and Panama
>> Vector similarity implementations that we need to carry. This not only
>> reduces maintenance cost, but avoids additional consideration and
>> experimentation for performance improvements.
>>
>> 2) Support for half float, Float::float16ToFloat and Float::floatToFloat16,
>> which will likely be beneficial in several places.
>>
>> More concretely, and somewhat orthogonal to the discussion of when, I
>> would like to create a meta-issue capturing the prerequisites to a
>> version bump.
>>
>> Your thoughts, comments, and feedback are very much welcome.
>>
>> -Chris.
>>
>> [*] we need at least an ECJ JDT dependency update, that supports
>> Java 21, https://www.eclipse.org/lists/eclipse-dev/msg12203.html
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: dev-help@lucene.apache.org
>>
> --
> Uwe Schindler
> Achterdiek 19, D-28357 Bremen
> https://www.thetaphi.de
> eMail: uwe@thetaphi.de
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: dev-help@lucene.apache.org
>


---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org

Hi,

thanks Chris. This is why I suggested the idea, to have the discussion
here. We are already close to Lucene 9.9. Do we want 9.10? We had that
long series of minor releases only int the 4.x branch (which ended in 4.10).

I have some comments inline:

>> On 3 Nov 2023, at 13:11, Uwe Schindler<uwe@thetaphi.de> wrote:
>>
>> Hi,
>>
>> I had another idea: Why not release main as 10.0.0 *NOW* and create branch_10x (with Java 17) minimum, stop working on 9.x, and move main branch to 21?
> I see now that 9.x has a minimum Java version of 11, and that _main_ has a minimum version of 17. I previously overlooked this ( I thought that 9.x was on 17, but it is not ). Ok, so your idea is actually quite inline with how things have happened in the past.
>
> For ease of reference, here are the dates of the last 4 major releases.
> 9.0.0 Dec 2021
> 8.0.0 Mar 2019
> 7.0.0 Sep 2017
> 6.0.0 Apr 2016
>
> If we release 10.0.0 now (with a minimum of 17) that drops the need to support Java 11 (since work in 9.x will mostly stop). I’m ok with this, and we get the benefits of dropping < Java 17. But can we be more ambitious in our approach here?
>
> I’ll defer to others about what is in _main_ to justify a major release or not - the driver for a release should be more than just the minimum Java version.
>
> Alternatively, what if we were to not release 10.0.0 for another while, say 3 - 6 months, and at the same time bump it to Java 21. In the meantime we can keep the 9.x updates coming. My motivation for suggesting this is that it appears that major Lucene versions seem to be around every 2 years or so, and if we release 10 with Java 17, the we’ll still be reluctant to use Java APIs and features between 17 and 21 for the next, likely, 2 years. An alternative to that is to release Lucene 11.0.0 sometime before the 2 year mark.
>
>> I would be happy to remove the MmapByteBuffer directory in Java 18.
> We can only do this when we move to a minimum Java > 17, so in your proposal that would be in _main_ some time post the fork for branch_10x. That seems ok.
Sorry this was a typo with version number. I meant Java 21 would no
longer require (Mapped-)ByteBufferIndexInput.
>> Unfortunately in Java 21 we still need a hack top compile the MemorySegment classes because of the preview flag. And for the incubator we also need the APIJAR files. But we can do this then without MR-JAR unless we need a new version for Java 22, 23 of vectors. My idea would be to patch in the api JAR during compile of "main" sourceset classes.
> Yeah, regardless of the minimum version bump some work is needed here :-( Where possible we should try to minimise it, but I agree we’ll likely need updates for the vector stuff in 22+.

I figured out it is not so easy, we need additional maintenance and
possibly a MR JAR also with Java 21:

* In Java 21, panama-foreign is still preview. So when compiling we
need the APIJAR.
* In the MR-JAR compilation we patch the APIJAR into the java.base
module (which we also need for incubating). The problem is: YOu
cannot patch the "java.base" module and at same time pass "--release
21". So In that code part we need to compile against actual class
library (I have no idea why patching is disallowed with --release).
It prints a cryptic error message, but makes no sense to me.
* Because of the inability to use "--release" we still need to compile
the Panama classes in a separate gradle sourceSet. But we can copy
the separate sourceSet output for 21 directly into the main JAR part
(but we can also let it live in versions/21.


This should not stop us from moving to 21, the details with how to build
the JRA/MR-JAR can be solved separately. You PR looks fine, I would keep
away from the MR-JAR sourceSets for now. We can clean the up later.

Keeping parts of the MR-JAR logic as suggested before helps with
backporting.

Uwe

>> Am 03.11.2023 um 13:20 schrieb Chris Hegarty:
>>> Hi,
>>>
>>> I would like to start the discussion and gather feedback on bumping the
>>> minimum Java version requirement to 21.
>>>
>>> I have no particular timeline in mind, but these kinda bumps often
>>> require dependency updates [*], small code refactorings, etc, and can
>>> take some time to plan and execute. It's best to at least have a plan
>>> for when, rather than if! Any bump would of course be limited to the
>>> _main_ branch, and therefore targeting a major Lucene release (no
>>> changes to branches targeting minor patch releases).
>>>
>>> I'm sure subscribers to this list are already familiar with the various
>>> goodies that have been added between Java 17 and 21, so I'll not
>>> enumerate them here, but rather callout just two particular benefits
>>> that I think are significant to the Lucene project.
>>>
>>> 1) Put a lower bound on the number of memory segment mmap and Panama
>>> Vector similarity implementations that we need to carry. This not only
>>> reduces maintenance cost, but avoids additional consideration and
>>> experimentation for performance improvements.
>>>
>>> 2) Support for half float, Float::float16ToFloat and Float::floatToFloat16,
>>> which will likely be beneficial in several places.
>>>
>>> More concretely, and somewhat orthogonal to the discussion of when, I
>>> would like to create a meta-issue capturing the prerequisites to a
>>> version bump.
>>>
>>> Your thoughts, comments, and feedback are very much welcome.
>>>
>>> -Chris.
>>>
>>> [*] we need at least an ECJ JDT dependency update, that supports
>>> Java 21,https://www.eclipse.org/lists/eclipse-dev/msg12203.html
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail:dev-unsubscribe@lucene.apache.org
>>> For additional commands, e-mail:dev-help@lucene.apache.org
>>>
>> --
>> Uwe Schindler
>> Achterdiek 19, D-28357 Bremen
>> https://www.thetaphi.de
>> eMail:uwe@thetaphi.de
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail:dev-unsubscribe@lucene.apache.org
>> For additional commands, e-mail:dev-help@lucene.apache.org
>>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail:dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail:dev-help@lucene.apache.org
>
--
Uwe Schindler
Achterdiek 19, D-28357 Bremen
https://www.thetaphi.de
eMail:uwe@thetaphi.de

Hi,

Great discussion, I agree with all that you have said. And that we will have to deal with the intricacies of the MR-JAR regardless of the outcome here, which is doable.

I would very much like to avoid supporting Java 17 (released in Sep 2021) in 2025. So far we have two possible approaches:

1. Release Lucene 10.0.0 now with Java 17 minimum. Bump _main_ to Java 21.

2. Release Lucene 9.9.0 soon(ish) with Java 11 minimum. Bump _main_ to Java 21, and release 10.0.0 in the first quarter of 2024.

Have I captured this correctly? Are there other alternatives that should be considered?

My issue with no.1 is that the 10.x train will likely live on for ~2yrs? In which case we’ll be supporting Java 17 until some time in late 2025, when Java 25 is released. This could be mitigated by releasing Lucene 11.0.0 earlier than 2yrs, say 1yr after 10.0.0.

The only concern I have with no.2 is that it could be considered an “aggressive” adoption of Java 21 - adoption sooner than the ecosystem can handle, e.g. are environments in which Lucene is deployed, and their transitive dependencies, ready to run on Java 21? By the time we’re ready to release 10.0.0, say March 2023, then I expect no issue with this.

I’m only putting dates on things to make the conversation more concrete, clearly these are all just illustrative.

-Chris.

P.S. This thread is starting to touch on the general release cadence of Lucene, which is a bigger topic than the subject line of this mail, but I think is great to think about these bumps in this context.


> On 5 Nov 2023, at 09:46, Uwe Schindler <uwe@thetaphi.de> wrote:
>
> Hi,
>
> thanks Chris. This is why I suggested the idea, to have the discussion here. We are already close to Lucene 9.9. Do we want 9.10? We had that long series of minor releases only int the 4.x branch (which ended in 4.10).
>
> I have some comments inline:
>
>
>>> On 3 Nov 2023, at 13:11, Uwe Schindler <uwe@thetaphi.de> <mailto:uwe@thetaphi.de> wrote:
>>>
>>> Hi,
>>>
>>> I had another idea: Why not release main as 10.0.0 *NOW* and create branch_10x (with Java 17) minimum, stop working on 9.x, and move main branch to 21?
>> I see now that 9.x has a minimum Java version of 11, and that _main_ has a minimum version of 17. I previously overlooked this ( I thought that 9.x was on 17, but it is not ). Ok, so your idea is actually quite inline with how things have happened in the past.
>>
>> For ease of reference, here are the dates of the last 4 major releases.
>> 9.0.0 Dec 2021
>> 8.0.0 Mar 2019
>> 7.0.0 Sep 2017
>> 6.0.0 Apr 2016
>>
>> If we release 10.0.0 now (with a minimum of 17) that drops the need to support Java 11 (since work in 9.x will mostly stop). I’m ok with this, and we get the benefits of dropping < Java 17. But can we be more ambitious in our approach here?
>>
>> I’ll defer to others about what is in _main_ to justify a major release or not - the driver for a release should be more than just the minimum Java version.
>>
>> Alternatively, what if we were to not release 10.0.0 for another while, say 3 - 6 months, and at the same time bump it to Java 21. In the meantime we can keep the 9.x updates coming. My motivation for suggesting this is that it appears that major Lucene versions seem to be around every 2 years or so, and if we release 10 with Java 17, the we’ll still be reluctant to use Java APIs and features between 17 and 21 for the next, likely, 2 years. An alternative to that is to release Lucene 11.0.0 sometime before the 2 year mark.
>>
>>> I would be happy to remove the MmapByteBuffer directory in Java 18.
>> We can only do this when we move to a minimum Java > 17, so in your proposal that would be in _main_ some time post the fork for branch_10x. That seems ok.
> Sorry this was a typo with version number. I meant Java 21 would no longer require (Mapped-)ByteBufferIndexInput.
>>> Unfortunately in Java 21 we still need a hack top compile the MemorySegment classes because of the preview flag. And for the incubator we also need the APIJAR files. But we can do this then without MR-JAR unless we need a new version for Java 22, 23 of vectors. My idea would be to patch in the api JAR during compile of "main" sourceset classes.
>> Yeah, regardless of the minimum version bump some work is needed here :-( Where possible we should try to minimise it, but I agree we’ll likely need updates for the vector stuff in 22+.
> I figured out it is not so easy, we need additional maintenance and possibly a MR JAR also with Java 21:
>
> In Java 21, panama-foreign is still preview. So when compiling we need the APIJAR.
> In the MR-JAR compilation we patch the APIJAR into the java.base module (which we also need for incubating). The problem is: YOu cannot patch the "java.base" module and at same time pass "--release 21". So In that code part we need to compile against actual class library (I have no idea why patching is disallowed with --release). It prints a cryptic error message, but makes no sense to me.
> Because of the inability to use "--release" we still need to compile the Panama classes in a separate gradle sourceSet. But we can copy the separate sourceSet output for 21 directly into the main JAR part (but we can also let it live in versions/21.
>
> This should not stop us from moving to 21, the details with how to build the JRA/MR-JAR can be solved separately. You PR looks fine, I would keep away from the MR-JAR sourceSets for now. We can clean the up later.
>
> Keeping parts of the MR-JAR logic as suggested before helps with backporting.
>
> Uwe
>
>>> Am 03.11.2023 um 13:20 schrieb Chris Hegarty:
>>>> Hi,
>>>>
>>>> I would like to start the discussion and gather feedback on bumping the
>>>> minimum Java version requirement to 21.
>>>>
>>>> I have no particular timeline in mind, but these kinda bumps often
>>>> require dependency updates [*], small code refactorings, etc, and can
>>>> take some time to plan and execute. It's best to at least have a plan
>>>> for when, rather than if! Any bump would of course be limited to the
>>>> _main_ branch, and therefore targeting a major Lucene release (no
>>>> changes to branches targeting minor patch releases).
>>>>
>>>> I'm sure subscribers to this list are already familiar with the various
>>>> goodies that have been added between Java 17 and 21, so I'll not
>>>> enumerate them here, but rather callout just two particular benefits
>>>> that I think are significant to the Lucene project.
>>>>
>>>> 1) Put a lower bound on the number of memory segment mmap and Panama
>>>> Vector similarity implementations that we need to carry. This not only
>>>> reduces maintenance cost, but avoids additional consideration and
>>>> experimentation for performance improvements.
>>>>
>>>> 2) Support for half float, Float::float16ToFloat and Float::floatToFloat16,
>>>> which will likely be beneficial in several places.
>>>>
>>>> More concretely, and somewhat orthogonal to the discussion of when, I
>>>> would like to create a meta-issue capturing the prerequisites to a
>>>> version bump.
>>>>
>>>> Your thoughts, comments, and feedback are very much welcome.
>>>>
>>>> -Chris.
>>>>
>>>> [*] we need at least an ECJ JDT dependency update, that supports
>>>> Java 21, https://www.eclipse.org/lists/eclipse-dev/msg12203.html
>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org <mailto:dev-unsubscribe@lucene.apache.org>
>>>> For additional commands, e-mail: dev-help@lucene.apache.org <mailto:dev-help@lucene.apache.org>
>>>>
>>> --
>>> Uwe Schindler
>>> Achterdiek 19, D-28357 Bremen
>>> https://www.thetaphi.de <https://www.thetaphi.de/>
>>> eMail: uwe@thetaphi.de <mailto:uwe@thetaphi.de>
>>>
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org <mailto:dev-unsubscribe@lucene.apache.org>
>>> For additional commands, e-mail: dev-help@lucene.apache.org <mailto:dev-help@lucene.apache.org>
>>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org <mailto:dev-unsubscribe@lucene.apache.org>
>> For additional commands, e-mail: dev-help@lucene.apache.org <mailto:dev-help@lucene.apache.org>
>>
> --
> Uwe Schindler
> Achterdiek 19, D-28357 Bremen
> https://www.thetaphi.de <https://www.thetaphi.de/>
> eMail: uwe@thetaphi.de <mailto:uwe@thetaphi.de>

On Mon, Nov 6, 2023 at 4:22?AM Chris Hegarty
<christopher.hegarty@elastic.co.invalid> wrote:
>
> Hi,
>
> Great discussion, I agree with all that you have said. And that we will have to deal with the intricacies of the MR-JAR regardless of the outcome here, which is doable.
>
> I would very much like to avoid supporting Java 17 (released in Sep 2021) in 2025. So far we have two possible approaches:
>
> 1. Release Lucene 10.0.0 now with Java 17 minimum. Bump _main_ to Java 21.
>
> 2. Release Lucene 9.9.0 soon(ish) with Java 11 minimum. Bump _main_ to Java 21, and release 10.0.0 in the first quarter of 2024.
>
> Have I captured this correctly? Are there other alternatives that should be considered?
>
> My issue with no.1 is that the 10.x train will likely live on for ~2yrs? In which case we’ll be supporting Java 17 until some time in late 2025, when Java 25 is released. This could be mitigated by releasing Lucene 11.0.0 earlier than 2yrs, say 1yr after 10.0.0.
>
> The only concern I have with no.2 is that it could be considered an “aggressive” adoption of Java 21 - adoption sooner than the ecosystem can handle, e.g. are environments in which Lucene is deployed, and their transitive dependencies, ready to run on Java 21? By the time we’re ready to release 10.0.0, say March 2023, then I expect no issue with this.

The problem is worse, historically jdk version X isn't adopted as a
minimum until it is already EOL. And the lucene major versions take an
eternity to get out there, code just sits in "main" branch for years
unreleased to nobody. It is really discouraging as a contributor to
contribute code that literally sits on the shelf for years, for no
good reason at all. So why delay?

The argument of "moving sooner than ecosystem can handle" is also
bogus in the same way. You mean versus the code sitting on the shelf
and being released to nobody?

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org

> > The only concern I have with no.2 is that it could be considered an “aggressive” adoption of Java 21 - adoption sooner than the ecosystem can handle, e.g. are environments in which Lucene is deployed, and their transitive dependencies, ready to run on Java 21? By the time we’re ready to release 10.0.0, say March 2023, then I expect no issue with this.
>

As an open source library from apache software foundation, with no
warranty, it is impossible to release too aggressively. Someone
doesn't like that we released version 10 because the minimum JDK
version won't run on their 486? They just keep using version 9, we
didn't hurt them by releasing 10. We can't force them to upgrade to 10
anyway.

But on the other hand, it gave a lot of other people a choice. They
get the choice to use newer code instead of no choice at all (that
code sitting on the shelf for years). Run "git blame
lucene/CHANGES.txt" if you think I am crazy. Here's a change I made
nearly two years ago, it just sits on the shelf.

84e4b85b094c lucene/CHANGES.txt (Robert Muir
2021-12-07 21:39:13 -0500 14) * LUCENE-10010: AutomatonQuery,
CompiledAutomaton, RunAutomaton, RegExp
b2e866b70366 lucene/CHANGES.txt (Robert Muir
2021-12-03 19:48:33 -0500 15) classes no longer determinize NFAs.
Instead it is the responsibility
b2e866b70366 lucene/CHANGES.txt (Robert Muir
2021-12-03 19:48:33 -0500 16) of the caller to determinize.
(Robert Muir)

I didn't backport that change, not because I am lazy, but because it
is the kind of change that deserves to be in a major release (hard to
wrap-your-head-around-type-of-change). But I didn't intend for it to
sit on the shelf for two years either.

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org

Hi Robert,

> On 6 Nov 2023, at 12:24, Robert Muir <rcmuir@gmail.com> wrote:
>
>> …
>> The only concern I have with no.2 is that it could be considered an “aggressive” adoption of Java 21 - adoption sooner than the ecosystem can handle, e.g. are environments in which Lucene is deployed, and their transitive dependencies, ready to run on Java 21? By the time we’re ready to release 10.0.0, say March 2023, then I expect no issue with this.
>
> The problem is worse, historically jdk version X isn't adopted as a
> minimum until it is already EOL. And the lucene major versions take an
> eternity to get out there, code just sits in "main" branch for years
> unreleased to nobody. It is really discouraging as a contributor to
> contribute code that literally sits on the shelf for years, for no
> good reason at all.

Agreed. I also feel discouraged by this approach too, and also wanna
avoid the “backport the world”, since it’s counterproductive.

> So why delay?
>
> The argument of "moving sooner than ecosystem can handle" is also
> bogus in the same way. You mean versus the code sitting on the shelf
> and being released to nobody?

Yes - sitting on the shelf is no good to anyone.

Ok, what I’m hearing are good arguments for releasing 10.0.0 *now*, with
a Java 17 minimum - this is what is in _main_ today.

If we do that, then we can follow up with _main_ later (after the 10.x
branch is created). That is, 1) bump _main_ to Java 21, and 2) decide
when a Lucene 11 is to be released (I would to see Lucene 11 ~1yr after
Lucene 10).

This is Uwe’s proposal, earlier in this thread.

-Chris.



---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org

For perspective, I'm still seeing java 11 as the norm for clients... 17 is
uncommon. Anything requiring 21 is likely to be difficult to sell. I am
however a small shop, and "migrating off of solr 6" and "trying out solr
cloud" is still a thing for some clients.

Just a datapoint/anecdote, possibly skewed.

On Mon, Nov 6, 2023 at 7:41?AM Chris Hegarty
<christopher.hegarty@elastic.co.invalid> wrote:

> Hi Robert,
>
> > On 6 Nov 2023, at 12:24, Robert Muir <rcmuir@gmail.com> wrote:
> >
> >> …
> >> The only concern I have with no.2 is that it could be considered an
> “aggressive” adoption of Java 21 - adoption sooner than the ecosystem can
> handle, e.g. are environments in which Lucene is deployed, and their
> transitive dependencies, ready to run on Java 21? By the time we’re ready
> to release 10.0.0, say March 2023, then I expect no issue with this.
> >
> > The problem is worse, historically jdk version X isn't adopted as a
> > minimum until it is already EOL. And the lucene major versions take an
> > eternity to get out there, code just sits in "main" branch for years
> > unreleased to nobody. It is really discouraging as a contributor to
> > contribute code that literally sits on the shelf for years, for no
> > good reason at all.
>
> Agreed. I also feel discouraged by this approach too, and also wanna
> avoid the “backport the world”, since it’s counterproductive.
>
> > So why delay?
> >
> > The argument of "moving sooner than ecosystem can handle" is also
> > bogus in the same way. You mean versus the code sitting on the shelf
> > and being released to nobody?
>
> Yes - sitting on the shelf is no good to anyone.
>
> Ok, what I’m hearing are good arguments for releasing 10.0.0 *now*, with
> a Java 17 minimum - this is what is in _main_ today.
>
> If we do that, then we can follow up with _main_ later (after the 10.x
> branch is created). That is, 1) bump _main_ to Java 21, and 2) decide
> when a Lucene 11 is to be released (I would to see Lucene 11 ~1yr after
> Lucene 10).
>
> This is Uwe’s proposal, earlier in this thread.
>
> -Chris.
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: dev-help@lucene.apache.org
>
>

--
http://www.needhamsoftware.com (work)
https://a.co/d/b2sZLD9 (my fantasy fiction book)

It's not just you - we have an internal JDK11 fork at BIG COMPANY for some
folks that can't get off the stick. To be fair it's challenging because
they have to shift all their dependencies. I think Spark was the one
mentioned by one group, but there is a JDK17-based release of Spark, so
clearly not a blocker, OTOH if you have to upgrade JDK, Lucene, Spark, who
knows what else, all at the same time, it becomes challenging. Still I
agree it's no reason to lag behind; we have to keep pushing forward
together. +1 to release 10 - easy for me to say, we need a RM to volunteer
and it will happen

On Mon, Nov 6, 2023 at 8:19?AM Gus Heck <gus.heck@gmail.com> wrote:

> For perspective, I'm still seeing java 11 as the norm for clients... 17 is
> uncommon. Anything requiring 21 is likely to be difficult to sell. I am
> however a small shop, and "migrating off of solr 6" and "trying out solr
> cloud" is still a thing for some clients.
>
> Just a datapoint/anecdote, possibly skewed.
>
> On Mon, Nov 6, 2023 at 7:41?AM Chris Hegarty
> <christopher.hegarty@elastic.co.invalid> wrote:
>
>> Hi Robert,
>>
>> > On 6 Nov 2023, at 12:24, Robert Muir <rcmuir@gmail.com> wrote:
>> >
>> >> …
>> >> The only concern I have with no.2 is that it could be considered an
>> “aggressive” adoption of Java 21 - adoption sooner than the ecosystem can
>> handle, e.g. are environments in which Lucene is deployed, and their
>> transitive dependencies, ready to run on Java 21? By the time we’re ready
>> to release 10.0.0, say March 2023, then I expect no issue with this.
>> >
>> > The problem is worse, historically jdk version X isn't adopted as a
>> > minimum until it is already EOL. And the lucene major versions take an
>> > eternity to get out there, code just sits in "main" branch for years
>> > unreleased to nobody. It is really discouraging as a contributor to
>> > contribute code that literally sits on the shelf for years, for no
>> > good reason at all.
>>
>> Agreed. I also feel discouraged by this approach too, and also wanna
>> avoid the “backport the world”, since it’s counterproductive.
>>
>> > So why delay?
>> >
>> > The argument of "moving sooner than ecosystem can handle" is also
>> > bogus in the same way. You mean versus the code sitting on the shelf
>> > and being released to nobody?
>>
>> Yes - sitting on the shelf is no good to anyone.
>>
>> Ok, what I’m hearing are good arguments for releasing 10.0.0 *now*, with
>> a Java 17 minimum - this is what is in _main_ today.
>>
>> If we do that, then we can follow up with _main_ later (after the 10.x
>> branch is created). That is, 1) bump _main_ to Java 21, and 2) decide
>> when a Lucene 11 is to be released (I would to see Lucene 11 ~1yr after
>> Lucene 10).
>>
>> This is Uwe’s proposal, earlier in this thread.
>>
>> -Chris.
>>
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: dev-help@lucene.apache.org
>>
>>
>
> --
> http://www.needhamsoftware.com (work)
> https://a.co/d/b2sZLD9 (my fantasy fiction book)
>

+1 to a 10.0 on JDK17.

There is no "agreement" anywhere to follow a 2-year cadence for major versions, even if that's been a pattern.
Adopting a new JDK with clear benefits or getting off an EOL JDK should be valid arguments for considering a new major.
If downstream wants to keep supporting 9.10.y into eternity after our 11.0 rlease, then it's open source :)

Jan

> 6. nov. 2023 kl. 13:40 skrev Chris Hegarty <christopher.hegarty@elastic.co.INVALID>:
>
> Hi Robert,
>
>> On 6 Nov 2023, at 12:24, Robert Muir <rcmuir@gmail.com> wrote:
>>
>>> …
>>> The only concern I have with no.2 is that it could be considered an “aggressive” adoption of Java 21 - adoption sooner than the ecosystem can handle, e.g. are environments in which Lucene is deployed, and their transitive dependencies, ready to run on Java 21? By the time we’re ready to release 10.0.0, say March 2023, then I expect no issue with this.
>>
>> The problem is worse, historically jdk version X isn't adopted as a
>> minimum until it is already EOL. And the lucene major versions take an
>> eternity to get out there, code just sits in "main" branch for years
>> unreleased to nobody. It is really discouraging as a contributor to
>> contribute code that literally sits on the shelf for years, for no
>> good reason at all.
>
> Agreed. I also feel discouraged by this approach too, and also wanna
> avoid the “backport the world”, since it’s counterproductive.
>
>> So why delay?
>>
>> The argument of "moving sooner than ecosystem can handle" is also
>> bogus in the same way. You mean versus the code sitting on the shelf
>> and being released to nobody?
>
> Yes - sitting on the shelf is no good to anyone.
>
> Ok, what I’m hearing are good arguments for releasing 10.0.0 *now*, with
> a Java 17 minimum - this is what is in _main_ today.
>
> If we do that, then we can follow up with _main_ later (after the 10.x
> branch is created). That is, 1) bump _main_ to Java 21, and 2) decide
> when a Lucene 11 is to be released (I would to see Lucene 11 ~1yr after
> Lucene 10).
>
> This is Uwe’s proposal, earlier in this thread.
>
> -Chris.
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: dev-help@lucene.apache.org
>


---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org

Yes, LexisNexis is running Java 11 and will probably move to Java 17 soon because of Spring Boot 3 requirements. We are running a few hundred Solr nodes, mostly 9.1. Probably a few 8.10 clusters out there.

wunder
Walter Underwood
wunder@wunderwood.org
http://observer.wunderwood.org/ (my blog)

> On Nov 6, 2023, at 5:18 AM, Gus Heck <gus.heck@gmail.com> wrote:
>
> For perspective, I'm still seeing java 11 as the norm for clients... 17 is uncommon. Anything requiring 21 is likely to be difficult to sell. I am however a small shop, and "migrating off of solr 6" and "trying out solr cloud" is still a thing for some clients.
>
> Just a datapoint/anecdote, possibly skewed.
>
> On Mon, Nov 6, 2023 at 7:41?AM Chris Hegarty <christopher.hegarty@elastic.co.invalid> wrote:
>> Hi Robert,
>>
>> > On 6 Nov 2023, at 12:24, Robert Muir <rcmuir@gmail.com <mailto:rcmuir@gmail.com>> wrote:
>> >
>> >> …
>> >> The only concern I have with no.2 is that it could be considered an “aggressive” adoption of Java 21 - adoption sooner than the ecosystem can handle, e.g. are environments in which Lucene is deployed, and their transitive dependencies, ready to run on Java 21? By the time we’re ready to release 10.0.0, say March 2023, then I expect no issue with this.
>> >
>> > The problem is worse, historically jdk version X isn't adopted as a
>> > minimum until it is already EOL. And the lucene major versions take an
>> > eternity to get out there, code just sits in "main" branch for years
>> > unreleased to nobody. It is really discouraging as a contributor to
>> > contribute code that literally sits on the shelf for years, for no
>> > good reason at all.
>>
>> Agreed. I also feel discouraged by this approach too, and also wanna
>> avoid the “backport the world”, since it’s counterproductive.
>>
>> > So why delay?
>> >
>> > The argument of "moving sooner than ecosystem can handle" is also
>> > bogus in the same way. You mean versus the code sitting on the shelf
>> > and being released to nobody?
>>
>> Yes - sitting on the shelf is no good to anyone.
>>
>> Ok, what I’m hearing are good arguments for releasing 10.0.0 *now*, with
>> a Java 17 minimum - this is what is in _main_ today.
>>
>> If we do that, then we can follow up with _main_ later (after the 10.x
>> branch is created). That is, 1) bump _main_ to Java 21, and 2) decide
>> when a Lucene 11 is to be released (I would to see Lucene 11 ~1yr after
>> Lucene 10).
>>
>> This is Uwe’s proposal, earlier in this thread.
>>
>> -Chris.
>>
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org <mailto:dev-unsubscribe@lucene.apache.org>
>> For additional commands, e-mail: dev-help@lucene.apache.org <mailto:dev-help@lucene.apache.org>
>>
>
>
> --
> http://www.needhamsoftware.com <http://www.needhamsoftware.com/> (work)
> https://a.co/d/b2sZLD9 (my fantasy fiction book)

> It's not just you - we have an internal JDK11 fork at BIG COMPANY for some
> folks that can't get off the stick.
>

The truth is - most large companies will be reluctant to upgrade unless
they see a benefit in doing so. Here, we can offer this benefit (call it a
carrot, if you mentioned the stick, Mike) - speedups to vector routines,
new directory implementations Uwe has been working on, probably more.

I'm myself fairly conservative too but I also think that those new APIs are
probably worth the investment (and potential pain) to upgrade.

Dawid

We love the performance improvements, but most of our upgrades are because of CVEs that aren’t backported. We need to upgrade Thing X to the next major version and that version requires a more recent Java.

Java versions for Solr are managed separately from the massive Java codebase, but we’d probably bump everything around the same time.

We would absolutely upgrade for a substantial performance improvement because of the number of nodes we run. Reduction in cost and better response time would make it worth it.

wunder
Walter Underwood
wunder@wunderwood.org
http://observer.wunderwood.org/ (my blog)

> On Nov 6, 2023, at 8:47 AM, Dawid Weiss <dawid.weiss@gmail.com> wrote:
>
>
>> It's not just you - we have an internal JDK11 fork at BIG COMPANY for some folks that can't get off the stick.
>
> The truth is - most large companies will be reluctant to upgrade unless they see a benefit in doing so. Here, we can offer this benefit (call it a carrot, if you mentioned the stick, Mike) - speedups to vector routines, new directory implementations Uwe has been working on, probably more.
>
> I'm myself fairly conservative too but I also think that those new APIs are probably worth the investment (and potential pain) to upgrade.
>
> Dawid