Mailing List Archive: The need for a Lucene 9.9.1 release

The need for a Lucene 9.9.1 release

Dec 9, 2023, 1:09 AM

Post #1 of 13 (271 views)

Hi,

We’ve encounter two very serious issues with the recent Lucene 9.9.0 release, both of which (even if taken by themselves) would warrant a 9.9.1. The issues are:

1. https://github.com/apache/lucene/issues/12895 - Corruption read on term dictionaries in Lucene 9.9

2. https://github.com/apache/lucene/issues/12898 - JVM SIGSEGV crash when compiling computeCommonPrefixLengthAndBuildHistogram Lucene 9.9.0

There is still a little investigation and work left to bring these issues to a point where we’re comfortable with proposing a solution. I would be hopeful that we’ll get there by early next week. If so, then a Lucene 9.9.1 release can be proposed.

Thanks,
-Chris.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org

Re: The need for a Lucene 9.9.1 release [ In reply to ]

christopher.hegarty at elastic

Dec 9, 2023, 2:12 AM

Post #2 of 13 (271 views)

Permalink

Oh, and I’m happy to be Release Manager for 9.9.1 (given my recent experience on 9.9.0)

-Chris.

> On 9 Dec 2023, at 09:09, Chris Hegarty <christopher.hegarty@elastic.co> wrote:
>
> Hi,
>
> We’ve encounter two very serious issues with the recent Lucene 9.9.0 release, both of which (even if taken by themselves) would warrant a 9.9.1. The issues are:
>
> 1. https://github.com/apache/lucene/issues/12895 - Corruption read on term dictionaries in Lucene 9.9
>
> 2. https://github.com/apache/lucene/issues/12898 - JVM SIGSEGV crash when compiling computeCommonPrefixLengthAndBuildHistogram Lucene 9.9.0
>
> There is still a little investigation and work left to bring these issues to a point where we’re comfortable with proposing a solution. I would be hopeful that we’ll get there by early next week. If so, then a Lucene 9.9.1 release can be proposed.
>
> Thanks,
> -Chris.

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org

Re: The need for a Lucene 9.9.1 release [ In reply to ]

javanna at apache

Dec 9, 2023, 2:30 AM

Post #3 of 13 (271 views)

Permalink

Based on the discussions in https://github.com/apache/lucene/issues/12895 ,
it seems like reverting the change that caused the corruption on read is
the quickest fix, so that we can speed up releasing 9.9.1. I opened a PR
for that: https://github.com/apache/lucene/pull/12899. Is there additional
testing that needs to be done to ensure that this is enough to address the
corruption?

Regarding a fix for the JVM SIGSEGV crash, how far are we from a fix that
protects Lucene from it? Should we wait for that to be included in 9.9.1?
Asking because the corruption above looks like it needs to be addressed
rather quickly. It would be great to include both, but I don't know how
long that delays 9.9.1.

Cheers
Luca

On Sat, Dec 9, 2023 at 11:13?AM Chris Hegarty
<christopher.hegarty@elastic.co.invalid> wrote:

> Oh, and I’m happy to be Release Manager for 9.9.1 (given my recent
> experience on 9.9.0)
>
> -Chris.
>
> > On 9 Dec 2023, at 09:09, Chris Hegarty <christopher.hegarty@elastic.co>
> wrote:
> >
> > Hi,
> >
> > We’ve encounter two very serious issues with the recent Lucene 9.9.0
> release, both of which (even if taken by themselves) would warrant a 9.9.1.
> The issues are:
> >
> > 1. https://github.com/apache/lucene/issues/12895 - Corruption read on
> term dictionaries in Lucene 9.9
> >
> > 2. https://github.com/apache/lucene/issues/12898 - JVM SIGSEGV crash
> when compiling computeCommonPrefixLengthAndBuildHistogram Lucene 9.9.0
> >
> > There is still a little investigation and work left to bring these
> issues to a point where we’re comfortable with proposing a solution. I
> would be hopeful that we’ll get there by early next week. If so, then a
> Lucene 9.9.1 release can be proposed.
> >
> > Thanks,
> > -Chris.
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: dev-help@lucene.apache.org
>
>

Re: The need for a Lucene 9.9.1 release [ In reply to ]

rcmuir at gmail

Dec 9, 2023, 3:31 AM

Post #4 of 13 (271 views)

Permalink

I don't understand use of the word corruption, isn't it just a bug in
intersect() that only affects wildcards etc? e.g. its not gonna merge
into new segments or impact written data in any way.

And i don't think we should rushout some bugfix release without any
test for this?

On Sat, Dec 9, 2023 at 5:30?AM Luca Cavanna <javanna@apache.org> wrote:
>
> Based on the discussions in https://github.com/apache/lucene/issues/12895 , it seems like reverting the change that caused the corruption on read is the quickest fix, so that we can speed up releasing 9.9.1. I opened a PR for that: https://github.com/apache/lucene/pull/12899. Is there additional testing that needs to be done to ensure that this is enough to address the corruption?
>
> Regarding a fix for the JVM SIGSEGV crash, how far are we from a fix that protects Lucene from it? Should we wait for that to be included in 9.9.1? Asking because the corruption above looks like it needs to be addressed rather quickly. It would be great to include both, but I don't know how long that delays 9.9.1.
>
> Cheers
> Luca
>
>
>
> On Sat, Dec 9, 2023 at 11:13?AM Chris Hegarty <christopher.hegarty@elastic.co.invalid> wrote:
>>
>> Oh, and I’m happy to be Release Manager for 9.9.1 (given my recent experience on 9.9.0)
>>
>> -Chris.
>>
>> > On 9 Dec 2023, at 09:09, Chris Hegarty <christopher.hegarty@elastic.co> wrote:
>> >
>> > Hi,
>> >
>> > We’ve encounter two very serious issues with the recent Lucene 9.9.0 release, both of which (even if taken by themselves) would warrant a 9.9.1. The issues are:
>> >
>> > 1. https://github.com/apache/lucene/issues/12895 - Corruption read on term dictionaries in Lucene 9.9
>> >
>> > 2. https://github.com/apache/lucene/issues/12898 - JVM SIGSEGV crash when compiling computeCommonPrefixLengthAndBuildHistogram Lucene 9.9.0
>> >
>> > There is still a little investigation and work left to bring these issues to a point where we’re comfortable with proposing a solution. I would be hopeful that we’ll get there by early next week. If so, then a Lucene 9.9.1 release can be proposed.
>> >
>> > Thanks,
>> > -Chris.
>>
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: dev-help@lucene.apache.org
>>

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org

Re: The need for a Lucene 9.9.1 release [ In reply to ]

javanna at apache

Dec 9, 2023, 5:01 AM

Post #5 of 13 (271 views)

Permalink

I believe your assessment that it is "only" a read problem is correct. I
can see how using the "corruption" wording may have caused confusion. It is
a severe bug though that affects multi term queries and I thought it's a
good idea to patch that, given that folks have reproduced it and found the
root cause. I agree on adding tests that cover it and not rushing anything
out, yet people upgrading to 9.9.0 are affected by it and that seems bad.
Thanks for the feedback.

On Sat, Dec 9, 2023 at 12:31?PM Robert Muir <rcmuir@gmail.com> wrote:

> I don't understand use of the word corruption, isn't it just a bug in
> intersect() that only affects wildcards etc? e.g. its not gonna merge
> into new segments or impact written data in any way.
>
> And i don't think we should rushout some bugfix release without any
> test for this?
>
> On Sat, Dec 9, 2023 at 5:30?AM Luca Cavanna <javanna@apache.org> wrote:
> >
> > Based on the discussions in
> https://github.com/apache/lucene/issues/12895 , it seems like reverting
> the change that caused the corruption on read is the quickest fix, so that
> we can speed up releasing 9.9.1. I opened a PR for that:
> https://github.com/apache/lucene/pull/12899. Is there additional testing
> that needs to be done to ensure that this is enough to address the
> corruption?
> >
> > Regarding a fix for the JVM SIGSEGV crash, how far are we from a fix
> that protects Lucene from it? Should we wait for that to be included in
> 9.9.1? Asking because the corruption above looks like it needs to be
> addressed rather quickly. It would be great to include both, but I don't
> know how long that delays 9.9.1.
> >
> > Cheers
> > Luca
> >
> >
> >
> > On Sat, Dec 9, 2023 at 11:13?AM Chris Hegarty
> <christopher.hegarty@elastic.co.invalid> wrote:
> >>
> >> Oh, and I’m happy to be Release Manager for 9.9.1 (given my recent
> experience on 9.9.0)
> >>
> >> -Chris.
> >>
> >> > On 9 Dec 2023, at 09:09, Chris Hegarty <
> christopher.hegarty@elastic.co> wrote:
> >> >
> >> > Hi,
> >> >
> >> > We’ve encounter two very serious issues with the recent Lucene 9.9.0
> release, both of which (even if taken by themselves) would warrant a 9.9.1.
> The issues are:
> >> >
> >> > 1. https://github.com/apache/lucene/issues/12895 - Corruption read
> on term dictionaries in Lucene 9.9
> >> >
> >> > 2. https://github.com/apache/lucene/issues/12898 - JVM SIGSEGV crash
> when compiling computeCommonPrefixLengthAndBuildHistogram Lucene 9.9.0
> >> >
> >> > There is still a little investigation and work left to bring these
> issues to a point where we’re comfortable with proposing a solution. I
> would be hopeful that we’ll get there by early next week. If so, then a
> Lucene 9.9.1 release can be proposed.
> >> >
> >> > Thanks,
> >> > -Chris.
> >>
> >>
> >>
> >> ---------------------------------------------------------------------
> >> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
> >> For additional commands, e-mail: dev-help@lucene.apache.org
> >>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: dev-help@lucene.apache.org
>
>

Re: The need for a Lucene 9.9.1 release [ In reply to ]

christopher.hegarty at elastic

Dec 9, 2023, 9:08 AM

Post #6 of 13 (269 views)

Permalink

FYI - I added the next bugfix version 9.9.1 to `branch_9_9`, in preparation for the upcoming bug fix release.

https://github.com/apache/lucene/commit/1617c0b3a5624adba6e7b380dfeb7fb89b8a2feb

-Chris.

> On 9 Dec 2023, at 09:09, Chris Hegarty <christopher.hegarty@elastic.co> wrote:
>
> Hi,
>
> We’ve encounter two very serious issues with the recent Lucene 9.9.0 release, both of which (even if taken by themselves) would warrant a 9.9.1. The issues are:
>
> 1. https://github.com/apache/lucene/issues/12895 - Corruption read on term dictionaries in Lucene 9.9
>
> 2. https://github.com/apache/lucene/issues/12898 - JVM SIGSEGV crash when compiling computeCommonPrefixLengthAndBuildHistogram Lucene 9.9.0
>
> There is still a little investigation and work left to bring these issues to a point where we’re comfortable with proposing a solution. I would be hopeful that we’ll get there by early next week. If so, then a Lucene 9.9.1 release can be proposed.
>
> Thanks,
> -Chris.

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org

Re: The need for a Lucene 9.9.1 release [ In reply to ]

christopher.hegarty at elastic

Dec 10, 2023, 12:46 PM

Post #7 of 13 (263 views)

Permalink

> On 9 Dec 2023, at 09:09, Chris Hegarty <christopher.hegarty@elastic.co> wrote:
>
> Hi,
>
> We’ve encounter two very serious issues with the recent Lucene 9.9.0 release, both of which (even if taken by themselves) would warrant a 9.9.1. The issues are:
>
> 1. https://github.com/apache/lucene/issues/12895 - Corruption read on term dictionaries in Lucene 9.9
>
> 2. https://github.com/apache/lucene/issues/12898 - JVM SIGSEGV crash when compiling computeCommonPrefixLengthAndBuildHistogram Lucene 9.9.0

I opened a small PR which reflows the code in computeCommonPrefixLengthAndBuildHistogram which has the affect of working around the JIT crash.

https://github.com/apache/lucene/pull/12903

-Chris.

> There is still a little investigation and work left to bring these issues to a point where we’re comfortable with proposing a solution. I would be hopeful that we’ll get there by early next week. If so, then a Lucene 9.9.1 release can be proposed.
>
> Thanks,
> -Chris.

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org

Re: The need for a Lucene 9.9.1 release [ In reply to ]

christopher.hegarty at elastic

Dec 11, 2023, 12:27 PM

Post #8 of 13 (261 views)

Permalink

Just a quick update on this...

> On 9 Dec 2023, at 09:09, Chris Hegarty <christopher.hegarty@elastic.co> wrote:
>
> Hi,
>
> We’ve encounter two very serious issues with the recent Lucene 9.9.0 release, both of which (even if taken by themselves) would warrant a 9.9.1. The issues are:
>
> 1. https://github.com/apache/lucene/issues/12895 - Corruption read on term dictionaries in Lucene 9.9

Great work has been done re-adding tests, creating a new test to reproduce, and also working on an underlying fix. It feels like we’re getting close! :-)

> 2. https://github.com/apache/lucene/issues/12898 - JVM SIGSEGV crash when compiling computeCommonPrefixLengthAndBuildHistogram Lucene 9.9.0

Merged to branch_9_9.

Once no.1 is merged, I’ll build a 9.9.1 RC1 and start a vote.

-Chris

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org

Re: The need for a Lucene 9.9.1 release [ In reply to ]

guofeng at apache

Dec 12, 2023, 4:53 AM

Post #9 of 13 (260 views)

Permalink

Heads up:

The bug fix PR (https://github.com/apache/lucene/pull/12900) has been merged to main, and backported to lucene_9x & lucene_9_9.

On 2023/12/11 20:27:48 Chris Hegarty wrote:
> Just a quick update on this...
>
> > On 9 Dec 2023, at 09:09, Chris Hegarty <christopher.hegarty@elastic.co> wrote:
> >
> > Hi,
> >
> > We’ve encounter two very serious issues with the recent Lucene 9.9.0 release, both of which (even if taken by themselves) would warrant a 9.9.1. The issues are:
> >
> > 1. https://github.com/apache/lucene/issues/12895 - Corruption read on term dictionaries in Lucene 9.9
>
> Great work has been done re-adding tests, creating a new test to reproduce, and also working on an underlying fix. It feels like we’re getting close! :-)
>
> > 2. https://github.com/apache/lucene/issues/12898 - JVM SIGSEGV crash when compiling computeCommonPrefixLengthAndBuildHistogram Lucene 9.9.0
>
> Merged to branch_9_9.
>
> Once no.1 is merged, I’ll build a 9.9.1 RC1 and start a vote.
>
> -Chris
>
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: dev-help@lucene.apache.org
>
>

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org

Re: The need for a Lucene 9.9.1 release [ In reply to ]

lucene at mikemccandless

Dec 12, 2023, 4:56 AM

Post #10 of 13 (260 views)

Permalink

Hi Chris,

I think we should also regenerate the FSTs for 9.9.1?

https://github.com/apache/lucene/pull/12924

Thanks,

Mike

On Tue, Dec 12, 2023 at 7:54?AM Guo Feng <guofeng@apache.org> wrote:

> Heads up:
>
> The bug fix PR (https://github.com/apache/lucene/pull/12900) has been
> merged to main, and backported to lucene_9x & lucene_9_9.
>
> On 2023/12/11 20:27:48 Chris Hegarty wrote:
> > Just a quick update on this...
> >
> > > On 9 Dec 2023, at 09:09, Chris Hegarty <christopher.hegarty@elastic.co>
> wrote:
> > >
> > > Hi,
> > >
> > > We’ve encounter two very serious issues with the recent Lucene 9.9.0
> release, both of which (even if taken by themselves) would warrant a 9.9.1.
> The issues are:
> > >
> > > 1. https://github.com/apache/lucene/issues/12895 - Corruption read on
> term dictionaries in Lucene 9.9
> >
> > Great work has been done re-adding tests, creating a new test to
> reproduce, and also working on an underlying fix. It feels like we’re
> getting close! :-)
> >
> > > 2. https://github.com/apache/lucene/issues/12898 - JVM SIGSEGV crash
> when compiling computeCommonPrefixLengthAndBuildHistogram Lucene 9.9.0
> >
> > Merged to branch_9_9.
> >
> > Once no.1 is merged, I’ll build a 9.9.1 RC1 and start a vote.
> >
> > -Chris
> >
> >
> >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
> > For additional commands, e-mail: dev-help@lucene.apache.org
> >
> >
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: dev-help@lucene.apache.org
>
>

Re: The need for a Lucene 9.9.1 release [ In reply to ]

christopher.hegarty at elastic

Dec 12, 2023, 6:16 AM

Post #11 of 13 (260 views)

Permalink

Hi Mike,

> On 12 Dec 2023, at 12:56, Michael McCandless <lucene@mikemccandless.com> wrote:
>
> Hi Chris,
>
> I think we should also regenerate the FSTs for 9.9.1?

Seems reasonable.

> https://github.com/apache/lucene/pull/12924

I added my comments and review on the PR.

-Chris.

> Thanks,
>
> Mike

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org

Re: The need for a Lucene 9.9.1 release [ In reply to ]

lucene at mikemccandless

Dec 12, 2023, 10:42 AM

Post #12 of 13 (255 views)

Permalink

OK this is merged now. Are there any other 9.9.1 blockers? I am trying to
pass all Monster tests but that can probably just run concurrently with
voting (optimistic concurrency!)?

Mike McCandless

http://blog.mikemccandless.com

On Tue, Dec 12, 2023 at 9:18?AM Chris Hegarty
<christopher.hegarty@elastic.co.invalid> wrote:

> Hi Mike,
>
> > On 12 Dec 2023, at 12:56, Michael McCandless <lucene@mikemccandless.com>
> wrote:
> >
> > Hi Chris,
> >
> > I think we should also regenerate the FSTs for 9.9.1?
>
> Seems reasonable.
>
> > https://github.com/apache/lucene/pull/12924
>
> I added my comments and review on the PR.
>
> -Chris.
>
> > Thanks,
> >
> > Mike
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: dev-help@lucene.apache.org
>
>

Re: The need for a Lucene 9.9.1 release [ In reply to ]

christopher.hegarty at elastic

Dec 13, 2023, 12:34 AM

Post #13 of 13 (252 views)

Permalink

Thanks,

I added a couple of 9.9.1 changelog entries, and will start the RC1 process.

-Chris.

> On 12 Dec 2023, at 18:42, Michael McCandless <lucene@mikemccandless.com> wrote:
>
> OK this is merged now. Are there any other 9.9.1 blockers? I am trying to pass all Monster tests but that can probably just run concurrently with voting (optimistic concurrency!)?
>
> Mike McCandless
>
> http://blog.mikemccandless.com
>
>
> On Tue, Dec 12, 2023 at 9:18?AM Chris Hegarty <christopher.hegarty@elastic.co.invalid> wrote:
> Hi Mike,
>
> > On 12 Dec 2023, at 12:56, Michael McCandless <lucene@mikemccandless.com> wrote:
> >
> > Hi Chris,
> >
> > I think we should also regenerate the FSTs for 9.9.1?
>
> Seems reasonable.
>
> > https://github.com/apache/lucene/pull/12924
>
> I added my comments and review on the PR.
>
> -Chris.
>
> > Thanks,
> >
> > Mike
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: dev-help@lucene.apache.org
>

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org