Mailing List Archive

Lucene 9.0.0 inconsistent index options
Hi


We have a long-standing index with some mandatory fields and some optional
fields that has been through multiple lucene upgrades without a full
rebuild and on testing out an upgrade from version 8.11.0 to 9.0.0, when
open an IndexWriter we are hitting the exception

Exception in thread "main" java.lang.IllegalArgumentException: cannot
change field "language" from index options=NONE to inconsistent index
options=DOCS
at
org.apache.lucene.index.FieldInfo.verifySameIndexOptions(FieldInfo.java:245)
at
org.apache.lucene.index.FieldInfos$FieldNumbers.verifySameSchema(FieldInfos.java:421)
at
org.apache.lucene.index.FieldInfos$FieldNumbers.addOrGet(FieldInfos.java:357)
at
org.apache.lucene.index.IndexWriter.getFieldNumberMap(IndexWriter.java:1263)
at org.apache.lucene.index.IndexWriter.<init>(IndexWriter.java:1116)

Where language is one of our optional fields.

Presumably this is at least somewhat related to "Index options can no
longer be changed dynamically" as mentioned at
https://lucene.apache.org/core/9_0_0/MIGRATE.html although it fails before
our code attempts to update the index, and we are not trying to change any
index options.

Adding some displays to IndexWriter and FieldInfos and logging rather than
throwing the exception I see

language curr=NONE, other=NONE
language curr=NONE, other=NONE
language curr=NONE, other=NONE
language curr=NONE, other=NONE
language curr=NONE, other=NONE
language curr=NONE, other=NONE
language curr=NONE, other=NONE
language curr=NONE, other=NONE
language curr=NONE, other=DOCS
language curr=NONE, other=NONE
language curr=NONE, other=NONE
language curr=NONE, other=NONE
language curr=NONE, other=NONE
language curr=NONE, other=NONE
language curr=NONE, other=NONE
language curr=NONE, other=NONE
language curr=NONE, other=NONE
language curr=NONE, other=NONE
language curr=NONE, other=DOCS
language curr=NONE, other=DOCS
language curr=NONE, other=DOCS
language curr=NONE, other=DOCS
language curr=NONE, other=DOCS
language curr=NONE, other=DOCS
language curr=NONE, other=DOCS
language curr=NONE, other=DOCS

where there is one line per segment. It logs the exception whenever
other=DOCS. Subset with segment info:

segment _x8(8.2.0):c31753/-1:[.diagnostics={timestamp=1565623850605,
lucene.version=8.2.0, java.vm.version=11.0.3+7, java.version=11.0.3,
mergeMaxNumSegments=-1, os.version=3.1.0-1.2-desktop,
java.vendor=AdoptOpenJDK, source=merge, os.arch=amd64, mergeFactor=10,
java.runtime.version=11.0.3+7,
os=Linux}]:[attributes={Lucene50StoredFieldsFormat.mode=BEST_SPEED}]

language curr=NONE, other=NONE

segment _y9(8.7.0):c43531/-1:[.diagnostics={timestamp=1604597581562,
lucene.version=8.7.0, java.vm.version=11.0.3+7, java.version=11.0.3,
mergeMaxNumSegments=-1, os.version=3.1.0-1.2-desktop,
java.vendor=AdoptOpenJDK, source=merge, os.arch=amd64, mergeFactor=10,
java.runtime.version=11.0.3+7,
os=Linux}]:[attributes={Lucene87StoredFieldsFormat.mode=BEST_SPEED}]

language curr=NONE, other=DOCS

NOT throwing java.lang.IllegalArgumentException: cannot change field
"language" from index options=NONE to inconsistent index options=DOCS


Some variation on an old-fashioned not set versus not present bug perhaps?


--
Ian.
Re: Lucene 9.0.0 inconsistent index options [ In reply to ]
This looks related to the new changes around schema validation. Lucene
now requires a field to either be absent from a document or be indexed
with the exact same options (index options, points dimensions, norms,
doc values type, etc.) as already indexed documents that also have
this field.

However it's a bug that Lucene fails to open an index that was legal
in Lucene 8. Can you file a JIRA issue?

On Mon, Dec 13, 2021 at 4:23 PM Ian Lea <ian.lea@gmail.com> wrote:
>
> Hi
>
>
> We have a long-standing index with some mandatory fields and some optional
> fields that has been through multiple lucene upgrades without a full
> rebuild and on testing out an upgrade from version 8.11.0 to 9.0.0, when
> open an IndexWriter we are hitting the exception
>
> Exception in thread "main" java.lang.IllegalArgumentException: cannot
> change field "language" from index options=NONE to inconsistent index
> options=DOCS
> at
> org.apache.lucene.index.FieldInfo.verifySameIndexOptions(FieldInfo.java:245)
> at
> org.apache.lucene.index.FieldInfos$FieldNumbers.verifySameSchema(FieldInfos.java:421)
> at
> org.apache.lucene.index.FieldInfos$FieldNumbers.addOrGet(FieldInfos.java:357)
> at
> org.apache.lucene.index.IndexWriter.getFieldNumberMap(IndexWriter.java:1263)
> at org.apache.lucene.index.IndexWriter.<init>(IndexWriter.java:1116)
>
> Where language is one of our optional fields.
>
> Presumably this is at least somewhat related to "Index options can no
> longer be changed dynamically" as mentioned at
> https://lucene.apache.org/core/9_0_0/MIGRATE.html although it fails before
> our code attempts to update the index, and we are not trying to change any
> index options.
>
> Adding some displays to IndexWriter and FieldInfos and logging rather than
> throwing the exception I see
>
> language curr=NONE, other=NONE
> language curr=NONE, other=NONE
> language curr=NONE, other=NONE
> language curr=NONE, other=NONE
> language curr=NONE, other=NONE
> language curr=NONE, other=NONE
> language curr=NONE, other=NONE
> language curr=NONE, other=NONE
> language curr=NONE, other=DOCS
> language curr=NONE, other=NONE
> language curr=NONE, other=NONE
> language curr=NONE, other=NONE
> language curr=NONE, other=NONE
> language curr=NONE, other=NONE
> language curr=NONE, other=NONE
> language curr=NONE, other=NONE
> language curr=NONE, other=NONE
> language curr=NONE, other=NONE
> language curr=NONE, other=DOCS
> language curr=NONE, other=DOCS
> language curr=NONE, other=DOCS
> language curr=NONE, other=DOCS
> language curr=NONE, other=DOCS
> language curr=NONE, other=DOCS
> language curr=NONE, other=DOCS
> language curr=NONE, other=DOCS
>
> where there is one line per segment. It logs the exception whenever
> other=DOCS. Subset with segment info:
>
> segment _x8(8.2.0):c31753/-1:[.diagnostics={timestamp=1565623850605,
> lucene.version=8.2.0, java.vm.version=11.0.3+7, java.version=11.0.3,
> mergeMaxNumSegments=-1, os.version=3.1.0-1.2-desktop,
> java.vendor=AdoptOpenJDK, source=merge, os.arch=amd64, mergeFactor=10,
> java.runtime.version=11.0.3+7,
> os=Linux}]:[attributes={Lucene50StoredFieldsFormat.mode=BEST_SPEED}]
>
> language curr=NONE, other=NONE
>
> segment _y9(8.7.0):c43531/-1:[.diagnostics={timestamp=1604597581562,
> lucene.version=8.7.0, java.vm.version=11.0.3+7, java.version=11.0.3,
> mergeMaxNumSegments=-1, os.version=3.1.0-1.2-desktop,
> java.vendor=AdoptOpenJDK, source=merge, os.arch=amd64, mergeFactor=10,
> java.runtime.version=11.0.3+7,
> os=Linux}]:[attributes={Lucene87StoredFieldsFormat.mode=BEST_SPEED}]
>
> language curr=NONE, other=DOCS
>
> NOT throwing java.lang.IllegalArgumentException: cannot change field
> "language" from index options=NONE to inconsistent index options=DOCS
>
>
> Some variation on an old-fashioned not set versus not present bug perhaps?
>
>
> --
> Ian.



--
Adrien

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Lucene 9.0.0 inconsistent index options [ In reply to ]
Strictly speaking, we could have opened an older index using Lucene 8
(say one that was created using Lucene 7, or 6) that would no longer
be valid in Lucene 9, at least according to the policy? I agree we
should try to fix this, just want to clarify the policy

On Tue, Dec 14, 2021 at 8:54 AM Adrien Grand <jpountz@gmail.com> wrote:
>
> This looks related to the new changes around schema validation. Lucene
> now requires a field to either be absent from a document or be indexed
> with the exact same options (index options, points dimensions, norms,
> doc values type, etc.) as already indexed documents that also have
> this field.
>
> However it's a bug that Lucene fails to open an index that was legal
> in Lucene 8. Can you file a JIRA issue?
>
> On Mon, Dec 13, 2021 at 4:23 PM Ian Lea <ian.lea@gmail.com> wrote:
> >
> > Hi
> >
> >
> > We have a long-standing index with some mandatory fields and some optional
> > fields that has been through multiple lucene upgrades without a full
> > rebuild and on testing out an upgrade from version 8.11.0 to 9.0.0, when
> > open an IndexWriter we are hitting the exception
> >
> > Exception in thread "main" java.lang.IllegalArgumentException: cannot
> > change field "language" from index options=NONE to inconsistent index
> > options=DOCS
> > at
> > org.apache.lucene.index.FieldInfo.verifySameIndexOptions(FieldInfo.java:245)
> > at
> > org.apache.lucene.index.FieldInfos$FieldNumbers.verifySameSchema(FieldInfos.java:421)
> > at
> > org.apache.lucene.index.FieldInfos$FieldNumbers.addOrGet(FieldInfos.java:357)
> > at
> > org.apache.lucene.index.IndexWriter.getFieldNumberMap(IndexWriter.java:1263)
> > at org.apache.lucene.index.IndexWriter.<init>(IndexWriter.java:1116)
> >
> > Where language is one of our optional fields.
> >
> > Presumably this is at least somewhat related to "Index options can no
> > longer be changed dynamically" as mentioned at
> > https://lucene.apache.org/core/9_0_0/MIGRATE.html although it fails before
> > our code attempts to update the index, and we are not trying to change any
> > index options.
> >
> > Adding some displays to IndexWriter and FieldInfos and logging rather than
> > throwing the exception I see
> >
> > language curr=NONE, other=NONE
> > language curr=NONE, other=NONE
> > language curr=NONE, other=NONE
> > language curr=NONE, other=NONE
> > language curr=NONE, other=NONE
> > language curr=NONE, other=NONE
> > language curr=NONE, other=NONE
> > language curr=NONE, other=NONE
> > language curr=NONE, other=DOCS
> > language curr=NONE, other=NONE
> > language curr=NONE, other=NONE
> > language curr=NONE, other=NONE
> > language curr=NONE, other=NONE
> > language curr=NONE, other=NONE
> > language curr=NONE, other=NONE
> > language curr=NONE, other=NONE
> > language curr=NONE, other=NONE
> > language curr=NONE, other=NONE
> > language curr=NONE, other=DOCS
> > language curr=NONE, other=DOCS
> > language curr=NONE, other=DOCS
> > language curr=NONE, other=DOCS
> > language curr=NONE, other=DOCS
> > language curr=NONE, other=DOCS
> > language curr=NONE, other=DOCS
> > language curr=NONE, other=DOCS
> >
> > where there is one line per segment. It logs the exception whenever
> > other=DOCS. Subset with segment info:
> >
> > segment _x8(8.2.0):c31753/-1:[.diagnostics={timestamp=1565623850605,
> > lucene.version=8.2.0, java.vm.version=11.0.3+7, java.version=11.0.3,
> > mergeMaxNumSegments=-1, os.version=3.1.0-1.2-desktop,
> > java.vendor=AdoptOpenJDK, source=merge, os.arch=amd64, mergeFactor=10,
> > java.runtime.version=11.0.3+7,
> > os=Linux}]:[attributes={Lucene50StoredFieldsFormat.mode=BEST_SPEED}]
> >
> > language curr=NONE, other=NONE
> >
> > segment _y9(8.7.0):c43531/-1:[.diagnostics={timestamp=1604597581562,
> > lucene.version=8.7.0, java.vm.version=11.0.3+7, java.version=11.0.3,
> > mergeMaxNumSegments=-1, os.version=3.1.0-1.2-desktop,
> > java.vendor=AdoptOpenJDK, source=merge, os.arch=amd64, mergeFactor=10,
> > java.runtime.version=11.0.3+7,
> > os=Linux}]:[attributes={Lucene87StoredFieldsFormat.mode=BEST_SPEED}]
> >
> > language curr=NONE, other=DOCS
> >
> > NOT throwing java.lang.IllegalArgumentException: cannot change field
> > "language" from index options=NONE to inconsistent index options=DOCS
> >
> >
> > Some variation on an old-fashioned not set versus not present bug perhaps?
> >
> >
> > --
> > Ian.
>
>
>
> --
> Adrien
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Lucene 9.0.0 inconsistent index options [ In reply to ]
Thanks for the response. https://issues.apache.org/jira/browse/LUCENE-10314

Will we still be able to decide, maybe years down the line, that we do want
to search on fieldX after all, and be able to change the code and reindex
the maybe small proportion of documents that have a value for fieldX
without having to create a new index from scratch? That happens.


--
Ian.


On Tue, Dec 14, 2021 at 1:54 PM Adrien Grand <jpountz@gmail.com> wrote:

> This looks related to the new changes around schema validation. Lucene
> now requires a field to either be absent from a document or be indexed
> with the exact same options (index options, points dimensions, norms,
> doc values type, etc.) as already indexed documents that also have
> this field.
>
> However it's a bug that Lucene fails to open an index that was legal
> in Lucene 8. Can you file a JIRA issue?
>
> On Mon, Dec 13, 2021 at 4:23 PM Ian Lea <ian.lea@gmail.com> wrote:
> >
> > Hi
> >
> >
> > We have a long-standing index with some mandatory fields and some
> optional
> > fields that has been through multiple lucene upgrades without a full
> > rebuild and on testing out an upgrade from version 8.11.0 to 9.0.0, when
> > open an IndexWriter we are hitting the exception
> >
> > Exception in thread "main" java.lang.IllegalArgumentException: cannot
> > change field "language" from index options=NONE to inconsistent index
> > options=DOCS
> > at
> >
> org.apache.lucene.index.FieldInfo.verifySameIndexOptions(FieldInfo.java:245)
> > at
> >
> org.apache.lucene.index.FieldInfos$FieldNumbers.verifySameSchema(FieldInfos.java:421)
> > at
> >
> org.apache.lucene.index.FieldInfos$FieldNumbers.addOrGet(FieldInfos.java:357)
> > at
> >
> org.apache.lucene.index.IndexWriter.getFieldNumberMap(IndexWriter.java:1263)
> > at
> org.apache.lucene.index.IndexWriter.<init>(IndexWriter.java:1116)
> >
> > Where language is one of our optional fields.
> >
> > Presumably this is at least somewhat related to "Index options can no
> > longer be changed dynamically" as mentioned at
> > https://lucene.apache.org/core/9_0_0/MIGRATE.html although it fails
> before
> > our code attempts to update the index, and we are not trying to change
> any
> > index options.
> >
> > Adding some displays to IndexWriter and FieldInfos and logging rather
> than
> > throwing the exception I see
> >
> > language curr=NONE, other=NONE
> > language curr=NONE, other=NONE
> > language curr=NONE, other=NONE
> > language curr=NONE, other=NONE
> > language curr=NONE, other=NONE
> > language curr=NONE, other=NONE
> > language curr=NONE, other=NONE
> > language curr=NONE, other=NONE
> > language curr=NONE, other=DOCS
> > language curr=NONE, other=NONE
> > language curr=NONE, other=NONE
> > language curr=NONE, other=NONE
> > language curr=NONE, other=NONE
> > language curr=NONE, other=NONE
> > language curr=NONE, other=NONE
> > language curr=NONE, other=NONE
> > language curr=NONE, other=NONE
> > language curr=NONE, other=NONE
> > language curr=NONE, other=DOCS
> > language curr=NONE, other=DOCS
> > language curr=NONE, other=DOCS
> > language curr=NONE, other=DOCS
> > language curr=NONE, other=DOCS
> > language curr=NONE, other=DOCS
> > language curr=NONE, other=DOCS
> > language curr=NONE, other=DOCS
> >
> > where there is one line per segment. It logs the exception whenever
> > other=DOCS. Subset with segment info:
> >
> > segment _x8(8.2.0):c31753/-1:[.diagnostics={timestamp=1565623850605,
> > lucene.version=8.2.0, java.vm.version=11.0.3+7, java.version=11.0.3,
> > mergeMaxNumSegments=-1, os.version=3.1.0-1.2-desktop,
> > java.vendor=AdoptOpenJDK, source=merge, os.arch=amd64, mergeFactor=10,
> > java.runtime.version=11.0.3+7,
> > os=Linux}]:[attributes={Lucene50StoredFieldsFormat.mode=BEST_SPEED}]
> >
> > language curr=NONE, other=NONE
> >
> > segment _y9(8.7.0):c43531/-1:[.diagnostics={timestamp=1604597581562,
> > lucene.version=8.7.0, java.vm.version=11.0.3+7, java.version=11.0.3,
> > mergeMaxNumSegments=-1, os.version=3.1.0-1.2-desktop,
> > java.vendor=AdoptOpenJDK, source=merge, os.arch=amd64, mergeFactor=10,
> > java.runtime.version=11.0.3+7,
> > os=Linux}]:[attributes={Lucene87StoredFieldsFormat.mode=BEST_SPEED}]
> >
> > language curr=NONE, other=DOCS
> >
> > NOT throwing java.lang.IllegalArgumentException: cannot change field
> > "language" from index options=NONE to inconsistent index options=DOCS
> >
> >
> > Some variation on an old-fashioned not set versus not present bug
> perhaps?
> >
> >
> > --
> > Ian.
>
>
>
> --
> Adrien
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>
Re: Lucene 9.0.0 inconsistent index options [ In reply to ]
I have a PR with a test and a possible fix for this:
https://github.com/apache/lucene/pull/12326, anyone for review?

Tomás

On Tue, Dec 14, 2021 at 7:11?AM Ian Lea <ian.lea@gmail.com> wrote:

> Thanks for the response.
> https://issues.apache.org/jira/browse/LUCENE-10314
>
> Will we still be able to decide, maybe years down the line, that we do want
> to search on fieldX after all, and be able to change the code and reindex
> the maybe small proportion of documents that have a value for fieldX
> without having to create a new index from scratch? That happens.
>
>
> --
> Ian.
>
>
> On Tue, Dec 14, 2021 at 1:54 PM Adrien Grand <jpountz@gmail.com> wrote:
>
> > This looks related to the new changes around schema validation. Lucene
> > now requires a field to either be absent from a document or be indexed
> > with the exact same options (index options, points dimensions, norms,
> > doc values type, etc.) as already indexed documents that also have
> > this field.
> >
> > However it's a bug that Lucene fails to open an index that was legal
> > in Lucene 8. Can you file a JIRA issue?
> >
> > On Mon, Dec 13, 2021 at 4:23 PM Ian Lea <ian.lea@gmail.com> wrote:
> > >
> > > Hi
> > >
> > >
> > > We have a long-standing index with some mandatory fields and some
> > optional
> > > fields that has been through multiple lucene upgrades without a full
> > > rebuild and on testing out an upgrade from version 8.11.0 to 9.0.0,
> when
> > > open an IndexWriter we are hitting the exception
> > >
> > > Exception in thread "main" java.lang.IllegalArgumentException: cannot
> > > change field "language" from index options=NONE to inconsistent index
> > > options=DOCS
> > > at
> > >
> >
> org.apache.lucene.index.FieldInfo.verifySameIndexOptions(FieldInfo.java:245)
> > > at
> > >
> >
> org.apache.lucene.index.FieldInfos$FieldNumbers.verifySameSchema(FieldInfos.java:421)
> > > at
> > >
> >
> org.apache.lucene.index.FieldInfos$FieldNumbers.addOrGet(FieldInfos.java:357)
> > > at
> > >
> >
> org.apache.lucene.index.IndexWriter.getFieldNumberMap(IndexWriter.java:1263)
> > > at
> > org.apache.lucene.index.IndexWriter.<init>(IndexWriter.java:1116)
> > >
> > > Where language is one of our optional fields.
> > >
> > > Presumably this is at least somewhat related to "Index options can no
> > > longer be changed dynamically" as mentioned at
> > > https://lucene.apache.org/core/9_0_0/MIGRATE.html although it fails
> > before
> > > our code attempts to update the index, and we are not trying to change
> > any
> > > index options.
> > >
> > > Adding some displays to IndexWriter and FieldInfos and logging rather
> > than
> > > throwing the exception I see
> > >
> > > language curr=NONE, other=NONE
> > > language curr=NONE, other=NONE
> > > language curr=NONE, other=NONE
> > > language curr=NONE, other=NONE
> > > language curr=NONE, other=NONE
> > > language curr=NONE, other=NONE
> > > language curr=NONE, other=NONE
> > > language curr=NONE, other=NONE
> > > language curr=NONE, other=DOCS
> > > language curr=NONE, other=NONE
> > > language curr=NONE, other=NONE
> > > language curr=NONE, other=NONE
> > > language curr=NONE, other=NONE
> > > language curr=NONE, other=NONE
> > > language curr=NONE, other=NONE
> > > language curr=NONE, other=NONE
> > > language curr=NONE, other=NONE
> > > language curr=NONE, other=NONE
> > > language curr=NONE, other=DOCS
> > > language curr=NONE, other=DOCS
> > > language curr=NONE, other=DOCS
> > > language curr=NONE, other=DOCS
> > > language curr=NONE, other=DOCS
> > > language curr=NONE, other=DOCS
> > > language curr=NONE, other=DOCS
> > > language curr=NONE, other=DOCS
> > >
> > > where there is one line per segment. It logs the exception whenever
> > > other=DOCS. Subset with segment info:
> > >
> > > segment _x8(8.2.0):c31753/-1:[.diagnostics={timestamp=1565623850605,
> > > lucene.version=8.2.0, java.vm.version=11.0.3+7, java.version=11.0.3,
> > > mergeMaxNumSegments=-1, os.version=3.1.0-1.2-desktop,
> > > java.vendor=AdoptOpenJDK, source=merge, os.arch=amd64, mergeFactor=10,
> > > java.runtime.version=11.0.3+7,
> > > os=Linux}]:[attributes={Lucene50StoredFieldsFormat.mode=BEST_SPEED}]
> > >
> > > language curr=NONE, other=NONE
> > >
> > > segment _y9(8.7.0):c43531/-1:[.diagnostics={timestamp=1604597581562,
> > > lucene.version=8.7.0, java.vm.version=11.0.3+7, java.version=11.0.3,
> > > mergeMaxNumSegments=-1, os.version=3.1.0-1.2-desktop,
> > > java.vendor=AdoptOpenJDK, source=merge, os.arch=amd64, mergeFactor=10,
> > > java.runtime.version=11.0.3+7,
> > > os=Linux}]:[attributes={Lucene87StoredFieldsFormat.mode=BEST_SPEED}]
> > >
> > > language curr=NONE, other=DOCS
> > >
> > > NOT throwing java.lang.IllegalArgumentException: cannot change field
> > > "language" from index options=NONE to inconsistent index options=DOCS
> > >
> > >
> > > Some variation on an old-fashioned not set versus not present bug
> > perhaps?
> > >
> > >
> > > --
> > > Ian.
> >
> >
> >
> > --
> > Adrien
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> > For additional commands, e-mail: java-user-help@lucene.apache.org
> >
> >
>