Mailing List Archive: How to highlight fields that are not stored?

Hi,
I am converting my application from
reading documents into memory, then indexing the documents
to streaming the documents to be indexed.

I quickly found out this required that the field NOT be stored.
I then quickly found out that my highlighting code requires the field to be stored.

I’ve been searching for an existing highlighter that doesn’t require the field to be stored,
and thought I’d found one in the FastVectorHighlighter,
but tests revealed this highlighter also requires the field to be stored,
though this requirement isn’t documented, or reflected in any returned exception.

I have been investigating using code like
Terms terms = reader.getTermVector(docID, fieldName);
TermsEnum termsEnum = terms.iterator();
BytesRef bytesRef = termsEnum.next();
PostingsEnum pe = termsEnum.postings(null, PostingsEnum.OFFSETS);

While this gives me the terms from the document, and the positions,
iterating over this, and matching to the queries I’m running,
seems cumbersome, and inefficient.

Any suggestions for highlighting query matches without the searched field being stored?

Thanks,
David Shifflett
Senior Lead Technologist
Enterprise Cross Domain Solutions (ECDS)
Booz Allen Hamilton

Sorry your problem statement makes no sense: you should be able to
store field data in the index without loading all your documents into
RAM while indexing. Maybe there is some constraint you are not telling
us about? Or you may be confused. In any case highlighting requires
the document in its uninverted form. Otherwise what text would you
highlight?

On Mon, Feb 13, 2023 at 3:46 PM Shifflett, David [USA]
<Shifflett_David@bah.com.invalid> wrote:
>
> Hi,
> I am converting my application from
> reading documents into memory, then indexing the documents
> to streaming the documents to be indexed.
>
> I quickly found out this required that the field NOT be stored.
> I then quickly found out that my highlighting code requires the field to be stored.
>
> I’ve been searching for an existing highlighter that doesn’t require the field to be stored,
> and thought I’d found one in the FastVectorHighlighter,
> but tests revealed this highlighter also requires the field to be stored,
> though this requirement isn’t documented, or reflected in any returned exception.
>
> I have been investigating using code like
> Terms terms = reader.getTermVector(docID, fieldName);
> TermsEnum termsEnum = terms.iterator();
> BytesRef bytesRef = termsEnum.next();
> PostingsEnum pe = termsEnum.postings(null, PostingsEnum.OFFSETS);
>
> While this gives me the terms from the document, and the positions,
> iterating over this, and matching to the queries I’m running,
> seems cumbersome, and inefficient.
>
> Any suggestions for highlighting query matches without the searched field being stored?
>
> Thanks,
> David Shifflett
> Senior Lead Technologist
> Enterprise Cross Domain Solutions (ECDS)
> Booz Allen Hamilton
>

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org