Hi there,
Hopefully this is the right audience for my question. I'm a developer
working on an effort to upgrade our Java app from Lucene 5 to Lucene 8 (or
later). While doing investigation into changes in these versions the main
thing that I'm struggling with is how to replace our current usage of
org.apache.lucene.search.Filter as we use this class pretty heavily, and
this class was previously deprecated and has been removed after version 5.
I've looked at the migration guide for Lucene 6
<https://lucene.apache.org/core/6_5_1/MIGRATE.html> and javadocs and I'm
just not understanding the intended path to migrate away from using Filter
and FilteredQuery. In the migration guide I see:
Removal of Filter and FilteredQuery (LUCENE-6301
<https://issues.apache.org/jira/browse/LUCENE-6301>,LUCENE-6583
<https://issues.apache.org/jira/browse/LUCENE-6583>)
Filter and FilteredQuery have been removed. Regular queries can be used
instead of filters as they have been optimized for the filtering case. And
you can construct a BooleanQuery with one MUST clause for the query, and
one FILTER clause for the filter in order to have similar behaviour to
FilteredQuery.
It is my understanding that in older versions of Lucene filters were
similar to queries, except they didn't participate in scoring, and I can
see now how to generate a query that doesn't apply to scoring by using a
BooleanQuery with the BooleanClause.Occur.FILTER option to effectively get
the same behavior. So that makes sense.
But another capability of the Filter class was the ability to provide a
DocIdSet to indicate which documents should be permitted in search results
Filter.getDocIdSet() :
<https://lucene.apache.org/core/5_5_0/core/org/apache/lucene/search/Filter.html#getDocIdSet(org.apache.lucene.index.LeafReaderContext,%20org.apache.lucene.util.Bits)>
public abstract DocIdSet getDocIdSet(LeafReaderContext context, Bits
acceptDocs) throws IOException;
We use this low-level capability to do filtering of documents based on
BitSets we populate in application code and then convert to DocIdSets when
running Lucene queries in certain contexts. We have an extension of the
Filter class that does exactly this, and it's pretty straighforward. Now
that Filter has been removed, is there a suggested Query implementation to
use that would provide similar behavior? I've looked at the implementations
of the Query class mentioned here:
https://lucene.apache.org/core/8_0_0/core/org/apache/lucene/search/Query.html
but I'm not seeing any that accept a DocIdSet or BitSet, or would be
relevant to the use case I described above. I've also looked at stack
overflow and other forums online to get insight into this problem but to no
avail.
If there's no existing Query implementation relevant to this use case,
would you suggest I write my own Query implementation similar to the old
FilteredQuery? Or might there be a better way to go about solving this
problem that scales better and is performant? We basically want to apply a
BitSet filter to every Lucene query that a user performs in certain
contexts. We have the ability to quickly and easily populate BitSet
instances representing all of the Lucene Doc IDs in the index, with
the bits turned on for those documents we want to be included in search
results.
If this has already been answered in a forum post, I apologize. Or if
there's a Lucene specific forum somewhere I could look at, if you could
kindly point me there, I would appreciate it.
Any help/insight is greatly appreciated.
Thanks,
Scott Robey
Hopefully this is the right audience for my question. I'm a developer
working on an effort to upgrade our Java app from Lucene 5 to Lucene 8 (or
later). While doing investigation into changes in these versions the main
thing that I'm struggling with is how to replace our current usage of
org.apache.lucene.search.Filter as we use this class pretty heavily, and
this class was previously deprecated and has been removed after version 5.
I've looked at the migration guide for Lucene 6
<https://lucene.apache.org/core/6_5_1/MIGRATE.html> and javadocs and I'm
just not understanding the intended path to migrate away from using Filter
and FilteredQuery. In the migration guide I see:
Removal of Filter and FilteredQuery (LUCENE-6301
<https://issues.apache.org/jira/browse/LUCENE-6301>,LUCENE-6583
<https://issues.apache.org/jira/browse/LUCENE-6583>)
Filter and FilteredQuery have been removed. Regular queries can be used
instead of filters as they have been optimized for the filtering case. And
you can construct a BooleanQuery with one MUST clause for the query, and
one FILTER clause for the filter in order to have similar behaviour to
FilteredQuery.
It is my understanding that in older versions of Lucene filters were
similar to queries, except they didn't participate in scoring, and I can
see now how to generate a query that doesn't apply to scoring by using a
BooleanQuery with the BooleanClause.Occur.FILTER option to effectively get
the same behavior. So that makes sense.
But another capability of the Filter class was the ability to provide a
DocIdSet to indicate which documents should be permitted in search results
Filter.getDocIdSet() :
<https://lucene.apache.org/core/5_5_0/core/org/apache/lucene/search/Filter.html#getDocIdSet(org.apache.lucene.index.LeafReaderContext,%20org.apache.lucene.util.Bits)>
public abstract DocIdSet getDocIdSet(LeafReaderContext context, Bits
acceptDocs) throws IOException;
We use this low-level capability to do filtering of documents based on
BitSets we populate in application code and then convert to DocIdSets when
running Lucene queries in certain contexts. We have an extension of the
Filter class that does exactly this, and it's pretty straighforward. Now
that Filter has been removed, is there a suggested Query implementation to
use that would provide similar behavior? I've looked at the implementations
of the Query class mentioned here:
https://lucene.apache.org/core/8_0_0/core/org/apache/lucene/search/Query.html
but I'm not seeing any that accept a DocIdSet or BitSet, or would be
relevant to the use case I described above. I've also looked at stack
overflow and other forums online to get insight into this problem but to no
avail.
If there's no existing Query implementation relevant to this use case,
would you suggest I write my own Query implementation similar to the old
FilteredQuery? Or might there be a better way to go about solving this
problem that scales better and is performant? We basically want to apply a
BitSet filter to every Lucene query that a user performs in certain
contexts. We have the ability to quickly and easily populate BitSet
instances representing all of the Lucene Doc IDs in the index, with
the bits turned on for those documents we want to be included in search
results.
If this has already been answered in a forum post, I apologize. Or if
there's a Lucene specific forum somewhere I could look at, if you could
kindly point me there, I would appreciate it.
Any help/insight is greatly appreciated.
Thanks,
Scott Robey