Mailing List Archive

Hi

I noticed that Lucene 9.1.0 supports filtering in nearest-neighbor
vector search, which is great :-)

I have found

http://url7093.wyona.com/ls/click?upn=JOH5Fjdv9AA9sbvUyiP84WWONyl36e4Tdd3VZFG-2B7pcYPJTPhVT3xqtcUDjPgQX5jI0WYWlJZX8h9NDC6okDRg-3D-3DHvvY_UMWFA-2BOn91WS4mEQPCWI9gZNzEZlJPmWPGP2CeMD7g4c-2Fpo3g6VPyd4ghH4X9o8sJ-2Bl292KOe2-2F30WmSZB1KHnF6KpgvICPKY8k5m30V-2FWrDvoSQLIaimtz2YHrzSMNV98es7-2BeXQS174-2B0EHPnQVtOYqUPojoZgGkqmovRXrz1dJlfs9dtFDGqSjpYaFiMaVBoiDTrpJ-2FUkuanwcx6R8UgvxBCq08DUa7vhRJqir7M-3D
http://url7093.wyona.com/ls/click?upn=JOH5Fjdv9AA9sbvUyiP84WWONyl36e4Tdd3VZFG-2B7pe76-2Fa6XWGARorEmYO8A-2BeVhPN2B1iPvCEp9XG8WpVE6w-3D-3DHR5k_UMWFA-2BOn91WS4mEQPCWI9gZNzEZlJPmWPGP2CeMD7g4c-2Fpo3g6VPyd4ghH4X9o8sJ-2Bl292KOe2-2F30WmSZB1KHiGuZh3RFDkfg6-2FfNpOG8Tly-2FHwK25rp-2F24-2BPYZV6e5mGIWVQ5bpJ0l9u4lRzO6rySncTjxQZEPOjzZIrDMh-2Fo17VBGBagQ0Gr6G-2BAySO2ZdtDBthWjE-2F7HwxioFG9XrEUBVS79a0mcaaPKM-2BdzT9Cc-3D

and

http://url7093.wyona.com/ls/click?upn=JOH5Fjdv9AA9sbvUyiP84Rv2gN4NEUQqv-2Fn4lbJOY6mZGeN60klU5tyssLGPfHHB3IBl2Fx9C7un4UF2pBgYYcEd15H8F-2FPYEn4LTL-2Bz8fMFeo4z-2BjB3yMGv345VDdnStdESYCXN-2FD-2BQOSZSTNQLbQ-3D-3D9uTB_UMWFA-2BOn91WS4mEQPCWI9gZNzEZlJPmWPGP2CeMD7g4c-2Fpo3g6VPyd4ghH4X9o8sJ-2Bl292KOe2-2F30WmSZB1KHkzFWMImz7LKaiDu0g-2BZTsPclbKiyBoQiJHrZiOk5CuKsixeOFVVfvwAjyEhV-2F5McxrC76Q-2F72ILNowoPMFyMwXdaUF-2FhFh6HF0aWgai16l9zSdZIETAq46vRruPFO9ZqlRY6XiSu-2FBiKe4r5xiM0vA-3D

but it is not really clear to me how to use it.

Is there some additional documentation / example / demo how I can
combine filtering with vector search?

Thanks

Michael



---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org

sorry for the URLs below.

I have tested Twilio SendGrid as outgoing server and it just rewrote the
URLs


https://issues.apache.org/jira/browse/SOLR-15947
https://issues.apache.org/jira/browse/LUCENE-10382

and

https://lucene.apache.org/core/9_1_0/demo/org/apache/lucene/demo/knn/package-summary.html

which I was not aware of, but disabled the tracking now and hope it will
be ok now.

Thanks

Michael

Am 09.05.22 um 15:12 schrieb Michael Wechner:
> Hi
>
> I noticed that Lucene 9.1.0 supports filtering in nearest-neighbor
> vector search, which is great :-)
>
> I have found
>
> http://url7093.wyona.com/ls/click?upn=JOH5Fjdv9AA9sbvUyiP84WWONyl36e4Tdd3VZFG-2B7pcYPJTPhVT3xqtcUDjPgQX5jI0WYWlJZX8h9NDC6okDRg-3D-3DHvvY_UMWFA-2BOn91WS4mEQPCWI9gZNzEZlJPmWPGP2CeMD7g4c-2Fpo3g6VPyd4ghH4X9o8sJ-2Bl292KOe2-2F30WmSZB1KHnF6KpgvICPKY8k5m30V-2FWrDvoSQLIaimtz2YHrzSMNV98es7-2BeXQS174-2B0EHPnQVtOYqUPojoZgGkqmovRXrz1dJlfs9dtFDGqSjpYaFiMaVBoiDTrpJ-2FUkuanwcx6R8UgvxBCq08DUa7vhRJqir7M-3D
>
> http://url7093.wyona.com/ls/click?upn=JOH5Fjdv9AA9sbvUyiP84WWONyl36e4Tdd3VZFG-2B7pe76-2Fa6XWGARorEmYO8A-2BeVhPN2B1iPvCEp9XG8WpVE6w-3D-3DHR5k_UMWFA-2BOn91WS4mEQPCWI9gZNzEZlJPmWPGP2CeMD7g4c-2Fpo3g6VPyd4ghH4X9o8sJ-2Bl292KOe2-2F30WmSZB1KHiGuZh3RFDkfg6-2FfNpOG8Tly-2FHwK25rp-2F24-2BPYZV6e5mGIWVQ5bpJ0l9u4lRzO6rySncTjxQZEPOjzZIrDMh-2Fo17VBGBagQ0Gr6G-2BAySO2ZdtDBthWjE-2F7HwxioFG9XrEUBVS79a0mcaaPKM-2BdzT9Cc-3D
>
>
> and
>
> http://url7093.wyona.com/ls/click?upn=JOH5Fjdv9AA9sbvUyiP84Rv2gN4NEUQqv-2Fn4lbJOY6mZGeN60klU5tyssLGPfHHB3IBl2Fx9C7un4UF2pBgYYcEd15H8F-2FPYEn4LTL-2Bz8fMFeo4z-2BjB3yMGv345VDdnStdESYCXN-2FD-2BQOSZSTNQLbQ-3D-3D9uTB_UMWFA-2BOn91WS4mEQPCWI9gZNzEZlJPmWPGP2CeMD7g4c-2Fpo3g6VPyd4ghH4X9o8sJ-2Bl292KOe2-2F30WmSZB1KHkzFWMImz7LKaiDu0g-2BZTsPclbKiyBoQiJHrZiOk5CuKsixeOFVVfvwAjyEhV-2F5McxrC76Q-2F72ILNowoPMFyMwXdaUF-2FhFh6HF0aWgai16l9zSdZIETAq46vRruPFO9ZqlRY6XiSu-2FBiKe4r5xiM0vA-3D
>
>
> but it is not really clear to me how to use it.
>
> Is there some additional documentation / example / demo how I can
> combine filtering with vector search?
>
> Thanks
>
> Michael
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>


---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org

Hello Michael, we don't yet have an example of kNN with filtering in the
demo package. This would be nice to add!

For now maybe looking at the unit tests could give a sense of how to use
it. Here's an example:
https://github.com/apache/lucene/blob/main/lucene/core/src/test/org/apache/lucene/search/TestKnnVectorQuery.java#L115-L127.
The idea is that KnnVectorQuery optionally accepts a Query as a filter, and
returns the k nearest vectors that also match the filter. Many people refer
to this as "kNN with prefiltering" (as opposed to "postfiltering", where
the filter is applied *after* the kNN search, so in the end you may receive
fewer than k matches).

Let us know if you run into any questions/ issues while trying it out!

Julie

On Mon, May 9, 2022 at 8:08 AM Michael Wechner <michael.wechner@wyona.com>
wrote:

> sorry for the URLs below.
>
> I have tested Twilio SendGrid as outgoing server and it just rewrote the
> URLs
>
>
> https://issues.apache.org/jira/browse/SOLR-15947
> https://issues.apache.org/jira/browse/LUCENE-10382
>
> and
>
>
> https://lucene.apache.org/core/9_1_0/demo/org/apache/lucene/demo/knn/package-summary.html
>
> which I was not aware of, but disabled the tracking now and hope it will
> be ok now.
>
> Thanks
>
> Michael
>
> Am 09.05.22 um 15:12 schrieb Michael Wechner:
> > Hi
> >
> > I noticed that Lucene 9.1.0 supports filtering in nearest-neighbor
> > vector search, which is great :-)
> >
> > I have found
> >
> >
> http://url7093.wyona.com/ls/click?upn=JOH5Fjdv9AA9sbvUyiP84WWONyl36e4Tdd3VZFG-2B7pcYPJTPhVT3xqtcUDjPgQX5jI0WYWlJZX8h9NDC6okDRg-3D-3DHvvY_UMWFA-2BOn91WS4mEQPCWI9gZNzEZlJPmWPGP2CeMD7g4c-2Fpo3g6VPyd4ghH4X9o8sJ-2Bl292KOe2-2F30WmSZB1KHnF6KpgvICPKY8k5m30V-2FWrDvoSQLIaimtz2YHrzSMNV98es7-2BeXQS174-2B0EHPnQVtOYqUPojoZgGkqmovRXrz1dJlfs9dtFDGqSjpYaFiMaVBoiDTrpJ-2FUkuanwcx6R8UgvxBCq08DUa7vhRJqir7M-3D
> >
> >
> http://url7093.wyona.com/ls/click?upn=JOH5Fjdv9AA9sbvUyiP84WWONyl36e4Tdd3VZFG-2B7pe76-2Fa6XWGARorEmYO8A-2BeVhPN2B1iPvCEp9XG8WpVE6w-3D-3DHR5k_UMWFA-2BOn91WS4mEQPCWI9gZNzEZlJPmWPGP2CeMD7g4c-2Fpo3g6VPyd4ghH4X9o8sJ-2Bl292KOe2-2F30WmSZB1KHiGuZh3RFDkfg6-2FfNpOG8Tly-2FHwK25rp-2F24-2BPYZV6e5mGIWVQ5bpJ0l9u4lRzO6rySncTjxQZEPOjzZIrDMh-2Fo17VBGBagQ0Gr6G-2BAySO2ZdtDBthWjE-2F7HwxioFG9XrEUBVS79a0mcaaPKM-2BdzT9Cc-3D
> >
> >
> > and
> >
> >
> http://url7093.wyona.com/ls/click?upn=JOH5Fjdv9AA9sbvUyiP84Rv2gN4NEUQqv-2Fn4lbJOY6mZGeN60klU5tyssLGPfHHB3IBl2Fx9C7un4UF2pBgYYcEd15H8F-2FPYEn4LTL-2Bz8fMFeo4z-2BjB3yMGv345VDdnStdESYCXN-2FD-2BQOSZSTNQLbQ-3D-3D9uTB_UMWFA-2BOn91WS4mEQPCWI9gZNzEZlJPmWPGP2CeMD7g4c-2Fpo3g6VPyd4ghH4X9o8sJ-2Bl292KOe2-2F30WmSZB1KHkzFWMImz7LKaiDu0g-2BZTsPclbKiyBoQiJHrZiOk5CuKsixeOFVVfvwAjyEhV-2F5McxrC76Q-2F72ILNowoPMFyMwXdaUF-2FhFh6HF0aWgai16l9zSdZIETAq46vRruPFO9ZqlRY6XiSu-2FBiKe4r5xiM0vA-3D
> >
> >
> > but it is not really clear to me how to use it.
> >
> > Is there some additional documentation / example / demo how I can
> > combine filtering with vector search?
> >
> > Thanks
> >
> > Michael
> >
> >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> > For additional commands, e-mail: java-user-help@lucene.apache.org
> >
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>

Hi Julie

Cool, thanks!

I try to apply it and if it works could create an example to the demo
package.

Will keep you posted :-)

Thanks

Michael

Am 11.05.22 um 02:13 schrieb Julie Tibshirani:
> Hello Michael, we don't yet have an example of kNN with filtering in the
> demo package. This would be nice to add!
>
> For now maybe looking at the unit tests could give a sense of how to use
> it. Here's an example:
> https://github.com/apache/lucene/blob/main/lucene/core/src/test/org/apache/lucene/search/TestKnnVectorQuery.java#L115-L127.
> The idea is that KnnVectorQuery optionally accepts a Query as a filter, and
> returns the k nearest vectors that also match the filter. Many people refer
> to this as "kNN with prefiltering" (as opposed to "postfiltering", where
> the filter is applied *after* the kNN search, so in the end you may receive
> fewer than k matches).
>
> Let us know if you run into any questions/ issues while trying it out!
>
> Julie
>
> On Mon, May 9, 2022 at 8:08 AM Michael Wechner <michael.wechner@wyona.com>
> wrote:
>
>> sorry for the URLs below.
>>
>> I have tested Twilio SendGrid as outgoing server and it just rewrote the
>> URLs
>>
>>
>> https://issues.apache.org/jira/browse/SOLR-15947
>> https://issues.apache.org/jira/browse/LUCENE-10382
>>
>> and
>>
>>
>> https://lucene.apache.org/core/9_1_0/demo/org/apache/lucene/demo/knn/package-summary.html
>>
>> which I was not aware of, but disabled the tracking now and hope it will
>> be ok now.
>>
>> Thanks
>>
>> Michael
>>
>> Am 09.05.22 um 15:12 schrieb Michael Wechner:
>>> Hi
>>>
>>> I noticed that Lucene 9.1.0 supports filtering in nearest-neighbor
>>> vector search, which is great :-)
>>>
>>> I have found
>>>
>>>
>> http://url7093.wyona.com/ls/click?upn=JOH5Fjdv9AA9sbvUyiP84WWONyl36e4Tdd3VZFG-2B7pcYPJTPhVT3xqtcUDjPgQX5jI0WYWlJZX8h9NDC6okDRg-3D-3DHvvY_UMWFA-2BOn91WS4mEQPCWI9gZNzEZlJPmWPGP2CeMD7g4c-2Fpo3g6VPyd4ghH4X9o8sJ-2Bl292KOe2-2F30WmSZB1KHnF6KpgvICPKY8k5m30V-2FWrDvoSQLIaimtz2YHrzSMNV98es7-2BeXQS174-2B0EHPnQVtOYqUPojoZgGkqmovRXrz1dJlfs9dtFDGqSjpYaFiMaVBoiDTrpJ-2FUkuanwcx6R8UgvxBCq08DUa7vhRJqir7M-3D
>>>
>> http://url7093.wyona.com/ls/click?upn=JOH5Fjdv9AA9sbvUyiP84WWONyl36e4Tdd3VZFG-2B7pe76-2Fa6XWGARorEmYO8A-2BeVhPN2B1iPvCEp9XG8WpVE6w-3D-3DHR5k_UMWFA-2BOn91WS4mEQPCWI9gZNzEZlJPmWPGP2CeMD7g4c-2Fpo3g6VPyd4ghH4X9o8sJ-2Bl292KOe2-2F30WmSZB1KHiGuZh3RFDkfg6-2FfNpOG8Tly-2FHwK25rp-2F24-2BPYZV6e5mGIWVQ5bpJ0l9u4lRzO6rySncTjxQZEPOjzZIrDMh-2Fo17VBGBagQ0Gr6G-2BAySO2ZdtDBthWjE-2F7HwxioFG9XrEUBVS79a0mcaaPKM-2BdzT9Cc-3D
>>>
>>> and
>>>
>>>
>> http://url7093.wyona.com/ls/click?upn=JOH5Fjdv9AA9sbvUyiP84Rv2gN4NEUQqv-2Fn4lbJOY6mZGeN60klU5tyssLGPfHHB3IBl2Fx9C7un4UF2pBgYYcEd15H8F-2FPYEn4LTL-2Bz8fMFeo4z-2BjB3yMGv345VDdnStdESYCXN-2FD-2BQOSZSTNQLbQ-3D-3D9uTB_UMWFA-2BOn91WS4mEQPCWI9gZNzEZlJPmWPGP2CeMD7g4c-2Fpo3g6VPyd4ghH4X9o8sJ-2Bl292KOe2-2F30WmSZB1KHkzFWMImz7LKaiDu0g-2BZTsPclbKiyBoQiJHrZiOk5CuKsixeOFVVfvwAjyEhV-2F5McxrC76Q-2F72ILNowoPMFyMwXdaUF-2FhFh6HF0aWgai16l9zSdZIETAq46vRruPFO9ZqlRY6XiSu-2FBiKe4r5xiM0vA-3D
>>>
>>> but it is not really clear to me how to use it.
>>>
>>> Is there some additional documentation / example / demo how I can
>>> combine filtering with vector search?
>>>
>>> Thanks
>>>
>>> Michael
>>>
>>>
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>
>>


---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org

Hi Julie

I got it running and it seems to work fine so far :-)

Re an example for the demo package, I guess this would go here

https://lucene.apache.org/core/9_1_0/demo/org/apache/lucene/demo/knn/package-summary.html

and I thought of something like "KnnVectorPreFilterQuery.java"

Query preFilterQuery =new TermQuery(new Term(TOPIC_FIELD,"general"));
if (filter !=null) {
log.info("Filter applied before the vector search: " + preFilterQuery);
}
Query query =new KnnVectorQuery(VECTOR_FIELD, queryVector, k, preFilterQuery);

TopDocs topDocs = searcher.search(query, k);

Does that make sense to you?

Thanks

Michael

Am 11.05.22 um 07:59 schrieb Michael Wechner:
> Hi Julie
>
> Cool, thanks!
>
> I try to apply it and if it works could create an example to the demo
> package.
>
> Will keep you posted :-)
>
> Thanks
>
> Michael
>
> Am 11.05.22 um 02:13 schrieb Julie Tibshirani:
>> Hello Michael, we don't yet have an example of kNN with filtering in the
>> demo package. This would be nice to add!
>>
>> For now maybe looking at the unit tests could give a sense of how to use
>> it. Here's an example:
>> https://github.com/apache/lucene/blob/main/lucene/core/src/test/org/apache/lucene/search/TestKnnVectorQuery.java#L115-L127.
>>
>> The idea is that KnnVectorQuery optionally accepts a Query as a
>> filter, and
>> returns the k nearest vectors that also match the filter. Many people
>> refer
>> to this as "kNN with prefiltering" (as opposed to "postfiltering", where
>> the filter is applied *after* the kNN search, so in the end you may
>> receive
>> fewer than k matches).
>>
>> Let us know if you run into any questions/ issues while trying it out!
>>
>> Julie
>>
>> On Mon, May 9, 2022 at 8:08 AM Michael Wechner
>> <michael.wechner@wyona.com>
>> wrote:
>>
>>> sorry for the URLs below.
>>>
>>> I have tested Twilio SendGrid as outgoing server and it just rewrote
>>> the
>>> URLs
>>>
>>>
>>> https://issues.apache.org/jira/browse/SOLR-15947
>>> https://issues.apache.org/jira/browse/LUCENE-10382
>>>
>>> and
>>>
>>>
>>> https://lucene.apache.org/core/9_1_0/demo/org/apache/lucene/demo/knn/package-summary.html
>>>
>>>
>>> which I was not aware of, but disabled the tracking now and hope it
>>> will
>>> be ok now.
>>>
>>> Thanks
>>>
>>> Michael
>>>
>>> Am 09.05.22 um 15:12 schrieb Michael Wechner:
>>>> Hi
>>>>
>>>> I noticed that Lucene 9.1.0 supports filtering in nearest-neighbor
>>>> vector search, which is great :-)
>>>>
>>>> I have found
>>>>
>>>>
>>> http://url7093.wyona.com/ls/click?upn=JOH5Fjdv9AA9sbvUyiP84WWONyl36e4Tdd3VZFG-2B7pcYPJTPhVT3xqtcUDjPgQX5jI0WYWlJZX8h9NDC6okDRg-3D-3DHvvY_UMWFA-2BOn91WS4mEQPCWI9gZNzEZlJPmWPGP2CeMD7g4c-2Fpo3g6VPyd4ghH4X9o8sJ-2Bl292KOe2-2F30WmSZB1KHnF6KpgvICPKY8k5m30V-2FWrDvoSQLIaimtz2YHrzSMNV98es7-2BeXQS174-2B0EHPnQVtOYqUPojoZgGkqmovRXrz1dJlfs9dtFDGqSjpYaFiMaVBoiDTrpJ-2FUkuanwcx6R8UgvxBCq08DUa7vhRJqir7M-3D
>>>
>>>>
>>> http://url7093.wyona.com/ls/click?upn=JOH5Fjdv9AA9sbvUyiP84WWONyl36e4Tdd3VZFG-2B7pe76-2Fa6XWGARorEmYO8A-2BeVhPN2B1iPvCEp9XG8WpVE6w-3D-3DHR5k_UMWFA-2BOn91WS4mEQPCWI9gZNzEZlJPmWPGP2CeMD7g4c-2Fpo3g6VPyd4ghH4X9o8sJ-2Bl292KOe2-2F30WmSZB1KHiGuZh3RFDkfg6-2FfNpOG8Tly-2FHwK25rp-2F24-2BPYZV6e5mGIWVQ5bpJ0l9u4lRzO6rySncTjxQZEPOjzZIrDMh-2Fo17VBGBagQ0Gr6G-2BAySO2ZdtDBthWjE-2F7HwxioFG9XrEUBVS79a0mcaaPKM-2BdzT9Cc-3D
>>>
>>>>
>>>> and
>>>>
>>>>
>>> http://url7093.wyona.com/ls/click?upn=JOH5Fjdv9AA9sbvUyiP84Rv2gN4NEUQqv-2Fn4lbJOY6mZGeN60klU5tyssLGPfHHB3IBl2Fx9C7un4UF2pBgYYcEd15H8F-2FPYEn4LTL-2Bz8fMFeo4z-2BjB3yMGv345VDdnStdESYCXN-2FD-2BQOSZSTNQLbQ-3D-3D9uTB_UMWFA-2BOn91WS4mEQPCWI9gZNzEZlJPmWPGP2CeMD7g4c-2Fpo3g6VPyd4ghH4X9o8sJ-2Bl292KOe2-2F30WmSZB1KHkzFWMImz7LKaiDu0g-2BZTsPclbKiyBoQiJHrZiOk5CuKsixeOFVVfvwAjyEhV-2F5McxrC76Q-2F72ILNowoPMFyMwXdaUF-2FhFh6HF0aWgai16l9zSdZIETAq46vRruPFO9ZqlRY6XiSu-2FBiKe4r5xiM0vA-3D
>>>
>>>>
>>>> but it is not really clear to me how to use it.
>>>>
>>>> Is there some additional documentation / example / demo how I can
>>>> combine filtering with vector search?
>>>>
>>>> Thanks
>>>>
>>>> Michael
>>>>
>>>>
>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>>>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>>>
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>>
>>>
>

Thanks Julie. I was able to implement vector search in Zulia with your
pointers. The pull request might be helpful to others:
https://github.com/zuliaio/zuliasearch/pull/70

Thanks,
Matt

On Fri, May 20, 2022 at 9:23 AM Michael Wechner <michael.wechner@wyona.com>
wrote:

> Hi Julie
>
> I got it running and it seems to work fine so far :-)
>
> Re an example for the demo package, I guess this would go here
>
>
> https://lucene.apache.org/core/9_1_0/demo/org/apache/lucene/demo/knn/package-summary.html
>
> and I thought of something like "KnnVectorPreFilterQuery.java"
>
> Query preFilterQuery =new TermQuery(new Term(TOPIC_FIELD,"general"));
> if (filter !=null) {
> log.info("Filter applied before the vector search: " +
> preFilterQuery);
> }
> Query query =new KnnVectorQuery(VECTOR_FIELD, queryVector, k,
> preFilterQuery);
>
> TopDocs topDocs = searcher.search(query, k);
>
> Does that make sense to you?
>
> Thanks
>
> Michael
>
> Am 11.05.22 um 07:59 schrieb Michael Wechner:
> > Hi Julie
> >
> > Cool, thanks!
> >
> > I try to apply it and if it works could create an example to the demo
> > package.
> >
> > Will keep you posted :-)
> >
> > Thanks
> >
> > Michael
> >
> > Am 11.05.22 um 02:13 schrieb Julie Tibshirani:
> >> Hello Michael, we don't yet have an example of kNN with filtering in the
> >> demo package. This would be nice to add!
> >>
> >> For now maybe looking at the unit tests could give a sense of how to use
> >> it. Here's an example:
> >>
> https://github.com/apache/lucene/blob/main/lucene/core/src/test/org/apache/lucene/search/TestKnnVectorQuery.java#L115-L127.
>
> >>
> >> The idea is that KnnVectorQuery optionally accepts a Query as a
> >> filter, and
> >> returns the k nearest vectors that also match the filter. Many people
> >> refer
> >> to this as "kNN with prefiltering" (as opposed to "postfiltering", where
> >> the filter is applied *after* the kNN search, so in the end you may
> >> receive
> >> fewer than k matches).
> >>
> >> Let us know if you run into any questions/ issues while trying it out!
> >>
> >> Julie
> >>
> >> On Mon, May 9, 2022 at 8:08 AM Michael Wechner
> >> <michael.wechner@wyona.com>
> >> wrote:
> >>
> >>> sorry for the URLs below.
> >>>
> >>> I have tested Twilio SendGrid as outgoing server and it just rewrote
> >>> the
> >>> URLs
> >>>
> >>>
> >>> https://issues.apache.org/jira/browse/SOLR-15947
> >>> https://issues.apache.org/jira/browse/LUCENE-10382
> >>>
> >>> and
> >>>
> >>>
> >>>
> https://lucene.apache.org/core/9_1_0/demo/org/apache/lucene/demo/knn/package-summary.html
> >>>
> >>>
> >>> which I was not aware of, but disabled the tracking now and hope it
> >>> will
> >>> be ok now.
> >>>
> >>> Thanks
> >>>
> >>> Michael
> >>>
> >>> Am 09.05.22 um 15:12 schrieb Michael Wechner:
> >>>> Hi
> >>>>
> >>>> I noticed that Lucene 9.1.0 supports filtering in nearest-neighbor
> >>>> vector search, which is great :-)
> >>>>
> >>>> I have found
> >>>>
> >>>>
> >>>
> http://url7093.wyona.com/ls/click?upn=JOH5Fjdv9AA9sbvUyiP84WWONyl36e4Tdd3VZFG-2B7pcYPJTPhVT3xqtcUDjPgQX5jI0WYWlJZX8h9NDC6okDRg-3D-3DHvvY_UMWFA-2BOn91WS4mEQPCWI9gZNzEZlJPmWPGP2CeMD7g4c-2Fpo3g6VPyd4ghH4X9o8sJ-2Bl292KOe2-2F30WmSZB1KHnF6KpgvICPKY8k5m30V-2FWrDvoSQLIaimtz2YHrzSMNV98es7-2BeXQS174-2B0EHPnQVtOYqUPojoZgGkqmovRXrz1dJlfs9dtFDGqSjpYaFiMaVBoiDTrpJ-2FUkuanwcx6R8UgvxBCq08DUa7vhRJqir7M-3D
> >>>
> >>>>
> >>>
> http://url7093.wyona.com/ls/click?upn=JOH5Fjdv9AA9sbvUyiP84WWONyl36e4Tdd3VZFG-2B7pe76-2Fa6XWGARorEmYO8A-2BeVhPN2B1iPvCEp9XG8WpVE6w-3D-3DHR5k_UMWFA-2BOn91WS4mEQPCWI9gZNzEZlJPmWPGP2CeMD7g4c-2Fpo3g6VPyd4ghH4X9o8sJ-2Bl292KOe2-2F30WmSZB1KHiGuZh3RFDkfg6-2FfNpOG8Tly-2FHwK25rp-2F24-2BPYZV6e5mGIWVQ5bpJ0l9u4lRzO6rySncTjxQZEPOjzZIrDMh-2Fo17VBGBagQ0Gr6G-2BAySO2ZdtDBthWjE-2F7HwxioFG9XrEUBVS79a0mcaaPKM-2BdzT9Cc-3D
> >>>
> >>>>
> >>>> and
> >>>>
> >>>>
> >>>
> http://url7093.wyona.com/ls/click?upn=JOH5Fjdv9AA9sbvUyiP84Rv2gN4NEUQqv-2Fn4lbJOY6mZGeN60klU5tyssLGPfHHB3IBl2Fx9C7un4UF2pBgYYcEd15H8F-2FPYEn4LTL-2Bz8fMFeo4z-2BjB3yMGv345VDdnStdESYCXN-2FD-2BQOSZSTNQLbQ-3D-3D9uTB_UMWFA-2BOn91WS4mEQPCWI9gZNzEZlJPmWPGP2CeMD7g4c-2Fpo3g6VPyd4ghH4X9o8sJ-2Bl292KOe2-2F30WmSZB1KHkzFWMImz7LKaiDu0g-2BZTsPclbKiyBoQiJHrZiOk5CuKsixeOFVVfvwAjyEhV-2F5McxrC76Q-2F72ILNowoPMFyMwXdaUF-2FhFh6HF0aWgai16l9zSdZIETAq46vRruPFO9ZqlRY6XiSu-2FBiKe4r5xiM0vA-3D
> >>>
> >>>>
> >>>> but it is not really clear to me how to use it.
> >>>>
> >>>> Is there some additional documentation / example / demo how I can
> >>>> combine filtering with vector search?
> >>>>
> >>>> Thanks
> >>>>
> >>>> Michael
> >>>>
> >>>>
> >>>>
> >>>> ---------------------------------------------------------------------
> >>>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> >>>> For additional commands, e-mail: java-user-help@lucene.apache.org
> >>>>
> >>>
> >>> ---------------------------------------------------------------------
> >>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> >>> For additional commands, e-mail: java-user-help@lucene.apache.org
> >>>
> >>>
> >
>

Michael -- I am not totally clear on what you have in mind. Would you be up
for opening a pull request in GitHub (or JIRA issue) to show the idea, and
we can discuss the details there?

Matt -- thanks for sharing!

Julie

On Sun, May 22, 2022 at 3:29 PM Matt Davis <kryptonics411@gmail.com> wrote:

> Thanks Julie. I was able to implement vector search in Zulia with your
> pointers. The pull request might be helpful to others:
> https://github.com/zuliaio/zuliasearch/pull/70
>
> Thanks,
> Matt
>
> On Fri, May 20, 2022 at 9:23 AM Michael Wechner <michael.wechner@wyona.com
> >
> wrote:
>
> > Hi Julie
> >
> > I got it running and it seems to work fine so far :-)
> >
> > Re an example for the demo package, I guess this would go here
> >
> >
> >
> https://lucene.apache.org/core/9_1_0/demo/org/apache/lucene/demo/knn/package-summary.html
> >
> > and I thought of something like "KnnVectorPreFilterQuery.java"
> >
> > Query preFilterQuery =new TermQuery(new Term(TOPIC_FIELD,"general"));
> > if (filter !=null) {
> > log.info("Filter applied before the vector search: " +
> > preFilterQuery);
> > }
> > Query query =new KnnVectorQuery(VECTOR_FIELD, queryVector, k,
> > preFilterQuery);
> >
> > TopDocs topDocs = searcher.search(query, k);
> >
> > Does that make sense to you?
> >
> > Thanks
> >
> > Michael
> >
> > Am 11.05.22 um 07:59 schrieb Michael Wechner:
> > > Hi Julie
> > >
> > > Cool, thanks!
> > >
> > > I try to apply it and if it works could create an example to the demo
> > > package.
> > >
> > > Will keep you posted :-)
> > >
> > > Thanks
> > >
> > > Michael
> > >
> > > Am 11.05.22 um 02:13 schrieb Julie Tibshirani:
> > >> Hello Michael, we don't yet have an example of kNN with filtering in
> the
> > >> demo package. This would be nice to add!
> > >>
> > >> For now maybe looking at the unit tests could give a sense of how to
> use
> > >> it. Here's an example:
> > >>
> >
> https://github.com/apache/lucene/blob/main/lucene/core/src/test/org/apache/lucene/search/TestKnnVectorQuery.java#L115-L127
> .
> >
> > >>
> > >> The idea is that KnnVectorQuery optionally accepts a Query as a
> > >> filter, and
> > >> returns the k nearest vectors that also match the filter. Many people
> > >> refer
> > >> to this as "kNN with prefiltering" (as opposed to "postfiltering",
> where
> > >> the filter is applied *after* the kNN search, so in the end you may
> > >> receive
> > >> fewer than k matches).
> > >>
> > >> Let us know if you run into any questions/ issues while trying it out!
> > >>
> > >> Julie
> > >>
> > >> On Mon, May 9, 2022 at 8:08 AM Michael Wechner
> > >> <michael.wechner@wyona.com>
> > >> wrote:
> > >>
> > >>> sorry for the URLs below.
> > >>>
> > >>> I have tested Twilio SendGrid as outgoing server and it just rewrote
> > >>> the
> > >>> URLs
> > >>>
> > >>>
> > >>> https://issues.apache.org/jira/browse/SOLR-15947
> > >>> https://issues.apache.org/jira/browse/LUCENE-10382
> > >>>
> > >>> and
> > >>>
> > >>>
> > >>>
> >
> https://lucene.apache.org/core/9_1_0/demo/org/apache/lucene/demo/knn/package-summary.html
> > >>>
> > >>>
> > >>> which I was not aware of, but disabled the tracking now and hope it
> > >>> will
> > >>> be ok now.
> > >>>
> > >>> Thanks
> > >>>
> > >>> Michael
> > >>>
> > >>> Am 09.05.22 um 15:12 schrieb Michael Wechner:
> > >>>> Hi
> > >>>>
> > >>>> I noticed that Lucene 9.1.0 supports filtering in nearest-neighbor
> > >>>> vector search, which is great :-)
> > >>>>
> > >>>> I have found
> > >>>>
> > >>>>
> > >>>
> >
> http://url7093.wyona.com/ls/click?upn=JOH5Fjdv9AA9sbvUyiP84WWONyl36e4Tdd3VZFG-2B7pcYPJTPhVT3xqtcUDjPgQX5jI0WYWlJZX8h9NDC6okDRg-3D-3DHvvY_UMWFA-2BOn91WS4mEQPCWI9gZNzEZlJPmWPGP2CeMD7g4c-2Fpo3g6VPyd4ghH4X9o8sJ-2Bl292KOe2-2F30WmSZB1KHnF6KpgvICPKY8k5m30V-2FWrDvoSQLIaimtz2YHrzSMNV98es7-2BeXQS174-2B0EHPnQVtOYqUPojoZgGkqmovRXrz1dJlfs9dtFDGqSjpYaFiMaVBoiDTrpJ-2FUkuanwcx6R8UgvxBCq08DUa7vhRJqir7M-3D
> > >>>
> > >>>>
> > >>>
> >
> http://url7093.wyona.com/ls/click?upn=JOH5Fjdv9AA9sbvUyiP84WWONyl36e4Tdd3VZFG-2B7pe76-2Fa6XWGARorEmYO8A-2BeVhPN2B1iPvCEp9XG8WpVE6w-3D-3DHR5k_UMWFA-2BOn91WS4mEQPCWI9gZNzEZlJPmWPGP2CeMD7g4c-2Fpo3g6VPyd4ghH4X9o8sJ-2Bl292KOe2-2F30WmSZB1KHiGuZh3RFDkfg6-2FfNpOG8Tly-2FHwK25rp-2F24-2BPYZV6e5mGIWVQ5bpJ0l9u4lRzO6rySncTjxQZEPOjzZIrDMh-2Fo17VBGBagQ0Gr6G-2BAySO2ZdtDBthWjE-2F7HwxioFG9XrEUBVS79a0mcaaPKM-2BdzT9Cc-3D
> > >>>
> > >>>>
> > >>>> and
> > >>>>
> > >>>>
> > >>>
> >
> http://url7093.wyona.com/ls/click?upn=JOH5Fjdv9AA9sbvUyiP84Rv2gN4NEUQqv-2Fn4lbJOY6mZGeN60klU5tyssLGPfHHB3IBl2Fx9C7un4UF2pBgYYcEd15H8F-2FPYEn4LTL-2Bz8fMFeo4z-2BjB3yMGv345VDdnStdESYCXN-2FD-2BQOSZSTNQLbQ-3D-3D9uTB_UMWFA-2BOn91WS4mEQPCWI9gZNzEZlJPmWPGP2CeMD7g4c-2Fpo3g6VPyd4ghH4X9o8sJ-2Bl292KOe2-2F30WmSZB1KHkzFWMImz7LKaiDu0g-2BZTsPclbKiyBoQiJHrZiOk5CuKsixeOFVVfvwAjyEhV-2F5McxrC76Q-2F72ILNowoPMFyMwXdaUF-2FhFh6HF0aWgai16l9zSdZIETAq46vRruPFO9ZqlRY6XiSu-2FBiKe4r5xiM0vA-3D
> > >>>
> > >>>>
> > >>>> but it is not really clear to me how to use it.
> > >>>>
> > >>>> Is there some additional documentation / example / demo how I can
> > >>>> combine filtering with vector search?
> > >>>>
> > >>>> Thanks
> > >>>>
> > >>>> Michael
> > >>>>
> > >>>>
> > >>>>
> > >>>>
> ---------------------------------------------------------------------
> > >>>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> > >>>> For additional commands, e-mail: java-user-help@lucene.apache.org
> > >>>>
> > >>>
> > >>> ---------------------------------------------------------------------
> > >>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> > >>> For additional commands, e-mail: java-user-help@lucene.apache.org
> > >>>
> > >>>
> > >
> >
>

yes, sure, will keep you posted :-)

Thanks

Michael

Am 24.05.22 um 02:10 schrieb Julie Tibshirani:
> Michael -- I am not totally clear on what you have in mind. Would you be up
> for opening a pull request in GitHub (or JIRA issue) to show the idea, and
> we can discuss the details there?
>
> Matt -- thanks for sharing!
>
> Julie
>
> On Sun, May 22, 2022 at 3:29 PM Matt Davis <kryptonics411@gmail.com> wrote:
>
>> Thanks Julie. I was able to implement vector search in Zulia with your
>> pointers. The pull request might be helpful to others:
>> https://github.com/zuliaio/zuliasearch/pull/70
>>
>> Thanks,
>> Matt
>>
>> On Fri, May 20, 2022 at 9:23 AM Michael Wechner <michael.wechner@wyona.com
>> wrote:
>>
>>> Hi Julie
>>>
>>> I got it running and it seems to work fine so far :-)
>>>
>>> Re an example for the demo package, I guess this would go here
>>>
>>>
>>>
>> https://lucene.apache.org/core/9_1_0/demo/org/apache/lucene/demo/knn/package-summary.html
>>> and I thought of something like "KnnVectorPreFilterQuery.java"
>>>
>>> Query preFilterQuery =new TermQuery(new Term(TOPIC_FIELD,"general"));
>>> if (filter !=null) {
>>> log.info("Filter applied before the vector search: " +
>>> preFilterQuery);
>>> }
>>> Query query =new KnnVectorQuery(VECTOR_FIELD, queryVector, k,
>>> preFilterQuery);
>>>
>>> TopDocs topDocs = searcher.search(query, k);
>>>
>>> Does that make sense to you?
>>>
>>> Thanks
>>>
>>> Michael
>>>
>>> Am 11.05.22 um 07:59 schrieb Michael Wechner:
>>>> Hi Julie
>>>>
>>>> Cool, thanks!
>>>>
>>>> I try to apply it and if it works could create an example to the demo
>>>> package.
>>>>
>>>> Will keep you posted :-)
>>>>
>>>> Thanks
>>>>
>>>> Michael
>>>>
>>>> Am 11.05.22 um 02:13 schrieb Julie Tibshirani:
>>>>> Hello Michael, we don't yet have an example of kNN with filtering in
>> the
>>>>> demo package. This would be nice to add!
>>>>>
>>>>> For now maybe looking at the unit tests could give a sense of how to
>> use
>>>>> it. Here's an example:
>>>>>
>> https://github.com/apache/lucene/blob/main/lucene/core/src/test/org/apache/lucene/search/TestKnnVectorQuery.java#L115-L127
>> .
>>>>> The idea is that KnnVectorQuery optionally accepts a Query as a
>>>>> filter, and
>>>>> returns the k nearest vectors that also match the filter. Many people
>>>>> refer
>>>>> to this as "kNN with prefiltering" (as opposed to "postfiltering",
>> where
>>>>> the filter is applied *after* the kNN search, so in the end you may
>>>>> receive
>>>>> fewer than k matches).
>>>>>
>>>>> Let us know if you run into any questions/ issues while trying it out!
>>>>>
>>>>> Julie
>>>>>
>>>>> On Mon, May 9, 2022 at 8:08 AM Michael Wechner
>>>>> <michael.wechner@wyona.com>
>>>>> wrote:
>>>>>
>>>>>> sorry for the URLs below.
>>>>>>
>>>>>> I have tested Twilio SendGrid as outgoing server and it just rewrote
>>>>>> the
>>>>>> URLs
>>>>>>
>>>>>>
>>>>>> https://issues.apache.org/jira/browse/SOLR-15947
>>>>>> https://issues.apache.org/jira/browse/LUCENE-10382
>>>>>>
>>>>>> and
>>>>>>
>>>>>>
>>>>>>
>> https://lucene.apache.org/core/9_1_0/demo/org/apache/lucene/demo/knn/package-summary.html
>>>>>>
>>>>>> which I was not aware of, but disabled the tracking now and hope it
>>>>>> will
>>>>>> be ok now.
>>>>>>
>>>>>> Thanks
>>>>>>
>>>>>> Michael
>>>>>>
>>>>>> Am 09.05.22 um 15:12 schrieb Michael Wechner:
>>>>>>> Hi
>>>>>>>
>>>>>>> I noticed that Lucene 9.1.0 supports filtering in nearest-neighbor
>>>>>>> vector search, which is great :-)
>>>>>>>
>>>>>>> I have found
>>>>>>>
>>>>>>>
>> http://url7093.wyona.com/ls/click?upn=JOH5Fjdv9AA9sbvUyiP84WWONyl36e4Tdd3VZFG-2B7pcYPJTPhVT3xqtcUDjPgQX5jI0WYWlJZX8h9NDC6okDRg-3D-3DHvvY_UMWFA-2BOn91WS4mEQPCWI9gZNzEZlJPmWPGP2CeMD7g4c-2Fpo3g6VPyd4ghH4X9o8sJ-2Bl292KOe2-2F30WmSZB1KHnF6KpgvICPKY8k5m30V-2FWrDvoSQLIaimtz2YHrzSMNV98es7-2BeXQS174-2B0EHPnQVtOYqUPojoZgGkqmovRXrz1dJlfs9dtFDGqSjpYaFiMaVBoiDTrpJ-2FUkuanwcx6R8UgvxBCq08DUa7vhRJqir7M-3D
>> http://url7093.wyona.com/ls/click?upn=JOH5Fjdv9AA9sbvUyiP84WWONyl36e4Tdd3VZFG-2B7pe76-2Fa6XWGARorEmYO8A-2BeVhPN2B1iPvCEp9XG8WpVE6w-3D-3DHR5k_UMWFA-2BOn91WS4mEQPCWI9gZNzEZlJPmWPGP2CeMD7g4c-2Fpo3g6VPyd4ghH4X9o8sJ-2Bl292KOe2-2F30WmSZB1KHiGuZh3RFDkfg6-2FfNpOG8Tly-2FHwK25rp-2F24-2BPYZV6e5mGIWVQ5bpJ0l9u4lRzO6rySncTjxQZEPOjzZIrDMh-2Fo17VBGBagQ0Gr6G-2BAySO2ZdtDBthWjE-2F7HwxioFG9XrEUBVS79a0mcaaPKM-2BdzT9Cc-3D
>>>>>>> and
>>>>>>>
>>>>>>>
>> http://url7093.wyona.com/ls/click?upn=JOH5Fjdv9AA9sbvUyiP84Rv2gN4NEUQqv-2Fn4lbJOY6mZGeN60klU5tyssLGPfHHB3IBl2Fx9C7un4UF2pBgYYcEd15H8F-2FPYEn4LTL-2Bz8fMFeo4z-2BjB3yMGv345VDdnStdESYCXN-2FD-2BQOSZSTNQLbQ-3D-3D9uTB_UMWFA-2BOn91WS4mEQPCWI9gZNzEZlJPmWPGP2CeMD7g4c-2Fpo3g6VPyd4ghH4X9o8sJ-2Bl292KOe2-2F30WmSZB1KHkzFWMImz7LKaiDu0g-2BZTsPclbKiyBoQiJHrZiOk5CuKsixeOFVVfvwAjyEhV-2F5McxrC76Q-2F72ILNowoPMFyMwXdaUF-2FhFh6HF0aWgai16l9zSdZIETAq46vRruPFO9ZqlRY6XiSu-2FBiKe4r5xiM0vA-3D
>>>>>>> but it is not really clear to me how to use it.
>>>>>>>
>>>>>>> Is there some additional documentation / example / demo how I can
>>>>>>> combine filtering with vector search?
>>>>>>>
>>>>>>> Thanks
>>>>>>>
>>>>>>> Michael
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>> ---------------------------------------------------------------------
>>>>>>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>>>>>>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>>>>>>
>>>>>> ---------------------------------------------------------------------
>>>>>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>>>>>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>>>>>
>>>>>>


---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org