Mailing List Archive

VectorField renamed to KnnVectorField?
Hi

In May 2021 I have done a Vector Search implementation based on Lucene 9.0.0-SNAPSHOT with the following code

FieldType vectorFieldType = VectorField.createHnswType(vector.length, VectorValues.SimilarityFunction.DOT_PRODUCT,16,500);
VectorField vectorField =new VectorField(VECTOR_FIELD, vector, vectorFieldType);
doc.add(vectorField)

and

class KnnWeightextends Weight {

KnnWeight() {
super(KnnQuery.this);
}

@Override public Scorer scorer(LeafReaderContext context)throws IOException {
log.debug("Get scorer.");
return new TopDocScorer(this, context.reader().searchNearestVectors(field,vector,topK,fanout));
}

whereas fanout is of type "int"

I have now updated Lucene source and rebuilt 9.0.0-SNAPSHOT and get various compile errors.

I assume VectorField got renamed to KnnVectorField, right?

Does somebody maybe have some sample code how Vector search is being implemented with the most recent Lucene code?

Thanks

Michael
Re: VectorField renamed to KnnVectorField? [ In reply to ]
I was able to update my code

-        FieldType vectorFieldType =
VectorField.createHnswType(vector.length,
VectorValues.SimilarityFunction.DOT_PRODUCT, 16, 500);
-        VectorField vectorField = new VectorField(VECTOR_FIELD, vector,
vectorFieldType);
+        FieldType vectorFieldType =
KnnVectorField.createFieldType(vector.length,
VectorSimilarityFunction.DOT_PRODUCT);
+        KnnVectorField vectorField = new KnnVectorField(VECTOR_FIELD,
vector, vectorFieldType);

and

-            return new TopDocScorer(this,
context.reader().searchNearestVectors(field, vector, topK, fanout));
+            return new TopDocScorer(this,
context.reader().searchNearestVectors(field, vector, topK, null));

the indexing and searching works again :-)

Thanks

Michael

Am 01.11.21 um 18:53 schrieb Michael Wechner:
> Hi
>
> In May 2021 I have done a Vector Search implementation based on Lucene 9.0.0-SNAPSHOT with the following code
>
> FieldType vectorFieldType = VectorField.createHnswType(vector.length, VectorValues.SimilarityFunction.DOT_PRODUCT,16,500);
> VectorField vectorField =new VectorField(VECTOR_FIELD, vector, vectorFieldType);
> doc.add(vectorField)
>
> and
>
> class KnnWeightextends Weight {
>
> KnnWeight() {
> super(KnnQuery.this);
> }
>
> @Override public Scorer scorer(LeafReaderContext context)throws IOException {
> log.debug("Get scorer.");
> return new TopDocScorer(this, context.reader().searchNearestVectors(field,vector,topK,fanout));
> }
>
> whereas fanout is of type "int"
>
> I have now updated Lucene source and rebuilt 9.0.0-SNAPSHOT and get various compile errors.
>
> I assume VectorField got renamed to KnnVectorField, right?
>
> Does somebody maybe have some sample code how Vector search is being implemented with the most recent Lucene code?
>
> Thanks
>
> Michael
Re: VectorField renamed to KnnVectorField? [ In reply to ]
Hi Michael,

Glad you got it working. There is also a KNN vector search demo that was
added not long ago. You might want to check it out. It has references for
example, to compute embeddings and build knn vector queries
<https://github.com/apache/lucene/blob/main/lucene/demo/src/java/org/apache/lucene/demo/SearchFiles.java#L272-L292>,
among other things.

- Search Files:
https://github.com/apache/lucene/blob/main/lucene/demo/src/java/org/apache/lucene/demo/SearchFiles.java

- Index Files:
https://github.com/apache/lucene/blob/main/lucene/demo/src/java/org/apache/lucene/demo/IndexFiles.java

- knn demo files:
https://github.com/apache/lucene/tree/main/lucene/demo/src/java/org/apache/lucene/demo/knn


- Vigya


On Mon, Nov 1, 2021 at 2:44 PM Michael Wechner <michael.wechner@wyona.com>
wrote:

> I was able to update my code
>
> - FieldType vectorFieldType =
> VectorField.createHnswType(vector.length,
> VectorValues.SimilarityFunction.DOT_PRODUCT, 16, 500);
> - VectorField vectorField = new VectorField(VECTOR_FIELD, vector,
> vectorFieldType);
> + FieldType vectorFieldType =
> KnnVectorField.createFieldType(vector.length,
> VectorSimilarityFunction.DOT_PRODUCT);
> + KnnVectorField vectorField = new KnnVectorField(VECTOR_FIELD,
> vector, vectorFieldType);
>
> and
>
> - return new TopDocScorer(this,
> context.reader().searchNearestVectors(field, vector, topK, fanout));
> + return new TopDocScorer(this,
> context.reader().searchNearestVectors(field, vector, topK, null));
>
> the indexing and searching works again :-)
>
> Thanks
>
> Michael
>
> Am 01.11.21 um 18:53 schrieb Michael Wechner:
>
> Hi
>
> In May 2021 I have done a Vector Search implementation based on Lucene 9.0.0-SNAPSHOT with the following code
>
> FieldType vectorFieldType = VectorField.createHnswType(vector.length, VectorValues.SimilarityFunction.DOT_PRODUCT, 16, 500);
> VectorField vectorField = new VectorField(VECTOR_FIELD, vector, vectorFieldType);
> doc.add(vectorField)
>
> and
> class KnnWeight extends Weight {
>
> KnnWeight() {
> super(KnnQuery.this);
> }
>
> @Override public Scorer scorer(LeafReaderContext context) throws IOException {
> log.debug("Get scorer.");
> return new TopDocScorer(this, context.reader().searchNearestVectors(field, vector, topK, fanout));
> }
>
> whereas fanout is of type "int"
>
> I have now updated Lucene source and rebuilt 9.0.0-SNAPSHOT and get various compile errors.
>
> I assume VectorField got renamed to KnnVectorField, right?
>
> Does somebody maybe have some sample code how Vector search is being implemented with the most recent Lucene code?
>
> Thanks
>
> Michael
>
>
>

--
- Vigya
Re: VectorField renamed to KnnVectorField? [ In reply to ]
Hi Vigya

Great, thank you very much for these links!

All the best

Michael

Am 02.11.21 um 01:45 schrieb Vigya Sharma:
> Hi Michael,
>
> Glad you got it working. There is also a KNN vector search demo that
> was added not long ago. You might want to check it out. It has
> references for example, to compute embeddings and build knn vector
> queries
> <https://github.com/apache/lucene/blob/main/lucene/demo/src/java/org/apache/lucene/demo/SearchFiles.java#L272-L292>,
> among other things.
>
> * Search Files:
> https://github.com/apache/lucene/blob/main/lucene/demo/src/java/org/apache/lucene/demo/SearchFiles.java
> <https://github.com/apache/lucene/blob/main/lucene/demo/src/java/org/apache/lucene/demo/SearchFiles.java>
>
> * Index Files:
> https://github.com/apache/lucene/blob/main/lucene/demo/src/java/org/apache/lucene/demo/IndexFiles.java
> <https://github.com/apache/lucene/blob/main/lucene/demo/src/java/org/apache/lucene/demo/IndexFiles.java>
>
> * knn demo files:
> https://github.com/apache/lucene/tree/main/lucene/demo/src/java/org/apache/lucene/demo/knn
> <https://github.com/apache/lucene/tree/main/lucene/demo/src/java/org/apache/lucene/demo/knn>
>
>
> - Vigya
>
>
> On Mon, Nov 1, 2021 at 2:44 PM Michael Wechner
> <michael.wechner@wyona.com <mailto:michael.wechner@wyona.com>> wrote:
>
> I was able to update my code
>
> -        FieldType vectorFieldType =
> VectorField.createHnswType(vector.length,
> VectorValues.SimilarityFunction.DOT_PRODUCT, 16, 500);
> -        VectorField vectorField = new VectorField(VECTOR_FIELD,
> vector, vectorFieldType);
> +        FieldType vectorFieldType =
> KnnVectorField.createFieldType(vector.length,
> VectorSimilarityFunction.DOT_PRODUCT);
> +        KnnVectorField vectorField = new
> KnnVectorField(VECTOR_FIELD, vector, vectorFieldType);
>
> and
>
> -            return new TopDocScorer(this,
> context.reader().searchNearestVectors(field, vector, topK, fanout));
> +            return new TopDocScorer(this,
> context.reader().searchNearestVectors(field, vector, topK, null));
>
> the indexing and searching works again :-)
>
> Thanks
>
> Michael
>
> Am 01.11.21 um 18:53 schrieb Michael Wechner:
>> Hi
>>
>> In May 2021 I have done a Vector Search implementation based on Lucene 9.0.0-SNAPSHOT with the following code
>>
>> FieldType vectorFieldType = VectorField.createHnswType(vector.length, VectorValues.SimilarityFunction.DOT_PRODUCT,16,500);
>> VectorField vectorField =new VectorField(VECTOR_FIELD, vector, vectorFieldType);
>> doc.add(vectorField)
>>
>> and
>>
>> class KnnWeightextends Weight {
>>
>> KnnWeight() {
>> super(KnnQuery.this);
>> }
>>
>> @Override public Scorer scorer(LeafReaderContext context)throws IOException {
>> log.debug("Get scorer.");
>> return new TopDocScorer(this, context.reader().searchNearestVectors(field,vector,topK,fanout));
>> }
>>
>> whereas fanout is of type "int"
>>
>> I have now updated Lucene source and rebuilt 9.0.0-SNAPSHOT and get various compile errors.
>>
>> I assume VectorField got renamed to KnnVectorField, right?
>>
>> Does somebody maybe have some sample code how Vector search is being implemented with the most recent Lucene code?
>>
>> Thanks
>>
>> Michael
>
>
>
> --
> - Vigya