Mailing List Archive: [VOTE] Dimension Limit for KNN Vectors

[VOTE] Dimension Limit for KNN Vectors

May 16, 2023, 1:50 AM

Post #1 of 46 (855 views)

Hi all,
we have finalized all the options proposed by the community and we are
ready to vote for the preferred one and then proceed with the
implementation.

*Option 1*
Keep it as it is (dimension limit hardcoded to 1024)
*Motivation*:
We are close to improving on many fronts. Given the criticality of Lucene
in computing infrastructure and the concerns raised by one of the most
active stewards of the project, I think we should keep working toward
improving the feature as is and move to up the limit after we can
demonstrate improvement unambiguously.

*Option 2*
make the limit configurable, for example through a system property
*Motivation*:
The system administrator can enforce a limit its users need to respect that
it's in line with whatever the admin decided to be acceptable for them.
The default can stay the current one.
This should open the doors for Apache Solr, Elasticsearch, OpenSearch, and
any sort of plugin development

*Option 3*
Move the max dimension limit lower level to a HNSW specific implementation.
Once there, this limit would not bind any other potential vector engine
alternative/evolution.
*Motivation:* There seem to be contradictory performance interpretations
about the current HNSW implementation. Some consider its performance ok,
some not, and it depends on the target data set and use case. Increasing
the max dimension limit where it is currently (in top level
FloatVectorValues) would not allow potential alternatives (e.g. for other
use-cases) to be based on a lower limit.

*Option 4*
Make it configurable and move it to an appropriate place.
In particular, a simple Integer.getInteger("lucene.hnsw.maxDimensions",
1024) should be enough.
*Motivation*:
Both are good and not mutually exclusive and could happen in any order.
Someone suggested to perfect what the _default_ limit should be, but I've
not seen an argument _against_ configurability. Especially in this way --
a toggle that doesn't bind Lucene's APIs in any way.

I'll keep this [VOTE] open for a week and then proceed to the
implementation.
--------------------------
*Alessandro Benedetti*
Director @ Sease Ltd.
*Apache Lucene/Solr Committer*
*Apache Solr PMC Member*

e-mail: a.benedetti@sease.io

*Sease* - Information Retrieval Applied
Consulting | Training | Open Source

Website: Sease.io <http://sease.io/>
LinkedIn <https://linkedin.com/company/sease-ltd> | Twitter
<https://twitter.com/seaseltd> | Youtube
<https://www.youtube.com/channel/UCDx86ZKLYNpI3gzMercM7BQ> | Github
<https://github.com/seaseltd>

Re: [VOTE] Dimension Limit for KNN Vectors [ In reply to ]

a.benedetti at sease

May 16, 2023, 1:51 AM

Post #2 of 46 (855 views)

Permalink

My vote goes to *Option 4*.
--------------------------
*Alessandro Benedetti*
Director @ Sease Ltd.
*Apache Lucene/Solr Committer*
*Apache Solr PMC Member*

e-mail: a.benedetti@sease.io

*Sease* - Information Retrieval Applied
Consulting | Training | Open Source

Website: Sease.io <http://sease.io/>
LinkedIn <https://linkedin.com/company/sease-ltd> | Twitter
<https://twitter.com/seaseltd> | Youtube
<https://www.youtube.com/channel/UCDx86ZKLYNpI3gzMercM7BQ> | Github
<https://github.com/seaseltd>

On Tue, 16 May 2023 at 09:50, Alessandro Benedetti <a.benedetti@sease.io>
wrote:

> Hi all,
> we have finalized all the options proposed by the community and we are
> ready to vote for the preferred one and then proceed with the
> implementation.
>
> *Option 1*
> Keep it as it is (dimension limit hardcoded to 1024)
> *Motivation*:
> We are close to improving on many fronts. Given the criticality of Lucene
> in computing infrastructure and the concerns raised by one of the most
> active stewards of the project, I think we should keep working toward
> improving the feature as is and move to up the limit after we can
> demonstrate improvement unambiguously.
>
> *Option 2*
> make the limit configurable, for example through a system property
> *Motivation*:
> The system administrator can enforce a limit its users need to respect
> that it's in line with whatever the admin decided to be acceptable for
> them.
> The default can stay the current one.
> This should open the doors for Apache Solr, Elasticsearch, OpenSearch, and
> any sort of plugin development
>
> *Option 3*
> Move the max dimension limit lower level to a HNSW specific
> implementation. Once there, this limit would not bind any other potential
> vector engine alternative/evolution.
> *Motivation:* There seem to be contradictory performance interpretations
> about the current HNSW implementation. Some consider its performance ok,
> some not, and it depends on the target data set and use case. Increasing
> the max dimension limit where it is currently (in top level
> FloatVectorValues) would not allow potential alternatives (e.g. for other
> use-cases) to be based on a lower limit.
>
> *Option 4*
> Make it configurable and move it to an appropriate place.
> In particular, a simple Integer.getInteger("lucene.hnsw.maxDimensions",
> 1024) should be enough.
> *Motivation*:
> Both are good and not mutually exclusive and could happen in any order.
> Someone suggested to perfect what the _default_ limit should be, but I've
> not seen an argument _against_ configurability. Especially in this way --
> a toggle that doesn't bind Lucene's APIs in any way.
>
> I'll keep this [VOTE] open for a week and then proceed to the
> implementation.
> --------------------------
> *Alessandro Benedetti*
> Director @ Sease Ltd.
> *Apache Lucene/Solr Committer*
> *Apache Solr PMC Member*
>
> e-mail: a.benedetti@sease.io
>
>
> *Sease* - Information Retrieval Applied
> Consulting | Training | Open Source
>
> Website: Sease.io <http://sease.io/>
> LinkedIn <https://linkedin.com/company/sease-ltd> | Twitter
> <https://twitter.com/seaseltd> | Youtube
> <https://www.youtube.com/channel/UCDx86ZKLYNpI3gzMercM7BQ> | Github
> <https://github.com/seaseltd>
>

Re: [VOTE] Dimension Limit for KNN Vectors [ In reply to ]

michael.wechner at wyona

May 16, 2023, 2:04 AM

Post #3 of 46 (855 views)

Permalink

Hi Alessandro

Thank you very much for summarizing and starting the vote.

I am not sure whether I really understand the difference between Option
2 and Option 4, or is it just about implementation details?

Thanks

Michael

Am 16.05.23 um 10:50 schrieb Alessandro Benedetti:
> Hi all,
> we have finalized all the options proposed by the community and we are
> ready to vote for the preferred one and then proceed with the
> implementation.
>
> *Option 1*
> Keep it as it is (dimension limit hardcoded to 1024)
> *Motivation*:
> We are close to improving on many fronts. Given?the criticality of
> Lucene in computing infrastructure and the concerns raised by one of
> the most active stewards of the project, I think we should keep
> working toward improving the feature as is and move to up the limit
> after we can demonstrate improvement unambiguously.
>
> *Option 2*
> make the limit configurable, for example through a system property
> *Motivation*:
> The system administrator can enforce a limit its users need to respect
> that it's in line with whatever the admin decided to be acceptable for
> them.
> The default can stay the current one.
> This should open the doors for Apache Solr, Elasticsearch, OpenSearch,
> and any sort of plugin development
>
> *Option 3*
> Move the max dimension limit lower level to a HNSW specific
> implementation. Once there, this limit would not bind any other
> potential vector engine alternative/evolution.*
> *
> *Motivation:*There seem to be contradictory performance
> interpretations about the current HNSW implementation. Some consider
> its performance ok, some not, and it depends on the target data set
> and use case. Increasing the max dimension limit where it is currently
> (in top level FloatVectorValues) would not allow
> potential?alternatives (e.g. for other use-cases) to be based on a
> lower limit.
>
> *Option 4*
> Make it configurable and move it to an appropriate place.
> In particular, a
> simple?Integer.getInteger("lucene.hnsw.maxDimensions", 1024) should be
> enough.
> *Motivation*:
> Both are good and not mutually exclusive and could happen in any order.
> Someone suggested to perfect what the _default_ limit should be, but
> I've not?seen an argument _against_ configurability.? Especially in
> this way -- a toggle that doesn't bind Lucene's APIs in any way.
>
> I'll keep this [VOTE] open for a week and then proceed to the
> implementation.
> --------------------------
> *Alessandro Benedetti*
> Director @ Sease Ltd.
> /Apache Lucene/Solr Committer/
> /Apache Solr PMC Member/
>
> e-mail: a.benedetti@sease.io/
> /
>
> *Sease*?- Information Retrieval Applied
> Consulting | Training | Open Source
>
> Website: Sease.io <http://sease.io/>
> LinkedIn <https://linkedin.com/company/sease-ltd>?| Twitter
> <https://twitter.com/seaseltd>?| Youtube
> <https://www.youtube.com/channel/UCDx86ZKLYNpI3gzMercM7BQ>?| Github
> <https://github.com/seaseltd>

Re: [VOTE] Dimension Limit for KNN Vectors [ In reply to ]

a.benedetti at sease

May 16, 2023, 2:24 AM

Post #4 of 46 (855 views)

Permalink

Option 4 also aim to refactor the limit in an appropriate place for the
code (short answer is Yes, implementation details)

Cheers

On Tue, 16 May 2023, 10:04 Michael Wechner, <michael.wechner@wyona.com>
wrote:

> Hi Alessandro
>
> Thank you very much for summarizing and starting the vote.
>
> I am not sure whether I really understand the difference between Option 2
> and Option 4, or is it just about implementation details?
>
> Thanks
>
> Michael
>
>
>
> Am 16.05.23 um 10:50 schrieb Alessandro Benedetti:
>
> Hi all,
> we have finalized all the options proposed by the community and we are
> ready to vote for the preferred one and then proceed with the
> implementation.
>
> *Option 1*
> Keep it as it is (dimension limit hardcoded to 1024)
> *Motivation*:
> We are close to improving on many fronts. Given the criticality of Lucene
> in computing infrastructure and the concerns raised by one of the most
> active stewards of the project, I think we should keep working toward
> improving the feature as is and move to up the limit after we can
> demonstrate improvement unambiguously.
>
> *Option 2*
> make the limit configurable, for example through a system property
> *Motivation*:
> The system administrator can enforce a limit its users need to respect
> that it's in line with whatever the admin decided to be acceptable for
> them.
> The default can stay the current one.
> This should open the doors for Apache Solr, Elasticsearch, OpenSearch, and
> any sort of plugin development
>
> *Option 3*
> Move the max dimension limit lower level to a HNSW specific
> implementation. Once there, this limit would not bind any other potential
> vector engine alternative/evolution.
> *Motivation:* There seem to be contradictory performance interpretations
> about the current HNSW implementation. Some consider its performance ok,
> some not, and it depends on the target data set and use case. Increasing
> the max dimension limit where it is currently (in top level
> FloatVectorValues) would not allow potential alternatives (e.g. for other
> use-cases) to be based on a lower limit.
>
> *Option 4*
> Make it configurable and move it to an appropriate place.
> In particular, a simple Integer.getInteger("lucene.hnsw.maxDimensions",
> 1024) should be enough.
> *Motivation*:
> Both are good and not mutually exclusive and could happen in any order.
> Someone suggested to perfect what the _default_ limit should be, but I've
> not seen an argument _against_ configurability. Especially in this way --
> a toggle that doesn't bind Lucene's APIs in any way.
>
> I'll keep this [VOTE] open for a week and then proceed to the
> implementation.
> --------------------------
> *Alessandro Benedetti*
> Director @ Sease Ltd.
> *Apache Lucene/Solr Committer*
> *Apache Solr PMC Member*
>
> e-mail: a.benedetti@sease.io
>
>
> *Sease* - Information Retrieval Applied
> Consulting | Training | Open Source
>
> Website: Sease.io <http://sease.io/>
> LinkedIn <https://linkedin.com/company/sease-ltd> | Twitter
> <https://twitter.com/seaseltd> | Youtube
> <https://www.youtube.com/channel/UCDx86ZKLYNpI3gzMercM7BQ> | Github
> <https://github.com/seaseltd>
>
>
>

Re: [VOTE] Dimension Limit for KNN Vectors [ In reply to ]

a.benedetti at sease

May 16, 2023, 2:28 AM

Post #5 of 46 (855 views)

Permalink

For simplicity's sake, let's consider Option 2 and 4 as equivalent as they
are not mutually exclusive and just differ on a minor implementation
detail.

On Tue, 16 May 2023, 10:24 Alessandro Benedetti, <a.benedetti@sease.io>
wrote:

> Option 4 also aim to refactor the limit in an appropriate place for the
> code (short answer is Yes, implementation details)
>
> Cheers
>
> On Tue, 16 May 2023, 10:04 Michael Wechner, <michael.wechner@wyona.com>
> wrote:
>
>> Hi Alessandro
>>
>> Thank you very much for summarizing and starting the vote.
>>
>> I am not sure whether I really understand the difference between Option 2
>> and Option 4, or is it just about implementation details?
>>
>> Thanks
>>
>> Michael
>>
>>
>>
>> Am 16.05.23 um 10:50 schrieb Alessandro Benedetti:
>>
>> Hi all,
>> we have finalized all the options proposed by the community and we are
>> ready to vote for the preferred one and then proceed with the
>> implementation.
>>
>> *Option 1*
>> Keep it as it is (dimension limit hardcoded to 1024)
>> *Motivation*:
>> We are close to improving on many fronts. Given the criticality of Lucene
>> in computing infrastructure and the concerns raised by one of the most
>> active stewards of the project, I think we should keep working toward
>> improving the feature as is and move to up the limit after we can
>> demonstrate improvement unambiguously.
>>
>> *Option 2*
>> make the limit configurable, for example through a system property
>> *Motivation*:
>> The system administrator can enforce a limit its users need to respect
>> that it's in line with whatever the admin decided to be acceptable for
>> them.
>> The default can stay the current one.
>> This should open the doors for Apache Solr, Elasticsearch, OpenSearch,
>> and any sort of plugin development
>>
>> *Option 3*
>> Move the max dimension limit lower level to a HNSW specific
>> implementation. Once there, this limit would not bind any other potential
>> vector engine alternative/evolution.
>> *Motivation:* There seem to be contradictory performance interpretations
>> about the current HNSW implementation. Some consider its performance ok,
>> some not, and it depends on the target data set and use case. Increasing
>> the max dimension limit where it is currently (in top level
>> FloatVectorValues) would not allow potential alternatives (e.g. for other
>> use-cases) to be based on a lower limit.
>>
>> *Option 4*
>> Make it configurable and move it to an appropriate place.
>> In particular, a simple Integer.getInteger("lucene.hnsw.maxDimensions",
>> 1024) should be enough.
>> *Motivation*:
>> Both are good and not mutually exclusive and could happen in any order.
>> Someone suggested to perfect what the _default_ limit should be, but I've
>> not seen an argument _against_ configurability. Especially in this way --
>> a toggle that doesn't bind Lucene's APIs in any way.
>>
>> I'll keep this [VOTE] open for a week and then proceed to the
>> implementation.
>> --------------------------
>> *Alessandro Benedetti*
>> Director @ Sease Ltd.
>> *Apache Lucene/Solr Committer*
>> *Apache Solr PMC Member*
>>
>> e-mail: a.benedetti@sease.io
>>
>>
>> *Sease* - Information Retrieval Applied
>> Consulting | Training | Open Source
>>
>> Website: Sease.io <http://sease.io/>
>> LinkedIn <https://linkedin.com/company/sease-ltd> | Twitter
>> <https://twitter.com/seaseltd> | Youtube
>> <https://www.youtube.com/channel/UCDx86ZKLYNpI3gzMercM7BQ> | Github
>> <https://github.com/seaseltd>
>>
>>
>>

Re: [VOTE] Dimension Limit for KNN Vectors [ In reply to ]

rcmuir at gmail

May 16, 2023, 3:03 AM

Post #6 of 46 (855 views)

Permalink

i still feel -1 (veto) on increasing this limit. sending more emails does
not change the technical facts or make the veto go away.

On Tue, May 16, 2023 at 4:50?AM Alessandro Benedetti <a.benedetti@sease.io>
wrote:

> Hi all,
> we have finalized all the options proposed by the community and we are
> ready to vote for the preferred one and then proceed with the
> implementation.
>
> *Option 1*
> Keep it as it is (dimension limit hardcoded to 1024)
> *Motivation*:
> We are close to improving on many fronts. Given the criticality of Lucene
> in computing infrastructure and the concerns raised by one of the most
> active stewards of the project, I think we should keep working toward
> improving the feature as is and move to up the limit after we can
> demonstrate improvement unambiguously.
>
> *Option 2*
> make the limit configurable, for example through a system property
> *Motivation*:
> The system administrator can enforce a limit its users need to respect
> that it's in line with whatever the admin decided to be acceptable for
> them.
> The default can stay the current one.
> This should open the doors for Apache Solr, Elasticsearch, OpenSearch, and
> any sort of plugin development
>
> *Option 3*
> Move the max dimension limit lower level to a HNSW specific
> implementation. Once there, this limit would not bind any other potential
> vector engine alternative/evolution.
> *Motivation:* There seem to be contradictory performance interpretations
> about the current HNSW implementation. Some consider its performance ok,
> some not, and it depends on the target data set and use case. Increasing
> the max dimension limit where it is currently (in top level
> FloatVectorValues) would not allow potential alternatives (e.g. for other
> use-cases) to be based on a lower limit.
>
> *Option 4*
> Make it configurable and move it to an appropriate place.
> In particular, a simple Integer.getInteger("lucene.hnsw.maxDimensions",
> 1024) should be enough.
> *Motivation*:
> Both are good and not mutually exclusive and could happen in any order.
> Someone suggested to perfect what the _default_ limit should be, but I've
> not seen an argument _against_ configurability. Especially in this way --
> a toggle that doesn't bind Lucene's APIs in any way.
>
> I'll keep this [VOTE] open for a week and then proceed to the
> implementation.
> --------------------------
> *Alessandro Benedetti*
> Director @ Sease Ltd.
> *Apache Lucene/Solr Committer*
> *Apache Solr PMC Member*
>
> e-mail: a.benedetti@sease.io
>
>
> *Sease* - Information Retrieval Applied
> Consulting | Training | Open Source
>
> Website: Sease.io <http://sease.io/>
> LinkedIn <https://linkedin.com/company/sease-ltd> | Twitter
> <https://twitter.com/seaseltd> | Youtube
> <https://www.youtube.com/channel/UCDx86ZKLYNpI3gzMercM7BQ> | Github
> <https://github.com/seaseltd>
>

Re: [VOTE] Dimension Limit for KNN Vectors [ In reply to ]

michael.wechner at wyona

May 16, 2023, 3:05 AM

Post #7 of 46 (855 views)

Permalink

my non-binding vote goes to Option 2 resp. Option 4

Thanks

Michael Wechner

Am 16.05.23 um 10:51 schrieb Alessandro Benedetti:
> My vote goes to *Option 4*.
> --------------------------
> *Alessandro Benedetti*
> Director @ Sease Ltd.
> /Apache Lucene/Solr Committer/
> /Apache Solr PMC Member/
>
> e-mail: a.benedetti@sease.io/
> /
>
> *Sease*?- Information Retrieval Applied
> Consulting | Training | Open Source
>
> Website: Sease.io <http://sease.io/>
> LinkedIn <https://linkedin.com/company/sease-ltd>?| Twitter
> <https://twitter.com/seaseltd>?| Youtube
> <https://www.youtube.com/channel/UCDx86ZKLYNpI3gzMercM7BQ>?| Github
> <https://github.com/seaseltd>
>
>
> On Tue, 16 May 2023 at 09:50, Alessandro Benedetti
> <a.benedetti@sease.io> wrote:
>
> Hi all,
> we have finalized all the options proposed by the community and we
> are ready to vote for the preferred one and then proceed with the
> implementation.
>
> *Option 1*
> Keep it as it is (dimension limit hardcoded to 1024)
> *Motivation*:
> We are close to improving on many fronts. Given?the criticality of
> Lucene in computing infrastructure and the concerns raised by one
> of the most active stewards of the project, I think we should keep
> working toward improving the feature as is and move to up the
> limit after we can demonstrate improvement unambiguously.
>
> *Option 2*
> make the limit configurable, for example through a system property
> *Motivation*:
> The system administrator can enforce a limit its users need to
> respect that it's in line with whatever the admin decided to be
> acceptable for them.
> The default can stay the current one.
> This should open the doors for Apache Solr, Elasticsearch,
> OpenSearch, and any sort of plugin development
>
> *Option 3*
> Move the max dimension limit lower level to a HNSW specific
> implementation. Once there, this limit would not bind any other
> potential vector engine alternative/evolution.*
> *
> *Motivation:*There seem to be contradictory performance
> interpretations about the current HNSW implementation. Some
> consider its performance ok, some not, and it depends on the
> target data set and use case. Increasing the max dimension limit
> where it is currently (in top level FloatVectorValues) would not
> allow potential?alternatives (e.g. for other use-cases) to be
> based on a lower limit.
>
> *Option 4*
> Make it configurable and move it to an appropriate place.
> In particular, a
> simple?Integer.getInteger("lucene.hnsw.maxDimensions", 1024)
> should be enough.
> *Motivation*:
> Both are good and not mutually exclusive and could happen in any
> order.
> Someone suggested to perfect what the _default_ limit should be,
> but I've not?seen an argument _against_ configurability.?
> Especially in this way -- a toggle that doesn't bind Lucene's APIs
> in any way.
>
> I'll keep this [VOTE] open for a week and then proceed to the
> implementation.
> --------------------------
> *Alessandro Benedetti*
> Director @ Sease Ltd.
> /Apache Lucene/Solr Committer/
> /Apache Solr PMC Member/
>
> e-mail: a.benedetti@sease.io/
> /
>
> *Sease*?- Information Retrieval Applied
> Consulting | Training | Open Source
>
> Website: Sease.io <http://sease.io/>
> LinkedIn <https://linkedin.com/company/sease-ltd>?| Twitter
> <https://twitter.com/seaseltd>?| Youtube
> <https://www.youtube.com/channel/UCDx86ZKLYNpI3gzMercM7BQ>?|
> Github <https://github.com/seaseltd>
>

Re: [VOTE] Dimension Limit for KNN Vectors [ In reply to ]

gus.heck at gmail

May 16, 2023, 4:45 AM

Post #8 of 46 (855 views)

Permalink

Robert,

Can you explain in clear technical terms the standard that must be met for
performance? A benchmark that must run in X time on Y hardware for example
(and why that test is suitable)? Or some other reproducible criteria? So
far I've heard you give an *opinion* that it's unusable, but that's not a
technical criteria, others may have a different concept of what is usable
to them.

Forgive me if I misunderstand, but the essence of your argument has seemed
to be

"Performance isn't good enough, therefore we should force anyone who wants
to experiment with something bigger to fork the code base to do it"

Thus, it is necessary to have a clear unambiguous standard that anyone can
verify for "good enough". A clear standard would also focus efforts at
improvement.

Where are the goal posts?

FWIW I'm +1 on any of 2-4 since I believe the existence of a hard limit is
fundamentally counterproductive in an open source setting, as it will lead
to *fewer people* pushing the limits. Extremely few people are going to get
into the nitty-gritty of optimizing things unless they are staring at code
that they can prove does something interesting, but doesn't run fast enough
for their purposes. If people hit a hard limit, more of them give up and
never develop the code that will motivate them to look for optimizations.

-Gus

On Tue, May 16, 2023 at 6:04?AM Robert Muir <rcmuir@gmail.com> wrote:

> i still feel -1 (veto) on increasing this limit. sending more emails does
> not change the technical facts or make the veto go away.
>
> On Tue, May 16, 2023 at 4:50?AM Alessandro Benedetti <a.benedetti@sease.io>
> wrote:
>
>> Hi all,
>> we have finalized all the options proposed by the community and we are
>> ready to vote for the preferred one and then proceed with the
>> implementation.
>>
>> *Option 1*
>> Keep it as it is (dimension limit hardcoded to 1024)
>> *Motivation*:
>> We are close to improving on many fronts. Given the criticality of Lucene
>> in computing infrastructure and the concerns raised by one of the most
>> active stewards of the project, I think we should keep working toward
>> improving the feature as is and move to up the limit after we can
>> demonstrate improvement unambiguously.
>>
>> *Option 2*
>> make the limit configurable, for example through a system property
>> *Motivation*:
>> The system administrator can enforce a limit its users need to respect
>> that it's in line with whatever the admin decided to be acceptable for
>> them.
>> The default can stay the current one.
>> This should open the doors for Apache Solr, Elasticsearch, OpenSearch,
>> and any sort of plugin development
>>
>> *Option 3*
>> Move the max dimension limit lower level to a HNSW specific
>> implementation. Once there, this limit would not bind any other potential
>> vector engine alternative/evolution.
>> *Motivation:* There seem to be contradictory performance interpretations
>> about the current HNSW implementation. Some consider its performance ok,
>> some not, and it depends on the target data set and use case. Increasing
>> the max dimension limit where it is currently (in top level
>> FloatVectorValues) would not allow potential alternatives (e.g. for other
>> use-cases) to be based on a lower limit.
>>
>> *Option 4*
>> Make it configurable and move it to an appropriate place.
>> In particular, a simple Integer.getInteger("lucene.hnsw.maxDimensions",
>> 1024) should be enough.
>> *Motivation*:
>> Both are good and not mutually exclusive and could happen in any order.
>> Someone suggested to perfect what the _default_ limit should be, but I've
>> not seen an argument _against_ configurability. Especially in this way --
>> a toggle that doesn't bind Lucene's APIs in any way.
>>
>> I'll keep this [VOTE] open for a week and then proceed to the
>> implementation.
>> --------------------------
>> *Alessandro Benedetti*
>> Director @ Sease Ltd.
>> *Apache Lucene/Solr Committer*
>> *Apache Solr PMC Member*
>>
>> e-mail: a.benedetti@sease.io
>>
>>
>> *Sease* - Information Retrieval Applied
>> Consulting | Training | Open Source
>>
>> Website: Sease.io <http://sease.io/>
>> LinkedIn <https://linkedin.com/company/sease-ltd> | Twitter
>> <https://twitter.com/seaseltd> | Youtube
>> <https://www.youtube.com/channel/UCDx86ZKLYNpI3gzMercM7BQ> | Github
>> <https://github.com/seaseltd>
>>
>

--
http://www.needhamsoftware.com (work)
http://www.the111shift.com (play)

Re: [VOTE] Dimension Limit for KNN Vectors [ In reply to ]

michael.wechner at wyona

May 16, 2023, 5:01 AM

Post #9 of 46 (855 views)

Permalink

+1 to Gus' reply.

I think that Robert's veto or anyone else's veto is fair enough, but I
also think that anyone who is vetoing should be very clear about the
objectives / goals to be achieved, in order to get a +1.

If no clear objectives / goals can be defined and agreed on, then the
whole thing becomes arbitrary.

Therefore I would also be interested to know the objectives / goals to
be met that there will be a +1 re this vote?

Thanks

Michael

Am 16.05.23 um 13:45 schrieb Gus Heck:
> Robert,
>
> Can you explain in clear technical terms the standard that must be met
> for performance? A benchmark that must run in X time on Y hardware for
> example (and why that test is suitable)? Or some other reproducible
> criteria? So far I've heard you give an *opinion* that it's unusable,
> but that's not a technical criteria, others may have a different
> concept of what is usable to them.
>
> Forgive me if I misunderstand, but the essence of your argument has
> seemed to be
>
> "Performance isn't good enough, therefore we should force anyone who
> wants to experiment with something bigger to fork the code base to do it"
>
> Thus, it is necessary to have a clear unambiguous standard that anyone
> can verify for "good enough". A clear standard would also focus
> efforts at improvement.
>
> Where are the goal posts?
>
> FWIW I'm +1 on any of 2-4 since I believe the existence of a hard
> limit is fundamentally counterproductive in an open source setting, as
> it will lead to *fewer people* pushing the limits. Extremely few
> people are going to get into the nitty-gritty of optimizing things
> unless they are staring at code that they can prove does something
> interesting, but doesn't run fast enough for their purposes. If people
> hit a hard limit, more of them give up and never develop the code that
> will motivate them to look for optimizations.
>
> -Gus
>
> On Tue, May 16, 2023 at 6:04?AM Robert Muir <rcmuir@gmail.com> wrote:
>
> i still feel -1 (veto) on increasing this limit. sending more
> emails does not change the technical facts or make the veto go away.
>
> On Tue, May 16, 2023 at 4:50?AM Alessandro Benedetti
> <a.benedetti@sease.io> wrote:
>
> Hi all,
> we have finalized all the options proposed by the community
> and we are ready to vote for the preferred one and then
> proceed with the implementation.
>
> *Option 1*
> Keep it as it is (dimension limit hardcoded to 1024)
> *Motivation*:
> We are close to improving on many fronts. Given the
> criticality of Lucene in computing infrastructure and the
> concerns raised by one of the most active stewards of the
> project, I think we should keep working toward improving the
> feature as is and move to up the limit after we can
> demonstrate improvement unambiguously.
>
> *Option 2*
> make the limit configurable, for example through a system property
> *Motivation*:
> The system administrator can enforce a limit its users need to
> respect that it's in line with whatever the admin decided to
> be acceptable for them.
> The default can stay the current one.
> This should open the doors for Apache Solr, Elasticsearch,
> OpenSearch, and any sort of plugin development
>
> *Option 3*
> Move the max dimension limit lower level to a HNSW specific
> implementation. Once there, this limit would not bind any
> other potential vector engine alternative/evolution.*
> *
> *Motivation:*There seem to be contradictory performance
> interpretations about the current HNSW implementation. Some
> consider its performance ok, some not, and it depends on the
> target data set and use case. Increasing the max dimension
> limit where it is currently (in top level FloatVectorValues)
> would not allow potential alternatives (e.g. for other
> use-cases) to be based on a lower limit.
>
> *Option 4*
> Make it configurable and move it to an appropriate place.
> In particular, a
> simple Integer.getInteger("lucene.hnsw.maxDimensions", 1024)
> should be enough.
> *Motivation*:
> Both are good and not mutually exclusive and could happen in
> any order.
> Someone suggested to perfect what the _default_ limit should
> be, but I've not seen an argument _against_ configurability.
> Especially in this way -- a toggle that doesn't bind Lucene's
> APIs in any way.
>
> I'll keep this [VOTE] open for a week and then proceed to the
> implementation.
> --------------------------
> *Alessandro Benedetti*
> Director @ Sease Ltd.
> /Apache Lucene/Solr Committer/
> /Apache Solr PMC Member/
>
> e-mail: a.benedetti@sease.io/
> /
>
> *Sease* - Information Retrieval Applied
> Consulting | Training | Open Source
>
> Website: Sease.io <http://sease.io/>
> LinkedIn <https://linkedin.com/company/sease-ltd> | Twitter
> <https://twitter.com/seaseltd> | Youtube
> <https://www.youtube.com/channel/UCDx86ZKLYNpI3gzMercM7BQ> |
> Github <https://github.com/seaseltd>
>
>
>
> --
> http://www.needhamsoftware.com (work)
> http://www.the111shift.com (play)

Re: [VOTE] Dimension Limit for KNN Vectors [ In reply to ]

ben.w.trent at gmail

May 16, 2023, 5:26 AM

Post #10 of 46 (855 views)

Permalink

My vote is for option 3. Prevents Lucene from having the limit increased.
Allows others who implement a different codec to set a limit of their
choosing.

Though I don't know the historical reasons for putting specific
configuration items at the codec level. This limit is performance related
and various codec implementations would have different performance concerns.

On Tue, May 16, 2023, 8:02 AM Michael Wechner <michael.wechner@wyona.com>
wrote:

> +1 to Gus' reply.
>
> I think that Robert's veto or anyone else's veto is fair enough, but I
> also think that anyone who is vetoing should be very clear about the
> objectives / goals to be achieved, in order to get a +1.
>
> If no clear objectives / goals can be defined and agreed on, then the
> whole thing becomes arbitrary.
>
> Therefore I would also be interested to know the objectives / goals to be
> met that there will be a +1 re this vote?
>
> Thanks
>
> Michael
>
>
>
> Am 16.05.23 um 13:45 schrieb Gus Heck:
>
> Robert,
>
> Can you explain in clear technical terms the standard that must be met for
> performance? A benchmark that must run in X time on Y hardware for example
> (and why that test is suitable)? Or some other reproducible criteria? So
> far I've heard you give an *opinion* that it's unusable, but that's not a
> technical criteria, others may have a different concept of what is usable
> to them.
>
> Forgive me if I misunderstand, but the essence of your argument has seemed
> to be
>
> "Performance isn't good enough, therefore we should force anyone who wants
> to experiment with something bigger to fork the code base to do it"
>
> Thus, it is necessary to have a clear unambiguous standard that anyone can
> verify for "good enough". A clear standard would also focus efforts at
> improvement.
>
> Where are the goal posts?
>
> FWIW I'm +1 on any of 2-4 since I believe the existence of a hard limit is
> fundamentally counterproductive in an open source setting, as it will lead
> to *fewer people* pushing the limits. Extremely few people are going to
> get into the nitty-gritty of optimizing things unless they are staring at
> code that they can prove does something interesting, but doesn't run fast
> enough for their purposes. If people hit a hard limit, more of them give up
> and never develop the code that will motivate them to look for
> optimizations.
>
> -Gus
>
> On Tue, May 16, 2023 at 6:04?AM Robert Muir <rcmuir@gmail.com> wrote:
>
>> i still feel -1 (veto) on increasing this limit. sending more emails does
>> not change the technical facts or make the veto go away.
>>
>> On Tue, May 16, 2023 at 4:50?AM Alessandro Benedetti <
>> a.benedetti@sease.io> wrote:
>>
>>> Hi all,
>>> we have finalized all the options proposed by the community and we are
>>> ready to vote for the preferred one and then proceed with the
>>> implementation.
>>>
>>> *Option 1*
>>> Keep it as it is (dimension limit hardcoded to 1024)
>>> *Motivation*:
>>> We are close to improving on many fronts. Given the criticality of
>>> Lucene in computing infrastructure and the concerns raised by one of the
>>> most active stewards of the project, I think we should keep working toward
>>> improving the feature as is and move to up the limit after we can
>>> demonstrate improvement unambiguously.
>>>
>>> *Option 2*
>>> make the limit configurable, for example through a system property
>>> *Motivation*:
>>> The system administrator can enforce a limit its users need to respect
>>> that it's in line with whatever the admin decided to be acceptable for
>>> them.
>>> The default can stay the current one.
>>> This should open the doors for Apache Solr, Elasticsearch, OpenSearch,
>>> and any sort of plugin development
>>>
>>> *Option 3*
>>> Move the max dimension limit lower level to a HNSW specific
>>> implementation. Once there, this limit would not bind any other potential
>>> vector engine alternative/evolution.
>>> *Motivation:* There seem to be contradictory performance
>>> interpretations about the current HNSW implementation. Some consider its
>>> performance ok, some not, and it depends on the target data set and use
>>> case. Increasing the max dimension limit where it is currently (in top
>>> level FloatVectorValues) would not allow potential alternatives (e.g. for
>>> other use-cases) to be based on a lower limit.
>>>
>>> *Option 4*
>>> Make it configurable and move it to an appropriate place.
>>> In particular, a simple Integer.getInteger("lucene.hnsw.maxDimensions",
>>> 1024) should be enough.
>>> *Motivation*:
>>> Both are good and not mutually exclusive and could happen in any order.
>>> Someone suggested to perfect what the _default_ limit should be, but
>>> I've not seen an argument _against_ configurability. Especially in this
>>> way -- a toggle that doesn't bind Lucene's APIs in any way.
>>>
>>> I'll keep this [VOTE] open for a week and then proceed to the
>>> implementation.
>>> --------------------------
>>> *Alessandro Benedetti*
>>> Director @ Sease Ltd.
>>> *Apache Lucene/Solr Committer*
>>> *Apache Solr PMC Member*
>>>
>>> e-mail: a.benedetti@sease.io
>>>
>>>
>>> *Sease* - Information Retrieval Applied
>>> Consulting | Training | Open Source
>>>
>>> Website: Sease.io <http://sease.io/>
>>> LinkedIn <https://linkedin.com/company/sease-ltd> | Twitter
>>> <https://twitter.com/seaseltd> | Youtube
>>> <https://www.youtube.com/channel/UCDx86ZKLYNpI3gzMercM7BQ> | Github
>>> <https://github.com/seaseltd>
>>>
>>
>
> --
> http://www.needhamsoftware.com (work)
> http://www.the111shift.com (play)
>
>
>

Re: [VOTE] Dimension Limit for KNN Vectors [ In reply to ]

dawid.weiss at gmail

May 16, 2023, 6:37 AM

Post #11 of 46 (855 views)

Permalink

I'm for option 3 (limit at algorithm level), with the default there tunable
via property (option 4).

I understand Robert's concerns and I'd love to contribute a faster
implementation but the reality is - I can't do it at the moment. I feel
like experiments are good though and we shouldn't just ban people from
trying - if somebody changes the (sane) default and gets burned by
performance, perhaps it'll be an itch to work on speeding things up (much
like it's already happening with Jonathan's patch).

Dawid

On Tue, May 16, 2023 at 10:50?AM Alessandro Benedetti <a.benedetti@sease.io>
wrote:

> Hi all,
> we have finalized all the options proposed by the community and we are
> ready to vote for the preferred one and then proceed with the
> implementation.
>
> *Option 1*
> Keep it as it is (dimension limit hardcoded to 1024)
> *Motivation*:
> We are close to improving on many fronts. Given the criticality of Lucene
> in computing infrastructure and the concerns raised by one of the most
> active stewards of the project, I think we should keep working toward
> improving the feature as is and move to up the limit after we can
> demonstrate improvement unambiguously.
>
> *Option 2*
> make the limit configurable, for example through a system property
> *Motivation*:
> The system administrator can enforce a limit its users need to respect
> that it's in line with whatever the admin decided to be acceptable for
> them.
> The default can stay the current one.
> This should open the doors for Apache Solr, Elasticsearch, OpenSearch, and
> any sort of plugin development
>
> *Option 3*
> Move the max dimension limit lower level to a HNSW specific
> implementation. Once there, this limit would not bind any other potential
> vector engine alternative/evolution.
> *Motivation:* There seem to be contradictory performance interpretations
> about the current HNSW implementation. Some consider its performance ok,
> some not, and it depends on the target data set and use case. Increasing
> the max dimension limit where it is currently (in top level
> FloatVectorValues) would not allow potential alternatives (e.g. for other
> use-cases) to be based on a lower limit.
>
> *Option 4*
> Make it configurable and move it to an appropriate place.
> In particular, a simple Integer.getInteger("lucene.hnsw.maxDimensions",
> 1024) should be enough.
> *Motivation*:
> Both are good and not mutually exclusive and could happen in any order.
> Someone suggested to perfect what the _default_ limit should be, but I've
> not seen an argument _against_ configurability. Especially in this way --
> a toggle that doesn't bind Lucene's APIs in any way.
>
> I'll keep this [VOTE] open for a week and then proceed to the
> implementation.
> --------------------------
> *Alessandro Benedetti*
> Director @ Sease Ltd.
> *Apache Lucene/Solr Committer*
> *Apache Solr PMC Member*
>
> e-mail: a.benedetti@sease.io
>
>
> *Sease* - Information Retrieval Applied
> Consulting | Training | Open Source
>
> Website: Sease.io <http://sease.io/>
> LinkedIn <https://linkedin.com/company/sease-ltd> | Twitter
> <https://twitter.com/seaseltd> | Youtube
> <https://www.youtube.com/channel/UCDx86ZKLYNpI3gzMercM7BQ> | Github
> <https://github.com/seaseltd>
>

Re: [VOTE] Dimension Limit for KNN Vectors [ In reply to ]

uwe at thetaphi

May 16, 2023, 6:56 AM

Post #12 of 46 (855 views)

Permalink

I agree with Dawid,

I am +1 for those two options in combination:

* option 3 (make limit an HNSW specific thing). New formats may use
other limits (lower or higher).
* option 4 (make a system property with HNSW prefix). Adding the
system property must be done in same way like new properties for
MMAP directory (including access controller) so it can be denied by
system admin to be set in code (see
https://github.com/apache/lucene/blob/f53eb28af053d7612f7e4d1b2de05d33dc410645/lucene/core/src/java/org/apache/lucene/store/MMapDirectory.java#L327-L346
for example). Care has to be taken that the static initializers
won't fail is system properties cannot be read/set (system
adminitrator enforces default -> see mmap code). It also has to be
made sure that an index written with raised limit can still be read
without the limit, so the limit should not be glued into the file
format. Otherwise I disagree with option 4.

In short: I am fine with making it configurable only for HNSW if the
limit is not glued into index format. The default should only be there
to by default prevent people from doing wrong things, but changing
default should not break reading/modifiying those indexes.

Uwe

Am 16.05.2023 um 15:37 schrieb Dawid Weiss:
>
> I'm for option 3 (limit at algorithm level), with the default there
> tunable via property (option 4).
>
> I understand Robert's concerns and I'd love to contribute a faster
> implementation but the reality is - I can't do it at the moment. I
> feel like experiments are good though and we shouldn't just ban people
> from trying - if somebody changes the (sane) default and gets burned
> by performance, perhaps it'll be an itch to work on speeding things up
> (much like it's already happening with Jonathan's patch).
>
> Dawid
>
> On Tue, May 16, 2023 at 10:50?AM Alessandro Benedetti
> <a.benedetti@sease.io> wrote:
>
> Hi all,
> we have finalized all the options proposed by the community and we
> are ready to vote for the preferred one and then proceed with the
> implementation.
>
> *Option 1*
> Keep it as it is (dimension limit hardcoded to 1024)
> *Motivation*:
> We are close to improving on many fronts. Given the criticality of
> Lucene in computing infrastructure and the concerns raised by one
> of the most active stewards of the project, I think we should keep
> working toward improving the feature as is and move to up the
> limit after we can demonstrate improvement unambiguously.
>
> *Option 2*
> make the limit configurable, for example through a system property
> *Motivation*:
> The system administrator can enforce a limit its users need to
> respect that it's in line with whatever the admin decided to be
> acceptable for them.
> The default can stay the current one.
> This should open the doors for Apache Solr, Elasticsearch,
> OpenSearch, and any sort of plugin development
>
> *Option 3*
> Move the max dimension limit lower level to a HNSW specific
> implementation. Once there, this limit would not bind any other
> potential vector engine alternative/evolution.*
> *
> *Motivation:*There seem to be contradictory performance
> interpretations about the current HNSW implementation. Some
> consider its performance ok, some not, and it depends on the
> target data set and use case. Increasing the max dimension limit
> where it is currently (in top level FloatVectorValues) would not
> allow potential alternatives (e.g. for other use-cases) to be
> based on a lower limit.
>
> *Option 4*
> Make it configurable and move it to an appropriate place.
> In particular, a
> simple Integer.getInteger("lucene.hnsw.maxDimensions", 1024)
> should be enough.
> *Motivation*:
> Both are good and not mutually exclusive and could happen in any
> order.
> Someone suggested to perfect what the _default_ limit should be,
> but I've not seen an argument _against_ configurability.
> Especially in this way -- a toggle that doesn't bind Lucene's APIs
> in any way.
>
> I'll keep this [VOTE] open for a week and then proceed to the
> implementation.
> --------------------------
> *Alessandro Benedetti*
> Director @ Sease Ltd.
> /Apache Lucene/Solr Committer/
> /Apache Solr PMC Member/
>
> e-mail: a.benedetti@sease.io/
> /
>
> *Sease* - Information Retrieval Applied
> Consulting | Training | Open Source
>
> Website: Sease.io <http://sease.io/>
> LinkedIn <https://linkedin.com/company/sease-ltd> | Twitter
> <https://twitter.com/seaseltd> | Youtube
> <https://www.youtube.com/channel/UCDx86ZKLYNpI3gzMercM7BQ> |
> Github <https://github.com/seaseltd>
>
--
Uwe Schindler
Achterdiek 19, D-28357 Bremen
https://www.thetaphi.de
eMail:uwe@thetaphi.de

Re: [VOTE] Dimension Limit for KNN Vectors [ In reply to ]

houston at apache

May 16, 2023, 7:09 AM

Post #13 of 46 (855 views)

Permalink

+1 on the combination of #3 and #4.

Also good things to make sure of Uwe, thanks for calling those out.
(Especially about the limit only being used on write, not on read).

- Houston

On Tue, May 16, 2023 at 9:57?AM Uwe Schindler <uwe@thetaphi.de> wrote:

> I agree with Dawid,
>
> I am +1 for those two options in combination:
>
> - option 3 (make limit an HNSW specific thing). New formats may use
> other limits (lower or higher).
> - option 4 (make a system property with HNSW prefix). Adding the
> system property must be done in same way like new properties for MMAP
> directory (including access controller) so it can be denied by system admin
> to be set in code (see
> https://github.com/apache/lucene/blob/f53eb28af053d7612f7e4d1b2de05d33dc410645/lucene/core/src/java/org/apache/lucene/store/MMapDirectory.java#L327-L346
> for example). Care has to be taken that the static initializers won't fail
> is system properties cannot be read/set (system adminitrator enforces
> default -> see mmap code). It also has to be made sure that an index
> written with raised limit can still be read without the limit, so the limit
> should not be glued into the file format. Otherwise I disagree with option
> 4.
>
> In short: I am fine with making it configurable only for HNSW if the limit
> is not glued into index format. The default should only be there to by
> default prevent people from doing wrong things, but changing default should
> not break reading/modifiying those indexes.
>
> Uwe
> Am 16.05.2023 um 15:37 schrieb Dawid Weiss:
>
>
> I'm for option 3 (limit at algorithm level), with the default there
> tunable via property (option 4).
>
> I understand Robert's concerns and I'd love to contribute a faster
> implementation but the reality is - I can't do it at the moment. I feel
> like experiments are good though and we shouldn't just ban people from
> trying - if somebody changes the (sane) default and gets burned by
> performance, perhaps it'll be an itch to work on speeding things up (much
> like it's already happening with Jonathan's patch).
>
> Dawid
>
> On Tue, May 16, 2023 at 10:50?AM Alessandro Benedetti <
> a.benedetti@sease.io> wrote:
>
>> Hi all,
>> we have finalized all the options proposed by the community and we are
>> ready to vote for the preferred one and then proceed with the
>> implementation.
>>
>> *Option 1*
>> Keep it as it is (dimension limit hardcoded to 1024)
>> *Motivation*:
>> We are close to improving on many fronts. Given the criticality of Lucene
>> in computing infrastructure and the concerns raised by one of the most
>> active stewards of the project, I think we should keep working toward
>> improving the feature as is and move to up the limit after we can
>> demonstrate improvement unambiguously.
>>
>> *Option 2*
>> make the limit configurable, for example through a system property
>> *Motivation*:
>> The system administrator can enforce a limit its users need to respect
>> that it's in line with whatever the admin decided to be acceptable for
>> them.
>> The default can stay the current one.
>> This should open the doors for Apache Solr, Elasticsearch, OpenSearch,
>> and any sort of plugin development
>>
>> *Option 3*
>> Move the max dimension limit lower level to a HNSW specific
>> implementation. Once there, this limit would not bind any other potential
>> vector engine alternative/evolution.
>> *Motivation:* There seem to be contradictory performance interpretations
>> about the current HNSW implementation. Some consider its performance ok,
>> some not, and it depends on the target data set and use case. Increasing
>> the max dimension limit where it is currently (in top level
>> FloatVectorValues) would not allow potential alternatives (e.g. for other
>> use-cases) to be based on a lower limit.
>>
>> *Option 4*
>> Make it configurable and move it to an appropriate place.
>> In particular, a simple Integer.getInteger("lucene.hnsw.maxDimensions",
>> 1024) should be enough.
>> *Motivation*:
>> Both are good and not mutually exclusive and could happen in any order.
>> Someone suggested to perfect what the _default_ limit should be, but I've
>> not seen an argument _against_ configurability. Especially in this way --
>> a toggle that doesn't bind Lucene's APIs in any way.
>>
>> I'll keep this [VOTE] open for a week and then proceed to the
>> implementation.
>> --------------------------
>> *Alessandro Benedetti*
>> Director @ Sease Ltd.
>> *Apache Lucene/Solr Committer*
>> *Apache Solr PMC Member*
>>
>> e-mail: a.benedetti@sease.io
>>
>>
>> *Sease* - Information Retrieval Applied
>> Consulting | Training | Open Source
>>
>> Website: Sease.io <http://sease.io/>
>> LinkedIn <https://linkedin.com/company/sease-ltd> | Twitter
>> <https://twitter.com/seaseltd> | Youtube
>> <https://www.youtube.com/channel/UCDx86ZKLYNpI3gzMercM7BQ> | Github
>> <https://github.com/seaseltd>
>>
> --
> Uwe Schindler
> Achterdiek 19, D-28357 Bremenhttps://www.thetaphi.de
> eMail: uwe@thetaphi.de
>
>

Re: [VOTE] Dimension Limit for KNN Vectors [ In reply to ]

marcuseagan at gmail

May 16, 2023, 7:26 AM

Post #14 of 46 (855 views)

Permalink

Given that Robert has put in his veto, aren’t we clear on what we need to
do for him to change his mind? He’s been pretty clear and the rules of veto
are cut and dry.

Most of the people that have contributed to kNN vectors recently are not
even on the thread. I think improving the feature should be the focus of
the Lucene community at this juncture.

On Tue, May 16, 2023 at 7:09 AM Houston Putman <houston@apache.org> wrote:

> +1 on the combination of #3 and #4.
>
> Also good things to make sure of Uwe, thanks for calling those out.
> (Especially about the limit only being used on write, not on read).
>
> - Houston
>
> On Tue, May 16, 2023 at 9:57?AM Uwe Schindler <uwe@thetaphi.de> wrote:
>
>> I agree with Dawid,
>>
>> I am +1 for those two options in combination:
>>
>> - option 3 (make limit an HNSW specific thing). New formats may use
>> other limits (lower or higher).
>> - option 4 (make a system property with HNSW prefix). Adding the
>> system property must be done in same way like new properties for MMAP
>> directory (including access controller) so it can be denied by system admin
>> to be set in code (see
>> https://github.com/apache/lucene/blob/f53eb28af053d7612f7e4d1b2de05d33dc410645/lucene/core/src/java/org/apache/lucene/store/MMapDirectory.java#L327-L346
>> for example). Care has to be taken that the static initializers won't fail
>> is system properties cannot be read/set (system adminitrator enforces
>> default -> see mmap code). It also has to be made sure that an index
>> written with raised limit can still be read without the limit, so the limit
>> should not be glued into the file format. Otherwise I disagree with option
>> 4.
>>
>> In short: I am fine with making it configurable only for HNSW if the
>> limit is not glued into index format. The default should only be there to
>> by default prevent people from doing wrong things, but changing default
>> should not break reading/modifiying those indexes.
>>
>> Uwe
>> Am 16.05.2023 um 15:37 schrieb Dawid Weiss:
>>
>>
>> I'm for option 3 (limit at algorithm level), with the default there
>> tunable via property (option 4).
>>
>> I understand Robert's concerns and I'd love to contribute a faster
>> implementation but the reality is - I can't do it at the moment. I feel
>> like experiments are good though and we shouldn't just ban people from
>> trying - if somebody changes the (sane) default and gets burned by
>> performance, perhaps it'll be an itch to work on speeding things up (much
>> like it's already happening with Jonathan's patch).
>>
>> Dawid
>>
>> On Tue, May 16, 2023 at 10:50?AM Alessandro Benedetti <
>> a.benedetti@sease.io> wrote:
>>
>>> Hi all,
>>> we have finalized all the options proposed by the community and we are
>>> ready to vote for the preferred one and then proceed with the
>>> implementation.
>>>
>>> *Option 1*
>>> Keep it as it is (dimension limit hardcoded to 1024)
>>> *Motivation*:
>>> We are close to improving on many fronts. Given the criticality of
>>> Lucene in computing infrastructure and the concerns raised by one of the
>>> most active stewards of the project, I think we should keep working toward
>>> improving the feature as is and move to up the limit after we can
>>> demonstrate improvement unambiguously.
>>>
>>> *Option 2*
>>> make the limit configurable, for example through a system property
>>> *Motivation*:
>>> The system administrator can enforce a limit its users need to respect
>>> that it's in line with whatever the admin decided to be acceptable for
>>> them.
>>> The default can stay the current one.
>>> This should open the doors for Apache Solr, Elasticsearch, OpenSearch,
>>> and any sort of plugin development
>>>
>>> *Option 3*
>>> Move the max dimension limit lower level to a HNSW specific
>>> implementation. Once there, this limit would not bind any other potential
>>> vector engine alternative/evolution.
>>> *Motivation:* There seem to be contradictory performance
>>> interpretations about the current HNSW implementation. Some consider its
>>> performance ok, some not, and it depends on the target data set and use
>>> case. Increasing the max dimension limit where it is currently (in top
>>> level FloatVectorValues) would not allow potential alternatives (e.g. for
>>> other use-cases) to be based on a lower limit.
>>>
>>> *Option 4*
>>> Make it configurable and move it to an appropriate place.
>>> In particular, a simple Integer.getInteger("lucene.hnsw.maxDimensions",
>>> 1024) should be enough.
>>> *Motivation*:
>>> Both are good and not mutually exclusive and could happen in any order.
>>> Someone suggested to perfect what the _default_ limit should be, but
>>> I've not seen an argument _against_ configurability. Especially in this
>>> way -- a toggle that doesn't bind Lucene's APIs in any way.
>>>
>>> I'll keep this [VOTE] open for a week and then proceed to the
>>> implementation.
>>> --------------------------
>>> *Alessandro Benedetti*
>>> Director @ Sease Ltd.
>>> *Apache Lucene/Solr Committer*
>>> *Apache Solr PMC Member*
>>>
>>> e-mail: a.benedetti@sease.io
>>>
>>>
>>> *Sease* - Information Retrieval Applied
>>> Consulting | Training | Open Source
>>>
>>> Website: Sease.io <http://sease.io/>
>>> LinkedIn <https://linkedin.com/company/sease-ltd> | Twitter
>>> <https://twitter.com/seaseltd> | Youtube
>>> <https://www.youtube.com/channel/UCDx86ZKLYNpI3gzMercM7BQ> | Github
>>> <https://github.com/seaseltd>
>>>
>> --
>> Uwe SchindlerAchterdiek 19, D-28357 Bremen <https://www.google.com/maps/search/Achterdiek+19,+D-28357+Bremen?entry=gmail&source=g>https://www.thetaphi.de
>> eMail: uwe@thetaphi.de
>>
>> --
Marcus Eagan

Re: [VOTE] Dimension Limit for KNN Vectors [ In reply to ]

a.benedetti at sease

May 16, 2023, 7:46 AM

Post #15 of 46 (855 views)

Permalink

Hi Marcus,
I am afraid at this stage Robert's opinion counts just as any other
opinion, a single vote for option 1.
We are collecting a community's feedback here, we are not changing any code
nor voting for a yes/no.
Once the voting is finished, we'll operate an action depending on the
community's choice.
If the action involves making a change and someone(Robert or whoever) feels
to veto it, he/she will need to motivate the veto with technical merit.

In response to Uwe point:

>
>> On Tue, May 16, 2023 at 9:57?AM Uwe Schindler <uwe@thetaphi.de> wrote:
>>
>>> I agree with Dawid,
>>>
>>> I am +1 for those two options in combination:
>>>
>>> - option 3 (make limit an HNSW specific thing). New formats may use
>>> other limits (lower or higher).
>>> - option 4 (make a system property with HNSW prefix). Adding the
>>> system property must be done in same way like new properties for MMAP
>>> directory (including access controller) so it can be denied by system admin
>>> to be set in code (see
>>> https://github.com/apache/lucene/blob/f53eb28af053d7612f7e4d1b2de05d33dc410645/lucene/core/src/java/org/apache/lucene/store/MMapDirectory.java#L327-L346
>>> for example). Care has to be taken that the static initializers won't fail
>>> is system properties cannot be read/set (system adminitrator enforces
>>> default -> see mmap code). It also has to be made sure that an index
>>> written with raised limit can still be read without the limit, so the limit
>>> should not be glued into the file format. Otherwise I disagree with option
>>> 4.
>>>
>>> In short: I am fine with making it configurable only for HNSW if the
>>> limit is not glued into index format. The default should only be there to
>>> by default prevent people from doing wrong things, but changing default
>>> should not break reading/modifiying those indexes.
>>>
>>> Uwe
>>>
>>> Thanks Uwe, that's very useful!
Just to fully understand it, right now the limit is not written in any file
format, so you just want this behavior to be maintained right?

Re: [VOTE] Dimension Limit for KNN Vectors [ In reply to ]

gus.heck at gmail

May 16, 2023, 7:51 AM

Post #16 of 46 (855 views)

Permalink

Actually, I had wondered if this is a proper vote thread or not, normally
those are yes/no on a single option.

On Tue, May 16, 2023 at 10:47?AM Alessandro Benedetti <a.benedetti@sease.io>
wrote:

> Hi Marcus,
> I am afraid at this stage Robert's opinion counts just as any other
> opinion, a single vote for option 1.
> We are collecting a community's feedback here, we are not changing any
> code nor voting for a yes/no.
> Once the voting is finished, we'll operate an action depending on the
> community's choice.
> If the action involves making a change and someone(Robert or whoever)
> feels to veto it, he/she will need to motivate the veto with technical
> merit.
>
> In response to Uwe point:
>
>>
>>> On Tue, May 16, 2023 at 9:57?AM Uwe Schindler <uwe@thetaphi.de> wrote:
>>>
>>>> I agree with Dawid,
>>>>
>>>> I am +1 for those two options in combination:
>>>>
>>>> - option 3 (make limit an HNSW specific thing). New formats may use
>>>> other limits (lower or higher).
>>>> - option 4 (make a system property with HNSW prefix). Adding the
>>>> system property must be done in same way like new properties for MMAP
>>>> directory (including access controller) so it can be denied by system admin
>>>> to be set in code (see
>>>> https://github.com/apache/lucene/blob/f53eb28af053d7612f7e4d1b2de05d33dc410645/lucene/core/src/java/org/apache/lucene/store/MMapDirectory.java#L327-L346
>>>> for example). Care has to be taken that the static initializers won't fail
>>>> is system properties cannot be read/set (system adminitrator enforces
>>>> default -> see mmap code). It also has to be made sure that an index
>>>> written with raised limit can still be read without the limit, so the limit
>>>> should not be glued into the file format. Otherwise I disagree with option
>>>> 4.
>>>>
>>>> In short: I am fine with making it configurable only for HNSW if the
>>>> limit is not glued into index format. The default should only be there to
>>>> by default prevent people from doing wrong things, but changing default
>>>> should not break reading/modifiying those indexes.
>>>>
>>>> Uwe
>>>>
>>>> Thanks Uwe, that's very useful!
> Just to fully understand it, right now the limit is not written in any
> file format, so you just want this behavior to be maintained right?
>
>

--
http://www.needhamsoftware.com (work)
http://www.the111shift.com (play)

Re: [VOTE] Dimension Limit for KNN Vectors [ In reply to ]

a.benedetti at sease

May 16, 2023, 8:06 AM

Post #17 of 46 (855 views)

Permalink

Even if the options can be basically summarised in two groups: make it
configurable VS not making it configurable and leave it be, when I
collected the options from people I ended up with these four and I didn't
want to collapse any of them (potentially making the proposer feel
diminished).

--------------------------
*Alessandro Benedetti*
Director @ Sease Ltd.
*Apache Lucene/Solr Committer*
*Apache Solr PMC Member*

e-mail: a.benedetti@sease.io

*Sease* - Information Retrieval Applied
Consulting | Training | Open Source

Website: Sease.io <http://sease.io/>
LinkedIn <https://linkedin.com/company/sease-ltd> | Twitter
<https://twitter.com/seaseltd> | Youtube
<https://www.youtube.com/channel/UCDx86ZKLYNpI3gzMercM7BQ> | Github
<https://github.com/seaseltd>

On Tue, 16 May 2023 at 15:54, Gus Heck <gus.heck@gmail.com> wrote:

> Actually, I had wondered if this is a proper vote thread or not, normally
> those are yes/no on a single option.
>
> On Tue, May 16, 2023 at 10:47?AM Alessandro Benedetti <
> a.benedetti@sease.io> wrote:
>
>> Hi Marcus,
>> I am afraid at this stage Robert's opinion counts just as any other
>> opinion, a single vote for option 1.
>> We are collecting a community's feedback here, we are not changing any
>> code nor voting for a yes/no.
>> Once the voting is finished, we'll operate an action depending on the
>> community's choice.
>> If the action involves making a change and someone(Robert or whoever)
>> feels to veto it, he/she will need to motivate the veto with technical
>> merit.
>>
>> In response to Uwe point:
>>
>>>
>>>> On Tue, May 16, 2023 at 9:57?AM Uwe Schindler <uwe@thetaphi.de> wrote:
>>>>
>>>>> I agree with Dawid,
>>>>>
>>>>> I am +1 for those two options in combination:
>>>>>
>>>>> - option 3 (make limit an HNSW specific thing). New formats may
>>>>> use other limits (lower or higher).
>>>>> - option 4 (make a system property with HNSW prefix). Adding the
>>>>> system property must be done in same way like new properties for MMAP
>>>>> directory (including access controller) so it can be denied by system admin
>>>>> to be set in code (see
>>>>> https://github.com/apache/lucene/blob/f53eb28af053d7612f7e4d1b2de05d33dc410645/lucene/core/src/java/org/apache/lucene/store/MMapDirectory.java#L327-L346
>>>>> for example). Care has to be taken that the static initializers won't fail
>>>>> is system properties cannot be read/set (system adminitrator enforces
>>>>> default -> see mmap code). It also has to be made sure that an index
>>>>> written with raised limit can still be read without the limit, so the limit
>>>>> should not be glued into the file format. Otherwise I disagree with option
>>>>> 4.
>>>>>
>>>>> In short: I am fine with making it configurable only for HNSW if the
>>>>> limit is not glued into index format. The default should only be there to
>>>>> by default prevent people from doing wrong things, but changing default
>>>>> should not break reading/modifiying those indexes.
>>>>>
>>>>> Uwe
>>>>>
>>>>> Thanks Uwe, that's very useful!
>> Just to fully understand it, right now the limit is not written in any
>> file format, so you just want this behavior to be maintained right?
>>
>>
>
> --
> http://www.needhamsoftware.com (work)
> http://www.the111shift.com (play)
>

Re: [VOTE] Dimension Limit for KNN Vectors [ In reply to ]

jbellis at gmail

May 16, 2023, 4:22 PM

Post #18 of 46 (855 views)

Permalink

My non-binding vote:

Option 2 = Option 4 > Option 1 > Option 3

Explanation: Lucene's somewhat arbitrary limit of 1024 does not currently
affect the raw, low-level HNSW, which is what I am plugging into
Cassandra. The only option that would break this code is option 3.

P.S. I mentioned this in another thread, but I'm happily throwing OpenAI's
ada embeddings of dimension 1536 at it and I've tested more.

On Tue, May 16, 2023 at 3:50?AM Alessandro Benedetti <a.benedetti@sease.io>
wrote:

> Hi all,
> we have finalized all the options proposed by the community and we are
> ready to vote for the preferred one and then proceed with the
> implementation.
>
> *Option 1*
> Keep it as it is (dimension limit hardcoded to 1024)
> *Motivation*:
> We are close to improving on many fronts. Given the criticality of Lucene
> in computing infrastructure and the concerns raised by one of the most
> active stewards of the project, I think we should keep working toward
> improving the feature as is and move to up the limit after we can
> demonstrate improvement unambiguously.
>
> *Option 2*
> make the limit configurable, for example through a system property
> *Motivation*:
> The system administrator can enforce a limit its users need to respect
> that it's in line with whatever the admin decided to be acceptable for
> them.
> The default can stay the current one.
> This should open the doors for Apache Solr, Elasticsearch, OpenSearch, and
> any sort of plugin development
>
> *Option 3*
> Move the max dimension limit lower level to a HNSW specific
> implementation. Once there, this limit would not bind any other potential
> vector engine alternative/evolution.
> *Motivation:* There seem to be contradictory performance interpretations
> about the current HNSW implementation. Some consider its performance ok,
> some not, and it depends on the target data set and use case. Increasing
> the max dimension limit where it is currently (in top level
> FloatVectorValues) would not allow potential alternatives (e.g. for other
> use-cases) to be based on a lower limit.
>
> *Option 4*
> Make it configurable and move it to an appropriate place.
> In particular, a simple Integer.getInteger("lucene.hnsw.maxDimensions",
> 1024) should be enough.
> *Motivation*:
> Both are good and not mutually exclusive and could happen in any order.
> Someone suggested to perfect what the _default_ limit should be, but I've
> not seen an argument _against_ configurability. Especially in this way --
> a toggle that doesn't bind Lucene's APIs in any way.
>
> I'll keep this [VOTE] open for a week and then proceed to the
> implementation.
> --------------------------
> *Alessandro Benedetti*
> Director @ Sease Ltd.
> *Apache Lucene/Solr Committer*
> *Apache Solr PMC Member*
>
> e-mail: a.benedetti@sease.io
>
>
> *Sease* - Information Retrieval Applied
> Consulting | Training | Open Source
>
> Website: Sease.io <http://sease.io/>
> LinkedIn <https://linkedin.com/company/sease-ltd> | Twitter
> <https://twitter.com/seaseltd> | Youtube
> <https://www.youtube.com/channel/UCDx86ZKLYNpI3gzMercM7BQ> | Github
> <https://github.com/seaseltd>
>

--
Jonathan Ellis
co-founder, http://www.datastax.com
@spyced

RE: [VOTE] Dimension Limit for KNN Vectors [ In reply to ]

pandu at s2search

May 16, 2023, 4:27 PM

Post #19 of 46 (855 views)

Permalink

Hi all,

Great to have this discussion!

My votes are for 2 and 4!

Best,

Pandu

On 2023/05/16 08:50:24 Alessandro Benedetti wrote:
> Hi all,
> we have finalized all the options proposed by the community and we are
> ready to vote for the preferred one and then proceed with the
> implementation.
>
> *Option 1*
> Keep it as it is (dimension limit hardcoded to 1024)
> *Motivation*:
> We are close to improving on many fronts. Given the criticality of Lucene
> in computing infrastructure and the concerns raised by one of the most
> active stewards of the project, I think we should keep working toward
> improving the feature as is and move to up the limit after we can
> demonstrate improvement unambiguously.
>
> *Option 2*
> make the limit configurable, for example through a system property
> *Motivation*:
> The system administrator can enforce a limit its users need to respect that
> it's in line with whatever the admin decided to be acceptable for them.
> The default can stay the current one.
> This should open the doors for Apache Solr, Elasticsearch, OpenSearch, and
> any sort of plugin development
>
> *Option 3*
> Move the max dimension limit lower level to a HNSW specific implementation.
> Once there, this limit would not bind any other potential vector engine
> alternative/evolution.
> *Motivation:* There seem to be contradictory performance interpretations
> about the current HNSW implementation. Some consider its performance ok,
> some not, and it depends on the target data set and use case. Increasing
> the max dimension limit where it is currently (in top level
> FloatVectorValues) would not allow potential alternatives (e.g. for other
> use-cases) to be based on a lower limit.
>
> *Option 4*
> Make it configurable and move it to an appropriate place.
> In particular, a simple Integer.getInteger("lucene.hnsw.maxDimensions",
> 1024) should be enough.
> *Motivation*:
> Both are good and not mutually exclusive and could happen in any order.
> Someone suggested to perfect what the _default_ limit should be, but I've
> not seen an argument _against_ configurability. Especially in this way --
> a toggle that doesn't bind Lucene's APIs in any way.
>
> I'll keep this [VOTE] open for a week and then proceed to the
> implementation.
> --------------------------
> *Alessandro Benedetti*
> Director @ Sease Ltd.
> *Apache Lucene/Solr Committer*
> *Apache Solr PMC Member*
>
> e-mail: a.benedetti@sease.io
>
>
> *Sease* - Information Retrieval Applied
> Consulting | Training | Open Source
>
> Website: Sease.io <http://sease.io/>
> LinkedIn <https://linkedin.com/company/sease-ltd> | Twitter
> <https://twitter.com/seaseltd> | Youtube
> <https://www.youtube.com/channel/UCDx86ZKLYNpI3gzMercM7BQ> | Github
> <https://github.com/seaseltd>
>

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org

RE: [VOTE] Dimension Limit for KNN Vectors [ In reply to ]

pandu at s2search

May 16, 2023, 4:38 PM

Post #20 of 46 (855 views)

Permalink

Re: [VOTE] Dimension Limit for KNN Vectors [ In reply to ]

rcmuir at gmail

May 16, 2023, 5:53 PM

Post #21 of 46 (855 views)

Permalink

Gus, I think i explained myself multiple times on issues and in this
thread. the performance is unacceptable, everyone knows it, but nobody is
talking about.
I don't need to explain myself time and time again here.
You don't seem to understand the technical issues (at least you sure as
fuck don't know how service loading works or you wouldnt have opened
https://github.com/apache/lucene/issues/12300 ????)

I'm just the only one here completely unconstrained by any of silicon
valley's influences to speak my true mind, without any repercussions, so I
do it. Don't give any fucks about ChatGPT.

I'm standing by my technical veto. If you bypass it, I'll revert the
offending commit.

As far as fixing the technical performance, I just opened an issue with
some ideas to at least improve cpu usage by a factor of N. It does not help
with the crazy heap memory usage or other issues of KNN implementation
causing shit like OOM on merge. But it is one step:
https://github.com/apache/lucene/issues/12302

On Tue, May 16, 2023 at 7:45?AM Gus Heck <gus.heck@gmail.com> wrote:

> Robert,
>
> Can you explain in clear technical terms the standard that must be met for
> performance? A benchmark that must run in X time on Y hardware for example
> (and why that test is suitable)? Or some other reproducible criteria? So
> far I've heard you give an *opinion* that it's unusable, but that's not a
> technical criteria, others may have a different concept of what is usable
> to them.
>
> Forgive me if I misunderstand, but the essence of your argument has seemed
> to be
>
> "Performance isn't good enough, therefore we should force anyone who wants
> to experiment with something bigger to fork the code base to do it"
>
> Thus, it is necessary to have a clear unambiguous standard that anyone can
> verify for "good enough". A clear standard would also focus efforts at
> improvement.
>
> Where are the goal posts?
>
> FWIW I'm +1 on any of 2-4 since I believe the existence of a hard limit is
> fundamentally counterproductive in an open source setting, as it will lead
> to *fewer people* pushing the limits. Extremely few people are going to
> get into the nitty-gritty of optimizing things unless they are staring at
> code that they can prove does something interesting, but doesn't run fast
> enough for their purposes. If people hit a hard limit, more of them give up
> and never develop the code that will motivate them to look for
> optimizations.
>
> -Gus
>
> On Tue, May 16, 2023 at 6:04?AM Robert Muir <rcmuir@gmail.com> wrote:
>
>> i still feel -1 (veto) on increasing this limit. sending more emails does
>> not change the technical facts or make the veto go away.
>>
>> On Tue, May 16, 2023 at 4:50?AM Alessandro Benedetti <
>> a.benedetti@sease.io> wrote:
>>
>>> Hi all,
>>> we have finalized all the options proposed by the community and we are
>>> ready to vote for the preferred one and then proceed with the
>>> implementation.
>>>
>>> *Option 1*
>>> Keep it as it is (dimension limit hardcoded to 1024)
>>> *Motivation*:
>>> We are close to improving on many fronts. Given the criticality of
>>> Lucene in computing infrastructure and the concerns raised by one of the
>>> most active stewards of the project, I think we should keep working toward
>>> improving the feature as is and move to up the limit after we can
>>> demonstrate improvement unambiguously.
>>>
>>> *Option 2*
>>> make the limit configurable, for example through a system property
>>> *Motivation*:
>>> The system administrator can enforce a limit its users need to respect
>>> that it's in line with whatever the admin decided to be acceptable for
>>> them.
>>> The default can stay the current one.
>>> This should open the doors for Apache Solr, Elasticsearch, OpenSearch,
>>> and any sort of plugin development
>>>
>>> *Option 3*
>>> Move the max dimension limit lower level to a HNSW specific
>>> implementation. Once there, this limit would not bind any other potential
>>> vector engine alternative/evolution.
>>> *Motivation:* There seem to be contradictory performance
>>> interpretations about the current HNSW implementation. Some consider its
>>> performance ok, some not, and it depends on the target data set and use
>>> case. Increasing the max dimension limit where it is currently (in top
>>> level FloatVectorValues) would not allow potential alternatives (e.g. for
>>> other use-cases) to be based on a lower limit.
>>>
>>> *Option 4*
>>> Make it configurable and move it to an appropriate place.
>>> In particular, a simple Integer.getInteger("lucene.hnsw.maxDimensions",
>>> 1024) should be enough.
>>> *Motivation*:
>>> Both are good and not mutually exclusive and could happen in any order.
>>> Someone suggested to perfect what the _default_ limit should be, but
>>> I've not seen an argument _against_ configurability. Especially in this
>>> way -- a toggle that doesn't bind Lucene's APIs in any way.
>>>
>>> I'll keep this [VOTE] open for a week and then proceed to the
>>> implementation.
>>> --------------------------
>>> *Alessandro Benedetti*
>>> Director @ Sease Ltd.
>>> *Apache Lucene/Solr Committer*
>>> *Apache Solr PMC Member*
>>>
>>> e-mail: a.benedetti@sease.io
>>>
>>>
>>> *Sease* - Information Retrieval Applied
>>> Consulting | Training | Open Source
>>>
>>> Website: Sease.io <http://sease.io/>
>>> LinkedIn <https://linkedin.com/company/sease-ltd> | Twitter
>>> <https://twitter.com/seaseltd> | Youtube
>>> <https://www.youtube.com/channel/UCDx86ZKLYNpI3gzMercM7BQ> | Github
>>> <https://github.com/seaseltd>
>>>
>>
>
> --
> http://www.needhamsoftware.com (work)
> http://www.the111shift.com (play)
>

Re: [VOTE] Dimension Limit for KNN Vectors [ In reply to ]

rcmuir at gmail

May 16, 2023, 6:50 PM

Post #22 of 46 (855 views)

Permalink

by the way, i agree with the idea to MOVE THE LIMIT UNCHANGED to the
hsnw-specific code.

This way, someone can write alternative codec with vectors using some other
completely different approach that incorporates a different more
appropriate limit (maybe lower, maybe higher) depending upon their
tradeoffs. We should encourage this as I think it is the "only true fix" to
the scalability issues: use a scalable algorithm! Also, alternative codecs
don't force the project into many years of index backwards compatibility,
which is really my penultimate concern. We can lock ourselves into a truly
bad place and become irrelevant (especially with scalar code implementing
all this vector stuff, it is really senseless). In the meantime I suggest
we try to reduce pain for the default codec with the current implementation
if possible. If it is not possible, we need a new codec that performs.

On Tue, May 16, 2023 at 8:53?PM Robert Muir <rcmuir@gmail.com> wrote:

> Gus, I think i explained myself multiple times on issues and in this
> thread. the performance is unacceptable, everyone knows it, but nobody is
> talking about.
> I don't need to explain myself time and time again here.
> You don't seem to understand the technical issues (at least you sure as
> fuck don't know how service loading works or you wouldnt have opened
> https://github.com/apache/lucene/issues/12300 ????)
>
> I'm just the only one here completely unconstrained by any of silicon
> valley's influences to speak my true mind, without any repercussions, so I
> do it. Don't give any fucks about ChatGPT.
>
> I'm standing by my technical veto. If you bypass it, I'll revert the
> offending commit.
>
> As far as fixing the technical performance, I just opened an issue with
> some ideas to at least improve cpu usage by a factor of N. It does not help
> with the crazy heap memory usage or other issues of KNN implementation
> causing shit like OOM on merge. But it is one step:
> https://github.com/apache/lucene/issues/12302
>
>
>
> On Tue, May 16, 2023 at 7:45?AM Gus Heck <gus.heck@gmail.com> wrote:
>
>> Robert,
>>
>> Can you explain in clear technical terms the standard that must be met
>> for performance? A benchmark that must run in X time on Y hardware for
>> example (and why that test is suitable)? Or some other reproducible
>> criteria? So far I've heard you give an *opinion* that it's unusable, but
>> that's not a technical criteria, others may have a different concept of
>> what is usable to them.
>>
>> Forgive me if I misunderstand, but the essence of your argument has
>> seemed to be
>>
>> "Performance isn't good enough, therefore we should force anyone who
>> wants to experiment with something bigger to fork the code base to do it"
>>
>> Thus, it is necessary to have a clear unambiguous standard that anyone
>> can verify for "good enough". A clear standard would also focus efforts at
>> improvement.
>>
>> Where are the goal posts?
>>
>> FWIW I'm +1 on any of 2-4 since I believe the existence of a hard limit
>> is fundamentally counterproductive in an open source setting, as it will
>> lead to *fewer people* pushing the limits. Extremely few people are
>> going to get into the nitty-gritty of optimizing things unless they are
>> staring at code that they can prove does something interesting, but doesn't
>> run fast enough for their purposes. If people hit a hard limit, more of
>> them give up and never develop the code that will motivate them to look for
>> optimizations.
>>
>> -Gus
>>
>> On Tue, May 16, 2023 at 6:04?AM Robert Muir <rcmuir@gmail.com> wrote:
>>
>>> i still feel -1 (veto) on increasing this limit. sending more emails
>>> does not change the technical facts or make the veto go away.
>>>
>>> On Tue, May 16, 2023 at 4:50?AM Alessandro Benedetti <
>>> a.benedetti@sease.io> wrote:
>>>
>>>> Hi all,
>>>> we have finalized all the options proposed by the community and we are
>>>> ready to vote for the preferred one and then proceed with the
>>>> implementation.
>>>>
>>>> *Option 1*
>>>> Keep it as it is (dimension limit hardcoded to 1024)
>>>> *Motivation*:
>>>> We are close to improving on many fronts. Given the criticality of
>>>> Lucene in computing infrastructure and the concerns raised by one of the
>>>> most active stewards of the project, I think we should keep working toward
>>>> improving the feature as is and move to up the limit after we can
>>>> demonstrate improvement unambiguously.
>>>>
>>>> *Option 2*
>>>> make the limit configurable, for example through a system property
>>>> *Motivation*:
>>>> The system administrator can enforce a limit its users need to respect
>>>> that it's in line with whatever the admin decided to be acceptable for
>>>> them.
>>>> The default can stay the current one.
>>>> This should open the doors for Apache Solr, Elasticsearch, OpenSearch,
>>>> and any sort of plugin development
>>>>
>>>> *Option 3*
>>>> Move the max dimension limit lower level to a HNSW specific
>>>> implementation. Once there, this limit would not bind any other potential
>>>> vector engine alternative/evolution.
>>>> *Motivation:* There seem to be contradictory performance
>>>> interpretations about the current HNSW implementation. Some consider its
>>>> performance ok, some not, and it depends on the target data set and use
>>>> case. Increasing the max dimension limit where it is currently (in top
>>>> level FloatVectorValues) would not allow potential alternatives (e.g. for
>>>> other use-cases) to be based on a lower limit.
>>>>
>>>> *Option 4*
>>>> Make it configurable and move it to an appropriate place.
>>>> In particular, a simple Integer.getInteger("lucene.hnsw.maxDimensions",
>>>> 1024) should be enough.
>>>> *Motivation*:
>>>> Both are good and not mutually exclusive and could happen in any order.
>>>> Someone suggested to perfect what the _default_ limit should be, but
>>>> I've not seen an argument _against_ configurability. Especially in this
>>>> way -- a toggle that doesn't bind Lucene's APIs in any way.
>>>>
>>>> I'll keep this [VOTE] open for a week and then proceed to the
>>>> implementation.
>>>> --------------------------
>>>> *Alessandro Benedetti*
>>>> Director @ Sease Ltd.
>>>> *Apache Lucene/Solr Committer*
>>>> *Apache Solr PMC Member*
>>>>
>>>> e-mail: a.benedetti@sease.io
>>>>
>>>>
>>>> *Sease* - Information Retrieval Applied
>>>> Consulting | Training | Open Source
>>>>
>>>> Website: Sease.io <http://sease.io/>
>>>> LinkedIn <https://linkedin.com/company/sease-ltd> | Twitter
>>>> <https://twitter.com/seaseltd> | Youtube
>>>> <https://www.youtube.com/channel/UCDx86ZKLYNpI3gzMercM7BQ> | Github
>>>> <https://github.com/seaseltd>
>>>>
>>>
>>
>> --
>> http://www.needhamsoftware.com (work)
>> http://www.the111shift.com (play)
>>
>

Re: [VOTE] Dimension Limit for KNN Vectors [ In reply to ]

dsmiley at apache

May 16, 2023, 7:49 PM

Post #23 of 46 (855 views)

Permalink

Robert, I have not heard from you (or anyone) an argument against System
property based configurability (as I described in Option 4 via a System
property). Uwe notes wisely some care must be taken to ensure it actually
works. Sure, of course. What concerns do you have with this?

~ David Smiley
Apache Lucene/Solr Search Developer
http://www.linkedin.com/in/davidwsmiley

On Tue, May 16, 2023 at 9:50?PM Robert Muir <rcmuir@gmail.com> wrote:

> by the way, i agree with the idea to MOVE THE LIMIT UNCHANGED to the
> hsnw-specific code.
>
> This way, someone can write alternative codec with vectors using some
> other completely different approach that incorporates a different more
> appropriate limit (maybe lower, maybe higher) depending upon their
> tradeoffs. We should encourage this as I think it is the "only true fix" to
> the scalability issues: use a scalable algorithm! Also, alternative codecs
> don't force the project into many years of index backwards compatibility,
> which is really my penultimate concern. We can lock ourselves into a truly
> bad place and become irrelevant (especially with scalar code implementing
> all this vector stuff, it is really senseless). In the meantime I suggest
> we try to reduce pain for the default codec with the current implementation
> if possible. If it is not possible, we need a new codec that performs.
>
> On Tue, May 16, 2023 at 8:53?PM Robert Muir <rcmuir@gmail.com> wrote:
>
>> Gus, I think i explained myself multiple times on issues and in this
>> thread. the performance is unacceptable, everyone knows it, but nobody is
>> talking about.
>> I don't need to explain myself time and time again here.
>> You don't seem to understand the technical issues (at least you sure as
>> fuck don't know how service loading works or you wouldnt have opened
>> https://github.com/apache/lucene/issues/12300 ????)
>>
>> I'm just the only one here completely unconstrained by any of silicon
>> valley's influences to speak my true mind, without any repercussions, so I
>> do it. Don't give any fucks about ChatGPT.
>>
>> I'm standing by my technical veto. If you bypass it, I'll revert the
>> offending commit.
>>
>> As far as fixing the technical performance, I just opened an issue with
>> some ideas to at least improve cpu usage by a factor of N. It does not help
>> with the crazy heap memory usage or other issues of KNN implementation
>> causing shit like OOM on merge. But it is one step:
>> https://github.com/apache/lucene/issues/12302
>>
>>
>>
>> On Tue, May 16, 2023 at 7:45?AM Gus Heck <gus.heck@gmail.com> wrote:
>>
>>> Robert,
>>>
>>> Can you explain in clear technical terms the standard that must be met
>>> for performance? A benchmark that must run in X time on Y hardware for
>>> example (and why that test is suitable)? Or some other reproducible
>>> criteria? So far I've heard you give an *opinion* that it's unusable, but
>>> that's not a technical criteria, others may have a different concept of
>>> what is usable to them.
>>>
>>> Forgive me if I misunderstand, but the essence of your argument has
>>> seemed to be
>>>
>>> "Performance isn't good enough, therefore we should force anyone who
>>> wants to experiment with something bigger to fork the code base to do it"
>>>
>>> Thus, it is necessary to have a clear unambiguous standard that anyone
>>> can verify for "good enough". A clear standard would also focus efforts at
>>> improvement.
>>>
>>> Where are the goal posts?
>>>
>>> FWIW I'm +1 on any of 2-4 since I believe the existence of a hard limit
>>> is fundamentally counterproductive in an open source setting, as it will
>>> lead to *fewer people* pushing the limits. Extremely few people are
>>> going to get into the nitty-gritty of optimizing things unless they are
>>> staring at code that they can prove does something interesting, but doesn't
>>> run fast enough for their purposes. If people hit a hard limit, more of
>>> them give up and never develop the code that will motivate them to look for
>>> optimizations.
>>>
>>> -Gus
>>>
>>> On Tue, May 16, 2023 at 6:04?AM Robert Muir <rcmuir@gmail.com> wrote:
>>>
>>>> i still feel -1 (veto) on increasing this limit. sending more emails
>>>> does not change the technical facts or make the veto go away.
>>>>
>>>> On Tue, May 16, 2023 at 4:50?AM Alessandro Benedetti <
>>>> a.benedetti@sease.io> wrote:
>>>>
>>>>> Hi all,
>>>>> we have finalized all the options proposed by the community and we are
>>>>> ready to vote for the preferred one and then proceed with the
>>>>> implementation.
>>>>>
>>>>> *Option 1*
>>>>> Keep it as it is (dimension limit hardcoded to 1024)
>>>>> *Motivation*:
>>>>> We are close to improving on many fronts. Given the criticality of
>>>>> Lucene in computing infrastructure and the concerns raised by one of the
>>>>> most active stewards of the project, I think we should keep working toward
>>>>> improving the feature as is and move to up the limit after we can
>>>>> demonstrate improvement unambiguously.
>>>>>
>>>>> *Option 2*
>>>>> make the limit configurable, for example through a system property
>>>>> *Motivation*:
>>>>> The system administrator can enforce a limit its users need to respect
>>>>> that it's in line with whatever the admin decided to be acceptable for
>>>>> them.
>>>>> The default can stay the current one.
>>>>> This should open the doors for Apache Solr, Elasticsearch, OpenSearch,
>>>>> and any sort of plugin development
>>>>>
>>>>> *Option 3*
>>>>> Move the max dimension limit lower level to a HNSW specific
>>>>> implementation. Once there, this limit would not bind any other potential
>>>>> vector engine alternative/evolution.
>>>>> *Motivation:* There seem to be contradictory performance
>>>>> interpretations about the current HNSW implementation. Some consider its
>>>>> performance ok, some not, and it depends on the target data set and use
>>>>> case. Increasing the max dimension limit where it is currently (in top
>>>>> level FloatVectorValues) would not allow potential alternatives (e.g. for
>>>>> other use-cases) to be based on a lower limit.
>>>>>
>>>>> *Option 4*
>>>>> Make it configurable and move it to an appropriate place.
>>>>> In particular, a
>>>>> simple Integer.getInteger("lucene.hnsw.maxDimensions", 1024) should be
>>>>> enough.
>>>>> *Motivation*:
>>>>> Both are good and not mutually exclusive and could happen in any order.
>>>>> Someone suggested to perfect what the _default_ limit should be, but
>>>>> I've not seen an argument _against_ configurability. Especially in this
>>>>> way -- a toggle that doesn't bind Lucene's APIs in any way.
>>>>>
>>>>> I'll keep this [VOTE] open for a week and then proceed to the
>>>>> implementation.
>>>>> --------------------------
>>>>> *Alessandro Benedetti*
>>>>> Director @ Sease Ltd.
>>>>> *Apache Lucene/Solr Committer*
>>>>> *Apache Solr PMC Member*
>>>>>
>>>>> e-mail: a.benedetti@sease.io
>>>>>
>>>>>
>>>>> *Sease* - Information Retrieval Applied
>>>>> Consulting | Training | Open Source
>>>>>
>>>>> Website: Sease.io <http://sease.io/>
>>>>> LinkedIn <https://linkedin.com/company/sease-ltd> | Twitter
>>>>> <https://twitter.com/seaseltd> | Youtube
>>>>> <https://www.youtube.com/channel/UCDx86ZKLYNpI3gzMercM7BQ> | Github
>>>>> <https://github.com/seaseltd>
>>>>>
>>>>
>>>
>>> --
>>> http://www.needhamsoftware.com (work)
>>> http://www.the111shift.com (play)
>>>
>>

Re: [VOTE] Dimension Limit for KNN Vectors [ In reply to ]

rcmuir at gmail

May 16, 2023, 7:58 PM

Post #24 of 46 (855 views)

Permalink

My problem is that it impacts the default codec which is supported by our
backwards compatibility policy for many years. We can't just let the user
determine backwards compatibility with a sysprop. how will checkindex work?
We have to have bounds and also allow for more performant implementations
that might have different limitations. And I'm pretty sure we want a faster
implementation than what we have in the future, and it will probably have
different limits.

For other codecs, it is fine to have a different limit as I already said,
as it is implementation dependent. And honestly the stuff in lucene/codecs
can be more "Fast and loose" because it doesn't require the extensive index
back compat guarantee.

Again, penultimate concern is that index back compat guarantee. When it
comes to limits, the proper way is not to just keep bumping them without
technical reasons, instead the correct approach is to fix the technical
problems and make them irrelevant. Great example here (merged this
morning):
https://github.com/apache/lucene/commit/f53eb28af053d7612f7e4d1b2de05d33dc410645

On Tue, May 16, 2023 at 10:49?PM David Smiley <dsmiley@apache.org> wrote:

> Robert, I have not heard from you (or anyone) an argument against System
> property based configurability (as I described in Option 4 via a System
> property). Uwe notes wisely some care must be taken to ensure it actually
> works. Sure, of course. What concerns do you have with this?
>
> ~ David Smiley
> Apache Lucene/Solr Search Developer
> http://www.linkedin.com/in/davidwsmiley
>
>
> On Tue, May 16, 2023 at 9:50?PM Robert Muir <rcmuir@gmail.com> wrote:
>
>> by the way, i agree with the idea to MOVE THE LIMIT UNCHANGED to the
>> hsnw-specific code.
>>
>> This way, someone can write alternative codec with vectors using some
>> other completely different approach that incorporates a different more
>> appropriate limit (maybe lower, maybe higher) depending upon their
>> tradeoffs. We should encourage this as I think it is the "only true fix" to
>> the scalability issues: use a scalable algorithm! Also, alternative codecs
>> don't force the project into many years of index backwards compatibility,
>> which is really my penultimate concern. We can lock ourselves into a truly
>> bad place and become irrelevant (especially with scalar code implementing
>> all this vector stuff, it is really senseless). In the meantime I suggest
>> we try to reduce pain for the default codec with the current implementation
>> if possible. If it is not possible, we need a new codec that performs.
>>
>> On Tue, May 16, 2023 at 8:53?PM Robert Muir <rcmuir@gmail.com> wrote:
>>
>>> Gus, I think i explained myself multiple times on issues and in this
>>> thread. the performance is unacceptable, everyone knows it, but nobody is
>>> talking about.
>>> I don't need to explain myself time and time again here.
>>> You don't seem to understand the technical issues (at least you sure as
>>> fuck don't know how service loading works or you wouldnt have opened
>>> https://github.com/apache/lucene/issues/12300 ????)
>>>
>>> I'm just the only one here completely unconstrained by any of silicon
>>> valley's influences to speak my true mind, without any repercussions, so I
>>> do it. Don't give any fucks about ChatGPT.
>>>
>>> I'm standing by my technical veto. If you bypass it, I'll revert the
>>> offending commit.
>>>
>>> As far as fixing the technical performance, I just opened an issue with
>>> some ideas to at least improve cpu usage by a factor of N. It does not help
>>> with the crazy heap memory usage or other issues of KNN implementation
>>> causing shit like OOM on merge. But it is one step:
>>> https://github.com/apache/lucene/issues/12302
>>>
>>>
>>>
>>> On Tue, May 16, 2023 at 7:45?AM Gus Heck <gus.heck@gmail.com> wrote:
>>>
>>>> Robert,
>>>>
>>>> Can you explain in clear technical terms the standard that must be met
>>>> for performance? A benchmark that must run in X time on Y hardware for
>>>> example (and why that test is suitable)? Or some other reproducible
>>>> criteria? So far I've heard you give an *opinion* that it's unusable, but
>>>> that's not a technical criteria, others may have a different concept of
>>>> what is usable to them.
>>>>
>>>> Forgive me if I misunderstand, but the essence of your argument has
>>>> seemed to be
>>>>
>>>> "Performance isn't good enough, therefore we should force anyone who
>>>> wants to experiment with something bigger to fork the code base to do it"
>>>>
>>>> Thus, it is necessary to have a clear unambiguous standard that anyone
>>>> can verify for "good enough". A clear standard would also focus efforts at
>>>> improvement.
>>>>
>>>> Where are the goal posts?
>>>>
>>>> FWIW I'm +1 on any of 2-4 since I believe the existence of a hard limit
>>>> is fundamentally counterproductive in an open source setting, as it will
>>>> lead to *fewer people* pushing the limits. Extremely few people are
>>>> going to get into the nitty-gritty of optimizing things unless they are
>>>> staring at code that they can prove does something interesting, but doesn't
>>>> run fast enough for their purposes. If people hit a hard limit, more of
>>>> them give up and never develop the code that will motivate them to look for
>>>> optimizations.
>>>>
>>>> -Gus
>>>>
>>>> On Tue, May 16, 2023 at 6:04?AM Robert Muir <rcmuir@gmail.com> wrote:
>>>>
>>>>> i still feel -1 (veto) on increasing this limit. sending more emails
>>>>> does not change the technical facts or make the veto go away.
>>>>>
>>>>> On Tue, May 16, 2023 at 4:50?AM Alessandro Benedetti <
>>>>> a.benedetti@sease.io> wrote:
>>>>>
>>>>>> Hi all,
>>>>>> we have finalized all the options proposed by the community and we
>>>>>> are ready to vote for the preferred one and then proceed with the
>>>>>> implementation.
>>>>>>
>>>>>> *Option 1*
>>>>>> Keep it as it is (dimension limit hardcoded to 1024)
>>>>>> *Motivation*:
>>>>>> We are close to improving on many fronts. Given the criticality of
>>>>>> Lucene in computing infrastructure and the concerns raised by one of the
>>>>>> most active stewards of the project, I think we should keep working toward
>>>>>> improving the feature as is and move to up the limit after we can
>>>>>> demonstrate improvement unambiguously.
>>>>>>
>>>>>> *Option 2*
>>>>>> make the limit configurable, for example through a system property
>>>>>> *Motivation*:
>>>>>> The system administrator can enforce a limit its users need to
>>>>>> respect that it's in line with whatever the admin decided to be acceptable
>>>>>> for them.
>>>>>> The default can stay the current one.
>>>>>> This should open the doors for Apache Solr, Elasticsearch,
>>>>>> OpenSearch, and any sort of plugin development
>>>>>>
>>>>>> *Option 3*
>>>>>> Move the max dimension limit lower level to a HNSW specific
>>>>>> implementation. Once there, this limit would not bind any other potential
>>>>>> vector engine alternative/evolution.
>>>>>> *Motivation:* There seem to be contradictory performance
>>>>>> interpretations about the current HNSW implementation. Some consider its
>>>>>> performance ok, some not, and it depends on the target data set and use
>>>>>> case. Increasing the max dimension limit where it is currently (in top
>>>>>> level FloatVectorValues) would not allow potential alternatives (e.g. for
>>>>>> other use-cases) to be based on a lower limit.
>>>>>>
>>>>>> *Option 4*
>>>>>> Make it configurable and move it to an appropriate place.
>>>>>> In particular, a
>>>>>> simple Integer.getInteger("lucene.hnsw.maxDimensions", 1024) should be
>>>>>> enough.
>>>>>> *Motivation*:
>>>>>> Both are good and not mutually exclusive and could happen in any
>>>>>> order.
>>>>>> Someone suggested to perfect what the _default_ limit should be, but
>>>>>> I've not seen an argument _against_ configurability. Especially in this
>>>>>> way -- a toggle that doesn't bind Lucene's APIs in any way.
>>>>>>
>>>>>> I'll keep this [VOTE] open for a week and then proceed to the
>>>>>> implementation.
>>>>>> --------------------------
>>>>>> *Alessandro Benedetti*
>>>>>> Director @ Sease Ltd.
>>>>>> *Apache Lucene/Solr Committer*
>>>>>> *Apache Solr PMC Member*
>>>>>>
>>>>>> e-mail: a.benedetti@sease.io
>>>>>>
>>>>>>
>>>>>> *Sease* - Information Retrieval Applied
>>>>>> Consulting | Training | Open Source
>>>>>>
>>>>>> Website: Sease.io <http://sease.io/>
>>>>>> LinkedIn <https://linkedin.com/company/sease-ltd> | Twitter
>>>>>> <https://twitter.com/seaseltd> | Youtube
>>>>>> <https://www.youtube.com/channel/UCDx86ZKLYNpI3gzMercM7BQ> | Github
>>>>>> <https://github.com/seaseltd>
>>>>>>
>>>>>
>>>>
>>>> --
>>>> http://www.needhamsoftware.com (work)
>>>> http://www.the111shift.com (play)
>>>>
>>>

Re: [VOTE] Dimension Limit for KNN Vectors [ In reply to ]

gus.heck at gmail

May 16, 2023, 7:59 PM

Post #25 of 46 (855 views)

Permalink

Hi Robert,

If you read the issue I opened more carefully you'll see I had all the
service loading stuff sorted just fine. It's the silent eating of the
security exceptions by URLClassPath that I think is a useful thing to point
out. If anything, that ticket is more about being surprised by Security
manager behavior than service loading. I thought it would be good if anyone
else who doesn't know that bit of (IMHO obscure) trivia didn't have to
spend a long time hunting down the same thing I did if they encounter a
misconfigured security policy. If you think it could be worded better I'm
all ears.

Also, I didn't question a single thing you said or ask you to repeat
anything. I asked for a very specific detail you have not yet provided, and
that's what's your goal post. When is it good enough?

I do disagree with you at a "how open source works best" level, favoring
enablement. I don't think I've disagreed with a single one of your
technical claims or reported experiences.

Awesome if you've found an improvement. :) If it works as well as you
expect, is it enough to change your mind?

-Gus

On Tue, May 16, 2023 at 8:54?PM Robert Muir <rcmuir@gmail.com> wrote:

> Gus, I think i explained myself multiple times on issues and in this
> thread. the performance is unacceptable, everyone knows it, but nobody is
> talking about.
> I don't need to explain myself time and time again here.
> You don't seem to understand the technical issues (at least you sure as
> fuck don't know how service loading works or you wouldnt have opened
> https://github.com/apache/lucene/issues/12300 ????)
>
> I'm just the only one here completely unconstrained by any of silicon
> valley's influences to speak my true mind, without any repercussions, so I
> do it. Don't give any fucks about ChatGPT.
>
> I'm standing by my technical veto. If you bypass it, I'll revert the
> offending commit.
>
> As far as fixing the technical performance, I just opened an issue with
> some ideas to at least improve cpu usage by a factor of N. It does not help
> with the crazy heap memory usage or other issues of KNN implementation
> causing shit like OOM on merge. But it is one step:
> https://github.com/apache/lucene/issues/12302
>
>
>
> On Tue, May 16, 2023 at 7:45?AM Gus Heck <gus.heck@gmail.com> wrote:
>
>> Robert,
>>
>> Can you explain in clear technical terms the standard that must be met
>> for performance? A benchmark that must run in X time on Y hardware for
>> example (and why that test is suitable)? Or some other reproducible
>> criteria? So far I've heard you give an *opinion* that it's unusable, but
>> that's not a technical criteria, others may have a different concept of
>> what is usable to them.
>>
>> Forgive me if I misunderstand, but the essence of your argument has
>> seemed to be
>>
>> "Performance isn't good enough, therefore we should force anyone who
>> wants to experiment with something bigger to fork the code base to do it"
>>
>> Thus, it is necessary to have a clear unambiguous standard that anyone
>> can verify for "good enough". A clear standard would also focus efforts at
>> improvement.
>>
>> Where are the goal posts?
>>
>> FWIW I'm +1 on any of 2-4 since I believe the existence of a hard limit
>> is fundamentally counterproductive in an open source setting, as it will
>> lead to *fewer people* pushing the limits. Extremely few people are
>> going to get into the nitty-gritty of optimizing things unless they are
>> staring at code that they can prove does something interesting, but doesn't
>> run fast enough for their purposes. If people hit a hard limit, more of
>> them give up and never develop the code that will motivate them to look for
>> optimizations.
>>
>> -Gus
>>
>> On Tue, May 16, 2023 at 6:04?AM Robert Muir <rcmuir@gmail.com> wrote:
>>
>>> i still feel -1 (veto) on increasing this limit. sending more emails
>>> does not change the technical facts or make the veto go away.
>>>
>>> On Tue, May 16, 2023 at 4:50?AM Alessandro Benedetti <
>>> a.benedetti@sease.io> wrote:
>>>
>>>> Hi all,
>>>> we have finalized all the options proposed by the community and we are
>>>> ready to vote for the preferred one and then proceed with the
>>>> implementation.
>>>>
>>>> *Option 1*
>>>> Keep it as it is (dimension limit hardcoded to 1024)
>>>> *Motivation*:
>>>> We are close to improving on many fronts. Given the criticality of
>>>> Lucene in computing infrastructure and the concerns raised by one of the
>>>> most active stewards of the project, I think we should keep working toward
>>>> improving the feature as is and move to up the limit after we can
>>>> demonstrate improvement unambiguously.
>>>>
>>>> *Option 2*
>>>> make the limit configurable, for example through a system property
>>>> *Motivation*:
>>>> The system administrator can enforce a limit its users need to respect
>>>> that it's in line with whatever the admin decided to be acceptable for
>>>> them.
>>>> The default can stay the current one.
>>>> This should open the doors for Apache Solr, Elasticsearch, OpenSearch,
>>>> and any sort of plugin development
>>>>
>>>> *Option 3*
>>>> Move the max dimension limit lower level to a HNSW specific
>>>> implementation. Once there, this limit would not bind any other potential
>>>> vector engine alternative/evolution.
>>>> *Motivation:* There seem to be contradictory performance
>>>> interpretations about the current HNSW implementation. Some consider its
>>>> performance ok, some not, and it depends on the target data set and use
>>>> case. Increasing the max dimension limit where it is currently (in top
>>>> level FloatVectorValues) would not allow potential alternatives (e.g. for
>>>> other use-cases) to be based on a lower limit.
>>>>
>>>> *Option 4*
>>>> Make it configurable and move it to an appropriate place.
>>>> In particular, a simple Integer.getInteger("lucene.hnsw.maxDimensions",
>>>> 1024) should be enough.
>>>> *Motivation*:
>>>> Both are good and not mutually exclusive and could happen in any order.
>>>> Someone suggested to perfect what the _default_ limit should be, but
>>>> I've not seen an argument _against_ configurability. Especially in this
>>>> way -- a toggle that doesn't bind Lucene's APIs in any way.
>>>>
>>>> I'll keep this [VOTE] open for a week and then proceed to the
>>>> implementation.
>>>> --------------------------
>>>> *Alessandro Benedetti*
>>>> Director @ Sease Ltd.
>>>> *Apache Lucene/Solr Committer*
>>>> *Apache Solr PMC Member*
>>>>
>>>> e-mail: a.benedetti@sease.io
>>>>
>>>>
>>>> *Sease* - Information Retrieval Applied
>>>> Consulting | Training | Open Source
>>>>
>>>> Website: Sease.io <http://sease.io/>
>>>> LinkedIn <https://linkedin.com/company/sease-ltd> | Twitter
>>>> <https://twitter.com/seaseltd> | Youtube
>>>> <https://www.youtube.com/channel/UCDx86ZKLYNpI3gzMercM7BQ> | Github
>>>> <https://github.com/seaseltd>
>>>>
>>>
>>
>> --
>> http://www.needhamsoftware.com (work)
>> http://www.the111shift.com (play)
>>
>

--
http://www.needhamsoftware.com (work)
http://www.the111shift.com (play)