Putting ChatGPT aside, what are the implications of (1) removing the limit,
or (2) increasing the limit, or (3) make it configurable at the app's
discretion? The configuration can even be in the form of a VectorEncoder
impl which will decide on the size of the vectors, thereby making it
clearer that this is an expert setting and puts it at the hands of the app
to decide how to handle those large vectors.
Will bigger vectors require an algorithmic change (I understand that it
might benefit from one, I ask if it's required besides performance gains)?
If not, then why do you object to making it an "app problem"? If 2048 dims
vectors require 2GB IW RAM buffer, what's wrong with documenting it and
letting the app choose whether they want/can do it or not?
Do we limit the size of stored fields, or binary doc values? Do we prevent
anyone from using a Codec which loads these big byte[] into memory?
> Also, please let's only disucss SEARCH. lucene is a SEARCH ENGINE
> LIBRARY. not a vector database or whatever trash is being proposed
> here.
The problem being discussed *is* related to search, not lexical search, but
rather semantic search, or search in vector space, however you want to call
it. Are you saying that Lucene should only focus on lexical search
scenarios? Or are you saying that semantic search scenarios don't need to
index >1024 dimension vectors in order to produce high quality results?
I personally don't understand why not letting apps index bigger vectors, if
all that it takes is using bigger RAM buffers. Improvements will come
later, especially as more and more applications will try it. If we prevent
it, then we might never see these improvements cause no one will even
attempt to do it with Lucene, and IMO it's not a direction we want to head.
While ChatGPT itself might be a hype, I don't think that big vectors are,
and if the only technical reason we have for not supporting them is a
bigger RAM buffer, then I think we should allow it.
On Sun, Apr 9, 2023 at 1:59?PM Robert Muir <rcmuir@gmail.com> wrote:
> Also, please let's only disucss SEARCH. lucene is a SEARCH ENGINE
> LIBRARY. not a vector database or whatever trash is being proposed
> here.
>
> i think we should table this and revisit it after chatgpt hype has
> dissipated.
>
> this hype is causing ppl to behave irrationally, it is why i can't
> converse with basically anyone on this thread because they are all
> stating crazy things that don't make sense.
>
> On Sun, Apr 9, 2023 at 6:25?AM Robert Muir <rcmuir@gmail.com> wrote:
> >
> > Yes, its very clear that folks on this thread are ignoring reason
> > entirely and completely swooned by chatgpt-hype.
> > And what happens when they make chatgpt-8 that uses even more dimensions?
> > backwards compatibility decisions can't be made by garbage hype such
> > as cryptocurrency or chatgpt.
> > Trying to convince me we should bump it because of chatgpt, well, i
> > think it has the opposite effect.
> >
> > Please, lemme see real technical arguments why this limit needs to be
> > bumped. not including trash like chatgpt.
> >
> > On Sat, Apr 8, 2023 at 7:50?PM Marcus Eagan <marcuseagan@gmail.com>
> wrote:
> > >
> > > Given the massive amounts of funding going into the development and
> investigation of the project, I think it would be good to at least have
> Lucene be a part of the conversation. Simply because academics typically
> focus on vectors <= 784 dimensions does not mean all users will. A large
> swathe of very important users of the Lucene project never exceed 500k
> documents, though they are shifting to other search engines to try out very
> popular embeddings.
> > >
> > > I think giving our users the opportunity to build chat bots or LLM
> memory machines using Lucene is a positive development, even if some
> datasets won't be able to work well. We don't limit the number of fields
> someone can add in most cases, though we did just undeprecate that API to
> better support multi-tenancy. But people still add so many fields and can
> crash their clusters with mapping explosions when unlimited. The limit to
> vectors feels similar. I expect more people to dig into Lucene due to its
> openness and robustness as they run into problems. Today, they are forced
> to consider other engines that are more permissive.
> > >
> > > Not everyone important or valuable Lucene workload is in the millions
> of documents. Many of them only have lots of queries or computationally
> expensive access patterns for B-trees. We can document that it is very
> ill-advised to make a deployment with vectors too large. What others will
> do with it is on them.
> > >
> > >
> > > On Sat, Apr 8, 2023 at 2:29?PM Adrien Grand <jpountz@gmail.com> wrote:
> > >>
> > >> As Dawid pointed out earlier on this thread, this is the rule for
> > >> Apache projects: a single -1 vote on a code change is a veto and
> > >> cannot be overridden. Furthermore, Robert is one of the people on this
> > >> project who worked the most on debugging subtle bugs, making Lucene
> > >> more robust and improving our test framework, so I'm listening when he
> > >> voices quality concerns.
> > >>
> > >> The argument against removing/raising the limit that resonates with me
> > >> the most is that it is a one-way door. As MikeS highlighted earlier on
> > >> this thread, implementations may want to take advantage of the fact
> > >> that there is a limit at some point too. This is why I don't want to
> > >> remove the limit and would prefer a slight increase, such as 2048 as
> > >> suggested in the original issue, which would enable most of the things
> > >> that users who have been asking about raising the limit would like to
> > >> do.
> > >>
> > >> I agree that the merge-time memory usage and slow indexing rate are
> > >> not great. But it's still possible to index multi-million vector
> > >> datasets with a 4GB heap without hitting OOMEs regardless of the
> > >> number of dimensions, and the feedback I'm seeing is that many users
> > >> are still interested in indexing multi-million vector datasets despite
> > >> the slow indexing rate. I wish we could do better, and vector indexing
> > >> is certainly more expert than text indexing, but it still is usable in
> > >> my opinion. I understand how giving Lucene more information about
> > >> vectors prior to indexing (e.g. clustering information as Jim pointed
> > >> out) could help make merging faster and more memory-efficient, but I
> > >> would really like to avoid making it a requirement for indexing
> > >> vectors as it also makes this feature much harder to use.
> > >>
> > >> On Sat, Apr 8, 2023 at 9:28?PM Alessandro Benedetti
> > >> <a.benedetti@sease.io> wrote:
> > >> >
> > >> > I am very attentive to listen opinions but I am un-convinced here
> and I an not sure that a single person opinion should be allowed to be
> detrimental for such an important project.
> > >> >
> > >> > The limit as far as I know is literally just raising an exception.
> > >> > Removing it won't alter in any way the current performance for
> users in low dimensional space.
> > >> > Removing it will just enable more users to use Lucene.
> > >> >
> > >> > If new users in certain situations will be unhappy with the
> performance, they may contribute improvements.
> > >> > This is how you make progress.
> > >> >
> > >> > If it's a reputation thing, trust me that not allowing users to
> play with high dimensional space will equally damage it.
> > >> >
> > >> > To me it's really a no brainer.
> > >> > Removing the limit and enable people to use high dimensional
> vectors will take minutes.
> > >> > Improving the hnsw implementation can take months.
> > >> > Pick one to begin with...
> > >> >
> > >> > And there's no-one paying me here, no company interest whatsoever,
> actually I pay people to contribute, I am just convinced it's a good idea.
> > >> >
> > >> >
> > >> > On Sat, 8 Apr 2023, 18:57 Robert Muir, <rcmuir@gmail.com> wrote:
> > >> >>
> > >> >> I disagree with your categorization. I put in plenty of work and
> > >> >> experienced plenty of pain myself, writing tests and fighting these
> > >> >> issues, after i saw that, two releases in a row, vector indexing
> fell
> > >> >> over and hit integer overflows etc on small datasets:
> > >> >>
> > >> >> https://github.com/apache/lucene/pull/11905
> > >> >>
> > >> >> Attacking me isn't helping the situation.
> > >> >>
> > >> >> PS: when i said the "one guy who wrote the code" I didn't mean it
> in
> > >> >> any kind of demeaning fashion really. I meant to describe the
> current
> > >> >> state of usability with respect to indexing a few million docs with
> > >> >> high dimensions. You can scroll up the thread and see that at least
> > >> >> one other committer on the project experienced similar pain as me.
> > >> >> Then, think about users who aren't committers trying to use the
> > >> >> functionality!
> > >> >>
> > >> >> On Sat, Apr 8, 2023 at 12:51?PM Michael Sokolov <
> msokolov@gmail.com> wrote:
> > >> >> >
> > >> >> > What you said about increasing dimensions requiring a bigger ram
> buffer on merge is wrong. That's the point I was trying to make. Your
> concerns about merge costs are not wrong, but your conclusion that we need
> to limit dimensions is not justified.
> > >> >> >
> > >> >> > You complain that hnsw sucks it doesn't scale, but when I show
> it scales linearly with dimension you just ignore that and complain about
> something entirely different.
> > >> >> >
> > >> >> > You demand that people run all kinds of tests to prove you wrong
> but when they do, you don't listen and you won't put in the work yourself
> or complain that it's too hard.
> > >> >> >
> > >> >> > Then you complain about people not meeting you half way. Wow
> > >> >> >
> > >> >> > On Sat, Apr 8, 2023, 12:40 PM Robert Muir <rcmuir@gmail.com>
> wrote:
> > >> >> >>
> > >> >> >> On Sat, Apr 8, 2023 at 8:33?AM Michael Wechner
> > >> >> >> <michael.wechner@wyona.com> wrote:
> > >> >> >> >
> > >> >> >> > What exactly do you consider reasonable?
> > >> >> >>
> > >> >> >> Let's begin a real discussion by being HONEST about the current
> > >> >> >> status. Please put politically correct or your own company's
> wishes
> > >> >> >> aside, we know it's not in a good state.
> > >> >> >>
> > >> >> >> Current status is the one guy who wrote the code can set a
> > >> >> >> multi-gigabyte ram buffer and index a small dataset with 1024
> > >> >> >> dimensions in HOURS (i didn't ask what hardware).
> > >> >> >>
> > >> >> >> My concerns are everyone else except the one guy, I want it to
> be
> > >> >> >> usable. Increasing dimensions just means even bigger
> multi-gigabyte
> > >> >> >> ram buffer and bigger heap to avoid OOM on merge.
> > >> >> >> It is also a permanent backwards compatibility decision, we
> have to
> > >> >> >> support it once we do this and we can't just say "oops" and
> flip it
> > >> >> >> back.
> > >> >> >>
> > >> >> >> It is unclear to me, if the multi-gigabyte ram buffer is really
> to
> > >> >> >> avoid merges because they are so slow and it would be DAYS
> otherwise,
> > >> >> >> or if its to avoid merges so it doesn't hit OOM.
> > >> >> >> Also from personal experience, it takes trial and error (means
> > >> >> >> experiencing OOM on merge!!!) before you get those heap values
> correct
> > >> >> >> for your dataset. This usually means starting over which is
> > >> >> >> frustrating and wastes more time.
> > >> >> >>
> > >> >> >> Jim mentioned some ideas about the memory usage in IndexWriter,
> seems
> > >> >> >> to me like its a good idea. maybe the multigigabyte ram buffer
> can be
> > >> >> >> avoided in this way and performance improved by writing bigger
> > >> >> >> segments with lucene's defaults. But this doesn't mean we can
> simply
> > >> >> >> ignore the horrors of what happens on merge. merging needs to
> scale so
> > >> >> >> that indexing really scales.
> > >> >> >>
> > >> >> >> At least it shouldnt spike RAM on trivial data amounts and
> cause OOM,
> > >> >> >> and definitely it shouldnt burn hours and hours of CPU in O(n^2)
> > >> >> >> fashion when indexing.
> > >> >> >>
> > >> >> >>
> ---------------------------------------------------------------------
> > >> >> >> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
> > >> >> >> For additional commands, e-mail: dev-help@lucene.apache.org
> > >> >> >>
> > >> >>
> > >> >>
> ---------------------------------------------------------------------
> > >> >> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
> > >> >> For additional commands, e-mail: dev-help@lucene.apache.org
> > >> >>
> > >>
> > >>
> > >> --
> > >> Adrien
> > >>
> > >> ---------------------------------------------------------------------
> > >> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
> > >> For additional commands, e-mail: dev-help@lucene.apache.org
> > >>
> > >
> > >
> > > --
> > > Marcus Eagan
> > >
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: dev-help@lucene.apache.org
>
>