Mailing List Archive

TLDR: use the lucene suggest/ package. Start with building suggester
from your query logs (either a file or index them).
These have a lot of flexibility about how the matches happen, for
example pure prefixes, edit distance typos, infix matching, analysis
chain, even now Japanese input-method integration :)

Run that suggester on the user input, retrieving say, the top 5-10
matches of relevant query suggestions.
return those in the UI (typical autosuggest-type field), but also run
a search on the first one.

The user gets the instant-search experience, but when they type 'tes',
you search on 'tesla' (if that's the top-suggested query, the
highlighted one in the autocomplete). if they arrow-down to another
suggestion such as 'test' or type a 't' or use the mouse or whatever,
then the process runs again and they see the results for that.

IMO for most cases this leads to a saner experience than trying to
rank all documents based on a prefix 'tes': the problem is there is
still too much query ambiguity, not really any "keywords" yet, so
trying to rank those documents won't be very useful. Instead you try
to "interact" with the user to present results in a useful way that
they can navigate.

On the other hand if you really want to just search on prefixes and
jumble up the results (perhaps because you are gonna just sort by some
custom document feature instead of relevance), then you can do that if
you really want. You can use the n-gram/edge-ngram/shingle filters in
the analysis package for that.

On Wed, Oct 6, 2021 at 5:37 PM Michael Wechner
<michael.wechner@wyona.com> wrote:
>
> Hi
>
> I am trying to implement a search with Lucene similar to what for
> example various "Note Apps" (e.g. "Google Keep" or "Samsung Notes") are
> offering, that with every new letter typed a new search is being executed.
>
> For example when I type "tes", then all documents are being returned
> containing the word "test" or "tesla" and when I continue typing, for
> example "tesö" and there are no documents containing the string "tesö",
> then the app will tell me that there are no matches.
>
> I have found a couple of articles related to this kind of search, for
> example
>
> https://stackoverflow.com/questions/10828825/incremental-search-using-lucene
>
> https://stackoverflow.com/questions/120180/how-to-do-query-auto-completion-suggestions-in-lucene
>
> but would be great to know whether there exist other possibilities or
> what the best practice is?
>
> I am even not sure what the right term for this kind of search is, is it
> really "incremental search" or something else?
>
> Looking forward to your feedback and will be happy to extend the Lucene
> FAQ once I understand better :-)
>
> Thanks
>
> Michael
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org

Thanks very much for your feedback!

I will try it :-)

As I wrote I would like to add a summary to the Lucene FAQ
(https://cwiki.apache.org/confluence/display/lucene/lucenefaq)

Would the following questions make sense?

     - "Does Lucene support incremental search?"

     - "Does Lucene support auto completion suggestions?"

Or would other other terms / or another wording make more sense?

Thanks

Michael



Am 07.10.21 um 01:14 schrieb Robert Muir:
> TLDR: use the lucene suggest/ package. Start with building suggester
> from your query logs (either a file or index them).
> These have a lot of flexibility about how the matches happen, for
> example pure prefixes, edit distance typos, infix matching, analysis
> chain, even now Japanese input-method integration :)
>
> Run that suggester on the user input, retrieving say, the top 5-10
> matches of relevant query suggestions.
> return those in the UI (typical autosuggest-type field), but also run
> a search on the first one.
>
> The user gets the instant-search experience, but when they type 'tes',
> you search on 'tesla' (if that's the top-suggested query, the
> highlighted one in the autocomplete). if they arrow-down to another
> suggestion such as 'test' or type a 't' or use the mouse or whatever,
> then the process runs again and they see the results for that.
>
> IMO for most cases this leads to a saner experience than trying to
> rank all documents based on a prefix 'tes': the problem is there is
> still too much query ambiguity, not really any "keywords" yet, so
> trying to rank those documents won't be very useful. Instead you try
> to "interact" with the user to present results in a useful way that
> they can navigate.
>
> On the other hand if you really want to just search on prefixes and
> jumble up the results (perhaps because you are gonna just sort by some
> custom document feature instead of relevance), then you can do that if
> you really want. You can use the n-gram/edge-ngram/shingle filters in
> the analysis package for that.
>
> On Wed, Oct 6, 2021 at 5:37 PM Michael Wechner
> <michael.wechner@wyona.com> wrote:
>> Hi
>>
>> I am trying to implement a search with Lucene similar to what for
>> example various "Note Apps" (e.g. "Google Keep" or "Samsung Notes") are
>> offering, that with every new letter typed a new search is being executed.
>>
>> For example when I type "tes", then all documents are being returned
>> containing the word "test" or "tesla" and when I continue typing, for
>> example "tesö" and there are no documents containing the string "tesö",
>> then the app will tell me that there are no matches.
>>
>> I have found a couple of articles related to this kind of search, for
>> example
>>
>> https://stackoverflow.com/questions/10828825/incremental-search-using-lucene
>>
>> https://stackoverflow.com/questions/120180/how-to-do-query-auto-completion-suggestions-in-lucene
>>
>> but would be great to know whether there exist other possibilities or
>> what the best practice is?
>>
>> I am even not sure what the right term for this kind of search is, is it
>> really "incremental search" or something else?
>>
>> Looking forward to your feedback and will be happy to extend the Lucene
>> FAQ once I understand better :-)
>>
>> Thanks
>>
>> Michael
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>


---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org

Thank you for offering to add to the FAQ! Indeed it should mention the
suggester capability. I think you have permissions to edit that wiki?
Please go ahead and I think add a link to the suggest module javadocs

On Thu, Oct 7, 2021 at 2:30 AM Michael Wechner
<michael.wechner@wyona.com> wrote:
>
> Thanks very much for your feedback!
>
> I will try it :-)
>
> As I wrote I would like to add a summary to the Lucene FAQ
> (https://cwiki.apache.org/confluence/display/lucene/lucenefaq)
>
> Would the following questions make sense?
>
> - "Does Lucene support incremental search?"
>
> - "Does Lucene support auto completion suggestions?"
>
> Or would other other terms / or another wording make more sense?
>
> Thanks
>
> Michael
>
>
>
> Am 07.10.21 um 01:14 schrieb Robert Muir:
> > TLDR: use the lucene suggest/ package. Start with building suggester
> > from your query logs (either a file or index them).
> > These have a lot of flexibility about how the matches happen, for
> > example pure prefixes, edit distance typos, infix matching, analysis
> > chain, even now Japanese input-method integration :)
> >
> > Run that suggester on the user input, retrieving say, the top 5-10
> > matches of relevant query suggestions.
> > return those in the UI (typical autosuggest-type field), but also run
> > a search on the first one.
> >
> > The user gets the instant-search experience, but when they type 'tes',
> > you search on 'tesla' (if that's the top-suggested query, the
> > highlighted one in the autocomplete). if they arrow-down to another
> > suggestion such as 'test' or type a 't' or use the mouse or whatever,
> > then the process runs again and they see the results for that.
> >
> > IMO for most cases this leads to a saner experience than trying to
> > rank all documents based on a prefix 'tes': the problem is there is
> > still too much query ambiguity, not really any "keywords" yet, so
> > trying to rank those documents won't be very useful. Instead you try
> > to "interact" with the user to present results in a useful way that
> > they can navigate.
> >
> > On the other hand if you really want to just search on prefixes and
> > jumble up the results (perhaps because you are gonna just sort by some
> > custom document feature instead of relevance), then you can do that if
> > you really want. You can use the n-gram/edge-ngram/shingle filters in
> > the analysis package for that.
> >
> > On Wed, Oct 6, 2021 at 5:37 PM Michael Wechner
> > <michael.wechner@wyona.com> wrote:
> >> Hi
> >>
> >> I am trying to implement a search with Lucene similar to what for
> >> example various "Note Apps" (e.g. "Google Keep" or "Samsung Notes") are
> >> offering, that with every new letter typed a new search is being executed.
> >>
> >> For example when I type "tes", then all documents are being returned
> >> containing the word "test" or "tesla" and when I continue typing, for
> >> example "tesö" and there are no documents containing the string "tesö",
> >> then the app will tell me that there are no matches.
> >>
> >> I have found a couple of articles related to this kind of search, for
> >> example
> >>
> >> https://stackoverflow.com/questions/10828825/incremental-search-using-lucene
> >>
> >> https://stackoverflow.com/questions/120180/how-to-do-query-auto-completion-suggestions-in-lucene
> >>
> >> but would be great to know whether there exist other possibilities or
> >> what the best practice is?
> >>
> >> I am even not sure what the right term for this kind of search is, is it
> >> really "incremental search" or something else?
> >>
> >> Looking forward to your feedback and will be happy to extend the Lucene
> >> FAQ once I understand better :-)
> >>
> >> Thanks
> >>
> >> Michael
> >>
> >> ---------------------------------------------------------------------
> >> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> >> For additional commands, e-mail: java-user-help@lucene.apache.org
> >>
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> > For additional commands, e-mail: java-user-help@lucene.apache.org
> >
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org

Am 08.10.21 um 18:49 schrieb Michael Sokolov:
> Thank you for offering to add to the FAQ! Indeed it should mention the
> suggester capability. I think you have permissions to edit that wiki?

yes :-)

> Please go ahead and I think add a link to the suggest module javadocs

ok, will do!

Thanks

Michael

>
> On Thu, Oct 7, 2021 at 2:30 AM Michael Wechner
> <michael.wechner@wyona.com> wrote:
>> Thanks very much for your feedback!
>>
>> I will try it :-)
>>
>> As I wrote I would like to add a summary to the Lucene FAQ
>> (https://cwiki.apache.org/confluence/display/lucene/lucenefaq)
>>
>> Would the following questions make sense?
>>
>> - "Does Lucene support incremental search?"
>>
>> - "Does Lucene support auto completion suggestions?"
>>
>> Or would other other terms / or another wording make more sense?
>>
>> Thanks
>>
>> Michael
>>
>>
>>
>> Am 07.10.21 um 01:14 schrieb Robert Muir:
>>> TLDR: use the lucene suggest/ package. Start with building suggester
>>> from your query logs (either a file or index them).
>>> These have a lot of flexibility about how the matches happen, for
>>> example pure prefixes, edit distance typos, infix matching, analysis
>>> chain, even now Japanese input-method integration :)
>>>
>>> Run that suggester on the user input, retrieving say, the top 5-10
>>> matches of relevant query suggestions.
>>> return those in the UI (typical autosuggest-type field), but also run
>>> a search on the first one.
>>>
>>> The user gets the instant-search experience, but when they type 'tes',
>>> you search on 'tesla' (if that's the top-suggested query, the
>>> highlighted one in the autocomplete). if they arrow-down to another
>>> suggestion such as 'test' or type a 't' or use the mouse or whatever,
>>> then the process runs again and they see the results for that.
>>>
>>> IMO for most cases this leads to a saner experience than trying to
>>> rank all documents based on a prefix 'tes': the problem is there is
>>> still too much query ambiguity, not really any "keywords" yet, so
>>> trying to rank those documents won't be very useful. Instead you try
>>> to "interact" with the user to present results in a useful way that
>>> they can navigate.
>>>
>>> On the other hand if you really want to just search on prefixes and
>>> jumble up the results (perhaps because you are gonna just sort by some
>>> custom document feature instead of relevance), then you can do that if
>>> you really want. You can use the n-gram/edge-ngram/shingle filters in
>>> the analysis package for that.
>>>
>>> On Wed, Oct 6, 2021 at 5:37 PM Michael Wechner
>>> <michael.wechner@wyona.com> wrote:
>>>> Hi
>>>>
>>>> I am trying to implement a search with Lucene similar to what for
>>>> example various "Note Apps" (e.g. "Google Keep" or "Samsung Notes") are
>>>> offering, that with every new letter typed a new search is being executed.
>>>>
>>>> For example when I type "tes", then all documents are being returned
>>>> containing the word "test" or "tesla" and when I continue typing, for
>>>> example "tesö" and there are no documents containing the string "tesö",
>>>> then the app will tell me that there are no matches.
>>>>
>>>> I have found a couple of articles related to this kind of search, for
>>>> example
>>>>
>>>> https://stackoverflow.com/questions/10828825/incremental-search-using-lucene
>>>>
>>>> https://stackoverflow.com/questions/120180/how-to-do-query-auto-completion-suggestions-in-lucene
>>>>
>>>> but would be great to know whether there exist other possibilities or
>>>> what the best practice is?
>>>>
>>>> I am even not sure what the right term for this kind of search is, is it
>>>> really "incremental search" or something else?
>>>>
>>>> Looking forward to your feedback and will be happy to extend the Lucene
>>>> FAQ once I understand better :-)
>>>>
>>>> Thanks
>>>>
>>>> Michael
>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>>>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>


---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org

I have added a QnA

https://cwiki.apache.org/confluence/display/LUCENE/LuceneFAQ#LuceneFAQ-DoesLucenesupportauto-suggest/autocomplete?

I will also try to provide an example, for example

https://medium.com/@ekaterinamihailova/in-memory-search-and-autocomplete-with-lucene-8-5-f2df1bc71c36
https://github.com/Da-Bulgaria/e-prescriptions/blob/master/src/main/java/bg/ehealth/prescriptions/services/icd/ICDService.java

or

https://stackoverflow.com/questions/24968697/how-to-implement-auto-suggest-using-lucenes-new-analyzinginfixsuggester-api

but first need to check whether these examples are according to Lucene
8.10.1 suggest API

If you know any simple, recent examples, please let me know

Thanks

Michael


Am 08.10.21 um 21:40 schrieb Michael Wechner:
>
>
> Am 08.10.21 um 18:49 schrieb Michael Sokolov:
>> Thank you for offering to add to the FAQ! Indeed it should mention the
>> suggester capability. I think you have permissions to edit that wiki?
>
> yes :-)
>
>> Please go ahead and I think add a link to the suggest module javadocs
>
> ok, will do!
>
> Thanks
>
> Michael
>
>>
>> On Thu, Oct 7, 2021 at 2:30 AM Michael Wechner
>> <michael.wechner@wyona.com> wrote:
>>> Thanks very much for your feedback!
>>>
>>> I will try it :-)
>>>
>>> As I wrote I would like to add a summary to the Lucene FAQ
>>> (https://cwiki.apache.org/confluence/display/lucene/lucenefaq)
>>>
>>> Would the following questions make sense?
>>>
>>>        - "Does Lucene support incremental search?"
>>>
>>>        - "Does Lucene support auto completion suggestions?"
>>>
>>> Or would other other terms / or another wording make more sense?
>>>
>>> Thanks
>>>
>>> Michael
>>>
>>>
>>>
>>> Am 07.10.21 um 01:14 schrieb Robert Muir:
>>>> TLDR: use the lucene suggest/ package. Start with building suggester
>>>> from your query logs (either a file or index them).
>>>> These have a lot of flexibility about how the matches happen, for
>>>> example pure prefixes, edit distance typos, infix matching, analysis
>>>> chain, even now Japanese input-method integration :)
>>>>
>>>> Run that suggester on the user input, retrieving say, the top 5-10
>>>> matches of relevant query suggestions.
>>>> return those in the UI (typical autosuggest-type field), but also run
>>>> a search on the first one.
>>>>
>>>> The user gets the instant-search experience, but when they type 'tes',
>>>> you search on 'tesla' (if that's the top-suggested query, the
>>>> highlighted one in the autocomplete). if they arrow-down to another
>>>> suggestion such as 'test' or type a 't' or use the mouse or whatever,
>>>> then the process runs again and they see the results for that.
>>>>
>>>> IMO for most cases this leads to a saner experience than trying to
>>>> rank all documents based on a prefix 'tes': the problem is there is
>>>> still too much query ambiguity, not really any "keywords" yet, so
>>>> trying to rank those documents won't be very useful. Instead you try
>>>> to "interact" with the user to present results in a useful way that
>>>> they can navigate.
>>>>
>>>> On the other hand if you really want to just search on prefixes and
>>>> jumble up the results (perhaps because you are gonna just sort by some
>>>> custom document feature instead of relevance), then you can do that if
>>>> you really want. You can use the n-gram/edge-ngram/shingle filters in
>>>> the analysis package for that.
>>>>
>>>> On Wed, Oct 6, 2021 at 5:37 PM Michael Wechner
>>>> <michael.wechner@wyona.com> wrote:
>>>>> Hi
>>>>>
>>>>> I am trying to implement a search with Lucene similar to what for
>>>>> example various "Note Apps" (e.g. "Google Keep" or "Samsung
>>>>> Notes") are
>>>>> offering, that with every new letter typed a new search is being
>>>>> executed.
>>>>>
>>>>> For example when I type "tes", then all documents are being returned
>>>>> containing the word "test" or "tesla" and when I continue typing, for
>>>>> example "tesö" and there are no documents containing the string
>>>>> "tesö",
>>>>> then the app will tell me that there are no matches.
>>>>>
>>>>> I have found a couple of articles related to this kind of search, for
>>>>> example
>>>>>
>>>>> https://stackoverflow.com/questions/10828825/incremental-search-using-lucene
>>>>>
>>>>>
>>>>> https://stackoverflow.com/questions/120180/how-to-do-query-auto-completion-suggestions-in-lucene
>>>>>
>>>>>
>>>>> but would be great to know whether there exist other possibilities or
>>>>> what the best practice is?
>>>>>
>>>>> I am even not sure what the right term for this kind of search is,
>>>>> is it
>>>>> really "incremental search" or something else?
>>>>>
>>>>> Looking forward to your feedback and will be happy to extend the
>>>>> Lucene
>>>>> FAQ once I understand better :-)
>>>>>
>>>>> Thanks
>>>>>
>>>>> Michael
>>>>>
>>>>> ---------------------------------------------------------------------
>>>>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>>>>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>>>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>>>
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>


---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org