Mailing List Archive

FWIW a related PR was just merged that allows to introspect query
execution: https://issues.apache.org/jira/browse/LUCENE-9965. It's
different from your use-case though in that it is debugging information for
a single query rather than statistical information across lots of user
queries (and the approach on that other issue makes things much slower so
you wouldn't like to enable it in production).

Out of curiosity, what are you doing with this information about which
execution path is chosen?

On Wed, Jun 9, 2021 at 2:14 PM Egor Moraru <moraru.egor@gmail.com> wrote:

> Hi,
>
> At my current project we wanted to monitor for a specific field the
> fraction of indexed vs doc values queries executed by
> IndexOrDocValuesQuery.
>
> We ended up forking IndexOrDocValuesQuery and passing a listener that
> is notified when the query execution path is decided.
>
> Do you think this is something the community might be interested in?
>
> Kind regards,
> Egor Moraru.
>


--
Adrien

Hi Adrien,

In this specific use case our data is encoded as points and also is stored
as doc values.

We use information about which execution path is chosen to decide
if we can get away with storing this data only once and using one of
the queries.

On Wed, Jun 9, 2021 at 10:39 PM Adrien Grand <jpountz@gmail.com> wrote:

> FWIW a related PR was just merged that allows to introspect query
> execution: https://issues.apache.org/jira/browse/LUCENE-9965. It's
> different from your use-case though in that it is debugging information for
> a single query rather than statistical information across lots of user
> queries (and the approach on that other issue makes things much slower so
> you wouldn't like to enable it in production).
>
> Out of curiosity, what are you doing with this information about which
> execution path is chosen?
>
> On Wed, Jun 9, 2021 at 2:14 PM Egor Moraru <moraru.egor@gmail.com> wrote:
>
> > Hi,
> >
> > At my current project we wanted to monitor for a specific field the
> > fraction of indexed vs doc values queries executed by
> > IndexOrDocValuesQuery.
> >
> > We ended up forking IndexOrDocValuesQuery and passing a listener that
> > is notified when the query execution path is decided.
> >
> > Do you think this is something the community might be interested in?
> >
> > Kind regards,
> > Egor Moraru.
> >
>
>
> --
> Adrien
>


--

Kind regards,
Egor Moraru.

This sounds useful indeed!

Is there a way we could do it that wouldn't require forking
IndexOrDocValuesQuery? E.g. could we have query wrappers that we would use
on both the index query and the doc-value query in order to be able to
count how many times they have been used? We could add something like that
to lucene/sandbox.

On Thu, Jun 10, 2021 at 2:51 PM Egor Moraru <moraru.egor@gmail.com> wrote:

> Hi Adrien,
>
> In this specific use case our data is encoded as points and also is stored
> as doc values.
>
> We use information about which execution path is chosen to decide
> if we can get away with storing this data only once and using one of
> the queries.
>
> On Wed, Jun 9, 2021 at 10:39 PM Adrien Grand <jpountz@gmail.com> wrote:
>
> > FWIW a related PR was just merged that allows to introspect query
> > execution: https://issues.apache.org/jira/browse/LUCENE-9965. It's
> > different from your use-case though in that it is debugging information
> for
> > a single query rather than statistical information across lots of user
> > queries (and the approach on that other issue makes things much slower so
> > you wouldn't like to enable it in production).
> >
> > Out of curiosity, what are you doing with this information about which
> > execution path is chosen?
> >
> > On Wed, Jun 9, 2021 at 2:14 PM Egor Moraru <moraru.egor@gmail.com>
> wrote:
> >
> > > Hi,
> > >
> > > At my current project we wanted to monitor for a specific field the
> > > fraction of indexed vs doc values queries executed by
> > > IndexOrDocValuesQuery.
> > >
> > > We ended up forking IndexOrDocValuesQuery and passing a listener that
> > > is notified when the query execution path is decided.
> > >
> > > Do you think this is something the community might be interested in?
> > >
> > > Kind regards,
> > > Egor Moraru.
> > >
> >
> >
> > --
> > Adrien
> >
>
>
> --
>
> Kind regards,
> Egor Moraru.
>


--
Adrien

That should be doable and it is less invasive.

I'll give it a try, Thanks!

On Thu, Jun 10, 2021 at 2:21 PM Adrien Grand <jpountz@gmail.com> wrote:

> This sounds useful indeed!
>
> Is there a way we could do it that wouldn't require forking
> IndexOrDocValuesQuery? E.g. could we have query wrappers that we would use
> on both the index query and the doc-value query in order to be able to
> count how many times they have been used? We could add something like that
> to lucene/sandbox.
>
> On Thu, Jun 10, 2021 at 2:51 PM Egor Moraru <moraru.egor@gmail.com> wrote:
>
> > Hi Adrien,
> >
> > In this specific use case our data is encoded as points and also is
> stored
> > as doc values.
> >
> > We use information about which execution path is chosen to decide
> > if we can get away with storing this data only once and using one of
> > the queries.
> >
> > On Wed, Jun 9, 2021 at 10:39 PM Adrien Grand <jpountz@gmail.com> wrote:
> >
> > > FWIW a related PR was just merged that allows to introspect query
> > > execution: https://issues.apache.org/jira/browse/LUCENE-9965. It's
> > > different from your use-case though in that it is debugging information
> > for
> > > a single query rather than statistical information across lots of user
> > > queries (and the approach on that other issue makes things much slower
> so
> > > you wouldn't like to enable it in production).
> > >
> > > Out of curiosity, what are you doing with this information about which
> > > execution path is chosen?
> > >
> > > On Wed, Jun 9, 2021 at 2:14 PM Egor Moraru <moraru.egor@gmail.com>
> > wrote:
> > >
> > > > Hi,
> > > >
> > > > At my current project we wanted to monitor for a specific field the
> > > > fraction of indexed vs doc values queries executed by
> > > > IndexOrDocValuesQuery.
> > > >
> > > > We ended up forking IndexOrDocValuesQuery and passing a listener that
> > > > is notified when the query execution path is decided.
> > > >
> > > > Do you think this is something the community might be interested in?
> > > >
> > > > Kind regards,
> > > > Egor Moraru.
> > > >
> > >
> > >
> > > --
> > > Adrien
> > >
> >
> >
> > --
> >
> > Kind regards,
> > Egor Moraru.
> >
>
>
> --
> Adrien
>


--

Kind regards,
Egor Moraru.