Mailing List Archive: Expressions greedy advanceExact implementation

Expressions greedy advanceExact implementation

Oct 25, 2022, 10:13 AM

Post #1 of 8 (326 views)

ExpressionFunctionValueSource lazily evaluates in doubleValues: an
expression like

condition ? f1 : f2

will only evaluate one of f1 or f2.

At the same time, the advanceExact() call is greedy -- when you
advance that expression it will also advance both f1 and f2. But
here's the thing: it always returns true, regardless of whether f1 and
f2 advance. Which makes sense from the point of view of the lazy
evaluation -- if condition is true we don't care whether f2 advances
or not.

My question is whether we could defer these child advanceExact calls
until ExpressionFunctionValues.doubleValue()?

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org

Re: Expressions greedy advanceExact implementation [ In reply to ]

rcmuir at gmail

Oct 25, 2022, 7:00 PM

Post #2 of 8 (326 views)

Permalink

Iirc the expressions acts like a simple scripting engine where it just
compiles bytecode for your expression and you are able to bind variables
that you pass to the method... I don't know of an easy way to do this.

On Tue, Oct 25, 2022, 1:13 PM Michael Sokolov <msokolov@gmail.com> wrote:

> ExpressionFunctionValueSource lazily evaluates in doubleValues: an
> expression like
>
> condition ? f1 : f2
>
> will only evaluate one of f1 or f2.
>
> At the same time, the advanceExact() call is greedy -- when you
> advance that expression it will also advance both f1 and f2. But
> here's the thing: it always returns true, regardless of whether f1 and
> f2 advance. Which makes sense from the point of view of the lazy
> evaluation -- if condition is true we don't care whether f2 advances
> or not.
>
> My question is whether we could defer these child advanceExact calls
> until ExpressionFunctionValues.doubleValue()?
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: dev-help@lucene.apache.org
>
>

Re: Expressions greedy advanceExact implementation [ In reply to ]

mkhl at apache

Oct 26, 2022, 12:51 AM

Post #3 of 8 (325 views)

Permalink

Hello, Michael.
I suppose you can bind f2 to custom lazy implementation of DoubleValuesSource,
which defer advanceExact() by storing doc num and returning true always,
and actually advancing on doubleValue() only.

On Tue, Oct 25, 2022 at 8:13 PM Michael Sokolov <msokolov@gmail.com> wrote:

> ExpressionFunctionValueSource lazily evaluates in doubleValues: an
> expression like
>
> condition ? f1 : f2
>
> will only evaluate one of f1 or f2.
>
> At the same time, the advanceExact() call is greedy -- when you
> advance that expression it will also advance both f1 and f2. But
> here's the thing: it always returns true, regardless of whether f1 and
> f2 advance. Which makes sense from the point of view of the lazy
> evaluation -- if condition is true we don't care whether f2 advances
> or not.
>
> My question is whether we could defer these child advanceExact calls
> until ExpressionFunctionValues.doubleValue()?
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: dev-help@lucene.apache.org
>
>

--
Sincerely yours
Mikhail Khludnev

Re: Expressions greedy advanceExact implementation [ In reply to ]

msokolov at gmail

Oct 26, 2022, 6:58 AM

Post #4 of 8 (325 views)

Permalink

Hi, yes, makes sense Mikhail, that will address most of the problem.
But I also think, given the way Expressions work (they always return
true from advanceExact) there is no reason for them to advance their
operands. This shifts the burden/concern from the developer who no
longer has to think as hard about this :) - let me post a PR that
shows

On Wed, Oct 26, 2022 at 3:52 AM Mikhail Khludnev <mkhl@apache.org> wrote:
>
> Hello, Michael.
> I suppose you can bind f2 to custom lazy implementation of DoubleValuesSource, which defer advanceExact() by storing doc num and returning true always, and actually advancing on doubleValue() only.
>
> On Tue, Oct 25, 2022 at 8:13 PM Michael Sokolov <msokolov@gmail.com> wrote:
>>
>> ExpressionFunctionValueSource lazily evaluates in doubleValues: an
>> expression like
>>
>> condition ? f1 : f2
>>
>> will only evaluate one of f1 or f2.
>>
>> At the same time, the advanceExact() call is greedy -- when you
>> advance that expression it will also advance both f1 and f2. But
>> here's the thing: it always returns true, regardless of whether f1 and
>> f2 advance. Which makes sense from the point of view of the lazy
>> evaluation -- if condition is true we don't care whether f2 advances
>> or not.
>>
>> My question is whether we could defer these child advanceExact calls
>> until ExpressionFunctionValues.doubleValue()?
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: dev-help@lucene.apache.org
>>
>
>
> --
> Sincerely yours
> Mikhail Khludnev

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org

Re: Expressions greedy advanceExact implementation [ In reply to ]

msokolov at gmail

Oct 26, 2022, 7:23 AM

Post #5 of 8 (325 views)

Permalink

see https://github.com/apache/lucene/pull/11878 ... it doesn't do what
I initially asked for (still advances all of the operands), but it
delays until doubleValue() is called, which is safe and could have
some impact

On Wed, Oct 26, 2022 at 9:58 AM Michael Sokolov <msokolov@gmail.com> wrote:
>
> Hi, yes, makes sense Mikhail, that will address most of the problem.
> But I also think, given the way Expressions work (they always return
> true from advanceExact) there is no reason for them to advance their
> operands. This shifts the burden/concern from the developer who no
> longer has to think as hard about this :) - let me post a PR that
> shows
>
> On Wed, Oct 26, 2022 at 3:52 AM Mikhail Khludnev <mkhl@apache.org> wrote:
> >
> > Hello, Michael.
> > I suppose you can bind f2 to custom lazy implementation of DoubleValuesSource, which defer advanceExact() by storing doc num and returning true always, and actually advancing on doubleValue() only.
> >
> > On Tue, Oct 25, 2022 at 8:13 PM Michael Sokolov <msokolov@gmail.com> wrote:
> >>
> >> ExpressionFunctionValueSource lazily evaluates in doubleValues: an
> >> expression like
> >>
> >> condition ? f1 : f2
> >>
> >> will only evaluate one of f1 or f2.
> >>
> >> At the same time, the advanceExact() call is greedy -- when you
> >> advance that expression it will also advance both f1 and f2. But
> >> here's the thing: it always returns true, regardless of whether f1 and
> >> f2 advance. Which makes sense from the point of view of the lazy
> >> evaluation -- if condition is true we don't care whether f2 advances
> >> or not.
> >>
> >> My question is whether we could defer these child advanceExact calls
> >> until ExpressionFunctionValues.doubleValue()?
> >>
> >> ---------------------------------------------------------------------
> >> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
> >> For additional commands, e-mail: dev-help@lucene.apache.org
> >>
> >
> >
> > --
> > Sincerely yours
> > Mikhail Khludnev

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org

Re: Expressions greedy advanceExact implementation [ In reply to ]

rcmuir at gmail

Oct 26, 2022, 7:31 AM

Post #6 of 8 (325 views)

Permalink

I think deferring the advance call like this is fine and harmless,
only because this DoubleValues "caches" the result for the current
doc, so its idempotent anyway.

Yes, about "advancing all the operands" as I mentioned, expressions
has no clue about this. If you wanted to change it, you'd have to push
both advancing AND caching down lower into the actual compiled
expression code.

I think this would add way too much complexity, especially when it
would only improve the ternary "if" feature in such cases.

On Wed, Oct 26, 2022 at 10:23 AM Michael Sokolov <msokolov@gmail.com> wrote:
>
> see https://github.com/apache/lucene/pull/11878 ... it doesn't do what
> I initially asked for (still advances all of the operands), but it
> delays until doubleValue() is called, which is safe and could have
> some impact
>
> On Wed, Oct 26, 2022 at 9:58 AM Michael Sokolov <msokolov@gmail.com> wrote:
> >
> > Hi, yes, makes sense Mikhail, that will address most of the problem.
> > But I also think, given the way Expressions work (they always return
> > true from advanceExact) there is no reason for them to advance their
> > operands. This shifts the burden/concern from the developer who no
> > longer has to think as hard about this :) - let me post a PR that
> > shows
> >
> > On Wed, Oct 26, 2022 at 3:52 AM Mikhail Khludnev <mkhl@apache.org> wrote:
> > >
> > > Hello, Michael.
> > > I suppose you can bind f2 to custom lazy implementation of DoubleValuesSource, which defer advanceExact() by storing doc num and returning true always, and actually advancing on doubleValue() only.
> > >
> > > On Tue, Oct 25, 2022 at 8:13 PM Michael Sokolov <msokolov@gmail.com> wrote:
> > >>
> > >> ExpressionFunctionValueSource lazily evaluates in doubleValues: an
> > >> expression like
> > >>
> > >> condition ? f1 : f2
> > >>
> > >> will only evaluate one of f1 or f2.
> > >>
> > >> At the same time, the advanceExact() call is greedy -- when you
> > >> advance that expression it will also advance both f1 and f2. But
> > >> here's the thing: it always returns true, regardless of whether f1 and
> > >> f2 advance. Which makes sense from the point of view of the lazy
> > >> evaluation -- if condition is true we don't care whether f2 advances
> > >> or not.
> > >>
> > >> My question is whether we could defer these child advanceExact calls
> > >> until ExpressionFunctionValues.doubleValue()?
> > >>
> > >> ---------------------------------------------------------------------
> > >> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
> > >> For additional commands, e-mail: dev-help@lucene.apache.org
> > >>
> > >
> > >
> > > --
> > > Sincerely yours
> > > Mikhail Khludnev
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: dev-help@lucene.apache.org
>

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org

Re: Expressions greedy advanceExact implementation [ In reply to ]

msokolov at gmail

Oct 26, 2022, 11:05 AM

Post #7 of 8 (324 views)

Permalink

Thanks, yeah I thought so too. Merged

On Wed, Oct 26, 2022 at 10:31 AM Robert Muir <rcmuir@gmail.com> wrote:
>
> I think deferring the advance call like this is fine and harmless,
> only because this DoubleValues "caches" the result for the current
> doc, so its idempotent anyway.
>
> Yes, about "advancing all the operands" as I mentioned, expressions
> has no clue about this. If you wanted to change it, you'd have to push
> both advancing AND caching down lower into the actual compiled
> expression code.
>
> I think this would add way too much complexity, especially when it
> would only improve the ternary "if" feature in such cases.
>
> On Wed, Oct 26, 2022 at 10:23 AM Michael Sokolov <msokolov@gmail.com> wrote:
> >
> > see https://github.com/apache/lucene/pull/11878 ... it doesn't do what
> > I initially asked for (still advances all of the operands), but it
> > delays until doubleValue() is called, which is safe and could have
> > some impact
> >
> > On Wed, Oct 26, 2022 at 9:58 AM Michael Sokolov <msokolov@gmail.com> wrote:
> > >
> > > Hi, yes, makes sense Mikhail, that will address most of the problem.
> > > But I also think, given the way Expressions work (they always return
> > > true from advanceExact) there is no reason for them to advance their
> > > operands. This shifts the burden/concern from the developer who no
> > > longer has to think as hard about this :) - let me post a PR that
> > > shows
> > >
> > > On Wed, Oct 26, 2022 at 3:52 AM Mikhail Khludnev <mkhl@apache.org> wrote:
> > > >
> > > > Hello, Michael.
> > > > I suppose you can bind f2 to custom lazy implementation of DoubleValuesSource, which defer advanceExact() by storing doc num and returning true always, and actually advancing on doubleValue() only.
> > > >
> > > > On Tue, Oct 25, 2022 at 8:13 PM Michael Sokolov <msokolov@gmail.com> wrote:
> > > >>
> > > >> ExpressionFunctionValueSource lazily evaluates in doubleValues: an
> > > >> expression like
> > > >>
> > > >> condition ? f1 : f2
> > > >>
> > > >> will only evaluate one of f1 or f2.
> > > >>
> > > >> At the same time, the advanceExact() call is greedy -- when you
> > > >> advance that expression it will also advance both f1 and f2. But
> > > >> here's the thing: it always returns true, regardless of whether f1 and
> > > >> f2 advance. Which makes sense from the point of view of the lazy
> > > >> evaluation -- if condition is true we don't care whether f2 advances
> > > >> or not.
> > > >>
> > > >> My question is whether we could defer these child advanceExact calls
> > > >> until ExpressionFunctionValues.doubleValue()?
> > > >>
> > > >> ---------------------------------------------------------------------
> > > >> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
> > > >> For additional commands, e-mail: dev-help@lucene.apache.org
> > > >>
> > > >
> > > >
> > > > --
> > > > Sincerely yours
> > > > Mikhail Khludnev
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
> > For additional commands, e-mail: dev-help@lucene.apache.org
> >
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: dev-help@lucene.apache.org
>

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org

Re: Expressions greedy advanceExact implementation [ In reply to ]

gsmiller at gmail

Sep 15, 2023, 8:45 AM

Post #8 of 8 (181 views)

Permalink

Reviving this thread with another thought…

I think we can improve on this last solution and lazily advance an
expression’s referenced double values without needing to push complexity
down into compiled expression.

What if we do something like this?
https://github.com/apache/lucene/pull/12560

Cheers,
-Greg

On Wed, Oct 26, 2022 at 11:06 Michael Sokolov <msokolov@gmail.com> wrote:

> Thanks, yeah I thought so too. Merged
>
> On Wed, Oct 26, 2022 at 10:31 AM Robert Muir <rcmuir@gmail.com> wrote:
> >
> > I think deferring the advance call like this is fine and harmless,
> > only because this DoubleValues "caches" the result for the current
> > doc, so its idempotent anyway.
> >
> > Yes, about "advancing all the operands" as I mentioned, expressions
> > has no clue about this. If you wanted to change it, you'd have to push
> > both advancing AND caching down lower into the actual compiled
> > expression code.
> >
> > I think this would add way too much complexity, especially when it
> > would only improve the ternary "if" feature in such cases.
> >
> > On Wed, Oct 26, 2022 at 10:23 AM Michael Sokolov <msokolov@gmail.com>
> wrote:
> > >
> > > see https://github.com/apache/lucene/pull/11878 ... it doesn't do what
> > > I initially asked for (still advances all of the operands), but it
> > > delays until doubleValue() is called, which is safe and could have
> > > some impact
> > >
> > > On Wed, Oct 26, 2022 at 9:58 AM Michael Sokolov <msokolov@gmail.com>
> wrote:
> > > >
> > > > Hi, yes, makes sense Mikhail, that will address most of the problem.
> > > > But I also think, given the way Expressions work (they always return
> > > > true from advanceExact) there is no reason for them to advance their
> > > > operands. This shifts the burden/concern from the developer who no
> > > > longer has to think as hard about this :) - let me post a PR that
> > > > shows
> > > >
> > > > On Wed, Oct 26, 2022 at 3:52 AM Mikhail Khludnev <mkhl@apache.org>
> wrote:
> > > > >
> > > > > Hello, Michael.
> > > > > I suppose you can bind f2 to custom lazy implementation of
> DoubleValuesSource, which defer advanceExact() by storing doc num and
> returning true always, and actually advancing on doubleValue() only.
> > > > >
> > > > > On Tue, Oct 25, 2022 at 8:13 PM Michael Sokolov <
> msokolov@gmail.com> wrote:
> > > > >>
> > > > >> ExpressionFunctionValueSource lazily evaluates in doubleValues: an
> > > > >> expression like
> > > > >>
> > > > >> condition ? f1 : f2
> > > > >>
> > > > >> will only evaluate one of f1 or f2.
> > > > >>
> > > > >> At the same time, the advanceExact() call is greedy -- when you
> > > > >> advance that expression it will also advance both f1 and f2. But
> > > > >> here's the thing: it always returns true, regardless of whether
> f1 and
> > > > >> f2 advance. Which makes sense from the point of view of the lazy
> > > > >> evaluation -- if condition is true we don't care whether f2
> advances
> > > > >> or not.
> > > > >>
> > > > >> My question is whether we could defer these child advanceExact
> calls
> > > > >> until ExpressionFunctionValues.doubleValue()?
> > > > >>
> > > > >>
> ---------------------------------------------------------------------
> > > > >> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
> > > > >> For additional commands, e-mail: dev-help@lucene.apache.org
> > > > >>
> > > > >
> > > > >
> > > > > --
> > > > > Sincerely yours
> > > > > Mikhail Khludnev
> > >
> > > ---------------------------------------------------------------------
> > > To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
> > > For additional commands, e-mail: dev-help@lucene.apache.org
> > >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
> > For additional commands, e-mail: dev-help@lucene.apache.org
> >
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: dev-help@lucene.apache.org
>
>