Mailing List Archive

Slower fetch document after upgrade >=8.7
Hi
BEST_SPEED has been changed in LUCENE-9447 and LUCENE-9486. For this
reason, retrieving data from elasticsearch has slowed down by 10-20%. When
there is a lot of data, this is critical
Can developers leave the choice of which codec to use: LZ4(16kB) (old
BEST_SPEED) or LZ4 with preset dict(BEST_SPEED_SAVING_DISKSIZE)? Or make
more flexible settings?

Otherwise, such changes may be a blocker or will have to spend money on
buying new hardware
Re: Slower fetch document after upgrade >=8.7 [ In reply to ]
Thanks for the feedback.

We don't want to offer too many choices, as it complicates backward
compatibility testing, and want to stick to two options at most.

Since this is the second time I'm seeing this feedback, I'm inclined to
reduce the block size for BEST_SPEED in order to trade a bit of compression
ratio for better decompression speed. I have some changes to stored fields
on my plate, I'll include this change as well.

On Thu, Apr 8, 2021 at 7:04 AM ?????? ???????? <mihaylovnikitos@gmail.com>
wrote:

> Hi
> BEST_SPEED has been changed in LUCENE-9447 and LUCENE-9486. For this
> reason, retrieving data from elasticsearch has slowed down by 10-20%. When
> there is a lot of data, this is critical
> Can developers leave the choice of which codec to use: LZ4(16kB) (old
> BEST_SPEED) or LZ4 with preset dict(BEST_SPEED_SAVING_DISKSIZE)? Or make
> more flexible settings?
>
> Otherwise, such changes may be a blocker or will have to spend money on
> buying new hardware
>


--
Adrien
Re: Slower fetch document after upgrade >=8.7 [ In reply to ]
Thanks for the reply.

The problem of understanding. You can make flexible settings for
advanced developers, leaving two facets by default. In tests, check
these facets
Never change them so that the developers themselves explicitly set the
settings. IMHO, I think this will help to avoid such problems

OK. Have a ticket?

??, 8 ???. 2021 ?. ? 13:52, Adrien Grand <jpountz@gmail.com>:
>
> Thanks for the feedback.
>
> We don't want to offer too many choices, as it complicates backward
> compatibility testing, and want to stick to two options at most.
>
> Since this is the second time I'm seeing this feedback, I'm inclined to
> reduce the block size for BEST_SPEED in order to trade a bit of compression
> ratio for better decompression speed. I have some changes to stored fields
> on my plate, I'll include this change as well.
>
> On Thu, Apr 8, 2021 at 7:04 AM ?????? ???????? <mihaylovnikitos@gmail.com>
> wrote:
>
> > Hi
> > BEST_SPEED has been changed in LUCENE-9447 and LUCENE-9486. For this
> > reason, retrieving data from elasticsearch has slowed down by 10-20%. When
> > there is a lot of data, this is critical
> > Can developers leave the choice of which codec to use: LZ4(16kB) (old
> > BEST_SPEED) or LZ4 with preset dict(BEST_SPEED_SAVING_DISKSIZE)? Or make
> > more flexible settings?
> >
> > Otherwise, such changes may be a blocker or will have to spend money on
> > buying new hardware
> >
>
>
> --
> Adrien

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Slower fetch document after upgrade >=8.7 [ In reply to ]
Actually, we don't plan to have flexible settings even for advanced
developers. Our stance on these discussions is that we should be
opinionated about the default codec and not offer any options. Rather than
exposing advanced settings for advanced users, these advanced users can
build their own codec and take care of backward compatibility themselves.

On Thu, Apr 8, 2021 at 10:11 AM ?????? ???????? <mihaylovnikitos@gmail.com>
wrote:

> Thanks for the reply.
>
> The problem of understanding. You can make flexible settings for
> advanced developers, leaving two facets by default. In tests, check
> these facets
> Never change them so that the developers themselves explicitly set the
> settings. IMHO, I think this will help to avoid such problems
>
> OK. Have a ticket?
>
> ??, 8 ???. 2021 ?. ? 13:52, Adrien Grand <jpountz@gmail.com>:
> >
> > Thanks for the feedback.
> >
> > We don't want to offer too many choices, as it complicates backward
> > compatibility testing, and want to stick to two options at most.
> >
> > Since this is the second time I'm seeing this feedback, I'm inclined to
> > reduce the block size for BEST_SPEED in order to trade a bit of
> compression
> > ratio for better decompression speed. I have some changes to stored
> fields
> > on my plate, I'll include this change as well.
> >
> > On Thu, Apr 8, 2021 at 7:04 AM ?????? ???????? <
> mihaylovnikitos@gmail.com>
> > wrote:
> >
> > > Hi
> > > BEST_SPEED has been changed in LUCENE-9447 and LUCENE-9486. For this
> > > reason, retrieving data from elasticsearch has slowed down by 10-20%.
> When
> > > there is a lot of data, this is critical
> > > Can developers leave the choice of which codec to use: LZ4(16kB) (old
> > > BEST_SPEED) or LZ4 with preset dict(BEST_SPEED_SAVING_DISKSIZE)? Or
> make
> > > more flexible settings?
> > >
> > > Otherwise, such changes may be a blocker or will have to spend money on
> > > buying new hardware
> > >
> >
> >
> > --
> > Adrien
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>

--
Adrien
Re: Slower fetch document after upgrade >=8.7 [ In reply to ]
Thank you. Understood the policy

> I have some changes to stored fields on my plate, I'll include this change as well.

Is there a ticket for this change?

??, 8 ???. 2021 ?. ? 15:30, Adrien Grand <jpountz@gmail.com>:
>
> Actually, we don't plan to have flexible settings even for advanced
> developers. Our stance on these discussions is that we should be
> opinionated about the default codec and not offer any options. Rather than
> exposing advanced settings for advanced users, these advanced users can
> build their own codec and take care of backward compatibility themselves.
>
> On Thu, Apr 8, 2021 at 10:11 AM ?????? ???????? <mihaylovnikitos@gmail.com>
> wrote:
>
> > Thanks for the reply.
> >
> > The problem of understanding. You can make flexible settings for
> > advanced developers, leaving two facets by default. In tests, check
> > these facets
> > Never change them so that the developers themselves explicitly set the
> > settings. IMHO, I think this will help to avoid such problems
> >
> > OK. Have a ticket?
> >
> > ??, 8 ???. 2021 ?. ? 13:52, Adrien Grand <jpountz@gmail.com>:
> > >
> > > Thanks for the feedback.
> > >
> > > We don't want to offer too many choices, as it complicates backward
> > > compatibility testing, and want to stick to two options at most.
> > >
> > > Since this is the second time I'm seeing this feedback, I'm inclined to
> > > reduce the block size for BEST_SPEED in order to trade a bit of
> > compression
> > > ratio for better decompression speed. I have some changes to stored
> > fields
> > > on my plate, I'll include this change as well.
> > >
> > > On Thu, Apr 8, 2021 at 7:04 AM ?????? ???????? <
> > mihaylovnikitos@gmail.com>
> > > wrote:
> > >
> > > > Hi
> > > > BEST_SPEED has been changed in LUCENE-9447 and LUCENE-9486. For this
> > > > reason, retrieving data from elasticsearch has slowed down by 10-20%.
> > When
> > > > there is a lot of data, this is critical
> > > > Can developers leave the choice of which codec to use: LZ4(16kB) (old
> > > > BEST_SPEED) or LZ4 with preset dict(BEST_SPEED_SAVING_DISKSIZE)? Or
> > make
> > > > more flexible settings?
> > > >
> > > > Otherwise, such changes may be a blocker or will have to spend money on
> > > > buying new hardware
> > > >
> > >
> > >
> > > --
> > > Adrien
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> > For additional commands, e-mail: java-user-help@lucene.apache.org
> >
> >
>
> --
> Adrien

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Slower fetch document after upgrade >=8.7 [ In reply to ]
I just opened https://issues.apache.org/jira/browse/LUCENE-9917.

On Thu, Apr 8, 2021 at 4:34 PM ?????? ???????? <mihaylovnikitos@gmail.com>
wrote:

> Thank you. Understood the policy
>
> > I have some changes to stored fields on my plate, I'll include this
> change as well.
>
> Is there a ticket for this change?
>
> ??, 8 ???. 2021 ?. ? 15:30, Adrien Grand <jpountz@gmail.com>:
> >
> > Actually, we don't plan to have flexible settings even for advanced
> > developers. Our stance on these discussions is that we should be
> > opinionated about the default codec and not offer any options. Rather
> than
> > exposing advanced settings for advanced users, these advanced users can
> > build their own codec and take care of backward compatibility themselves.
> >
> > On Thu, Apr 8, 2021 at 10:11 AM ?????? ???????? <
> mihaylovnikitos@gmail.com>
> > wrote:
> >
> > > Thanks for the reply.
> > >
> > > The problem of understanding. You can make flexible settings for
> > > advanced developers, leaving two facets by default. In tests, check
> > > these facets
> > > Never change them so that the developers themselves explicitly set the
> > > settings. IMHO, I think this will help to avoid such problems
> > >
> > > OK. Have a ticket?
> > >
> > > ??, 8 ???. 2021 ?. ? 13:52, Adrien Grand <jpountz@gmail.com>:
> > > >
> > > > Thanks for the feedback.
> > > >
> > > > We don't want to offer too many choices, as it complicates backward
> > > > compatibility testing, and want to stick to two options at most.
> > > >
> > > > Since this is the second time I'm seeing this feedback, I'm inclined
> to
> > > > reduce the block size for BEST_SPEED in order to trade a bit of
> > > compression
> > > > ratio for better decompression speed. I have some changes to stored
> > > fields
> > > > on my plate, I'll include this change as well.
> > > >
> > > > On Thu, Apr 8, 2021 at 7:04 AM ?????? ???????? <
> > > mihaylovnikitos@gmail.com>
> > > > wrote:
> > > >
> > > > > Hi
> > > > > BEST_SPEED has been changed in LUCENE-9447 and LUCENE-9486. For
> this
> > > > > reason, retrieving data from elasticsearch has slowed down by
> 10-20%.
> > > When
> > > > > there is a lot of data, this is critical
> > > > > Can developers leave the choice of which codec to use: LZ4(16kB)
> (old
> > > > > BEST_SPEED) or LZ4 with preset dict(BEST_SPEED_SAVING_DISKSIZE)? Or
> > > make
> > > > > more flexible settings?
> > > > >
> > > > > Otherwise, such changes may be a blocker or will have to spend
> money on
> > > > > buying new hardware
> > > > >
> > > >
> > > >
> > > > --
> > > > Adrien
> > >
> > > ---------------------------------------------------------------------
> > > To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> > > For additional commands, e-mail: java-user-help@lucene.apache.org
> > >
> > >
> >
> > --
> > Adrien
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>

--
Adrien