Mailing List Archive: How to Safely Rollback after Upgrading Lucene?

How to Safely Rollback after Upgrading Lucene?

Jun 23, 2023, 2:34 PM

Post #1 of 5 (258 views)

Hello,

I have a service that creates and manages Lucene indices. The service is
using Lucene 8 and I want to upgrade to Lucene 9, and I would like to be
able to rollback the upgrade in case I encounter any issues (*). The
problem is that the older version of the service won't be able to read the
segments created after the upgrade because they would be created using
Lucene 9's codec.

One potential solution is to keep writing new indices using Lucene 8's
codecs after upgrading to Lucene 9, and only start using Lucene 9's codec
after some time. However this doesn't seem to be officially supported: the
lucene-backward-codecs package provides older codecs, but they are
read-only.

My questions are:
1. I can try copying over the actual codecs from Lucene 8 instead of using
lucene-backward-codecs, but is this actually supposed to work, or will
there be undefined behaviors when the codec is used with Lucene 9?
2. Is there an alternative solution to support rollbacks?

Any help would be greatly appreciated!

Best regards,
Yixun Xu

(*) I'm not really worried about the Lucene upgrade itself, but I have less
confidence in the changes I make alongside the upgrade. For example, if I
make some changes to my code to address compilation breaks brought by the
Lucene upgrade, then I'd like to keep the option to immediately rollback
the service in case those changes I made end up causing serious issues.

Re: How to Safely Rollback after Upgrading Lucene? [ In reply to ]

stevenschlansker at gmail

Jun 23, 2023, 5:18 PM

Post #2 of 5 (258 views)

Permalink

Hi,

> On Jun 23, 2023, at 2:34 PM, Yixun Xu <yixunx@gmail.com> wrote:
>
> Hello,
>
> I have a service that creates and manages Lucene indices. The service is
> using Lucene 8 and I want to upgrade to Lucene 9, and I would like to be
> able to rollback the upgrade in case I encounter any issues (*). The
> problem is that the older version of the service won't be able to read the
> segments created after the upgrade because they would be created using
> Lucene 9's codec.
>
> One potential solution is to keep writing new indices using Lucene 8's
> codecs after upgrading to Lucene 9, and only start using Lucene 9's codec
> after some time. However this doesn't seem to be officially supported: the
> lucene-backward-codecs package provides older codecs, but they are
> read-only.
>
> My questions are:
> 1. I can try copying over the actual codecs from Lucene 8 instead of using
> lucene-backward-codecs, but is this actually supposed to work, or will
> there be undefined behaviors when the codec is used with Lucene 9?
> 2. Is there an alternative solution to support rollbacks?

This is probably not the solution you are looking for, but the way we manage this is to
have a higher level logical version (including Lucene codes, backwards-incompatible document format changes, etc).
We read all indexing operations off of a Kafka topic, and each index format version independently keeps Kafka offsets.
So, after upgrading to index format X+1, rolling back to index format X will also rewind the Kafka consumers and they
will replay any missed indexing operations immediately.

>
> Any help would be greatly appreciated!
>
> Best regards,
> Yixun Xu

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org

Re: How to Safely Rollback after Upgrading Lucene? [ In reply to ]

yixunx at gmail

Jun 24, 2023, 8:28 AM

Post #3 of 5 (258 views)

Permalink

Thank you for the suggestion. We do keep the original data around in an
upstream database, so we are also able to reindex the data after the
rollback, but I'm curious if there are ways to avoid the need to reindex
for some time. We'll eventually start using the new codecs for sure, at
which time we'll lose the ability to rollback without reindex anyways, but
ideally I'd like to make that a separate step from the Lucene upgrade, and
possibly apply the codec upgrade gradually per index.

Best regards,
Yixun Xu

On Fri, Jun 23, 2023 at 8:19?PM Steven Schlansker <
stevenschlansker@gmail.com> wrote:

> Hi,
>
> > On Jun 23, 2023, at 2:34 PM, Yixun Xu <yixunx@gmail.com> wrote:
> >
> > Hello,
> >
> > I have a service that creates and manages Lucene indices. The service is
> > using Lucene 8 and I want to upgrade to Lucene 9, and I would like to be
> > able to rollback the upgrade in case I encounter any issues (*). The
> > problem is that the older version of the service won't be able to read
> the
> > segments created after the upgrade because they would be created using
> > Lucene 9's codec.
> >
> > One potential solution is to keep writing new indices using Lucene 8's
> > codecs after upgrading to Lucene 9, and only start using Lucene 9's codec
> > after some time. However this doesn't seem to be officially supported:
> the
> > lucene-backward-codecs package provides older codecs, but they are
> > read-only.
> >
> > My questions are:
> > 1. I can try copying over the actual codecs from Lucene 8 instead of
> using
> > lucene-backward-codecs, but is this actually supposed to work, or will
> > there be undefined behaviors when the codec is used with Lucene 9?
> > 2. Is there an alternative solution to support rollbacks?
>
> This is probably not the solution you are looking for, but the way we
> manage this is to
> have a higher level logical version (including Lucene codes,
> backwards-incompatible document format changes, etc).
> We read all indexing operations off of a Kafka topic, and each index
> format version independently keeps Kafka offsets.
> So, after upgrading to index format X+1, rolling back to index format X
> will also rewind the Kafka consumers and they
> will replay any missed indexing operations immediately.
>
> >
> > Any help would be greatly appreciated!
> >
> > Best regards,
> > Yixun Xu
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>

Re: How to Safely Rollback after Upgrading Lucene? [ In reply to ]

worah.gautam at gmail

Jun 24, 2023, 11:42 AM

Post #4 of 5 (258 views)

Permalink

Hi Yixun,

Given that Lucene supports file format backwards compatibility for X-1
major version release with version X application code,
https://cwiki.apache.org/confluence/display/lucene/BackwardsCompatibility

Here is something you can try:

Keep using Lucene 8 indexes and Lucene 8 application code as your baseline.
Then switch just the application code to Lucene 9 (this will presumably be
slightly different as some APIs must've changed)
If something goes wrong, just switch back to Lucene 8 application code,
else, just keep using Lucene 9 code.
Then, switch to Lucene 9 indexes, and they should just work with Lucene 9
application code (I have some small concerns here around whether or not the
exact same Lucene APIs used to read Lucene 8 indexes would work with Lucene
9 indexes, if not the application code would change here slightly).

I would first perform these steps in just a single dev box with a static
index and check the flow.

Hope this helps.

On Sat, Jun 24, 2023 at 8:28?AM Yixun Xu <yixunx@gmail.com> wrote:

> Thank you for the suggestion. We do keep the original data around in an
> upstream database, so we are also able to reindex the data after the
> rollback, but I'm curious if there are ways to avoid the need to reindex
> for some time. We'll eventually start using the new codecs for sure, at
> which time we'll lose the ability to rollback without reindex anyways, but
> ideally I'd like to make that a separate step from the Lucene upgrade, and
> possibly apply the codec upgrade gradually per index.
>
> Best regards,
> Yixun Xu
>
>
> On Fri, Jun 23, 2023 at 8:19?PM Steven Schlansker <
> stevenschlansker@gmail.com> wrote:
>
> > Hi,
> >
> > > On Jun 23, 2023, at 2:34 PM, Yixun Xu <yixunx@gmail.com> wrote:
> > >
> > > Hello,
> > >
> > > I have a service that creates and manages Lucene indices. The service
> is
> > > using Lucene 8 and I want to upgrade to Lucene 9, and I would like to
> be
> > > able to rollback the upgrade in case I encounter any issues (*). The
> > > problem is that the older version of the service won't be able to read
> > the
> > > segments created after the upgrade because they would be created using
> > > Lucene 9's codec.
> > >
> > > One potential solution is to keep writing new indices using Lucene 8's
> > > codecs after upgrading to Lucene 9, and only start using Lucene 9's
> codec
> > > after some time. However this doesn't seem to be officially supported:
> > the
> > > lucene-backward-codecs package provides older codecs, but they are
> > > read-only.
> > >
> > > My questions are:
> > > 1. I can try copying over the actual codecs from Lucene 8 instead of
> > using
> > > lucene-backward-codecs, but is this actually supposed to work, or will
> > > there be undefined behaviors when the codec is used with Lucene 9?
> > > 2. Is there an alternative solution to support rollbacks?
> >
> > This is probably not the solution you are looking for, but the way we
> > manage this is to
> > have a higher level logical version (including Lucene codes,
> > backwards-incompatible document format changes, etc).
> > We read all indexing operations off of a Kafka topic, and each index
> > format version independently keeps Kafka offsets.
> > So, after upgrading to index format X+1, rolling back to index format X
> > will also rewind the Kafka consumers and they
> > will replay any missed indexing operations immediately.
> >
> > >
> > > Any help would be greatly appreciated!
> > >
> > > Best regards,
> > > Yixun Xu
> >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> > For additional commands, e-mail: java-user-help@lucene.apache.org
> >
> >
>

Re: How to Safely Rollback after Upgrading Lucene? [ In reply to ]

yixunx at gmail

Jun 26, 2023, 7:16 PM

Post #5 of 5 (253 views)

Permalink

That makes sense, thank you. These steps would work, and the caveat is that
my service doesn't have a very clear separation between "indexing code" and
"application code", so I'll likely need to do some refactoring before using
this solution, but I'll look into it.

I also found that Lucene has unit tests that use the older codecs for
writing (the *RWCodec
<https://github.com/apache/lucene/blob/b50b969d2539309d113d9c283e4cb68e84977412/lucene/backward-codecs/src/test/org/apache/lucene/backward_codecs/lucene87/Lucene87RWCodec.java>),
so I suppose I can also copy those into my code to keep writing indices in
Lucene 8's format, and that would be another potential solution.

Thanks everyone for the help!

Best regards,
Yixun Xu

On Sat, Jun 24, 2023 at 2:42?PM Gautam Worah <worah.gautam@gmail.com> wrote:

> Hi Yixun,
>
> Given that Lucene supports file format backwards compatibility for X-1
> major version release with version X application code,
> https://cwiki.apache.org/confluence/display/lucene/BackwardsCompatibility
>
> Here is something you can try:
>
> Keep using Lucene 8 indexes and Lucene 8 application code as your baseline.
> Then switch just the application code to Lucene 9 (this will presumably be
> slightly different as some APIs must've changed)
> If something goes wrong, just switch back to Lucene 8 application code,
> else, just keep using Lucene 9 code.
> Then, switch to Lucene 9 indexes, and they should just work with Lucene 9
> application code (I have some small concerns here around whether or not the
> exact same Lucene APIs used to read Lucene 8 indexes would work with Lucene
> 9 indexes, if not the application code would change here slightly).
>
> I would first perform these steps in just a single dev box with a static
> index and check the flow.
>
> Hope this helps.
>
>
> On Sat, Jun 24, 2023 at 8:28?AM Yixun Xu <yixunx@gmail.com> wrote:
>
> > Thank you for the suggestion. We do keep the original data around in an
> > upstream database, so we are also able to reindex the data after the
> > rollback, but I'm curious if there are ways to avoid the need to reindex
> > for some time. We'll eventually start using the new codecs for sure, at
> > which time we'll lose the ability to rollback without reindex anyways,
> but
> > ideally I'd like to make that a separate step from the Lucene upgrade,
> and
> > possibly apply the codec upgrade gradually per index.
> >
> > Best regards,
> > Yixun Xu
> >
> >
> > On Fri, Jun 23, 2023 at 8:19?PM Steven Schlansker <
> > stevenschlansker@gmail.com> wrote:
> >
> > > Hi,
> > >
> > > > On Jun 23, 2023, at 2:34 PM, Yixun Xu <yixunx@gmail.com> wrote:
> > > >
> > > > Hello,
> > > >
> > > > I have a service that creates and manages Lucene indices. The service
> > is
> > > > using Lucene 8 and I want to upgrade to Lucene 9, and I would like to
> > be
> > > > able to rollback the upgrade in case I encounter any issues (*). The
> > > > problem is that the older version of the service won't be able to
> read
> > > the
> > > > segments created after the upgrade because they would be created
> using
> > > > Lucene 9's codec.
> > > >
> > > > One potential solution is to keep writing new indices using Lucene
> 8's
> > > > codecs after upgrading to Lucene 9, and only start using Lucene 9's
> > codec
> > > > after some time. However this doesn't seem to be officially
> supported:
> > > the
> > > > lucene-backward-codecs package provides older codecs, but they are
> > > > read-only.
> > > >
> > > > My questions are:
> > > > 1. I can try copying over the actual codecs from Lucene 8 instead of
> > > using
> > > > lucene-backward-codecs, but is this actually supposed to work, or
> will
> > > > there be undefined behaviors when the codec is used with Lucene 9?
> > > > 2. Is there an alternative solution to support rollbacks?
> > >
> > > This is probably not the solution you are looking for, but the way we
> > > manage this is to
> > > have a higher level logical version (including Lucene codes,
> > > backwards-incompatible document format changes, etc).
> > > We read all indexing operations off of a Kafka topic, and each index
> > > format version independently keeps Kafka offsets.
> > > So, after upgrading to index format X+1, rolling back to index format X
> > > will also rewind the Kafka consumers and they
> > > will replay any missed indexing operations immediately.
> > >
> > > >
> > > > Any help would be greatly appreciated!
> > > >
> > > > Best regards,
> > > > Yixun Xu
> > >
> > >
> > > ---------------------------------------------------------------------
> > > To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> > > For additional commands, e-mail: java-user-help@lucene.apache.org
> > >
> > >
> >
>