Mailing List Archive: Proposal for the Lucene Dependency after git repo split

Proposal for the Lucene Dependency after git repo split

Feb 25, 2021, 1:50 PM

Post #1 of 13 (819 views)

Hey everyone,

Currently there is discussion going on, in SOLR-14762
<https://issues.apache.org/jira/browse/SOLR-14762>, regarding the split of
the lucene-solr repo into individual repos for Solr and Lucene. There seems
to be agreement that we shouldn't wait for a Lucene release to do the
split, and instead split now and release whenever that happens.

The biggest issue that arises there is that Solr's master branch is
obviously based on Lucene's master branch, since they are currently the
same. So when the split happens, Solr master will have to depend on Lucene
9.0-SNAPSHOT. We can have solr merely depend on the lucene snapshot, but
that will result in inconsistent builds, depending on whatever cached
dependencies each dev has locally. Personally, I think that will cause a
bunch of build errors and headaches for everyone trying to maintain Solr.

There is another option though. We could instead do an *alpha* "release" of
lucene-solr 9.0 right before the repo is split. Therefore Solr can reliably
depend on a stable version of lucene until 9.0 is truly released. (And
lucene can use a stable version of Solr, if it sees a need for that). There
would be no guarantees for using this alpha release, and we don't have to
advertise it at all.

It's not perfect, but I think it would be preferable to depending on an
ever-changing SNAPSHOT lucene.

- Houston

Re: Proposal for the Lucene Dependency after git repo split [ In reply to ]

gus.heck at gmail

Feb 25, 2021, 2:48 PM

Post #2 of 13 (819 views)

Permalink

Until the first feature that wants to add something on both fronts... Is it
possible for Lucene to publish nightly snapshots? I know there is some
level of support for snapshots in central, though I don't know what
their usage policies are. If that's too restricted is there an artifact
repo controlled by the ASF that could be used? (An implementation of Apache
Archiva?) This would have the added benefit of allowing solr to detect when
Lucene breaks something before its released.

On Thu, Feb 25, 2021 at 4:50 PM Houston Putman <houstonputman@gmail.com>
wrote:

> Hey everyone,
>
> Currently there is discussion going on, in SOLR-14762
> <https://issues.apache.org/jira/browse/SOLR-14762>, regarding the split
> of the lucene-solr repo into individual repos for Solr and Lucene. There
> seems to be agreement that we shouldn't wait for a Lucene release to do the
> split, and instead split now and release whenever that happens.
>
> The biggest issue that arises there is that Solr's master branch is
> obviously based on Lucene's master branch, since they are currently the
> same. So when the split happens, Solr master will have to depend on Lucene
> 9.0-SNAPSHOT. We can have solr merely depend on the lucene snapshot, but
> that will result in inconsistent builds, depending on whatever cached
> dependencies each dev has locally. Personally, I think that will cause a
> bunch of build errors and headaches for everyone trying to maintain Solr.
>
> There is another option though. We could instead do an *alpha* "release"
> of lucene-solr 9.0 right before the repo is split. Therefore Solr can
> reliably depend on a stable version of lucene until 9.0 is truly released.
> (And lucene can use a stable version of Solr, if it sees a need for that).
> There would be no guarantees for using this alpha release, and we don't
> have to advertise it at all.
>
> It's not perfect, but I think it would be preferable to depending on an
> ever-changing SNAPSHOT lucene.
>
> - Houston
>

--
http://www.needhamsoftware.com (work)
http://www.the111shift.com (play)

Re: Proposal for the Lucene Dependency after git repo split [ In reply to ]

houstonputman at gmail

Feb 25, 2021, 3:02 PM

Post #3 of 13 (819 views)

Permalink

Once the projects are on separate release cadences there wont be an ability
to “add on both fronts” anymore. You will have to add to lucene, wait for a
release, then add to Solr once Solr upgrades its lucene dependency to that
new version. I dont imagine that we are going to keep Solr master/main, or
even 8x, 9x, etc, depending on Lucene snaphsots in perpetuity. After it
becomes possible (when lucene 9.0 is released) we should only be using
released lucene versions as dependencies for every version branch in Solr.

On Thu, Feb 25, 2021 at 5:49 PM Gus Heck <gus.heck@gmail.com> wrote:

> Until the first feature that wants to add something on both fronts... Is
> it possible for Lucene to publish nightly snapshots? I know there is some
> level of support for snapshots in central, though I don't know what
> their usage policies are. If that's too restricted is there an artifact
> repo controlled by the ASF that could be used? (An implementation of Apache
> Archiva?) This would have the added benefit of allowing solr to detect when
> Lucene breaks something before its released.
>
> On Thu, Feb 25, 2021 at 4:50 PM Houston Putman <houstonputman@gmail.com>
> wrote:
>
>> Hey everyone,
>>
>> Currently there is discussion going on, in SOLR-14762
>> <https://issues.apache.org/jira/browse/SOLR-14762>, regarding the split
>> of the lucene-solr repo into individual repos for Solr and Lucene. There
>> seems to be agreement that we shouldn't wait for a Lucene release to do the
>> split, and instead split now and release whenever that happens.
>>
>> The biggest issue that arises there is that Solr's master branch is
>> obviously based on Lucene's master branch, since they are currently the
>> same. So when the split happens, Solr master will have to depend on Lucene
>> 9.0-SNAPSHOT. We can have solr merely depend on the lucene snapshot, but
>> that will result in inconsistent builds, depending on whatever cached
>> dependencies each dev has locally. Personally, I think that will cause a
>> bunch of build errors and headaches for everyone trying to maintain Solr.
>>
>> There is another option though. We could instead do an *alpha* "release"
>> of lucene-solr 9.0 right before the repo is split. Therefore Solr can
>> reliably depend on a stable version of lucene until 9.0 is truly released.
>> (And lucene can use a stable version of Solr, if it sees a need for that).
>> There would be no guarantees for using this alpha release, and we don't
>> have to advertise it at all.
>>
>> It's not perfect, but I think it would be preferable to depending on an
>> ever-changing SNAPSHOT lucene.
>>
>> - Houston
>>
>
>
> --
> http://www.needhamsoftware.com (work)
> http://www.the111shift.com (play)
>

Re: Proposal for the Lucene Dependency after git repo split [ In reply to ]

mdrob at mdrob

Feb 25, 2021, 3:04 PM

Post #4 of 13 (819 views)

Permalink

It is possible to publish snapshots into the Apache Nexus repository. That
said, I think it is a bad idea for Solr to depend on Lucene snapshots
because that constrains the ability to do releases. Either you have to wait
for a Lucene release and then you can cut over, or you have to figure out
what changes you need to roll back.

Features today rarely touch both fronts anyway, they usually land in Lucene
first and then percolate into Solr. For an easy example, we can see how
WAND was developed recently.

On Thu, Feb 25, 2021 at 5:02 PM Houston Putman <houstonputman@gmail.com>
wrote:

> Once the projects are on separate release cadences there wont be an
> ability to “add on both fronts” anymore. You will have to add to lucene,
> wait for a release, then add to Solr once Solr upgrades its lucene
> dependency to that new version. I dont imagine that we are going to keep
> Solr master/main, or even 8x, 9x, etc, depending on Lucene snaphsots in
> perpetuity. After it becomes possible (when lucene 9.0 is released) we
> should only be using released lucene versions as dependencies for every
> version branch in Solr.
>
> On Thu, Feb 25, 2021 at 5:49 PM Gus Heck <gus.heck@gmail.com> wrote:
>
>> Until the first feature that wants to add something on both fronts... Is
>> it possible for Lucene to publish nightly snapshots? I know there is some
>> level of support for snapshots in central, though I don't know what
>> their usage policies are. If that's too restricted is there an artifact
>> repo controlled by the ASF that could be used? (An implementation of Apache
>> Archiva?) This would have the added benefit of allowing solr to detect when
>> Lucene breaks something before its released.
>>
>> On Thu, Feb 25, 2021 at 4:50 PM Houston Putman <houstonputman@gmail.com>
>> wrote:
>>
>>> Hey everyone,
>>>
>>> Currently there is discussion going on, in SOLR-14762
>>> <https://issues.apache.org/jira/browse/SOLR-14762>, regarding the split
>>> of the lucene-solr repo into individual repos for Solr and Lucene. There
>>> seems to be agreement that we shouldn't wait for a Lucene release to do the
>>> split, and instead split now and release whenever that happens.
>>>
>>> The biggest issue that arises there is that Solr's master branch is
>>> obviously based on Lucene's master branch, since they are currently the
>>> same. So when the split happens, Solr master will have to depend on Lucene
>>> 9.0-SNAPSHOT. We can have solr merely depend on the lucene snapshot, but
>>> that will result in inconsistent builds, depending on whatever cached
>>> dependencies each dev has locally. Personally, I think that will cause a
>>> bunch of build errors and headaches for everyone trying to maintain Solr.
>>>
>>> There is another option though. We could instead do an *alpha*
>>> "release" of lucene-solr 9.0 right before the repo is split. Therefore Solr
>>> can reliably depend on a stable version of lucene until 9.0 is truly
>>> released. (And lucene can use a stable version of Solr, if it sees a need
>>> for that). There would be no guarantees for using this alpha release, and
>>> we don't have to advertise it at all.
>>>
>>> It's not perfect, but I think it would be preferable to depending on an
>>> ever-changing SNAPSHOT lucene.
>>>
>>> - Houston
>>>
>>
>>
>> --
>> http://www.needhamsoftware.com (work)
>> http://www.the111shift.com (play)
>>
>

Re: Proposal for the Lucene Dependency after git repo split [ In reply to ]

jan.asf at cominvent

Feb 26, 2021, 1:41 AM

Post #5 of 13 (817 views)

Permalink

I think we can start off with depending on lucene-SNAPSHOT in main branch. We already have those, right?
https://repository.apache.org/#nexus-search;gav~org.apache.lucene~lucene-core~9.0.0-SNAPSHOT~~
If it becomes too flaky for Solr, we can push an internal 9.0-alpha-SNAPSHOT to the nexus snapshots directory and keep it stable for Solr to depend upon.

For post-9.0 development though, I think Solr main branch should depend on a stable lucene release.

The Solr project should probably find a good strategy for
1) Monitoring Lucene CHANGES.txt and update a backlog for what Solr MUST or SHOULD do in response.
2) When and how to upgrade Lucene dependency version in main. After each Lucene release? Once there is a need for it in Solr? Before each Solr release?

Jan

> 26. feb. 2021 kl. 00:04 skrev Mike Drob <mdrob@mdrob.com>:
>
> It is possible to publish snapshots into the Apache Nexus repository. That said, I think it is a bad idea for Solr to depend on Lucene snapshots because that constrains the ability to do releases. Either you have to wait for a Lucene release and then you can cut over, or you have to figure out what changes you need to roll back.
>
> Features today rarely touch both fronts anyway, they usually land in Lucene first and then percolate into Solr. For an easy example, we can see how WAND was developed recently.
>
> On Thu, Feb 25, 2021 at 5:02 PM Houston Putman <houstonputman@gmail.com <mailto:houstonputman@gmail.com>> wrote:
> Once the projects are on separate release cadences there wont be an ability to “add on both fronts” anymore. You will have to add to lucene, wait for a release, then add to Solr once Solr upgrades its lucene dependency to that new version. I dont imagine that we are going to keep Solr master/main, or even 8x, 9x, etc, depending on Lucene snaphsots in perpetuity. After it becomes possible (when lucene 9.0 is released) we should only be using released lucene versions as dependencies for every version branch in Solr.
>
> On Thu, Feb 25, 2021 at 5:49 PM Gus Heck <gus.heck@gmail.com <mailto:gus.heck@gmail.com>> wrote:
> Until the first feature that wants to add something on both fronts... Is it possible for Lucene to publish nightly snapshots? I know there is some level of support for snapshots in central, though I don't know what their usage policies are. If that's too restricted is there an artifact repo controlled by the ASF that could be used? (An implementation of Apache Archiva?) This would have the added benefit of allowing solr to detect when Lucene breaks something before its released.
>
> On Thu, Feb 25, 2021 at 4:50 PM Houston Putman <houstonputman@gmail.com <mailto:houstonputman@gmail.com>> wrote:
> Hey everyone,
>
> Currently there is discussion going on, in SOLR-14762 <https://issues.apache.org/jira/browse/SOLR-14762>, regarding the split of the lucene-solr repo into individual repos for Solr and Lucene. There seems to be agreement that we shouldn't wait for a Lucene release to do the split, and instead split now and release whenever that happens.
>
> The biggest issue that arises there is that Solr's master branch is obviously based on Lucene's master branch, since they are currently the same. So when the split happens, Solr master will have to depend on Lucene 9.0-SNAPSHOT. We can have solr merely depend on the lucene snapshot, but that will result in inconsistent builds, depending on whatever cached dependencies each dev has locally. Personally, I think that will cause a bunch of build errors and headaches for everyone trying to maintain Solr.
>
> There is another option though. We could instead do an alpha "release" of lucene-solr 9.0 right before the repo is split. Therefore Solr can reliably depend on a stable version of lucene until 9.0 is truly released. (And lucene can use a stable version of Solr, if it sees a need for that). There would be no guarantees for using this alpha release, and we don't have to advertise it at all.
>
> It's not perfect, but I think it would be preferable to depending on an ever-changing SNAPSHOT lucene.
>
> - Houston
>
>
> --
> http://www.needhamsoftware.com <http://www.needhamsoftware.com/> (work)
> http://www.the111shift.com <http://www.the111shift.com/> (play)

Re: Proposal for the Lucene Dependency after git repo split [ In reply to ]

gus.heck at gmail

Feb 26, 2021, 5:22 AM

Post #6 of 13 (817 views)

Permalink

Except I just finished helping a contributor with a feature that touches
both and I know for a fact that it was developed for his customer who was
using solr (payload inequalities)... and have another in the works (the
AQP).... Not being able to enhance lucene to support a feature in solr is
an issue IMHO.

On Thu, Feb 25, 2021 at 6:05 PM Mike Drob <mdrob@mdrob.com> wrote:

> It is possible to publish snapshots into the Apache Nexus repository. That
> said, I think it is a bad idea for Solr to depend on Lucene snapshots
> because that constrains the ability to do releases. Either you have to wait
> for a Lucene release and then you can cut over, or you have to figure out
> what changes you need to roll back.
>
> Features today rarely touch both fronts anyway, they usually land in
> Lucene first and then percolate into Solr. For an easy example, we can see
> how WAND was developed recently.
>
> On Thu, Feb 25, 2021 at 5:02 PM Houston Putman <houstonputman@gmail.com>
> wrote:
>
>> Once the projects are on separate release cadences there wont be an
>> ability to “add on both fronts” anymore. You will have to add to lucene,
>> wait for a release, then add to Solr once Solr upgrades its lucene
>> dependency to that new version. I dont imagine that we are going to keep
>> Solr master/main, or even 8x, 9x, etc, depending on Lucene snaphsots in
>> perpetuity. After it becomes possible (when lucene 9.0 is released) we
>> should only be using released lucene versions as dependencies for every
>> version branch in Solr.
>>
>> On Thu, Feb 25, 2021 at 5:49 PM Gus Heck <gus.heck@gmail.com> wrote:
>>
>>> Until the first feature that wants to add something on both fronts... Is
>>> it possible for Lucene to publish nightly snapshots? I know there is some
>>> level of support for snapshots in central, though I don't know what
>>> their usage policies are. If that's too restricted is there an artifact
>>> repo controlled by the ASF that could be used? (An implementation of Apache
>>> Archiva?) This would have the added benefit of allowing solr to detect when
>>> Lucene breaks something before its released.
>>>
>>> On Thu, Feb 25, 2021 at 4:50 PM Houston Putman <houstonputman@gmail.com>
>>> wrote:
>>>
>>>> Hey everyone,
>>>>
>>>> Currently there is discussion going on, in SOLR-14762
>>>> <https://issues.apache.org/jira/browse/SOLR-14762>, regarding the
>>>> split of the lucene-solr repo into individual repos for Solr and Lucene.
>>>> There seems to be agreement that we shouldn't wait for a Lucene release to
>>>> do the split, and instead split now and release whenever that happens.
>>>>
>>>> The biggest issue that arises there is that Solr's master branch is
>>>> obviously based on Lucene's master branch, since they are currently the
>>>> same. So when the split happens, Solr master will have to depend on Lucene
>>>> 9.0-SNAPSHOT. We can have solr merely depend on the lucene snapshot, but
>>>> that will result in inconsistent builds, depending on whatever cached
>>>> dependencies each dev has locally. Personally, I think that will cause a
>>>> bunch of build errors and headaches for everyone trying to maintain Solr.
>>>>
>>>> There is another option though. We could instead do an *alpha*
>>>> "release" of lucene-solr 9.0 right before the repo is split. Therefore Solr
>>>> can reliably depend on a stable version of lucene until 9.0 is truly
>>>> released. (And lucene can use a stable version of Solr, if it sees a need
>>>> for that). There would be no guarantees for using this alpha release, and
>>>> we don't have to advertise it at all.
>>>>
>>>> It's not perfect, but I think it would be preferable to depending on an
>>>> ever-changing SNAPSHOT lucene.
>>>>
>>>> - Houston
>>>>
>>>
>>>
>>> --
>>> http://www.needhamsoftware.com (work)
>>> http://www.the111shift.com (play)
>>>
>>

--
http://www.needhamsoftware.com (work)
http://www.the111shift.com (play)

Re: Proposal for the Lucene Dependency after git repo split [ In reply to ]

jan.asf at cominvent

Feb 26, 2021, 6:51 AM

Post #7 of 13 (817 views)

Permalink

The developer workflow for adding something to both Lucene and Solr would be as any other dependency, right?
So say we are on Luene 9.0. This is the process in my head:
Adapt Lucene as needed, and "mvn install" lucene-9.1-SNAPSHOT to your local laptop (whatever command that is in gradle)
Make your Solr feature branch depend on lucene-9.1-SNAPSHOT instead of lucene-9.0.0 -hopefully Gradle will pick the local version over Apache Nexus version
Iterate 1-2 until happy
Make a Lucene PR and eventually commit the Lucene change
After next Jenkins build the feature is in Apache Nexus snapshot as lucene-9.1-SNAPSHOT
Now the Solr Pull Request will compile and can be tested by others
Wait until Lucene 9.1 release
Upgrade Solr's lucene dependency on 'main'
Merge Solr PR
Backporting will be a similar process, i.e. first backport and release in Lucene, then backport in Sol
Hmm, as I wrote this list I can understand why so many features were added only to Solr and not to Lucene in the early days :)

Jan

> 26. feb. 2021 kl. 14:22 skrev Gus Heck <gus.heck@gmail.com>:
>
> Except I just finished helping a contributor with a feature that touches both and I know for a fact that it was developed for his customer who was using solr (payload inequalities)... and have another in the works (the AQP).... Not being able to enhance lucene to support a feature in solr is an issue IMHO.
>
> On Thu, Feb 25, 2021 at 6:05 PM Mike Drob <mdrob@mdrob.com <mailto:mdrob@mdrob.com>> wrote:
> It is possible to publish snapshots into the Apache Nexus repository. That said, I think it is a bad idea for Solr to depend on Lucene snapshots because that constrains the ability to do releases. Either you have to wait for a Lucene release and then you can cut over, or you have to figure out what changes you need to roll back.
>
> Features today rarely touch both fronts anyway, they usually land in Lucene first and then percolate into Solr. For an easy example, we can see how WAND was developed recently.
>
> On Thu, Feb 25, 2021 at 5:02 PM Houston Putman <houstonputman@gmail.com <mailto:houstonputman@gmail.com>> wrote:
> Once the projects are on separate release cadences there wont be an ability to “add on both fronts” anymore. You will have to add to lucene, wait for a release, then add to Solr once Solr upgrades its lucene dependency to that new version. I dont imagine that we are going to keep Solr master/main, or even 8x, 9x, etc, depending on Lucene snaphsots in perpetuity. After it becomes possible (when lucene 9.0 is released) we should only be using released lucene versions as dependencies for every version branch in Solr.
>
> On Thu, Feb 25, 2021 at 5:49 PM Gus Heck <gus.heck@gmail.com <mailto:gus.heck@gmail.com>> wrote:
> Until the first feature that wants to add something on both fronts... Is it possible for Lucene to publish nightly snapshots? I know there is some level of support for snapshots in central, though I don't know what their usage policies are. If that's too restricted is there an artifact repo controlled by the ASF that could be used? (An implementation of Apache Archiva?) This would have the added benefit of allowing solr to detect when Lucene breaks something before its released.
>
> On Thu, Feb 25, 2021 at 4:50 PM Houston Putman <houstonputman@gmail.com <mailto:houstonputman@gmail.com>> wrote:
> Hey everyone,
>
> Currently there is discussion going on, in SOLR-14762 <https://issues.apache.org/jira/browse/SOLR-14762>, regarding the split of the lucene-solr repo into individual repos for Solr and Lucene. There seems to be agreement that we shouldn't wait for a Lucene release to do the split, and instead split now and release whenever that happens.
>
> The biggest issue that arises there is that Solr's master branch is obviously based on Lucene's master branch, since they are currently the same. So when the split happens, Solr master will have to depend on Lucene 9.0-SNAPSHOT. We can have solr merely depend on the lucene snapshot, but that will result in inconsistent builds, depending on whatever cached dependencies each dev has locally. Personally, I think that will cause a bunch of build errors and headaches for everyone trying to maintain Solr.
>
> There is another option though. We could instead do an alpha "release" of lucene-solr 9.0 right before the repo is split. Therefore Solr can reliably depend on a stable version of lucene until 9.0 is truly released. (And lucene can use a stable version of Solr, if it sees a need for that). There would be no guarantees for using this alpha release, and we don't have to advertise it at all.
>
> It's not perfect, but I think it would be preferable to depending on an ever-changing SNAPSHOT lucene.
>
> - Houston
>
>
> --
> http://www.needhamsoftware.com <http://www.needhamsoftware.com/> (work)
> http://www.the111shift.com <http://www.the111shift.com/> (play)
>
>
> --
> http://www.needhamsoftware.com <http://www.needhamsoftware.com/> (work)
> http://www.the111shift.com <http://www.the111shift.com/> (play)

Re: Proposal for the Lucene Dependency after git repo split [ In reply to ]

mdrob at mdrob

Feb 26, 2021, 9:43 AM

Post #8 of 13 (817 views)

Permalink

The part of this process that I really don't like is that Solr then still
becomes beholden to Lucene's release schedule. We don't know how long step
7 takes, and will be effectively blocked from making our own releases until
that happens.

On Fri, Feb 26, 2021 at 8:51 AM Jan Høydahl <jan.asf@cominvent.com> wrote:

> The developer workflow for adding something to both Lucene and Solr would
> be as any other dependency, right?
> So say we are on Luene 9.0. This is the process in my head:
>
> 1. Adapt Lucene as needed, and "mvn install" lucene-9.1-SNAPSHOT to
> your local laptop (whatever command that is in gradle)
> 2. Make your Solr feature branch depend on lucene-9.1-SNAPSHOT instead
> of lucene-9.0.0 -hopefully Gradle will pick the local version over Apache
> Nexus version
> 3. Iterate 1-2 until happy
> 4. Make a Lucene PR and eventually commit the Lucene change
> 5. After next Jenkins build the feature is in Apache Nexus snapshot as
> lucene-9.1-SNAPSHOT
> 6. Now the Solr Pull Request will compile and can be tested by others
> 7. Wait until Lucene 9.1 release
> 8. Upgrade Solr's lucene dependency on 'main'
> 9. Merge Solr PR
> 10. Backporting will be a similar process, i.e. first backport and
> release in Lucene, then backport in Sol
>
> Hmm, as I wrote this list I can understand why so many features were added
> only to Solr and not to Lucene in the early days :)
>
> Jan
>
> 26. feb. 2021 kl. 14:22 skrev Gus Heck <gus.heck@gmail.com>:
>
> Except I just finished helping a contributor with a feature that touches
> both and I know for a fact that it was developed for his customer who was
> using solr (payload inequalities)... and have another in the works (the
> AQP).... Not being able to enhance lucene to support a feature in solr is
> an issue IMHO.
>
> On Thu, Feb 25, 2021 at 6:05 PM Mike Drob <mdrob@mdrob.com> wrote:
>
>> It is possible to publish snapshots into the Apache Nexus repository.
>> That said, I think it is a bad idea for Solr to depend on Lucene snapshots
>> because that constrains the ability to do releases. Either you have to wait
>> for a Lucene release and then you can cut over, or you have to figure out
>> what changes you need to roll back.
>>
>> Features today rarely touch both fronts anyway, they usually land in
>> Lucene first and then percolate into Solr. For an easy example, we can see
>> how WAND was developed recently.
>>
>> On Thu, Feb 25, 2021 at 5:02 PM Houston Putman <houstonputman@gmail.com>
>> wrote:
>>
>>> Once the projects are on separate release cadences there wont be an
>>> ability to “add on both fronts” anymore. You will have to add to lucene,
>>> wait for a release, then add to Solr once Solr upgrades its lucene
>>> dependency to that new version. I dont imagine that we are going to keep
>>> Solr master/main, or even 8x, 9x, etc, depending on Lucene snaphsots in
>>> perpetuity. After it becomes possible (when lucene 9.0 is released) we
>>> should only be using released lucene versions as dependencies for every
>>> version branch in Solr.
>>>
>>> On Thu, Feb 25, 2021 at 5:49 PM Gus Heck <gus.heck@gmail.com> wrote:
>>>
>>>> Until the first feature that wants to add something on both fronts...
>>>> Is it possible for Lucene to publish nightly snapshots? I know there is
>>>> some level of support for snapshots in central, though I don't know what
>>>> their usage policies are. If that's too restricted is there an artifact
>>>> repo controlled by the ASF that could be used? (An implementation of Apache
>>>> Archiva?) This would have the added benefit of allowing solr to detect when
>>>> Lucene breaks something before its released.
>>>>
>>>> On Thu, Feb 25, 2021 at 4:50 PM Houston Putman <houstonputman@gmail.com>
>>>> wrote:
>>>>
>>>>> Hey everyone,
>>>>>
>>>>> Currently there is discussion going on, in SOLR-14762
>>>>> <https://issues.apache.org/jira/browse/SOLR-14762>, regarding the
>>>>> split of the lucene-solr repo into individual repos for Solr and Lucene.
>>>>> There seems to be agreement that we shouldn't wait for a Lucene release to
>>>>> do the split, and instead split now and release whenever that happens.
>>>>>
>>>>> The biggest issue that arises there is that Solr's master branch is
>>>>> obviously based on Lucene's master branch, since they are currently the
>>>>> same. So when the split happens, Solr master will have to depend on Lucene
>>>>> 9.0-SNAPSHOT. We can have solr merely depend on the lucene snapshot, but
>>>>> that will result in inconsistent builds, depending on whatever cached
>>>>> dependencies each dev has locally. Personally, I think that will cause a
>>>>> bunch of build errors and headaches for everyone trying to maintain Solr.
>>>>>
>>>>> There is another option though. We could instead do an *alpha*
>>>>> "release" of lucene-solr 9.0 right before the repo is split. Therefore Solr
>>>>> can reliably depend on a stable version of lucene until 9.0 is truly
>>>>> released. (And lucene can use a stable version of Solr, if it sees a need
>>>>> for that). There would be no guarantees for using this alpha release, and
>>>>> we don't have to advertise it at all.
>>>>>
>>>>> It's not perfect, but I think it would be preferable to depending on
>>>>> an ever-changing SNAPSHOT lucene.
>>>>>
>>>>> - Houston
>>>>>
>>>>
>>>>
>>>> --
>>>> http://www.needhamsoftware.com (work)
>>>> http://www.the111shift.com (play)
>>>>
>>>
>
> --
> http://www.needhamsoftware.com (work)
> http://www.the111shift.com (play)
>
>
>

Re: Proposal for the Lucene Dependency after git repo split [ In reply to ]

houstonputman at gmail

Feb 26, 2021, 10:26 AM

Post #9 of 13 (817 views)

Permalink

I don't think Jan's workflow blocks Solr releases on Lucene releases. It
just blocks this one feature branch merge on a Lucene release. Multiple
Solr releases can happen between step 6 and step 7.

I completely agree with that being the workflow going forward with separate
repos, Jan. It will unfortunately be a pain to integrate changes that
affect both Lucene and Solr, but I think that's just a consequence of
splitting the projects.

Neither option gives us everything we want, so here are the pros and cons
in my opinion.

Using a snapshot lucene version
- Easier to make changes to lucene and solr concurrently
- Solr releases are blocked until the snapshot version being depended on is
released.
- Builds may break at any time, and possibly for different sets of users
depending on dependency caches.

Using a released lucene version
- Harder to update lucene and solr concurrently
- Solr can make releases independent of Lucene's release schedule
- Builds are stable and consistent.

Personally I think stability and the ability to own our own release
schedule outweigh the benefits of being able to iterate on both projects
concurrently. But it's definitely something that we should decide on as a
community.

On Fri, Feb 26, 2021 at 12:43 PM Mike Drob <mdrob@mdrob.com> wrote:

> The part of this process that I really don't like is that Solr then still
> becomes beholden to Lucene's release schedule. We don't know how long step
> 7 takes, and will be effectively blocked from making our own releases until
> that happens.
>
> On Fri, Feb 26, 2021 at 8:51 AM Jan Høydahl <jan.asf@cominvent.com> wrote:
>
>> The developer workflow for adding something to both Lucene and Solr would
>> be as any other dependency, right?
>> So say we are on Luene 9.0. This is the process in my head:
>>
>> 1. Adapt Lucene as needed, and "mvn install" lucene-9.1-SNAPSHOT to
>> your local laptop (whatever command that is in gradle)
>> 2. Make your Solr feature branch depend on lucene-9.1-SNAPSHOT
>> instead of lucene-9.0.0 -hopefully Gradle will pick the local version over
>> Apache Nexus version
>> 3. Iterate 1-2 until happy
>> 4. Make a Lucene PR and eventually commit the Lucene change
>> 5. After next Jenkins build the feature is in Apache Nexus snapshot
>> as lucene-9.1-SNAPSHOT
>> 6. Now the Solr Pull Request will compile and can be tested by others
>> 7. Wait until Lucene 9.1 release
>> 8. Upgrade Solr's lucene dependency on 'main'
>> 9. Merge Solr PR
>> 10. Backporting will be a similar process, i.e. first backport and
>> release in Lucene, then backport in Sol
>>
>> Hmm, as I wrote this list I can understand why so many features were
>> added only to Solr and not to Lucene in the early days :)
>>
>> Jan
>>
>> 26. feb. 2021 kl. 14:22 skrev Gus Heck <gus.heck@gmail.com>:
>>
>> Except I just finished helping a contributor with a feature that touches
>> both and I know for a fact that it was developed for his customer who was
>> using solr (payload inequalities)... and have another in the works (the
>> AQP).... Not being able to enhance lucene to support a feature in solr is
>> an issue IMHO.
>>
>> On Thu, Feb 25, 2021 at 6:05 PM Mike Drob <mdrob@mdrob.com> wrote:
>>
>>> It is possible to publish snapshots into the Apache Nexus repository.
>>> That said, I think it is a bad idea for Solr to depend on Lucene snapshots
>>> because that constrains the ability to do releases. Either you have to wait
>>> for a Lucene release and then you can cut over, or you have to figure out
>>> what changes you need to roll back.
>>>
>>> Features today rarely touch both fronts anyway, they usually land in
>>> Lucene first and then percolate into Solr. For an easy example, we can see
>>> how WAND was developed recently.
>>>
>>> On Thu, Feb 25, 2021 at 5:02 PM Houston Putman <houstonputman@gmail.com>
>>> wrote:
>>>
>>>> Once the projects are on separate release cadences there wont be an
>>>> ability to “add on both fronts” anymore. You will have to add to lucene,
>>>> wait for a release, then add to Solr once Solr upgrades its lucene
>>>> dependency to that new version. I dont imagine that we are going to keep
>>>> Solr master/main, or even 8x, 9x, etc, depending on Lucene snaphsots in
>>>> perpetuity. After it becomes possible (when lucene 9.0 is released) we
>>>> should only be using released lucene versions as dependencies for every
>>>> version branch in Solr.
>>>>
>>>> On Thu, Feb 25, 2021 at 5:49 PM Gus Heck <gus.heck@gmail.com> wrote:
>>>>
>>>>> Until the first feature that wants to add something on both fronts...
>>>>> Is it possible for Lucene to publish nightly snapshots? I know there is
>>>>> some level of support for snapshots in central, though I don't know what
>>>>> their usage policies are. If that's too restricted is there an artifact
>>>>> repo controlled by the ASF that could be used? (An implementation of Apache
>>>>> Archiva?) This would have the added benefit of allowing solr to detect when
>>>>> Lucene breaks something before its released.
>>>>>
>>>>> On Thu, Feb 25, 2021 at 4:50 PM Houston Putman <
>>>>> houstonputman@gmail.com> wrote:
>>>>>
>>>>>> Hey everyone,
>>>>>>
>>>>>> Currently there is discussion going on, in SOLR-14762
>>>>>> <https://issues.apache.org/jira/browse/SOLR-14762>, regarding the
>>>>>> split of the lucene-solr repo into individual repos for Solr and Lucene.
>>>>>> There seems to be agreement that we shouldn't wait for a Lucene release to
>>>>>> do the split, and instead split now and release whenever that happens.
>>>>>>
>>>>>> The biggest issue that arises there is that Solr's master branch is
>>>>>> obviously based on Lucene's master branch, since they are currently the
>>>>>> same. So when the split happens, Solr master will have to depend on Lucene
>>>>>> 9.0-SNAPSHOT. We can have solr merely depend on the lucene snapshot, but
>>>>>> that will result in inconsistent builds, depending on whatever cached
>>>>>> dependencies each dev has locally. Personally, I think that will cause a
>>>>>> bunch of build errors and headaches for everyone trying to maintain Solr.
>>>>>>
>>>>>> There is another option though. We could instead do an *alpha*
>>>>>> "release" of lucene-solr 9.0 right before the repo is split. Therefore Solr
>>>>>> can reliably depend on a stable version of lucene until 9.0 is truly
>>>>>> released. (And lucene can use a stable version of Solr, if it sees a need
>>>>>> for that). There would be no guarantees for using this alpha release, and
>>>>>> we don't have to advertise it at all.
>>>>>>
>>>>>> It's not perfect, but I think it would be preferable to depending on
>>>>>> an ever-changing SNAPSHOT lucene.
>>>>>>
>>>>>> - Houston
>>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> http://www.needhamsoftware.com (work)
>>>>> http://www.the111shift.com (play)
>>>>>
>>>>
>>
>> --
>> http://www.needhamsoftware.com (work)
>> http://www.the111shift.com (play)
>>
>>
>>

Re: Proposal for the Lucene Dependency after git repo split [ In reply to ]

epugh at opensourceconnections

Feb 26, 2021, 10:29 AM

Post #10 of 13 (817 views)

Permalink

Plus, isn’t this a reason for folks in the Solr side to continue to be involved in the Lucene project? It’s the inverse of the days when folks wanted to cut releases of Lucene, and were waiting for Solr to be ready!

> On Feb 26, 2021, at 1:26 PM, Houston Putman <houstonputman@gmail.com> wrote:
>
> I don't think Jan's workflow blocks Solr releases on Lucene releases. It just blocks this one feature branch merge on a Lucene release. Multiple Solr releases can happen between step 6 and step 7.
>
> I completely agree with that being the workflow going forward with separate repos, Jan. It will unfortunately be a pain to integrate changes that affect both Lucene and Solr, but I think that's just a consequence of splitting the projects.
>
> Neither option gives us everything we want, so here are the pros and cons in my opinion.
>
> Using a snapshot lucene version
> - Easier to make changes to lucene and solr concurrently
> - Solr releases are blocked until the snapshot version being depended on is released.
> - Builds may break at any time, and possibly for different sets of users depending on dependency caches.
>
> Using a released lucene version
> - Harder to update lucene and solr concurrently
> - Solr can make releases independent of Lucene's release schedule
> - Builds are stable and consistent.
>
> Personally I think stability and the ability to own our own release schedule outweigh the benefits of being able to iterate on both projects concurrently. But it's definitely something that we should decide on as a community.
>
> On Fri, Feb 26, 2021 at 12:43 PM Mike Drob <mdrob@mdrob.com <mailto:mdrob@mdrob.com>> wrote:
> The part of this process that I really don't like is that Solr then still becomes beholden to Lucene's release schedule. We don't know how long step 7 takes, and will be effectively blocked from making our own releases until that happens.
>
> On Fri, Feb 26, 2021 at 8:51 AM Jan Høydahl <jan.asf@cominvent.com <mailto:jan.asf@cominvent.com>> wrote:
> The developer workflow for adding something to both Lucene and Solr would be as any other dependency, right?
> So say we are on Luene 9.0. This is the process in my head:
> Adapt Lucene as needed, and "mvn install" lucene-9.1-SNAPSHOT to your local laptop (whatever command that is in gradle)
> Make your Solr feature branch depend on lucene-9.1-SNAPSHOT instead of lucene-9.0.0 -hopefully Gradle will pick the local version over Apache Nexus version
> Iterate 1-2 until happy
> Make a Lucene PR and eventually commit the Lucene change
> After next Jenkins build the feature is in Apache Nexus snapshot as lucene-9.1-SNAPSHOT
> Now the Solr Pull Request will compile and can be tested by others
> Wait until Lucene 9.1 release
> Upgrade Solr's lucene dependency on 'main'
> Merge Solr PR
> Backporting will be a similar process, i.e. first backport and release in Lucene, then backport in Sol
> Hmm, as I wrote this list I can understand why so many features were added only to Solr and not to Lucene in the early days :)
>
> Jan
>
>> 26. feb. 2021 kl. 14:22 skrev Gus Heck <gus.heck@gmail.com <mailto:gus.heck@gmail.com>>:
>>
>> Except I just finished helping a contributor with a feature that touches both and I know for a fact that it was developed for his customer who was using solr (payload inequalities)... and have another in the works (the AQP).... Not being able to enhance lucene to support a feature in solr is an issue IMHO.
>>
>> On Thu, Feb 25, 2021 at 6:05 PM Mike Drob <mdrob@mdrob.com <mailto:mdrob@mdrob.com>> wrote:
>> It is possible to publish snapshots into the Apache Nexus repository. That said, I think it is a bad idea for Solr to depend on Lucene snapshots because that constrains the ability to do releases. Either you have to wait for a Lucene release and then you can cut over, or you have to figure out what changes you need to roll back.
>>
>> Features today rarely touch both fronts anyway, they usually land in Lucene first and then percolate into Solr. For an easy example, we can see how WAND was developed recently.
>>
>> On Thu, Feb 25, 2021 at 5:02 PM Houston Putman <houstonputman@gmail.com <mailto:houstonputman@gmail.com>> wrote:
>> Once the projects are on separate release cadences there wont be an ability to “add on both fronts” anymore. You will have to add to lucene, wait for a release, then add to Solr once Solr upgrades its lucene dependency to that new version. I dont imagine that we are going to keep Solr master/main, or even 8x, 9x, etc, depending on Lucene snaphsots in perpetuity. After it becomes possible (when lucene 9.0 is released) we should only be using released lucene versions as dependencies for every version branch in Solr.
>>
>> On Thu, Feb 25, 2021 at 5:49 PM Gus Heck <gus.heck@gmail.com <mailto:gus.heck@gmail.com>> wrote:
>> Until the first feature that wants to add something on both fronts... Is it possible for Lucene to publish nightly snapshots? I know there is some level of support for snapshots in central, though I don't know what their usage policies are. If that's too restricted is there an artifact repo controlled by the ASF that could be used? (An implementation of Apache Archiva?) This would have the added benefit of allowing solr to detect when Lucene breaks something before its released.
>>
>> On Thu, Feb 25, 2021 at 4:50 PM Houston Putman <houstonputman@gmail.com <mailto:houstonputman@gmail.com>> wrote:
>> Hey everyone,
>>
>> Currently there is discussion going on, in SOLR-14762 <https://issues.apache.org/jira/browse/SOLR-14762>, regarding the split of the lucene-solr repo into individual repos for Solr and Lucene. There seems to be agreement that we shouldn't wait for a Lucene release to do the split, and instead split now and release whenever that happens.
>>
>> The biggest issue that arises there is that Solr's master branch is obviously based on Lucene's master branch, since they are currently the same. So when the split happens, Solr master will have to depend on Lucene 9.0-SNAPSHOT. We can have solr merely depend on the lucene snapshot, but that will result in inconsistent builds, depending on whatever cached dependencies each dev has locally. Personally, I think that will cause a bunch of build errors and headaches for everyone trying to maintain Solr.
>>
>> There is another option though. We could instead do an alpha "release" of lucene-solr 9.0 right before the repo is split. Therefore Solr can reliably depend on a stable version of lucene until 9.0 is truly released. (And lucene can use a stable version of Solr, if it sees a need for that). There would be no guarantees for using this alpha release, and we don't have to advertise it at all.
>>
>> It's not perfect, but I think it would be preferable to depending on an ever-changing SNAPSHOT lucene.
>>
>> - Houston
>>
>>
>> --
>> http://www.needhamsoftware.com <http://www.needhamsoftware.com/> (work)
>> http://www.the111shift.com <http://www.the111shift.com/> (play)
>>
>>
>> --
>> http://www.needhamsoftware.com <http://www.needhamsoftware.com/> (work)
>> http://www.the111shift.com <http://www.the111shift.com/> (play)
>

_______________________
Eric Pugh | Founder & CEO | OpenSource Connections, LLC | 434.466.1467 | http://www.opensourceconnections.com <http://www.opensourceconnections.com/> | My Free/Busy <http://tinyurl.com/eric-cal>
Co-Author: Apache Solr Enterprise Search Server, 3rd Ed <https://www.packtpub.com/big-data-and-business-intelligence/apache-solr-enterprise-search-server-third-edition-raw>
This e-mail and all contents, including attachments, is considered to be Company Confidential unless explicitly stated otherwise, regardless of whether attachments are marked as such.

Re: Proposal for the Lucene Dependency after git repo split [ In reply to ]

jpountz at gmail

Feb 26, 2021, 1:32 PM

Post #11 of 13 (817 views)

Permalink

FYI Elasticsearch has been regularly depending on builds of specific
commits of Lucene for this case of features that need changes both in
Lucene and Elasticsearch.

The workflow usually looks like this:
- Do work in Lucene.
- When it becomes clear that the next release of Lucene should happen
before the next feature freeze of Elasticsearch, we do a new build of
Lucene and upgrade Elasticsearch to it.
- Do work in Elasticsearch.
- When a new Lucene release is out, upgrade Elasticsearch to this Lucene
release.

We have done this dozens of times and it has worked well for us. We do the
same when a vote for a new Lucene release is about to start in order to
check whether it breaks anything in Elasticsearch.

The rest of the time (which is most of the time) Elasticsearch depends on
an actual Lucene release.

Le ven. 26 févr. 2021 à 19:29, Eric Pugh <epugh@opensourceconnections.com>
a écrit :

> Plus, isn’t this a reason for folks in the Solr side to continue to be
> involved in the Lucene project? It’s the inverse of the days when folks
> wanted to cut releases of Lucene, and were waiting for Solr to be ready!
>
>
> On Feb 26, 2021, at 1:26 PM, Houston Putman <houstonputman@gmail.com>
> wrote:
>
> I don't think Jan's workflow blocks Solr releases on Lucene releases. It
> just blocks this one feature branch merge on a Lucene release. Multiple
> Solr releases can happen between step 6 and step 7.
>
> I completely agree with that being the workflow going forward with
> separate repos, Jan. It will unfortunately be a pain to integrate changes
> that affect both Lucene and Solr, but I think that's just a consequence of
> splitting the projects.
>
> Neither option gives us everything we want, so here are the pros and cons
> in my opinion.
>
> Using a snapshot lucene version
> - Easier to make changes to lucene and solr concurrently
> - Solr releases are blocked until the snapshot version being depended on
> is released.
> - Builds may break at any time, and possibly for different sets of users
> depending on dependency caches.
>
> Using a released lucene version
> - Harder to update lucene and solr concurrently
> - Solr can make releases independent of Lucene's release schedule
> - Builds are stable and consistent.
>
> Personally I think stability and the ability to own our own release
> schedule outweigh the benefits of being able to iterate on both projects
> concurrently. But it's definitely something that we should decide on as a
> community.
>
> On Fri, Feb 26, 2021 at 12:43 PM Mike Drob <mdrob@mdrob.com> wrote:
>
>> The part of this process that I really don't like is that Solr then still
>> becomes beholden to Lucene's release schedule. We don't know how long step
>> 7 takes, and will be effectively blocked from making our own releases until
>> that happens.
>>
>> On Fri, Feb 26, 2021 at 8:51 AM Jan Høydahl <jan.asf@cominvent.com>
>> wrote:
>>
>>> The developer workflow for adding something to both Lucene and Solr
>>> would be as any other dependency, right?
>>> So say we are on Luene 9.0. This is the process in my head:
>>>
>>> 1. Adapt Lucene as needed, and "mvn install" lucene-9.1-SNAPSHOT to
>>> your local laptop (whatever command that is in gradle)
>>> 2. Make your Solr feature branch depend on lucene-9.1-SNAPSHOT
>>> instead of lucene-9.0.0 -hopefully Gradle will pick the local version over
>>> Apache Nexus version
>>> 3. Iterate 1-2 until happy
>>> 4. Make a Lucene PR and eventually commit the Lucene change
>>> 5. After next Jenkins build the feature is in Apache Nexus snapshot
>>> as lucene-9.1-SNAPSHOT
>>> 6. Now the Solr Pull Request will compile and can be tested by others
>>> 7. Wait until Lucene 9.1 release
>>> 8. Upgrade Solr's lucene dependency on 'main'
>>> 9. Merge Solr PR
>>> 10. Backporting will be a similar process, i.e. first backport and
>>> release in Lucene, then backport in Sol
>>>
>>> Hmm, as I wrote this list I can understand why so many features were
>>> added only to Solr and not to Lucene in the early days :)
>>>
>>> Jan
>>>
>>> 26. feb. 2021 kl. 14:22 skrev Gus Heck <gus.heck@gmail.com>:
>>>
>>> Except I just finished helping a contributor with a feature that touches
>>> both and I know for a fact that it was developed for his customer who was
>>> using solr (payload inequalities)... and have another in the works (the
>>> AQP).... Not being able to enhance lucene to support a feature in solr is
>>> an issue IMHO.
>>>
>>> On Thu, Feb 25, 2021 at 6:05 PM Mike Drob <mdrob@mdrob.com> wrote:
>>>
>>>> It is possible to publish snapshots into the Apache Nexus repository.
>>>> That said, I think it is a bad idea for Solr to depend on Lucene snapshots
>>>> because that constrains the ability to do releases. Either you have to wait
>>>> for a Lucene release and then you can cut over, or you have to figure out
>>>> what changes you need to roll back.
>>>>
>>>> Features today rarely touch both fronts anyway, they usually land in
>>>> Lucene first and then percolate into Solr. For an easy example, we can see
>>>> how WAND was developed recently.
>>>>
>>>> On Thu, Feb 25, 2021 at 5:02 PM Houston Putman <houstonputman@gmail.com>
>>>> wrote:
>>>>
>>>>> Once the projects are on separate release cadences there wont be an
>>>>> ability to “add on both fronts” anymore. You will have to add to lucene,
>>>>> wait for a release, then add to Solr once Solr upgrades its lucene
>>>>> dependency to that new version. I dont imagine that we are going to keep
>>>>> Solr master/main, or even 8x, 9x, etc, depending on Lucene snaphsots in
>>>>> perpetuity. After it becomes possible (when lucene 9.0 is released) we
>>>>> should only be using released lucene versions as dependencies for every
>>>>> version branch in Solr.
>>>>>
>>>>> On Thu, Feb 25, 2021 at 5:49 PM Gus Heck <gus.heck@gmail.com> wrote:
>>>>>
>>>>>> Until the first feature that wants to add something on both fronts...
>>>>>> Is it possible for Lucene to publish nightly snapshots? I know there is
>>>>>> some level of support for snapshots in central, though I don't know what
>>>>>> their usage policies are. If that's too restricted is there an artifact
>>>>>> repo controlled by the ASF that could be used? (An implementation of Apache
>>>>>> Archiva?) This would have the added benefit of allowing solr to detect when
>>>>>> Lucene breaks something before its released.
>>>>>>
>>>>>> On Thu, Feb 25, 2021 at 4:50 PM Houston Putman <
>>>>>> houstonputman@gmail.com> wrote:
>>>>>>
>>>>>>> Hey everyone,
>>>>>>>
>>>>>>> Currently there is discussion going on, in SOLR-14762
>>>>>>> <https://issues.apache.org/jira/browse/SOLR-14762>, regarding the
>>>>>>> split of the lucene-solr repo into individual repos for Solr and Lucene.
>>>>>>> There seems to be agreement that we shouldn't wait for a Lucene release to
>>>>>>> do the split, and instead split now and release whenever that happens.
>>>>>>>
>>>>>>> The biggest issue that arises there is that Solr's master branch is
>>>>>>> obviously based on Lucene's master branch, since they are currently the
>>>>>>> same. So when the split happens, Solr master will have to depend on Lucene
>>>>>>> 9.0-SNAPSHOT. We can have solr merely depend on the lucene snapshot, but
>>>>>>> that will result in inconsistent builds, depending on whatever cached
>>>>>>> dependencies each dev has locally. Personally, I think that will cause a
>>>>>>> bunch of build errors and headaches for everyone trying to maintain Solr.
>>>>>>>
>>>>>>> There is another option though. We could instead do an *alpha*
>>>>>>> "release" of lucene-solr 9.0 right before the repo is split. Therefore Solr
>>>>>>> can reliably depend on a stable version of lucene until 9.0 is truly
>>>>>>> released. (And lucene can use a stable version of Solr, if it sees a need
>>>>>>> for that). There would be no guarantees for using this alpha release, and
>>>>>>> we don't have to advertise it at all.
>>>>>>>
>>>>>>> It's not perfect, but I think it would be preferable to depending on
>>>>>>> an ever-changing SNAPSHOT lucene.
>>>>>>>
>>>>>>> - Houston
>>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> http://www.needhamsoftware.com (work)
>>>>>> http://www.the111shift.com (play)
>>>>>>
>>>>>
>>>
>>> --
>>> http://www.needhamsoftware.com (work)
>>> http://www.the111shift.com (play)
>>>
>>>
>>>
> _______________________
> *Eric Pugh **| *Founder & CEO | OpenSource Connections, LLC | 434.466.1467
> | http://www.opensourceconnections.com | My Free/Busy
> <http://tinyurl.com/eric-cal>
> Co-Author: Apache Solr Enterprise Search Server, 3rd Ed
> <https://www.packtpub.com/big-data-and-business-intelligence/apache-solr-enterprise-search-server-third-edition-raw>
> This e-mail and all contents, including attachments, is considered to be
> Company Confidential unless explicitly stated otherwise, regardless
> of whether attachments are marked as such.
>
>

Re: Proposal for the Lucene Dependency after git repo split [ In reply to ]

ichattopadhyaya at gmail

Feb 26, 2021, 7:28 PM

Post #12 of 13 (817 views)

Permalink

I would prefer Lucene as a git submodule inside Solr. That way, I can work
with solr and Lucene together easily. The commit to which Lucene is pegged
at could be from a release branch or a tag.

On Sat, 27 Feb, 2021, 3:03 am Adrien Grand, <jpountz@gmail.com> wrote:

> FYI Elasticsearch has been regularly depending on builds of specific
> commits of Lucene for this case of features that need changes both in
> Lucene and Elasticsearch.
>
> The workflow usually looks like this:
> - Do work in Lucene.
> - When it becomes clear that the next release of Lucene should happen
> before the next feature freeze of Elasticsearch, we do a new build of
> Lucene and upgrade Elasticsearch to it.
> - Do work in Elasticsearch.
> - When a new Lucene release is out, upgrade Elasticsearch to this Lucene
> release.
>
> We have done this dozens of times and it has worked well for us. We do the
> same when a vote for a new Lucene release is about to start in order to
> check whether it breaks anything in Elasticsearch.
>
> The rest of the time (which is most of the time) Elasticsearch depends on
> an actual Lucene release.
>
> Le ven. 26 févr. 2021 à 19:29, Eric Pugh <epugh@opensourceconnections.com>
> a écrit :
>
>> Plus, isn’t this a reason for folks in the Solr side to continue to be
>> involved in the Lucene project? It’s the inverse of the days when folks
>> wanted to cut releases of Lucene, and were waiting for Solr to be ready!
>>
>>
>> On Feb 26, 2021, at 1:26 PM, Houston Putman <houstonputman@gmail.com>
>> wrote:
>>
>> I don't think Jan's workflow blocks Solr releases on Lucene releases. It
>> just blocks this one feature branch merge on a Lucene release. Multiple
>> Solr releases can happen between step 6 and step 7.
>>
>> I completely agree with that being the workflow going forward with
>> separate repos, Jan. It will unfortunately be a pain to integrate changes
>> that affect both Lucene and Solr, but I think that's just a consequence of
>> splitting the projects.
>>
>> Neither option gives us everything we want, so here are the pros and cons
>> in my opinion.
>>
>> Using a snapshot lucene version
>> - Easier to make changes to lucene and solr concurrently
>> - Solr releases are blocked until the snapshot version being depended on
>> is released.
>> - Builds may break at any time, and possibly for different sets of users
>> depending on dependency caches.
>>
>> Using a released lucene version
>> - Harder to update lucene and solr concurrently
>> - Solr can make releases independent of Lucene's release schedule
>> - Builds are stable and consistent.
>>
>> Personally I think stability and the ability to own our own release
>> schedule outweigh the benefits of being able to iterate on both projects
>> concurrently. But it's definitely something that we should decide on as a
>> community.
>>
>> On Fri, Feb 26, 2021 at 12:43 PM Mike Drob <mdrob@mdrob.com> wrote:
>>
>>> The part of this process that I really don't like is that Solr then
>>> still becomes beholden to Lucene's release schedule. We don't know how long
>>> step 7 takes, and will be effectively blocked from making our own releases
>>> until that happens.
>>>
>>> On Fri, Feb 26, 2021 at 8:51 AM Jan Høydahl <jan.asf@cominvent.com>
>>> wrote:
>>>
>>>> The developer workflow for adding something to both Lucene and Solr
>>>> would be as any other dependency, right?
>>>> So say we are on Luene 9.0. This is the process in my head:
>>>>
>>>> 1. Adapt Lucene as needed, and "mvn install" lucene-9.1-SNAPSHOT to
>>>> your local laptop (whatever command that is in gradle)
>>>> 2. Make your Solr feature branch depend on lucene-9.1-SNAPSHOT
>>>> instead of lucene-9.0.0 -hopefully Gradle will pick the local version over
>>>> Apache Nexus version
>>>> 3. Iterate 1-2 until happy
>>>> 4. Make a Lucene PR and eventually commit the Lucene change
>>>> 5. After next Jenkins build the feature is in Apache Nexus snapshot
>>>> as lucene-9.1-SNAPSHOT
>>>> 6. Now the Solr Pull Request will compile and can be tested by
>>>> others
>>>> 7. Wait until Lucene 9.1 release
>>>> 8. Upgrade Solr's lucene dependency on 'main'
>>>> 9. Merge Solr PR
>>>> 10. Backporting will be a similar process, i.e. first backport and
>>>> release in Lucene, then backport in Sol
>>>>
>>>> Hmm, as I wrote this list I can understand why so many features were
>>>> added only to Solr and not to Lucene in the early days :)
>>>>
>>>> Jan
>>>>
>>>> 26. feb. 2021 kl. 14:22 skrev Gus Heck <gus.heck@gmail.com>:
>>>>
>>>> Except I just finished helping a contributor with a feature that
>>>> touches both and I know for a fact that it was developed for his customer
>>>> who was using solr (payload inequalities)... and have another in the works
>>>> (the AQP).... Not being able to enhance lucene to support a feature in solr
>>>> is an issue IMHO.
>>>>
>>>> On Thu, Feb 25, 2021 at 6:05 PM Mike Drob <mdrob@mdrob.com> wrote:
>>>>
>>>>> It is possible to publish snapshots into the Apache Nexus repository.
>>>>> That said, I think it is a bad idea for Solr to depend on Lucene snapshots
>>>>> because that constrains the ability to do releases. Either you have to wait
>>>>> for a Lucene release and then you can cut over, or you have to figure out
>>>>> what changes you need to roll back.
>>>>>
>>>>> Features today rarely touch both fronts anyway, they usually land in
>>>>> Lucene first and then percolate into Solr. For an easy example, we can see
>>>>> how WAND was developed recently.
>>>>>
>>>>> On Thu, Feb 25, 2021 at 5:02 PM Houston Putman <
>>>>> houstonputman@gmail.com> wrote:
>>>>>
>>>>>> Once the projects are on separate release cadences there wont be an
>>>>>> ability to “add on both fronts” anymore. You will have to add to lucene,
>>>>>> wait for a release, then add to Solr once Solr upgrades its lucene
>>>>>> dependency to that new version. I dont imagine that we are going to keep
>>>>>> Solr master/main, or even 8x, 9x, etc, depending on Lucene snaphsots in
>>>>>> perpetuity. After it becomes possible (when lucene 9.0 is released) we
>>>>>> should only be using released lucene versions as dependencies for every
>>>>>> version branch in Solr.
>>>>>>
>>>>>> On Thu, Feb 25, 2021 at 5:49 PM Gus Heck <gus.heck@gmail.com> wrote:
>>>>>>
>>>>>>> Until the first feature that wants to add something on both
>>>>>>> fronts... Is it possible for Lucene to publish nightly snapshots? I know
>>>>>>> there is some level of support for snapshots in central, though I don't
>>>>>>> know what their usage policies are. If that's too restricted is there an
>>>>>>> artifact repo controlled by the ASF that could be used? (An implementation
>>>>>>> of Apache Archiva?) This would have the added benefit of allowing solr to
>>>>>>> detect when Lucene breaks something before its released.
>>>>>>>
>>>>>>> On Thu, Feb 25, 2021 at 4:50 PM Houston Putman <
>>>>>>> houstonputman@gmail.com> wrote:
>>>>>>>
>>>>>>>> Hey everyone,
>>>>>>>>
>>>>>>>> Currently there is discussion going on, in SOLR-14762
>>>>>>>> <https://issues.apache.org/jira/browse/SOLR-14762>, regarding the
>>>>>>>> split of the lucene-solr repo into individual repos for Solr and Lucene.
>>>>>>>> There seems to be agreement that we shouldn't wait for a Lucene release to
>>>>>>>> do the split, and instead split now and release whenever that happens.
>>>>>>>>
>>>>>>>> The biggest issue that arises there is that Solr's master branch is
>>>>>>>> obviously based on Lucene's master branch, since they are currently the
>>>>>>>> same. So when the split happens, Solr master will have to depend on Lucene
>>>>>>>> 9.0-SNAPSHOT. We can have solr merely depend on the lucene snapshot, but
>>>>>>>> that will result in inconsistent builds, depending on whatever cached
>>>>>>>> dependencies each dev has locally. Personally, I think that will cause a
>>>>>>>> bunch of build errors and headaches for everyone trying to maintain Solr.
>>>>>>>>
>>>>>>>> There is another option though. We could instead do an *alpha*
>>>>>>>> "release" of lucene-solr 9.0 right before the repo is split. Therefore Solr
>>>>>>>> can reliably depend on a stable version of lucene until 9.0 is truly
>>>>>>>> released. (And lucene can use a stable version of Solr, if it sees a need
>>>>>>>> for that). There would be no guarantees for using this alpha release, and
>>>>>>>> we don't have to advertise it at all.
>>>>>>>>
>>>>>>>> It's not perfect, but I think it would be preferable to depending
>>>>>>>> on an ever-changing SNAPSHOT lucene.
>>>>>>>>
>>>>>>>> - Houston
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> http://www.needhamsoftware.com (work)
>>>>>>> http://www.the111shift.com (play)
>>>>>>>
>>>>>>
>>>>
>>>> --
>>>> http://www.needhamsoftware.com (work)
>>>> http://www.the111shift.com (play)
>>>>
>>>>
>>>>
>> _______________________
>> *Eric Pugh **| *Founder & CEO | OpenSource Connections, LLC | 434.466.1467
>> | http://www.opensourceconnections.com | My Free/Busy
>> <http://tinyurl.com/eric-cal>
>> Co-Author: Apache Solr Enterprise Search Server, 3rd Ed
>> <https://www.packtpub.com/big-data-and-business-intelligence/apache-solr-enterprise-search-server-third-edition-raw>
>> This e-mail and all contents, including attachments, is considered to be
>> Company Confidential unless explicitly stated otherwise, regardless
>> of whether attachments are marked as such.
>>
>>

Re: Proposal for the Lucene Dependency after git repo split [ In reply to ]

jan.asf at cominvent

Feb 27, 2021, 2:58 AM

Post #13 of 13 (815 views)

Permalink

Please, no sub modules! It is honestly a mess. And besides, for 95% of Solr development work there are no Lucene changes, so having to compile Lucene every time is not logical.
I suppose you would be able to do some surgery on your local setup though, removing lucene dep from gradle and instead adding your Lucene checkout as a module dependency in your IDE?

Jan

> 27. feb. 2021 kl. 04:28 skrev Ishan Chattopadhyaya <ichattopadhyaya@gmail.com>:
>
> I would prefer Lucene as a git submodule inside Solr. That way, I can work with solr and Lucene together easily. The commit to which Lucene is pegged at could be from a release branch or a tag.
>
> On Sat, 27 Feb, 2021, 3:03 am Adrien Grand, <jpountz@gmail.com <mailto:jpountz@gmail.com>> wrote:
> FYI Elasticsearch has been regularly depending on builds of specific commits of Lucene for this case of features that need changes both in Lucene and Elasticsearch.
>
> The workflow usually looks like this:
> - Do work in Lucene.
> - When it becomes clear that the next release of Lucene should happen before the next feature freeze of Elasticsearch, we do a new build of Lucene and upgrade Elasticsearch to it.
> - Do work in Elasticsearch.
> - When a new Lucene release is out, upgrade Elasticsearch to this Lucene release.
>
> We have done this dozens of times and it has worked well for us. We do the same when a vote for a new Lucene release is about to start in order to check whether it breaks anything in Elasticsearch.
>
> The rest of the time (which is most of the time) Elasticsearch depends on an actual Lucene release.
>
> Le ven. 26 févr. 2021 à 19:29, Eric Pugh <epugh@opensourceconnections.com <mailto:epugh@opensourceconnections.com>> a écrit :
> Plus, isn’t this a reason for folks in the Solr side to continue to be involved in the Lucene project? It’s the inverse of the days when folks wanted to cut releases of Lucene, and were waiting for Solr to be ready!
>
>
>> On Feb 26, 2021, at 1:26 PM, Houston Putman <houstonputman@gmail.com <mailto:houstonputman@gmail.com>> wrote:
>>
>> I don't think Jan's workflow blocks Solr releases on Lucene releases. It just blocks this one feature branch merge on a Lucene release. Multiple Solr releases can happen between step 6 and step 7.
>>
>> I completely agree with that being the workflow going forward with separate repos, Jan. It will unfortunately be a pain to integrate changes that affect both Lucene and Solr, but I think that's just a consequence of splitting the projects.
>>
>> Neither option gives us everything we want, so here are the pros and cons in my opinion.
>>
>> Using a snapshot lucene version
>> - Easier to make changes to lucene and solr concurrently
>> - Solr releases are blocked until the snapshot version being depended on is released.
>> - Builds may break at any time, and possibly for different sets of users depending on dependency caches.
>>
>> Using a released lucene version
>> - Harder to update lucene and solr concurrently
>> - Solr can make releases independent of Lucene's release schedule
>> - Builds are stable and consistent.
>>
>> Personally I think stability and the ability to own our own release schedule outweigh the benefits of being able to iterate on both projects concurrently. But it's definitely something that we should decide on as a community.
>>
>> On Fri, Feb 26, 2021 at 12:43 PM Mike Drob <mdrob@mdrob.com <mailto:mdrob@mdrob.com>> wrote:
>> The part of this process that I really don't like is that Solr then still becomes beholden to Lucene's release schedule. We don't know how long step 7 takes, and will be effectively blocked from making our own releases until that happens.
>>
>> On Fri, Feb 26, 2021 at 8:51 AM Jan Høydahl <jan.asf@cominvent.com <mailto:jan.asf@cominvent.com>> wrote:
>> The developer workflow for adding something to both Lucene and Solr would be as any other dependency, right?
>> So say we are on Luene 9.0. This is the process in my head:
>> Adapt Lucene as needed, and "mvn install" lucene-9.1-SNAPSHOT to your local laptop (whatever command that is in gradle)
>> Make your Solr feature branch depend on lucene-9.1-SNAPSHOT instead of lucene-9.0.0 -hopefully Gradle will pick the local version over Apache Nexus version
>> Iterate 1-2 until happy
>> Make a Lucene PR and eventually commit the Lucene change
>> After next Jenkins build the feature is in Apache Nexus snapshot as lucene-9.1-SNAPSHOT
>> Now the Solr Pull Request will compile and can be tested by others
>> Wait until Lucene 9.1 release
>> Upgrade Solr's lucene dependency on 'main'
>> Merge Solr PR
>> Backporting will be a similar process, i.e. first backport and release in Lucene, then backport in Sol
>> Hmm, as I wrote this list I can understand why so many features were added only to Solr and not to Lucene in the early days :)
>>
>> Jan
>>
>>> 26. feb. 2021 kl. 14:22 skrev Gus Heck <gus.heck@gmail.com <mailto:gus.heck@gmail.com>>:
>>>
>>> Except I just finished helping a contributor with a feature that touches both and I know for a fact that it was developed for his customer who was using solr (payload inequalities)... and have another in the works (the AQP).... Not being able to enhance lucene to support a feature in solr is an issue IMHO.
>>>
>>> On Thu, Feb 25, 2021 at 6:05 PM Mike Drob <mdrob@mdrob.com <mailto:mdrob@mdrob.com>> wrote:
>>> It is possible to publish snapshots into the Apache Nexus repository. That said, I think it is a bad idea for Solr to depend on Lucene snapshots because that constrains the ability to do releases. Either you have to wait for a Lucene release and then you can cut over, or you have to figure out what changes you need to roll back.
>>>
>>> Features today rarely touch both fronts anyway, they usually land in Lucene first and then percolate into Solr. For an easy example, we can see how WAND was developed recently.
>>>
>>> On Thu, Feb 25, 2021 at 5:02 PM Houston Putman <houstonputman@gmail.com <mailto:houstonputman@gmail.com>> wrote:
>>> Once the projects are on separate release cadences there wont be an ability to “add on both fronts” anymore. You will have to add to lucene, wait for a release, then add to Solr once Solr upgrades its lucene dependency to that new version. I dont imagine that we are going to keep Solr master/main, or even 8x, 9x, etc, depending on Lucene snaphsots in perpetuity. After it becomes possible (when lucene 9.0 is released) we should only be using released lucene versions as dependencies for every version branch in Solr.
>>>
>>> On Thu, Feb 25, 2021 at 5:49 PM Gus Heck <gus.heck@gmail.com <mailto:gus.heck@gmail.com>> wrote:
>>> Until the first feature that wants to add something on both fronts... Is it possible for Lucene to publish nightly snapshots? I know there is some level of support for snapshots in central, though I don't know what their usage policies are. If that's too restricted is there an artifact repo controlled by the ASF that could be used? (An implementation of Apache Archiva?) This would have the added benefit of allowing solr to detect when Lucene breaks something before its released.
>>>
>>> On Thu, Feb 25, 2021 at 4:50 PM Houston Putman <houstonputman@gmail.com <mailto:houstonputman@gmail.com>> wrote:
>>> Hey everyone,
>>>
>>> Currently there is discussion going on, in SOLR-14762 <https://issues.apache.org/jira/browse/SOLR-14762>, regarding the split of the lucene-solr repo into individual repos for Solr and Lucene. There seems to be agreement that we shouldn't wait for a Lucene release to do the split, and instead split now and release whenever that happens.
>>>
>>> The biggest issue that arises there is that Solr's master branch is obviously based on Lucene's master branch, since they are currently the same. So when the split happens, Solr master will have to depend on Lucene 9.0-SNAPSHOT. We can have solr merely depend on the lucene snapshot, but that will result in inconsistent builds, depending on whatever cached dependencies each dev has locally. Personally, I think that will cause a bunch of build errors and headaches for everyone trying to maintain Solr.
>>>
>>> There is another option though. We could instead do an alpha "release" of lucene-solr 9.0 right before the repo is split. Therefore Solr can reliably depend on a stable version of lucene until 9.0 is truly released. (And lucene can use a stable version of Solr, if it sees a need for that). There would be no guarantees for using this alpha release, and we don't have to advertise it at all.
>>>
>>> It's not perfect, but I think it would be preferable to depending on an ever-changing SNAPSHOT lucene.
>>>
>>> - Houston
>>>
>>>
>>> --
>>> http://www.needhamsoftware.com <http://www.needhamsoftware.com/> (work)
>>> http://www.the111shift.com <http://www.the111shift.com/> (play)
>>>
>>>
>>> --
>>> http://www.needhamsoftware.com <http://www.needhamsoftware.com/> (work)
>>> http://www.the111shift.com <http://www.the111shift.com/> (play)
>>
>
> _______________________
> Eric Pugh | Founder & CEO | OpenSource Connections, LLC | 434.466.1467 | http://www.opensourceconnections.com <http://www.opensourceconnections.com/> | My Free/Busy <http://tinyurl.com/eric-cal>
> Co-Author: Apache Solr Enterprise Search Server, 3rd Ed <https://www.packtpub.com/big-data-and-business-intelligence/apache-solr-enterprise-search-server-third-edition-raw>
> This e-mail and all contents, including attachments, is considered to be Company Confidential unless explicitly stated otherwise, regardless of whether attachments are marked as such.
>