Mailing List Archive

Separate git repo(s) for Solr modules
Hi Devs,

As we discussed over the last few months, there seems a need to move
non-core pieces away from the Solr core module. The contribs are presently
a good place, but it makes sense to have a separate git repository hosting
such modules. Some candidates that come to mind are the present day contrib
modules, upcoming HDFS support module (separated away from solr-core),
other first party packages. Along with that, there is also a need for a
repository for hosting WIP modules/sub-projects.

I propose that we apply for the creation of two new git repositories:
1. solr-extras (or lucene-solr-extras)
2. solr-sandbox (or lucene-solr-sandbox)

Well tested, well supported modules/sub-projects can be released straight
away from *solr-extras*. The first party packages can be built from this
location and shipped with Solr (or be available for install using the
package manager CLI).

New, unproved, beta, unstable modules can be hosted on *solr-sandbox* (and
graduate to solr-extras once stable).

Please let me know if there are any questions/concerns with this approach.

Thanks and regards,
Ishan
Re: Separate git repo(s) for Solr modules [ In reply to ]
I think it is a good idea to have solr-extras and solr-sandbox. However, I
think it's fine if some projects in solr-extras need to be migrated into
their own repos at some point.

I don't necessarily agree with moving all contrib modules out of the main
repo however. I think it makes the most sense to leave the "first-party"
plugin contribs inside the repo, so that they are released at the same
cadence as Solr itself, and compatibility is guaranteed whenever changes
are made to Solr core. Anything not in the main repo or released in the
same cadence as Solr is guaranteed to become stale with time. And if we do
have "first-party" plugins, then it stands to reason that those are
supposed to be supported on day-1 of a Solr release.

Some contrib modules aren't plugins, such as the Prometheus Exporter, so
this would make sense to be moved to something like solr-extras.

- Houston

On Wed, Jan 13, 2021 at 10:30 AM Ishan Chattopadhyaya <
ichattopadhyaya@gmail.com> wrote:

> Hi Devs,
>
> As we discussed over the last few months, there seems a need to move
> non-core pieces away from the Solr core module. The contribs are presently
> a good place, but it makes sense to have a separate git repository hosting
> such modules. Some candidates that come to mind are the present day contrib
> modules, upcoming HDFS support module (separated away from solr-core),
> other first party packages. Along with that, there is also a need for a
> repository for hosting WIP modules/sub-projects.
>
> I propose that we apply for the creation of two new git repositories:
> 1. solr-extras (or lucene-solr-extras)
> 2. solr-sandbox (or lucene-solr-sandbox)
>
> Well tested, well supported modules/sub-projects can be released straight
> away from *solr-extras*. The first party packages can be built from this
> location and shipped with Solr (or be available for install using the
> package manager CLI).
>
> New, unproved, beta, unstable modules can be hosted on *solr-sandbox*
> (and graduate to solr-extras once stable).
>
> Please let me know if there are any questions/concerns with this approach.
>
> Thanks and regards,
> Ishan
>
Re: Separate git repo(s) for Solr modules [ In reply to ]
+1

This is also an opportunity to create the distinction between first party
supported packages and the other plug-ins.



On Wed, 13 Jan 2021, 21:00 Ishan Chattopadhyaya, <ichattopadhyaya@gmail.com>
wrote:

> Hi Devs,
>
> As we discussed over the last few months, there seems a need to move
> non-core pieces away from the Solr core module. The contribs are presently
> a good place, but it makes sense to have a separate git repository hosting
> such modules. Some candidates that come to mind are the present day contrib
> modules, upcoming HDFS support module (separated away from solr-core),
> other first party packages. Along with that, there is also a need for a
> repository for hosting WIP modules/sub-projects.
>
> I propose that we apply for the creation of two new git repositories:
> 1. solr-extras (or lucene-solr-extras)
> 2. solr-sandbox (or lucene-solr-sandbox)
>
> Well tested, well supported modules/sub-projects can be released straight
> away from *solr-extras*. The first party packages can be built from this
> location and shipped with Solr (or be available for install using the
> package manager CLI).
>
> New, unproved, beta, unstable modules can be hosted on *solr-sandbox*
> (and graduate to solr-extras once stable).
>
> Please let me know if there are any questions/concerns with this approach.
>
> Thanks and regards,
> Ishan
>
Re: Separate git repo(s) for Solr modules [ In reply to ]
Agere with Houston 100%. I think it's a good idea to have an official repo
for things that are related to Solr and, at the same time, don't belong
within the Solr codebase such as the prometheus exporter or, eventually,
the cross-dc work that Anshum is working on (once it "graduates" from
sandbox I guess). For the "first party" plugins (things like
analysis-extra, langid, ltr), it's better to keep them together with the
codebase, so that we can easily guarantee compatibility and releases
together with Solr.

On Wed, Jan 13, 2021 at 8:26 AM Atri Sharma <atri@apache.org> wrote:

> +1
>
> This is also an opportunity to create the distinction between first party
> supported packages and the other plug-ins.
>
>
>
> On Wed, 13 Jan 2021, 21:00 Ishan Chattopadhyaya, <
> ichattopadhyaya@gmail.com> wrote:
>
>> Hi Devs,
>>
>> As we discussed over the last few months, there seems a need to move
>> non-core pieces away from the Solr core module. The contribs are presently
>> a good place, but it makes sense to have a separate git repository hosting
>> such modules. Some candidates that come to mind are the present day contrib
>> modules, upcoming HDFS support module (separated away from solr-core),
>> other first party packages. Along with that, there is also a need for a
>> repository for hosting WIP modules/sub-projects.
>>
>> I propose that we apply for the creation of two new git repositories:
>> 1. solr-extras (or lucene-solr-extras)
>> 2. solr-sandbox (or lucene-solr-sandbox)
>>
>> Well tested, well supported modules/sub-projects can be released straight
>> away from *solr-extras*. The first party packages can be built from this
>> location and shipped with Solr (or be available for install using the
>> package manager CLI).
>>
>> New, unproved, beta, unstable modules can be hosted on *solr-sandbox*
>> (and graduate to solr-extras once stable).
>>
>> Please let me know if there are any questions/concerns with this approach.
>>
>> Thanks and regards,
>> Ishan
>>
>
Re: Separate git repo(s) for Solr modules [ In reply to ]
Both paths have an appeal -- do contribs stay or move? I recommend not
moving existing contribs there until we explore how we can maintain
compatibility with new Solr releases. (what Houston & Tomas point to
below). For example, *perhaps* a "git sub-module" pointing to Solr might
be a key mechanism? Could CI automatically keep it up to date? Someone
needs to try before we really know.

~ David Smiley
Apache Lucene/Solr Search Developer
http://www.linkedin.com/in/davidwsmiley


On Wed, Jan 13, 2021 at 10:30 AM Ishan Chattopadhyaya <
ichattopadhyaya@gmail.com> wrote:

> Hi Devs,
>
> As we discussed over the last few months, there seems a need to move
> non-core pieces away from the Solr core module. The contribs are presently
> a good place, but it makes sense to have a separate git repository hosting
> such modules. Some candidates that come to mind are the present day contrib
> modules, upcoming HDFS support module (separated away from solr-core),
> other first party packages. Along with that, there is also a need for a
> repository for hosting WIP modules/sub-projects.
>
> I propose that we apply for the creation of two new git repositories:
> 1. solr-extras (or lucene-solr-extras)
> 2. solr-sandbox (or lucene-solr-sandbox)
>
> Well tested, well supported modules/sub-projects can be released straight
> away from *solr-extras*. The first party packages can be built from this
> location and shipped with Solr (or be available for install using the
> package manager CLI).
>
> New, unproved, beta, unstable modules can be hosted on *solr-sandbox*
> (and graduate to solr-extras once stable).
>
> Please let me know if there are any questions/concerns with this approach.
>
> Thanks and regards,
> Ishan
>
Re: Separate git repo(s) for Solr modules [ In reply to ]
Sounds good, thanks Houston for bringing it up. I'll take a stab at a few
first party packages soon and then we can take a call where they should
belong (maybe on a case to case basis).

Anshum has applied for the sandbox repo, I'll request for the extras repo
soon.

Thanks!

On Thu, 14 Jan, 2021, 4:04 am David Smiley, <dsmiley@apache.org> wrote:

> Both paths have an appeal -- do contribs stay or move? I recommend not
> moving existing contribs there until we explore how we can maintain
> compatibility with new Solr releases. (what Houston & Tomas point to
> below). For example, *perhaps* a "git sub-module" pointing to Solr might
> be a key mechanism? Could CI automatically keep it up to date? Someone
> needs to try before we really know.
>
> ~ David Smiley
> Apache Lucene/Solr Search Developer
> http://www.linkedin.com/in/davidwsmiley
>
>
> On Wed, Jan 13, 2021 at 10:30 AM Ishan Chattopadhyaya <
> ichattopadhyaya@gmail.com> wrote:
>
>> Hi Devs,
>>
>> As we discussed over the last few months, there seems a need to move
>> non-core pieces away from the Solr core module. The contribs are presently
>> a good place, but it makes sense to have a separate git repository hosting
>> such modules. Some candidates that come to mind are the present day contrib
>> modules, upcoming HDFS support module (separated away from solr-core),
>> other first party packages. Along with that, there is also a need for a
>> repository for hosting WIP modules/sub-projects.
>>
>> I propose that we apply for the creation of two new git repositories:
>> 1. solr-extras (or lucene-solr-extras)
>> 2. solr-sandbox (or lucene-solr-sandbox)
>>
>> Well tested, well supported modules/sub-projects can be released straight
>> away from *solr-extras*. The first party packages can be built from this
>> location and shipped with Solr (or be available for install using the
>> package manager CLI).
>>
>> New, unproved, beta, unstable modules can be hosted on *solr-sandbox*
>> (and graduate to solr-extras once stable).
>>
>> Please let me know if there are any questions/concerns with this approach.
>>
>> Thanks and regards,
>> Ishan
>>
>
Re: Separate git repo(s) for Solr modules [ In reply to ]
: As we discussed over the last few months, there seems a need to move
: non-core pieces away from the Solr core module. The contribs are presently
: a good place, but it makes sense to have a separate git repository hosting
: such modules. Some candidates that come to mind are the present day contrib

can you explain why it makes sense to have a separate git repo for these
things?

I can think of lots of reasons why it makes sense to have a single
repo for all things solr (unified CI that quickly identifies if core
changes break "first order" plugins, shared feature branches & monotomic
commits of code that affects APIs and impls of those APIs, etc...) but
I've yet to see any concrete specifics of why multiple git repos are
"better" then just having distinct sub-projects (with distinct artifacts)
in the same repo other then "it makes sense"

why does it make sense?

why can't the ideas of "solr-sandbox" and "solr-extras" just be
directories in the "solr repo" ? ... what value is gained by making them
new repos?


-Hoss
http://www.lucidworks.com/

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
Re: Separate git repo(s) for Solr modules [ In reply to ]
Tentative +1 to Hoss' questions. I agree with his summary of the
potential risks here, and I share his ignorance of the perceived
benefits.

(I thought for a time that this was driven by interest in having
release cadences independent from Solr-core releases. I'm all for
that goal, but if that's the motivation I'm not sure what the obstacle
is to doing that with a single repo - all build systems these days
support versioning and releasing modules independent of one another.
But maybe that was never a driving factor here.)

I know there have been a lot of discussions about this, and I know the
repo has already been created. So maybe it's too late to object even
if I wanted to, which I'm not sure I do. But can someone that
understands the motivation please summarize what multiple-repos gets
us over a single repo that outweighs the "cons" that Hoss mentioned?

Jason

On Thu, Jan 14, 2021 at 12:34 PM Chris Hostetter
<hossman_lucene@fucit.org> wrote:
>
>
> : As we discussed over the last few months, there seems a need to move
> : non-core pieces away from the Solr core module. The contribs are presently
> : a good place, but it makes sense to have a separate git repository hosting
> : such modules. Some candidates that come to mind are the present day contrib
>
> can you explain why it makes sense to have a separate git repo for these
> things?
>
> I can think of lots of reasons why it makes sense to have a single
> repo for all things solr (unified CI that quickly identifies if core
> changes break "first order" plugins, shared feature branches & monotomic
> commits of code that affects APIs and impls of those APIs, etc...) but
> I've yet to see any concrete specifics of why multiple git repos are
> "better" then just having distinct sub-projects (with distinct artifacts)
> in the same repo other then "it makes sense"
>
> why does it make sense?
>
> why can't the ideas of "solr-sandbox" and "solr-extras" just be
> directories in the "solr repo" ? ... what value is gained by making them
> new repos?
>
>
> -Hoss
> http://www.lucidworks.com/
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: dev-help@lucene.apache.org
>

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
Re: Separate git repo(s) for Solr modules [ In reply to ]
I haven't been able to follow up on creation of the extras repo, but more
importantly I wanted to respond to Hoss. I'm out on an emergency for a week
or so, shall resume then. If there's a decision on this until then, I shall
accept it.

On Mon, 25 Jan, 2021, 9:04 am Jason Gerlowski, <gerlowskija@gmail.com>
wrote:

> Tentative +1 to Hoss' questions. I agree with his summary of the
> potential risks here, and I share his ignorance of the perceived
> benefits.
>
> (I thought for a time that this was driven by interest in having
> release cadences independent from Solr-core releases. I'm all for
> that goal, but if that's the motivation I'm not sure what the obstacle
> is to doing that with a single repo - all build systems these days
> support versioning and releasing modules independent of one another.
> But maybe that was never a driving factor here.)
>
> I know there have been a lot of discussions about this, and I know the
> repo has already been created. So maybe it's too late to object even
> if I wanted to, which I'm not sure I do. But can someone that
> understands the motivation please summarize what multiple-repos gets
> us over a single repo that outweighs the "cons" that Hoss mentioned?
>
> Jason
>
> On Thu, Jan 14, 2021 at 12:34 PM Chris Hostetter
> <hossman_lucene@fucit.org> wrote:
> >
> >
> > : As we discussed over the last few months, there seems a need to move
> > : non-core pieces away from the Solr core module. The contribs are
> presently
> > : a good place, but it makes sense to have a separate git repository
> hosting
> > : such modules. Some candidates that come to mind are the present day
> contrib
> >
> > can you explain why it makes sense to have a separate git repo for these
> > things?
> >
> > I can think of lots of reasons why it makes sense to have a single
> > repo for all things solr (unified CI that quickly identifies if core
> > changes break "first order" plugins, shared feature branches & monotomic
> > commits of code that affects APIs and impls of those APIs, etc...) but
> > I've yet to see any concrete specifics of why multiple git repos are
> > "better" then just having distinct sub-projects (with distinct artifacts)
> > in the same repo other then "it makes sense"
> >
> > why does it make sense?
> >
> > why can't the ideas of "solr-sandbox" and "solr-extras" just be
> > directories in the "solr repo" ? ... what value is gained by making them
> > new repos?
> >
> >
> > -Hoss
> > http://www.lucidworks.com/
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
> > For additional commands, e-mail: dev-help@lucene.apache.org
> >
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: dev-help@lucene.apache.org
>
>
Re: Separate git repo(s) for Solr modules [ In reply to ]
Hoss,
One important reason why separate repo for solr-extras is a good idea, as
opposed to conrib modules, is that separate repo can be used to test
against many Solr versions. Imagine a component that works across Solr
versions 6x through 9x from day one. I can't imagine such a component being
part of the lucene-solr repo itself.

Also, today, contrib modules are shipped with Solr, which we don't want for
new things we might want to build.

On Mon, Jan 25, 2021 at 4:25 PM Ishan Chattopadhyaya <
ichattopadhyaya@gmail.com> wrote:

> I haven't been able to follow up on creation of the extras repo, but more
> importantly I wanted to respond to Hoss. I'm out on an emergency for a week
> or so, shall resume then. If there's a decision on this until then, I shall
> accept it.
>
> On Mon, 25 Jan, 2021, 9:04 am Jason Gerlowski, <gerlowskija@gmail.com>
> wrote:
>
>> Tentative +1 to Hoss' questions. I agree with his summary of the
>> potential risks here, and I share his ignorance of the perceived
>> benefits.
>>
>> (I thought for a time that this was driven by interest in having
>> release cadences independent from Solr-core releases. I'm all for
>> that goal, but if that's the motivation I'm not sure what the obstacle
>> is to doing that with a single repo - all build systems these days
>> support versioning and releasing modules independent of one another.
>> But maybe that was never a driving factor here.)
>>
>> I know there have been a lot of discussions about this, and I know the
>> repo has already been created. So maybe it's too late to object even
>> if I wanted to, which I'm not sure I do. But can someone that
>> understands the motivation please summarize what multiple-repos gets
>> us over a single repo that outweighs the "cons" that Hoss mentioned?
>>
>> Jason
>>
>> On Thu, Jan 14, 2021 at 12:34 PM Chris Hostetter
>> <hossman_lucene@fucit.org> wrote:
>> >
>> >
>> > : As we discussed over the last few months, there seems a need to move
>> > : non-core pieces away from the Solr core module. The contribs are
>> presently
>> > : a good place, but it makes sense to have a separate git repository
>> hosting
>> > : such modules. Some candidates that come to mind are the present day
>> contrib
>> >
>> > can you explain why it makes sense to have a separate git repo for these
>> > things?
>> >
>> > I can think of lots of reasons why it makes sense to have a single
>> > repo for all things solr (unified CI that quickly identifies if core
>> > changes break "first order" plugins, shared feature branches & monotomic
>> > commits of code that affects APIs and impls of those APIs, etc...) but
>> > I've yet to see any concrete specifics of why multiple git repos are
>> > "better" then just having distinct sub-projects (with distinct
>> artifacts)
>> > in the same repo other then "it makes sense"
>> >
>> > why does it make sense?
>> >
>> > why can't the ideas of "solr-sandbox" and "solr-extras" just be
>> > directories in the "solr repo" ? ... what value is gained by making them
>> > new repos?
>> >
>> >
>> > -Hoss
>> > http://www.lucidworks.com/
>> >
>> > ---------------------------------------------------------------------
>> > To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
>> > For additional commands, e-mail: dev-help@lucene.apache.org
>> >
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: dev-help@lucene.apache.org
>>
>>
Re: Separate git repo(s) for Solr modules [ In reply to ]
Hi,

> One important reason why separate repo for solr-extras is a good idea, as opposed to conrib modules, is that separate repo can be used to test against many Solr versions. Imagine a component that works across Solr versions 6x through 9x from day one. I can't imagine such a component being part of the lucene-solr repo itself.

For independent tools such as solr-operator or prometheus-exporter, or some future standalone DIH, it makes sense with separate repo and release cadence. However it will be much easier for our 1st party packages to remain in the monorepo, keeping all tests in-sync with core.

> Also, today, contrib modules are shipped with Solr, which we don't want for new things we might want to build.

Remember that Solr main tarball does not need to include all our packages. We can easily extend our releae process to release e.g. solr-cell-9.5, solr-analysis-9.5, solr-auth-xyz-9.5 in our release repo together with the 9.5 release, i.e. release every package with every Solr version. Then we do not need a compatibility matrix, since users will be able to install the latest package. And each release can ship with a pre-installed package repo for that version.

I imagine it will be very very hard to provide cross-major-version compatibility for some plugins. It may get easier as more higher-level APIs are build to shield plugins from low-level inner code, but we're not there.

Jan


> 15. feb. 2021 kl. 15:17 skrev Ishan Chattopadhyaya <ichattopadhyaya@gmail.com>:
>
> Hoss,
> One important reason why separate repo for solr-extras is a good idea, as opposed to conrib modules, is that separate repo can be used to test against many Solr versions. Imagine a component that works across Solr versions 6x through 9x from day one. I can't imagine such a component being part of the lucene-solr repo itself.
>
> Also, today, contrib modules are shipped with Solr, which we don't want for new things we might want to build.
>
> On Mon, Jan 25, 2021 at 4:25 PM Ishan Chattopadhyaya <ichattopadhyaya@gmail.com <mailto:ichattopadhyaya@gmail.com>> wrote:
> I haven't been able to follow up on creation of the extras repo, but more importantly I wanted to respond to Hoss. I'm out on an emergency for a week or so, shall resume then. If there's a decision on this until then, I shall accept it.
>
> On Mon, 25 Jan, 2021, 9:04 am Jason Gerlowski, <gerlowskija@gmail.com <mailto:gerlowskija@gmail.com>> wrote:
> Tentative +1 to Hoss' questions. I agree with his summary of the
> potential risks here, and I share his ignorance of the perceived
> benefits.
>
> (I thought for a time that this was driven by interest in having
> release cadences independent from Solr-core releases. I'm all for
> that goal, but if that's the motivation I'm not sure what the obstacle
> is to doing that with a single repo - all build systems these days
> support versioning and releasing modules independent of one another.
> But maybe that was never a driving factor here.)
>
> I know there have been a lot of discussions about this, and I know the
> repo has already been created. So maybe it's too late to object even
> if I wanted to, which I'm not sure I do. But can someone that
> understands the motivation please summarize what multiple-repos gets
> us over a single repo that outweighs the "cons" that Hoss mentioned?
>
> Jason
>
> On Thu, Jan 14, 2021 at 12:34 PM Chris Hostetter
> <hossman_lucene@fucit.org <mailto:hossman_lucene@fucit.org>> wrote:
> >
> >
> > : As we discussed over the last few months, there seems a need to move
> > : non-core pieces away from the Solr core module. The contribs are presently
> > : a good place, but it makes sense to have a separate git repository hosting
> > : such modules. Some candidates that come to mind are the present day contrib
> >
> > can you explain why it makes sense to have a separate git repo for these
> > things?
> >
> > I can think of lots of reasons why it makes sense to have a single
> > repo for all things solr (unified CI that quickly identifies if core
> > changes break "first order" plugins, shared feature branches & monotomic
> > commits of code that affects APIs and impls of those APIs, etc...) but
> > I've yet to see any concrete specifics of why multiple git repos are
> > "better" then just having distinct sub-projects (with distinct artifacts)
> > in the same repo other then "it makes sense"
> >
> > why does it make sense?
> >
> > why can't the ideas of "solr-sandbox" and "solr-extras" just be
> > directories in the "solr repo" ? ... what value is gained by making them
> > new repos?
> >
> >
> > -Hoss
> > http://www.lucidworks.com/ <http://www.lucidworks.com/>
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org <mailto:dev-unsubscribe@lucene.apache.org>
> > For additional commands, e-mail: dev-help@lucene.apache.org <mailto:dev-help@lucene.apache.org>
> >
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org <mailto:dev-unsubscribe@lucene.apache.org>
> For additional commands, e-mail: dev-help@lucene.apache.org <mailto:dev-help@lucene.apache.org>
>
Re: Separate git repo(s) for Solr modules [ In reply to ]
I think embracing Docker is how we can identify if a plugin works in
multiple versions of Solr. The setup of such a test would start Solr via
Docker and install the plugin into a Solr server using the package
manager. The test code itself would interact with Solr purely via
SolrClient. This would be an integration / smoke level test for the
plugin. I test my plugins at work similar to this way at a "smoke"
level, which follows "integration". What's cool is that we can take the
very same test and either have it run via an embedded Solr (which we call
an integration test) or a remote'ed Solr (run via Docker, which we call our
"smoke" test). Naturally, if a test makes assumptions that only work in
embedded, then it won't run in smoke mode, so we have annotations to
categorize the tests. I'd like to work on some JUnit "Rule" for Solr,
similar to what I have at work (and have done in the past) --
https://issues.apache.org/jira/browse/SOLR-11872

~ David Smiley
Apache Lucene/Solr Search Developer
http://www.linkedin.com/in/davidwsmiley


On Mon, Feb 15, 2021 at 9:17 AM Ishan Chattopadhyaya <
ichattopadhyaya@gmail.com> wrote:

> Hoss,
> One important reason why separate repo for solr-extras is a good idea, as
> opposed to conrib modules, is that separate repo can be used to test
> against many Solr versions. Imagine a component that works across Solr
> versions 6x through 9x from day one. I can't imagine such a component being
> part of the lucene-solr repo itself.
>
> Also, today, contrib modules are shipped with Solr, which we don't want
> for new things we might want to build.
>
> On Mon, Jan 25, 2021 at 4:25 PM Ishan Chattopadhyaya <
> ichattopadhyaya@gmail.com> wrote:
>
>> I haven't been able to follow up on creation of the extras repo, but more
>> importantly I wanted to respond to Hoss. I'm out on an emergency for a week
>> or so, shall resume then. If there's a decision on this until then, I shall
>> accept it.
>>
>> On Mon, 25 Jan, 2021, 9:04 am Jason Gerlowski, <gerlowskija@gmail.com>
>> wrote:
>>
>>> Tentative +1 to Hoss' questions. I agree with his summary of the
>>> potential risks here, and I share his ignorance of the perceived
>>> benefits.
>>>
>>> (I thought for a time that this was driven by interest in having
>>> release cadences independent from Solr-core releases. I'm all for
>>> that goal, but if that's the motivation I'm not sure what the obstacle
>>> is to doing that with a single repo - all build systems these days
>>> support versioning and releasing modules independent of one another.
>>> But maybe that was never a driving factor here.)
>>>
>>> I know there have been a lot of discussions about this, and I know the
>>> repo has already been created. So maybe it's too late to object even
>>> if I wanted to, which I'm not sure I do. But can someone that
>>> understands the motivation please summarize what multiple-repos gets
>>> us over a single repo that outweighs the "cons" that Hoss mentioned?
>>>
>>> Jason
>>>
>>> On Thu, Jan 14, 2021 at 12:34 PM Chris Hostetter
>>> <hossman_lucene@fucit.org> wrote:
>>> >
>>> >
>>> > : As we discussed over the last few months, there seems a need to move
>>> > : non-core pieces away from the Solr core module. The contribs are
>>> presently
>>> > : a good place, but it makes sense to have a separate git repository
>>> hosting
>>> > : such modules. Some candidates that come to mind are the present day
>>> contrib
>>> >
>>> > can you explain why it makes sense to have a separate git repo for
>>> these
>>> > things?
>>> >
>>> > I can think of lots of reasons why it makes sense to have a single
>>> > repo for all things solr (unified CI that quickly identifies if core
>>> > changes break "first order" plugins, shared feature branches &
>>> monotomic
>>> > commits of code that affects APIs and impls of those APIs, etc...) but
>>> > I've yet to see any concrete specifics of why multiple git repos are
>>> > "better" then just having distinct sub-projects (with distinct
>>> artifacts)
>>> > in the same repo other then "it makes sense"
>>> >
>>> > why does it make sense?
>>> >
>>> > why can't the ideas of "solr-sandbox" and "solr-extras" just be
>>> > directories in the "solr repo" ? ... what value is gained by making
>>> them
>>> > new repos?
>>> >
>>> >
>>> > -Hoss
>>> > http://www.lucidworks.com/
>>> >
>>> > ---------------------------------------------------------------------
>>> > To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
>>> > For additional commands, e-mail: dev-help@lucene.apache.org
>>> >
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
>>> For additional commands, e-mail: dev-help@lucene.apache.org
>>>
>>>
Re: Separate git repo(s) for Solr modules [ In reply to ]
> separate repo can be used to test against many Solr versions. Imagine a component that works across Solr versions 6x through 9x from day one. I can't imagine such a component being part of the lucene-solr repo itself.

It'll be great to test contrib's on multiple Solr versions, but I
can't see any blockers for this that are same-repo specific. Can you
elaborate on what makes this tricky in a same-repo setup? AFAICT you
can set up multiple gradle test tasks that rely on different
solr-test-framework versions and run tests that way - regardless of
where the code lives. (Or alternatively, docker images for various
Solr versions as David suggested.)

I think Jan's got a good point about how do-able it'll be in practice
to have plugins support multiple major release versions. That said
though, a lot of these questions (docker tests vs otherwise, cross
major-version plugins?) will likely sort themselves out when it comes
time to implement.

On Tue, Feb 16, 2021 at 12:21 AM David Smiley <dsmiley@apache.org> wrote:
>
> I think embracing Docker is how we can identify if a plugin works in multiple versions of Solr. The setup of such a test would start Solr via Docker and install the plugin into a Solr server using the package manager. The test code itself would interact with Solr purely via SolrClient. This would be an integration / smoke level test for the plugin. I test my plugins at work similar to this way at a "smoke" level, which follows "integration". What's cool is that we can take the very same test and either have it run via an embedded Solr (which we call an integration test) or a remote'ed Solr (run via Docker, which we call our "smoke" test). Naturally, if a test makes assumptions that only work in embedded, then it won't run in smoke mode, so we have annotations to categorize the tests. I'd like to work on some JUnit "Rule" for Solr, similar to what I have at work (and have done in the past) -- https://issues.apache.org/jira/browse/SOLR-11872
>
> ~ David Smiley
> Apache Lucene/Solr Search Developer
> http://www.linkedin.com/in/davidwsmiley
>
>
> On Mon, Feb 15, 2021 at 9:17 AM Ishan Chattopadhyaya <ichattopadhyaya@gmail.com> wrote:
>>
>> Hoss,
>> One important reason why separate repo for solr-extras is a good idea, as opposed to conrib modules, is that separate repo can be used to test against many Solr versions. Imagine a component that works across Solr versions 6x through 9x from day one. I can't imagine such a component being part of the lucene-solr repo itself.
>>
>> Also, today, contrib modules are shipped with Solr, which we don't want for new things we might want to build.
>>
>> On Mon, Jan 25, 2021 at 4:25 PM Ishan Chattopadhyaya <ichattopadhyaya@gmail.com> wrote:
>>>
>>> I haven't been able to follow up on creation of the extras repo, but more importantly I wanted to respond to Hoss. I'm out on an emergency for a week or so, shall resume then. If there's a decision on this until then, I shall accept it.
>>>
>>> On Mon, 25 Jan, 2021, 9:04 am Jason Gerlowski, <gerlowskija@gmail.com> wrote:
>>>>
>>>> Tentative +1 to Hoss' questions. I agree with his summary of the
>>>> potential risks here, and I share his ignorance of the perceived
>>>> benefits.
>>>>
>>>> (I thought for a time that this was driven by interest in having
>>>> release cadences independent from Solr-core releases. I'm all for
>>>> that goal, but if that's the motivation I'm not sure what the obstacle
>>>> is to doing that with a single repo - all build systems these days
>>>> support versioning and releasing modules independent of one another.
>>>> But maybe that was never a driving factor here.)
>>>>
>>>> I know there have been a lot of discussions about this, and I know the
>>>> repo has already been created. So maybe it's too late to object even
>>>> if I wanted to, which I'm not sure I do. But can someone that
>>>> understands the motivation please summarize what multiple-repos gets
>>>> us over a single repo that outweighs the "cons" that Hoss mentioned?
>>>>
>>>> Jason
>>>>
>>>> On Thu, Jan 14, 2021 at 12:34 PM Chris Hostetter
>>>> <hossman_lucene@fucit.org> wrote:
>>>> >
>>>> >
>>>> > : As we discussed over the last few months, there seems a need to move
>>>> > : non-core pieces away from the Solr core module. The contribs are presently
>>>> > : a good place, but it makes sense to have a separate git repository hosting
>>>> > : such modules. Some candidates that come to mind are the present day contrib
>>>> >
>>>> > can you explain why it makes sense to have a separate git repo for these
>>>> > things?
>>>> >
>>>> > I can think of lots of reasons why it makes sense to have a single
>>>> > repo for all things solr (unified CI that quickly identifies if core
>>>> > changes break "first order" plugins, shared feature branches & monotomic
>>>> > commits of code that affects APIs and impls of those APIs, etc...) but
>>>> > I've yet to see any concrete specifics of why multiple git repos are
>>>> > "better" then just having distinct sub-projects (with distinct artifacts)
>>>> > in the same repo other then "it makes sense"
>>>> >
>>>> > why does it make sense?
>>>> >
>>>> > why can't the ideas of "solr-sandbox" and "solr-extras" just be
>>>> > directories in the "solr repo" ? ... what value is gained by making them
>>>> > new repos?
>>>> >
>>>> >
>>>> > -Hoss
>>>> > http://www.lucidworks.com/
>>>> >
>>>> > ---------------------------------------------------------------------
>>>> > To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
>>>> > For additional commands, e-mail: dev-help@lucene.apache.org
>>>> >
>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
>>>> For additional commands, e-mail: dev-help@lucene.apache.org
>>>>

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
Re: Separate git repo(s) for Solr modules [ In reply to ]
Imagine a component that wants to have its own release cadence. Say, a 1.x
line of the component that supports certain features (across Solr 6x to 9x)
and 2.x line that supports certain other features (which are a big upgrade
from previous 1.x). A Solr UI can be a good example, where say 1.x uses
React and 2.x uses AngularJS.
Shoving such a component into lucene-solr repo makes no sense, given its
branching strategy is based on master / branch_8x (and later it will become
branch_9x and master).

On Tue, Feb 16, 2021 at 5:19 PM Jason Gerlowski <gerlowskija@gmail.com>
wrote:

> > separate repo can be used to test against many Solr versions. Imagine a
> component that works across Solr versions 6x through 9x from day one. I
> can't imagine such a component being part of the lucene-solr repo itself.
>
> It'll be great to test contrib's on multiple Solr versions, but I
> can't see any blockers for this that are same-repo specific. Can you
> elaborate on what makes this tricky in a same-repo setup? AFAICT you
> can set up multiple gradle test tasks that rely on different
> solr-test-framework versions and run tests that way - regardless of
> where the code lives. (Or alternatively, docker images for various
> Solr versions as David suggested.)
>
> I think Jan's got a good point about how do-able it'll be in practice
> to have plugins support multiple major release versions. That said
> though, a lot of these questions (docker tests vs otherwise, cross
> major-version plugins?) will likely sort themselves out when it comes
> time to implement.
>
> On Tue, Feb 16, 2021 at 12:21 AM David Smiley <dsmiley@apache.org> wrote:
> >
> > I think embracing Docker is how we can identify if a plugin works in
> multiple versions of Solr. The setup of such a test would start Solr via
> Docker and install the plugin into a Solr server using the package
> manager. The test code itself would interact with Solr purely via
> SolrClient. This would be an integration / smoke level test for the
> plugin. I test my plugins at work similar to this way at a "smoke" level,
> which follows "integration". What's cool is that we can take the very same
> test and either have it run via an embedded Solr (which we call an
> integration test) or a remote'ed Solr (run via Docker, which we call our
> "smoke" test). Naturally, if a test makes assumptions that only work in
> embedded, then it won't run in smoke mode, so we have annotations to
> categorize the tests. I'd like to work on some JUnit "Rule" for Solr,
> similar to what I have at work (and have done in the past) --
> https://issues.apache.org/jira/browse/SOLR-11872
> >
> > ~ David Smiley
> > Apache Lucene/Solr Search Developer
> > http://www.linkedin.com/in/davidwsmiley
> >
> >
> > On Mon, Feb 15, 2021 at 9:17 AM Ishan Chattopadhyaya <
> ichattopadhyaya@gmail.com> wrote:
> >>
> >> Hoss,
> >> One important reason why separate repo for solr-extras is a good idea,
> as opposed to conrib modules, is that separate repo can be used to test
> against many Solr versions. Imagine a component that works across Solr
> versions 6x through 9x from day one. I can't imagine such a component being
> part of the lucene-solr repo itself.
> >>
> >> Also, today, contrib modules are shipped with Solr, which we don't want
> for new things we might want to build.
> >>
> >> On Mon, Jan 25, 2021 at 4:25 PM Ishan Chattopadhyaya <
> ichattopadhyaya@gmail.com> wrote:
> >>>
> >>> I haven't been able to follow up on creation of the extras repo, but
> more importantly I wanted to respond to Hoss. I'm out on an emergency for a
> week or so, shall resume then. If there's a decision on this until then, I
> shall accept it.
> >>>
> >>> On Mon, 25 Jan, 2021, 9:04 am Jason Gerlowski, <gerlowskija@gmail.com>
> wrote:
> >>>>
> >>>> Tentative +1 to Hoss' questions. I agree with his summary of the
> >>>> potential risks here, and I share his ignorance of the perceived
> >>>> benefits.
> >>>>
> >>>> (I thought for a time that this was driven by interest in having
> >>>> release cadences independent from Solr-core releases. I'm all for
> >>>> that goal, but if that's the motivation I'm not sure what the obstacle
> >>>> is to doing that with a single repo - all build systems these days
> >>>> support versioning and releasing modules independent of one another.
> >>>> But maybe that was never a driving factor here.)
> >>>>
> >>>> I know there have been a lot of discussions about this, and I know the
> >>>> repo has already been created. So maybe it's too late to object even
> >>>> if I wanted to, which I'm not sure I do. But can someone that
> >>>> understands the motivation please summarize what multiple-repos gets
> >>>> us over a single repo that outweighs the "cons" that Hoss mentioned?
> >>>>
> >>>> Jason
> >>>>
> >>>> On Thu, Jan 14, 2021 at 12:34 PM Chris Hostetter
> >>>> <hossman_lucene@fucit.org> wrote:
> >>>> >
> >>>> >
> >>>> > : As we discussed over the last few months, there seems a need to
> move
> >>>> > : non-core pieces away from the Solr core module. The contribs are
> presently
> >>>> > : a good place, but it makes sense to have a separate git
> repository hosting
> >>>> > : such modules. Some candidates that come to mind are the present
> day contrib
> >>>> >
> >>>> > can you explain why it makes sense to have a separate git repo for
> these
> >>>> > things?
> >>>> >
> >>>> > I can think of lots of reasons why it makes sense to have a single
> >>>> > repo for all things solr (unified CI that quickly identifies if core
> >>>> > changes break "first order" plugins, shared feature branches &
> monotomic
> >>>> > commits of code that affects APIs and impls of those APIs, etc...)
> but
> >>>> > I've yet to see any concrete specifics of why multiple git repos are
> >>>> > "better" then just having distinct sub-projects (with distinct
> artifacts)
> >>>> > in the same repo other then "it makes sense"
> >>>> >
> >>>> > why does it make sense?
> >>>> >
> >>>> > why can't the ideas of "solr-sandbox" and "solr-extras" just be
> >>>> > directories in the "solr repo" ? ... what value is gained by making
> them
> >>>> > new repos?
> >>>> >
> >>>> >
> >>>> > -Hoss
> >>>> > http://www.lucidworks.com/
> >>>> >
> >>>> >
> ---------------------------------------------------------------------
> >>>> > To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
> >>>> > For additional commands, e-mail: dev-help@lucene.apache.org
> >>>> >
> >>>>
> >>>> ---------------------------------------------------------------------
> >>>> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
> >>>> For additional commands, e-mail: dev-help@lucene.apache.org
> >>>>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: dev-help@lucene.apache.org
>
>
Re: Separate git repo(s) for Solr modules [ In reply to ]
Testing across multiple versions is always very difficult ;-). I recently saw this very interesting approach to using our Dockerized Solr’s to test a component against a number of previous versions of Solr. https://github.com/querqy/querqy/pull/147. I’m hopeful it could be a model for other packages to adopt.

Trying to maintain across multiple versions is kind of a Sisyphean task, and I don’t think plays to open source communities strengths. It’s hard enough to keep all components of Solr up to date with the latest Lucene and the latest Solr…. (Try out wt=xlsx Response Writer, it’s currently broken on master) . Reminds me of the Apache Gump project ;-)

If something is really going to be backcompatible across certain versions, then maybe having it’s own repo makes sense, but I suspect it means those components may go stale…. Look at DIH and Velocity components that are moved over to their own repos, both are getting stale, and I’d argue it’s because they don’t live in the main project where all of us have oversight and easy access.


> On Feb 16, 2021, at 7:44 AM, Ishan Chattopadhyaya <ichattopadhyaya@gmail.com> wrote:
>
> Imagine a component that wants to have its own release cadence. Say, a 1.x line of the component that supports certain features (across Solr 6x to 9x) and 2.x line that supports certain other features (which are a big upgrade from previous 1.x). A Solr UI can be a good example, where say 1.x uses React and 2.x uses AngularJS.
> Shoving such a component into lucene-solr repo makes no sense, given its branching strategy is based on master / branch_8x (and later it will become branch_9x and master).
>
> On Tue, Feb 16, 2021 at 5:19 PM Jason Gerlowski <gerlowskija@gmail.com <mailto:gerlowskija@gmail.com>> wrote:
> > separate repo can be used to test against many Solr versions. Imagine a component that works across Solr versions 6x through 9x from day one. I can't imagine such a component being part of the lucene-solr repo itself.
>
> It'll be great to test contrib's on multiple Solr versions, but I
> can't see any blockers for this that are same-repo specific. Can you
> elaborate on what makes this tricky in a same-repo setup? AFAICT you
> can set up multiple gradle test tasks that rely on different
> solr-test-framework versions and run tests that way - regardless of
> where the code lives. (Or alternatively, docker images for various
> Solr versions as David suggested.)
>
> I think Jan's got a good point about how do-able it'll be in practice
> to have plugins support multiple major release versions. That said
> though, a lot of these questions (docker tests vs otherwise, cross
> major-version plugins?) will likely sort themselves out when it comes
> time to implement.
>
> On Tue, Feb 16, 2021 at 12:21 AM David Smiley <dsmiley@apache.org <mailto:dsmiley@apache.org>> wrote:
> >
> > I think embracing Docker is how we can identify if a plugin works in multiple versions of Solr. The setup of such a test would start Solr via Docker and install the plugin into a Solr server using the package manager. The test code itself would interact with Solr purely via SolrClient. This would be an integration / smoke level test for the plugin. I test my plugins at work similar to this way at a "smoke" level, which follows "integration". What's cool is that we can take the very same test and either have it run via an embedded Solr (which we call an integration test) or a remote'ed Solr (run via Docker, which we call our "smoke" test). Naturally, if a test makes assumptions that only work in embedded, then it won't run in smoke mode, so we have annotations to categorize the tests. I'd like to work on some JUnit "Rule" for Solr, similar to what I have at work (and have done in the past) -- https://issues.apache.org/jira/browse/SOLR-11872 <https://issues.apache.org/jira/browse/SOLR-11872>
> >
> > ~ David Smiley
> > Apache Lucene/Solr Search Developer
> > http://www.linkedin.com/in/davidwsmiley <http://www.linkedin.com/in/davidwsmiley>
> >
> >
> > On Mon, Feb 15, 2021 at 9:17 AM Ishan Chattopadhyaya <ichattopadhyaya@gmail.com <mailto:ichattopadhyaya@gmail.com>> wrote:
> >>
> >> Hoss,
> >> One important reason why separate repo for solr-extras is a good idea, as opposed to conrib modules, is that separate repo can be used to test against many Solr versions. Imagine a component that works across Solr versions 6x through 9x from day one. I can't imagine such a component being part of the lucene-solr repo itself.
> >>
> >> Also, today, contrib modules are shipped with Solr, which we don't want for new things we might want to build.
> >>
> >> On Mon, Jan 25, 2021 at 4:25 PM Ishan Chattopadhyaya <ichattopadhyaya@gmail.com <mailto:ichattopadhyaya@gmail.com>> wrote:
> >>>
> >>> I haven't been able to follow up on creation of the extras repo, but more importantly I wanted to respond to Hoss. I'm out on an emergency for a week or so, shall resume then. If there's a decision on this until then, I shall accept it.
> >>>
> >>> On Mon, 25 Jan, 2021, 9:04 am Jason Gerlowski, <gerlowskija@gmail.com <mailto:gerlowskija@gmail.com>> wrote:
> >>>>
> >>>> Tentative +1 to Hoss' questions. I agree with his summary of the
> >>>> potential risks here, and I share his ignorance of the perceived
> >>>> benefits.
> >>>>
> >>>> (I thought for a time that this was driven by interest in having
> >>>> release cadences independent from Solr-core releases. I'm all for
> >>>> that goal, but if that's the motivation I'm not sure what the obstacle
> >>>> is to doing that with a single repo - all build systems these days
> >>>> support versioning and releasing modules independent of one another.
> >>>> But maybe that was never a driving factor here.)
> >>>>
> >>>> I know there have been a lot of discussions about this, and I know the
> >>>> repo has already been created. So maybe it's too late to object even
> >>>> if I wanted to, which I'm not sure I do. But can someone that
> >>>> understands the motivation please summarize what multiple-repos gets
> >>>> us over a single repo that outweighs the "cons" that Hoss mentioned?
> >>>>
> >>>> Jason
> >>>>
> >>>> On Thu, Jan 14, 2021 at 12:34 PM Chris Hostetter
> >>>> <hossman_lucene@fucit.org <mailto:hossman_lucene@fucit.org>> wrote:
> >>>> >
> >>>> >
> >>>> > : As we discussed over the last few months, there seems a need to move
> >>>> > : non-core pieces away from the Solr core module. The contribs are presently
> >>>> > : a good place, but it makes sense to have a separate git repository hosting
> >>>> > : such modules. Some candidates that come to mind are the present day contrib
> >>>> >
> >>>> > can you explain why it makes sense to have a separate git repo for these
> >>>> > things?
> >>>> >
> >>>> > I can think of lots of reasons why it makes sense to have a single
> >>>> > repo for all things solr (unified CI that quickly identifies if core
> >>>> > changes break "first order" plugins, shared feature branches & monotomic
> >>>> > commits of code that affects APIs and impls of those APIs, etc...) but
> >>>> > I've yet to see any concrete specifics of why multiple git repos are
> >>>> > "better" then just having distinct sub-projects (with distinct artifacts)
> >>>> > in the same repo other then "it makes sense"
> >>>> >
> >>>> > why does it make sense?
> >>>> >
> >>>> > why can't the ideas of "solr-sandbox" and "solr-extras" just be
> >>>> > directories in the "solr repo" ? ... what value is gained by making them
> >>>> > new repos?
> >>>> >
> >>>> >
> >>>> > -Hoss
> >>>> > http://www.lucidworks.com/ <http://www.lucidworks.com/>
> >>>> >
> >>>> > ---------------------------------------------------------------------
> >>>> > To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org <mailto:dev-unsubscribe@lucene.apache.org>
> >>>> > For additional commands, e-mail: dev-help@lucene.apache.org <mailto:dev-help@lucene.apache.org>
> >>>> >
> >>>>
> >>>> ---------------------------------------------------------------------
> >>>> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org <mailto:dev-unsubscribe@lucene.apache.org>
> >>>> For additional commands, e-mail: dev-help@lucene.apache.org <mailto:dev-help@lucene.apache.org>
> >>>>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org <mailto:dev-unsubscribe@lucene.apache.org>
> For additional commands, e-mail: dev-help@lucene.apache.org <mailto:dev-help@lucene.apache.org>
>

_______________________
Eric Pugh | Founder & CEO | OpenSource Connections, LLC | 434.466.1467 | http://www.opensourceconnections.com <http://www.opensourceconnections.com/> | My Free/Busy <http://tinyurl.com/eric-cal>
Co-Author: Apache Solr Enterprise Search Server, 3rd Ed <https://www.packtpub.com/big-data-and-business-intelligence/apache-solr-enterprise-search-server-third-edition-raw>
This e-mail and all contents, including attachments, is considered to be Company Confidential unless explicitly stated otherwise, regardless of whether attachments are marked as such.
Re: Separate git repo(s) for Solr modules [ In reply to ]
On Tue, Feb 16, 2021 at 8:38 AM Eric Pugh <epugh@opensourceconnections.com>
wrote:

> Testing across multiple versions is always very difficult ;-). I recently
> saw this very interesting approach to using our Dockerized Solr’s to test a
> component against a number of previous versions of Solr.
> https://github.com/querqy/querqy/pull/147. I’m hopeful it could be a
> model for other packages to adopt.
>

Thanks for the link to that Querqy PR. That is *very* similar to what I do
at work (minus multi-version testing), and also similar to how I test
multiple versions in one of my pet projects by using a CI build matrix of a
configurable dependency. I didn't know Testcontainer.org has its own Solr
module -- https://www.testcontainers.org/modules/solr/ but we implemented
that ourselves; not hard.


> Trying to maintain across multiple versions is kind of a Sisyphean task,
> and I don’t think plays to open source communities strengths. It’s hard
> enough to keep all components of Solr up to date with the latest Lucene and
> the latest Solr…. (Try out wt=xlsx Response Writer, it’s currently broken
> on master) . Reminds me of the Apache Gump project ;-)
>
> If something is really going to be backcompatible across certain versions,
> then maybe having it’s own repo makes sense,
>

I'd prefer that a module only leave our "contribs" area when the
concerns/limitations become real. Doing it prematurely could lead to
atrophy of the module....


> but I suspect it means those components may go stale…. Look at DIH and
> Velocity components that are moved over to their own repos, both are
> getting stale, and I’d argue it’s because they don’t live in the main
> project where all of us have oversight and easy access.
>

Agreed :-(
Re: Separate git repo(s) for Solr modules [ In reply to ]
> Shoving such a component into lucene-solr repo makes no sense, given its branching strategy is based on master / branch_8x

I get how this could cause issues - I def hadn't thought much about
multi-version support and branching. But I don't think moving plugins
to a separate repo solves that problem for us. If our first class
plugins are set up to have different release "lines" that don't line
up with major Solr versions, it's only a matter of time before two of
those plugins have branch requirements that conflict. Unless I'm
missing something here?

> I'd prefer that a module only leave our "contribs" area when the concerns/limitations become real. Doing it prematurely could lead to atrophy of the module....

+1 to David's comment. I def hadn't considered the branching-scheme
issues that Ishan brought up in his last reply, and they seem like
valid concerns to me. But the risk and the downsides of "atrophy" are
serious enough that I'd vote to not risk them until this starts to
cause us issues in practice. Even if, for now, that means we won't be
able to build a single plugin jar that supports (e.g.) 3 major Solr
versions. IMO that's a much smaller loss.

On Tue, Feb 16, 2021 at 9:40 AM David Smiley <dsmiley@apache.org> wrote:
>
> On Tue, Feb 16, 2021 at 8:38 AM Eric Pugh <epugh@opensourceconnections.com> wrote:
>>
>> Testing across multiple versions is always very difficult ;-). I recently saw this very interesting approach to using our Dockerized Solr’s to test a component against a number of previous versions of Solr. https://github.com/querqy/querqy/pull/147. I’m hopeful it could be a model for other packages to adopt.
>
>
> Thanks for the link to that Querqy PR. That is *very* similar to what I do at work (minus multi-version testing), and also similar to how I test multiple versions in one of my pet projects by using a CI build matrix of a configurable dependency. I didn't know Testcontainer.org has its own Solr module -- https://www.testcontainers.org/modules/solr/ but we implemented that ourselves; not hard.
>
>>
>> Trying to maintain across multiple versions is kind of a Sisyphean task, and I don’t think plays to open source communities strengths. It’s hard enough to keep all components of Solr up to date with the latest Lucene and the latest Solr…. (Try out wt=xlsx Response Writer, it’s currently broken on master) . Reminds me of the Apache Gump project ;-)
>>
>> If something is really going to be backcompatible across certain versions, then maybe having it’s own repo makes sense,
>
>
> I'd prefer that a module only leave our "contribs" area when the concerns/limitations become real. Doing it prematurely could lead to atrophy of the module....
>
>>
>> but I suspect it means those components may go stale…. Look at DIH and Velocity components that are moved over to their own repos, both are getting stale, and I’d argue it’s because they don’t live in the main project where all of us have oversight and easy access.
>
>
> Agreed :-(
>

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
Re: Separate git repo(s) for Solr modules [ In reply to ]
I'm still struggling to understand the workflow when I'm working on a
feature that spans lucene and solr.

I'm yet to see an argument against sub-modules

On Wed, Feb 17, 2021 at 3:18 AM Jason Gerlowski <gerlowskija@gmail.com>
wrote:

> > Shoving such a component into lucene-solr repo makes no sense, given its
> branching strategy is based on master / branch_8x
>
> I get how this could cause issues - I def hadn't thought much about
> multi-version support and branching. But I don't think moving plugins
> to a separate repo solves that problem for us. If our first class
> plugins are set up to have different release "lines" that don't line
> up with major Solr versions, it's only a matter of time before two of
> those plugins have branch requirements that conflict. Unless I'm
> missing something here?
>
> > I'd prefer that a module only leave our "contribs" area when the
> concerns/limitations become real. Doing it prematurely could lead to
> atrophy of the module....
>
> +1 to David's comment. I def hadn't considered the branching-scheme
> issues that Ishan brought up in his last reply, and they seem like
> valid concerns to me. But the risk and the downsides of "atrophy" are
> serious enough that I'd vote to not risk them until this starts to
> cause us issues in practice. Even if, for now, that means we won't be
> able to build a single plugin jar that supports (e.g.) 3 major Solr
> versions. IMO that's a much smaller loss.
>
> On Tue, Feb 16, 2021 at 9:40 AM David Smiley <dsmiley@apache.org> wrote:
> >
> > On Tue, Feb 16, 2021 at 8:38 AM Eric Pugh <
> epugh@opensourceconnections.com> wrote:
> >>
> >> Testing across multiple versions is always very difficult ;-). I
> recently saw this very interesting approach to using our Dockerized Solr’s
> to test a component against a number of previous versions of Solr.
> https://github.com/querqy/querqy/pull/147. I’m hopeful it could be a
> model for other packages to adopt.
> >
> >
> > Thanks for the link to that Querqy PR. That is *very* similar to what I
> do at work (minus multi-version testing), and also similar to how I test
> multiple versions in one of my pet projects by using a CI build matrix of a
> configurable dependency. I didn't know Testcontainer.org has its own Solr
> module -- https://www.testcontainers.org/modules/solr/ but we implemented
> that ourselves; not hard.
> >
> >>
> >> Trying to maintain across multiple versions is kind of a Sisyphean
> task, and I don’t think plays to open source communities strengths. It’s
> hard enough to keep all components of Solr up to date with the latest
> Lucene and the latest Solr…. (Try out wt=xlsx Response Writer, it’s
> currently broken on master) . Reminds me of the Apache Gump project ;-)
> >>
> >> If something is really going to be backcompatible across certain
> versions, then maybe having it’s own repo makes sense,
> >
> >
> > I'd prefer that a module only leave our "contribs" area when the
> concerns/limitations become real. Doing it prematurely could lead to
> atrophy of the module....
> >
> >>
> >> but I suspect it means those components may go stale…. Look at DIH
> and Velocity components that are moved over to their own repos, both are
> getting stale, and I’d argue it’s because they don’t live in the main
> project where all of us have oversight and easy access.
> >
> >
> > Agreed :-(
> >
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: dev-help@lucene.apache.org
>
>

--
-----------------------------------------------------
Noble Paul
Re: Separate git repo(s) for Solr modules [ In reply to ]
In the future we wont be able to “work on both at the same time”, once
Lucene 9 is cut. Why not pull that bandaid now?

On Mon, May 3, 2021 at 11:32 PM Noble Paul <noble.paul@gmail.com> wrote:

> I'm still struggling to understand the workflow when I'm working on a
> feature that spans lucene and solr.
>
> I'm yet to see an argument against sub-modules
>
> On Wed, Feb 17, 2021 at 3:18 AM Jason Gerlowski <gerlowskija@gmail.com>
> wrote:
>
>> > Shoving such a component into lucene-solr repo makes no sense, given
>> its branching strategy is based on master / branch_8x
>>
>> I get how this could cause issues - I def hadn't thought much about
>> multi-version support and branching. But I don't think moving plugins
>> to a separate repo solves that problem for us. If our first class
>> plugins are set up to have different release "lines" that don't line
>> up with major Solr versions, it's only a matter of time before two of
>> those plugins have branch requirements that conflict. Unless I'm
>> missing something here?
>>
>> > I'd prefer that a module only leave our "contribs" area when the
>> concerns/limitations become real. Doing it prematurely could lead to
>> atrophy of the module....
>>
>> +1 to David's comment. I def hadn't considered the branching-scheme
>> issues that Ishan brought up in his last reply, and they seem like
>> valid concerns to me. But the risk and the downsides of "atrophy" are
>> serious enough that I'd vote to not risk them until this starts to
>> cause us issues in practice. Even if, for now, that means we won't be
>> able to build a single plugin jar that supports (e.g.) 3 major Solr
>> versions. IMO that's a much smaller loss.
>>
>> On Tue, Feb 16, 2021 at 9:40 AM David Smiley <dsmiley@apache.org> wrote:
>> >
>> > On Tue, Feb 16, 2021 at 8:38 AM Eric Pugh <
>> epugh@opensourceconnections.com> wrote:
>> >>
>> >> Testing across multiple versions is always very difficult ;-). I
>> recently saw this very interesting approach to using our Dockerized Solr’s
>> to test a component against a number of previous versions of Solr.
>> https://github.com/querqy/querqy/pull/147. I’m hopeful it could be a
>> model for other packages to adopt.
>> >
>> >
>> > Thanks for the link to that Querqy PR. That is *very* similar to what
>> I do at work (minus multi-version testing), and also similar to how I test
>> multiple versions in one of my pet projects by using a CI build matrix of a
>> configurable dependency. I didn't know Testcontainer.org has its own Solr
>> module -- https://www.testcontainers.org/modules/solr/ but we
>> implemented that ourselves; not hard.
>> >
>> >>
>> >> Trying to maintain across multiple versions is kind of a Sisyphean
>> task, and I don’t think plays to open source communities strengths. It’s
>> hard enough to keep all components of Solr up to date with the latest
>> Lucene and the latest Solr…. (Try out wt=xlsx Response Writer, it’s
>> currently broken on master) . Reminds me of the Apache Gump project ;-)
>> >>
>> >> If something is really going to be backcompatible across certain
>> versions, then maybe having it’s own repo makes sense,
>> >
>> >
>> > I'd prefer that a module only leave our "contribs" area when the
>> concerns/limitations become real. Doing it prematurely could lead to
>> atrophy of the module....
>> >
>> >>
>> >> but I suspect it means those components may go stale…. Look at DIH
>> and Velocity components that are moved over to their own repos, both are
>> getting stale, and I’d argue it’s because they don’t live in the main
>> project where all of us have oversight and easy access.
>> >
>> >
>> > Agreed :-(
>> >
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: dev-help@lucene.apache.org
>>
>>
>
> --
> -----------------------------------------------------
> Noble Paul
>
Re: Separate git repo(s) for Solr modules [ In reply to ]
@Houston

So, Are you suggesting we should not do that ?

On Tue, May 4, 2021 at 2:35 PM Houston Putman <houstonputman@gmail.com>
wrote:

> In the future we wont be able to “work on both at the same time”, once
> Lucene 9 is cut. Why not pull that bandaid now?
>
> On Mon, May 3, 2021 at 11:32 PM Noble Paul <noble.paul@gmail.com> wrote:
>
>> I'm still struggling to understand the workflow when I'm working on a
>> feature that spans lucene and solr.
>>
>> I'm yet to see an argument against sub-modules
>>
>> On Wed, Feb 17, 2021 at 3:18 AM Jason Gerlowski <gerlowskija@gmail.com>
>> wrote:
>>
>>> > Shoving such a component into lucene-solr repo makes no sense, given
>>> its branching strategy is based on master / branch_8x
>>>
>>> I get how this could cause issues - I def hadn't thought much about
>>> multi-version support and branching. But I don't think moving plugins
>>> to a separate repo solves that problem for us. If our first class
>>> plugins are set up to have different release "lines" that don't line
>>> up with major Solr versions, it's only a matter of time before two of
>>> those plugins have branch requirements that conflict. Unless I'm
>>> missing something here?
>>>
>>> > I'd prefer that a module only leave our "contribs" area when the
>>> concerns/limitations become real. Doing it prematurely could lead to
>>> atrophy of the module....
>>>
>>> +1 to David's comment. I def hadn't considered the branching-scheme
>>> issues that Ishan brought up in his last reply, and they seem like
>>> valid concerns to me. But the risk and the downsides of "atrophy" are
>>> serious enough that I'd vote to not risk them until this starts to
>>> cause us issues in practice. Even if, for now, that means we won't be
>>> able to build a single plugin jar that supports (e.g.) 3 major Solr
>>> versions. IMO that's a much smaller loss.
>>>
>>> On Tue, Feb 16, 2021 at 9:40 AM David Smiley <dsmiley@apache.org> wrote:
>>> >
>>> > On Tue, Feb 16, 2021 at 8:38 AM Eric Pugh <
>>> epugh@opensourceconnections.com> wrote:
>>> >>
>>> >> Testing across multiple versions is always very difficult ;-). I
>>> recently saw this very interesting approach to using our Dockerized Solr’s
>>> to test a component against a number of previous versions of Solr.
>>> https://github.com/querqy/querqy/pull/147. I’m hopeful it could be a
>>> model for other packages to adopt.
>>> >
>>> >
>>> > Thanks for the link to that Querqy PR. That is *very* similar to what
>>> I do at work (minus multi-version testing), and also similar to how I test
>>> multiple versions in one of my pet projects by using a CI build matrix of a
>>> configurable dependency. I didn't know Testcontainer.org has its own Solr
>>> module -- https://www.testcontainers.org/modules/solr/ but we
>>> implemented that ourselves; not hard.
>>> >
>>> >>
>>> >> Trying to maintain across multiple versions is kind of a Sisyphean
>>> task, and I don’t think plays to open source communities strengths. It’s
>>> hard enough to keep all components of Solr up to date with the latest
>>> Lucene and the latest Solr…. (Try out wt=xlsx Response Writer, it’s
>>> currently broken on master) . Reminds me of the Apache Gump project ;-)
>>> >>
>>> >> If something is really going to be backcompatible across certain
>>> versions, then maybe having it’s own repo makes sense,
>>> >
>>> >
>>> > I'd prefer that a module only leave our "contribs" area when the
>>> concerns/limitations become real. Doing it prematurely could lead to
>>> atrophy of the module....
>>> >
>>> >>
>>> >> but I suspect it means those components may go stale…. Look at DIH
>>> and Velocity components that are moved over to their own repos, both are
>>> getting stale, and I’d argue it’s because they don’t live in the main
>>> project where all of us have oversight and easy access.
>>> >
>>> >
>>> > Agreed :-(
>>> >
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
>>> For additional commands, e-mail: dev-help@lucene.apache.org
>>>
>>>
>>
>> --
>> -----------------------------------------------------
>> Noble Paul
>>
>

--
-----------------------------------------------------
Noble Paul
Re: Separate git repo(s) for Solr modules [ In reply to ]
As with any dependency on any project, you update the dependency project
first then consume the updated dependency in Solr.

If the idea is to be able to modify Lucene and Solr in parallel, then the
project split is counterproductive.

From the Solr perspective, Lucene and Zookerper are really two “similar”
dependencies and IMO we should think about them in that way.

Ilan

On Tue 4 May 2021 at 09:45, Noble Paul <noble.paul@gmail.com> wrote:

> @Houston
>
> So, Are you suggesting we should not do that ?
>
> On Tue, May 4, 2021 at 2:35 PM Houston Putman <houstonputman@gmail.com>
> wrote:
>
>> In the future we wont be able to “work on both at the same time”, once
>> Lucene 9 is cut. Why not pull that bandaid now?
>>
>> On Mon, May 3, 2021 at 11:32 PM Noble Paul <noble.paul@gmail.com> wrote:
>>
>>> I'm still struggling to understand the workflow when I'm working on a
>>> feature that spans lucene and solr.
>>>
>>> I'm yet to see an argument against sub-modules
>>>
>>> On Wed, Feb 17, 2021 at 3:18 AM Jason Gerlowski <gerlowskija@gmail.com>
>>> wrote:
>>>
>>>> > Shoving such a component into lucene-solr repo makes no sense, given
>>>> its branching strategy is based on master / branch_8x
>>>>
>>>> I get how this could cause issues - I def hadn't thought much about
>>>> multi-version support and branching. But I don't think moving plugins
>>>> to a separate repo solves that problem for us. If our first class
>>>> plugins are set up to have different release "lines" that don't line
>>>> up with major Solr versions, it's only a matter of time before two of
>>>> those plugins have branch requirements that conflict. Unless I'm
>>>> missing something here?
>>>>
>>>> > I'd prefer that a module only leave our "contribs" area when the
>>>> concerns/limitations become real. Doing it prematurely could lead to
>>>> atrophy of the module....
>>>>
>>>> +1 to David's comment. I def hadn't considered the branching-scheme
>>>> issues that Ishan brought up in his last reply, and they seem like
>>>> valid concerns to me. But the risk and the downsides of "atrophy" are
>>>> serious enough that I'd vote to not risk them until this starts to
>>>> cause us issues in practice. Even if, for now, that means we won't be
>>>> able to build a single plugin jar that supports (e.g.) 3 major Solr
>>>> versions. IMO that's a much smaller loss.
>>>>
>>>> On Tue, Feb 16, 2021 at 9:40 AM David Smiley <dsmiley@apache.org>
>>>> wrote:
>>>> >
>>>> > On Tue, Feb 16, 2021 at 8:38 AM Eric Pugh <
>>>> epugh@opensourceconnections.com> wrote:
>>>> >>
>>>> >> Testing across multiple versions is always very difficult ;-). I
>>>> recently saw this very interesting approach to using our Dockerized Solr’s
>>>> to test a component against a number of previous versions of Solr.
>>>> https://github.com/querqy/querqy/pull/147. I’m hopeful it could be a
>>>> model for other packages to adopt.
>>>> >
>>>> >
>>>> > Thanks for the link to that Querqy PR. That is *very* similar to
>>>> what I do at work (minus multi-version testing), and also similar to how I
>>>> test multiple versions in one of my pet projects by using a CI build matrix
>>>> of a configurable dependency. I didn't know Testcontainer.org has its own
>>>> Solr module -- https://www.testcontainers.org/modules/solr/ but we
>>>> implemented that ourselves; not hard.
>>>> >
>>>> >>
>>>> >> Trying to maintain across multiple versions is kind of a Sisyphean
>>>> task, and I don’t think plays to open source communities strengths. It’s
>>>> hard enough to keep all components of Solr up to date with the latest
>>>> Lucene and the latest Solr…. (Try out wt=xlsx Response Writer, it’s
>>>> currently broken on master) . Reminds me of the Apache Gump project ;-)
>>>> >>
>>>> >> If something is really going to be backcompatible across certain
>>>> versions, then maybe having it’s own repo makes sense,
>>>> >
>>>> >
>>>> > I'd prefer that a module only leave our "contribs" area when the
>>>> concerns/limitations become real. Doing it prematurely could lead to
>>>> atrophy of the module....
>>>> >
>>>> >>
>>>> >> but I suspect it means those components may go stale…. Look at DIH
>>>> and Velocity components that are moved over to their own repos, both are
>>>> getting stale, and I’d argue it’s because they don’t live in the main
>>>> project where all of us have oversight and easy access.
>>>> >
>>>> >
>>>> > Agreed :-(
>>>> >
>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
>>>> For additional commands, e-mail: dev-help@lucene.apache.org
>>>>
>>>>
>>>
>>> --
>>> -----------------------------------------------------
>>> Noble Paul
>>>
>>
>
> --
> -----------------------------------------------------
> Noble Paul
>
Re: Separate git repo(s) for Solr modules [ In reply to ]
I completely agree with Ilan.

Keeping the projects in the same repo was the only long term viable way of
having Solr depend on a snapshot version of Lucene.

And I'm sorry if I misspoke earlier. You can always "work on both at
the same time" even when we rely on a release version of Lucene,
but you won't be able to commit the Solr code until a Lucene release has
been made that includes those changes.

Also I think discussion has taken over the wrong thread. The thread was
originally about modules for various Solr plugins and projects.

- Houston

On Tue, May 4, 2021 at 7:02 AM Ilan Ginzburg <ilansolr@gmail.com> wrote:

> As with any dependency on any project, you update the dependency project
> first then consume the updated dependency in Solr.
>
> If the idea is to be able to modify Lucene and Solr in parallel, then the
> project split is counterproductive.
>
> From the Solr perspective, Lucene and Zookerper are really two “similar”
> dependencies and IMO we should think about them in that way.
>
> Ilan
>
> On Tue 4 May 2021 at 09:45, Noble Paul <noble.paul@gmail.com> wrote:
>
>> @Houston
>>
>> So, Are you suggesting we should not do that ?
>>
>> On Tue, May 4, 2021 at 2:35 PM Houston Putman <houstonputman@gmail.com>
>> wrote:
>>
>>> In the future we wont be able to “work on both at the same time”, once
>>> Lucene 9 is cut. Why not pull that bandaid now?
>>>
>>> On Mon, May 3, 2021 at 11:32 PM Noble Paul <noble.paul@gmail.com> wrote:
>>>
>>>> I'm still struggling to understand the workflow when I'm working on a
>>>> feature that spans lucene and solr.
>>>>
>>>> I'm yet to see an argument against sub-modules
>>>>
>>>> On Wed, Feb 17, 2021 at 3:18 AM Jason Gerlowski <gerlowskija@gmail.com>
>>>> wrote:
>>>>
>>>>> > Shoving such a component into lucene-solr repo makes no sense, given
>>>>> its branching strategy is based on master / branch_8x
>>>>>
>>>>> I get how this could cause issues - I def hadn't thought much about
>>>>> multi-version support and branching. But I don't think moving plugins
>>>>> to a separate repo solves that problem for us. If our first class
>>>>> plugins are set up to have different release "lines" that don't line
>>>>> up with major Solr versions, it's only a matter of time before two of
>>>>> those plugins have branch requirements that conflict. Unless I'm
>>>>> missing something here?
>>>>>
>>>>> > I'd prefer that a module only leave our "contribs" area when the
>>>>> concerns/limitations become real. Doing it prematurely could lead to
>>>>> atrophy of the module....
>>>>>
>>>>> +1 to David's comment. I def hadn't considered the branching-scheme
>>>>> issues that Ishan brought up in his last reply, and they seem like
>>>>> valid concerns to me. But the risk and the downsides of "atrophy" are
>>>>> serious enough that I'd vote to not risk them until this starts to
>>>>> cause us issues in practice. Even if, for now, that means we won't be
>>>>> able to build a single plugin jar that supports (e.g.) 3 major Solr
>>>>> versions. IMO that's a much smaller loss.
>>>>>
>>>>> On Tue, Feb 16, 2021 at 9:40 AM David Smiley <dsmiley@apache.org>
>>>>> wrote:
>>>>> >
>>>>> > On Tue, Feb 16, 2021 at 8:38 AM Eric Pugh <
>>>>> epugh@opensourceconnections.com> wrote:
>>>>> >>
>>>>> >> Testing across multiple versions is always very difficult ;-). I
>>>>> recently saw this very interesting approach to using our Dockerized Solr’s
>>>>> to test a component against a number of previous versions of Solr.
>>>>> https://github.com/querqy/querqy/pull/147. I’m hopeful it could be a
>>>>> model for other packages to adopt.
>>>>> >
>>>>> >
>>>>> > Thanks for the link to that Querqy PR. That is *very* similar to
>>>>> what I do at work (minus multi-version testing), and also similar to how I
>>>>> test multiple versions in one of my pet projects by using a CI build matrix
>>>>> of a configurable dependency. I didn't know Testcontainer.org has its own
>>>>> Solr module -- https://www.testcontainers.org/modules/solr/ but we
>>>>> implemented that ourselves; not hard.
>>>>> >
>>>>> >>
>>>>> >> Trying to maintain across multiple versions is kind of a Sisyphean
>>>>> task, and I don’t think plays to open source communities strengths. It’s
>>>>> hard enough to keep all components of Solr up to date with the latest
>>>>> Lucene and the latest Solr…. (Try out wt=xlsx Response Writer, it’s
>>>>> currently broken on master) . Reminds me of the Apache Gump project ;-)
>>>>> >>
>>>>> >> If something is really going to be backcompatible across certain
>>>>> versions, then maybe having it’s own repo makes sense,
>>>>> >
>>>>> >
>>>>> > I'd prefer that a module only leave our "contribs" area when the
>>>>> concerns/limitations become real. Doing it prematurely could lead to
>>>>> atrophy of the module....
>>>>> >
>>>>> >>
>>>>> >> but I suspect it means those components may go stale…. Look at
>>>>> DIH and Velocity components that are moved over to their own repos, both
>>>>> are getting stale, and I’d argue it’s because they don’t live in the main
>>>>> project where all of us have oversight and easy access.
>>>>> >
>>>>> >
>>>>> > Agreed :-(
>>>>> >
>>>>>
>>>>> ---------------------------------------------------------------------
>>>>> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
>>>>> For additional commands, e-mail: dev-help@lucene.apache.org
>>>>>
>>>>>
>>>>
>>>> --
>>>> -----------------------------------------------------
>>>> Noble Paul
>>>>
>>>
>>
>> --
>> -----------------------------------------------------
>> Noble Paul
>>
>
Re: Separate git repo(s) for Solr modules [ In reply to ]
@Ilan Ginzburg <ilansolr@gmail.com>

I don't think the project split is counter productive because we have
lucene as a sub module. Solr does not use lucene like a simple library.
It's an integral part of Solr


On Tue, May 4, 2021 at 10:02 PM Ilan Ginzburg <ilansolr@gmail.com> wrote:

> As with any dependency on any project, you update the dependency project
> first then consume the updated dependency in Solr.
>
> If the idea is to be able to modify Lucene and Solr in parallel, then the
> project split is counterproductive.
>
> From the Solr perspective, Lucene and Zookerper are really two “similar”
> dependencies and IMO we should think about them in that way.
>
> Ilan
>
> On Tue 4 May 2021 at 09:45, Noble Paul <noble.paul@gmail.com> wrote:
>
>> @Houston
>>
>> So, Are you suggesting we should not do that ?
>>
>> On Tue, May 4, 2021 at 2:35 PM Houston Putman <houstonputman@gmail.com>
>> wrote:
>>
>>> In the future we wont be able to “work on both at the same time”, once
>>> Lucene 9 is cut. Why not pull that bandaid now?
>>>
>>> On Mon, May 3, 2021 at 11:32 PM Noble Paul <noble.paul@gmail.com> wrote:
>>>
>>>> I'm still struggling to understand the workflow when I'm working on a
>>>> feature that spans lucene and solr.
>>>>
>>>> I'm yet to see an argument against sub-modules
>>>>
>>>> On Wed, Feb 17, 2021 at 3:18 AM Jason Gerlowski <gerlowskija@gmail.com>
>>>> wrote:
>>>>
>>>>> > Shoving such a component into lucene-solr repo makes no sense, given
>>>>> its branching strategy is based on master / branch_8x
>>>>>
>>>>> I get how this could cause issues - I def hadn't thought much about
>>>>> multi-version support and branching. But I don't think moving plugins
>>>>> to a separate repo solves that problem for us. If our first class
>>>>> plugins are set up to have different release "lines" that don't line
>>>>> up with major Solr versions, it's only a matter of time before two of
>>>>> those plugins have branch requirements that conflict. Unless I'm
>>>>> missing something here?
>>>>>
>>>>> > I'd prefer that a module only leave our "contribs" area when the
>>>>> concerns/limitations become real. Doing it prematurely could lead to
>>>>> atrophy of the module....
>>>>>
>>>>> +1 to David's comment. I def hadn't considered the branching-scheme
>>>>> issues that Ishan brought up in his last reply, and they seem like
>>>>> valid concerns to me. But the risk and the downsides of "atrophy" are
>>>>> serious enough that I'd vote to not risk them until this starts to
>>>>> cause us issues in practice. Even if, for now, that means we won't be
>>>>> able to build a single plugin jar that supports (e.g.) 3 major Solr
>>>>> versions. IMO that's a much smaller loss.
>>>>>
>>>>> On Tue, Feb 16, 2021 at 9:40 AM David Smiley <dsmiley@apache.org>
>>>>> wrote:
>>>>> >
>>>>> > On Tue, Feb 16, 2021 at 8:38 AM Eric Pugh <
>>>>> epugh@opensourceconnections.com> wrote:
>>>>> >>
>>>>> >> Testing across multiple versions is always very difficult ;-). I
>>>>> recently saw this very interesting approach to using our Dockerized Solr’s
>>>>> to test a component against a number of previous versions of Solr.
>>>>> https://github.com/querqy/querqy/pull/147. I’m hopeful it could be a
>>>>> model for other packages to adopt.
>>>>> >
>>>>> >
>>>>> > Thanks for the link to that Querqy PR. That is *very* similar to
>>>>> what I do at work (minus multi-version testing), and also similar to how I
>>>>> test multiple versions in one of my pet projects by using a CI build matrix
>>>>> of a configurable dependency. I didn't know Testcontainer.org has its own
>>>>> Solr module -- https://www.testcontainers.org/modules/solr/ but we
>>>>> implemented that ourselves; not hard.
>>>>> >
>>>>> >>
>>>>> >> Trying to maintain across multiple versions is kind of a Sisyphean
>>>>> task, and I don’t think plays to open source communities strengths. It’s
>>>>> hard enough to keep all components of Solr up to date with the latest
>>>>> Lucene and the latest Solr…. (Try out wt=xlsx Response Writer, it’s
>>>>> currently broken on master) . Reminds me of the Apache Gump project ;-)
>>>>> >>
>>>>> >> If something is really going to be backcompatible across certain
>>>>> versions, then maybe having it’s own repo makes sense,
>>>>> >
>>>>> >
>>>>> > I'd prefer that a module only leave our "contribs" area when the
>>>>> concerns/limitations become real. Doing it prematurely could lead to
>>>>> atrophy of the module....
>>>>> >
>>>>> >>
>>>>> >> but I suspect it means those components may go stale…. Look at
>>>>> DIH and Velocity components that are moved over to their own repos, both
>>>>> are getting stale, and I’d argue it’s because they don’t live in the main
>>>>> project where all of us have oversight and easy access.
>>>>> >
>>>>> >
>>>>> > Agreed :-(
>>>>> >
>>>>>
>>>>> ---------------------------------------------------------------------
>>>>> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
>>>>> For additional commands, e-mail: dev-help@lucene.apache.org
>>>>>
>>>>>
>>>>
>>>> --
>>>> -----------------------------------------------------
>>>> Noble Paul
>>>>
>>>
>>
>> --
>> -----------------------------------------------------
>> Noble Paul
>>
>

--
-----------------------------------------------------
Noble Paul
Re: Separate git repo(s) for Solr modules [ In reply to ]
I am a bit confused -- if Lucene was to become a sub module of Solr, are we
essentially forking Lucene?

I am in agreement with Ilan and Houston -- Solr should be depending on
Lucene only as a dependency.


On Tue, 4 May 2021, 19:52 Noble Paul, <noble.paul@gmail.com> wrote:

> @Ilan Ginzburg <ilansolr@gmail.com>
>
> I don't think the project split is counter productive because we have
> lucene as a sub module. Solr does not use lucene like a simple library.
> It's an integral part of Solr
>
>
> On Tue, May 4, 2021 at 10:02 PM Ilan Ginzburg <ilansolr@gmail.com> wrote:
>
>> As with any dependency on any project, you update the dependency project
>> first then consume the updated dependency in Solr.
>>
>> If the idea is to be able to modify Lucene and Solr in parallel, then the
>> project split is counterproductive.
>>
>> From the Solr perspective, Lucene and Zookerper are really two “similar”
>> dependencies and IMO we should think about them in that way.
>>
>> Ilan
>>
>> On Tue 4 May 2021 at 09:45, Noble Paul <noble.paul@gmail.com> wrote:
>>
>>> @Houston
>>>
>>> So, Are you suggesting we should not do that ?
>>>
>>> On Tue, May 4, 2021 at 2:35 PM Houston Putman <houstonputman@gmail.com>
>>> wrote:
>>>
>>>> In the future we wont be able to “work on both at the same time”, once
>>>> Lucene 9 is cut. Why not pull that bandaid now?
>>>>
>>>> On Mon, May 3, 2021 at 11:32 PM Noble Paul <noble.paul@gmail.com>
>>>> wrote:
>>>>
>>>>> I'm still struggling to understand the workflow when I'm working on a
>>>>> feature that spans lucene and solr.
>>>>>
>>>>> I'm yet to see an argument against sub-modules
>>>>>
>>>>> On Wed, Feb 17, 2021 at 3:18 AM Jason Gerlowski <gerlowskija@gmail.com>
>>>>> wrote:
>>>>>
>>>>>> > Shoving such a component into lucene-solr repo makes no sense,
>>>>>> given its branching strategy is based on master / branch_8x
>>>>>>
>>>>>> I get how this could cause issues - I def hadn't thought much about
>>>>>> multi-version support and branching. But I don't think moving plugins
>>>>>> to a separate repo solves that problem for us. If our first class
>>>>>> plugins are set up to have different release "lines" that don't line
>>>>>> up with major Solr versions, it's only a matter of time before two of
>>>>>> those plugins have branch requirements that conflict. Unless I'm
>>>>>> missing something here?
>>>>>>
>>>>>> > I'd prefer that a module only leave our "contribs" area when the
>>>>>> concerns/limitations become real. Doing it prematurely could lead to
>>>>>> atrophy of the module....
>>>>>>
>>>>>> +1 to David's comment. I def hadn't considered the branching-scheme
>>>>>> issues that Ishan brought up in his last reply, and they seem like
>>>>>> valid concerns to me. But the risk and the downsides of "atrophy" are
>>>>>> serious enough that I'd vote to not risk them until this starts to
>>>>>> cause us issues in practice. Even if, for now, that means we won't be
>>>>>> able to build a single plugin jar that supports (e.g.) 3 major Solr
>>>>>> versions. IMO that's a much smaller loss.
>>>>>>
>>>>>> On Tue, Feb 16, 2021 at 9:40 AM David Smiley <dsmiley@apache.org>
>>>>>> wrote:
>>>>>> >
>>>>>> > On Tue, Feb 16, 2021 at 8:38 AM Eric Pugh <
>>>>>> epugh@opensourceconnections.com> wrote:
>>>>>> >>
>>>>>> >> Testing across multiple versions is always very difficult ;-). I
>>>>>> recently saw this very interesting approach to using our Dockerized Solr’s
>>>>>> to test a component against a number of previous versions of Solr.
>>>>>> https://github.com/querqy/querqy/pull/147. I’m hopeful it could be a
>>>>>> model for other packages to adopt.
>>>>>> >
>>>>>> >
>>>>>> > Thanks for the link to that Querqy PR. That is *very* similar to
>>>>>> what I do at work (minus multi-version testing), and also similar to how I
>>>>>> test multiple versions in one of my pet projects by using a CI build matrix
>>>>>> of a configurable dependency. I didn't know Testcontainer.org has its own
>>>>>> Solr module -- https://www.testcontainers.org/modules/solr/ but we
>>>>>> implemented that ourselves; not hard.
>>>>>> >
>>>>>> >>
>>>>>> >> Trying to maintain across multiple versions is kind of a Sisyphean
>>>>>> task, and I don’t think plays to open source communities strengths. It’s
>>>>>> hard enough to keep all components of Solr up to date with the latest
>>>>>> Lucene and the latest Solr…. (Try out wt=xlsx Response Writer, it’s
>>>>>> currently broken on master) . Reminds me of the Apache Gump project ;-)
>>>>>> >>
>>>>>> >> If something is really going to be backcompatible across certain
>>>>>> versions, then maybe having it’s own repo makes sense,
>>>>>> >
>>>>>> >
>>>>>> > I'd prefer that a module only leave our "contribs" area when the
>>>>>> concerns/limitations become real. Doing it prematurely could lead to
>>>>>> atrophy of the module....
>>>>>> >
>>>>>> >>
>>>>>> >> but I suspect it means those components may go stale…. Look at
>>>>>> DIH and Velocity components that are moved over to their own repos, both
>>>>>> are getting stale, and I’d argue it’s because they don’t live in the main
>>>>>> project where all of us have oversight and easy access.
>>>>>> >
>>>>>> >
>>>>>> > Agreed :-(
>>>>>> >
>>>>>>
>>>>>> ---------------------------------------------------------------------
>>>>>> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
>>>>>> For additional commands, e-mail: dev-help@lucene.apache.org
>>>>>>
>>>>>>
>>>>>
>>>>> --
>>>>> -----------------------------------------------------
>>>>> Noble Paul
>>>>>
>>>>
>>>
>>> --
>>> -----------------------------------------------------
>>> Noble Paul
>>>
>>
>
> --
> -----------------------------------------------------
> Noble Paul
>
Re: Separate git repo(s) for Solr modules [ In reply to ]
A submodule is nothing more than a checkout of a different repository
(at a given commit). It's not a fork, unless you clone that repository
and place it somewhere else.

In any case, I agree Solr should stick to official Lucene releases. If
there is a need to have a "sticky" intermediate tagged release of
Lucene for internal (development branch) purposes, Cassandra showed
what seems like an adequate place to host it until an official Lucene
release is out.

If you're working on both projects at once and wish to modify code in both,
IDEs in many cases help you out by substituting code/classpaths from multiple
sources. I know Eclipse can do it, I'm pretty sure you could try to do
the same with
IntelliJ.

Dawid

On Tue, May 4, 2021 at 4:40 PM Atri Sharma <atri@apache.org> wrote:
>
> I am a bit confused -- if Lucene was to become a sub module of Solr, are we essentially forking Lucene?
>
> I am in agreement with Ilan and Houston -- Solr should be depending on Lucene only as a dependency.
>
>
> On Tue, 4 May 2021, 19:52 Noble Paul, <noble.paul@gmail.com> wrote:
>>
>> @Ilan Ginzburg
>>
>> I don't think the project split is counter productive because we have lucene as a sub module. Solr does not use lucene like a simple library. It's an integral part of Solr
>>
>>
>> On Tue, May 4, 2021 at 10:02 PM Ilan Ginzburg <ilansolr@gmail.com> wrote:
>>>
>>> As with any dependency on any project, you update the dependency project first then consume the updated dependency in Solr.
>>>
>>> If the idea is to be able to modify Lucene and Solr in parallel, then the project split is counterproductive.
>>>
>>> From the Solr perspective, Lucene and Zookerper are really two “similar”
>>> dependencies and IMO we should think about them in that way.
>>>
>>> Ilan
>>>
>>> On Tue 4 May 2021 at 09:45, Noble Paul <noble.paul@gmail.com> wrote:
>>>>
>>>> @Houston
>>>>
>>>> So, Are you suggesting we should not do that ?
>>>>
>>>> On Tue, May 4, 2021 at 2:35 PM Houston Putman <houstonputman@gmail.com> wrote:
>>>>>
>>>>> In the future we wont be able to “work on both at the same time”, once Lucene 9 is cut. Why not pull that bandaid now?
>>>>>
>>>>> On Mon, May 3, 2021 at 11:32 PM Noble Paul <noble.paul@gmail.com> wrote:
>>>>>>
>>>>>> I'm still struggling to understand the workflow when I'm working on a feature that spans lucene and solr.
>>>>>>
>>>>>> I'm yet to see an argument against sub-modules
>>>>>>
>>>>>> On Wed, Feb 17, 2021 at 3:18 AM Jason Gerlowski <gerlowskija@gmail.com> wrote:
>>>>>>>
>>>>>>> > Shoving such a component into lucene-solr repo makes no sense, given its branching strategy is based on master / branch_8x
>>>>>>>
>>>>>>> I get how this could cause issues - I def hadn't thought much about
>>>>>>> multi-version support and branching. But I don't think moving plugins
>>>>>>> to a separate repo solves that problem for us. If our first class
>>>>>>> plugins are set up to have different release "lines" that don't line
>>>>>>> up with major Solr versions, it's only a matter of time before two of
>>>>>>> those plugins have branch requirements that conflict. Unless I'm
>>>>>>> missing something here?
>>>>>>>
>>>>>>> > I'd prefer that a module only leave our "contribs" area when the concerns/limitations become real. Doing it prematurely could lead to atrophy of the module....
>>>>>>>
>>>>>>> +1 to David's comment. I def hadn't considered the branching-scheme
>>>>>>> issues that Ishan brought up in his last reply, and they seem like
>>>>>>> valid concerns to me. But the risk and the downsides of "atrophy" are
>>>>>>> serious enough that I'd vote to not risk them until this starts to
>>>>>>> cause us issues in practice. Even if, for now, that means we won't be
>>>>>>> able to build a single plugin jar that supports (e.g.) 3 major Solr
>>>>>>> versions. IMO that's a much smaller loss.
>>>>>>>
>>>>>>> On Tue, Feb 16, 2021 at 9:40 AM David Smiley <dsmiley@apache.org> wrote:
>>>>>>> >
>>>>>>> > On Tue, Feb 16, 2021 at 8:38 AM Eric Pugh <epugh@opensourceconnections.com> wrote:
>>>>>>> >>
>>>>>>> >> Testing across multiple versions is always very difficult ;-). I recently saw this very interesting approach to using our Dockerized Solr’s to test a component against a number of previous versions of Solr. https://github.com/querqy/querqy/pull/147. I’m hopeful it could be a model for other packages to adopt.
>>>>>>> >
>>>>>>> >
>>>>>>> > Thanks for the link to that Querqy PR. That is *very* similar to what I do at work (minus multi-version testing), and also similar to how I test multiple versions in one of my pet projects by using a CI build matrix of a configurable dependency. I didn't know Testcontainer.org has its own Solr module -- https://www.testcontainers.org/modules/solr/ but we implemented that ourselves; not hard.
>>>>>>> >
>>>>>>> >>
>>>>>>> >> Trying to maintain across multiple versions is kind of a Sisyphean task, and I don’t think plays to open source communities strengths. It’s hard enough to keep all components of Solr up to date with the latest Lucene and the latest Solr…. (Try out wt=xlsx Response Writer, it’s currently broken on master) . Reminds me of the Apache Gump project ;-)
>>>>>>> >>
>>>>>>> >> If something is really going to be backcompatible across certain versions, then maybe having it’s own repo makes sense,
>>>>>>> >
>>>>>>> >
>>>>>>> > I'd prefer that a module only leave our "contribs" area when the concerns/limitations become real. Doing it prematurely could lead to atrophy of the module....
>>>>>>> >
>>>>>>> >>
>>>>>>> >> but I suspect it means those components may go stale…. Look at DIH and Velocity components that are moved over to their own repos, both are getting stale, and I’d argue it’s because they don’t live in the main project where all of us have oversight and easy access.
>>>>>>> >
>>>>>>> >
>>>>>>> > Agreed :-(
>>>>>>> >
>>>>>>>
>>>>>>> ---------------------------------------------------------------------
>>>>>>> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
>>>>>>> For additional commands, e-mail: dev-help@lucene.apache.org
>>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> -----------------------------------------------------
>>>>>> Noble Paul
>>>>
>>>>
>>>>
>>>> --
>>>> -----------------------------------------------------
>>>> Noble Paul
>>
>>
>>
>> --
>> -----------------------------------------------------
>> Noble Paul

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org

1 2  View All