Mailing List Archive

Solr Docker discussion
There's a few decisions that need to be ironed out around the Solr docker
image before 9.0 is released. This is because the community has decided
that Solr should start releasing it's own docker images starting with 9.0.

Below is the current state of the ongoing discussions for the Solr Docker
image. Please feel free to correct me or fill in any information I may be
missing.

Where does this image live?

There are two options for this really.

- _/solr - docker run solr:9.0 (Official Docker Image)
- apache/solr - docker run apache/solr:9.0

The benefits of the first are 1) the nice usability of being able to
plainly specify "solr" and 2) the "Docker Official Images" badge on
DockerHub. The downsides are that there are very strict requirements with
creating Official Docker Images, which would complicate and require
separating the way that we build release docker images and local docker
images.

The benefits of using the apache namespace is that we can build the image
in any way that we want. We would be able to build release and local docker
images the exact same way. The downside is the loss of the "Docker Official
Images" badge.

*I think there is some consensus that choosing the "apache/solr" location
is fine, and worth the added flexibility we get in the build process.*

Legal Stuff

There are a few legal questions we need to keep in mind when creating this
process and doing a release for the first time.

- Source release - The apache policy is: (from Michael Sokolov)

“Every ASF release MUST contain one or more source packages, which MUST
> be sufficient for a user to build and test the release provided they have
> access to the appropriate platform and tools.”

For the docker build this is fine as long as the solr/docker gradle
module is included in the source release. As one can always rebuild the
same image running gradlew docker using the source.

-

Jan Høydahl mentioned that the docker file layers should be limited, but
I'm not exactly sure what this means or entails. Maybe he can expand on
this.

Artifacts within the Image

As mentioned above in the "Image Location" section, the goal of including
the docker build process inside the Solr project is to make development
easier by providing an easy way to build the official-style docker image
with local source code. In order to achieve "official-style" images for
local builds, we want to make the build process for local images as close
as possible to the process for building official release images.

Currently the solr-docker-image, and a majority of "Docker Official
Images", the officially released Solr binaries are downloaded from mirrors
and validated within the Dockerfiles. This makes it easy to ensure to users
that the 9.0 solr docker image contains the 9.0 solr release. This process
doesn't fit very well with local builds, because there is nowhere to
download local builds from, and validation isn't required.

*The current opinion in the community is to abandon the "Docker Official
Images" style process of downloading and validating official binaries, and
instead having the release manager use the local-build image creation with
the final release source.* This should result in the same docker image in
the end, however there is no trust built into the docker image itself.
Instead we are likely going to document a way for users to verify the
docker-image contents themselves.

I am not sure what the user-side verification process would look like for
the image, but I definitely think it is something that we should look into.
Maybe Docker will allow us to use these as official images if we can script
out this verification and make it easy for them to do? Just a thought, I'm
not sure if that would actually work.
Re: Solr Docker discussion [ In reply to ]
>
> Currently the solr-docker-image, and a majority of "Docker Official
> Images", the officially released Solr binaries are downloaded from mirrors
> and validated within the Dockerfiles. This makes it easy to ensure to users
> that the 9.0 solr docker image contains the 9.0 solr release. This process
> doesn't fit very well with local builds, because there is nowhere to
> download local builds from, and validation isn't required.
>
> *The current opinion in the community is to abandon the "Docker Official
> Images" style process of downloading and validating official binaries, and
> instead having the release manager use the local-build image creation with
> the final release source.* This should result in the same docker image in
> the end, however there is no trust built into the docker image itself.
> Instead we are likely going to document a way for users to verify the
> docker-image contents themselves.
>

Before we abandon the official process of downloading/validating official
binaries, I think there is a good reason to keep the ability to download an
"official" Apache Solr release and use it in the "official" Solr
convenience Docker image.

Docker images are static point in time copies of an OS and all supporting
packages (like Java) when built. Periodically Docker images should be
rebuilt to pick up the latest security and bug enhancements in the base
image. Just like any OS should run `apt upgrade` or `yum update`
periodically to ensure it is up to date.

My proposal is to periodically rebuild the "official" Solr convenience
Docker image based on the "official" Solr release to ensure we keep the
Docker images up to date. The idea being that we have a list of "supported"
versions of Solr (ie: 8.5, 8.6, 8.7) and periodically (ie: daily, weekly)
the Docker images are rebuilt. Once a new release is made (ie: 8.8) it gets
added to this rebuilding matrix. This ensures that the "official" Solr
convenience Docker image is reasonably up to date with regards to the base
image security updates.

My understanding (from a few years ago) is that the Docker official images
are rebuilt when the base image is updated. This was automatic from what I
understand. If we move away from the "Docker official image" way of doing
things, we should still ensure that we can provide high quality secure
Docker images to the community.

We should not need a new Apache Solr release (ie: 8.7.1, 8.8) to update the
Docker image to pick up the latest base image changes.

It is important to be able to build and test locally with a Docker image
that matches what an "official" Docker image is. This should still be a
goal, but we should be able to rebuild the Solr docker image without
rebuilding all of Solr.

PS - I chatted a bit with Houston on Slack about this topic and hopefully
captured all the context correctly.

Kevin Risden


On Fri, Jan 15, 2021 at 12:45 PM Houston Putman <houstonputman@gmail.com>
wrote:

> There's a few decisions that need to be ironed out around the Solr docker
> image before 9.0 is released. This is because the community has decided
> that Solr should start releasing it's own docker images starting with 9.0.
>
> Below is the current state of the ongoing discussions for the Solr Docker
> image. Please feel free to correct me or fill in any information I may be
> missing.
>
> Where does this image live?
>
> There are two options for this really.
>
> - _/solr - docker run solr:9.0 (Official Docker Image)
> - apache/solr - docker run apache/solr:9.0
>
> The benefits of the first are 1) the nice usability of being able to
> plainly specify "solr" and 2) the "Docker Official Images" badge on
> DockerHub. The downsides are that there are very strict requirements with
> creating Official Docker Images, which would complicate and require
> separating the way that we build release docker images and local docker
> images.
>
> The benefits of using the apache namespace is that we can build the image
> in any way that we want. We would be able to build release and local docker
> images the exact same way. The downside is the loss of the "Docker Official
> Images" badge.
>
> *I think there is some consensus that choosing the "apache/solr" location
> is fine, and worth the added flexibility we get in the build process.*
>
> Legal Stuff
>
> There are a few legal questions we need to keep in mind when creating this
> process and doing a release for the first time.
>
> - Source release - The apache policy is: (from Michael Sokolov)
>
> “Every ASF release MUST contain one or more source packages, which
>> MUST be sufficient for a user to build and test the release provided they
>> have access to the appropriate platform and tools.”
>
> For the docker build this is fine as long as the solr/docker gradle
> module is included in the source release. As one can always rebuild the
> same image running gradlew docker using the source.
>
> -
>
> Jan Høydahl mentioned that the docker file layers should be limited,
> but I'm not exactly sure what this means or entails. Maybe he can expand on
> this.
>
> Artifacts within the Image
>
> As mentioned above in the "Image Location" section, the goal of including
> the docker build process inside the Solr project is to make development
> easier by providing an easy way to build the official-style docker image
> with local source code. In order to achieve "official-style" images for
> local builds, we want to make the build process for local images as close
> as possible to the process for building official release images.
>
> Currently the solr-docker-image, and a majority of "Docker Official
> Images", the officially released Solr binaries are downloaded from mirrors
> and validated within the Dockerfiles. This makes it easy to ensure to users
> that the 9.0 solr docker image contains the 9.0 solr release. This process
> doesn't fit very well with local builds, because there is nowhere to
> download local builds from, and validation isn't required.
>
> *The current opinion in the community is to abandon the "Docker Official
> Images" style process of downloading and validating official binaries, and
> instead having the release manager use the local-build image creation with
> the final release source.* This should result in the same docker image in
> the end, however there is no trust built into the docker image itself.
> Instead we are likely going to document a way for users to verify the
> docker-image contents themselves.
>
> I am not sure what the user-side verification process would look like for
> the image, but I definitely think it is something that we should look into.
> Maybe Docker will allow us to use these as official images if we can script
> out this verification and make it easy for them to do? Just a thought, I'm
> not sure if that would actually work.
>
Re: Solr Docker discussion [ In reply to ]
Thanks for bringing up this issue Kevin.

Periodically re-building docker images is certainly a feature we could
support, and probably should to automatically keep up with security fixes.
We could even automate it pretty easily in Jenkins.

We could also build in support in the gradle commands to instead of
building a TGZ from source, download and verify the "official" TGZ, to
build the image with. That way release images are always built with the
same exact binaries. The Dockerfile wouldn't need to change at all between
local and release, it still merely expects a TGZ to be passed in the
context; gradle can determine if it needs to be built from scratch or
downloaded.

This still likely wouldn't be good enough to make the image an "official
docker image" but it gets us to essentially the same end-state image. The
only difference is the downloading and verification are happening in gradle
instead of the Dockerfile.

- Houston

On Fri, Jan 15, 2021 at 2:05 PM Kevin Risden <krisden@apache.org> wrote:

> Currently the solr-docker-image, and a majority of "Docker Official
>> Images", the officially released Solr binaries are downloaded from mirrors
>> and validated within the Dockerfiles. This makes it easy to ensure to users
>> that the 9.0 solr docker image contains the 9.0 solr release. This process
>> doesn't fit very well with local builds, because there is nowhere to
>> download local builds from, and validation isn't required.
>>
>> *The current opinion in the community is to abandon the "Docker Official
>> Images" style process of downloading and validating official binaries, and
>> instead having the release manager use the local-build image creation with
>> the final release source.* This should result in the same docker image
>> in the end, however there is no trust built into the docker image itself.
>> Instead we are likely going to document a way for users to verify the
>> docker-image contents themselves.
>>
>
> Before we abandon the official process of downloading/validating official
> binaries, I think there is a good reason to keep the ability to download an
> "official" Apache Solr release and use it in the "official" Solr
> convenience Docker image.
>
> Docker images are static point in time copies of an OS and all supporting
> packages (like Java) when built. Periodically Docker images should be
> rebuilt to pick up the latest security and bug enhancements in the base
> image. Just like any OS should run `apt upgrade` or `yum update`
> periodically to ensure it is up to date.
>
> My proposal is to periodically rebuild the "official" Solr convenience
> Docker image based on the "official" Solr release to ensure we keep the
> Docker images up to date. The idea being that we have a list of "supported"
> versions of Solr (ie: 8.5, 8.6, 8.7) and periodically (ie: daily, weekly)
> the Docker images are rebuilt. Once a new release is made (ie: 8.8) it gets
> added to this rebuilding matrix. This ensures that the "official" Solr
> convenience Docker image is reasonably up to date with regards to the base
> image security updates.
>
> My understanding (from a few years ago) is that the Docker official images
> are rebuilt when the base image is updated. This was automatic from what I
> understand. If we move away from the "Docker official image" way of doing
> things, we should still ensure that we can provide high quality secure
> Docker images to the community.
>
> We should not need a new Apache Solr release (ie: 8.7.1, 8.8) to update
> the Docker image to pick up the latest base image changes.
>
> It is important to be able to build and test locally with a Docker image
> that matches what an "official" Docker image is. This should still be a
> goal, but we should be able to rebuild the Solr docker image without
> rebuilding all of Solr.
>
> PS - I chatted a bit with Houston on Slack about this topic and hopefully
> captured all the context correctly.
>
> Kevin Risden
>
>
> On Fri, Jan 15, 2021 at 12:45 PM Houston Putman <houstonputman@gmail.com>
> wrote:
>
>> There's a few decisions that need to be ironed out around the Solr docker
>> image before 9.0 is released. This is because the community has decided
>> that Solr should start releasing it's own docker images starting with 9.0.
>>
>> Below is the current state of the ongoing discussions for the Solr Docker
>> image. Please feel free to correct me or fill in any information I may be
>> missing.
>>
>> Where does this image live?
>>
>> There are two options for this really.
>>
>> - _/solr - docker run solr:9.0 (Official Docker Image)
>> - apache/solr - docker run apache/solr:9.0
>>
>> The benefits of the first are 1) the nice usability of being able to
>> plainly specify "solr" and 2) the "Docker Official Images" badge on
>> DockerHub. The downsides are that there are very strict requirements with
>> creating Official Docker Images, which would complicate and require
>> separating the way that we build release docker images and local docker
>> images.
>>
>> The benefits of using the apache namespace is that we can build the image
>> in any way that we want. We would be able to build release and local docker
>> images the exact same way. The downside is the loss of the "Docker Official
>> Images" badge.
>>
>> *I think there is some consensus that choosing the "apache/solr" location
>> is fine, and worth the added flexibility we get in the build process.*
>>
>> Legal Stuff
>>
>> There are a few legal questions we need to keep in mind when creating
>> this process and doing a release for the first time.
>>
>> - Source release - The apache policy is: (from Michael Sokolov)
>>
>> “Every ASF release MUST contain one or more source packages, which
>>> MUST be sufficient for a user to build and test the release provided they
>>> have access to the appropriate platform and tools.”
>>
>> For the docker build this is fine as long as the solr/docker gradle
>> module is included in the source release. As one can always rebuild the
>> same image running gradlew docker using the source.
>>
>> -
>>
>> Jan Høydahl mentioned that the docker file layers should be limited,
>> but I'm not exactly sure what this means or entails. Maybe he can expand on
>> this.
>>
>> Artifacts within the Image
>>
>> As mentioned above in the "Image Location" section, the goal of including
>> the docker build process inside the Solr project is to make development
>> easier by providing an easy way to build the official-style docker image
>> with local source code. In order to achieve "official-style" images for
>> local builds, we want to make the build process for local images as close
>> as possible to the process for building official release images.
>>
>> Currently the solr-docker-image, and a majority of "Docker Official
>> Images", the officially released Solr binaries are downloaded from mirrors
>> and validated within the Dockerfiles. This makes it easy to ensure to users
>> that the 9.0 solr docker image contains the 9.0 solr release. This process
>> doesn't fit very well with local builds, because there is nowhere to
>> download local builds from, and validation isn't required.
>>
>> *The current opinion in the community is to abandon the "Docker Official
>> Images" style process of downloading and validating official binaries, and
>> instead having the release manager use the local-build image creation with
>> the final release source.* This should result in the same docker image
>> in the end, however there is no trust built into the docker image itself.
>> Instead we are likely going to document a way for users to verify the
>> docker-image contents themselves.
>>
>> I am not sure what the user-side verification process would look like for
>> the image, but I definitely think it is something that we should look into.
>> Maybe Docker will allow us to use these as official images if we can script
>> out this verification and make it easy for them to do? Just a thought, I'm
>> not sure if that would actually work.
>>
>
Re: Solr Docker discussion [ In reply to ]
I'm curious about how tags will work when updating the base image for a
released image? The image for a tag should be immutable (IMHO), and I think
people would be surprised if 8.8.0 suddenly changed even if it was for a
good reason such as fixing a CVE in the base image. But based on what Kevin
said, perhaps there's already precedence for this with the official images?

On Fri, Jan 15, 2021 at 1:51 PM Houston Putman <houstonputman@gmail.com>
wrote:

> Thanks for bringing up this issue Kevin.
>
> Periodically re-building docker images is certainly a feature we could
> support, and probably should to automatically keep up with security fixes.
> We could even automate it pretty easily in Jenkins.
>
> We could also build in support in the gradle commands to instead of
> building a TGZ from source, download and verify the "official" TGZ, to
> build the image with. That way release images are always built with the
> same exact binaries. The Dockerfile wouldn't need to change at all between
> local and release, it still merely expects a TGZ to be passed in the
> context; gradle can determine if it needs to be built from scratch or
> downloaded.
>
> This still likely wouldn't be good enough to make the image an "official
> docker image" but it gets us to essentially the same end-state image. The
> only difference is the downloading and verification are happening in gradle
> instead of the Dockerfile.
>
> - Houston
>
> On Fri, Jan 15, 2021 at 2:05 PM Kevin Risden <krisden@apache.org> wrote:
>
>> Currently the solr-docker-image, and a majority of "Docker Official
>>> Images", the officially released Solr binaries are downloaded from mirrors
>>> and validated within the Dockerfiles. This makes it easy to ensure to users
>>> that the 9.0 solr docker image contains the 9.0 solr release. This process
>>> doesn't fit very well with local builds, because there is nowhere to
>>> download local builds from, and validation isn't required.
>>>
>>> *The current opinion in the community is to abandon the "Docker Official
>>> Images" style process of downloading and validating official binaries, and
>>> instead having the release manager use the local-build image creation with
>>> the final release source.* This should result in the same docker image
>>> in the end, however there is no trust built into the docker image itself.
>>> Instead we are likely going to document a way for users to verify the
>>> docker-image contents themselves.
>>>
>>
>> Before we abandon the official process of downloading/validating official
>> binaries, I think there is a good reason to keep the ability to download an
>> "official" Apache Solr release and use it in the "official" Solr
>> convenience Docker image.
>>
>> Docker images are static point in time copies of an OS and all supporting
>> packages (like Java) when built. Periodically Docker images should be
>> rebuilt to pick up the latest security and bug enhancements in the base
>> image. Just like any OS should run `apt upgrade` or `yum update`
>> periodically to ensure it is up to date.
>>
>> My proposal is to periodically rebuild the "official" Solr convenience
>> Docker image based on the "official" Solr release to ensure we keep the
>> Docker images up to date. The idea being that we have a list of "supported"
>> versions of Solr (ie: 8.5, 8.6, 8.7) and periodically (ie: daily, weekly)
>> the Docker images are rebuilt. Once a new release is made (ie: 8.8) it gets
>> added to this rebuilding matrix. This ensures that the "official" Solr
>> convenience Docker image is reasonably up to date with regards to the base
>> image security updates.
>>
>> My understanding (from a few years ago) is that the Docker official
>> images are rebuilt when the base image is updated. This was automatic from
>> what I understand. If we move away from the "Docker official image" way of
>> doing things, we should still ensure that we can provide high quality
>> secure Docker images to the community.
>>
>> We should not need a new Apache Solr release (ie: 8.7.1, 8.8) to update
>> the Docker image to pick up the latest base image changes.
>>
>> It is important to be able to build and test locally with a Docker image
>> that matches what an "official" Docker image is. This should still be a
>> goal, but we should be able to rebuild the Solr docker image without
>> rebuilding all of Solr.
>>
>> PS - I chatted a bit with Houston on Slack about this topic and hopefully
>> captured all the context correctly.
>>
>> Kevin Risden
>>
>>
>> On Fri, Jan 15, 2021 at 12:45 PM Houston Putman <houstonputman@gmail.com>
>> wrote:
>>
>>> There's a few decisions that need to be ironed out around the Solr
>>> docker image before 9.0 is released. This is because the community has
>>> decided that Solr should start releasing it's own docker images starting
>>> with 9.0.
>>>
>>> Below is the current state of the ongoing discussions for the Solr
>>> Docker image. Please feel free to correct me or fill in any information I
>>> may be missing.
>>>
>>> Where does this image live?
>>>
>>> There are two options for this really.
>>>
>>> - _/solr - docker run solr:9.0 (Official Docker Image)
>>> - apache/solr - docker run apache/solr:9.0
>>>
>>> The benefits of the first are 1) the nice usability of being able to
>>> plainly specify "solr" and 2) the "Docker Official Images" badge on
>>> DockerHub. The downsides are that there are very strict requirements with
>>> creating Official Docker Images, which would complicate and require
>>> separating the way that we build release docker images and local docker
>>> images.
>>>
>>> The benefits of using the apache namespace is that we can build the
>>> image in any way that we want. We would be able to build release and local
>>> docker images the exact same way. The downside is the loss of the "Docker
>>> Official Images" badge.
>>>
>>> *I think there is some consensus that choosing the "apache/solr"
>>> location is fine, and worth the added flexibility we get in the build
>>> process.*
>>>
>>> Legal Stuff
>>>
>>> There are a few legal questions we need to keep in mind when creating
>>> this process and doing a release for the first time.
>>>
>>> - Source release - The apache policy is: (from Michael Sokolov)
>>>
>>> “Every ASF release MUST contain one or more source packages, which
>>>> MUST be sufficient for a user to build and test the release provided they
>>>> have access to the appropriate platform and tools.”
>>>
>>> For the docker build this is fine as long as the solr/docker gradle
>>> module is included in the source release. As one can always rebuild the
>>> same image running gradlew docker using the source.
>>>
>>> -
>>>
>>> Jan Høydahl mentioned that the docker file layers should be limited,
>>> but I'm not exactly sure what this means or entails. Maybe he can expand on
>>> this.
>>>
>>> Artifacts within the Image
>>>
>>> As mentioned above in the "Image Location" section, the goal of
>>> including the docker build process inside the Solr project is to make
>>> development easier by providing an easy way to build the official-style
>>> docker image with local source code. In order to achieve "official-style"
>>> images for local builds, we want to make the build process for local images
>>> as close as possible to the process for building official release images.
>>>
>>> Currently the solr-docker-image, and a majority of "Docker Official
>>> Images", the officially released Solr binaries are downloaded from mirrors
>>> and validated within the Dockerfiles. This makes it easy to ensure to users
>>> that the 9.0 solr docker image contains the 9.0 solr release. This process
>>> doesn't fit very well with local builds, because there is nowhere to
>>> download local builds from, and validation isn't required.
>>>
>>> *The current opinion in the community is to abandon the "Docker Official
>>> Images" style process of downloading and validating official binaries, and
>>> instead having the release manager use the local-build image creation with
>>> the final release source.* This should result in the same docker image
>>> in the end, however there is no trust built into the docker image itself.
>>> Instead we are likely going to document a way for users to verify the
>>> docker-image contents themselves.
>>>
>>> I am not sure what the user-side verification process would look like
>>> for the image, but I definitely think it is something that we should look
>>> into. Maybe Docker will allow us to use these as official images if we can
>>> script out this verification and make it easy for them to do? Just a
>>> thought, I'm not sure if that would actually work.
>>>
>>
Re: Solr Docker discussion [ In reply to ]
Great summary Houston!

Could also be that docker team is willing to provide a “link” from official _/solr to apache/solr if we can convince them of solid quality. Think they do this for elastic images already.

Since Docker images contain Linux and Java, which we would not be allowed to release as part of Solr, I have seen discussions in various ASF lists stating that it can be argued that the only binaries we do “release” are the layers built by out Dockerfile, i.e. what comes after the (runtime) FROM line. So we should be careful with what extra software we add in dockerfile. I did a check earlier and think we are in good shape.

We currently re-build a bunch of older solr docked images every time we release a new, but I don’t think there is any automatic refresh of images outside a release. Great idea to kick off refresh of images from Jenkins.

We can also publish nightly “master” images but since they are not officially voted releases they must be clearly labelled as unofficial and not advertised on the web page, only for dev purposes.

Jan Høydahl

> 16. jan. 2021 kl. 00:36 skrev Timothy Potter <thelabdude@gmail.com>:
> ?
> I'm curious about how tags will work when updating the base image for a released image? The image for a tag should be immutable (IMHO), and I think people would be surprised if 8.8.0 suddenly changed even if it was for a good reason such as fixing a CVE in the base image. But based on what Kevin said, perhaps there's already precedence for this with the official images?
>
> On Fri, Jan 15, 2021 at 1:51 PM Houston Putman <houstonputman@gmail.com> wrote:
>> Thanks for bringing up this issue Kevin.
>>
>> Periodically re-building docker images is certainly a feature we could support, and probably should to automatically keep up with security fixes. We could even automate it pretty easily in Jenkins.
>>
>> We could also build in support in the gradle commands to instead of building a TGZ from source, download and verify the "official" TGZ, to build the image with. That way release images are always built with the same exact binaries. The Dockerfile wouldn't need to change at all between local and release, it still merely expects a TGZ to be passed in the context; gradle can determine if it needs to be built from scratch or downloaded.
>>
>> This still likely wouldn't be good enough to make the image an "official docker image" but it gets us to essentially the same end-state image. The only difference is the downloading and verification are happening in gradle instead of the Dockerfile.
>>
>> - Houston
>>
>> On Fri, Jan 15, 2021 at 2:05 PM Kevin Risden <krisden@apache.org> wrote:
>>>> Currently the solr-docker-image, and a majority of "Docker Official Images", the officially released Solr binaries are downloaded from mirrors and validated within the Dockerfiles. This makes it easy to ensure to users that the 9.0 solr docker image contains the 9.0 solr release. This process doesn't fit very well with local builds, because there is nowhere to download local builds from, and validation isn't required.
>>>>
>>>> The current opinion in the community is to abandon the "Docker Official Images" style process of downloading and validating official binaries, and instead having the release manager use the local-build image creation with the final release source. This should result in the same docker image in the end, however there is no trust built into the docker image itself. Instead we are likely going to document a way for users to verify the docker-image contents themselves.
>>>
>>> Before we abandon the official process of downloading/validating official binaries, I think there is a good reason to keep the ability to download an "official" Apache Solr release and use it in the "official" Solr convenience Docker image.
>>>
>>> Docker images are static point in time copies of an OS and all supporting packages (like Java) when built. Periodically Docker images should be rebuilt to pick up the latest security and bug enhancements in the base image. Just like any OS should run `apt upgrade` or `yum update` periodically to ensure it is up to date.
>>>
>>> My proposal is to periodically rebuild the "official" Solr convenience Docker image based on the "official" Solr release to ensure we keep the Docker images up to date. The idea being that we have a list of "supported" versions of Solr (ie: 8.5, 8.6, 8.7) and periodically (ie: daily, weekly) the Docker images are rebuilt. Once a new release is made (ie: 8.8) it gets added to this rebuilding matrix. This ensures that the "official" Solr convenience Docker image is reasonably up to date with regards to the base image security updates.
>>>
>>> My understanding (from a few years ago) is that the Docker official images are rebuilt when the base image is updated. This was automatic from what I understand. If we move away from the "Docker official image" way of doing things, we should still ensure that we can provide high quality secure Docker images to the community.
>>>
>>> We should not need a new Apache Solr release (ie: 8.7.1, 8.8) to update the Docker image to pick up the latest base image changes.
>>>
>>> It is important to be able to build and test locally with a Docker image that matches what an "official" Docker image is. This should still be a goal, but we should be able to rebuild the Solr docker image without rebuilding all of Solr.
>>>
>>> PS - I chatted a bit with Houston on Slack about this topic and hopefully captured all the context correctly.
>>>
>>> Kevin Risden
>>>
>>>
>>> On Fri, Jan 15, 2021 at 12:45 PM Houston Putman <houstonputman@gmail.com> wrote:
>>>> There's a few decisions that need to be ironed out around the Solr docker image before 9.0 is released. This is because the community has decided that Solr should start releasing it's own docker images starting with 9.0.
>>>>
>>>> Below is the current state of the ongoing discussions for the Solr Docker image. Please feel free to correct me or fill in any information I may be missing.
>>>>
>>>> Where does this image live?
>>>>
>>>> There are two options for this really.
>>>> _/solr - docker run solr:9.0 (Official Docker Image)
>>>> apache/solr - docker run apache/solr:9.0
>>>> The benefits of the first are 1) the nice usability of being able to plainly specify "solr" and 2) the "Docker Official Images" badge on DockerHub. The downsides are that there are very strict requirements with creating Official Docker Images, which would complicate and require separating the way that we build release docker images and local docker images.
>>>>
>>>> The benefits of using the apache namespace is that we can build the image in any way that we want. We would be able to build release and local docker images the exact same way. The downside is the loss of the "Docker Official Images" badge.
>>>>
>>>> I think there is some consensus that choosing the "apache/solr" location is fine, and worth the added flexibility we get in the build process.
>>>>
>>>> Legal Stuff
>>>>
>>>> There are a few legal questions we need to keep in mind when creating this process and doing a release for the first time.
>>>> Source release - The apache policy is: (from Michael Sokolov)
>>>>> “Every ASF release MUST contain one or more source packages, which MUST be sufficient for a user to build and test the release provided they have access to the appropriate platform and tools.”
>>>> For the docker build this is fine as long as the solr/docker gradle module is included in the source release. As one can always rebuild the same image running gradlew docker using the source.
>>>> Jan Høydahl mentioned that the docker file layers should be limited, but I'm not exactly sure what this means or entails. Maybe he can expand on this.
>>>> Artifacts within the Image
>>>>
>>>> As mentioned above in the "Image Location" section, the goal of including the docker build process inside the Solr project is to make development easier by providing an easy way to build the official-style docker image with local source code. In order to achieve "official-style" images for local builds, we want to make the build process for local images as close as possible to the process for building official release images.
>>>>
>>>> Currently the solr-docker-image, and a majority of "Docker Official Images", the officially released Solr binaries are downloaded from mirrors and validated within the Dockerfiles. This makes it easy to ensure to users that the 9.0 solr docker image contains the 9.0 solr release. This process doesn't fit very well with local builds, because there is nowhere to download local builds from, and validation isn't required.
>>>>
>>>> The current opinion in the community is to abandon the "Docker Official Images" style process of downloading and validating official binaries, and instead having the release manager use the local-build image creation with the final release source. This should result in the same docker image in the end, however there is no trust built into the docker image itself. Instead we are likely going to document a way for users to verify the docker-image contents themselves.
>>>>
>>>> I am not sure what the user-side verification process would look like for the image, but I definitely think it is something that we should look into. Maybe Docker will allow us to use these as official images if we can script out this verification and make it easy for them to do? Just a thought, I'm not sure if that would actually work.
Re: Solr Docker discussion [ In reply to ]
Periodically rebuilding old images seems challenging in this new setup
compared to docker-solr. I suppose it could be done with a script that
loops over release branches to check them out to then execute the
appropriate gradle task to build & push. But the requirement to use an
identical Solr binary (same input tgz as was originally produced) is the
real challenge, because we can't use the same Dockerfile. The input TGZ is
gone. Maybe we could grab /opt/solr/* from the previous image, which can
be referenced in a FROM?

BTW I updated the Dockerfile significantly in Houston's recent PR. I think
it's a step in this direction.

~ David Smiley
Apache Lucene/Solr Search Developer
http://www.linkedin.com/in/davidwsmiley


On Sat, Jan 16, 2021 at 7:36 AM Jan Høydahl <jan.asf@cominvent.com> wrote:

> Great summary Houston!
>
> Could also be that docker team is willing to provide a “link” from
> official _/solr to apache/solr if we can convince them of solid quality.
> Think they do this for elastic images already.
>
> Since Docker images contain Linux and Java, which we would not be allowed
> to release as part of Solr, I have seen discussions in various ASF lists
> stating that it can be argued that the only binaries we do “release” are
> the layers built by out Dockerfile, i.e. what comes after the (runtime)
> FROM line. So we should be careful with what extra software we add in
> dockerfile. I did a check earlier and think we are in good shape.
>
> We currently re-build a bunch of older solr docked images every time we
> release a new, but I don’t think there is any automatic refresh of images
> outside a release. Great idea to kick off refresh of images from Jenkins.
>
> We can also publish nightly “master” images but since they are not
> officially voted releases they must be clearly labelled as unofficial and
> not advertised on the web page, only for dev purposes.
>
> Jan Høydahl
>
> 16. jan. 2021 kl. 00:36 skrev Timothy Potter <thelabdude@gmail.com>:
>
> ?
> I'm curious about how tags will work when updating the base image for a
> released image? The image for a tag should be immutable (IMHO), and I think
> people would be surprised if 8.8.0 suddenly changed even if it was for a
> good reason such as fixing a CVE in the base image. But based on what Kevin
> said, perhaps there's already precedence for this with the official images?
>
> On Fri, Jan 15, 2021 at 1:51 PM Houston Putman <houstonputman@gmail.com>
> wrote:
>
>> Thanks for bringing up this issue Kevin.
>>
>> Periodically re-building docker images is certainly a feature we could
>> support, and probably should to automatically keep up with security fixes.
>> We could even automate it pretty easily in Jenkins.
>>
>> We could also build in support in the gradle commands to instead of
>> building a TGZ from source, download and verify the "official" TGZ, to
>> build the image with. That way release images are always built with the
>> same exact binaries. The Dockerfile wouldn't need to change at all between
>> local and release, it still merely expects a TGZ to be passed in the
>> context; gradle can determine if it needs to be built from scratch or
>> downloaded.
>>
>> This still likely wouldn't be good enough to make the image an "official
>> docker image" but it gets us to essentially the same end-state image. The
>> only difference is the downloading and verification are happening in gradle
>> instead of the Dockerfile.
>>
>> - Houston
>>
>> On Fri, Jan 15, 2021 at 2:05 PM Kevin Risden <krisden@apache.org> wrote:
>>
>>> Currently the solr-docker-image, and a majority of "Docker Official
>>>> Images", the officially released Solr binaries are downloaded from mirrors
>>>> and validated within the Dockerfiles. This makes it easy to ensure to users
>>>> that the 9.0 solr docker image contains the 9.0 solr release. This process
>>>> doesn't fit very well with local builds, because there is nowhere to
>>>> download local builds from, and validation isn't required.
>>>>
>>>> *The current opinion in the community is to abandon the "Docker
>>>> Official Images" style process of downloading and validating official
>>>> binaries, and instead having the release manager use the local-build image
>>>> creation with the final release source.* This should result in the
>>>> same docker image in the end, however there is no trust built into the
>>>> docker image itself. Instead we are likely going to document a way for
>>>> users to verify the docker-image contents themselves.
>>>>
>>>
>>> Before we abandon the official process of downloading/validating
>>> official binaries, I think there is a good reason to keep the ability to
>>> download an "official" Apache Solr release and use it in the "official"
>>> Solr convenience Docker image.
>>>
>>> Docker images are static point in time copies of an OS and all
>>> supporting packages (like Java) when built. Periodically Docker images
>>> should be rebuilt to pick up the latest security and bug enhancements in
>>> the base image. Just like any OS should run `apt upgrade` or `yum update`
>>> periodically to ensure it is up to date.
>>>
>>> My proposal is to periodically rebuild the "official" Solr convenience
>>> Docker image based on the "official" Solr release to ensure we keep the
>>> Docker images up to date. The idea being that we have a list of "supported"
>>> versions of Solr (ie: 8.5, 8.6, 8.7) and periodically (ie: daily, weekly)
>>> the Docker images are rebuilt. Once a new release is made (ie: 8.8) it gets
>>> added to this rebuilding matrix. This ensures that the "official" Solr
>>> convenience Docker image is reasonably up to date with regards to the base
>>> image security updates.
>>>
>>> My understanding (from a few years ago) is that the Docker official
>>> images are rebuilt when the base image is updated. This was automatic from
>>> what I understand. If we move away from the "Docker official image" way of
>>> doing things, we should still ensure that we can provide high quality
>>> secure Docker images to the community.
>>>
>>> We should not need a new Apache Solr release (ie: 8.7.1, 8.8) to update
>>> the Docker image to pick up the latest base image changes.
>>>
>>> It is important to be able to build and test locally with a Docker image
>>> that matches what an "official" Docker image is. This should still be a
>>> goal, but we should be able to rebuild the Solr docker image without
>>> rebuilding all of Solr.
>>>
>>> PS - I chatted a bit with Houston on Slack about this topic and
>>> hopefully captured all the context correctly.
>>>
>>> Kevin Risden
>>>
>>>
>>> On Fri, Jan 15, 2021 at 12:45 PM Houston Putman <houstonputman@gmail.com>
>>> wrote:
>>>
>>>> There's a few decisions that need to be ironed out around the Solr
>>>> docker image before 9.0 is released. This is because the community has
>>>> decided that Solr should start releasing it's own docker images starting
>>>> with 9.0.
>>>>
>>>> Below is the current state of the ongoing discussions for the Solr
>>>> Docker image. Please feel free to correct me or fill in any information I
>>>> may be missing.
>>>>
>>>> Where does this image live?
>>>>
>>>> There are two options for this really.
>>>>
>>>> - _/solr - docker run solr:9.0 (Official Docker Image)
>>>> - apache/solr - docker run apache/solr:9.0
>>>>
>>>> The benefits of the first are 1) the nice usability of being able to
>>>> plainly specify "solr" and 2) the "Docker Official Images" badge on
>>>> DockerHub. The downsides are that there are very strict requirements with
>>>> creating Official Docker Images, which would complicate and require
>>>> separating the way that we build release docker images and local docker
>>>> images.
>>>>
>>>> The benefits of using the apache namespace is that we can build the
>>>> image in any way that we want. We would be able to build release and local
>>>> docker images the exact same way. The downside is the loss of the "Docker
>>>> Official Images" badge.
>>>>
>>>> *I think there is some consensus that choosing the "apache/solr"
>>>> location is fine, and worth the added flexibility we get in the build
>>>> process.*
>>>>
>>>> Legal Stuff
>>>>
>>>> There are a few legal questions we need to keep in mind when creating
>>>> this process and doing a release for the first time.
>>>>
>>>> - Source release - The apache policy is: (from Michael Sokolov)
>>>>
>>>> “Every ASF release MUST contain one or more source packages, which
>>>>> MUST be sufficient for a user to build and test the release provided they
>>>>> have access to the appropriate platform and tools.”
>>>>
>>>> For the docker build this is fine as long as the solr/docker gradle
>>>> module is included in the source release. As one can always rebuild the
>>>> same image running gradlew docker using the source.
>>>>
>>>> -
>>>>
>>>> Jan Høydahl mentioned that the docker file layers should be
>>>> limited, but I'm not exactly sure what this means or entails. Maybe he can
>>>> expand on this.
>>>>
>>>> Artifacts within the Image
>>>>
>>>> As mentioned above in the "Image Location" section, the goal of
>>>> including the docker build process inside the Solr project is to make
>>>> development easier by providing an easy way to build the official-style
>>>> docker image with local source code. In order to achieve "official-style"
>>>> images for local builds, we want to make the build process for local images
>>>> as close as possible to the process for building official release images.
>>>>
>>>> Currently the solr-docker-image, and a majority of "Docker Official
>>>> Images", the officially released Solr binaries are downloaded from mirrors
>>>> and validated within the Dockerfiles. This makes it easy to ensure to users
>>>> that the 9.0 solr docker image contains the 9.0 solr release. This process
>>>> doesn't fit very well with local builds, because there is nowhere to
>>>> download local builds from, and validation isn't required.
>>>>
>>>> *The current opinion in the community is to abandon the "Docker
>>>> Official Images" style process of downloading and validating official
>>>> binaries, and instead having the release manager use the local-build image
>>>> creation with the final release source.* This should result in the
>>>> same docker image in the end, however there is no trust built into the
>>>> docker image itself. Instead we are likely going to document a way for
>>>> users to verify the docker-image contents themselves.
>>>>
>>>> I am not sure what the user-side verification process would look like
>>>> for the image, but I definitely think it is something that we should look
>>>> into. Maybe Docker will allow us to use these as official images if we can
>>>> script out this verification and make it easy for them to do? Just a
>>>> thought, I'm not sure if that would actually work.
>>>>
>>>
Re: Solr Docker discussion [ In reply to ]
Jan, thanks for clarifying your point on the Dockerfile layers, that makes
a lot of sense.

David, for your point on using the same Solr binary, I think we could
possibly use the same Dockerfile. The difference would be the input Solr
tgz that dockerTar puts into the "releases" sub-directory. Instead of
pulling in from the solr/packaging module, as you would for a local build,
you could instead download the Solr tgz from official mirrors. The docker
context tgz would be different, but the actual solr binary would be exactly
the same between all "release" builds.

I do think the "official release" docker image can be targeted via a
separate PR. This one already has a pretty big scope as it is. (Not to say
that your changes are bad David, I think they are fine to be included.)

Also Jan, I think having the nightly "master" images will be wonderful for
nightly integration and load tests!

- Houston

On Sun, Jan 17, 2021 at 3:28 PM David Smiley <dsmiley@apache.org> wrote:

> Periodically rebuilding old images seems challenging in this new setup
> compared to docker-solr. I suppose it could be done with a script that
> loops over release branches to check them out to then execute the
> appropriate gradle task to build & push. But the requirement to use an
> identical Solr binary (same input tgz as was originally produced) is the
> real challenge, because we can't use the same Dockerfile. The input TGZ is
> gone. Maybe we could grab /opt/solr/* from the previous image, which can
> be referenced in a FROM?
>
> BTW I updated the Dockerfile significantly in Houston's recent PR. I
> think it's a step in this direction.
>
> ~ David Smiley
> Apache Lucene/Solr Search Developer
> http://www.linkedin.com/in/davidwsmiley
>
>
> On Sat, Jan 16, 2021 at 7:36 AM Jan Høydahl <jan.asf@cominvent.com> wrote:
>
>> Great summary Houston!
>>
>> Could also be that docker team is willing to provide a “link” from
>> official _/solr to apache/solr if we can convince them of solid quality.
>> Think they do this for elastic images already.
>>
>> Since Docker images contain Linux and Java, which we would not be allowed
>> to release as part of Solr, I have seen discussions in various ASF lists
>> stating that it can be argued that the only binaries we do “release” are
>> the layers built by out Dockerfile, i.e. what comes after the (runtime)
>> FROM line. So we should be careful with what extra software we add in
>> dockerfile. I did a check earlier and think we are in good shape.
>>
>> We currently re-build a bunch of older solr docked images every time we
>> release a new, but I don’t think there is any automatic refresh of images
>> outside a release. Great idea to kick off refresh of images from Jenkins.
>>
>> We can also publish nightly “master” images but since they are not
>> officially voted releases they must be clearly labelled as unofficial and
>> not advertised on the web page, only for dev purposes.
>>
>> Jan Høydahl
>>
>> 16. jan. 2021 kl. 00:36 skrev Timothy Potter <thelabdude@gmail.com>:
>>
>> ?
>> I'm curious about how tags will work when updating the base image for a
>> released image? The image for a tag should be immutable (IMHO), and I think
>> people would be surprised if 8.8.0 suddenly changed even if it was for a
>> good reason such as fixing a CVE in the base image. But based on what Kevin
>> said, perhaps there's already precedence for this with the official images?
>>
>> On Fri, Jan 15, 2021 at 1:51 PM Houston Putman <houstonputman@gmail.com>
>> wrote:
>>
>>> Thanks for bringing up this issue Kevin.
>>>
>>> Periodically re-building docker images is certainly a feature we could
>>> support, and probably should to automatically keep up with security fixes.
>>> We could even automate it pretty easily in Jenkins.
>>>
>>> We could also build in support in the gradle commands to instead of
>>> building a TGZ from source, download and verify the "official" TGZ, to
>>> build the image with. That way release images are always built with the
>>> same exact binaries. The Dockerfile wouldn't need to change at all between
>>> local and release, it still merely expects a TGZ to be passed in the
>>> context; gradle can determine if it needs to be built from scratch or
>>> downloaded.
>>>
>>> This still likely wouldn't be good enough to make the image an "official
>>> docker image" but it gets us to essentially the same end-state image. The
>>> only difference is the downloading and verification are happening in gradle
>>> instead of the Dockerfile.
>>>
>>> - Houston
>>>
>>> On Fri, Jan 15, 2021 at 2:05 PM Kevin Risden <krisden@apache.org> wrote:
>>>
>>>> Currently the solr-docker-image, and a majority of "Docker Official
>>>>> Images", the officially released Solr binaries are downloaded from mirrors
>>>>> and validated within the Dockerfiles. This makes it easy to ensure to users
>>>>> that the 9.0 solr docker image contains the 9.0 solr release. This process
>>>>> doesn't fit very well with local builds, because there is nowhere to
>>>>> download local builds from, and validation isn't required.
>>>>>
>>>>> *The current opinion in the community is to abandon the "Docker
>>>>> Official Images" style process of downloading and validating official
>>>>> binaries, and instead having the release manager use the local-build image
>>>>> creation with the final release source.* This should result in the
>>>>> same docker image in the end, however there is no trust built into the
>>>>> docker image itself. Instead we are likely going to document a way for
>>>>> users to verify the docker-image contents themselves.
>>>>>
>>>>
>>>> Before we abandon the official process of downloading/validating
>>>> official binaries, I think there is a good reason to keep the ability to
>>>> download an "official" Apache Solr release and use it in the "official"
>>>> Solr convenience Docker image.
>>>>
>>>> Docker images are static point in time copies of an OS and all
>>>> supporting packages (like Java) when built. Periodically Docker images
>>>> should be rebuilt to pick up the latest security and bug enhancements in
>>>> the base image. Just like any OS should run `apt upgrade` or `yum update`
>>>> periodically to ensure it is up to date.
>>>>
>>>> My proposal is to periodically rebuild the "official" Solr convenience
>>>> Docker image based on the "official" Solr release to ensure we keep the
>>>> Docker images up to date. The idea being that we have a list of "supported"
>>>> versions of Solr (ie: 8.5, 8.6, 8.7) and periodically (ie: daily, weekly)
>>>> the Docker images are rebuilt. Once a new release is made (ie: 8.8) it gets
>>>> added to this rebuilding matrix. This ensures that the "official" Solr
>>>> convenience Docker image is reasonably up to date with regards to the base
>>>> image security updates.
>>>>
>>>> My understanding (from a few years ago) is that the Docker official
>>>> images are rebuilt when the base image is updated. This was automatic from
>>>> what I understand. If we move away from the "Docker official image" way of
>>>> doing things, we should still ensure that we can provide high quality
>>>> secure Docker images to the community.
>>>>
>>>> We should not need a new Apache Solr release (ie: 8.7.1, 8.8) to update
>>>> the Docker image to pick up the latest base image changes.
>>>>
>>>> It is important to be able to build and test locally with a Docker
>>>> image that matches what an "official" Docker image is. This should still be
>>>> a goal, but we should be able to rebuild the Solr docker image without
>>>> rebuilding all of Solr.
>>>>
>>>> PS - I chatted a bit with Houston on Slack about this topic and
>>>> hopefully captured all the context correctly.
>>>>
>>>> Kevin Risden
>>>>
>>>>
>>>> On Fri, Jan 15, 2021 at 12:45 PM Houston Putman <
>>>> houstonputman@gmail.com> wrote:
>>>>
>>>>> There's a few decisions that need to be ironed out around the Solr
>>>>> docker image before 9.0 is released. This is because the community has
>>>>> decided that Solr should start releasing it's own docker images starting
>>>>> with 9.0.
>>>>>
>>>>> Below is the current state of the ongoing discussions for the Solr
>>>>> Docker image. Please feel free to correct me or fill in any information I
>>>>> may be missing.
>>>>>
>>>>> Where does this image live?
>>>>>
>>>>> There are two options for this really.
>>>>>
>>>>> - _/solr - docker run solr:9.0 (Official Docker Image)
>>>>> - apache/solr - docker run apache/solr:9.0
>>>>>
>>>>> The benefits of the first are 1) the nice usability of being able to
>>>>> plainly specify "solr" and 2) the "Docker Official Images" badge on
>>>>> DockerHub. The downsides are that there are very strict requirements with
>>>>> creating Official Docker Images, which would complicate and require
>>>>> separating the way that we build release docker images and local docker
>>>>> images.
>>>>>
>>>>> The benefits of using the apache namespace is that we can build the
>>>>> image in any way that we want. We would be able to build release and local
>>>>> docker images the exact same way. The downside is the loss of the "Docker
>>>>> Official Images" badge.
>>>>>
>>>>> *I think there is some consensus that choosing the "apache/solr"
>>>>> location is fine, and worth the added flexibility we get in the build
>>>>> process.*
>>>>>
>>>>> Legal Stuff
>>>>>
>>>>> There are a few legal questions we need to keep in mind when creating
>>>>> this process and doing a release for the first time.
>>>>>
>>>>> - Source release - The apache policy is: (from Michael Sokolov)
>>>>>
>>>>> “Every ASF release MUST contain one or more source packages, which
>>>>>> MUST be sufficient for a user to build and test the release provided they
>>>>>> have access to the appropriate platform and tools.”
>>>>>
>>>>> For the docker build this is fine as long as the solr/docker
>>>>> gradle module is included in the source release. As one can always rebuild
>>>>> the same image running gradlew docker using the source.
>>>>>
>>>>> -
>>>>>
>>>>> Jan Høydahl mentioned that the docker file layers should be
>>>>> limited, but I'm not exactly sure what this means or entails. Maybe he can
>>>>> expand on this.
>>>>>
>>>>> Artifacts within the Image
>>>>>
>>>>> As mentioned above in the "Image Location" section, the goal of
>>>>> including the docker build process inside the Solr project is to make
>>>>> development easier by providing an easy way to build the official-style
>>>>> docker image with local source code. In order to achieve "official-style"
>>>>> images for local builds, we want to make the build process for local images
>>>>> as close as possible to the process for building official release images.
>>>>>
>>>>> Currently the solr-docker-image, and a majority of "Docker Official
>>>>> Images", the officially released Solr binaries are downloaded from mirrors
>>>>> and validated within the Dockerfiles. This makes it easy to ensure to users
>>>>> that the 9.0 solr docker image contains the 9.0 solr release. This process
>>>>> doesn't fit very well with local builds, because there is nowhere to
>>>>> download local builds from, and validation isn't required.
>>>>>
>>>>> *The current opinion in the community is to abandon the "Docker
>>>>> Official Images" style process of downloading and validating official
>>>>> binaries, and instead having the release manager use the local-build image
>>>>> creation with the final release source.* This should result in the
>>>>> same docker image in the end, however there is no trust built into the
>>>>> docker image itself. Instead we are likely going to document a way for
>>>>> users to verify the docker-image contents themselves.
>>>>>
>>>>> I am not sure what the user-side verification process would look like
>>>>> for the image, but I definitely think it is something that we should look
>>>>> into. Maybe Docker will allow us to use these as official images if we can
>>>>> script out this verification and make it easy for them to do? Just a
>>>>> thought, I'm not sure if that would actually work.
>>>>>
>>>>
Re: Solr Docker discussion [ In reply to ]
: - _/solr - docker run solr:9.0 (Official Docker Image)
: - apache/solr - docker run apache/solr:9.0
:
: The benefits of the first are 1) the nice usability of being able to
: plainly specify "solr" and 2) the "Docker Official Images" badge on
: DockerHub. The downsides are that there are very strict requirements with
: creating Official Docker Images, which would complicate and require
: separating the way that we build release docker images and local docker
: images.

I don't know that this is strictly true? I think it would be possible to
support "local" images and "release" images fairly easily w/ a small
amount of tooling, but more on this idea farther below...


: *I think there is some consensus that choosing the "apache/solr" location
: is fine, and worth the added flexibility we get in the build process.*
...
: *The current opinion in the community is to abandon the "Docker Official
: Images" style process of downloading and validating official binaries, and
: instead having the release manager use the local-build image creation with
: the final release source.* This should result in the same docker image in
: the end, however there is no trust built into the docker image itself.
: Instead we are likely going to document a way for users to verify the
: docker-image contents themselves.

"Citation Needed" :)

Seriously though ... can you point me to when/where there has been
discussion/concensus about this?

And which community exactly are you refering to? -- Because I suspect that
on the subject of where/how to build the Docker images and if/how-much it
matters wether they live in _/solr vs apache/solr you may find a
discrepency between the "community of solr comitters" and the "docker user
community". Keep in mind: Mak didn't jump through the neccessary hoops
all those years ago to create _/solr just because he thought it would be
fun -- it would have been a lot easier to just create & publish
makuk66/solr images, but there was a (docker) community desire to have
"Docker official" _/solr images.

I think it would be presumptuous to assume that (docker) community desire
is going to go away just because the (apache solr) community is now
willing to publish apache/solr images.


With that in mind, I *think* there is a relatively straight forward way we
can have a "best of both worlds" type situation....

DISCLAIMER: I am not a docker expect, nor am i an expert on the docker-hub
rules for "official" images -- the idea below is just based on how the
current docker-solr/docker-solr github repo works. (I should also mention
I'm not up to speed on the docker related bits in lucene-solr master for
building "local" images, for now I'm just ignoring that and assuming it's
fair game to start over from scratch with how building docker images from
a source release/checkout work)

The crux of my suggestion is that we keep a Dockerfile.template in
the lucene-solr repo, on master (and any future branches) which has
templated values for the SOLR_VERSION and SHA512 of the
"solr-SOLR_VERSION.tgz" file names.

as part of the "gradle assemble" process, right after gradle creates
./packaging/build/distributions/solr-FOO.tgz and
./packaging/build/distributions/solr-FOO.tgz.sha512, the
Dockerfile.template should be used to create Dockerfile with the
corisponding file names and SHA512 values hardcoded into it.

the Dockerfile(.template) should have COPY directive that looks something
like...

COPY ./solr-FOO.tgz* /opt/solr/dist-archive

...followed by a RUN command invoking bash conditionals to download
solr-FOO.tgz from SOLR_DOWNLOAD_URL into /opt/solr/dist-archive if and
only if /opt/solr/dist-archive/solr-FOO.tgz doesn't already exist -- and
if it does exist to touch some file /opt/solr/dist-archive/this.is.a.local.build.txt

RUN script logic after this should attempt GPG validation (via KEYS & asc
files fetched from dist.apache.org) if and only if
/opt/solr/dist-archive/this.is.a.local.build.txt does not exist. (Either
way the hardcoded SHA512 value should be used to validate the tgz file
matches expectations)


This should allow us to have "Local" docker builds that don't require
any network access or downloading special files either by putting the
(post-templating) Dockerfile directly in ./packaging/build/distributions/
and telling people to use that as the context for "docker build" or by
some other gradle helper logic for constructing a special "docker context
dir" for people to build from where gradle copies the solr-FOO.tgz file.


"Official" docker-solr builds could use the exact same Dockerfile --
committed to the docker-solr/docker-solr (or some future
apache/docker-solr) repo by the RM via some release publishing scripts

(Any other "files" in the lucene-solr git repo that we want to be part
of docker images could likewise be copied to the "local" docker context
dir by gradle and/or to the apache/docker-solr repo by the release
publishing scripts -- but I would argue that as much as possible those
files should "live" inside the solr-FOO.tgz binary release)


Does anyone see any holes in this idea that i've missed? ... the need for
a "pre-processing" step to convert Dockerfile.template -> Dockerfile may
seem like a disadvantage of this process compared to other "simpler" ways
to build a "local" docker image (w/o needing any hard coded values) but no
matter what type of local docker image build process we'd like to have,
users will have to run *some* gradle command to compile the source code,
etc... so it doesn't seem like that much of a headache to add this if it
means the resulting Dockerfiles are also "viable" for use as official
_/solr Dockerfiles


-Hoss
http://www.lucidworks.com/

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
Re: Solr Docker discussion [ In reply to ]
I'd recommend building apache/solr:x.y.z as part of the normal solr release, then import those images into the official images library so that it also becomes solr:x.y.z. That way, you can move all the development/build/test work into your world (and deal with issues prior to release), and simplify the official publication step; and both can then be automated. This would remove a lot of toil from maintainers, and reduce the delay in image availability for our users, while retaining the "official image" badge and first-of-1600-matches listing on docker hub. It's then up to you which name you choose to promote in documentation, and whether to delay the release announcement till after the official image publication.

I don't know exactly what it takes for this to happen on the official-image library side, but it seems that Elastisearch does this [1]. I suggest Jan/David coordinate with @Tianon to see what is required in terms of verification, base images choice, rebuilding requirements and trigger mechanisms, and PR process changes. I would expect it to be fairly straightforward.

Then there is a question about how to roll this out. Do it the new way from the next Solr major version, and maintain the current repo for older version for N years? Or have the new way produce older supported versions like 7 and 8? The latter would be nice but perhaps more work and more difficult to do in a backwards-compatible way.

The docker-solr github org and repo namespace: it'd probably be best if they went away over time, assuming you can get what you need from the apache account in terms of administrative access.

On the regular rebuilding of images: Per [2] I do believe the library team do rebuilds when a base image is updated, and thus tags are not immutable; but I don't have data to prove if and how often this happens. You can argue the desirability of that both ways; but given that an application like Solr, especially running in a JVM, is quite isolated from the host OS, I think it's more a feature than a bug, and would suggest we continue doing that for apache/solr builds. It does mean you have to have some automated tests to ensure that a new candidate image actually works before tagging it. Somewhat related: for docker-solr I also backported fixes to the helper scripts to images for older Solr versions, because the repo templating made that easy. If these scripts are moving into the Solr tree itself, then that won't be happening automatically, and I think that's fine.

As to how exactly best implement the image building and publishing in your branch structure, build scripts and CI pipelines, I don't know; I'm not sufficiently familiar with those. But be careful with things like COPYing of tgz's, as that will lead to docker image layer wastage; doing something like a wget and extract in a single RUN command avoids that.

There should also be a discussion on what base images to use and what variants to offer. On the one extreme is the notion that it should not matter what the base is because you only care about the application, on the other extreme there are always users asking for full OS images, slim variants, alpine versions, debian/redhat/whoknows base OS distributions, and all possible versions and distributions of Java, on all possible machine architectures. That leads to exploding build times, test matrices and maintenance overhead. I'd be tempted to just pick one official Java based on a current debian:slim, build for amd64 and 64bit arm and call it a day; but I've not looked at the Java image landscape recently. Having a Dockerfile example that users could use to install a Solr tgz on their own different base images might be useful.

In response to Hoss' question about the initial choice to use _/solr and the github docker-solr org: I did that because there was a need for users to have Solr in a docker image that they could trust to be legitimate, would receive regular updates, would be respectful of backwards compatibility and would be functional enough for most use-cases. I could not have it in apache's namespace because that didn't exist, and the Solr project then wasn't really into Docker, or providing binary packaging for other third-party platforms. I did not want it in my or some other person's namespace (including my own) because that might lack credibility and would be problematic when they were no longer willing or able to provide updates. I did not want it in a company namespace because that could lead to misaligned interests. So by using _/solr, and setting up the docker-solr org, and getting buy-in from some Solr committers, I aimed to concentrate effort in a single community, and build trust and confidence in the image, and set things up so that they could easily be taken up by other Solr committers, or become part of the Apache Solr project, in future. So, it was a function of the circumstances at the time, and seems to have worked out pretty much as envisaged.

-- Martijn

[1] https://github.com/docker-library/elasticsearch/blob/master/7/Dockerfile
[2] https://github.com/docker-library/official-images#library-definition-files
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
Re: Solr Docker discussion [ In reply to ]
: I'd recommend building apache/solr:x.y.z as part of the normal solr
: release, then import those images into the official images library so
...
: I don't know exactly what it takes for this to happen on the
: official-image library side, but it seems that Elastisearch does this

Oh wow ... ok, yeah: I had no idea that was allowed by docker-library
based on the things I'd read about "official" docker images. (per the
links in the comments, the Dockerfile for the ES base image even uses
multi-stage builds which would help seriously simplify the our 'official'
image build process)

Yeah, if this is something docker-library will allow us to do as well --
define _/solr images directly 'FROM' apache/solr images -- and that
those are/can be reproducibly built from a (multistage) Dockerfile in our
git source repo identified by our release tag -- then by all means that
seems like the best way to go.

(I suspect the one finicky bit may be ensuring that docker-library is
ok with us using '--build-arg' to set the version when building our base
apache/solr images so that by building docker images from source checkout
defaults to building a snapshot)


-Hoss
http://www.lucidworks.com/

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
Re: Solr Docker discussion [ In reply to ]
Thanks for your input Hoss and Martijn.

So I've taken a look at the Elastic docker images [1], and they are
structured much like official images, except for the use of multi-stage
builds. They have no ARGs, and the docker context is merely the directory
in which the Dockerfile sits. With this in mind, and wanting to still take
advantage of the Official Docker Image, we will likely need to do some sort
of templating, as Hoss suggested. This can be automated using gradle fairly
easily I would imagine, and we could ensure that the only changes it makes
is to the part of the image that downloads Solr. That way there is very
little room for differences to occur between local and release images.

I do think that the current PR can be merged, regardless of our intent for
the release build, because the current way of building isn't going to be
acceptable for an official docker image release. I have made a ticket to
determine a release process for Solr docker images (9.0+) [2], this way we
can continue on with other parts of the Solr docker improvements while
being able to work on that in parallel.

To respond to the point about different architectures or operating system
images, I agree with you Martijn. I think we should support one operating
system by default, which would hopefully work with multiple architectures.
Dockerhub seems to support this for official images [3]. However if users
would want different operating system images, we will have given them the
ability to specify a custom base image and the dockerfile instructions
should be clear enough to migrate to other operating systems. That way they
can easily build their own.

- Houston

[1] https://github.com/elastic/dockerfiles/tree/v7.10.1/elasticsearch
[2] https://issues.apache.org/jira/browse/SOLR-15102
[3] https://github.com/docker-library/official-images#multiple-architectures

On Wed, Jan 20, 2021 at 11:55 AM Chris Hostetter <hossman_lucene@fucit.org>
wrote:

>
> : I'd recommend building apache/solr:x.y.z as part of the normal solr
> : release, then import those images into the official images library so
> ...
> : I don't know exactly what it takes for this to happen on the
> : official-image library side, but it seems that Elastisearch does this
>
> Oh wow ... ok, yeah: I had no idea that was allowed by docker-library
> based on the things I'd read about "official" docker images. (per the
> links in the comments, the Dockerfile for the ES base image even uses
> multi-stage builds which would help seriously simplify the our 'official'
> image build process)
>
> Yeah, if this is something docker-library will allow us to do as well --
> define _/solr images directly 'FROM' apache/solr images -- and that
> those are/can be reproducibly built from a (multistage) Dockerfile in our
> git source repo identified by our release tag -- then by all means that
> seems like the best way to go.
>
> (I suspect the one finicky bit may be ensuring that docker-library is
> ok with us using '--build-arg' to set the version when building our base
> apache/solr images so that by building docker images from source checkout
> defaults to building a snapshot)
>
>
> -Hoss
> http://www.lucidworks.com/
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: dev-help@lucene.apache.org
>
>
Re: Solr Docker discussion [ In reply to ]
: So I've taken a look at the Elastic docker images [1], and they are
: structured much like official images, except for the use of multi-stage
: builds. They have no ARGs, and the docker context is merely the directory
: in which the Dockerfile sits. With this in mind, and wanting to still take
: advantage of the Official Docker Image, we will likely need to do some sort
: of templating, as Hoss suggested. This can be automated using gradle fairly
: easily I would imagine, and we could ensure that the only changes it makes
: is to the part of the image that downloads Solr. That way there is very
: little room for differences to occur between local and release images.

If docker-library is willing to allow _/solr:X.Y images that are just
simple 'FROM apache/solr:X.Y' wrappers, and the apache/solr iamges can be
built with multi-stage builds, then i'm not sure we would really need
templating of the Dockerfile -- or to download a solr.tgz file.

The Dockerfile could have an initial stage that runs "gradle dist" to
compile the java source files -- using a build AEG to override the
default X.Y.Z-SNAPSHOT version variables -- while a later stage could
then take the resulting solr.tgz from the first stage and setup the file
layout for the final image.

So users wanting to build a "snapshot" docker image from their local
(possibly patched) checkout could just do...

docker build -f solr/docker/Dockerfile .

Or from a remote git branch:

docker build -f solr/docker/Dockerfile https://gitbox.apache.org/repos/asf/lucene-solr.git#branch_9x

while the official apache/solr:X.Y.Z image would could be reproducibly
buildable via...

docker build --build-arg SOLR_VERSION=X.Y.Z -f solr/docker/Dockerfile https://gitbox.apache.org/repos/asf/lucene-solr.git#releases/lucene-solr/X.Y.Z

Same Dockerfile in all use cases, no need for intermediate downloading
from any remote servers.


-Hoss
http://www.lucidworks.com/

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
Re: Solr Docker discussion [ In reply to ]
FYI: I've implemented a PoC of this idea and opened SOLR-15127...

https://issues.apache.org/jira/browse/SOLR-15127


: Date: Fri, 22 Jan 2021 14:37:19 -0700 (MST)
: From: Chris Hostetter <hossman_lucene@fucit.org>
: To: Solr/Lucene Dev <dev@lucene.apache.org>
: Subject: Re: Solr Docker discussion
:
:
: : So I've taken a look at the Elastic docker images [1], and they are
: : structured much like official images, except for the use of multi-stage
: : builds. They have no ARGs, and the docker context is merely the directory
: : in which the Dockerfile sits. With this in mind, and wanting to still take
: : advantage of the Official Docker Image, we will likely need to do some sort
: : of templating, as Hoss suggested. This can be automated using gradle fairly
: : easily I would imagine, and we could ensure that the only changes it makes
: : is to the part of the image that downloads Solr. That way there is very
: : little room for differences to occur between local and release images.
:
: If docker-library is willing to allow _/solr:X.Y images that are just
: simple 'FROM apache/solr:X.Y' wrappers, and the apache/solr iamges can be
: built with multi-stage builds, then i'm not sure we would really need
: templating of the Dockerfile -- or to download a solr.tgz file.
:
: The Dockerfile could have an initial stage that runs "gradle dist" to
: compile the java source files -- using a build AEG to override the
: default X.Y.Z-SNAPSHOT version variables -- while a later stage could
: then take the resulting solr.tgz from the first stage and setup the file
: layout for the final image.
:
: So users wanting to build a "snapshot" docker image from their local
: (possibly patched) checkout could just do...
:
: docker build -f solr/docker/Dockerfile .
:
: Or from a remote git branch:
:
: docker build -f solr/docker/Dockerfile https://gitbox.apache.org/repos/asf/lucene-solr.git#branch_9x
:
: while the official apache/solr:X.Y.Z image would could be reproducibly
: buildable via...
:
: docker build --build-arg SOLR_VERSION=X.Y.Z -f solr/docker/Dockerfile https://gitbox.apache.org/repos/asf/lucene-solr.git#releases/lucene-solr/X.Y.Z
:
: Same Dockerfile in all use cases, no need for intermediate downloading
: from any remote servers.
:
:
: -Hoss
: http://www.lucidworks.com/
:

-Hoss
http://www.lucidworks.com/

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org