Mailing List Archive

Recreating CI with Quibble
I'm trying to recreate a CI build to debug things locally. I've followed
the instructions under
https://doc.wikimedia.org/quibble/#reproducing-a-ci-build, but the command
for listing Docker images doesn't work. There are no images with "quibble"
in the name. Where can I find the available Quibble images?

*Sebastian Berlin*
Utvecklare/*Developer*
Wikimedia Sverige (WMSE)

E-post/*E-Mail*: sebastian.berlin@wikimedia.se
Telefon/*Phone*: (+46) 0707 - 92 03 84
Re: Recreating CI with Quibble [ In reply to ]
Hi, indeed the API request mentioned in the docs doesn't show up the
quibble images.

The docker registry mentioned in the docs is this one here [1] so you can
search in this page for the available versions of the quibble images.
For example this one [2] is built using debian buster and php 8.1.

[1] https://docker-registry.wikimedia.org/
[2] https://docker-registry.wikimedia.org/releng/quibble-buster-php81/tags/

On Fri, Apr 19, 2024 at 10:43?AM Sebastian Berlin <
sebastian.berlin@wikimedia.se> wrote:

> I'm trying to recreate a CI build to debug things locally. I've followed
> the instructions under
> https://doc.wikimedia.org/quibble/#reproducing-a-ci-build, but the
> command for listing Docker images doesn't work. There are no images with
> "quibble" in the name. Where can I find the available Quibble images?
>
> *Sebastian Berlin*
> Utvecklare/*Developer*
> Wikimedia Sverige (WMSE)
>
> E-post/*E-Mail*: sebastian.berlin@wikimedia.se
> Telefon/*Phone*: (+46) 0707 - 92 03 84
> _______________________________________________
> Wikitech-l mailing list -- wikitech-l@lists.wikimedia.org
> To unsubscribe send an email to wikitech-l-leave@lists.wikimedia.org
> https://lists.wikimedia.org/postorius/lists/wikitech-l.lists.wikimedia.org/
Re: Recreating CI with Quibble [ In reply to ]
Hi Sebastian,

I've also been playing around with debugging CI images on my local machine
recently. I've pushed an update to the Quibble docs on Mediawiki [1].
Hopefully that's helpful - feedback and changes welcome!

Thanks,

Arthur

[1] https://www.mediawiki.org/wiki/Continuous_integration/Quibble

Am Fr., 19. Apr. 2024 um 10:56 Uhr schrieb Yiannis Giannelos <
jgiannelos@wikimedia.org>:

> Hi, indeed the API request mentioned in the docs doesn't show up the
> quibble images.
>
> The docker registry mentioned in the docs is this one here [1] so you can
> search in this page for the available versions of the quibble images.
> For example this one [2] is built using debian buster and php 8.1.
>
> [1] https://docker-registry.wikimedia.org/
> [2]
> https://docker-registry.wikimedia.org/releng/quibble-buster-php81/tags/
>
> On Fri, Apr 19, 2024 at 10:43?AM Sebastian Berlin <
> sebastian.berlin@wikimedia.se> wrote:
>
>> I'm trying to recreate a CI build to debug things locally. I've followed
>> the instructions under
>> https://doc.wikimedia.org/quibble/#reproducing-a-ci-build, but the
>> command for listing Docker images doesn't work. There are no images with
>> "quibble" in the name. Where can I find the available Quibble images?
>>
>> *Sebastian Berlin*
>> Utvecklare/*Developer*
>> Wikimedia Sverige (WMSE)
>>
>> E-post/*E-Mail*: sebastian.berlin@wikimedia.se
>> Telefon/*Phone*: (+46) 0707 - 92 03 84
>> _______________________________________________
>> Wikitech-l mailing list -- wikitech-l@lists.wikimedia.org
>> To unsubscribe send an email to wikitech-l-leave@lists.wikimedia.org
>>
>> https://lists.wikimedia.org/postorius/lists/wikitech-l.lists.wikimedia.org/
>
> _______________________________________________
> Wikitech-l mailing list -- wikitech-l@lists.wikimedia.org
> To unsubscribe send an email to wikitech-l-leave@lists.wikimedia.org
> https://lists.wikimedia.org/postorius/lists/wikitech-l.lists.wikimedia.org/



--
Arthur Taylor
Senior Software Engineer

Wikimedia Deutschland e. V. | Tempelhofer Ufer 23-24 | 10963 Berlin
Phone: +49 (0)30-577 11 62-0
https://wikimedia.de

Keep up to date! Current news and exciting stories about Wikimedia,
Wikipedia and Free Knowledge in our newsletter (in German): Subscribe now
<https://www.wikimedia.de/newsletter/>.

Imagine a world in which every single human being can freely share in the
sum of all knowledge. Help us to achieve our vision!
https://spenden.wikimedia.de

Wikimedia Deutschland — Gesellschaft zur Förderung Freien Wissens e. V.
Eingetragen im Vereinsregister des Amtsgerichts Charlottenburg, VR 23855 B.
Als gemeinnützig anerkannt durch das Finanzamt für Körperschaften I Berlin,
Steuernummer 27/029/42207. Geschäftsführende Vorstände: Franziska Heine,
Dr. Christian Humborg
Re: Recreating CI with Quibble [ In reply to ]
Thank you both ????

I'll have a look at the updated documentation.

*Sebastian Berlin*
Utvecklare/*Developer*
Wikimedia Sverige (WMSE)

E-post/*E-Mail*: sebastian.berlin@wikimedia.se
Telefon/*Phone*: (+46) 0707 - 92 03 84


On Fri, 19 Apr 2024 at 11:03, Arthur Taylor <arthur.taylor@wikimedia.de>
wrote:

> Hi Sebastian,
>
> I've also been playing around with debugging CI images on my local machine
> recently. I've pushed an update to the Quibble docs on Mediawiki [1].
> Hopefully that's helpful - feedback and changes welcome!
>
> Thanks,
>
> Arthur
>
> [1] https://www.mediawiki.org/wiki/Continuous_integration/Quibble
>
> Am Fr., 19. Apr. 2024 um 10:56 Uhr schrieb Yiannis Giannelos <
> jgiannelos@wikimedia.org>:
>
>> Hi, indeed the API request mentioned in the docs doesn't show up the
>> quibble images.
>>
>> The docker registry mentioned in the docs is this one here [1] so you can
>> search in this page for the available versions of the quibble images.
>> For example this one [2] is built using debian buster and php 8.1.
>>
>> [1] https://docker-registry.wikimedia.org/
>> [2]
>> https://docker-registry.wikimedia.org/releng/quibble-buster-php81/tags/
>>
>> On Fri, Apr 19, 2024 at 10:43?AM Sebastian Berlin <
>> sebastian.berlin@wikimedia.se> wrote:
>>
>>> I'm trying to recreate a CI build to debug things locally. I've followed
>>> the instructions under
>>> https://doc.wikimedia.org/quibble/#reproducing-a-ci-build, but the
>>> command for listing Docker images doesn't work. There are no images with
>>> "quibble" in the name. Where can I find the available Quibble images?
>>>
>>> *Sebastian Berlin*
>>> Utvecklare/*Developer*
>>> Wikimedia Sverige (WMSE)
>>>
>>> E-post/*E-Mail*: sebastian.berlin@wikimedia.se
>>> Telefon/*Phone*: (+46) 0707 - 92 03 84
>>> _______________________________________________
>>> Wikitech-l mailing list -- wikitech-l@lists.wikimedia.org
>>> To unsubscribe send an email to wikitech-l-leave@lists.wikimedia.org
>>>
>>> https://lists.wikimedia.org/postorius/lists/wikitech-l.lists.wikimedia.org/
>>
>> _______________________________________________
>> Wikitech-l mailing list -- wikitech-l@lists.wikimedia.org
>> To unsubscribe send an email to wikitech-l-leave@lists.wikimedia.org
>>
>> https://lists.wikimedia.org/postorius/lists/wikitech-l.lists.wikimedia.org/
>
>
>
> --
> Arthur Taylor
> Senior Software Engineer
>
> Wikimedia Deutschland e. V. | Tempelhofer Ufer 23-24 | 10963 Berlin
> Phone: +49 (0)30-577 11 62-0
> https://wikimedia.de
>
> Keep up to date! Current news and exciting stories about Wikimedia,
> Wikipedia and Free Knowledge in our newsletter (in German): Subscribe now
> <https://www.wikimedia.de/newsletter/>.
>
> Imagine a world in which every single human being can freely share in the
> sum of all knowledge. Help us to achieve our vision!
> https://spenden.wikimedia.de
>
> Wikimedia Deutschland — Gesellschaft zur Förderung Freien Wissens e. V.
> Eingetragen im Vereinsregister des Amtsgerichts Charlottenburg, VR 23855 B.
> Als gemeinnützig anerkannt durch das Finanzamt für Körperschaften I Berlin,
> Steuernummer 27/029/42207. Geschäftsführende Vorstände: Franziska Heine,
> Dr. Christian Humborg
> _______________________________________________
> Wikitech-l mailing list -- wikitech-l@lists.wikimedia.org
> To unsubscribe send an email to wikitech-l-leave@lists.wikimedia.org
> https://lists.wikimedia.org/postorius/lists/wikitech-l.lists.wikimedia.org/
Re: Recreating CI with Quibble [ In reply to ]
Le 19/04/2024 à 11:03, Arthur Taylor a écrit :
> Hi Sebastian,
>
> I've also been playing around with debugging CI images on my local
> machine recently. I've pushed an update to the Quibble docs on
> Mediawiki [1]. Hopefully that's helpful - feedback and changes welcome!
>
> Thanks,
>
> Arthur
>
> [1] https://www.mediawiki.org/wiki/Continuous_integration/Quibble

Hello,

That documentation was merely an helper for people maintaining the CI
stack linking to the releasing doc which is in the repository, how to
rebuild the CI and then switch the jobs.

The Quibble manual / user documentation is inside the source repository
under the doc/source directory
https://gerrit.wikimedia.org/g/integration/quibble/+/refs/heads/master/doc/source
.  It ends up being published at https://doc.wikimedia.org/quibble/
which is the canonical place.

The wiki edit at
https://www.mediawiki.org/w/index.php?title=Continuous_integration/Quibble&diff=prev&oldid=6476384
<https://www.mediawiki.org/w/index.php?title=Continuous_integration/Quibble&diff=prev&oldid=6476384>
is amazing, and it would be really great to have it moved to the source
repository.  I like the Quabble.py script to pause before running tests,
though one can also ask Quibble to run bash as a test command: `quibble
-c bash`.

Antoine "hashar" Musso
_______________________________________________
Wikitech-l mailing list -- wikitech-l@lists.wikimedia.org
To unsubscribe send an email to wikitech-l-leave@lists.wikimedia.org
https://lists.wikimedia.org/postorius/lists/wikitech-l.lists.wikimedia.org/
Re: Recreating CI with Quibble [ In reply to ]
Le 19/04/2024 à 10:42, Sebastian Berlin a écrit :
> I'm trying to recreate a CI build to debug things locally. I've
> followed the instructions under
> https://doc.wikimedia.org/quibble/#reproducing-a-ci-build, but the
> command for listing Docker images doesn't work. There are no images
> with "quibble" in the name. Where can I find the available Quibble images?

Hello,

Quibble is not very user friendly. I started it during the 2017 Vienna
hackathon as a quick hack to unify the complicated stack of repos,
shell, python, Jenkins job into a single place. That has served to
reduce the CI complexity and made the jobs quite simpler.

Overall I can't say it is very pleasant to use as a human, there are a
few oddities such as:

- cleaning the dependencies and reinstalling everytime (use --skip-deps)
- misnamed options such as --skip-zuul to prevent refetching and doing a
git-clean of everything
- one need to set ZUUL_PROJECT to trigger the logic that runs
extension/skin tests
And more :)

I have sent a few patches recently to slightly improve it such as:

- automatically deleting the autogenerated LocalSettings.php instead of
having the MediaWiki installer to bail out after some minutes.

- Adding --change <gerrit change number> which retrieve from Gerrit the
metadata of the given change and initialize the ZUUL_* environment
variables (though I think one still has to set ZUUL_PROJECT to trigger
the proper logic ...)

Back on topic, the documentation about reproducing a CI build deserves
to be refreshed:

- the /v2/_catalog registry endpoint has evolved and is paginated
nowadays, see
https://distribution.github.io/distribution/spec/api/#listing-repositories
or look at which image is currently defined in CI (
https://gerrit.wikimedia.org/g/integration/config/+/refs/heads/master/jjb/mediawiki.yaml
).

Note that CI does a merge commit of the propose patch or patches against
the tip of the target branch. That is done by processes named
zuul-merger which exposes the resulting merges using git-daemon. Those
repositories are not publicly available, though they should be reachable
from a WMCS instance.

I am more than happy to assist and chat about Quibble. I will very
warmly welcome any help on that front :)

Antoine "hashar" Musso
_______________________________________________
Wikitech-l mailing list -- wikitech-l@lists.wikimedia.org
To unsubscribe send an email to wikitech-l-leave@lists.wikimedia.org
https://lists.wikimedia.org/postorius/lists/wikitech-l.lists.wikimedia.org/
Re: Recreating CI with Quibble [ In reply to ]
Hi Antoine,

Thanks for the positive feedback! I'll make a PR to move my documentation
edits over to the Quibble repo - thanks for the pointer there.

Happy you like the Quabble script - it's definitely not intended as a
long-term solution for anything - I just wrote it as a quick way to get my
local setup to do what I wanted. I would be happy to take a look at
contributing to Quibble itself and making it a bit more friendly for
debugging purposes.

I tried `quibble -c bash` this morning, or more completely I tried:

```
docker run --entrypoint=quibble-with-supervisord --tmpfs
/workspace/db:size=320M --volume "$(pwd)"/src:/workspace/src
--volume "$(pwd)"/cache:/cache --volume "$(pwd)"/log:/workspace/log
--volume "$(pwd)"/ref:/srv/git:ro --cap-add=SYS_PTRACE --security-opt
seccomp=unconfined --env-file=env-Wikibase-Master-20240410
--init --rm
docker-registry.wikimedia.org/releng/quibble-buster-php74:1.7.0-s1
--packages-source composer --db mysql --db-dir /workspace/db
--git-parallel=8 -c bash
```

and didn't get a shell. The command just exited immediately:

```
...
INFO:quibble.commands:>>> Start: User commands: bash
INFO:quibble.commands:User commands, working directory: /workspace/src
INFO:quibble.commands:bash
INFO:quibble.commands:<<< Finish: User commands: bash, in 0.006 s
INFO:quibble.commands:Shutting down backends: <ExternalWebserver
http://127.0.0.1:9413 /workspace/src> <Xvfb :94> <ChromeWebDriver :94>
...
```

so I'm not sure that addresses my wish to have an interactive debugging
environment with the backends running. But maybe I'm doing something wrong
there. Pointers welcome.

Will try and get to the documentation this week,

Many thanks,

Arthur


Am Fr., 19. Apr. 2024 um 14:50 Uhr schrieb Antoine Musso <hashar@free.fr>:

> Le 19/04/2024 à 11:03, Arthur Taylor a écrit :
> > Hi Sebastian,
> >
> > I've also been playing around with debugging CI images on my local
> > machine recently. I've pushed an update to the Quibble docs on
> > Mediawiki [1]. Hopefully that's helpful - feedback and changes welcome!
> >
> > Thanks,
> >
> > Arthur
> >
> > [1] https://www.mediawiki.org/wiki/Continuous_integration/Quibble
>
> Hello,
>
> That documentation was merely an helper for people maintaining the CI
> stack linking to the releasing doc which is in the repository, how to
> rebuild the CI and then switch the jobs.
>
> The Quibble manual / user documentation is inside the source repository
> under the doc/source directory
>
> https://gerrit.wikimedia.org/g/integration/quibble/+/refs/heads/master/doc/source
> . It ends up being published at https://doc.wikimedia.org/quibble/
> which is the canonical place.
>
> The wiki edit at
>
> https://www.mediawiki.org/w/index.php?title=Continuous_integration/Quibble&diff=prev&oldid=6476384
> <
> https://www.mediawiki.org/w/index.php?title=Continuous_integration/Quibble&diff=prev&oldid=6476384>
>
> is amazing, and it would be really great to have it moved to the source
> repository. I like the Quabble.py script to pause before running tests,
> though one can also ask Quibble to run bash as a test command: `quibble
> -c bash`.
>
> Antoine "hashar" Musso
>
>

--
Arthur Taylor
Senior Software Engineer

Wikimedia Deutschland e. V. | Tempelhofer Ufer 23-24 | 10963 Berlin
Phone: +49 (0)30-577 11 62-0
https://wikimedia.de

Keep up to date! Current news and exciting stories about Wikimedia,
Wikipedia and Free Knowledge in our newsletter (in German): Subscribe now
<https://www.wikimedia.de/newsletter/>.

Imagine a world in which every single human being can freely share in the
sum of all knowledge. Help us to achieve our vision!
https://spenden.wikimedia.de

Wikimedia Deutschland — Gesellschaft zur Förderung Freien Wissens e. V.
Eingetragen im Vereinsregister des Amtsgerichts Charlottenburg, VR 23855 B.
Als gemeinnützig anerkannt durch das Finanzamt für Körperschaften I Berlin,
Steuernummer 27/029/42207. Geschäftsführende Vorstände: Franziska Heine,
Dr. Christian Humborg
Re: Recreating CI with Quibble [ In reply to ]
Hi again,

My bad - `quibble -c bash` does launch a shell with the services running. I
just needed to add `-it` to the docker invocation. That makes things much
simpler - thank you!

Arthur


Am Mo., 22. Apr. 2024 um 09:03 Uhr schrieb Arthur Taylor <
arthur.taylor@wikimedia.de>:

> Hi Antoine,
>
> Thanks for the positive feedback! I'll make a PR to move my documentation
> edits over to the Quibble repo - thanks for the pointer there.
>
> Happy you like the Quabble script - it's definitely not intended as a
> long-term solution for anything - I just wrote it as a quick way to get my
> local setup to do what I wanted. I would be happy to take a look at
> contributing to Quibble itself and making it a bit more friendly for
> debugging purposes.
>
> I tried `quibble -c bash` this morning, or more completely I tried:
>
> ```
> docker run --entrypoint=quibble-with-supervisord --tmpfs
> /workspace/db:size=320M --volume "$(pwd)"/src:/workspace/src
> --volume "$(pwd)"/cache:/cache --volume "$(pwd)"/log:/workspace/log
> --volume "$(pwd)"/ref:/srv/git:ro --cap-add=SYS_PTRACE --security-opt
> seccomp=unconfined --env-file=env-Wikibase-Master-20240410
> --init --rm
> docker-registry.wikimedia.org/releng/quibble-buster-php74:1.7.0-s1
> --packages-source composer --db mysql --db-dir /workspace/db
> --git-parallel=8 -c bash
> ```
>
> and didn't get a shell. The command just exited immediately:
>
> ```
> ...
> INFO:quibble.commands:>>> Start: User commands: bash
> INFO:quibble.commands:User commands, working directory: /workspace/src
> INFO:quibble.commands:bash
> INFO:quibble.commands:<<< Finish: User commands: bash, in 0.006 s
> INFO:quibble.commands:Shutting down backends: <ExternalWebserver
> http://127.0.0.1:9413 /workspace/src> <Xvfb :94> <ChromeWebDriver :94>
> ...
> ```
>
> so I'm not sure that addresses my wish to have an interactive debugging
> environment with the backends running. But maybe I'm doing something wrong
> there. Pointers welcome.
>
> Will try and get to the documentation this week,
>
> Many thanks,
>
> Arthur
>
>
> Am Fr., 19. Apr. 2024 um 14:50 Uhr schrieb Antoine Musso <hashar@free.fr>:
>
>> Le 19/04/2024 à 11:03, Arthur Taylor a écrit :
>> > Hi Sebastian,
>> >
>> > I've also been playing around with debugging CI images on my local
>> > machine recently. I've pushed an update to the Quibble docs on
>> > Mediawiki [1]. Hopefully that's helpful - feedback and changes welcome!
>> >
>> > Thanks,
>> >
>> > Arthur
>> >
>> > [1] https://www.mediawiki.org/wiki/Continuous_integration/Quibble
>>
>> Hello,
>>
>> That documentation was merely an helper for people maintaining the CI
>> stack linking to the releasing doc which is in the repository, how to
>> rebuild the CI and then switch the jobs.
>>
>> The Quibble manual / user documentation is inside the source repository
>> under the doc/source directory
>>
>> https://gerrit.wikimedia.org/g/integration/quibble/+/refs/heads/master/doc/source
>> . It ends up being published at https://doc.wikimedia.org/quibble/
>> which is the canonical place.
>>
>> The wiki edit at
>>
>> https://www.mediawiki.org/w/index.php?title=Continuous_integration/Quibble&diff=prev&oldid=6476384
>> <
>> https://www.mediawiki.org/w/index.php?title=Continuous_integration/Quibble&diff=prev&oldid=6476384>
>>
>> is amazing, and it would be really great to have it moved to the source
>> repository. I like the Quabble.py script to pause before running tests,
>> though one can also ask Quibble to run bash as a test command: `quibble
>> -c bash`.
>>
>> Antoine "hashar" Musso
>>
>>
>
> --
> Arthur Taylor
> Senior Software Engineer
>
> Wikimedia Deutschland e. V. | Tempelhofer Ufer 23-24 | 10963 Berlin
> Phone: +49 (0)30-577 11 62-0
> https://wikimedia.de
>
> Keep up to date! Current news and exciting stories about Wikimedia,
> Wikipedia and Free Knowledge in our newsletter (in German): Subscribe now
> <https://www.wikimedia.de/newsletter/>.
>
> Imagine a world in which every single human being can freely share in the
> sum of all knowledge. Help us to achieve our vision!
> https://spenden.wikimedia.de
>
> Wikimedia Deutschland — Gesellschaft zur Förderung Freien Wissens e. V.
> Eingetragen im Vereinsregister des Amtsgerichts Charlottenburg, VR 23855 B.
> Als gemeinnützig anerkannt durch das Finanzamt für Körperschaften I Berlin,
> Steuernummer 27/029/42207. Geschäftsführende Vorstände: Franziska Heine,
> Dr. Christian Humborg
>


--
Arthur Taylor
Senior Software Engineer

Wikimedia Deutschland e. V. | Tempelhofer Ufer 23-24 | 10963 Berlin
Phone: +49 (0)30-577 11 62-0
https://wikimedia.de

Keep up to date! Current news and exciting stories about Wikimedia,
Wikipedia and Free Knowledge in our newsletter (in German): Subscribe now
<https://www.wikimedia.de/newsletter/>.

Imagine a world in which every single human being can freely share in the
sum of all knowledge. Help us to achieve our vision!
https://spenden.wikimedia.de

Wikimedia Deutschland — Gesellschaft zur Förderung Freien Wissens e. V.
Eingetragen im Vereinsregister des Amtsgerichts Charlottenburg, VR 23855 B.
Als gemeinnützig anerkannt durch das Finanzamt für Körperschaften I Berlin,
Steuernummer 27/029/42207. Geschäftsführende Vorstände: Franziska Heine,
Dr. Christian Humborg