Mailing List Archive

For the general case of Python projects, I’d argue that a better solution
is to adopt the lockfile pattern (package-lock.json, composer.lock,
Cargo.lock, etc.) and pin *all* dependencies, and only update them when the
new versions have been tested and are known to work. pip-tools
<https://pypi.org/project/pip-tools/> can help with that, for example (
requirements.in specifies “loose” dependencies; pip-compile creates a
pinned requirements.txt; pip-sync installs it; pip-compile -U upgrades
requirements.txt later; you check both requirements.in and requirements.txt
into version control.) But I don’t know if that applies in your
integration/config case.

Am Do., 4. Mai 2023 um 18:08 Uhr schrieb Antoine Musso <hashar@free.fr>:

> Hello,
>
> This is for python projects.
>
> Today, May 4th, urllib3 <https://pypi.org/project/urllib3/#history> has
> released a new major version 2.0.2 which breaks the extremely popular
> requests <https://pypi.org/project/requests/> library.
>
> The fix is to pin urllib3<2 to prevent the new major version from being
> installed (example
> <https://gerrit.wikimedia.org/r/c/integration/config/+/915736/1/tox.ini>).
>
> https://phabricator.wikimedia.org/T335977
>
> Upstream issue: https://github.com/psf/requests/issues/6432
>
>
> Antoine "hashar" Musso
> Wikimedia Release Engineering
> _______________________________________________
> Wikitech-l mailing list -- wikitech-l@lists.wikimedia.org
> To unsubscribe send an email to wikitech-l-leave@lists.wikimedia.org
> https://lists.wikimedia.org/postorius/lists/wikitech-l.lists.wikimedia.org/



--
Lucas Werkmeister (he/er)
Software Engineer

Wikimedia Deutschland e. V. | Tempelhofer Ufer 23-24 | 10963 Berlin
Phone: +49 (0)30-577 11 62-0
https://wikimedia.de

Imagine a world in which every single human being can freely share in the
sum of all knowledge. Help us to achieve our vision!
https://spenden.wikimedia.de

Wikimedia Deutschland - Gesellschaft zur Förderung Freien Wissens e. V.
Eingetragen im Vereinsregister des Amtsgerichts Berlin-Charlottenburg unter
der Nummer 23855 B. Als gemeinnützig anerkannt durch das Finanzamt für
Körperschaften I Berlin, Steuernummer 27/029/42207.

You can also create an empty virtual env, install all requirements and then
do
pip freeze > requirements.txt

That should take care of pinning

Am Fr., 5. Mai 2023 um 13:11 Uhr schrieb Lucas Werkmeister <
lucas.werkmeister@wikimedia.de>:

> For the general case of Python projects, I’d argue that a better solution
> is to adopt the lockfile pattern (package-lock.json, composer.lock,
> Cargo.lock, etc.) and pin *all* dependencies, and only update them when
> the new versions have been tested and are known to work. pip-tools
> <https://pypi.org/project/pip-tools/> can help with that, for example (
> requirements.in specifies “loose” dependencies; pip-compile creates a
> pinned requirements.txt; pip-sync installs it; pip-compile -U upgrades
> requirements.txt later; you check both requirements.in and
> requirements.txt into version control.) But I don’t know if that applies
> in your integration/config case.
>
> Am Do., 4. Mai 2023 um 18:08 Uhr schrieb Antoine Musso <hashar@free.fr>:
>
>> Hello,
>>
>> This is for python projects.
>>
>> Today, May 4th, urllib3 <https://pypi.org/project/urllib3/#history> has
>> released a new major version 2.0.2 which breaks the extremely popular
>> requests <https://pypi.org/project/requests/> library.
>>
>> The fix is to pin urllib3<2 to prevent the new major version from being
>> installed (example
>> <https://gerrit.wikimedia.org/r/c/integration/config/+/915736/1/tox.ini>
>> ).
>>
>> https://phabricator.wikimedia.org/T335977
>>
>> Upstream issue: https://github.com/psf/requests/issues/6432
>>
>>
>> Antoine "hashar" Musso
>> Wikimedia Release Engineering
>> _______________________________________________
>> Wikitech-l mailing list -- wikitech-l@lists.wikimedia.org
>> To unsubscribe send an email to wikitech-l-leave@lists.wikimedia.org
>>
>> https://lists.wikimedia.org/postorius/lists/wikitech-l.lists.wikimedia.org/
>
>
>
> --
> Lucas Werkmeister (he/er)
> Software Engineer
>
> Wikimedia Deutschland e. V. | Tempelhofer Ufer 23-24 | 10963 Berlin
> Phone: +49 (0)30-577 11 62-0
> https://wikimedia.de
>
> Imagine a world in which every single human being can freely share in the
> sum of all knowledge. Help us to achieve our vision!
> https://spenden.wikimedia.de
>
> Wikimedia Deutschland - Gesellschaft zur Förderung Freien Wissens e. V.
> Eingetragen im Vereinsregister des Amtsgerichts Berlin-Charlottenburg unter
> der Nummer 23855 B. Als gemeinnützig anerkannt durch das Finanzamt für
> Körperschaften I Berlin, Steuernummer 27/029/42207.
> _______________________________________________
> Wikitech-l mailing list -- wikitech-l@lists.wikimedia.org
> To unsubscribe send an email to wikitech-l-leave@lists.wikimedia.org
> https://lists.wikimedia.org/postorius/lists/wikitech-l.lists.wikimedia.org/



--
Amir (he/him)

A word of warning: using `pip freeze` to populate requirements.txt can
result in a hard to read (very long) file and other issues:
https://medium.com/@tomagee/pip-freeze-requirements-txt-considered-harmful-f0bce66cf895
.

*Sebastian Berlin*
Utvecklare/*Developer*
Wikimedia Sverige (WMSE)

E-post/*E-Mail*: sebastian.berlin@wikimedia.se
Telefon/*Phone*: (+46) 0707 - 92 03 84


On Fri, 5 May 2023 at 13:17, Amir Sarabadani <ladsgroup@gmail.com> wrote:

> You can also create an empty virtual env, install all requirements and
> then do
> pip freeze > requirements.txt
>
> That should take care of pinning
>
> Am Fr., 5. Mai 2023 um 13:11 Uhr schrieb Lucas Werkmeister <
> lucas.werkmeister@wikimedia.de>:
>
>> For the general case of Python projects, I’d argue that a better solution
>> is to adopt the lockfile pattern (package-lock.json, composer.lock,
>> Cargo.lock, etc.) and pin *all* dependencies, and only update them when
>> the new versions have been tested and are known to work. pip-tools
>> <https://pypi.org/project/pip-tools/> can help with that, for example (
>> requirements.in specifies “loose” dependencies; pip-compile creates a
>> pinned requirements.txt; pip-sync installs it; pip-compile -U upgrades
>> requirements.txt later; you check both requirements.in and
>> requirements.txt into version control.) But I don’t know if that applies
>> in your integration/config case.
>>
>> Am Do., 4. Mai 2023 um 18:08 Uhr schrieb Antoine Musso <hashar@free.fr>:
>>
>>> Hello,
>>>
>>> This is for python projects.
>>>
>>> Today, May 4th, urllib3 <https://pypi.org/project/urllib3/#history> has
>>> released a new major version 2.0.2 which breaks the extremely popular
>>> requests <https://pypi.org/project/requests/> library.
>>>
>>> The fix is to pin urllib3<2 to prevent the new major version from being
>>> installed (example
>>> <https://gerrit.wikimedia.org/r/c/integration/config/+/915736/1/tox.ini>
>>> ).
>>>
>>> https://phabricator.wikimedia.org/T335977
>>>
>>> Upstream issue: https://github.com/psf/requests/issues/6432
>>>
>>>
>>> Antoine "hashar" Musso
>>> Wikimedia Release Engineering
>>> _______________________________________________
>>> Wikitech-l mailing list -- wikitech-l@lists.wikimedia.org
>>> To unsubscribe send an email to wikitech-l-leave@lists.wikimedia.org
>>>
>>> https://lists.wikimedia.org/postorius/lists/wikitech-l.lists.wikimedia.org/
>>
>>
>>
>> --
>> Lucas Werkmeister (he/er)
>> Software Engineer
>>
>> Wikimedia Deutschland e. V. | Tempelhofer Ufer 23-24 | 10963 Berlin
>> Phone: +49 (0)30-577 11 62-0
>> https://wikimedia.de
>>
>> Imagine a world in which every single human being can freely share in the
>> sum of all knowledge. Help us to achieve our vision!
>> https://spenden.wikimedia.de
>>
>> Wikimedia Deutschland - Gesellschaft zur Förderung Freien Wissens e. V.
>> Eingetragen im Vereinsregister des Amtsgerichts Berlin-Charlottenburg unter
>> der Nummer 23855 B. Als gemeinnützig anerkannt durch das Finanzamt für
>> Körperschaften I Berlin, Steuernummer 27/029/42207.
>> _______________________________________________
>> Wikitech-l mailing list -- wikitech-l@lists.wikimedia.org
>> To unsubscribe send an email to wikitech-l-leave@lists.wikimedia.org
>>
>> https://lists.wikimedia.org/postorius/lists/wikitech-l.lists.wikimedia.org/
>
>
>
> --
> Amir (he/him)
>
> _______________________________________________
> Wikitech-l mailing list -- wikitech-l@lists.wikimedia.org
> To unsubscribe send an email to wikitech-l-leave@lists.wikimedia.org
> https://lists.wikimedia.org/postorius/lists/wikitech-l.lists.wikimedia.org/

Poetry is a modern lockfile-based packaging and dependency management tool
worth looking into. It also supports exporting dependencies into a
requirements.txt file, should you need that (nice if you want to
containerize an app without bloating the image with Poetry, for instance).

https://python-poetry.org/ <https://python-poetry.org/>

--
Slavina Stefanova (she/her)
Software Engineer - Technical Engagement

Wikimedia Foundation


On Fri, May 5, 2023 at 1:38?PM Sebastian Berlin <
sebastian.berlin@wikimedia.se> wrote:

> A word of warning: using `pip freeze` to populate requirements.txt can
> result in a hard to read (very long) file and other issues:
> https://medium.com/@tomagee/pip-freeze-requirements-txt-considered-harmful-f0bce66cf895
> .
>
> *Sebastian Berlin*
> Utvecklare/*Developer*
> Wikimedia Sverige (WMSE)
>
> E-post/*E-Mail*: sebastian.berlin@wikimedia.se
> Telefon/*Phone*: (+46) 0707 - 92 03 84
>
>
> On Fri, 5 May 2023 at 13:17, Amir Sarabadani <ladsgroup@gmail.com> wrote:
>
>> You can also create an empty virtual env, install all requirements and
>> then do
>> pip freeze > requirements.txt
>>
>> That should take care of pinning
>>
>> Am Fr., 5. Mai 2023 um 13:11 Uhr schrieb Lucas Werkmeister <
>> lucas.werkmeister@wikimedia.de>:
>>
>>> For the general case of Python projects, I’d argue that a better
>>> solution is to adopt the lockfile pattern (package-lock.json,
>>> composer.lock, Cargo.lock, etc.) and pin *all* dependencies, and only
>>> update them when the new versions have been tested and are known to work.
>>> pip-tools <https://pypi.org/project/pip-tools/> can help with that, for
>>> example (requirements.in specifies “loose” dependencies; pip-compile
>>> creates a pinned requirements.txt; pip-sync installs it; pip-compile -U
>>> upgrades requirements.txt later; you check both requirements.in and
>>> requirements.txt into version control.) But I don’t know if that
>>> applies in your integration/config case.
>>>
>>> Am Do., 4. Mai 2023 um 18:08 Uhr schrieb Antoine Musso <hashar@free.fr>:
>>>
>>>> Hello,
>>>>
>>>> This is for python projects.
>>>>
>>>> Today, May 4th, urllib3 <https://pypi.org/project/urllib3/#history>
>>>> has released a new major version 2.0.2 which breaks the extremely popular
>>>> requests <https://pypi.org/project/requests/> library.
>>>>
>>>> The fix is to pin urllib3<2 to prevent the new major version from being
>>>> installed (example
>>>> <https://gerrit.wikimedia.org/r/c/integration/config/+/915736/1/tox.ini>
>>>> ).
>>>>
>>>> https://phabricator.wikimedia.org/T335977
>>>>
>>>> Upstream issue: https://github.com/psf/requests/issues/6432
>>>>
>>>>
>>>> Antoine "hashar" Musso
>>>> Wikimedia Release Engineering
>>>> _______________________________________________
>>>> Wikitech-l mailing list -- wikitech-l@lists.wikimedia.org
>>>> To unsubscribe send an email to wikitech-l-leave@lists.wikimedia.org
>>>>
>>>> https://lists.wikimedia.org/postorius/lists/wikitech-l.lists.wikimedia.org/
>>>
>>>
>>>
>>> --
>>> Lucas Werkmeister (he/er)
>>> Software Engineer
>>>
>>> Wikimedia Deutschland e. V. | Tempelhofer Ufer 23-24 | 10963 Berlin
>>> Phone: +49 (0)30-577 11 62-0
>>> https://wikimedia.de
>>>
>>> Imagine a world in which every single human being can freely share in
>>> the sum of all knowledge. Help us to achieve our vision!
>>> https://spenden.wikimedia.de
>>>
>>> Wikimedia Deutschland - Gesellschaft zur Förderung Freien Wissens e. V.
>>> Eingetragen im Vereinsregister des Amtsgerichts Berlin-Charlottenburg unter
>>> der Nummer 23855 B. Als gemeinnützig anerkannt durch das Finanzamt für
>>> Körperschaften I Berlin, Steuernummer 27/029/42207.
>>> _______________________________________________
>>> Wikitech-l mailing list -- wikitech-l@lists.wikimedia.org
>>> To unsubscribe send an email to wikitech-l-leave@lists.wikimedia.org
>>>
>>> https://lists.wikimedia.org/postorius/lists/wikitech-l.lists.wikimedia.org/
>>
>>
>>
>> --
>> Amir (he/him)
>>
>> _______________________________________________
>> Wikitech-l mailing list -- wikitech-l@lists.wikimedia.org
>> To unsubscribe send an email to wikitech-l-leave@lists.wikimedia.org
>>
>> https://lists.wikimedia.org/postorius/lists/wikitech-l.lists.wikimedia.org/
>
> _______________________________________________
> Wikitech-l mailing list -- wikitech-l@lists.wikimedia.org
> To unsubscribe send an email to wikitech-l-leave@lists.wikimedia.org
> https://lists.wikimedia.org/postorius/lists/wikitech-l.lists.wikimedia.org/

Tangent: is it worthwhile to establish a consensus for best practices with package pinning and package management for Python projects in the Wikimedia ecosystem? When I last worked on a python project (https://wikitech.wikimedia.org/wiki/Add_Link) I found it confusing that we have so many different tools and approaches for doing these things, and seems like we'd benefit from having a standard, supported way. (Or maybe that already exists and I haven't found it?)

Kosta

> On 5. May 2023, at 13:51, Slavina Stefanova <sstefanova@wikimedia.org> wrote:
>
> Poetry is a modern lockfile-based packaging and dependency management tool worth looking into. It also supports exporting dependencies into a requirements.txt file, should you need that (nice if you want to containerize an app without bloating the image with Poetry, for instance).
>
> https://python-poetry.org/  <https://python-poetry.org/>
>
> --
> Slavina Stefanova (she/her)
> Software Engineer - Technical Engagement
>
> Wikimedia Foundation
>
>
> On Fri, May 5, 2023 at 1:38?PM Sebastian Berlin <sebastian.berlin@wikimedia.se <mailto:sebastian.berlin@wikimedia.se>> wrote:
>> A word of warning: using `pip freeze` to populate requirements.txt can result in a hard to read (very long) file and other issues: https://medium.com/@tomagee/pip-freeze-requirements-txt-considered-harmful-f0bce66cf895.
>>
>> Sebastian Berlin
>> Utvecklare/Developer
>> Wikimedia Sverige (WMSE)
>>
>> E-post/E-Mail: sebastian.berlin@wikimedia.se <mailto:sebastian.berlin@wikimedia.se>
>> Telefon/Phone: (+46) 0707 - 92 03 84
>>
>>
>> On Fri, 5 May 2023 at 13:17, Amir Sarabadani <ladsgroup@gmail.com <mailto:ladsgroup@gmail.com>> wrote:
>>> You can also create an empty virtual env, install all requirements and then do
>>> pip freeze > requirements.txt
>>>
>>> That should take care of pinning
>>>
>>> Am Fr., 5. Mai 2023 um 13:11 Uhr schrieb Lucas Werkmeister <lucas.werkmeister@wikimedia.de <mailto:lucas.werkmeister@wikimedia.de>>:
>>>> For the general case of Python projects, I’d argue that a better solution is to adopt the lockfile pattern (package-lock.json, composer.lock, Cargo.lock, etc.) and pin all dependencies, and only update them when the new versions have been tested and are known to work. pip-tools <https://pypi.org/project/pip-tools/> can help with that, for example (requirements.in <http://requirements.in/> specifies “loose” dependencies; pip-compile creates a pinned requirements.txt; pip-sync installs it; pip-compile -U upgrades requirements.txt later; you check both requirements.in <http://requirements.in/> and requirements.txt into version control.) But I don’t know if that applies in your integration/config case.
>>>>
>>>> Am Do., 4. Mai 2023 um 18:08 Uhr schrieb Antoine Musso <hashar@free.fr <mailto:hashar@free.fr>>:
>>>>> Hello,
>>>>>
>>>>> This is for python projects.
>>>>>
>>>>> Today, May 4th, urllib3 <https://pypi.org/project/urllib3/#history> has released a new major version 2.0.2 which breaks the extremely popular requests <https://pypi.org/project/requests/> library.
>>>>>
>>>>> The fix is to pin urllib3<2 to prevent the new major version from being installed (example <https://gerrit.wikimedia.org/r/c/integration/config/+/915736/1/tox.ini>).
>>>>>
>>>>> https://phabricator.wikimedia.org/T335977
>>>>>
>>>>> Upstream issue: https://github.com/psf/requests/issues/6432
>>>>>
>>>>>
>>>>>
>>>>> Antoine "hashar" Musso
>>>>> Wikimedia Release Engineering
>>>>> _______________________________________________
>>>>> Wikitech-l mailing list -- wikitech-l@lists.wikimedia.org <mailto:wikitech-l@lists.wikimedia.org>
>>>>> To unsubscribe send an email to wikitech-l-leave@lists.wikimedia.org <mailto:wikitech-l-leave@lists.wikimedia.org>
>>>>> https://lists.wikimedia.org/postorius/lists/wikitech-l.lists.wikimedia.org/
>>>>
>>>>
>>>> --
>>>> Lucas Werkmeister (he/er)
>>>> Software Engineer
>>>>
>>>> Wikimedia Deutschland e. V. | Tempelhofer Ufer 23-24 | 10963 Berlin
>>>> Phone: +49 (0)30-577 11 62-0
>>>> https://wikimedia.de <https://wikimedia.de/>
>>>>
>>>> Imagine a world in which every single human being can freely share in the sum of all knowledge. Help us to achieve our vision!
>>>> https://spenden.wikimedia.de <https://spenden.wikimedia.de/>
>>>>
>>>> Wikimedia Deutschland - Gesellschaft zur Förderung Freien Wissens e. V. Eingetragen im Vereinsregister des Amtsgerichts Berlin-Charlottenburg unter der Nummer 23855 B. Als gemeinnützig anerkannt durch das Finanzamt für Körperschaften I Berlin, Steuernummer 27/029/42207.
>>>> _______________________________________________
>>>> Wikitech-l mailing list -- wikitech-l@lists.wikimedia.org <mailto:wikitech-l@lists.wikimedia.org>
>>>> To unsubscribe send an email to wikitech-l-leave@lists.wikimedia.org <mailto:wikitech-l-leave@lists.wikimedia.org>
>>>> https://lists.wikimedia.org/postorius/lists/wikitech-l.lists.wikimedia.org/
>>>
>>>
>>> --
>>> Amir (he/him)
>>>
>>> _______________________________________________
>>> Wikitech-l mailing list -- wikitech-l@lists.wikimedia.org <mailto:wikitech-l@lists.wikimedia.org>
>>> To unsubscribe send an email to wikitech-l-leave@lists.wikimedia.org <mailto:wikitech-l-leave@lists.wikimedia.org>
>>> https://lists.wikimedia.org/postorius/lists/wikitech-l.lists.wikimedia.org/
>> _______________________________________________
>> Wikitech-l mailing list -- wikitech-l@lists.wikimedia.org <mailto:wikitech-l@lists.wikimedia.org>
>> To unsubscribe send an email to wikitech-l-leave@lists.wikimedia.org <mailto:wikitech-l-leave@lists.wikimedia.org>
>> https://lists.wikimedia.org/postorius/lists/wikitech-l.lists.wikimedia.org/
> _______________________________________________
> Wikitech-l mailing list -- wikitech-l@lists.wikimedia.org
> To unsubscribe send an email to wikitech-l-leave@lists.wikimedia.org
> https://lists.wikimedia.org/postorius/lists/wikitech-l.lists.wikimedia.org/

For Java, we run an instance of Archiva: https://archiva.wikimedia.org/

It's not a perfect approach but I think we can and should move in that
direction with all our other ecosystems (python, Javascript, PHP). Our
reduction in security-relevant surface area alone would be worth it.

On Fri, May 5, 2023 at 11:18?AM Kosta Harlan <kharlan@wikimedia.org> wrote:

> Tangent: is it worthwhile to establish a consensus for best practices with
> package pinning and package management for Python projects in the Wikimedia
> ecosystem? When I last worked on a python project (
> https://wikitech.wikimedia.org/wiki/Add_Link) I found it confusing that
> we have so many different tools and approaches for doing these things, and
> seems like we'd benefit from having a standard, supported way. (Or maybe
> that already exists and I haven't found it?)
>
> Kosta
>
> On 5. May 2023, at 13:51, Slavina Stefanova <sstefanova@wikimedia.org>
> wrote:
>
> Poetry is a modern lockfile-based packaging and dependency management tool
> worth looking into. It also supports exporting dependencies into a
> requirements.txt file, should you need that (nice if you want to
> containerize an app without bloating the image with Poetry, for instance).
>
> https://python-poetry.org/ <https://python-poetry.org/>
>
> --
> Slavina Stefanova (she/her)
> Software Engineer - Technical Engagement
>
> Wikimedia Foundation
>
>
> On Fri, May 5, 2023 at 1:38?PM Sebastian Berlin <
> sebastian.berlin@wikimedia.se> wrote:
>
>> A word of warning: using `pip freeze` to populate requirements.txt can
>> result in a hard to read (very long) file and other issues:
>> https://medium.com/@tomagee/pip-freeze-requirements-txt-considered-harmful-f0bce66cf895
>> .
>>
>> *Sebastian Berlin*
>> Utvecklare/*Developer*
>> Wikimedia Sverige (WMSE)
>>
>> E-post/*E-Mail*: sebastian.berlin@wikimedia.se
>> Telefon/*Phone*: (+46) 0707 - 92 03 84
>>
>>
>> On Fri, 5 May 2023 at 13:17, Amir Sarabadani <ladsgroup@gmail.com> wrote:
>>
>>> You can also create an empty virtual env, install all requirements and
>>> then do
>>> pip freeze > requirements.txt
>>>
>>> That should take care of pinning
>>>
>>> Am Fr., 5. Mai 2023 um 13:11 Uhr schrieb Lucas Werkmeister <
>>> lucas.werkmeister@wikimedia.de>:
>>>
>>>> For the general case of Python projects, I’d argue that a better
>>>> solution is to adopt the lockfile pattern (package-lock.json,
>>>> composer.lock, Cargo.lock, etc.) and pin *all* dependencies, and only
>>>> update them when the new versions have been tested and are known to work.
>>>> pip-tools <https://pypi.org/project/pip-tools/> can help with that,
>>>> for example (requirements.in specifies “loose” dependencies;
>>>> pip-compile creates a pinned requirements.txt; pip-sync installs it; pip-compile
>>>> -U upgrades requirements.txt later; you check both requirements.in and
>>>> requirements.txt into version control.) But I don’t know if that
>>>> applies in your integration/config case.
>>>>
>>>> Am Do., 4. Mai 2023 um 18:08 Uhr schrieb Antoine Musso <hashar@free.fr
>>>> >:
>>>>
>>>>> Hello,
>>>>>
>>>>> This is for python projects.
>>>>>
>>>>> Today, May 4th, urllib3 <https://pypi.org/project/urllib3/#history>
>>>>> has released a new major version 2.0.2 which breaks the extremely popular
>>>>> requests <https://pypi.org/project/requests/> library.
>>>>>
>>>>> The fix is to pin urllib3<2 to prevent the new major version from
>>>>> being installed (example
>>>>> <https://gerrit.wikimedia.org/r/c/integration/config/+/915736/1/tox.ini>
>>>>> ).
>>>>>
>>>>> https://phabricator.wikimedia.org/T335977
>>>>>
>>>>> Upstream issue: https://github.com/psf/requests/issues/6432
>>>>>
>>>>>
>>>>> Antoine "hashar" Musso
>>>>> Wikimedia Release Engineering
>>>>> _______________________________________________
>>>>> Wikitech-l mailing list -- wikitech-l@lists.wikimedia.org
>>>>> To unsubscribe send an email to wikitech-l-leave@lists.wikimedia.org
>>>>>
>>>>> https://lists.wikimedia.org/postorius/lists/wikitech-l.lists.wikimedia.org/
>>>>
>>>>
>>>>
>>>> --
>>>> Lucas Werkmeister (he/er)
>>>> Software Engineer
>>>>
>>>> Wikimedia Deutschland e. V. | Tempelhofer Ufer 23-24 | 10963 Berlin
>>>> Phone: +49 (0)30-577 11 62-0
>>>> https://wikimedia.de
>>>>
>>>> Imagine a world in which every single human being can freely share in
>>>> the sum of all knowledge. Help us to achieve our vision!
>>>> https://spenden.wikimedia.de
>>>>
>>>> Wikimedia Deutschland - Gesellschaft zur Förderung Freien Wissens e. V.
>>>> Eingetragen im Vereinsregister des Amtsgerichts Berlin-Charlottenburg unter
>>>> der Nummer 23855 B. Als gemeinnützig anerkannt durch das Finanzamt für
>>>> Körperschaften I Berlin, Steuernummer 27/029/42207.
>>>> _______________________________________________
>>>> Wikitech-l mailing list -- wikitech-l@lists.wikimedia.org
>>>> To unsubscribe send an email to wikitech-l-leave@lists.wikimedia.org
>>>>
>>>> https://lists.wikimedia.org/postorius/lists/wikitech-l.lists.wikimedia.org/
>>>
>>>
>>>
>>> --
>>> Amir (he/him)
>>>
>>> _______________________________________________
>>> Wikitech-l mailing list -- wikitech-l@lists.wikimedia.org
>>> To unsubscribe send an email to wikitech-l-leave@lists.wikimedia.org
>>>
>>> https://lists.wikimedia.org/postorius/lists/wikitech-l.lists.wikimedia.org/
>>
>> _______________________________________________
>> Wikitech-l mailing list -- wikitech-l@lists.wikimedia.org
>> To unsubscribe send an email to wikitech-l-leave@lists.wikimedia.org
>>
>> https://lists.wikimedia.org/postorius/lists/wikitech-l.lists.wikimedia.org/
>
> _______________________________________________
> Wikitech-l mailing list -- wikitech-l@lists.wikimedia.org
> To unsubscribe send an email to wikitech-l-leave@lists.wikimedia.org
> https://lists.wikimedia.org/postorius/lists/wikitech-l.lists.wikimedia.org/
>
>
> _______________________________________________
> Wikitech-l mailing list -- wikitech-l@lists.wikimedia.org
> To unsubscribe send an email to wikitech-l-leave@lists.wikimedia.org
> https://lists.wikimedia.org/postorius/lists/wikitech-l.lists.wikimedia.org/

>
> Tangent: is it worthwhile to establish a consensus for best practices with
> package pinning and package management for Python projects in the Wikimedia
> ecosystem? When I last worked on a python project (
> https://wikitech.wikimedia.org/wiki/Add_Link) I found it confusing that
> we have so many different tools and approaches for doing these things, and
> seems like we'd benefit from having a standard, supported way. (Or maybe
> that already exists and I haven't found it?)


I'm working on an "Essential Tools for Managing Python Development
Environments
<https://docs.google.com/document/d/1pl6QCDWGebGjRrHixgQNES8qRf-8PCzwLSsFj7jI1HY/edit#heading=h.i5ou2ywgmb09>"
tutorial that will be published to the wikis when ready. Maybe that could
be expanded upon? In my experience though, it can be hard to get people to
agree on following a standard, especially when there are so many different
options and many folks already have their favorite tools and workflows. But
it would be nice to have a set of recommendations to reduce the cognitive
load.

--
Slavina Stefanova (she/her)
Software Engineer - Technical Engagement

Wikimedia Foundation


On Fri, May 5, 2023 at 5:18?PM Kosta Harlan <kharlan@wikimedia.org> wrote:

> Tangent: is it worthwhile to establish a consensus for best practices with
> package pinning and package management for Python projects in the Wikimedia
> ecosystem? When I last worked on a python project (
> https://wikitech.wikimedia.org/wiki/Add_Link) I found it confusing that
> we have so many different tools and approaches for doing these things, and
> seems like we'd benefit from having a standard, supported way. (Or maybe
> that already exists and I haven't found it?)
>
> Kosta
>
> On 5. May 2023, at 13:51, Slavina Stefanova <sstefanova@wikimedia.org>
> wrote:
>
> Poetry is a modern lockfile-based packaging and dependency management tool
> worth looking into. It also supports exporting dependencies into a
> requirements.txt file, should you need that (nice if you want to
> containerize an app without bloating the image with Poetry, for instance).
>
> https://python-poetry.org/ <https://python-poetry.org/>
>
> --
> Slavina Stefanova (she/her)
> Software Engineer - Technical Engagement
>
> Wikimedia Foundation
>
>
> On Fri, May 5, 2023 at 1:38?PM Sebastian Berlin <
> sebastian.berlin@wikimedia.se> wrote:
>
>> A word of warning: using `pip freeze` to populate requirements.txt can
>> result in a hard to read (very long) file and other issues:
>> https://medium.com/@tomagee/pip-freeze-requirements-txt-considered-harmful-f0bce66cf895
>> .
>>
>> *Sebastian Berlin*
>> Utvecklare/*Developer*
>> Wikimedia Sverige (WMSE)
>>
>> E-post/*E-Mail*: sebastian.berlin@wikimedia.se
>> Telefon/*Phone*: (+46) 0707 - 92 03 84
>>
>>
>> On Fri, 5 May 2023 at 13:17, Amir Sarabadani <ladsgroup@gmail.com> wrote:
>>
>>> You can also create an empty virtual env, install all requirements and
>>> then do
>>> pip freeze > requirements.txt
>>>
>>> That should take care of pinning
>>>
>>> Am Fr., 5. Mai 2023 um 13:11 Uhr schrieb Lucas Werkmeister <
>>> lucas.werkmeister@wikimedia.de>:
>>>
>>>> For the general case of Python projects, I’d argue that a better
>>>> solution is to adopt the lockfile pattern (package-lock.json,
>>>> composer.lock, Cargo.lock, etc.) and pin *all* dependencies, and only
>>>> update them when the new versions have been tested and are known to work.
>>>> pip-tools <https://pypi.org/project/pip-tools/> can help with that,
>>>> for example (requirements.in specifies “loose” dependencies;
>>>> pip-compile creates a pinned requirements.txt; pip-sync installs it; pip-compile
>>>> -U upgrades requirements.txt later; you check both requirements.in and
>>>> requirements.txt into version control.) But I don’t know if that
>>>> applies in your integration/config case.
>>>>
>>>> Am Do., 4. Mai 2023 um 18:08 Uhr schrieb Antoine Musso <hashar@free.fr
>>>> >:
>>>>
>>>>> Hello,
>>>>>
>>>>> This is for python projects.
>>>>>
>>>>> Today, May 4th, urllib3 <https://pypi.org/project/urllib3/#history>
>>>>> has released a new major version 2.0.2 which breaks the extremely popular
>>>>> requests <https://pypi.org/project/requests/> library.
>>>>>
>>>>> The fix is to pin urllib3<2 to prevent the new major version from
>>>>> being installed (example
>>>>> <https://gerrit.wikimedia.org/r/c/integration/config/+/915736/1/tox.ini>
>>>>> ).
>>>>>
>>>>> https://phabricator.wikimedia.org/T335977
>>>>>
>>>>> Upstream issue: https://github.com/psf/requests/issues/6432
>>>>>
>>>>>
>>>>> Antoine "hashar" Musso
>>>>> Wikimedia Release Engineering
>>>>> _______________________________________________
>>>>> Wikitech-l mailing list -- wikitech-l@lists.wikimedia.org
>>>>> To unsubscribe send an email to wikitech-l-leave@lists.wikimedia.org
>>>>>
>>>>> https://lists.wikimedia.org/postorius/lists/wikitech-l.lists.wikimedia.org/
>>>>
>>>>
>>>>
>>>> --
>>>> Lucas Werkmeister (he/er)
>>>> Software Engineer
>>>>
>>>> Wikimedia Deutschland e. V. | Tempelhofer Ufer 23-24 | 10963 Berlin
>>>> Phone: +49 (0)30-577 11 62-0
>>>> https://wikimedia.de
>>>>
>>>> Imagine a world in which every single human being can freely share in
>>>> the sum of all knowledge. Help us to achieve our vision!
>>>> https://spenden.wikimedia.de
>>>>
>>>> Wikimedia Deutschland - Gesellschaft zur Förderung Freien Wissens e. V.
>>>> Eingetragen im Vereinsregister des Amtsgerichts Berlin-Charlottenburg unter
>>>> der Nummer 23855 B. Als gemeinnützig anerkannt durch das Finanzamt für
>>>> Körperschaften I Berlin, Steuernummer 27/029/42207.
>>>> _______________________________________________
>>>> Wikitech-l mailing list -- wikitech-l@lists.wikimedia.org
>>>> To unsubscribe send an email to wikitech-l-leave@lists.wikimedia.org
>>>>
>>>> https://lists.wikimedia.org/postorius/lists/wikitech-l.lists.wikimedia.org/
>>>
>>>
>>>
>>> --
>>> Amir (he/him)
>>>
>>> _______________________________________________
>>> Wikitech-l mailing list -- wikitech-l@lists.wikimedia.org
>>> To unsubscribe send an email to wikitech-l-leave@lists.wikimedia.org
>>>
>>> https://lists.wikimedia.org/postorius/lists/wikitech-l.lists.wikimedia.org/
>>
>> _______________________________________________
>> Wikitech-l mailing list -- wikitech-l@lists.wikimedia.org
>> To unsubscribe send an email to wikitech-l-leave@lists.wikimedia.org
>>
>> https://lists.wikimedia.org/postorius/lists/wikitech-l.lists.wikimedia.org/
>
> _______________________________________________
> Wikitech-l mailing list -- wikitech-l@lists.wikimedia.org
> To unsubscribe send an email to wikitech-l-leave@lists.wikimedia.org
> https://lists.wikimedia.org/postorius/lists/wikitech-l.lists.wikimedia.org/
>
>
> _______________________________________________
> Wikitech-l mailing list -- wikitech-l@lists.wikimedia.org
> To unsubscribe send an email to wikitech-l-leave@lists.wikimedia.org
> https://lists.wikimedia.org/postorius/lists/wikitech-l.lists.wikimedia.org/

> On 5. May 2023, at 17:30, Slavina Stefanova <sstefanova@wikimedia.org> wrote:
>
>> Tangent: is it worthwhile to establish a consensus for best practices with package pinning and package management for Python projects in the Wikimedia ecosystem? When I last worked on a python project (https://wikitech.wikimedia.org/wiki/Add_Link) I found it confusing that we have so many different tools and approaches for doing these things, and seems like we'd benefit from having a standard, supported way. (Or maybe that already exists and I haven't found it?)
>
> I'm working on an "Essential Tools for Managing Python Development Environments <https://docs.google.com/document/d/1pl6QCDWGebGjRrHixgQNES8qRf-8PCzwLSsFj7jI1HY/edit#heading=h.i5ou2ywgmb09>" tutorial that will be published to the wikis when ready. Maybe that could be expanded upon? In my experience though, it can be hard to get people to agree on following a standard, especially when there are so many different options and many folks already have their favorite tools and workflows. But it would be nice to have a set of recommendations to reduce the cognitive load.

Having something like that on wiki would be great. Thanks for working on it!

Kosta

> Tangent: is it worthwhile to establish a consensus for best practices
with package pinning and package management for Python projects in the
Wikimedia ecosystem?
Yes! That would be awesome. I have spent a lot of time floundering in this
area trying to make decisions; it'd be nice if we had a good guideline
established.

> I'm working on an "Essential Tools for Managing Python Development
Environments
<https://docs.google.com/document/d/1pl6QCDWGebGjRrHixgQNES8qRf-8PCzwLSsFj7jI1HY/edit#heading=h.i5ou2ywgmb09>"
tutorial
Awesome! Did you consider conda envs? FWIW, we rely on conda envs
<https://gitlab.wikimedia.org/repos/data-engineering/workflow_utils#building-project-conda-distribution-environments>
in the Data Engineering world to work around the lack of ability to
securely run docker images in production. We didn't try pyenv, mainly
because conda gets us more than just python :)

> Poetry is a modern lockfile-based packaging and dependency management
tool worth looking into
Poetry was nice, but I found that it wasn't as comprehensive (yet?) as ye
old setuptools based stuff. I can't quite recall what, but I think it was
around support for datafiles and scripts? But, this was 1.5 years ago so
maybe things are better now.

Thank you!

On Fri, May 5, 2023 at 11:32?AM Slavina Stefanova <sstefanova@wikimedia.org>
wrote:

> Tangent: is it worthwhile to establish a consensus for best practices with
>> package pinning and package management for Python projects in the Wikimedia
>> ecosystem? When I last worked on a python project (
>> https://wikitech.wikimedia.org/wiki/Add_Link) I found it confusing that
>> we have so many different tools and approaches for doing these things, and
>> seems like we'd benefit from having a standard, supported way. (Or maybe
>> that already exists and I haven't found it?)
>
>
> I'm working on an "Essential Tools for Managing Python Development
> Environments
> <https://docs.google.com/document/d/1pl6QCDWGebGjRrHixgQNES8qRf-8PCzwLSsFj7jI1HY/edit#heading=h.i5ou2ywgmb09>"
> tutorial that will be published to the wikis when ready. Maybe that could
> be expanded upon? In my experience though, it can be hard to get people to
> agree on following a standard, especially when there are so many different
> options and many folks already have their favorite tools and workflows. But
> it would be nice to have a set of recommendations to reduce the cognitive
> load.
>
> --
> Slavina Stefanova (she/her)
> Software Engineer - Technical Engagement
>
> Wikimedia Foundation
>
>
> On Fri, May 5, 2023 at 5:18?PM Kosta Harlan <kharlan@wikimedia.org> wrote:
>
>> Tangent: is it worthwhile to establish a consensus for best practices
>> with package pinning and package management for Python projects in the
>> Wikimedia ecosystem? When I last worked on a python project (
>> https://wikitech.wikimedia.org/wiki/Add_Link) I found it confusing that
>> we have so many different tools and approaches for doing these things, and
>> seems like we'd benefit from having a standard, supported way. (Or maybe
>> that already exists and I haven't found it?)
>>
>> Kosta
>>
>> On 5. May 2023, at 13:51, Slavina Stefanova <sstefanova@wikimedia.org>
>> wrote:
>>
>> Poetry is a modern lockfile-based packaging and dependency management
>> tool worth looking into. It also supports exporting dependencies into a
>> requirements.txt file, should you need that (nice if you want to
>> containerize an app without bloating the image with Poetry, for instance).
>>
>> https://python-poetry.org/ <https://python-poetry.org/>
>>
>> --
>> Slavina Stefanova (she/her)
>> Software Engineer - Technical Engagement
>>
>> Wikimedia Foundation
>>
>>
>> On Fri, May 5, 2023 at 1:38?PM Sebastian Berlin <
>> sebastian.berlin@wikimedia.se> wrote:
>>
>>> A word of warning: using `pip freeze` to populate requirements.txt can
>>> result in a hard to read (very long) file and other issues:
>>> https://medium.com/@tomagee/pip-freeze-requirements-txt-considered-harmful-f0bce66cf895
>>> .
>>>
>>> *Sebastian Berlin*
>>> Utvecklare/*Developer*
>>> Wikimedia Sverige (WMSE)
>>>
>>> E-post/*E-Mail*: sebastian.berlin@wikimedia.se
>>> Telefon/*Phone*: (+46) 0707 - 92 03 84
>>>
>>>
>>> On Fri, 5 May 2023 at 13:17, Amir Sarabadani <ladsgroup@gmail.com>
>>> wrote:
>>>
>>>> You can also create an empty virtual env, install all requirements and
>>>> then do
>>>> pip freeze > requirements.txt
>>>>
>>>> That should take care of pinning
>>>>
>>>> Am Fr., 5. Mai 2023 um 13:11 Uhr schrieb Lucas Werkmeister <
>>>> lucas.werkmeister@wikimedia.de>:
>>>>
>>>>> For the general case of Python projects, I’d argue that a better
>>>>> solution is to adopt the lockfile pattern (package-lock.json,
>>>>> composer.lock, Cargo.lock, etc.) and pin *all* dependencies, and only
>>>>> update them when the new versions have been tested and are known to work.
>>>>> pip-tools <https://pypi.org/project/pip-tools/> can help with that,
>>>>> for example (requirements.in specifies “loose” dependencies;
>>>>> pip-compile creates a pinned requirements.txt; pip-sync installs it; pip-compile
>>>>> -U upgrades requirements.txt later; you check both requirements.in
>>>>> and requirements.txt into version control.) But I don’t know if that
>>>>> applies in your integration/config case.
>>>>>
>>>>> Am Do., 4. Mai 2023 um 18:08 Uhr schrieb Antoine Musso <hashar@free.fr
>>>>> >:
>>>>>
>>>>>> Hello,
>>>>>>
>>>>>> This is for python projects.
>>>>>>
>>>>>> Today, May 4th, urllib3 <https://pypi.org/project/urllib3/#history>
>>>>>> has released a new major version 2.0.2 which breaks the extremely popular
>>>>>> requests <https://pypi.org/project/requests/> library.
>>>>>>
>>>>>> The fix is to pin urllib3<2 to prevent the new major version from
>>>>>> being installed (example
>>>>>> <https://gerrit.wikimedia.org/r/c/integration/config/+/915736/1/tox.ini>
>>>>>> ).
>>>>>>
>>>>>> https://phabricator.wikimedia.org/T335977
>>>>>>
>>>>>> Upstream issue: https://github.com/psf/requests/issues/6432
>>>>>>
>>>>>>
>>>>>> Antoine "hashar" Musso
>>>>>> Wikimedia Release Engineering
>>>>>> _______________________________________________
>>>>>> Wikitech-l mailing list -- wikitech-l@lists.wikimedia.org
>>>>>> To unsubscribe send an email to wikitech-l-leave@lists.wikimedia.org
>>>>>>
>>>>>> https://lists.wikimedia.org/postorius/lists/wikitech-l.lists.wikimedia.org/
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Lucas Werkmeister (he/er)
>>>>> Software Engineer
>>>>>
>>>>> Wikimedia Deutschland e. V. | Tempelhofer Ufer 23-24 | 10963 Berlin
>>>>> Phone: +49 (0)30-577 11 62-0
>>>>> https://wikimedia.de
>>>>>
>>>>> Imagine a world in which every single human being can freely share in
>>>>> the sum of all knowledge. Help us to achieve our vision!
>>>>> https://spenden.wikimedia.de
>>>>>
>>>>> Wikimedia Deutschland - Gesellschaft zur Förderung Freien Wissens e.
>>>>> V. Eingetragen im Vereinsregister des Amtsgerichts Berlin-Charlottenburg
>>>>> unter der Nummer 23855 B. Als gemeinnützig anerkannt durch das Finanzamt
>>>>> für Körperschaften I Berlin, Steuernummer 27/029/42207.
>>>>> _______________________________________________
>>>>> Wikitech-l mailing list -- wikitech-l@lists.wikimedia.org
>>>>> To unsubscribe send an email to wikitech-l-leave@lists.wikimedia.org
>>>>>
>>>>> https://lists.wikimedia.org/postorius/lists/wikitech-l.lists.wikimedia.org/
>>>>
>>>>
>>>>
>>>> --
>>>> Amir (he/him)
>>>>
>>>> _______________________________________________
>>>> Wikitech-l mailing list -- wikitech-l@lists.wikimedia.org
>>>> To unsubscribe send an email to wikitech-l-leave@lists.wikimedia.org
>>>>
>>>> https://lists.wikimedia.org/postorius/lists/wikitech-l.lists.wikimedia.org/
>>>
>>> _______________________________________________
>>> Wikitech-l mailing list -- wikitech-l@lists.wikimedia.org
>>> To unsubscribe send an email to wikitech-l-leave@lists.wikimedia.org
>>>
>>> https://lists.wikimedia.org/postorius/lists/wikitech-l.lists.wikimedia.org/
>>
>> _______________________________________________
>> Wikitech-l mailing list -- wikitech-l@lists.wikimedia.org
>> To unsubscribe send an email to wikitech-l-leave@lists.wikimedia.org
>>
>> https://lists.wikimedia.org/postorius/lists/wikitech-l.lists.wikimedia.org/
>>
>>
>> _______________________________________________
>> Wikitech-l mailing list -- wikitech-l@lists.wikimedia.org
>> To unsubscribe send an email to wikitech-l-leave@lists.wikimedia.org
>>
>> https://lists.wikimedia.org/postorius/lists/wikitech-l.lists.wikimedia.org/
>
> _______________________________________________
> Wikitech-l mailing list -- wikitech-l@lists.wikimedia.org
> To unsubscribe send an email to wikitech-l-leave@lists.wikimedia.org
> https://lists.wikimedia.org/postorius/lists/wikitech-l.lists.wikimedia.org/

> For Java, we run an instance of Archiva: https://archiva.wikimedia.org/
> It's not a perfect approach but I think we can and should move in that
direction with all our other ecosystems

Gitlab package registries may help us here!



On Mon, May 8, 2023 at 8:59?AM Andrew Otto <otto@wikimedia.org> wrote:

> > Tangent: is it worthwhile to establish a consensus for best practices
> with package pinning and package management for Python projects in the
> Wikimedia ecosystem?
> Yes! That would be awesome. I have spent a lot of time floundering in this
> area trying to make decisions; it'd be nice if we had a good guideline
> established.
>
> > I'm working on an "Essential Tools for Managing Python Development
> Environments
> <https://docs.google.com/document/d/1pl6QCDWGebGjRrHixgQNES8qRf-8PCzwLSsFj7jI1HY/edit#heading=h.i5ou2ywgmb09>"
> tutorial
> Awesome! Did you consider conda envs? FWIW, we rely on conda envs
> <https://gitlab.wikimedia.org/repos/data-engineering/workflow_utils#building-project-conda-distribution-environments>
> in the Data Engineering world to work around the lack of ability to
> securely run docker images in production. We didn't try pyenv, mainly
> because conda gets us more than just python :)
>
> > Poetry is a modern lockfile-based packaging and dependency management
> tool worth looking into
> Poetry was nice, but I found that it wasn't as comprehensive (yet?) as ye
> old setuptools based stuff. I can't quite recall what, but I think it was
> around support for datafiles and scripts? But, this was 1.5 years ago so
> maybe things are better now.
>
> Thank you!
>
> On Fri, May 5, 2023 at 11:32?AM Slavina Stefanova <
> sstefanova@wikimedia.org> wrote:
>
>> Tangent: is it worthwhile to establish a consensus for best practices
>>> with package pinning and package management for Python projects in the
>>> Wikimedia ecosystem? When I last worked on a python project (
>>> https://wikitech.wikimedia.org/wiki/Add_Link) I found it confusing that
>>> we have so many different tools and approaches for doing these things, and
>>> seems like we'd benefit from having a standard, supported way. (Or maybe
>>> that already exists and I haven't found it?)
>>
>>
>> I'm working on an "Essential Tools for Managing Python Development
>> Environments
>> <https://docs.google.com/document/d/1pl6QCDWGebGjRrHixgQNES8qRf-8PCzwLSsFj7jI1HY/edit#heading=h.i5ou2ywgmb09>"
>> tutorial that will be published to the wikis when ready. Maybe that could
>> be expanded upon? In my experience though, it can be hard to get people to
>> agree on following a standard, especially when there are so many different
>> options and many folks already have their favorite tools and workflows. But
>> it would be nice to have a set of recommendations to reduce the cognitive
>> load.
>>
>> --
>> Slavina Stefanova (she/her)
>> Software Engineer - Technical Engagement
>>
>> Wikimedia Foundation
>>
>>
>> On Fri, May 5, 2023 at 5:18?PM Kosta Harlan <kharlan@wikimedia.org>
>> wrote:
>>
>>> Tangent: is it worthwhile to establish a consensus for best practices
>>> with package pinning and package management for Python projects in the
>>> Wikimedia ecosystem? When I last worked on a python project (
>>> https://wikitech.wikimedia.org/wiki/Add_Link) I found it confusing that
>>> we have so many different tools and approaches for doing these things, and
>>> seems like we'd benefit from having a standard, supported way. (Or maybe
>>> that already exists and I haven't found it?)
>>>
>>> Kosta
>>>
>>> On 5. May 2023, at 13:51, Slavina Stefanova <sstefanova@wikimedia.org>
>>> wrote:
>>>
>>> Poetry is a modern lockfile-based packaging and dependency management
>>> tool worth looking into. It also supports exporting dependencies into a
>>> requirements.txt file, should you need that (nice if you want to
>>> containerize an app without bloating the image with Poetry, for instance).
>>>
>>> https://python-poetry.org/ <https://python-poetry.org/>
>>>
>>> --
>>> Slavina Stefanova (she/her)
>>> Software Engineer - Technical Engagement
>>>
>>> Wikimedia Foundation
>>>
>>>
>>> On Fri, May 5, 2023 at 1:38?PM Sebastian Berlin <
>>> sebastian.berlin@wikimedia.se> wrote:
>>>
>>>> A word of warning: using `pip freeze` to populate requirements.txt can
>>>> result in a hard to read (very long) file and other issues:
>>>> https://medium.com/@tomagee/pip-freeze-requirements-txt-considered-harmful-f0bce66cf895
>>>> .
>>>>
>>>> *Sebastian Berlin*
>>>> Utvecklare/*Developer*
>>>> Wikimedia Sverige (WMSE)
>>>>
>>>> E-post/*E-Mail*: sebastian.berlin@wikimedia.se
>>>> Telefon/*Phone*: (+46) 0707 - 92 03 84
>>>>
>>>>
>>>> On Fri, 5 May 2023 at 13:17, Amir Sarabadani <ladsgroup@gmail.com>
>>>> wrote:
>>>>
>>>>> You can also create an empty virtual env, install all requirements and
>>>>> then do
>>>>> pip freeze > requirements.txt
>>>>>
>>>>> That should take care of pinning
>>>>>
>>>>> Am Fr., 5. Mai 2023 um 13:11 Uhr schrieb Lucas Werkmeister <
>>>>> lucas.werkmeister@wikimedia.de>:
>>>>>
>>>>>> For the general case of Python projects, I’d argue that a better
>>>>>> solution is to adopt the lockfile pattern (package-lock.json,
>>>>>> composer.lock, Cargo.lock, etc.) and pin *all* dependencies, and
>>>>>> only update them when the new versions have been tested and are known to
>>>>>> work. pip-tools <https://pypi.org/project/pip-tools/> can help with
>>>>>> that, for example (requirements.in specifies “loose” dependencies;
>>>>>> pip-compile creates a pinned requirements.txt; pip-sync installs it; pip-compile
>>>>>> -U upgrades requirements.txt later; you check both requirements.in
>>>>>> and requirements.txt into version control.) But I don’t know if that
>>>>>> applies in your integration/config case.
>>>>>>
>>>>>> Am Do., 4. Mai 2023 um 18:08 Uhr schrieb Antoine Musso <
>>>>>> hashar@free.fr>:
>>>>>>
>>>>>>> Hello,
>>>>>>>
>>>>>>> This is for python projects.
>>>>>>>
>>>>>>> Today, May 4th, urllib3 <https://pypi.org/project/urllib3/#history>
>>>>>>> has released a new major version 2.0.2 which breaks the extremely popular
>>>>>>> requests <https://pypi.org/project/requests/> library.
>>>>>>>
>>>>>>> The fix is to pin urllib3<2 to prevent the new major version from
>>>>>>> being installed (example
>>>>>>> <https://gerrit.wikimedia.org/r/c/integration/config/+/915736/1/tox.ini>
>>>>>>> ).
>>>>>>>
>>>>>>> https://phabricator.wikimedia.org/T335977
>>>>>>>
>>>>>>> Upstream issue: https://github.com/psf/requests/issues/6432
>>>>>>>
>>>>>>>
>>>>>>> Antoine "hashar" Musso
>>>>>>> Wikimedia Release Engineering
>>>>>>> _______________________________________________
>>>>>>> Wikitech-l mailing list -- wikitech-l@lists.wikimedia.org
>>>>>>> To unsubscribe send an email to wikitech-l-leave@lists.wikimedia.org
>>>>>>>
>>>>>>> https://lists.wikimedia.org/postorius/lists/wikitech-l.lists.wikimedia.org/
>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Lucas Werkmeister (he/er)
>>>>>> Software Engineer
>>>>>>
>>>>>> Wikimedia Deutschland e. V. | Tempelhofer Ufer 23-24 | 10963 Berlin
>>>>>> Phone: +49 (0)30-577 11 62-0
>>>>>> https://wikimedia.de
>>>>>>
>>>>>> Imagine a world in which every single human being can freely share in
>>>>>> the sum of all knowledge. Help us to achieve our vision!
>>>>>> https://spenden.wikimedia.de
>>>>>>
>>>>>> Wikimedia Deutschland - Gesellschaft zur Förderung Freien Wissens e.
>>>>>> V. Eingetragen im Vereinsregister des Amtsgerichts Berlin-Charlottenburg
>>>>>> unter der Nummer 23855 B. Als gemeinnützig anerkannt durch das Finanzamt
>>>>>> für Körperschaften I Berlin, Steuernummer 27/029/42207.
>>>>>> _______________________________________________
>>>>>> Wikitech-l mailing list -- wikitech-l@lists.wikimedia.org
>>>>>> To unsubscribe send an email to wikitech-l-leave@lists.wikimedia.org
>>>>>>
>>>>>> https://lists.wikimedia.org/postorius/lists/wikitech-l.lists.wikimedia.org/
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Amir (he/him)
>>>>>
>>>>> _______________________________________________
>>>>> Wikitech-l mailing list -- wikitech-l@lists.wikimedia.org
>>>>> To unsubscribe send an email to wikitech-l-leave@lists.wikimedia.org
>>>>>
>>>>> https://lists.wikimedia.org/postorius/lists/wikitech-l.lists.wikimedia.org/
>>>>
>>>> _______________________________________________
>>>> Wikitech-l mailing list -- wikitech-l@lists.wikimedia.org
>>>> To unsubscribe send an email to wikitech-l-leave@lists.wikimedia.org
>>>>
>>>> https://lists.wikimedia.org/postorius/lists/wikitech-l.lists.wikimedia.org/
>>>
>>> _______________________________________________
>>> Wikitech-l mailing list -- wikitech-l@lists.wikimedia.org
>>> To unsubscribe send an email to wikitech-l-leave@lists.wikimedia.org
>>>
>>> https://lists.wikimedia.org/postorius/lists/wikitech-l.lists.wikimedia.org/
>>>
>>>
>>> _______________________________________________
>>> Wikitech-l mailing list -- wikitech-l@lists.wikimedia.org
>>> To unsubscribe send an email to wikitech-l-leave@lists.wikimedia.org
>>>
>>> https://lists.wikimedia.org/postorius/lists/wikitech-l.lists.wikimedia.org/
>>
>> _______________________________________________
>> Wikitech-l mailing list -- wikitech-l@lists.wikimedia.org
>> To unsubscribe send an email to wikitech-l-leave@lists.wikimedia.org
>>
>> https://lists.wikimedia.org/postorius/lists/wikitech-l.lists.wikimedia.org/
>
>

>
> > I'm working on an "Essential Tools for Managing Python Development
> Environments
> <https://docs.google.com/document/d/1pl6QCDWGebGjRrHixgQNES8qRf-8PCzwLSsFj7jI1HY/edit#heading=h.i5ou2ywgmb09>"
> tutorial
> Awesome! Did you consider conda envs? FWIW, we rely on conda envs
> <https://gitlab.wikimedia.org/repos/data-engineering/workflow_utils#building-project-conda-distribution-environments> in
> the Data Engineering world to work around the lack of ability to
> securely run docker images in production.
> We didn't try pyenv, mainly because conda gets us more than just python :)


I tried conda a few years ago, but for my own use cases, I found it mostly
got in my way. I know it's big with data and science folks though, so I
will add it to an "other tools to be aware of" section. But now I'm curious
about how conda enables running docker safely in production. :)

For anyone interested in the (bewildering complexity of the current) Python
packaging ecosystem, I can recommend this blog post by PyPA member and PSF
fellow Pradyum Gedam:
https://pradyunsg.me/blog/2023/01/21/thoughts-on-python-packaging/
<https://pradyunsg.me/blog/2023/01/21/thoughts-on-python-packaging/>


--
Slavina Stefanova (she/her)
Software Engineer - Technical Engagement

Wikimedia Foundation


On Mon, May 8, 2023 at 3:03?PM Andrew Otto <otto@wikimedia.org> wrote:

> > For Java, we run an instance of Archiva: https://archiva.wikimedia.org/
> > It's not a perfect approach but I think we can and should move in that
> direction with all our other ecosystems
>
> Gitlab package registries may help us here!
>
>
>
> On Mon, May 8, 2023 at 8:59?AM Andrew Otto <otto@wikimedia.org> wrote:
>
>> > Tangent: is it worthwhile to establish a consensus for best practices
>> with package pinning and package management for Python projects in the
>> Wikimedia ecosystem?
>> Yes! That would be awesome. I have spent a lot of time floundering in
>> this area trying to make decisions; it'd be nice if we had a good guideline
>> established.
>>
>> > I'm working on an "Essential Tools for Managing Python Development
>> Environments
>> <https://docs.google.com/document/d/1pl6QCDWGebGjRrHixgQNES8qRf-8PCzwLSsFj7jI1HY/edit#heading=h.i5ou2ywgmb09>"
>> tutorial
>> Awesome! Did you consider conda envs? FWIW, we rely on conda envs
>> <https://gitlab.wikimedia.org/repos/data-engineering/workflow_utils#building-project-conda-distribution-environments>
>> in the Data Engineering world to work around the lack of ability to
>> securely run docker images in production. We didn't try pyenv, mainly
>> because conda gets us more than just python :)
>>
>> > Poetry is a modern lockfile-based packaging and dependency management
>> tool worth looking into
>> Poetry was nice, but I found that it wasn't as comprehensive (yet?) as ye
>> old setuptools based stuff. I can't quite recall what, but I think it was
>> around support for datafiles and scripts? But, this was 1.5 years ago so
>> maybe things are better now.
>>
>> Thank you!
>>
>> On Fri, May 5, 2023 at 11:32?AM Slavina Stefanova <
>> sstefanova@wikimedia.org> wrote:
>>
>>> Tangent: is it worthwhile to establish a consensus for best practices
>>>> with package pinning and package management for Python projects in the
>>>> Wikimedia ecosystem? When I last worked on a python project (
>>>> https://wikitech.wikimedia.org/wiki/Add_Link) I found it confusing
>>>> that we have so many different tools and approaches for doing these things,
>>>> and seems like we'd benefit from having a standard, supported way. (Or
>>>> maybe that already exists and I haven't found it?)
>>>
>>>
>>> I'm working on an "Essential Tools for Managing Python Development
>>> Environments
>>> <https://docs.google.com/document/d/1pl6QCDWGebGjRrHixgQNES8qRf-8PCzwLSsFj7jI1HY/edit#heading=h.i5ou2ywgmb09>"
>>> tutorial that will be published to the wikis when ready. Maybe that could
>>> be expanded upon? In my experience though, it can be hard to get people to
>>> agree on following a standard, especially when there are so many different
>>> options and many folks already have their favorite tools and workflows. But
>>> it would be nice to have a set of recommendations to reduce the cognitive
>>> load.
>>>
>>> --
>>> Slavina Stefanova (she/her)
>>> Software Engineer - Technical Engagement
>>>
>>> Wikimedia Foundation
>>>
>>>
>>> On Fri, May 5, 2023 at 5:18?PM Kosta Harlan <kharlan@wikimedia.org>
>>> wrote:
>>>
>>>> Tangent: is it worthwhile to establish a consensus for best practices
>>>> with package pinning and package management for Python projects in the
>>>> Wikimedia ecosystem? When I last worked on a python project (
>>>> https://wikitech.wikimedia.org/wiki/Add_Link) I found it confusing
>>>> that we have so many different tools and approaches for doing these things,
>>>> and seems like we'd benefit from having a standard, supported way. (Or
>>>> maybe that already exists and I haven't found it?)
>>>>
>>>> Kosta
>>>>
>>>> On 5. May 2023, at 13:51, Slavina Stefanova <sstefanova@wikimedia.org>
>>>> wrote:
>>>>
>>>> Poetry is a modern lockfile-based packaging and dependency management
>>>> tool worth looking into. It also supports exporting dependencies into a
>>>> requirements.txt file, should you need that (nice if you want to
>>>> containerize an app without bloating the image with Poetry, for instance).
>>>>
>>>> https://python-poetry.org/ <https://python-poetry.org/>
>>>>
>>>> --
>>>> Slavina Stefanova (she/her)
>>>> Software Engineer - Technical Engagement
>>>>
>>>> Wikimedia Foundation
>>>>
>>>>
>>>> On Fri, May 5, 2023 at 1:38?PM Sebastian Berlin <
>>>> sebastian.berlin@wikimedia.se> wrote:
>>>>
>>>>> A word of warning: using `pip freeze` to populate requirements.txt can
>>>>> result in a hard to read (very long) file and other issues:
>>>>> https://medium.com/@tomagee/pip-freeze-requirements-txt-considered-harmful-f0bce66cf895
>>>>> .
>>>>>
>>>>> *Sebastian Berlin*
>>>>> Utvecklare/*Developer*
>>>>> Wikimedia Sverige (WMSE)
>>>>>
>>>>> E-post/*E-Mail*: sebastian.berlin@wikimedia.se
>>>>> Telefon/*Phone*: (+46) 0707 - 92 03 84
>>>>>
>>>>>
>>>>> On Fri, 5 May 2023 at 13:17, Amir Sarabadani <ladsgroup@gmail.com>
>>>>> wrote:
>>>>>
>>>>>> You can also create an empty virtual env, install all requirements
>>>>>> and then do
>>>>>> pip freeze > requirements.txt
>>>>>>
>>>>>> That should take care of pinning
>>>>>>
>>>>>> Am Fr., 5. Mai 2023 um 13:11 Uhr schrieb Lucas Werkmeister <
>>>>>> lucas.werkmeister@wikimedia.de>:
>>>>>>
>>>>>>> For the general case of Python projects, I’d argue that a better
>>>>>>> solution is to adopt the lockfile pattern (package-lock.json,
>>>>>>> composer.lock, Cargo.lock, etc.) and pin *all* dependencies, and
>>>>>>> only update them when the new versions have been tested and are known to
>>>>>>> work. pip-tools <https://pypi.org/project/pip-tools/> can help with
>>>>>>> that, for example (requirements.in specifies “loose” dependencies;
>>>>>>> pip-compile creates a pinned requirements.txt; pip-sync installs
>>>>>>> it; pip-compile -U upgrades requirements.txt later; you check both
>>>>>>> requirements.in and requirements.txt into version control.) But I
>>>>>>> don’t know if that applies in your integration/config case.
>>>>>>>
>>>>>>> Am Do., 4. Mai 2023 um 18:08 Uhr schrieb Antoine Musso <
>>>>>>> hashar@free.fr>:
>>>>>>>
>>>>>>>> Hello,
>>>>>>>>
>>>>>>>> This is for python projects.
>>>>>>>>
>>>>>>>> Today, May 4th, urllib3 <https://pypi.org/project/urllib3/#history>
>>>>>>>> has released a new major version 2.0.2 which breaks the extremely popular
>>>>>>>> requests <https://pypi.org/project/requests/> library.
>>>>>>>>
>>>>>>>> The fix is to pin urllib3<2 to prevent the new major version from
>>>>>>>> being installed (example
>>>>>>>> <https://gerrit.wikimedia.org/r/c/integration/config/+/915736/1/tox.ini>
>>>>>>>> ).
>>>>>>>>
>>>>>>>> https://phabricator.wikimedia.org/T335977
>>>>>>>>
>>>>>>>> Upstream issue: https://github.com/psf/requests/issues/6432
>>>>>>>>
>>>>>>>>
>>>>>>>> Antoine "hashar" Musso
>>>>>>>> Wikimedia Release Engineering
>>>>>>>> _______________________________________________
>>>>>>>> Wikitech-l mailing list -- wikitech-l@lists.wikimedia.org
>>>>>>>> To unsubscribe send an email to
>>>>>>>> wikitech-l-leave@lists.wikimedia.org
>>>>>>>>
>>>>>>>> https://lists.wikimedia.org/postorius/lists/wikitech-l.lists.wikimedia.org/
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> Lucas Werkmeister (he/er)
>>>>>>> Software Engineer
>>>>>>>
>>>>>>> Wikimedia Deutschland e. V. | Tempelhofer Ufer 23-24 | 10963 Berlin
>>>>>>> Phone: +49 (0)30-577 11 62-0
>>>>>>> https://wikimedia.de
>>>>>>>
>>>>>>> Imagine a world in which every single human being can freely share
>>>>>>> in the sum of all knowledge. Help us to achieve our vision!
>>>>>>> https://spenden.wikimedia.de
>>>>>>>
>>>>>>> Wikimedia Deutschland - Gesellschaft zur Förderung Freien Wissens e.
>>>>>>> V. Eingetragen im Vereinsregister des Amtsgerichts Berlin-Charlottenburg
>>>>>>> unter der Nummer 23855 B. Als gemeinnützig anerkannt durch das Finanzamt
>>>>>>> für Körperschaften I Berlin, Steuernummer 27/029/42207.
>>>>>>> _______________________________________________
>>>>>>> Wikitech-l mailing list -- wikitech-l@lists.wikimedia.org
>>>>>>> To unsubscribe send an email to wikitech-l-leave@lists.wikimedia.org
>>>>>>>
>>>>>>> https://lists.wikimedia.org/postorius/lists/wikitech-l.lists.wikimedia.org/
>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Amir (he/him)
>>>>>>
>>>>>> _______________________________________________
>>>>>> Wikitech-l mailing list -- wikitech-l@lists.wikimedia.org
>>>>>> To unsubscribe send an email to wikitech-l-leave@lists.wikimedia.org
>>>>>>
>>>>>> https://lists.wikimedia.org/postorius/lists/wikitech-l.lists.wikimedia.org/
>>>>>
>>>>> _______________________________________________
>>>>> Wikitech-l mailing list -- wikitech-l@lists.wikimedia.org
>>>>> To unsubscribe send an email to wikitech-l-leave@lists.wikimedia.org
>>>>>
>>>>> https://lists.wikimedia.org/postorius/lists/wikitech-l.lists.wikimedia.org/
>>>>
>>>> _______________________________________________
>>>> Wikitech-l mailing list -- wikitech-l@lists.wikimedia.org
>>>> To unsubscribe send an email to wikitech-l-leave@lists.wikimedia.org
>>>>
>>>> https://lists.wikimedia.org/postorius/lists/wikitech-l.lists.wikimedia.org/
>>>>
>>>>
>>>> _______________________________________________
>>>> Wikitech-l mailing list -- wikitech-l@lists.wikimedia.org
>>>> To unsubscribe send an email to wikitech-l-leave@lists.wikimedia.org
>>>>
>>>> https://lists.wikimedia.org/postorius/lists/wikitech-l.lists.wikimedia.org/
>>>
>>> _______________________________________________
>>> Wikitech-l mailing list -- wikitech-l@lists.wikimedia.org
>>> To unsubscribe send an email to wikitech-l-leave@lists.wikimedia.org
>>>
>>> https://lists.wikimedia.org/postorius/lists/wikitech-l.lists.wikimedia.org/
>>
>> _______________________________________________
> Wikitech-l mailing list -- wikitech-l@lists.wikimedia.org
> To unsubscribe send an email to wikitech-l-leave@lists.wikimedia.org
> https://lists.wikimedia.org/postorius/lists/wikitech-l.lists.wikimedia.org/

> But now I'm curious about how conda enables running docker safely in
production. :)

It doesn't enable running docker, we just use packed conda envs instead of
docker images. This only really works because we build and run the conda
envs on the same OS. See conda-pack <https://conda.github.io/conda-pack/>.

We'd prefer to use docker if we could.

On Thu, May 11, 2023 at 3:19?AM Slavina Stefanova <sstefanova@wikimedia.org>
wrote:

> > I'm working on an "Essential Tools for Managing Python Development
>> Environments
>> <https://docs.google.com/document/d/1pl6QCDWGebGjRrHixgQNES8qRf-8PCzwLSsFj7jI1HY/edit#heading=h.i5ou2ywgmb09>"
>> tutorial
>> Awesome! Did you consider conda envs? FWIW, we rely on conda envs
>> <https://gitlab.wikimedia.org/repos/data-engineering/workflow_utils#building-project-conda-distribution-environments> in
>> the Data Engineering world to work around the lack of ability to
>> securely run docker images in production.
>> We didn't try pyenv, mainly because conda gets us more than just python :)
>
>
> I tried conda a few years ago, but for my own use cases, I found it mostly
> got in my way. I know it's big with data and science folks though, so I
> will add it to an "other tools to be aware of" section. But now I'm curious
> about how conda enables running docker safely in production. :)
>
> For anyone interested in the (bewildering complexity of the current)
> Python packaging ecosystem, I can recommend this blog post by
> PyPA member and PSF fellow Pradyum Gedam:
> https://pradyunsg.me/blog/2023/01/21/thoughts-on-python-packaging/
> <https://pradyunsg.me/blog/2023/01/21/thoughts-on-python-packaging/>
>
>
> --
> Slavina Stefanova (she/her)
> Software Engineer - Technical Engagement
>
> Wikimedia Foundation
>
>
> On Mon, May 8, 2023 at 3:03?PM Andrew Otto <otto@wikimedia.org> wrote:
>
>> > For Java, we run an instance of Archiva: https://archiva.wikimedia.org/
>> > It's not a perfect approach but I think we can and should move in that
>> direction with all our other ecosystems
>>
>> Gitlab package registries may help us here!
>>
>>
>>
>> On Mon, May 8, 2023 at 8:59?AM Andrew Otto <otto@wikimedia.org> wrote:
>>
>>> > Tangent: is it worthwhile to establish a consensus for best practices
>>> with package pinning and package management for Python projects in the
>>> Wikimedia ecosystem?
>>> Yes! That would be awesome. I have spent a lot of time floundering in
>>> this area trying to make decisions; it'd be nice if we had a good guideline
>>> established.
>>>
>>> > I'm working on an "Essential Tools for Managing Python Development
>>> Environments
>>> <https://docs.google.com/document/d/1pl6QCDWGebGjRrHixgQNES8qRf-8PCzwLSsFj7jI1HY/edit#heading=h.i5ou2ywgmb09>"
>>> tutorial
>>> Awesome! Did you consider conda envs? FWIW, we rely on conda envs
>>> <https://gitlab.wikimedia.org/repos/data-engineering/workflow_utils#building-project-conda-distribution-environments>
>>> in the Data Engineering world to work around the lack of ability to
>>> securely run docker images in production. We didn't try pyenv, mainly
>>> because conda gets us more than just python :)
>>>
>>> > Poetry is a modern lockfile-based packaging and dependency management
>>> tool worth looking into
>>> Poetry was nice, but I found that it wasn't as comprehensive (yet?) as
>>> ye old setuptools based stuff. I can't quite recall what, but I think it
>>> was around support for datafiles and scripts? But, this was 1.5 years ago
>>> so maybe things are better now.
>>>
>>> Thank you!
>>>
>>> On Fri, May 5, 2023 at 11:32?AM Slavina Stefanova <
>>> sstefanova@wikimedia.org> wrote:
>>>
>>>> Tangent: is it worthwhile to establish a consensus for best practices
>>>>> with package pinning and package management for Python projects in the
>>>>> Wikimedia ecosystem? When I last worked on a python project (
>>>>> https://wikitech.wikimedia.org/wiki/Add_Link) I found it confusing
>>>>> that we have so many different tools and approaches for doing these things,
>>>>> and seems like we'd benefit from having a standard, supported way. (Or
>>>>> maybe that already exists and I haven't found it?)
>>>>
>>>>
>>>> I'm working on an "Essential Tools for Managing Python Development
>>>> Environments
>>>> <https://docs.google.com/document/d/1pl6QCDWGebGjRrHixgQNES8qRf-8PCzwLSsFj7jI1HY/edit#heading=h.i5ou2ywgmb09>"
>>>> tutorial that will be published to the wikis when ready. Maybe that could
>>>> be expanded upon? In my experience though, it can be hard to get people to
>>>> agree on following a standard, especially when there are so many different
>>>> options and many folks already have their favorite tools and workflows. But
>>>> it would be nice to have a set of recommendations to reduce the cognitive
>>>> load.
>>>>
>>>> --
>>>> Slavina Stefanova (she/her)
>>>> Software Engineer - Technical Engagement
>>>>
>>>> Wikimedia Foundation
>>>>
>>>>
>>>> On Fri, May 5, 2023 at 5:18?PM Kosta Harlan <kharlan@wikimedia.org>
>>>> wrote:
>>>>
>>>>> Tangent: is it worthwhile to establish a consensus for best practices
>>>>> with package pinning and package management for Python projects in the
>>>>> Wikimedia ecosystem? When I last worked on a python project (
>>>>> https://wikitech.wikimedia.org/wiki/Add_Link) I found it confusing
>>>>> that we have so many different tools and approaches for doing these things,
>>>>> and seems like we'd benefit from having a standard, supported way. (Or
>>>>> maybe that already exists and I haven't found it?)
>>>>>
>>>>> Kosta
>>>>>
>>>>> On 5. May 2023, at 13:51, Slavina Stefanova <sstefanova@wikimedia.org>
>>>>> wrote:
>>>>>
>>>>> Poetry is a modern lockfile-based packaging and dependency management
>>>>> tool worth looking into. It also supports exporting dependencies into a
>>>>> requirements.txt file, should you need that (nice if you want to
>>>>> containerize an app without bloating the image with Poetry, for instance).
>>>>>
>>>>> https://python-poetry.org/ <https://python-poetry.org/>
>>>>>
>>>>> --
>>>>> Slavina Stefanova (she/her)
>>>>> Software Engineer - Technical Engagement
>>>>>
>>>>> Wikimedia Foundation
>>>>>
>>>>>
>>>>> On Fri, May 5, 2023 at 1:38?PM Sebastian Berlin <
>>>>> sebastian.berlin@wikimedia.se> wrote:
>>>>>
>>>>>> A word of warning: using `pip freeze` to populate requirements.txt
>>>>>> can result in a hard to read (very long) file and other issues:
>>>>>> https://medium.com/@tomagee/pip-freeze-requirements-txt-considered-harmful-f0bce66cf895
>>>>>> .
>>>>>>
>>>>>> *Sebastian Berlin*
>>>>>> Utvecklare/*Developer*
>>>>>> Wikimedia Sverige (WMSE)
>>>>>>
>>>>>> E-post/*E-Mail*: sebastian.berlin@wikimedia.se
>>>>>> Telefon/*Phone*: (+46) 0707 - 92 03 84
>>>>>>
>>>>>>
>>>>>> On Fri, 5 May 2023 at 13:17, Amir Sarabadani <ladsgroup@gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>>> You can also create an empty virtual env, install all requirements
>>>>>>> and then do
>>>>>>> pip freeze > requirements.txt
>>>>>>>
>>>>>>> That should take care of pinning
>>>>>>>
>>>>>>> Am Fr., 5. Mai 2023 um 13:11 Uhr schrieb Lucas Werkmeister <
>>>>>>> lucas.werkmeister@wikimedia.de>:
>>>>>>>
>>>>>>>> For the general case of Python projects, I’d argue that a better
>>>>>>>> solution is to adopt the lockfile pattern (package-lock.json,
>>>>>>>> composer.lock, Cargo.lock, etc.) and pin *all* dependencies, and
>>>>>>>> only update them when the new versions have been tested and are known to
>>>>>>>> work. pip-tools <https://pypi.org/project/pip-tools/> can help
>>>>>>>> with that, for example (requirements.in specifies “loose”
>>>>>>>> dependencies; pip-compile creates a pinned requirements.txt;
>>>>>>>> pip-sync installs it; pip-compile -U upgrades requirements.txt
>>>>>>>> later; you check both requirements.in and requirements.txt into
>>>>>>>> version control.) But I don’t know if that applies in your
>>>>>>>> integration/config case.
>>>>>>>>
>>>>>>>> Am Do., 4. Mai 2023 um 18:08 Uhr schrieb Antoine Musso <
>>>>>>>> hashar@free.fr>:
>>>>>>>>
>>>>>>>>> Hello,
>>>>>>>>>
>>>>>>>>> This is for python projects.
>>>>>>>>>
>>>>>>>>> Today, May 4th, urllib3
>>>>>>>>> <https://pypi.org/project/urllib3/#history> has released a new
>>>>>>>>> major version 2.0.2 which breaks the extremely popular requests
>>>>>>>>> <https://pypi.org/project/requests/> library.
>>>>>>>>>
>>>>>>>>> The fix is to pin urllib3<2 to prevent the new major version from
>>>>>>>>> being installed (example
>>>>>>>>> <https://gerrit.wikimedia.org/r/c/integration/config/+/915736/1/tox.ini>
>>>>>>>>> ).
>>>>>>>>>
>>>>>>>>> https://phabricator.wikimedia.org/T335977
>>>>>>>>>
>>>>>>>>> Upstream issue: https://github.com/psf/requests/issues/6432
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Antoine "hashar" Musso
>>>>>>>>> Wikimedia Release Engineering
>>>>>>>>> _______________________________________________
>>>>>>>>> Wikitech-l mailing list -- wikitech-l@lists.wikimedia.org
>>>>>>>>> To unsubscribe send an email to
>>>>>>>>> wikitech-l-leave@lists.wikimedia.org
>>>>>>>>>
>>>>>>>>> https://lists.wikimedia.org/postorius/lists/wikitech-l.lists.wikimedia.org/
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> Lucas Werkmeister (he/er)
>>>>>>>> Software Engineer
>>>>>>>>
>>>>>>>> Wikimedia Deutschland e. V. | Tempelhofer Ufer 23-24 | 10963 Berlin
>>>>>>>> Phone: +49 (0)30-577 11 62-0
>>>>>>>> https://wikimedia.de
>>>>>>>>
>>>>>>>> Imagine a world in which every single human being can freely share
>>>>>>>> in the sum of all knowledge. Help us to achieve our vision!
>>>>>>>> https://spenden.wikimedia.de
>>>>>>>>
>>>>>>>> Wikimedia Deutschland - Gesellschaft zur Förderung Freien Wissens
>>>>>>>> e. V. Eingetragen im Vereinsregister des Amtsgerichts Berlin-Charlottenburg
>>>>>>>> unter der Nummer 23855 B. Als gemeinnützig anerkannt durch das Finanzamt
>>>>>>>> für Körperschaften I Berlin, Steuernummer 27/029/42207.
>>>>>>>> _______________________________________________
>>>>>>>> Wikitech-l mailing list -- wikitech-l@lists.wikimedia.org
>>>>>>>> To unsubscribe send an email to
>>>>>>>> wikitech-l-leave@lists.wikimedia.org
>>>>>>>>
>>>>>>>> https://lists.wikimedia.org/postorius/lists/wikitech-l.lists.wikimedia.org/
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> Amir (he/him)
>>>>>>>
>>>>>>> _______________________________________________
>>>>>>> Wikitech-l mailing list -- wikitech-l@lists.wikimedia.org
>>>>>>> To unsubscribe send an email to wikitech-l-leave@lists.wikimedia.org
>>>>>>>
>>>>>>> https://lists.wikimedia.org/postorius/lists/wikitech-l.lists.wikimedia.org/
>>>>>>
>>>>>> _______________________________________________
>>>>>> Wikitech-l mailing list -- wikitech-l@lists.wikimedia.org
>>>>>> To unsubscribe send an email to wikitech-l-leave@lists.wikimedia.org
>>>>>>
>>>>>> https://lists.wikimedia.org/postorius/lists/wikitech-l.lists.wikimedia.org/
>>>>>
>>>>> _______________________________________________
>>>>> Wikitech-l mailing list -- wikitech-l@lists.wikimedia.org
>>>>> To unsubscribe send an email to wikitech-l-leave@lists.wikimedia.org
>>>>>
>>>>> https://lists.wikimedia.org/postorius/lists/wikitech-l.lists.wikimedia.org/
>>>>>
>>>>>
>>>>> _______________________________________________
>>>>> Wikitech-l mailing list -- wikitech-l@lists.wikimedia.org
>>>>> To unsubscribe send an email to wikitech-l-leave@lists.wikimedia.org
>>>>>
>>>>> https://lists.wikimedia.org/postorius/lists/wikitech-l.lists.wikimedia.org/
>>>>
>>>> _______________________________________________
>>>> Wikitech-l mailing list -- wikitech-l@lists.wikimedia.org
>>>> To unsubscribe send an email to wikitech-l-leave@lists.wikimedia.org
>>>>
>>>> https://lists.wikimedia.org/postorius/lists/wikitech-l.lists.wikimedia.org/
>>>
>>> _______________________________________________
>> Wikitech-l mailing list -- wikitech-l@lists.wikimedia.org
>> To unsubscribe send an email to wikitech-l-leave@lists.wikimedia.org
>>
>> https://lists.wikimedia.org/postorius/lists/wikitech-l.lists.wikimedia.org/
>
> _______________________________________________
> Wikitech-l mailing list -- wikitech-l@lists.wikimedia.org
> To unsubscribe send an email to wikitech-l-leave@lists.wikimedia.org
> https://lists.wikimedia.org/postorius/lists/wikitech-l.lists.wikimedia.org/