Mailing List Archive

[PATCH 1/2] python-utils-r1.eclass: Enable hardlinking of identical pyc files for py3.9+
Python 3.9 includes a new feature for compileall to automatically hardlink
different optimization level cache files if they are identical. Make use of it
for python_optimize for some space savings.
This however does not cover distutils use cases and python itself.
---
 eclass/python-utils-r1.eclass | 5 ++++-
 1 file changed, 4 insertions(+), 1 deletion(-)

Note that I haven't actually tested this on py3.9, as I'm skipping that version.
It appears to do what it's meant to with py3.10

diff --git a/eclass/python-utils-r1.eclass b/eclass/python-utils-r1.eclass
index 3dbf221eac5..e2f05606993 100644
--- a/eclass/python-utils-r1.eclass
+++ b/eclass/python-utils-r1.eclass
@@ -626,12 +626,15 @@ python_optimize() {
                                "${PYTHON}" -m compileall -q -f -d "${instpath}" "${d}"
                                "${PYTHON}" -OO -m compileall -q -f -d "${instpath}" "${d}"
                                ;;
-                       python*|pypy3)
+                       python3.[5678]|pypy3)
                                # both levels of optimization are separate since 3.5
                                "${PYTHON}" -m compileall -q -f -d "${instpath}" "${d}"
                                "${PYTHON}" -O -m compileall -q -f -d "${instpath}" "${d}"
                                "${PYTHON}" -OO -m compileall -q -f -d "${instpath}" "${d}"
                                ;;
+                       python*)
+                               "${PYTHON}" -m compileall -o 0 -o 1 -o 2 --hardlink-dupes -q -f -d "${instpath}" "${d}"
+                               ;;
                        *)
                                "${PYTHON}" -m compileall -q -f -d "${instpath}" "${d}"
                                ;;
Re: [PATCH 1/2] python-utils-r1.eclass: Enable hardlinking of identical pyc files for py3.9+ [ In reply to ]
On Wed, 2021-06-23 at 12:36 +0300, Mart Raudsepp wrote:
> Python 3.9 includes a new feature for compileall to automatically hardlink
> different optimization level cache files if they are identical. Make use of it
> for python_optimize for some space savings.
> This however does not cover distutils use cases and python itself.
> ---
>  eclass/python-utils-r1.eclass | 5 ++++-
>  1 file changed, 4 insertions(+), 1 deletion(-)
>
> Note that I haven't actually tested this on py3.9, as I'm skipping that version.
> It appears to do what it's meant to with py3.10
>
> diff --git a/eclass/python-utils-r1.eclass b/eclass/python-utils-r1.eclass
> index 3dbf221eac5..e2f05606993 100644
> --- a/eclass/python-utils-r1.eclass
> +++ b/eclass/python-utils-r1.eclass
> @@ -626,12 +626,15 @@ python_optimize() {
>                                 "${PYTHON}" -m compileall -q -f -d "${instpath}" "${d}"
>                                 "${PYTHON}" -OO -m compileall -q -f -d "${instpath}" "${d}"
>                                 ;;
> -                       python*|pypy3)
> +                       python3.[5678]|pypy3)

We don't really support < 3.8 anymore but I can update that while
merging, I guess.

>                                 # both levels of optimization are separate since 3.5
>                                 "${PYTHON}" -m compileall -q -f -d "${instpath}" "${d}"
>                                 "${PYTHON}" -O -m compileall -q -f -d "${instpath}" "${d}"
>                                 "${PYTHON}" -OO -m compileall -q -f -d "${instpath}" "${d}"
>                                 ;;
> +                       python*)
> +                               "${PYTHON}" -m compileall -o 0 -o 1 -o 2 --hardlink-dupes -q -f -d "${instpath}" "${d}"
> +                               ;;
>                         *)
>                                 "${PYTHON}" -m compileall -q -f -d "${instpath}" "${d}"
>                                 ;;
>
>
>

--
Best regards,
Micha? Górny
Re: [PATCH 1/2] python-utils-r1.eclass: Enable hardlinking of identical pyc files for py3.9+ [ In reply to ]
Ühel kenal päeval, K, 23.06.2021 kell 11:56, kirjutas Micha? Górny:
> > -                       python*|pypy3)
> > +                       python3.[5678]|pypy3)
>
> We don't really support < 3.8 anymore but I can update that while
> merging, I guess.

Yeah, this is originating from the previous block having python3.[34],
so the changeset is making sure the 3.9 and 3.10 cases end up using the
added path, not about 3.5/3.6, for which the code path is kept the same
with this.

I don't foresee myself working on making distutils cases work with this
anytime soon. Maybe we should file a bug on that. It's a bit surprising
this wasn't done there too upstream, I guess those who made this change
only cared about it for python itself and they are running compileall
manually over all the caches.


Mart
Re: [PATCH 1/2] python-utils-r1.eclass: Enable hardlinking of identical pyc files for py3.9+ [ In reply to ]
On Wed, 2021-06-23 at 13:19 +0300, Mart Raudsepp wrote:
> Ühel kenal päeval, K, 23.06.2021 kell 11:56, kirjutas Micha? Górny:
> > > -                       python*|pypy3)
> > > +                       python3.[5678]|pypy3)
> >
> > We don't really support < 3.8 anymore but I can update that while
> > merging, I guess.
>
> Yeah, this is originating from the previous block having python3.[34],
> so the changeset is making sure the 3.9 and 3.10 cases end up using the
> added path, not about 3.5/3.6, for which the code path is kept the same
> with this.
>
> I don't foresee myself working on making distutils cases work with this
> anytime soon. Maybe we should file a bug on that. It's a bit surprising
> this wasn't done there too upstream, I guess those who made this change
> only cared about it for python itself and they are running compileall
> manually over all the caches.
>

The problem roughly is that distutils is unmaintained for years,
and upstream plans to remove it entirely in Python 3.12. If anyone's
going to work on patches, setuptools is probably where it would need
to land. But right now it's all a huge mess.

--
Best regards,
Micha? Górny