Mailing List Archive

[PATCH 2/2] python-utils-r1.eclass: Enable parallel bytecompile compilation
Python 3.5 added support for compileall to run parallel workers for
performing bytecode compilation. Make use of it to the extent
possible without refactoring the code too much to get different
paths into the same call for best possible parallelization.
---
 eclass/python-utils-r1.eclass | 16 +++++++++++-----
 1 file changed, 11 insertions(+), 5 deletions(-)

diff --git a/eclass/python-utils-r1.eclass b/eclass/python-utils-r1.eclass
index e2f05606993..f7a38f8c4e0 100644
--- a/eclass/python-utils-r1.eclass
+++ b/eclass/python-utils-r1.eclass
@@ -34,7 +34,7 @@ fi
 
 if [[ ! ${_PYTHON_UTILS_R1} ]]; then
 
-inherit toolchain-funcs
+inherit multiprocessing toolchain-funcs
 
 # @ECLASS-VARIABLE: _PYTHON_ALL_IMPLS
 # @INTERNAL
@@ -615,6 +615,12 @@ python_optimize() {
                debug-print "${FUNCNAME}: using sys.path: ${*/%/;}"
        fi
 
+       local jobs=$(makeopts_jobs "${MAKEOPTS}" INF)
+       if [[ ${jobs} == INF ]]; then
+               local nproc=$(get_nproc)
+               jobs=$(( nproc + 1 ))
+       fi
+
        local d
        for d; do
                # make sure to get a nice path without //
@@ -628,12 +634,12 @@ python_optimize() {
                                ;;
                        python3.[5678]|pypy3)
                                # both levels of optimization are separate since 3.5
-                               "${PYTHON}" -m compileall -q -f -d "${instpath}" "${d}"
-                               "${PYTHON}" -O -m compileall -q -f -d "${instpath}" "${d}"
-                               "${PYTHON}" -OO -m compileall -q -f -d "${instpath}" "${d}"
+                               "${PYTHON}" -m compileall -j "${jobs}" -q -f -d "${instpath}" "${d}"
+                               "${PYTHON}" -O -m compileall -j "${jobs}" -q -f -d "${instpath}" "${d}"
+                               "${PYTHON}" -OO -m compileall -j "${jobs}" -q -f -d "${instpath}" "${d}"
                                ;;
                        python*)
-                               "${PYTHON}" -m compileall -o 0 -o 1 -o 2 --hardlink-dupes -q -f -d "${instpath}" "${d}"
+                               "${PYTHON}" -m compileall -j "${jobs}" -o 0 -o 1 -o 2 --hardlink-dupes -q -f -d "${instpath}" "${d}"
                                ;;
                        *)
                                "${PYTHON}" -m compileall -q -f -d "${instpath}" "${d}"
Re: [PATCH 2/2] python-utils-r1.eclass: Enable parallel bytecompile compilation [ In reply to ]
On mercoled? 23 giugno 2021 11:36:33 CEST Mart Raudsepp wrote:
> + jobs=$(( nproc + 1 ))

I don't know who historically started the nproc+1 story (it looks to be in the
handbook too), but it has been demonstrated that nproc+1 is slower than nproc.

Agostino
Re: [PATCH 2/2] python-utils-r1.eclass: Enable parallel bytecompile compilation [ In reply to ]
Ühel kenal päeval, K, 23.06.2021 kell 11:43, kirjutas Agostino Sarubbo:
> On mercoledì 23 giugno 2021 11:36:33 CEST Mart Raudsepp wrote:
> > +               jobs=$(( nproc + 1 ))
>
> I don't know who historically started the nproc+1 story (it looks to
> be in the
> handbook too), but it has been demonstrated that nproc+1 is slower
> than nproc.

This is a copy of what distutils-r1.eclass is doing.


Mart
Re: [PATCH 2/2] python-utils-r1.eclass: Enable parallel bytecompile compilation [ In reply to ]
On Wed, 2021-06-23 at 11:43 +0200, Agostino Sarubbo wrote:
> On mercoledì 23 giugno 2021 11:36:33 CEST Mart Raudsepp wrote:
> > + jobs=$(( nproc + 1 ))
>
> I don't know who historically started the nproc+1 story (it looks to be in the
> handbook too), but it has been demonstrated that nproc+1 is slower than nproc.
>

Demonstrated on what? This largely depends on the task in question.
In this case, I guess nproc makes more sense and makes things simpler
as it's got little I/O overhead.

--
Best regards,
Micha? Górny