Mailing List Archive

Change by STINNER Victor <vstinner@python.org>:


----------
pull_requests: +26822
pull_request: https://github.com/python/cpython/pull/28410

_______________________________________
Python tracker <report@bugs.python.org>
<https://bugs.python.org/issue45020>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com

STINNER Victor <vstinner@python.org> added the comment:


New changeset 41551ee7e24fb6c58846836d3655dbb212281206 by Victor Stinner in branch 'main':
bpo-45020: Fix build out of source tree (GH-28410)
https://github.com/python/cpython/commit/41551ee7e24fb6c58846836d3655dbb212281206


----------

_______________________________________
Python tracker <report@bugs.python.org>
<https://bugs.python.org/issue45020>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com

Eric Snow <ericsnowcurrently@gmail.com> added the comment:

Thanks for fixing that, Victor!

----------

_______________________________________
Python tracker <report@bugs.python.org>
<https://bugs.python.org/issue45020>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com

Guido van Rossum <guido@python.org> added the comment:

(Ah, now I recall how the $(srcdir) mechanism works again. Generated files do *not* have a $(srcdir) prefix, since they are generated in the destination directory. Hence the fix for the buildbot failure a few days ago. I take back my skepticism about that.)

----------

_______________________________________
Python tracker <report@bugs.python.org>
<https://bugs.python.org/issue45020>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com

Guido van Rossum <guido@python.org> added the comment:

BTW, why does the script still run Programs/_freeze_module over all the modules to be frozen? Isn't that up to the Makefile or its Windows equivalent?

----------

_______________________________________
Python tracker <report@bugs.python.org>
<https://bugs.python.org/issue45020>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com

Eric Snow <ericsnowcurrently@gmail.com> added the comment:


New changeset 090591636c4f03ce06a039079bd7716a5b23631e by Eric Snow in branch 'main':
bpo-45020: Freeze os, site, and codecs. (gh-28398)
https://github.com/python/cpython/commit/090591636c4f03ce06a039079bd7716a5b23631e


----------

_______________________________________
Python tracker <report@bugs.python.org>
<https://bugs.python.org/issue45020>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com

Eric Snow <ericsnowcurrently@gmail.com> added the comment:

On Fri, Sep 17, 2021 at 4:22 PM Guido van Rossum <report@bugs.python.org> wrote:
> BTW, why does the script still run Programs/_freeze_module over all the modules to be frozen? Isn't that up to the Makefile or its Windows equivalent?

Yeah, it used to matter but we probably don't need to do that any longer.

----------

_______________________________________
Python tracker <report@bugs.python.org>
<https://bugs.python.org/issue45020>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com

Raymond Hettinger <raymond.hettinger@gmail.com> added the comment:

It would be nice to freeze argparse.py and its dependencies. For command-line tools, startup time is important.

----------
nosy: +rhettinger

_______________________________________
Python tracker <report@bugs.python.org>
<https://bugs.python.org/issue45020>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com

Guido van Rossum <guido@python.org> added the comment:

> It would be nice to freeze argparse.py and its dependencies. For command-line tools, startup time is important.

I quickly checked, and argparse has at least these dependencies:

argparse
re
enum
types
operator
functools
collections
keyword
reprlib
sre_compile
sre_parse
sre_constants
copyreg
gettext

Raymond, do you think we should also freeze the dependencies of runpy (so "python -m <module>" also starts faster)?

That would be

runpy
importlib
warnings
importlib.machinery
importlib.util
importlib._abc
contextlib
collections
keyword
operator
reprlib
functools
types

(I didn't dedupe this from the previous list.)

With all these modules frozen we'd probably run more risk of two things:

- frozen modules don't have a __file__ (and are missing a few other, less commonly used, attributes)
- editing a frozen module requires rebuilding, or running with -X frozen_modules=off

----------

_______________________________________
Python tracker <report@bugs.python.org>
<https://bugs.python.org/issue45020>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com

Raymond Hettinger <raymond.hettinger@gmail.com> added the comment:

> Raymond, do you think we should also freeze the dependencies
> of runpy (so "python -m <module>" also starts faster)?

Yes, please.

The '-m' load option can be considered core startup functionality. It is often the recommended way to launch command line tools outside of a virtual environment.

python -m pip install some_package

----------

_______________________________________
Python tracker <report@bugs.python.org>
<https://bugs.python.org/issue45020>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com

STINNER Victor <vstinner@python.org> added the comment:

runpy startup time matters a lot for "python3 -m module" startup time!

In Python 3.10 (bpo-41006 and bpo-41718), I reduced the number of modules imported by runpy:

"The runpy module now imports fewer modules. The python3 -m module-name command startup time is 1.4x faster in average. On Linux, python3 -I -m module-name imports 69 modules on Python 3.9, whereas it only imports 51 modules (-18) on Python 3.10. (Contributed by Victor Stinner in bpo-41006 and bpo-41718.)"

https://docs.python.org/dev/whatsnew/3.10.html#optimizations

--

For argparse, maybe it could use a few lazy imports to reduce the number of indirect impots on "import argparse"?

----------

_______________________________________
Python tracker <report@bugs.python.org>
<https://bugs.python.org/issue45020>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com

Guido van Rossum <guido@python.org> added the comment:

I'm still torn about the need for a __file__ attribute. Assuming it's more likely that people write code that *reads* stdlib source for some reason (maybe printing more context for error messages) than *writing* it (which would be truly strange), the amount of 3rd party code that might break due to the lack of __file__ can be expected to be larger than the amount of code (or number of users) that would be confused to having a __file__ pointing to a file that isn't actually read by the interpreter.

Okay, so I'm no longer torn. We should set __file__ for frozen modules based on the pre-computed location of the stdlib.

----------

_______________________________________
Python tracker <report@bugs.python.org>
<https://bugs.python.org/issue45020>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com

Brett Cannon <brett@python.org> added the comment:

What about if there isn't a pre-computed location for __file__? I could imagine a self-contained CPython build where there is no concept of a file location on disk for anything using this.

----------

_______________________________________
Python tracker <report@bugs.python.org>
<https://bugs.python.org/issue45020>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com

Marc-Andre Lemburg <mal@egenix.com> added the comment:

On 22.09.2021 20:47, Brett Cannon wrote:
> What about if there isn't a pre-computed location for __file__? I could imagine a self-contained CPython build where there is no concept of a file location on disk for anything using this.

This does work and is enough to make most code out there happy.

I use e.g. "<pyrun>/os.py" in PyRun. There is no os.py file to load,
but tracebacks and inspection tools work just fine with this.

----------

_______________________________________
Python tracker <report@bugs.python.org>
<https://bugs.python.org/issue45020>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com

Gregory Szorc <gregory.szorc@gmail.com> added the comment:

I'll throw out that __file__ is strictly optional, per https://docs.python.org/3/reference/datamodel.html. IMO it is more important to define importlib.abc.InspectLoader than __file__ (so tracebacks have source context).

Given that __file__ is optional, I'll throw out the idea of shipping modules without __file__/__cached__ and see what issues/pushback occurs. I anticipate this will likely result in having to restore advertising __file__ in frozen modules in 3.11 due to popular demand. But we /could/ use this as an opportunity to start attempting to ween people off __file__ and onto more appropriate APIs, like importlib.resources. (importlib.resources was introduced in 3.7 and I'm not sure that's old enough to force people onto without relying on a shim like https://pypi.org/project/importlib-resources/.)

Alternatively, defining a path hook referencing the python executable that knows how to service frozen modules could also work. zipimporter does this (__file__ is <zip_path>/<virtual_module_path>) and it is surprisingly not as brittle as one would think. It might be a good middle ground.

----------

_______________________________________
Python tracker <report@bugs.python.org>
<https://bugs.python.org/issue45020>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com

Change by Eric Snow <ericsnowcurrently@gmail.com>:


----------
pull_requests: +26923
pull_request: https://github.com/python/cpython/pull/28538

_______________________________________
Python tracker <report@bugs.python.org>
<https://bugs.python.org/issue45020>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com

Eric Snow <ericsnowcurrently@gmail.com> added the comment:


New changeset 7c801e0fa603b155eab3fd19698aa90854ac5a7b by Eric Snow in branch 'main':
bpo-45020: Fix some corner cases for frozen module generation. (gh-28538)
https://github.com/python/cpython/commit/7c801e0fa603b155eab3fd19698aa90854ac5a7b


----------

_______________________________________
Python tracker <report@bugs.python.org>
<https://bugs.python.org/issue45020>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com

Change by Eric Snow <ericsnowcurrently@gmail.com>:


----------
pull_requests: +26937
pull_request: https://github.com/python/cpython/pull/28554

_______________________________________
Python tracker <report@bugs.python.org>
<https://bugs.python.org/issue45020>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com

Marc-Andre Lemburg <mal@egenix.com> added the comment:

Eric, I noticed that you are freezing os.py. Please be aware that
this module is often being used as indicator for where the stdlib
was installed (the stdlib itself does this in site.py to read the
LICENSE and the test suite also uses os.__file__ in a couple of
places).

It may be worth changing the stdlib to pick a different module
as landmark for this purpose.

Also: Unless you have added the .__file__ attribute to frozen
modules, much of this landmark checking code will fail... which is
the reason I added the attribute to frozen modules in PyRun.

----------

_______________________________________
Python tracker <report@bugs.python.org>
<https://bugs.python.org/issue45020>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com

Guido van Rossum <guido@python.org> added the comment:

The plan is to add __file__ back, based on where the stdlib lives (we won’t do any stat() calls for frozen files).

----------

_______________________________________
Python tracker <report@bugs.python.org>
<https://bugs.python.org/issue45020>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com

STINNER Victor <vstinner@python.org> added the comment:

Marc-Andre: I suppose that you're talking about LANDMARK in Modules/getpath.c and PC/getpathp.c.

----------

_______________________________________
Python tracker <report@bugs.python.org>
<https://bugs.python.org/issue45020>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com

Marc-Andre Lemburg <mal@egenix.com> added the comment:

On 25.09.2021 18:20, STINNER Victor wrote:
>
> STINNER Victor <vstinner@python.org> added the comment:
>
> Marc-Andre: I suppose that you're talking about LANDMARK in Modules/getpath.c and PC/getpathp.c.

Now that you mention it: yes, that as well :-) But os.py is used in the
Python stdlib code as well, just search for "os.__file__" to see a few
such uses.

If you search for ".__file__" you'll find that there are quite a few
cases in the test suite expecting that attribute on other stdlib modules
as well. The attribute may officially be optional, but in reality a
lot of code expects to find it.

----------

_______________________________________
Python tracker <report@bugs.python.org>
<https://bugs.python.org/issue45020>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com

Change by Eric Snow <ericsnowcurrently@gmail.com>:


----------
pull_requests: +26965
pull_request: https://github.com/python/cpython/pull/28583

_______________________________________
Python tracker <report@bugs.python.org>
<https://bugs.python.org/issue45020>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com

Change by Barry A. Warsaw <barry@python.org>:


----------
nosy: +barry

_______________________________________
Python tracker <report@bugs.python.org>
<https://bugs.python.org/issue45020>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com

Change by Eric Snow <ericsnowcurrently@gmail.com>:


----------
pull_requests: +26972
pull_request: https://github.com/python/cpython/pull/28590

_______________________________________
Python tracker <report@bugs.python.org>
<https://bugs.python.org/issue45020>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com