Mailing List Archive

1 2 3 4 5 6  View All
[issue45020] Freeze all modules imported during startup. [ In reply to ]
Change by Inada Naoki <songofacandy@gmail.com>:


----------
nosy: +methane

_______________________________________
Python tracker <report@bugs.python.org>
<https://bugs.python.org/issue45020>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com
[issue45020] Freeze all modules imported during startup. [ In reply to ]
Eric Snow <ericsnowcurrently@gmail.com> added the comment:

On Fri, Aug 27, 2021 at 6:29 PM Guido van Rossum <report@bugs.python.org> wrote:
> The plot thickens. By searching my extensive GMail archives for Jeethu Rao I found
> an email from Sept. 14 to python-dev by Larry Hastings titled "Store startup modules
> as C structures for 20%+ startup speed improvement?"

Thanks for finding that, Guido!

On Fri, Aug 27, 2021 at 6:37 PM Guido van Rossum <report@bugs.python.org> wrote:
> Either way it's a suboptimal experience for people contributing to those modules. But
> we stand to gain a ~20% startup time improvement.

Agreed, and I think a solution shouldn't be too hard to reach.

On Fri, Aug 27, 2021 at 7:48 PM Larry Hastings <report@bugs.python.org> wrote:
> In experimenting with the prototype, I observed that simply calling stat() to ensure
> the frozen .py file hadn't changed on disk lost us about half the performance win
> from this approach.

Yeah, this is an approach others had suggested and I'd considered. We
have other solutions available that don't have that penalty.

On Fri, Aug 27, 2021 at 8:08 PM Larry Hastings <report@bugs.python.org> wrote:
> There should be a boolean flag that enables/disables cached copies of .py files from
> Lib/. You should be able to turn it off with either an environment variable or a
> command-line option, and when it's off it skips all the internal cached stuff and
> uses the normal .py / .pyc machinery.
>
> With that in place, it'd be great to pre-cache all the .py files automatically read in
> at startup.

Yeah, something along these lines should be good enough.

> [snip]
> But then I'm not sure this is a very good analogy--the workflow for making Clinic
> changes is very different from people hacking on Lib/*.py.

Agreed.

On Fri, Aug 27, 2021 at 10:06 PM Guido van Rossum
<report@bugs.python.org> wrote:
> [snip]
> FWIW in my attempts to time this, it looks like the perf benefits of Eric's approach are
> close to those of deep-freezing. And deep-freezing causes much more bloat of the
> source code and of the resulting binary.

The question of freeze vs deep-freeze (i.e. is deep-freeze better
enough) is one we can discuss separately, and your point here is
probably the fundamental center of that discussion. However, I don't
think it has a lot of bearing on the change proposed in this issue.

> [snip]
> I think the only solution here was hinted at in the python-dev thread from 2018: have
> a command-line flag to turn it on or off (e.g. -X deepfreeze=1/0) and have a policy for
> what the default for that flag should be (e.g. on by default in production builds, off by
> default in developer builds -- anything that doesn't use --enable-optimizations).

Agreed.

> [snip]
> it wasn't so clear that code objects should be immutable -- that realization came later,
> when Greg Stein proposed making them ROM-able. That didn't work out, but the
> notion that code objects should be strictly mutable (to the python user, at least)
> was born

This sounds like an interesting story. Do you have any mailing list
links handy? (Otherwise I can search the archives.)

> In fact, Eric's approach freezes everything in the encodings package, which turns out
> to be a lot of files and a lot of code (lots of simple data tables expressed in code), and
> I found that for basic startup time, it's best not to deep-freeze the encodings module
> except for __init__.py, aliases.py and utf_8.py.

Yeah, this is something to consider. FWIW, in my testing, dropping
encodings.* from the
list of frozen modules reduced the performance gains (from 20 ms to 21 ms).

-eric

----------

_______________________________________
Python tracker <report@bugs.python.org>
<https://bugs.python.org/issue45020>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com
[issue45020] Freeze all modules imported during startup. [ In reply to ]
Eric Snow <ericsnowcurrently@gmail.com> added the comment:

On Fri, Aug 27, 2021 at 11:14 PM Larry Hastings <report@bugs.python.org> wrote:
> [snip] On the other hand: if we made a viable tool that could consume some arbitrary
> set of .py files and produce a C file, and said C file could then be compiled into a
> shared library, end users could enjoy this speedup over the subset of the standard
> library their program used, and perhaps even their own source tree(s).

Yeah, that would be interesting to investigate.

On Sat, Aug 28, 2021 at 5:17 AM Marc-Andre Lemburg
<report@bugs.python.org> wrote:
> Eric's approach, as I understand it, is pretty much what PyRun does.
> [further details]

It's reassuring to hear that the approach is known to be viable. :)

> In fact, you save quite a bit of disk space compared to a full Python installation and
> additionally benefit from the memory mapping the OS does for sharing access to the
> marshal'ed byte code between processes.

That's a good point.

> That said, some things don't work with such an approach, e.g. a few packages
> include additional data files which they expect to find on disk. Since those are
> not available anymore, they fail.
>
> For PyRun I have patched some of those packages to include the data in form of
> Python modules instead, so that it gets frozen as well, e.g. the Python grammar files.

For stdlib modules it wouldn't be a big problem to set __file__ on
frozen modules.
Would that be enough to solve the problem?

On Sat, Aug 28, 2021 at 5:41 PM Gregory Szorc <report@bugs.python.org> wrote:
> When I investigated freezing the standard library for PyOxidizer, I ran into a rash
> of problems. The frozen importer doesn't behave like PathFinder. It doesn't
> (didn't?) set some common module-level attributes

This is mostly fixable for stdlib modules. Which attributes would
need to be added? Are there other missing behaviors?

> Also, when I last looked at the CPython source, the frozen importer performed
> a linear scan of its indexed C array performing strcmp() on each entry until it
> found what it was looking for. So adding hundreds of modules could result in
> sufficient overhead and justify using a more efficient lookup algorithm.
> (PyOxidizer uses Rust's HashMap to index modules by name.)

Yeah, we noticed this too. I wasn't sure it was something to worry
about at first because we're not freezing the entire stdlib. We're
freezing on the order of 10, plus all the (80+) encoding modules. I
figured we could look at an alternative to that linear search
afterward if it made sense.

> * Make sure you run unit tests against the frozen modules. If you don't do this, subtle differences in how the different importers behave will lead to problems.

We'll do what we already do with importlib: run the tests against both
the frozen and the source modules. Thanks for the reminder to do this
though!

On Sat, Aug 28, 2021 at 5:53 PM Gregory Szorc <report@bugs.python.org> wrote:
> Oh, PyOxidizer also ran into more general issues with the frozen importer in that
> it broke various importlib APIs. e.g. because the frozen importer only supports
> bytecode, you can't use .__loader__.get_source() to obtain the source of a module.
> This makes tracebacks more opaque and breaks legitimate API consumers relying
> on these importlib interfaces.

Good point. Supporting more of the FileLoader API on the frozen
loader is something to look into, at least for stdlib modules.

> The fundamental limitations with the frozen importer are why I implemented my
> own meta path importer (implemented in pure Rust), which is more fully featured,
> like the PathFinder importer that most people rely on today. That importer is
> available on PyPI (https://pypi.org/project/oxidized-importer/) and has its own API
> to facilitate PyOxidizer-like functionality
> (https://pyoxidizer.readthedocs.io/en/stable/oxidized_importer.html) if anyone
> wants to experiment with it.

Awesome! I'll take a look.

On Sat, Aug 28, 2021 at 6:14 PM Guido van Rossum <report@bugs.python.org> wrote:
> I agree that we should shore up the frozen importer -- probably in a separate PR though.
> (@Eric: do you think this is worth its own bpo issue?)

Yeah.

-eric

----------

_______________________________________
Python tracker <report@bugs.python.org>
<https://bugs.python.org/issue45020>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com
[issue45020] Freeze all modules imported during startup. [ In reply to ]
Eric Snow <ericsnowcurrently@gmail.com> added the comment:

At this point, here are the open questions I'm seeing:

+ The editing-stdlib-.py-files problem:
* use a command-line flag to opt-out of frozen modules?
* use a build flag to opt out (e.g. a configure flag or a new Py_NO_FROZEN or even Py_DEBUG)?
* ignore frozen modules if it's a dev build?
* (note: importlib._bootstrap and importlib._bootstrap_external must always be frozen, regardless of a flag)
* accommodate users of an installed Python that sometimes edit stdlib modules while debugging code?
* always emit a warning if a frozen module is ignored (in favor of the source module)?

+ Compatibility:
* set __file__ (and __path__) on frozen modules?
* store the hash of the source file for frozen modules (in the frozen .h files)?
* support more of the FileLoader API on the frozen loader?

+ Penalty for too many frozen modules:
* should we only freeze the UTF-8 encoding?
* should we use something other than linear search for looking up frozen modules?

FWIW, I think the ideal mechanism for a dev build will be to opt in to using frozen modules (instead of the source modules). Otherwise it is too easy for the unaware contributor to waste substantial time figuring out why their changes are not getting used.

Consequently, here's my order of preference for ignoring frozen modules:

1. use Py_DEBUG as an opt-out flag
(if we think contributors are editing stdlib modules on a build without Py_DEBUG then that isn't good enough)
2. automatically skip frozen modules if it's a dev build, with an explicit configure flag to opt in to frozen modules

----------

_______________________________________
Python tracker <report@bugs.python.org>
<https://bugs.python.org/issue45020>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com
[issue45020] Freeze all modules imported during startup. [ In reply to ]
Eric Snow <ericsnowcurrently@gmail.com> added the comment:

> * tricks to inject hooks ASAP (e.g. coverage.py swaps the encodings module) may lose their entry point

FWIW, I asked Ned Batchelder about this and he said this approach ("fullcoverage" [1]) was added to support running coverage on the stdlib. It doesn't affect other users of coverage.py. He didn't have any info on where this is used currently, though I'm pretty sure we do run coverage in CI. Furthermore, the devguide talks about running coverage.py on the stdlib and initially indicates that modules imported during startup are not covered. [2] However, it does have a section talking about using "fullcoverage" to cover startup modules.

I expect that any solution we make for contributors editing stdlib source files will resolve the potential issue for coverage.py.


[1] https://github.com/nedbat/coveragepy/tree/master/coverage/fullcoverage
[2] https://devguide.python.org/coverage/?highlight=coverage#measuring-coverage
[3] https://devguide.python.org/coverage/?highlight=coverage#coverage-results-for-modules-imported-early-on

----------

_______________________________________
Python tracker <report@bugs.python.org>
<https://bugs.python.org/issue45020>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com
[issue45020] Freeze all modules imported during startup. [ In reply to ]
Gregory Szorc <gregory.szorc@gmail.com> added the comment:

> For stdlib modules it wouldn't be a big problem to set __file__ on
> frozen modules.
> Would that be enough to solve the problem?

What do you set __file__ to? Do you still materialize the .py[c] files on disk and set to that? (This would make the most sense.) If you support setting __file__ on a frozen module, how does this work at the C level? A stdlib module would want a different __file__ from a 3rd party module using the frozen module API. This would seemingly require enhancements to the frozen modules C API or some kind of hackery only available to stdlib frozen modules.

>> When I investigated freezing the standard library for PyOxidizer, I ran into a rash
>> of problems. The frozen importer doesn't behave like PathFinder. It doesn't
>> (didn't?) set some common module-level attributes
> This is mostly fixable for stdlib modules. Which attributes would
> need to be added? Are there other missing behaviors?

It was a few years ago and I can't recall specifics. I just remember that after encountering a couple unrelated limitations with the frozen importer I decided it was far too primitive to work as a general importer and decided I'd have to roll my own.

> Good point. Supporting more of the FileLoader API on the frozen
> loader is something to look into, at least for stdlib modules.

>> I agree that we should shore up the frozen importer -- probably in a separate PR though.
>> (@Eric: do you think this is worth its own bpo issue?)
> Yeah.

I have some observations about the implications of this. I typed up a long comment but then realized someone would probably complain about me distracting from the technical parts of this issue. Which forum is most appropriate for this less technical commentary? (I want to lay out the case for building out an official importer far more advanced than frozen importer.)

----------

_______________________________________
Python tracker <report@bugs.python.org>
<https://bugs.python.org/issue45020>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com
[issue45020] Freeze all modules imported during startup. [ In reply to ]
Guido van Rossum <guido@python.org> added the comment:

> FWIW, I asked Ned Batchelder about this and he said this approach ("fullcoverage" [1]) was added to support running coverage on the stdlib. [...]

The docs you pointed out in [3] (where it talks about a *horrible hack you should never use" :-) should be amended with something explaining that "you need to comment out the line "<encodings.*>" from frozen_module.py for this to work, else the frozen version of the encodings module will take priority over the imposter from "fullcoverage"."

----------

_______________________________________
Python tracker <report@bugs.python.org>
<https://bugs.python.org/issue45020>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com
[issue45020] Freeze all modules imported during startup. [ In reply to ]
Guido van Rossum <guido@python.org> added the comment:

[Gregory Szorc]


> What do you set __file__ to? [...]

Exactly. I think it should not be set, just like it's not set for builtin modules.

> I have some observations about the implications of this. I typed up a long comment but then realized someone would probably complain about me distracting from the technical parts of this issue. Which forum is most appropriate for this less technical commentary? (I want to lay out the case for building out an official importer far more advanced than frozen importer.)

That seems to be something for python-dev. (There used to be an "import-sig" but the last mention I have from it is about that list shutting down for lack of traffic.) Or if you're still looking for more brainstorming you could try python-ideas first.

----------

_______________________________________
Python tracker <report@bugs.python.org>
<https://bugs.python.org/issue45020>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com
[issue45020] Freeze all modules imported during startup. [ In reply to ]
Guido van Rossum <guido@python.org> added the comment:

> At this point, here are the open questions I'm seeing:

> + The editing-stdlib-.py-files problem: [...]

> + Compatibility: [...]

? + Penalty for too many frozen modules: [...]

> FWIW, I think the ideal mechanism for a dev build will be to opt in to using frozen modules (instead of the source modules). Otherwise it is too easy for the unaware contributor to waste substantial time figuring out why their changes are not getting used.

Agreed. I don't care much about people (even myself) editing installed modules. But I care a lot about people who do a git checkout and build from source editing modules and testing them without doing an install (Personally I never install what I build from source except to test the installation process.)

> Consequently, here's my order of preference for ignoring frozen modules:

> 1. use Py_DEBUG as an opt-out flag
> (if we think contributors are editing stdlib modules on a build without Py_DEBUG then that isn't good enough)
> 2. automatically skip frozen modules if it's a dev build, with an explicit configure flag to opt in to frozen modules.

I propose to only opt in by default in **PGO builds**. After all what we're doing is another extreme optimization.

It should always be possible to opt in using some -X flag (e.g. to debug the freeze import loader) and it should also always be possible to opt *out* (for those cases where you want to edit an installed stdlib module in-place in anger).

I don't know how the -X mechanism works exactly but probably we could make the spelling

python -X freeze=on|off

with a dynamic default.

----------

_______________________________________
Python tracker <report@bugs.python.org>
<https://bugs.python.org/issue45020>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com
[issue45020] Freeze all modules imported during startup. [ In reply to ]
Guido van Rossum <guido@python.org> added the comment:

FWIW, I'd be okay with doing the -X flag in a separate PR.

----------

_______________________________________
Python tracker <report@bugs.python.org>
<https://bugs.python.org/issue45020>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com
[issue45020] Freeze all modules imported during startup. [ In reply to ]
Eric Snow <ericsnowcurrently@gmail.com> added the comment:

On Mon, Aug 30, 2021 at 2:22 PM Guido van Rossum <report@bugs.python.org> wrote:
> I propose to only opt in by default in **PGO builds**. After all what we're doing is another extreme optimization.
>
> It should always be possible to opt in using some -X flag (e.g. to debug the freeze import loader) and it should also always be possible to opt *out* (for those cases where you want to edit an installed stdlib module in-place in anger).
>
> I don't know how the -X mechanism works exactly but probably we could make the spelling
>
> python -X freeze=on|off
>
> with a dynamic default.

+1 to all that

----------

_______________________________________
Python tracker <report@bugs.python.org>
<https://bugs.python.org/issue45020>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com
[issue45020] Freeze all modules imported during startup. [ In reply to ]
Change by hai shi <shihai1991@126.com>:


----------
nosy: +shihai1991

_______________________________________
Python tracker <report@bugs.python.org>
<https://bugs.python.org/issue45020>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com
[issue45020] Freeze all modules imported during startup. [ In reply to ]
Brett Cannon <brett@python.org> added the comment:

> set __file__ (and __path__) on frozen modules?

See https://bugs.python.org/issue21736

----------

_______________________________________
Python tracker <report@bugs.python.org>
<https://bugs.python.org/issue45020>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com
[issue45020] Freeze all modules imported during startup. [ In reply to ]
Marc-Andre Lemburg <mal@egenix.com> added the comment:

On 31.08.2021 20:14, Brett Cannon wrote:
>
> Brett Cannon <brett@python.org> added the comment:
>
>> set __file__ (and __path__) on frozen modules?
>
> See https://bugs.python.org/issue21736

The patch on that ticket is straight from PyRun, where the
__file__ location is set in a way which signals that the file
does not exist, but instead is baked into the executable:

>>> import os
>>> os.__file__
'<pyrun>/os.py'

Not doing this breaks too many tests in the test suite for no
good reason, which is why I mentioned "practicality beats
purity" in the ticket.

----------

_______________________________________
Python tracker <report@bugs.python.org>
<https://bugs.python.org/issue45020>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com
[issue45020] Freeze all modules imported during startup. [ In reply to ]
Change by Eric Snow <ericsnowcurrently@gmail.com>:


----------
keywords: +patch
pull_requests: +26549
stage: needs patch -> patch review
pull_request: https://github.com/python/cpython/pull/28107

_______________________________________
Python tracker <report@bugs.python.org>
<https://bugs.python.org/issue45020>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com
[issue45020] Freeze all modules imported during startup. [ In reply to ]
Inada Naoki <songofacandy@gmail.com> added the comment:

I don't want all frozen header files to be committed in git repository.
Can't we just build them during build process?

----------

_______________________________________
Python tracker <report@bugs.python.org>
<https://bugs.python.org/issue45020>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com
[issue45020] Freeze all modules imported during startup. [ In reply to ]
Eric Snow <ericsnowcurrently@gmail.com> added the comment:

On Tue, Aug 31, 2021 at 10:05 PM Inada Naoki <report@bugs.python.org> wrote:
> I don't want all frozen header files to be committed in git repository.
> Can't we just build them during build process?

That's a good point (and an interesting one). Only two of the frozen
modules are necessary (_frozen_importlib_external and
_frozen_importlib_external, to bootstrap the import system). So for
those two it makes sense to have them in the git repository.

For all the rest it isn't necessary. The only advantage is that
contributors don't have to think about them and they will be
guaranteed to be there. However, if someone clones the repo they have
to build Python, so the frozen modules will get created anyway at that
point.

So I'm fine with not committing all those modules. This will require
that all those files be added to the .gitignore file. (I'll update my
PR accordingly.)

-eric

----------

_______________________________________
Python tracker <report@bugs.python.org>
<https://bugs.python.org/issue45020>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com
[issue45020] Freeze all modules imported during startup. [ In reply to ]
Eric Snow <ericsnowcurrently@gmail.com> added the comment:

On Tue, Aug 31, 2021 at 12:14 PM Brett Cannon <report@bugs.python.org> wrote:
> > set __file__ (and __path__) on frozen modules?
>
> See https://bugs.python.org/issue21736

Great! I'll take a look.

----------

_______________________________________
Python tracker <report@bugs.python.org>
<https://bugs.python.org/issue45020>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com
[issue45020] Freeze all modules imported during startup. [ In reply to ]
Nick Coghlan <ncoghlan@gmail.com> added the comment:

For the module metadata problem: one potential approach to that for "designed to be frozen" stdlib modules is to set the values directly in the module code, rather than trying to set them automatically in the frozen import machinery.

It should also be possible to delete the implicitly created metadata fields and use module level dynamic attribute retrieval to find the stdlib source code for tracebacks and introspection purposes without incurring any start up costs.

----------
nosy: +ncoghlan

_______________________________________
Python tracker <report@bugs.python.org>
<https://bugs.python.org/issue45020>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com
[issue45020] Freeze all modules imported during startup. [ In reply to ]
Marc-Andre Lemburg <mal@egenix.com> added the comment:

FWIW, I've not found the importer for frozen modules to be lacking
features. When using frozen modules, you don't expect to see source
code, so the whole part about finding source code is not really
relevant for that use case.

The only lacking part I found regarding frozen modules is support
for these in pkgutil.py. But that's easy to add:

--- /home/lemburg/egenix/projects/PyRun/Python-3.8.0/Lib/pkgutil.py
2019-10-14 1>
+++ ./Lib/pkgutil.py 2019-11-17 11:36:38.404752218 +0100
@@ -315,20 +315,27 @@
return self.etc[2]==imp.PKG_DIRECTORY

def get_code(self, fullname=None):
+ # eGenix PyRun needs pkgutil to also work for frozen modules,
+ # since pkgutil is used by the runpy module, which is needed
+ # to implement the -m command line switch.
+ if self.code is not None:
+ return self.code
fullname = self._fix_name(fullname)
- if self.code is None:
- mod_type = self.etc[2]
- if mod_type==imp.PY_SOURCE:
- source = self.get_source(fullname)
- self.code = compile(source, self.filename, 'exec')
- elif mod_type==imp.PY_COMPILED:
- self._reopen()
- try:
- self.code = read_code(self.file)
- finally:
- self.file.close()
- elif mod_type==imp.PKG_DIRECTORY:
- self.code = self._get_delegate().get_code()
+ mod_type = self.etc[2]
+ if mod_type == imp.PY_FROZEN:
+ self.code = imp.get_frozen_object(fullname)
+ return self.code
+ elif mod_type==imp.PY_SOURCE:
+ source = self.get_source(fullname)
+ self.code = compile(source, self.filename, 'exec')
+ elif mod_type==imp.PY_COMPILED:
+ self._reopen()
+ try:
+ self.code = read_code(self.file)
+ finally:
+ self.file.close()
+ elif mod_type==imp.PKG_DIRECTORY:
+ self.code = self._get_delegate().get_code()
return self.code

def get_source(self, fullname=None):

----------

_______________________________________
Python tracker <report@bugs.python.org>
<https://bugs.python.org/issue45020>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com
[issue45020] Freeze all modules imported during startup. [ In reply to ]
Change by Eric Snow <ericsnowcurrently@gmail.com>:


----------
dependencies: +De-couple the Windows builds from freezing modules., Marshal output isn't completely deterministic.

_______________________________________
Python tracker <report@bugs.python.org>
<https://bugs.python.org/issue45020>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com
[issue45020] Freeze all modules imported during startup. [ In reply to ]
Guido van Rossum <guido@python.org> added the comment:

If you reduce the number of modules being frozen you could probably manage
to land this (or most of it) before tackling those other issues.

----------

_______________________________________
Python tracker <report@bugs.python.org>
<https://bugs.python.org/issue45020>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com
[issue45020] Freeze all modules imported during startup. [ In reply to ]
Change by Eric Snow <ericsnowcurrently@gmail.com>:


----------
pull_requests: +26729
pull_request: https://github.com/python/cpython/pull/28320

_______________________________________
Python tracker <report@bugs.python.org>
<https://bugs.python.org/issue45020>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com
[issue45020] Freeze all modules imported during startup. [ In reply to ]
Eric Snow <ericsnowcurrently@gmail.com> added the comment:

On Mon, Sep 13, 2021 at 2:59 PM Guido van Rossum <report@bugs.python.org> wrote:
> If you reduce the number of modules being frozen you could probably manage
> to land this (or most of it) before tackling those other issues.

Yeah, that's what I'm doing. :)

----------

_______________________________________
Python tracker <report@bugs.python.org>
<https://bugs.python.org/issue45020>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com
[issue45020] Freeze all modules imported during startup. [ In reply to ]
Change by Eric Snow <ericsnowcurrently@gmail.com>:


----------
pull_requests: +26745
pull_request: https://github.com/python/cpython/pull/28335

_______________________________________
Python tracker <report@bugs.python.org>
<https://bugs.python.org/issue45020>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com

1 2 3 4 5 6  View All