Mailing List Archive

On Sun, 10 Apr 2022, 15:53 Guido van Rossum, <guido@python.org> wrote:

> On Sun, Apr 10, 2022 at 2:31 AM Daniel Pope <lord.mauve@gmail.com> wrote:
>
>> On Fri, 8 Apr 2022, 17:44 Guido van Rossum, <guido@python.org> wrote:
>>
>>> The interesting idea here seems to make "lazy imports" easier to
>>> implement by making them explicit in the code. So far, most lazy import
>>> frameworks for Python have done hacks with `__getattribute__` overrides.
>>>
>>
>> The value is more than ease of implementation. Having syntax for import
>> expressions makes them statically analysable, which is needed for type
>> checkers and IDE autocompletion.
>>
>
> This has been brought up a few times and I don't get it. Currently a use
> of an imported module is perfectly analyzable by all the static type
> checkers I know of (e.g. mypy, Pyre, pyright).
>

I was comparing a hypothetical import expression syntax with alternatives
like __getattribute__ hacks, which I take to mean
somemagicobject.package.module.attribute, or as other have suggested,
importlib.import_module(). I believe those are not statically analysable at
least without special casing them in the type checker.

An import expression would be just as statically analysable as a statement,
while being more ergonomic in certain situations.

>

On Sun, 10 Apr 2022 at 09:31, Daniel Pope <lord.mauve@gmail.com> wrote:
> I would like to bid again for (import package.module) as an expression. Instead of doing the import and assigning package to a variable package it would evaluate to the module object package.module.

I like this proposal and I agree with previous posters about hoisting
being undesirable.

And if lazy importing is slow then that is probably something that can
be optimized away; of course in tight loops or performance-sensitive
code you wouldn't use this functionality.

But perhaps using "from" makes the behavior more clear (and it is shorter):

start_date = from datetime.datetime(2012, 1, 6)

I think the behavior could be to simply keep importing until we have a
non-module value, rather than rely on putting the import name in
parentheses.

Cheers
_______________________________________________
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-leave@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/2NLRKDG3TZCQSALAJVHOM3VKZH45ZYWH/
Code of Conduct: http://python.org/psf/codeofconduct/

To me Steve nailed it... static type checker = lean and mean...

On Fri, Apr 8, 2022, 5:03 AM Steven D'Aprano <steve@pearwood.info> wrote:

> On Fri, Apr 08, 2022 at 08:24:40AM +0000, Malthe wrote:
>
> > But firstly, let me present the idea. It is very simple, that Python
> > should have declarative imports,
>
> I'm not sure I understand why you consider this "declarative".
>
> https://en.wikipedia.org/wiki/Declarative_programming
>
> As I see it, this proposal merely adds a unary operator for implicit
> imports.
>
>
>
> > For example, `some_regex = @re.compile(...)`.
> >
> > What happens then is that before anything else in that module, that
> > symbol is imported:
> >
> > from re import compile as _mangled_re_compile
> >
> > It must be the very first thing (hoisting) because when else would it
> > happen?
>
> On-demand. As you say yourself, this could be a dynamic import at the
> time of use.
>
>
> > It's been suggested before to have a shorthand syntax which
> > does a dynamic import at the time of using it but this brings me to
> > the twist:
> >
> > We want typing to pick up these imports.
>
> At the moment static type checkers have to look for two import
> statements: `import spam`, and `from spam import eggs`.
>
>
> With this proposal, they will still need to look for those two import
> statements, but they will also need to parse every expression looking
> for `@` as a unary operator:
>
> result = 2*x**3 - 3*y + @math.sin(x*y) - 5*x*y**2
>
> This does not help typing at all. It just means the type-checker has to
> do more work to recognise imports.
>
>
>
> > And this twist has a second
> > leg which is that we often need to import symbols simply in order to
> > type some argument type or return type. This leads to a great many
> > more imports to type.
>
> Code is read much more than it is written. I would much prefer to have
> explicit imports (by convention, if not necessity) in one place at the
> top of the page, than to mystery symbols where the implicit import could
> be buried far away.
>
> Right now, this is an error:
>
> # Start of module.
> obj = wibble.foo
>
> because wibble is not a built-in nor a global (excluding weird
> shenanigans committed by other modules), so the name "wibble" doesn't
> exist. But with hoisting, that import could be *anywhere* in the file.
> Even in dead code.
>
> # Start of module.
> obj = wibble.foo
> ...
> ...
> # five pages of code later
> for a in obj.method():
> while flag:
> if condition:
> @wibble
>
> That's right, with hoisting you can use a module before you import it.
> Mind. Blown.
>
> Have pity on beginners, casual Python programmers, and the people who
> have to help them. Don't implement this horror.
>
> If it were merely on-demand imports, then we would know that the import
> @wibble would have to appear in actual, executed code before you can use
> wibble. But with hoisting, we don't even have that promise.
>
> It is truly spooky action at a distance. And to save nothing but an
> import line.
>
>
> > A situation where this would come in really handy is in scripting such
> > as how we use Python in Apache Airflow to let users write out simple
> > workflows. A workflow definition which could be a 5-liner quickly
> > becomes a 20-liner – consider for example:
> >
> > default_args = {
> > "start_date": @datetime.datetime(...)
> > }
>
> That's just a three-liner, which becomes a four-liner:
>
> import datetime
> default_args = {
> "start_date": datetime.datetime(...)
> }
>
>
>
> --
> Steve
> _______________________________________________
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-leave@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-dev@python.org/message/GILL25XYSAPF4FN7LTZC7XLDB7ZX4E4Y/
> Code of Conduct: http://python.org/psf/codeofconduct/
>

On 10/04/2022 15:52, Guido van Rossum wrote:
> On Sun, Apr 10, 2022 at 2:31 AM Daniel Pope <lord.mauve@gmail.com> wrote:
>
> On Fri, 8 Apr 2022, 17:44 Guido van Rossum, <guido@python.org> wrote:
>
> The interesting idea here seems to make "lazy imports" easier
> to implement by making them explicit in the code. So far, most
> lazy import frameworks for Python have done hacks with
> `__getattribute__` overrides.
>
>
> The value is more than ease of implementation. Having syntax for
> import expressions makes them statically analysable, which is
> needed for type checkers and IDE autocompletion.
>
>
> This has been brought up a few times and I don't get it. Currently a
> use of an imported module is perfectly analyzable by all the static
> type checkers I know of (e.g. mypy, Pyre, pyright). For the static
> analyzer it makes no difference if I have
>
> import re
> .
> .
> .
> def foo(x):
>     if re.match(r"blah", x): ...
>
> or the hypothetical inline form:
>
> def foo(x):
>     if @re.match(r"blah", x): ...
>
> But I also see value in being able to type out code that uses
> modules not yet imported without breaking my flow to add an import
> statement. I don't yet trust IDEs to do this because I've been
> bitten by them doing so incorrectly in the past.
>
>
> I have too.

I see no reason why @re.match(...) couldn't just be equivalent to
__import__("re").match(...) (or a more optimised compiled version of
that). Any side-effects of importing the module (of which there probably
shouldn't be any for most modules) just happen at whenever the first
time the @re.match(...) is evaluated.

After the first call, and unless __import__ is overloaded, I also don't
think @re.match(...) would have to be significantly slower than
re.match(...). The latter already has to look up a global variable
(assuming the module was imported globally, which is almost always the
case). I don't think it would be too hard to optimise and Quicken.

Thanks Carl,

> On Apr 9, 2022, at 08:25, Carl Meyer <carl@oddbird.net> wrote:
>
> Our experience in practice, though, has been that universally lazy
> imports is somewhat easier to adopt than Strict Modules, and has had a
> much bigger overall impact on reducing startup time for big CLIs (and
> a big web server too; as you note it's not as serious an issue for a
> web server in production, but restart time still does make a
> difference to dev speed / experience.)

Excellent point about the impact of restarts and development time. That’s been an issue for us a bit, but not an overwhelming motivation to rewrite in other languages[1].

> Removing slow stuff happening
> at import time helps, but it'll never match the speed of not doing the
> import at all! We've seen startup time improvements up to 70% in
> real-world CLIs just by making imports lazy. We've also opened an
> issue to discuss the possibility of upstreaming this. [2]
>
> [1] https://github.com/facebookincubator/cinder/#strict-modules
> [2] https://bugs.python.org/issue46963

Post-GH-issues-migration link for the issue: https://github.com/python/cpython/issues/91119

I’ve put some questions and comments there, but I’m also really curious about the technical details for your lazy imports. Have you gotten as far as thinking about a PR or PEP?

-Barry

[1] Not that there aren’t other reasons folks give for rewriting, such as multicore performance, ecosystem alignment (e.g. SREs being more comfortable in Go), etc.

On Sun, Apr 10, 2022 at 2:39 AM Daniel Pope <lord.mauve@gmail.com> wrote:

> On Fri, 8 Apr 2022, 17:44 Guido van Rossum, <guido@python.org> wrote:
>
>> The interesting idea here seems to make "lazy imports" easier to
>> implement by making them explicit in the code. So far, most lazy import
>> frameworks for Python have done hacks with `__getattribute__` overrides.
>>
>
> The value is more than ease of implementation. Having syntax for import
> expressions makes them statically analysable, which is needed for type
> checkers and IDE autocompletion.
>
> But I also see value in being able to type out code that uses modules not
> yet imported without breaking my flow to add an import statement. I don't
> yet trust IDEs to do this because I've been bitten by them doing so
> incorrectly in the past.
>
> The key questions to me are
>> - What should the notation be?
>>
>
> I would like to bid again for (import package.module) as an expression.
> Instead of doing the import and assigning package to a variable package it
> would evaluate to the module object package.module.
>

That is an extremely subtle shift for what `import x.y` does compared to
`(import x.y)`. That requires a context switch of not only seeing `import`
in an expression context, but that the statement also acts differently in
terms of what is returned by the equivalent statement. I really don't to
try and teach that distinction to a newcomer. And I don't think the
ergonomics are great enough to warrant the context switch.

Plus you can do this today with imortlib.import_module(). Unless you're
suggesting the name also be bound in the scope it's run in? In which case
that's `(abc := importlib.import_module("collections.abc")).Mapping`.

-Brett


>
> The `as` form is not needed because no name is assigned and the `from`
> form isn't as valuable because you can just use attribute access afterwards.
>
> It isn't terse but it does make use of the import keyword and is thus
> instantly recognisable. It is even syntax highlighted correctly by much
> existing software. If we're using the import keyword then I think it has to
> look like this.
>
> But I concede that it isn't particularly elegant to type hint things with
>
> (import collections.abc).Mapping
>
> ...but not so inelegant that I couldn't see myself using it for a few
> one-off imports per module.
>
> A quirk is that it means there's a big difference between the statements
>
> import foo
>
> and
>
> (import foo)
>
> because one assigns a variable. I don't mind that; I don't think it is too
> ambiguous to a reader.
>
>
>
> _______________________________________________
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-leave@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-dev@python.org/message/T6V7WZAFEXGWMHXLHS7XHYXI5OPMOZKA/
> Code of Conduct: http://python.org/psf/codeofconduct/
>

On Mon, Apr 11, 2022 at 2:03 PM Brett Cannon <brett@python.org> wrote:

> That is an extremely subtle shift for what `import x.y` does compared to
> `(import x.y)`. That requires a context switch of not only seeing `import`
> in an expression context, but that the statement also acts differently in
> terms of what is returned by the equivalent statement. I really don't to
> try and teach that distinction to a newcomer.
>

Frankly, there are already an overwhelming-to-a-newcomer number of ways to
import modules. We really don't want nore!

-CHB


--
Christopher Barker, PhD (Chris)

Python Language Consulting
- Teaching
- Scientific Software Development
- Desktop GUI and Web Development
- wxPython, numpy, scipy, Cython