Mailing List Archive

Declarative imports
This is an idea which has been brought up before, sometimes introduced
as "heresy". But an interesting twist has surfaced now which is
typing.

But firstly, let me present the idea. It is very simple, that Python
should have declarative imports, usable anywhere using a simple
syntax, @<dotted-name>.

For example, `some_regex = @re.compile(...)`.

What happens then is that before anything else in that module, that
symbol is imported:

from re import compile as _mangled_re_compile

It must be the very first thing (hoisting) because when else would it
happen? It's been suggested before to have a shorthand syntax which
does a dynamic import at the time of using it but this brings me to
the twist:

We want typing to pick up these imports. And this twist has a second
leg which is that we often need to import symbols simply in order to
type some argument type or return type. This leads to a great many
more imports to type.

(Nevermind that if you want to take typing further, abstract
interfaces really make more sense rather than specific
implementations, but the point is the same.)

A situation where this would come in really handy is in scripting such
as how we use Python in Apache Airflow to let users write out simple
workflows. A workflow definition which could be a 5-liner quickly
becomes a 20-liner – consider for example:

default_args = {
"start_date": @datetime.datetime(...)
}

It's a lot more ergonomic from a user perspective (well perhaps for
some users and for some programs).

Thoughts?

Cheers
_______________________________________________
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-leave@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/UX6EJHLJNNLMFPWVPF5ANYHQSHDZK7SV/
Code of Conduct: http://python.org/psf/codeofconduct/
Re: Declarative imports [ In reply to ]
On Fri, 8 Apr 2022 at 18:29, Malthe <mborch@gmail.com> wrote:
>
> This is an idea which has been brought up before, sometimes introduced
> as "heresy". But an interesting twist has surfaced now which is
> typing.
>
> But firstly, let me present the idea. It is very simple, that Python
> should have declarative imports, usable anywhere using a simple
> syntax, @<dotted-name>.
>
> For example, `some_regex = @re.compile(...)`.
>
> What happens then is that before anything else in that module, that
> symbol is imported:
>
> from re import compile as _mangled_re_compile
>
> It must be the very first thing (hoisting) because when else would it
> happen?

JavaScript has this feature. ES6 modules are always imported
declaratively and always before the rest of the script is executed.

It is *extremely* annoying. Most of the time, it behaves the same way
as Python's dynamic imports, but any time it doesn't, the "imports
before the rest of the code" feature is nothing but hassle.

Here's an alternative: Have a whitelist of modules that you make use
of in this way. Any time you refer to a name that's on that whitelist
and doesn't have an assigned meaning in the module, insert an import
statement at the top of the script. It's not too hard to set up your
editor to be able to do this (eg in SciTE, I can press Ctrl+1 to
quickly run a script; a lot of editors let you configure a code
formatter that runs automatically on save, which would also work), and
then you retain full control of exactly where the importing happens.

ChrisA
_______________________________________________
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-leave@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/YUNFSHIONZNLZ6QS3GWF4QVLP4IWZY6W/
Code of Conduct: http://python.org/psf/codeofconduct/
Re: Declarative imports [ In reply to ]
On Fri, 8 Apr 2022 at 09:29, Malthe <mborch@gmail.com> wrote:

> A workflow definition which could be a 5-liner quickly
> becomes a 20-liner – consider for example:
>
> default_args = {
> "start_date": @datetime.datetime(...)
> }

Are you exaggerating for effect here or would this *actually* just expand to

from datetime import datetime
default_args = {
"start_date": datetime(...)
}

I'm not suggesting that there isn't a genuine benefit here, but your
example doesn't demonstrate what it is.

Paul
_______________________________________________
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-leave@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/RRLFDL2BDF3UV646GVZVRXUSGM7X5FRQ/
Code of Conduct: http://python.org/psf/codeofconduct/
Re: Declarative imports [ In reply to ]
On Fri, 8 Apr 2022 at 08:51, Paul Moore <p.f.moore@gmail.com> wrote:
> Are you exaggerating for effect here or would this *actually* just expand to
>
> from datetime import datetime
> default_args = {
> "start_date": datetime(...)
> }

Yes – and of course that is just a snippet, an actual complete script
will have lots of such imports.

The point is that in a scripting situation (especially one in which
you have many small scripts) the
top-level import requirement for simple imports like `datetime`
becomes rather verbose.

Thanks
_______________________________________________
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-leave@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/FVYRPSN6A7KN73MTQIF2UIU5W3HSUPGV/
Code of Conduct: http://python.org/psf/codeofconduct/
Re: Declarative imports [ In reply to ]
On Fri, Apr 08, 2022 at 08:24:40AM +0000, Malthe wrote:

> But firstly, let me present the idea. It is very simple, that Python
> should have declarative imports,

I'm not sure I understand why you consider this "declarative".

https://en.wikipedia.org/wiki/Declarative_programming

As I see it, this proposal merely adds a unary operator for implicit
imports.



> For example, `some_regex = @re.compile(...)`.
>
> What happens then is that before anything else in that module, that
> symbol is imported:
>
> from re import compile as _mangled_re_compile
>
> It must be the very first thing (hoisting) because when else would it
> happen?

On-demand. As you say yourself, this could be a dynamic import at the
time of use.


> It's been suggested before to have a shorthand syntax which
> does a dynamic import at the time of using it but this brings me to
> the twist:
>
> We want typing to pick up these imports.

At the moment static type checkers have to look for two import
statements: `import spam`, and `from spam import eggs`.


With this proposal, they will still need to look for those two import
statements, but they will also need to parse every expression looking
for `@` as a unary operator:

result = 2*x**3 - 3*y + @math.sin(x*y) - 5*x*y**2

This does not help typing at all. It just means the type-checker has to
do more work to recognise imports.



> And this twist has a second
> leg which is that we often need to import symbols simply in order to
> type some argument type or return type. This leads to a great many
> more imports to type.

Code is read much more than it is written. I would much prefer to have
explicit imports (by convention, if not necessity) in one place at the
top of the page, than to mystery symbols where the implicit import could
be buried far away.

Right now, this is an error:

# Start of module.
obj = wibble.foo

because wibble is not a built-in nor a global (excluding weird
shenanigans committed by other modules), so the name "wibble" doesn't
exist. But with hoisting, that import could be *anywhere* in the file.
Even in dead code.

# Start of module.
obj = wibble.foo
...
...
# five pages of code later
for a in obj.method():
while flag:
if condition:
@wibble

That's right, with hoisting you can use a module before you import it.
Mind. Blown.

Have pity on beginners, casual Python programmers, and the people who
have to help them. Don't implement this horror.

If it were merely on-demand imports, then we would know that the import
@wibble would have to appear in actual, executed code before you can use
wibble. But with hoisting, we don't even have that promise.

It is truly spooky action at a distance. And to save nothing but an
import line.


> A situation where this would come in really handy is in scripting such
> as how we use Python in Apache Airflow to let users write out simple
> workflows. A workflow definition which could be a 5-liner quickly
> becomes a 20-liner – consider for example:
>
> default_args = {
> "start_date": @datetime.datetime(...)
> }

That's just a three-liner, which becomes a four-liner:

import datetime
default_args = {
"start_date": datetime.datetime(...)
}



--
Steve
_______________________________________________
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-leave@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/GILL25XYSAPF4FN7LTZC7XLDB7ZX4E4Y/
Code of Conduct: http://python.org/psf/codeofconduct/
Re: Declarative imports [ In reply to ]
On Fri, 8 Apr 2022, 09:30 Malthe, <mborch@gmail.com> wrote:

> For example, `some_regex = @re.compile(...)`.
>

I like the idea of import expressions. I pitched it on Discourse recently:
https://discuss.python.org/t/import-expressions/11582

However, I do not see hoisting as something that should be done by Python.
In that thread it was suggested your IDE could do it, or maybe isort, which
seems fine because it's at the programmer's discretion and ends up being
explicit.

But, your proposed syntax is not usable because it is ambiguous. Exactly
what you propose is already used for decorators in a way that the parser
would not be able to distinguish a decorator from an import expression.
Consider:

@ham.spam() # import expression or decorator?

def eggs():
...

This currently parses as a decorator and for backwards compatibility that
must not change, which means that import expressions would be usable in
some contexts and not others purely based on what follows them.
Re: Declarative imports [ In reply to ]
On Fri, 8 Apr 2022 at 10:15, Daniel Pope <lord.mauve@gmail.com> wrote:
> But, your proposed syntax is not usable because it is ambiguous. Exactly what you propose is already used for decorators in a way that the parser would not be able to distinguish a decorator from an import expression. Consider:
>
> @ham.spam() # import expression or decorator?
>
> def eggs():
> ...
>
> This currently parses as a decorator and for backwards compatibility that must not change, which means that import expressions would be usable in some contexts and not others purely based on what follows them.

Perhaps `some_regex = re::compile(r"...")` could work.

That is, <dotted-path>::<name-to-import> to delineate the import.

@breakfast.ham::spam
def eggs(): pass

Cheers
_______________________________________________
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-leave@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/VXOSHFQDLPI5KXVNXWKFMRZOZHYJLZL5/
Code of Conduct: http://python.org/psf/codeofconduct/
Re: Declarative imports [ In reply to ]
On Fri, 8 Apr 2022 at 10:27, Malthe <mborch@gmail.com> wrote:
>
> On Fri, 8 Apr 2022 at 08:51, Paul Moore <p.f.moore@gmail.com> wrote:
> > Are you exaggerating for effect here or would this *actually* just expand to
> >
> > from datetime import datetime
> > default_args = {
> > "start_date": datetime(...)
> > }
>
> Yes – and of course that is just a snippet, an actual complete script
> will have lots of such imports.
>
> The point is that in a scripting situation (especially one in which
> you have many small scripts) the
> top-level import requirement for simple imports like `datetime`
> becomes rather verbose.

But Python's origin is in scripting situations, and explicit imports
for scripts has always been both the normal approach, and one of
Python's *strengths* ("Explicit is better than implicit"). Arguing
that explicit imports are a bad thing in (general) scripts is arguing
against decades of history and experience. If there's a reason why
*your specific context* would benefit from an abbreviated form, you
need to present it. But arguing that explicit imports are too verbose
for all cases of scripting isn't going to fly, frankly.

Paul
_______________________________________________
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-leave@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/TMJ4ZI7J62C64PZXQU3WRLFA6FAKPRVB/
Code of Conduct: http://python.org/psf/codeofconduct/
Re: Declarative imports [ In reply to ]
I do not think it's a "typing" improvement.

I think what I like most in the proposal (which might be implemented differently - I am not too tied to this) is that it potentially brings huge optimisations in import times when currently a lot of imports are done unnecessarily early - slowing down startup times of tools that require parsing of python code.

There are few examples of those:
* Airflow DAGs - this is one of the problems we have in Airflow (and this is I think one of the reasons why Malthe brought it) is nicely described here https://airflow.apache.org/docs/apache-airflow/stable/best-practices.html#top-level-python-code
* Tools like click-complete (where autocompletion suggestions are generated based on the decorators of Python code)
* Static Python Typing checks (mostly when you declare Type returned/used by function that is only used inside the function).

In all those cases, you really do not care about some of the imported code that is in some kind of "internal scope" and your python parsing does not **really** need to perform the imports to do the job.

* In Airflow DAGs - tasks which are defined as functions (and default arguments from the example from Malthe) are not executed at all when the DAGs are parsed by scheduler. The DAGs are parsed to get the structure of the DAG. In this case we could optimise the time it take to parse the DAG and generate the structure.

* In Click-complete - we only really care about names of the parameters for functions, but static typing for the function parmeters or returned value are not needed at all to generate complete suggestions. In this case importing time is essential, because you want to provide autocomplete suggestions **really quickly**

* In case of typing of parameters or return values, it leads quite often to circular imports generated. If - in your module - the only place you use a type is inside one or two functions, adding static typing involves adding top-level import for that type. And that has huge potential of generating circular imports. Before static typing in parameters or return values of those, local imports would do the job nicely, but with typing added to parameters and return values, you cannot do local imports in those functions that have those parameters or return values. This often leads to adding local imports elsewhere, but in fact a "logical" thing is that those imports should be local for the functions and any of the scopes that the functions are called. Top-level imports in this case are causing unnecessary cross-module dependencies and circular imports.

And some of those problems can currently be solved with local imports - but not all (default_values case by Malthe cannot).

I think this idea has really good "need" behind. Maybe the way to implement could be improved on, but I see clearly the use cases and needs the "general idea" aims to solve.
_______________________________________________
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-leave@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/ALXSM6753M7AT6MT7KVQULUCFX5G4XT7/
Code of Conduct: http://python.org/psf/codeofconduct/
Re: Declarative imports [ In reply to ]
I've read the rest of the thread so far, and agree strongly that we
can't do this at the language/runtime level.

However ...

On 4/8/2022 9:24 AM, Malthe wrote:
> This is an idea which has been brought up before, sometimes introduced
> as "heresy". But an interesting twist has surfaced now which is
> typing.
>
> [SNIP]
>
> We want typing to pick up these imports. And this twist has a second
> leg which is that we often need to import symbols simply in order to
> type some argument type or return type. This leads to a great many
> more imports to type.

Type checkers can do whatever they like. The syntax has to remain a
valid Python expression, but the semantics of it can include implicit
imports if that's what you want it to do.

I'd certainly make use of it on those occasions I actually put type
annotations in code (rather than .pyi files).

But it doesn't need the runtime to do anything differently if the module
isn't really going to be used at that point.

Cheers,
Steve
_______________________________________________
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-leave@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/57NDZA7B3Z4PLGS5JON5YXOP44KI4LVU/
Code of Conduct: http://python.org/psf/codeofconduct/
Re: Declarative imports [ In reply to ]
On Fri, 8 Apr 2022 at 11:26, Malthe <mborch@gmail.com> wrote:
> Perhaps `some_regex = re::compile(r"...")` could work.
>
> That is, <dotted-path>::<name-to-import> to delineate the import.

As I mentioned in the Discourse thread, this syntax chimes with Rust
(and C++, and also PHP, triggering fond memories of Paamayim
Nekudotayim).

There are two problems, sort of on the minor side:

First, there's one place in Python where this can occur and that is in
slices: xs[start::step] means the slice from start to the end with a
step of step. But xs[(module::name)] is currently a syntax error, so
you could just require parentheses to disambiguate the case. Still, it
is confusing if you write something slightly more complex like xs[1 +
math::factorial(n)] and you get NameError: name 'math' is not defined.

Secondly, it precludes importing just a module object, you have to
import a name from a module. Maybe using a module object in an
expression is niche and can be disregarded. Or a solution could be to
allow an expression like (re::) to import a module object, which I
guess we could get used to. Also fun would be to consider what ::name
does - access to current globals without the global keyword?

I think this discussion is better suited to the python-ideas mailing
list, or the Discourse thread I linked earlier.
_______________________________________________
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-leave@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/MO524TQHUR7PMUWOZ4PBQAXDMOOF2KFP/
Code of Conduct: http://python.org/psf/codeofconduct/
Re: Declarative imports [ In reply to ]
On Fri, 8 Apr 2022 at 12:23, Steve Dower <steve.dower@python.org> wrote:
>
> I've read the rest of the thread so far, and agree strongly that we
> can't do this at the language/runtime level.

You mean the hoisting, right?

I don't see any reason why an import expression without hoisting would
be impractical. But I'd like to hear your thoughts if you think it is.

Desirability on the other hand is subjective. I think I actually do
desire it, others are not bothered. I don't see strong arguments as to
why you definitely wouldn't want it in your language.

Design is hard, but designing this is definitely not as hard as
designing match/case or except* statements.
_______________________________________________
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-leave@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/G4TOKDQPJPKNEGPUGYMJFOHE7FSID2OY/
Code of Conduct: http://python.org/psf/codeofconduct/
Re: Declarative imports [ In reply to ]
On Fri, 8 Apr 2022 at 12:57, Daniel Pope <lord.mauve@gmail.com> wrote:
>
> On Fri, 8 Apr 2022 at 12:23, Steve Dower <steve.dower@python.org> wrote:
> >
> > I've read the rest of the thread so far, and agree strongly that we
> > can't do this at the language/runtime level.
>
> You mean the hoisting, right?
>
> I don't see any reason why an import expression without hoisting would
> be impractical. But I'd like to hear your thoughts if you think it is.
>
> Desirability on the other hand is subjective. I think I actually do
> desire it, others are not bothered. I don't see strong arguments as to
> why you definitely wouldn't want it in your language.

OK, I'll be explicit. I don't want this in Python. Having imports at
the top of a file makes it much easier to see your dependencies. The
current language features, combined with community wide style guides
that discourage the use of local imports, make this straightforward,
while still providing means of doing local imports if needed.

This proposal isn't just about having a new syntax to do an on-demand
import. It's also about normalising the idea that people can pull code
from other modules without declaring the intent to do so up front.
While I don't dispute that in some circumstances (notably the REPL,
and throwaway scripts[1]) not having to add import statements would be
nice, I don't see how we'd limit usage to cases like that - and I
*really* don't want to have to work out where some weird inline import
happened while debugging a priority 1 bug in a 10,000 line code base
at 2am...

Also, it should be possible to do something like this using the
existing import machinery:

from magic_import import on_demand_loader as OD
...
# Many lines of code later
default_args = {
"start_date": OD.datetime.datetime(...)
}


Trivial proof of concept implementation:

class OD:
def __getattr__(self, name):
return __import__(name)

OD = OD()

print(OD.datetime.datetime.now())

Paul

[1] Of course, today's throwaway script usually turns out to be next
month's key component in a mission-critical software stack :-(
_______________________________________________
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-leave@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/HC4UJPBWJHU5JWIPQQKNJRMN2QBXHJVC/
Code of Conduct: http://python.org/psf/codeofconduct/
Re: Declarative imports [ In reply to ]
On 4/8/2022 12:51 PM, Daniel Pope wrote:
> On Fri, 8 Apr 2022 at 12:23, Steve Dower <steve.dower@python.org> wrote:
>>
>> I've read the rest of the thread so far, and agree strongly that we
>> can't do this at the language/runtime level.
>
> You mean the hoisting, right?
>
> I don't see any reason why an import expression without hoisting would
> be impractical. But I'd like to hear your thoughts if you think it is.

Sure, __import__() or better yet, importlib.util.import_module() are
perfectly good ways to do an import in an expression. Both have exactly
the right amount of code-smell as well, and only a few surprising ways
to interfere with the behaviour of apparently completely unrelated code ;)

As others pointed out, implicitly defining variables that outlive the
evaluation of an expression is incredibly messy. It was bad enough
getting assignment expressions defined well enough to (a) not be a
terrible footgun and (b) be possible for the average developer to
predict what will happen when they're used (arguably this isn't true
either, but the "average" developer probably isn't doing things that
will hit the edge cases).

We don't want to have to figure something like that out again. Certainly
not for the sake of typing - the SC has already ruled that language
development won't be driven by the convenience of static type users.

Cheers,
Steve

_______________________________________________
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-leave@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/SRAIPD7DQH36DNBZVEAA6I56IZHLGA66/
Code of Conduct: http://python.org/psf/codeofconduct/
Re: Declarative imports [ In reply to ]
While it's different than you proposal, some people may like this
magic "smart imports" project which lazily imports a module when it's
used :-)

Project: https://pypi.org/project/smart-imports/

Replace:
---

import math
from my_project import calc
# 100500 other imports

def my_code(argument, function=calc):
return math.log(function(argument))
---

with:
---
import smart_imports
smart_imports.all()
# no any other imports

def my_code(argument, function=calc):
return math.log(function(argument))
---

I don't like it :-) I prefer explicit imports at the top!

Victor
_______________________________________________
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-leave@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/KLXXAFVPY6RMGJVIFPJ5UNIXNO2YG472/
Code of Conduct: http://python.org/psf/codeofconduct/
Re: Declarative imports [ In reply to ]
The interesting idea here seems to make "lazy imports" easier to implement
by making them explicit in the code. So far, most lazy import frameworks
for Python have done hacks with `__getattribute__` overrides. IIRC the
Cinder version even modifies the bytecode and/or the interpreter.
Disregarding the specific notation proposed, *if* people would be willing
to mark the points where they expect lazy imports explicitly, that would
make implementation much simpler.

The argument that "imports on top" makes code more readable seems pretty
weak to me. The current hacks to speed up startup already violate this rule
(imports inside functions), and in most cases I start reading or writing
code in the middle of a file (having gotten there via a search in my
editor) and the meaning of an import is either obvious (e.g. re.match(...))
or requires another annoying search to find the definition of a certain
unknown variable. Tools can easily show all imports a module does.

The key questions to me are
- What should the notation be?
- Will users be willing to use it?

--Guido



On Fri, Apr 8, 2022 at 1:26 AM Malthe <mborch@gmail.com> wrote:

> This is an idea which has been brought up before, sometimes introduced
> as "heresy". But an interesting twist has surfaced now which is
> typing.
>
> But firstly, let me present the idea. It is very simple, that Python
> should have declarative imports, usable anywhere using a simple
> syntax, @<dotted-name>.
>
> For example, `some_regex = @re.compile(...)`.
>
> What happens then is that before anything else in that module, that
> symbol is imported:
>
> from re import compile as _mangled_re_compile
>
> It must be the very first thing (hoisting) because when else would it
> happen? It's been suggested before to have a shorthand syntax which
> does a dynamic import at the time of using it but this brings me to
> the twist:
>
> We want typing to pick up these imports. And this twist has a second
> leg which is that we often need to import symbols simply in order to
> type some argument type or return type. This leads to a great many
> more imports to type.
>
> (Nevermind that if you want to take typing further, abstract
> interfaces really make more sense rather than specific
> implementations, but the point is the same.)
>
> A situation where this would come in really handy is in scripting such
> as how we use Python in Apache Airflow to let users write out simple
> workflows. A workflow definition which could be a 5-liner quickly
> becomes a 20-liner – consider for example:
>
> default_args = {
> "start_date": @datetime.datetime(...)
> }
>
> It's a lot more ergonomic from a user perspective (well perhaps for
> some users and for some programs).
>
> Thoughts?
>
> Cheers
> _______________________________________________
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-leave@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-dev@python.org/message/UX6EJHLJNNLMFPWVPF5ANYHQSHDZK7SV/
> Code of Conduct: http://python.org/psf/codeofconduct/
>


--
--Guido van Rossum (python.org/~guido)
*Pronouns: he/him **(why is my pronoun here?)*
<http://feministing.com/2015/02/03/how-using-they-as-a-singular-pronoun-can-change-the-world/>
Re: Declarative imports [ In reply to ]
On Fri, 8 Apr 2022 at 16:40, Guido van Rossum <guido@python.org> wrote:
> The interesting idea here seems to make "lazy imports" easier to implement by making them explicit in the code. So far, most lazy import frameworks for Python have done hacks with `__getattribute__` overrides. IIRC the Cinder version even modifies the bytecode and/or the interpreter. Disregarding the specific notation proposed, *if* people would be willing to mark the points where they expect lazy imports explicitly, that would make implementation much simpler.

Actually, to me the interesting idea is not so much lazy imports – I
think they should not be lazy, at least that was my initial thought. I
think they should be immediately resolved before anything else in that
module:

1. This would settle any discussion about performance impact (there
wouldn't be any).
2. This would enable IDEs, typers and other tooling to know the type
using existing import logic.
3. Catch errors early!

Notation is hard of course. Would users be willing to use it? I think
so, at least in Rust, people do use it and I think just in the right
places – typically for imports that are either "canonicalized" such as
`re.compile` is in Python, or used just once.

Cheers
_______________________________________________
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-leave@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/NKBWGPJRKYQX4Z4WDDD74CSV4XEBGTFD/
Code of Conduct: http://python.org/psf/codeofconduct/
Re: Declarative imports [ In reply to ]
You only get the ease-of-implementation benefit if you are willing to
explicitly mark every _use_ of a lazy-imported name as special (and
give the fully qualified name at every usage site). This is rather
more cumbersome (assuming multiple uses in a module) than just
explicitly marking an import as lazy in one location and then using
the imported name in multiple places normally.

Other "lazy import" solutions are trying to solve a problem where you
want the name to be usable (without special syntax or marking) in many
different places in a module, and visible in the module namespace
always -- but not actually imported until someone accesses/uses it.
The difficulty arises because in this case you need some kind of
placeholder for the "deferred import", but you need to avoid this
"deferred object" escaping and becoming visible to Python code without
being resolved first. Explicitly marking which imports are lazy is
fine if you want it (it's just a matter of syntax), but it doesn't do
anything to solve the problem of allowing usage of the lazy-imported
name to be transparent.

I agree that the idea that top-of-module imports help readability is
overstated; it sounds slightly Stockholm-syndrome-ish to me :)
Top-of-module imports are frankly a pain to maintain and a pain to
read (because they are often distant from the uses of the names). But
they are a necessary evil if you want a) namespaces and b) not
constantly retyping fully-qualified names at every usage site. Python
is pretty committed to namespaces at this point (and I wouldn't want
to change that), so that leaves the choice between top-of-module
imports vs fully qualifying every use of every name; pick your poison.
(Inline imports in a scope with multiple uses are a middle ground.)

Carl
_______________________________________________
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-leave@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/N4T2YMPHBLJXKCFA5CIPBFIZJJKO7SHR/
Code of Conduct: http://python.org/psf/codeofconduct/
Re: Declarative imports [ In reply to ]
Start up overhead due to imports is a real problem for some class of applications, e.g. CLIs, and I’ve seen a lot of hacks implemented to get Python CLIs to be more responsive. E.g. getting from invocation to —help output is a major UX problem.

It’s often more complicated than just imports alone though. Expensive module scope initializations and decorators contribute to this problem. Python start up time is one of the main drivers for rewriting CLIs in Go and other languages where I work. Note that this is much less of a problem for things like web services or other long running applications because that start up time is either amortized over the lifetime of the application, or aren’t directly visible to the end user.

Lazy imports *might* help with this and seems aligned with the common trick of moving imports into functions rather than at module scope. Faster CPython might help too. But these all feel like they aren’t tackling the start up problem head on[1]. Lots of ideas have been discussed over the years (I remember some in-depth ones at the Microsoft core sprint a few years ago), and I’m sure there are all kinds of other tricks that people use.

However, if start up time isn’t a direct benefit of on-demand imports (a.k.a. declarative imports), I’m not sure how actually useful or used they will be. I dunno, top-of-module imports never really bothered me that much.

-Barry

[1] I could be wrong about Faster CPython; ISTR there are some tickets on that project’s tracker that talk about start up times.

> On Apr 8, 2022, at 09:40, Guido van Rossum <guido@python.org> wrote:
>
> The interesting idea here seems to make "lazy imports" easier to implement by making them explicit in the code. So far, most lazy import frameworks for Python have done hacks with `__getattribute__` overrides. IIRC the Cinder version even modifies the bytecode and/or the interpreter. Disregarding the specific notation proposed, *if* people would be willing to mark the points where they expect lazy imports explicitly, that would make implementation much simpler.
>
> The argument that "imports on top" makes code more readable seems pretty weak to me. The current hacks to speed up startup already violate this rule (imports inside functions), and in most cases I start reading or writing code in the middle of a file (having gotten there via a search in my editor) and the meaning of an import is either obvious (e.g. re.match(...)) or requires another annoying search to find the definition of a certain unknown variable. Tools can easily show all imports a module does.
>
> The key questions to me are
> - What should the notation be?
> - Will users be willing to use it?
>
> --Guido
>
>
>
> On Fri, Apr 8, 2022 at 1:26 AM Malthe <mborch@gmail.com> wrote:
> This is an idea which has been brought up before, sometimes introduced
> as "heresy". But an interesting twist has surfaced now which is
> typing.
>
> But firstly, let me present the idea. It is very simple, that Python
> should have declarative imports, usable anywhere using a simple
> syntax, @<dotted-name>.
>
> For example, `some_regex = @re.compile(...)`.
>
> What happens then is that before anything else in that module, that
> symbol is imported:
>
> from re import compile as _mangled_re_compile
>
> It must be the very first thing (hoisting) because when else would it
> happen? It's been suggested before to have a shorthand syntax which
> does a dynamic import at the time of using it but this brings me to
> the twist:
>
> We want typing to pick up these imports. And this twist has a second
> leg which is that we often need to import symbols simply in order to
> type some argument type or return type. This leads to a great many
> more imports to type.
>
> (Nevermind that if you want to take typing further, abstract
> interfaces really make more sense rather than specific
> implementations, but the point is the same.)
>
> A situation where this would come in really handy is in scripting such
> as how we use Python in Apache Airflow to let users write out simple
> workflows. A workflow definition which could be a 5-liner quickly
> becomes a 20-liner – consider for example:
>
> default_args = {
> "start_date": @datetime.datetime(...)
> }
>
> It's a lot more ergonomic from a user perspective (well perhaps for
> some users and for some programs).
>
> Thoughts?
>
> Cheers
> _______________________________________________
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-leave@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/UX6EJHLJNNLMFPWVPF5ANYHQSHDZK7SV/
> Code of Conduct: http://python.org/psf/codeofconduct/
>
>
> --
> --Guido van Rossum (python.org/~guido)
> Pronouns: he/him (why is my pronoun here?)
> _______________________________________________
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-leave@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/BTWPAJRVQ24QX2Z7TIQPDYRXKJOGRPMU/
> Code of Conduct: http://python.org/psf/codeofconduct/
Re: Declarative imports [ In reply to ]
An interesting point in the lazy imports design space that I hadn't
previously considered could be:

- lazy imports are explicitly marked and usage of the imported name
within the module is transparent, but
- lazily imported names are _not_ visible in the module namespace;
they can't be accessed by other modules or re-exported; they are
internal-use-only within the module

This compromise would, I think, make it possible to implement lazy
imports entirely in the compiler (effectively as syntax sugar for an
inline import at every usage site), which is definitely an
implementation improvement.

I think in practice explicitly marking lazy imports would make it
somewhat harder to gain the benefits of lazy imports for e.g. speeding
up startup time in a large CLI, compared to an implicit/automatic
approach. But still could be usable to get significant benefits.

Carl
_______________________________________________
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-leave@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/ZT6CXQPFCWZD2M65YXCSAPPGNDGA6WNE/
Code of Conduct: http://python.org/psf/codeofconduct/
Re: Declarative imports [ In reply to ]
Hi Barry,

On Fri, Apr 8, 2022 at 12:44 PM Barry Warsaw <barry@python.org> wrote:
>
> Start up overhead due to imports is a real problem for some class of applications, e.g. CLIs, and I’ve seen a lot of hacks implemented to get Python CLIs to be more responsive. E.g. getting from invocation to —help output is a major UX problem.

Definitely, we have this same problem, and also the same symptom of
people pushing hard to rewrite Python CLIs in Go for this reason.

> It’s often more complicated than just imports alone though. Expensive module scope initializations and decorators contribute to this problem.

One of our projects that can prevent much of this expensive work being
done at import time is Strict Modules[1]. Currently it's only
available as part of Cinder, though we're hoping to make it
pip-installable as part of our project to make Cinder's features more
easily accessible.

Our experience in practice, though, has been that universally lazy
imports is somewhat easier to adopt than Strict Modules, and has had a
much bigger overall impact on reducing startup time for big CLIs (and
a big web server too; as you note it's not as serious an issue for a
web server in production, but restart time still does make a
difference to dev speed / experience.) Removing slow stuff happening
at import time helps, but it'll never match the speed of not doing the
import at all! We've seen startup time improvements up to 70% in
real-world CLIs just by making imports lazy. We've also opened an
issue to discuss the possibility of upstreaming this. [2]

[1] https://github.com/facebookincubator/cinder/#strict-modules
[2] https://bugs.python.org/issue46963
_______________________________________________
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-leave@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/62OTFJMAMQ2WHZ4H3TUEJTECMPJDQ557/
Code of Conduct: http://python.org/psf/codeofconduct/
Re: Declarative imports [ In reply to ]
Hi Malthe,

On Fri, Apr 8, 2022 at 12:04 PM Malthe <mborch@gmail.com> wrote:
> Actually, to me the interesting idea is not so much lazy imports – I
> think they should not be lazy, at least that was my initial thought. I
> think they should be immediately resolved before anything else in that
> module:

I'm +0.25 on your idea as simply a streamlined syntax for inline
imports (given actually finding an appropriate syntax, which I haven't
thought much about; @ probably doesn't work due to the conflict with
decorator syntax, but there might be other options.). If it existed I
would probably use it occasionally, but I don't feel a strong need for
it.

But I think your proposal is much stronger if you eliminate the
hoisting from it; with the hoisting I'd be -1. Out-of-source-order
execution like this is just quite surprising in the context of Python.

> 1. This would settle any discussion about performance impact (there
> wouldn't be any).

If the inline import is actually a performance problem because a
certain code path is very hot, the solution is simple: don't use the
inline import there, use a top-of-module import instead.

> 2. This would enable IDEs, typers and other tooling to know the type
> using existing import logic.

I don't think it enables any such thing. Static-analysis tooling has
only the source code to work with, runtime behavior doesn't affect it.
If the runtime executes these imports out-of-order, that won't make
the slightest difference to how easily IDEs and type checkers can
analyze the source code.

> 3. Catch errors early!

The very strong precedent in Python is that errors in code are caught
when the code runs, and the code runs more or less when you'd expect
it to, in source order. If you want to catch errors earlier, use a
static analysis tool to help catch them.

Carl
_______________________________________________
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-leave@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/ARI44O62CRMAF2IKPHJVLU5D2ADR2DP6/
Code of Conduct: http://python.org/psf/codeofconduct/
Re: Declarative imports [ In reply to ]
On Fri, Apr 8, 2022 at 1:26 AM Malthe <mborch@gmail.com> wrote:

> This is an idea which has been brought up before, sometimes introduced
> as "heresy". But an interesting twist has surfaced now which is
> typing.
>

What for? To save a few keystrokes?

Can't some IDE's add the import for you?

Please don't drag Python in the direction of line noise.
Re: Declarative imports [ In reply to ]
On Fri, 8 Apr 2022, 17:44 Guido van Rossum, <guido@python.org> wrote:

> The interesting idea here seems to make "lazy imports" easier to implement
> by making them explicit in the code. So far, most lazy import frameworks
> for Python have done hacks with `__getattribute__` overrides.
>

The value is more than ease of implementation. Having syntax for import
expressions makes them statically analysable, which is needed for type
checkers and IDE autocompletion.

But I also see value in being able to type out code that uses modules not
yet imported without breaking my flow to add an import statement. I don't
yet trust IDEs to do this because I've been bitten by them doing so
incorrectly in the past.

The key questions to me are
> - What should the notation be?
>

I would like to bid again for (import package.module) as an expression.
Instead of doing the import and assigning package to a variable package it
would evaluate to the module object package.module.

The `as` form is not needed because no name is assigned and the `from` form
isn't as valuable because you can just use attribute access afterwards.

It isn't terse but it does make use of the import keyword and is thus
instantly recognisable. It is even syntax highlighted correctly by much
existing software. If we're using the import keyword then I think it has to
look like this.

But I concede that it isn't particularly elegant to type hint things with

(import collections.abc).Mapping

...but not so inelegant that I couldn't see myself using it for a few
one-off imports per module.

A quirk is that it means there's a big difference between the statements

import foo

and

(import foo)

because one assigns a variable. I don't mind that; I don't think it is too
ambiguous to a reader.
Re: Declarative imports [ In reply to ]
On Sun, Apr 10, 2022 at 2:31 AM Daniel Pope <lord.mauve@gmail.com> wrote:

> On Fri, 8 Apr 2022, 17:44 Guido van Rossum, <guido@python.org> wrote:
>
>> The interesting idea here seems to make "lazy imports" easier to
>> implement by making them explicit in the code. So far, most lazy import
>> frameworks for Python have done hacks with `__getattribute__` overrides.
>>
>
> The value is more than ease of implementation. Having syntax for import
> expressions makes them statically analysable, which is needed for type
> checkers and IDE autocompletion.
>

This has been brought up a few times and I don't get it. Currently a use of
an imported module is perfectly analyzable by all the static type checkers
I know of (e.g. mypy, Pyre, pyright). For the static analyzer it makes no
difference if I have

import re
.
.
.
def foo(x):
if re.match(r"blah", x): ...

or the hypothetical inline form:

def foo(x):
if @re.match(r"blah", x): ...


> But I also see value in being able to type out code that uses modules not
> yet imported without breaking my flow to add an import statement. I don't
> yet trust IDEs to do this because I've been bitten by them doing so
> incorrectly in the past.
>

I have too.

--
--Guido van Rossum (python.org/~guido)
*Pronouns: he/him **(why is my pronoun here?)*
<http://feministing.com/2015/02/03/how-using-they-as-a-singular-pronoun-can-change-the-world/>

1 2  View All