Mailing List Archive

PEP 649: Deferred Evaluation Of Annotations Using Descriptors, round 2
Attached is my second draft of PEP 649.  The PEP and the prototype have
both seen a marked improvement since round 1 in January; PEP 649 now
allows annotations to refer to any variable they could see under stock
semantics:

* Local variables in the current function scope or in enclosing
function scopes become closures and use LOAD_DEFER.
* Class variables in the current class scope are made available using
a new mechanism, in which the class dict is attached to the bound
annotation function, then loaded into f_locals when the annotation
function is run.  Thus permitting LOAD_NAME opcodes to function
normally.


I look forward to your comments,


//arry/
Re: PEP 649: Deferred Evaluation Of Annotations Using Descriptors, round 2 [ In reply to ]
I like! I really appreciate the work you've put into this to get it
this far.

Questions and comments:


> PEP 563 also requires using ``eval()`` or ``typing.get_type_hints()``
> to examine annotations. Code updated to work with PEP 563 that calls
> ``eval()`` directly would have to be updated simply to remove the
> ``eval()`` call. Code using ``typing.get_type_hints()`` would
> continue to work unchanged, though future use of that function
> would become optional in most cases.

I think it is worth noting somewhere that string annotations are still
valid, and should still be evaluated if so.


> Because this PEP makes semantic changes to how annotations are
> evaluated, this PEP will be initially gated with a per-module
> ``from __future__ import co_annotations`` before it eventually
> becomes the default behavior.

Is it safe to assume that a module that does not import co_annotations,
but imports a module that does, will exhibit PEP 649 behavior when the
former accesses an annotation defined in the latter?


> * *Code that sets annotations on module or class attributes
> from inside any kind of flow control statement.* It's
> currently possible to set module and class attributes with
> annotations inside an ``if`` or ``try`` statement, and it works
> as one would expect. It's untenable to support this behavior
> when this PEP is active.

Is the following an example of the above?

@dataclass
class Foo:
if some_condition:
x: int
else:
x: float

If so, would the following still be valid?

if some_condition:
type_ = int
else:
type_ = float

@dataclass
class Foo:
x: type_


> * *Code in module or class scope that references or modifies the
> local* ``__annotations__`` *dict directly.* Currently, when
> setting annotations on module or class attributes, the generated
> code simply creates a local ``__annotations__`` dict, then sets
> mappings in it as needed. It's also possible for user code
> to directly modify this dict, though this doesn't seem like it's
> an intentional feature. Although it would be possible to support
> this after a fashion when this PEP was active, the semantics
> would likely be surprising and wouldn't make anyone happy.

I recognize the point you make later about its impact on static type
checkers. Setting that aside, I'm wondering about caes where
annotations can be dynamically generated, such as
dataclasses.make_dataclass(...). And, I could see reasons for
overwriting values in __annotations__, especially in the case where it
may be stored as a string and one wants to later affix its evaluated
value. These are considerations specific to runtime (dynamic) type
checking.

I wonder if it would make sense for each item in __annotations__ to be
evaluated separately on first access of each key, rather than all
__annotations__ on first access to the dict. Basically the dict would
act as a LazyDict. It could also provide the benefit of lessening the
expense of evaluating complex but otherwise unused annotations.


Paul


On Sun, 2021-04-11 at 18:55 -0700, Larry Hastings wrote:
>
> Attached is my second draft of PEP 649.  The PEP and the prototype
> have both seen a marked improvement since round 1 in January; PEP 649
> now allows annotations to refer to any variable they could see under
> stock semantics:
> * Local variables in the current function scope or in enclosing
> function scopes become closures and use LOAD_DEFER.
> * Class variables in the current class scope are made available using
> a new mechanism, in which the class dict is attached to the bound
> annotation function, then loaded into f_locals when the annotation
> function is run.  Thus permitting LOAD_NAME opcodes to function
> normally.
>
> I look forward to your comments,
>
> /arry
> _______________________________________________
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-leave@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-dev@python.org/message/QSASX6PZ3LIIFIANHQQFS752BJYFUFPY/
> Code of Conduct: http://python.org/psf/codeofconduct/
Re: PEP 649: Deferred Evaluation Of Annotations Using Descriptors, round 2 [ In reply to ]
On 4/11/21 7:55 PM, Paul Bryan wrote:
>> PEP 563 also requires using ``eval()`` or ``typing.get_type_hints()``
>> to examine annotations. Code updated to work with PEP 563 that calls
>> ``eval()`` directly would have to be updated simply to remove the
>> ``eval()`` call. Code using ``typing.get_type_hints()`` would
>> continue to work unchanged, though future use of that function
>> would become optional in most cases.
>
> I think it is worth noting somewhere that string annotations are still
> valid, and should still be evaluated if so.


That's not up to me, it's up to the static type checkers who created
that idiom.  But I assume they'll continue to support stringized
annotations, whether manually or automatically created.


>> Because this PEP makes semantic changes to how annotations are
>> evaluated, this PEP will be initially gated with a per-module
>> ``from __future__ import co_annotations`` before it eventually
>> becomes the default behavior.
>
> Is it safe to assume that a module that does not import
> co_annotations, but imports a module that does, will exhibit PEP 649
> behavior when the former accesses an annotation defined in the latter?

Yes.


>> * *Code that sets annotations on module or class attributes
>> from inside any kind of flow control statement.* It's
>> currently possible to set module and class attributes with
>> annotations inside an ``if`` or ``try`` statement, and it works
>> as one would expect. It's untenable to support this behavior
>> when this PEP is active.
>
> Is the following an example of the above?
>
> @dataclass
> class Foo:
> if some_condition:
> x: int
> else:
> x: float
> If so, would the following still be valid?
>
> if some_condition:
> type_ = int
> else:
> type_ = float
> @dataclass
> class Foo:
> x: type_

Your example was valid, and I think your workaround should be fine.  Do
you have a use case for this, or is this question motivated purely by
curiosity?


>> * *Code in module or class scope that references or modifies the
>> local* ``__annotations__`` *dict directly.* Currently, when
>> setting annotations on module or class attributes, the generated
>> code simply creates a local ``__annotations__`` dict, then sets
>> mappings in it as needed. It's also possible for user code
>> to directly modify this dict, though this doesn't seem like it's
>> an intentional feature. Although it would be possible to support
>> this after a fashion when this PEP was active, the semantics
>> would likely be surprising and wouldn't make anyone happy.
>
> I recognize the point you make later about its impact on static type
> checkers. Setting that aside, I'm wondering about caes where
> annotations can be dynamically generated, such as
> dataclasses.make_dataclass(...). And, I could see reasons for
> overwriting values in __annotations__, especially in the case where it
> may be stored as a string and one wants to later affix its evaluated
> value. These are considerations specific to runtime (dynamic) type
> checking.
It's fine to modify the __annotations__ dict after the creation of the
class or module.  It's code that modifies "__annotations__" from within
the class or module that is disallowed here.  Similarly for dataclasses;
once it creates a class object, it can explicitly set and / or modify
the annotations dict on that class.


> I wonder if it would make sense for each item in __annotations__ to be
> evaluated separately on first access /of each key/, rather than all
> __annotations__ on first access to the dict. Basically the dict would
> act as a LazyDict. It could also provide the benefit of lessening the
> expense of evaluating complex but otherwise unused annotations.

This would cause an immense proliferation of code objects (with some
pre-bound to function objects).  Rather than one code object per
annotation dict, it would create one code object per annotation key. 
Also, we don't have a "lazy dict" object built in to Python, so we'd
have to create one.

I don't have any problems that this would solve, so I'm not super
interested in it.  Personally I'd want to see a real compelling use case
for this feature before I'd consider adding it to Python.  Of course,
I'm not on the steering committee, so my opinion is only worth so much.


//arry/
Re: PEP 649: Deferred Evaluation Of Annotations Using Descriptors, round 2 [ In reply to ]
On Sun, 2021-04-11 at 23:34 -0700, Larry Hastings wrote:

> Your example was valid, and I think your workaround should be fine. 
> Do you have a use case for this, or is this question motivated purely
> by curiosity?

It took a few readings for me to understand the limitations in the PEP.
My example and workaround were mostly for me to confirm I had read it
correctly. 

> It's fine to modify the __annotations__ dict after the creation of
> the class or module.  It's code that modifies "__annotations__" from
> within the class or module that is disallowed here.  Similarly for
> dataclasses; once it creates a class object, it can explicitly set
> and / or modify the annotations dict on that class.

Thanks. I think this clarification should be added to the PEP.

Paul
Re: PEP 649: Deferred Evaluation Of Annotations Using Descriptors, round 2 [ In reply to ]
I'm a big fan of this PEP, for many reasons. But the fact that it
addresses some of the issues with get_type_hints() is very important.
dataclasses avoids calling get_type_hints() for performance reasons and
because it doesn't always succeed, see
https://github.com/python/typing/issues/508. I believe this issue is
fixed by PEP 649.

On 4/12/2021 2:34 AM, Larry Hastings wrote:
>
>
> On 4/11/21 7:55 PM, Paul Bryan wrote:
>> I recognize the point you make later about its impact on static type
>> checkers. Setting that aside, I'm wondering about caes where
>> annotations can be dynamically generated, such as
>> dataclasses.make_dataclass(...). And, I could see reasons for
>> overwriting values in __annotations__, especially in the case where
>> it may be stored as a string and one wants to later affix its
>> evaluated value. These are considerations specific to runtime
>> (dynamic) type checking.
> It's fine to modify the __annotations__ dict after the creation of the
> class or module.  It's code that modifies "__annotations__" from
> within the class or module that is disallowed here. Similarly for
> dataclasses; once it creates a class object, it can explicitly set and
> / or modify the annotations dict on that class.

There won't be any direct impact to make_dataclass(). It doesn't do
anything tricky here: it just builds up the annotations dictionary and
passes it as __annotations__ to the class namespace in
types.new_class(). After creating the class, it just applies the normal
dataclass() decorator.

Eric

_______________________________________________
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-leave@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/T4KPZKA57SPV2JQPEVZLYMPF45F2TFSG/
Code of Conduct: http://python.org/psf/codeofconduct/
Re: PEP 649: Deferred Evaluation Of Annotations Using Descriptors, round 2 [ In reply to ]
I still prefer PEP 563.
I will describe what we lost if PEP 597 is accepted and PEP 563 is rejected.

### Types not accessible in runtime

First of all, PEP 563 solves not only forward references.

Note that PEP 563 says: "we'll call any name imported or defined
within a `if TYPE_CHECKING: block` a forward reference, too."
https://www.python.org/dev/peps/pep-0563/#forward-references

PEP 563 solves all problems relating to types not accessible in runtime.
There are many reasons users can not get types used in annotations at runtime:

* To avoid circular import
* Types defined only in pyi files
* Optional dependency that is slow to import or hard to install

This is the most clear point where PEP 563 is better for some users.
See this example:

```
from dataclasses import dataclass

if 0:
from fakemod import FakeType

@dataclass
class C:
a : FakeType = 0
```

This works on PEP 563 semantics (Python 3.10a7). User can get
stringified annotation.

With stock semantics, it cause NameError when importing so author can
notice they need to quote "FakeType".

With PEP 649 semantics, author may not notice this annotation cause
error. User can not get any type hints at runtime.


### Type alias

Another PEP 563 benefit is user can see simple type alias.
Consider this example.

```
from typing import *

AliasType = Union[List[Dict[Tuple[int, str], Set[int]]], Tuple[str, List[str]]]

def f() -> AliasType:
pass

help(f)
```

Currently, help() calls `typing.get_type_hints()`. So it shows:

```
f() -> Union[List[Dict[Tuple[int, str], Set[int]]], Tuple[str, List[str]]]
```

But with PEP 563 semantics, we can stop evaluating annotations and
user can see more readable alias type.

```
f() -> AliasType
```

As PEP 597 says, eval() is slow. But it can avoidable in many cases
with PEP 563 semantics.
I am not sure but I expect dataclass can avoid eval() too in PEP 563 semantics.

Sphinx uses this feature already.
See https://www.sphinx-doc.org/en/master/usage/extensions/autodoc.html#confval-autodoc_type_aliases


### Relaxing annotation syntax

As discussed in PEP 647 thread, we can consider having different
syntax for annotation with PEP 597 semantics.


Regards,

On Mon, Apr 12, 2021 at 10:58 AM Larry Hastings <larry@hastings.org> wrote:
>
>
> Attached is my second draft of PEP 649. The PEP and the prototype have both seen a marked improvement since round 1 in January; PEP 649 now allows annotations to refer to any variable they could see under stock semantics:
>
> Local variables in the current function scope or in enclosing function scopes become closures and use LOAD_DEFER.
> Class variables in the current class scope are made available using a new mechanism, in which the class dict is attached to the bound annotation function, then loaded into f_locals when the annotation function is run. Thus permitting LOAD_NAME opcodes to function normally.
>
>
> I look forward to your comments,
>
>
> /arry
>
> _______________________________________________
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-leave@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/QSASX6PZ3LIIFIANHQQFS752BJYFUFPY/
> Code of Conduct: http://python.org/psf/codeofconduct/



--
Inada Naoki <songofacandy@gmail.com>
_______________________________________________
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-leave@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/KK5QG75RWSDBU4E36XAVSDPY5OUERA73/
Code of Conduct: http://python.org/psf/codeofconduct/
Re: PEP 649: Deferred Evaluation Of Annotations Using Descriptors, round 2 [ In reply to ]
On 4/12/21 4:50 PM, Inada Naoki wrote:
> As PEP 597 says, eval() is slow. But it can avoidable in many cases
> with PEP 563 semantics.

PEP 597 is "Add optional EncodingWarning".  You said PEP 597 in one
other place too.  Did you mean PEP 649 in both places?


Cheers,


//arry/
Re: PEP 649: Deferred Evaluation Of Annotations Using Descriptors, round 2 [ In reply to ]
On Tue, Apr 13, 2021 at 8:58 AM Larry Hastings <larry@hastings.org> wrote:
>
> On 4/12/21 4:50 PM, Inada Naoki wrote:
>
> As PEP 597 says, eval() is slow. But it can avoidable in many cases
> with PEP 563 semantics.
>
> PEP 597 is "Add optional EncodingWarning". You said PEP 597 in one other place too. Did you mean PEP 649 in both places?
>

You're right. I meant PEP 649 vs PEP 563. I'm sorry.

>
> Cheers,
>
>
> /arry



--
Inada Naoki <songofacandy@gmail.com>
_______________________________________________
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-leave@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/CLTMVQS7PMLEM237IAY4WCXC7M5DL7T6/
Code of Conduct: http://python.org/psf/codeofconduct/
Re: PEP 649: Deferred Evaluation Of Annotations Using Descriptors, round 2 [ In reply to ]
On 4/12/21 4:50 PM, Inada Naoki wrote:
> PEP 563 solves all problems relating to types not accessible in runtime.
> There are many reasons users can not get types used in annotations at runtime:
>
> * To avoid circular import
> * Types defined only in pyi files
> * Optional dependency that is slow to import or hard to install

It only "solves" these problems if you leave the annotation as a
string.  If PEP 563 is active, but you then use typing.get_type_hints()
to examine the actual Python value of the annotation, all of these
examples will fail with a NameError.  So, in this case, "solves the
problem" is a positive way of saying "hides a runtime error".

I don't know what the use cases are for examining type hints at runtime,
so I can't speak as to how convenient or inconvenient it is to deal with
them strictly as strings.  But it seems to me that examining annotations
as their actual Python values would be preferable.


> This is the most clear point where PEP 563 is better for some users.
> See this example:
>
> ```
> from dataclasses import dataclass
>
> if 0:
> from fakemod import FakeType
>
> @dataclass
> class C:
> a : FakeType = 0
> ```
>
> This works on PEP 563 semantics (Python 3.10a7). User can get
> stringified annotation.
>
> With stock semantics, it cause NameError when importing so author can
> notice they need to quote "FakeType".
>
> With PEP 649 semantics, author may not notice this annotation cause
> error. User can not get any type hints at runtime.

Again, by "works on PEP 563 semantics", you mean "doesn't raise an
error".  But the code /has/ an error.  It's just that it has been hidden
by PEP 563 semantics.

I don't agree that changing Python to automatically hide errors is an
improvement.  As the Zen says: "Errors should never pass silently."

This is really the heart of the debate over PEP 649 vs PEP 563. If you
examine an annotation, and it references an undefined symbol, should
that throw an error?  There is definitely a contingent of people who say
"no, that's inconvenient for us".  I think it should raise an error. 
Again from the Zen: "Special cases aren't special enough to break the
rules."  Annotations are expressions, and if evaluating an expression
fails because of an undefined name, it should raise a NameError.


> ### Type alias
>
> Another PEP 563 benefit is user can see simple type alias.
> Consider this example.
>
> ```
> from typing import *
>
> AliasType = Union[List[Dict[Tuple[int, str], Set[int]]], Tuple[str, List[str]]]
>
> def f() -> AliasType:
> pass
>
> help(f)
> ```
>
> Currently, help() calls `typing.get_type_hints()`. So it shows:
>
> ```
> f() -> Union[List[Dict[Tuple[int, str], Set[int]]], Tuple[str, List[str]]]
> ```
>
> But with PEP 563 semantics, we can stop evaluating annotations and
> user can see more readable alias type.
>
> ```
> f() -> AliasType
> ```

It's a matter of personal opinion whether "AliasType" or the full
definition is better here.  And it could lead to ambiguity, if the
programmer assigns to "AliasType" more than once.

Iif the programmer has a strong opinion that "AliasType" is better, they
could use an annotation of 'AliasType'--in quotes. Although I haven't
seen the topic discussed specifically, I assume that the static typing
analysis tools will continue to support manually stringized annotations
even if PEP 649 is accepted.

Either way, this hypothetical feature might be "nice-to-have", but I
don't think it's very important.  I would certainly forego this behavior
in favor of accepting PEP 649.


Cheers,


//arry/
Re: PEP 649: Deferred Evaluation Of Annotations Using Descriptors, round 2 [ In reply to ]
On Tue, Apr 13, 2021 at 9:57 AM Larry Hastings <larry@hastings.org> wrote:
>
>
> On 4/12/21 4:50 PM, Inada Naoki wrote:
>
> PEP 563 solves all problems relating to types not accessible in runtime.
> There are many reasons users can not get types used in annotations at runtime:
>
> * To avoid circular import
> * Types defined only in pyi files
> * Optional dependency that is slow to import or hard to install
>
> It only "solves" these problems if you leave the annotation as a string. If PEP 563 is active, but you then use typing.get_type_hints() to examine the actual Python value of the annotation, all of these examples will fail with a NameError. So, in this case, "solves the problem" is a positive way of saying "hides a runtime error".
>

Of course, "get type which is unavailable in runtime" is unsolvable
problem. PEP 597 doesn't solve it too. Author needs to quote the hint
manually, and `typing.get_type_hints()` raises NameError too.
And if author forget to quote, user can not get any type hints.


> I don't know what the use cases are for examining type hints at runtime, so I can't speak as to how convenient or inconvenient it is to deal with them strictly as strings. But it seems to me that examining annotations as their actual Python values would be preferable.
>

This is use cases for examining type hints at runtime and stringified
hints are OK.

* Sphinx autodoc
* help()
* IPython and other REPLS showing type hint in popup.


>
> ```
> from dataclasses import dataclass
>
> if 0:
> from fakemod import FakeType
>
> @dataclass
> class C:
> a : FakeType = 0
> ```
>
> This works on PEP 563 semantics (Python 3.10a7). User can get
> stringified annotation.
>
> With stock semantics, it cause NameError when importing so author can
> notice they need to quote "FakeType".
>
> With PEP 649 semantics, author may not notice this annotation cause
> error. User can not get any type hints at runtime.
>
> Again, by "works on PEP 563 semantics", you mean "doesn't raise an error". But the code has an error. It's just that it has been hidden by PEP 563 semantics.
>
> I don't agree that changing Python to automatically hide errors is an improvement. As the Zen says: "Errors should never pass silently."
>
> This is really the heart of the debate over PEP 649 vs PEP 563. If you examine an annotation, and it references an undefined symbol, should that throw an error? There is definitely a contingent of people who say "no, that's inconvenient for us". I think it should raise an error. Again from the Zen: "Special cases aren't special enough to break the rules." Annotations are expressions, and if evaluating an expression fails because of an undefined name, it should raise a NameError.
>

I agree that this is the heart of the debate. I think "annotations are
for type hitns". They are for:

* Static type checkers
* document.

So I don't think `if TYPE_CHECKING` idiom is violating Python Zen.


Regards,

--
Inada Naoki <songofacandy@gmail.com>
_______________________________________________
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-leave@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/DAMYYG5CY6MGRCAKIWRA4IKXW75DYXW6/
Code of Conduct: http://python.org/psf/codeofconduct/
Re: PEP 649: Deferred Evaluation Of Annotations Using Descriptors, round 2 [ In reply to ]
On Tue, 2021-04-13 at 10:47 +0900, Inada Naoki wrote:

> On Tue, Apr 13, 2021 at 9:57 AM Larry Hastings <larry@hastings.org>
> wrote:

> > This is really the heart of the debate over PEP 649 vs PEP 563.  If
> > you examine an annotation, and it references an undefined symbol,
> > should that throw an error?  There is definitely a contingent of
> > people who say "no, that's inconvenient for us".  I think it should
> > raise an error.  Again from the Zen: "Special cases aren't special
> > enough to break the rules."  Annotations are expressions, and if
> > evaluating an expression fails because of an undefined name, it
> > should raise a NameError.

> I agree that this is the heart of the debate. I think "annotations
> are
> for type hitns". They are for:
>
> * Static type checkers
> * document.

+ dynamic type validation, encoding and decoding (Pydantic, FastAPI,
Fondat, et al.)

Paul
Re: PEP 649: Deferred Evaluation Of Annotations Using Descriptors, round 2 [ In reply to ]
I've been thinking about this a bit, and I think that the way forward is
for Python to ignore the text of annotations ("relaxed annotation syntax"),
not to try and make it available as an expression.

To be honest, the most pressing issue with annotations is the clumsy way
that type variables have to be introduced. The current convention, `T =
TypeVar('T')`, is both verbose (why do I have to repeat the name?) and
widely misunderstood (many help request for mypy and pyright follow from
users making a mistaken association between two type variables that are
unrelated but share the same TypeVar definition). And relaxed annotation
syntax alone doesn't solve this.

Nevertheless I think that it's time to accept that annotations are for
types -- the intention of PEP 3107 was to experiment with different syntax
and semantics for types, and that experiment has resulted in the successful
adoption of a specific syntax for types that is wildly successful.





On Sun, Apr 11, 2021 at 7:02 PM Larry Hastings <larry@hastings.org> wrote:

>
> Attached is my second draft of PEP 649. The PEP and the prototype have
> both seen a marked improvement since round 1 in January; PEP 649 now allows
> annotations to refer to any variable they could see under stock semantics:
>
> - Local variables in the current function scope or in enclosing
> function scopes become closures and use LOAD_DEFER.
> - Class variables in the current class scope are made available using
> a new mechanism, in which the class dict is attached to the bound
> annotation function, then loaded into f_locals when the annotation function
> is run. Thus permitting LOAD_NAME opcodes to function normally.
>
>
> I look forward to your comments,
>
>
> */arry*
> _______________________________________________
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-leave@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-dev@python.org/message/QSASX6PZ3LIIFIANHQQFS752BJYFUFPY/
> Code of Conduct: http://python.org/psf/codeofconduct/
>


--
--Guido van Rossum (python.org/~guido)
*Pronouns: he/him **(why is my pronoun here?)*
<http://feministing.com/2015/02/03/how-using-they-as-a-singular-pronoun-can-change-the-world/>
Re: PEP 649: Deferred Evaluation Of Annotations Using Descriptors, round 2 [ In reply to ]
On Tue, Apr 13, 2021 at 11:18 AM Paul Bryan <pbryan@anode.ca> wrote:
>
> On Tue, 2021-04-13 at 10:47 +0900, Inada Naoki wrote:
>
> On Tue, Apr 13, 2021 at 9:57 AM Larry Hastings <larry@hastings.org> wrote:
>
>
> This is really the heart of the debate over PEP 649 vs PEP 563. If you examine an annotation, and it references an undefined symbol, should that throw an error? There is definitely a contingent of people who say "no, that's inconvenient for us". I think it should raise an error. Again from the Zen: "Special cases aren't special enough to break the rules." Annotations are expressions, and if evaluating an expression fails because of an undefined name, it should raise a NameError.
>
>
> I agree that this is the heart of the debate. I think "annotations are
> for type hitns". They are for:
>
> * Static type checkers
> * document.
>
>
> + dynamic type validation, encoding and decoding (Pydantic, FastAPI, Fondat, et al.)
>
> Paul
>

OK. It is important use case too.

Such use cases doesn't justify raising NameError instead of getting
stringified type hints for documents for document use cases.

On the other hand, if "dynamic type" is used heavily, eval()
performance can be a problem.

--
Inada Naoki <songofacandy@gmail.com>
_______________________________________________
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-leave@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/ICKGBL674MUOYBOGNGKKJDPHYD3TYGER/
Code of Conduct: http://python.org/psf/codeofconduct/
Re: PEP 649: Deferred Evaluation Of Annotations Using Descriptors, round 2 [ In reply to ]
On Tue, 2021-04-13 at 11:33 +0900, Inada Naoki wrote:
> On Tue, Apr 13, 2021 at 11:18 AM Paul Bryan <pbryan@anode.ca> wrote:
> >
> > On Tue, 2021-04-13 at 10:47 +0900, Inada Naoki wrote:
> >
> > On Tue, Apr 13, 2021 at 9:57 AM Larry Hastings <larry@hastings.org>
> > wrote:
> >
> >
> > This is really the heart of the debate over PEP 649 vs PEP 563.  If
> > you examine an annotation, and it references an undefined symbol,
> > should that throw an error?  There is definitely a contingent of
> > people who say "no, that's inconvenient for us".  I think it should
> > raise an error.  Again from the Zen: "Special cases aren't special
> > enough to break the rules."  Annotations are expressions, and if
> > evaluating an expression fails because of an undefined name, it
> > should raise a NameError.
> >
> >
> > I agree that this is the heart of the debate. I think "annotations
> > are
> > for type hitns". They are for:
> >
> > * Static type checkers
> > * document.
> >
> >
> > + dynamic type validation, encoding and decoding (Pydantic,
> > FastAPI, Fondat, et al.)
> >
> > Paul
> >
>
> OK. It is important use case too.
>
> Such use cases doesn't justify raising NameError instead of getting
> stringified type hints for documents for document use cases.
>
> On the other hand, if "dynamic type" is used heavily, eval()
> performance can be a problem.


In 3.9 this cost is paid once when a type is defined. However, in 3.10,
it gets expensive, because when the string is evaluated by
get_type_hints, its result is not stored/cached anywhere (repeated
calls to get_type_hints results in repeated evaluation). As a
workaround, I have code to "affix" the evaluated expression in
__annotations__ value. PEP 649 would resolve this and eliminate the
need for such a hack.

Paul
Re: PEP 649: Deferred Evaluation Of Annotations Using Descriptors, round 2 [ In reply to ]
On Mon, Apr 12, 2021 at 7:47 PM Paul Bryan <pbryan@anode.ca> wrote:

> In 3.9 this cost is paid once when a type is defined. However, in 3.10, it
> gets expensive, because when the string is evaluated by get_type_hints, its
> result is not stored/cached anywhere (repeated calls to get_type_hints
> results in repeated evaluation). As a workaround, I have code to "affix"
> the evaluated expression in __annotations__ value. PEP 649 would resolve
> this and eliminate the need for such a hack.
>

Why not submit a PR that adds caching to get_type_hints(), rather than
promote a paradigm shift?

--
--Guido van Rossum (python.org/~guido)
*Pronouns: he/him **(why is my pronoun here?)*
<http://feministing.com/2015/02/03/how-using-they-as-a-singular-pronoun-can-change-the-world/>
Re: PEP 649: Deferred Evaluation Of Annotations Using Descriptors, round 2 [ In reply to ]
On Mon, 2021-04-12 at 19:52 -0700, Guido van Rossum wrote:

> Why not submit a PR that adds caching to get_type_hints(), rather
> than promote a paradigm shift?

A couple of reasons:

1. In reviewing the code, I didn't find an obvious way to store cached
values. Anything but a non-trivial change would suggest the need for a
PEP of its own to document new behavior.

2. I've been hoping that PEP 649 would be adopted, making such a hack
or any plan to cache type hints moot.

Paul
Re: PEP 649: Deferred Evaluation Of Annotations Using Descriptors, round 2 [ In reply to ]
Hi,

Le 13/04/2021 à 04:24, Guido van Rossum a écrit :
> I've been thinking about this a bit, and I think that the way forward is
> for Python to ignore the text of annotations ("relaxed annotation
> syntax"), not to try and make it available as an expression.

Then, what's wrong with quoting? It's just 2 characters, and prevent the
user (or their IDE) from trying to parse them as Python syntax.

As a comparison: docstrings do get quoting, even though they also have
special semantics in the language.

Cheers,
Baptiste
_______________________________________________
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-leave@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/V66L56SNJRUSM7CIJVCJDIPNHNZTGMWJ/
Code of Conduct: http://python.org/psf/codeofconduct/
Re: PEP 649: Deferred Evaluation Of Annotations Using Descriptors, round 2 [ In reply to ]
Hi,

Le 12/04/2021 à 03:55, Larry Hastings a écrit :
>
> I look forward to your comments,

2 reading notes:

* in section "Annotations That Refer To Class Variables":

> If it's possible that an annotation function refers
> to class variables--if all these conditions are true:
>
> * The annotation function is being defined inside
> a class scope.
> * The generated code for the annotation function
> has at least one ``LOAD_NAME`` instruction.

I'm afraid I don't really understand the second condition. Would it be
possible to rephrase it in a less technical way, i.e. some condition on
the user code itself, not on what the implementation does with it.

* in section "Interactive REPL Shell":

> For the sake of simplicity, in this case we forego delayed evaluation.

This has the unpleasant consequence that any code using forward
references cannot be copy-pasted into the REPL. While such copy-pasting
is a very casual practice and does already often break, it is sometimes
useful in quick'n dirty prototyping. Would it be possible to specify
that in this case, a possible NameError in evaluation is caught, and the
annotation is set to None or some sentinel value?

Cheers,
Baptiste
_______________________________________________
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-leave@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/IPIJKJ6F4D634XT5ZXIKLD574QNWS2YB/
Code of Conduct: http://python.org/psf/codeofconduct/
Re: PEP 649: Deferred Evaluation Of Annotations Using Descriptors, round 2 [ In reply to ]
Hi Larry,

?On 4/12/21, 6:57 PM, "Larry Hastings" <larry@midwinter.com on behalf of larry@hastings.org> wrote:
Again, by "works on PEP 563 semantics", you mean "doesn't raise an error". But the code has an error. It's just that it has been hidden by PEP 563 semantics.
I don't agree that changing Python to automatically hide errors is an improvement. As the Zen says: "Errors should never pass silently."

This is really the heart of the debate over PEP 649 vs PEP 563. If you examine an annotation, and it references an undefined symbol, should that throw an error? There is definitely a contingent of people who say "no, that's inconvenient for us". I think it should raise an error. Again from the Zen: "Special cases aren't special enough to break the rules." Annotations are expressions, and if evaluating an expression fails because of an undefined name, it should raise a NameError.

Normally in Python, if you reference a symbol in a function definition line, the symbol must be defined at that point in module execution. Forward references are not permitted, and will raise `NameError`.

And yet you have implemented PEP 649, whose entire raison d'être is to implement a "special case" to "break the rules" by delaying evaluation of annotations such that a type annotation, unlike any other expression in the function definition line, may include forward reference names which will not be defined until later in the module.

The use case for `if TYPE_CHECKING` imports is effectively the same. They are just forward references to names in other modules which can't be imported eagerly, because e.g. it would cause a cycle. Those who have used type annotations in earnest are likely to confirm that such inter-module forward references are just as necessary as intra-module forward references for the usability of type annotations.

So it doesn't seem that we have here is a firm stand on principle of the Zen, it appears to rather be a disagreement about exactly where to draw the line on the "special case" that we all already seem to agree is needed.

The Zen argument seems to be a bit of a circular one: I have defined PEP 649 semantics in precisely this way, therefore code that works with PEP 649 does not have an error, and code that does not work with PEP 649 "has an error" which must be surfaced!

With PEP 563, although `get_type_hints()` cannot natively resolve inter-module forward references and raises `NameError`, it is possible to work around this by supplying a globals dict to `get_type_hints()` that has been augmented with those forward-referenced names. Under the current version of PEP 649, it becomes impossible to get access to such type annotations at runtime at all, without reverting to manually stringifying the annotation and then using something like `get_type_hints()`. So for users of type annotations who need `if TYPE_CHECKING` (which I think is most users of type annotations), the best-case overall effect of PEP 649 will be that a) some type annotations have to go back to being ugly strings in the source, and b) if type annotation values are needed at runtime, `get_type_hints()` will still be as necessary as it ever was.

It is possible for PEP 649 to draw the line differently and support both intra-module and inter-module forward references in annotations, by doing something like https://github.com/larryhastings/co_annotations/pull/3 and replacing unknown names with forward-reference markers, so the annotation values are still accessible at runtime. This meets the real needs of users of type annotations better, and gives up none of the benefits of PEP 649.

Carl



_______________________________________________
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-leave@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/DSZFE7XTRK2ESRJDPQPZIDP2I67E76WH/
Code of Conduct: http://python.org/psf/codeofconduct/
Re: PEP 649: Deferred Evaluation Of Annotations Using Descriptors, round 2 [ In reply to ]
On Tue, Apr 13, 2021 at 9:39 AM Baptiste Carvello <
devel2021@baptiste-carvello.net> wrote:

> Le 13/04/2021 à 04:24, Guido van Rossum a écrit :
> > I've been thinking about this a bit, and I think that the way forward is
> > for Python to ignore the text of annotations ("relaxed annotation
> > syntax"), not to try and make it available as an expression.
>
> Then, what's wrong with quoting? It's just 2 characters, and prevent the
> user (or their IDE) from trying to parse them as Python syntax.
>

Informal user research has shown high resistance to quoting.


> As a comparison: docstrings do get quoting, even though they also have
> special semantics in the language.
>

Not the same thing. Docstrings use English, which has no formal (enough)
syntax. The idea for annotations is that they *do* have a formal syntax, it
just evolves separately from that of Python itself.

--
--Guido van Rossum (python.org/~guido)
*Pronouns: he/him **(why is my pronoun here?)*
<http://feministing.com/2015/02/03/how-using-they-as-a-singular-pronoun-can-change-the-world/>
Re: PEP 649: Deferred Evaluation Of Annotations Using Descriptors, round 2 [ In reply to ]
On 4/12/21 7:24 PM, Guido van Rossum wrote:
> I've been thinking about this a bit, and I think that the way forward
> is for Python to ignore the text of annotations ("relaxed annotation
> syntax"), not to try and make it available as an expression.
>
> To be honest, the most pressing issue with annotations is the clumsy
> way that type variables have to be introduced. The current convention,
> `T = TypeVar('T')`, is both verbose (why do I have to repeat the
> name?) and widely misunderstood (many help request for mypy and
> pyright follow from users making a mistaken association between two
> type variables that are unrelated but share the same TypeVar
> definition). And relaxed annotation syntax alone doesn't solve this.
>
> Nevertheless I think that it's time to accept that annotations are for
> types -- the intention of PEP 3107 was to experiment with different
> syntax and semantics for types, and that experiment has resulted in
> the successful adoption of a specific syntax for types that is wildly
> successful.


I don't follow your reasoning.  I'm glad that type hints have found
success, but I don't see why that implies "and therefore we should
restrict the use of annotations solely for type hints". Annotations are
a useful, general-purpose feature of Python, with legitimate uses
besides type hints.  Why would it make Python better to restrict their
use now?


//arry/
Re: PEP 649: Deferred Evaluation Of Annotations Using Descriptors, round 2 [ In reply to ]
On Tue, Apr 13, 2021 at 12:32 PM Larry Hastings <larry@hastings.org> wrote:

>
> On 4/12/21 7:24 PM, Guido van Rossum wrote:
>
> I've been thinking about this a bit, and I think that the way forward is
> for Python to ignore the text of annotations ("relaxed annotation syntax"),
> not to try and make it available as an expression.
>
> To be honest, the most pressing issue with annotations is the clumsy way
> that type variables have to be introduced. The current convention, `T =
> TypeVar('T')`, is both verbose (why do I have to repeat the name?) and
> widely misunderstood (many help request for mypy and pyright follow from
> users making a mistaken association between two type variables that are
> unrelated but share the same TypeVar definition). And relaxed annotation
> syntax alone doesn't solve this.
>
> Nevertheless I think that it's time to accept that annotations are for
> types -- the intention of PEP 3107 was to experiment with different syntax
> and semantics for types, and that experiment has resulted in the successful
> adoption of a specific syntax for types that is wildly successful.
>
>
> I don't follow your reasoning. I'm glad that type hints have found
> success, but I don't see why that implies "and therefore we should restrict
> the use of annotations solely for type hints". Annotations are a useful,
> general-purpose feature of Python, with legitimate uses besides type
> hints. Why would it make Python better to restrict their use now?
>

Because typing is, to many folks, a Really Important Concept, and it's
confusing to use the same syntax ("x: blah blah") for different purposes,
in a way that makes it hard to tell whether a particular "blah blah" is
meant as a type or as something else -- because you have to know what's
introspecting the annotations before you can tell. And that introspection
could be signalled by a magical decorator, but it could also be implicit:
maybe you have a driver that calls a function based on a CLI entry point
name, and introspects that function even if it's not decorated.

OTOH, not requiring that annotations are syntactically valid expressions
might liberate such a CLI library too: you could write things like

def foo(prec: --precision int):
...


--
--Guido van Rossum (python.org/~guido)
*Pronouns: he/him **(why is my pronoun here?)*
<http://feministing.com/2015/02/03/how-using-they-as-a-singular-pronoun-can-change-the-world/>
Re: PEP 649: Deferred Evaluation Of Annotations Using Descriptors, round 2 [ In reply to ]
On 4/13/2021 4:21 AM, Baptiste Carvello wrote:
> Le 12/04/2021 à 03:55, Larry Hastings a écrit :

> * in section "Interactive REPL Shell":
>
>> For the sake of simplicity, in this case we forego delayed evaluation.

The intention of the code + codeop modules is that people should be able
to write interactive consoles that simulate the standard REPL. For example:

Python 3.10.0a7+ (heads/master-dirty:a9cf69df2e, Apr 12 2021, 15:36:39)
[MSC v.1900 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import code
>>> code.interact()
Python 3.10.0a7+ (heads/master-dirty:a9cf69df2e, Apr 12 2021, 15:36:39)
[MSC v.1900 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
(InteractiveConsole)
>>> # Call has not returned. Prompt is from code.InteractiveConsole.
>>> def f(x:int): -> float

>>> f.__annotations__ # should match REPL result

>>> ^Z

now exiting InteractiveConsole...
>>> Now back to repl

If the REPL compiles with "mode='single' and spec is changes to "when
mode is 'single'", then above should work. Larry, please test with your
proposed implementation.

--
Terry Jan Reedy


_______________________________________________
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-leave@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/KDFTZQMJRBRDBSUSVTMBDXTLCXIZXBFP/
Code of Conduct: http://python.org/psf/codeofconduct/
Re: PEP 649: Deferred Evaluation Of Annotations Using Descriptors, round 2 [ In reply to ]
On 4/13/21 3:28 PM, Terry Reedy wrote:
> On 4/13/2021 4:21 AM, Baptiste Carvello wrote:
>> Le 12/04/2021 à 03:55, Larry Hastings a écrit :
>
>> * in section "Interactive REPL Shell":
>>
>>> For the sake of simplicity, in this case we forego delayed evaluation.
>
> The intention of the code + codeop modules is that people should be
> able to write interactive consoles that simulate the standard REPL. 
> For example:
>
> Python 3.10.0a7+ (heads/master-dirty:a9cf69df2e, Apr 12 2021,
> 15:36:39) [MSC v.1900 64 bit (AMD64)] on win32
> Type "help", "copyright", "credits" or "license" for more information.
> >>> import code
> >>> code.interact()
> Python 3.10.0a7+ (heads/master-dirty:a9cf69df2e, Apr 12 2021,
> 15:36:39) [MSC v.1900 64 bit (AMD64)] on win32
> Type "help", "copyright", "credits" or "license" for more information.
> (InteractiveConsole)
> >>> # Call has not returned.  Prompt is from code.InteractiveConsole.
> >>> def f(x:int): -> float
>
> >>> f.__annotations__ # should match REPL result
>
> >>> ^Z
>
> now exiting InteractiveConsole...
> >>> Now back to repl
>
> If the REPL compiles with "mode='single' and spec is changes to "when
> mode is 'single'", then above should work.  Larry, please test with
> your proposed implementation.


A couple things!

1. I apologize if the PEP wasn't clear, but this section was talking
about the problem of /module/ annotations in the implicit __main__
module when using the interactive REPL. Annotations on other objects
(classes, functions, etc) defined in the interactive REPL work as
expected.
2. The above example has a minor bug: when defining a return annotation
on a function, the colon ending the function declaration goes
/after/ the return annotation.  It should have been "def f(x:int) ->
float:".
3. The above example works fine when run in my branch.
4. You need to "from __future__ import co_annotations" in order to
activate delayed evaluation of annotations using code objects in my
branch.  I added that (inside the code.interact() shell!) and it
still works fine.

So I'm not sure what problem you're proposing to solve with this "mode
is single" stuff.

* If you thought there was a problem with defining annotations on
functions and classes defined in the REPL, good news!, it was never
a problem.
* If you're solving the problem of defining annotations on the
interactive module /itself,/ I don't understand what your proposed
solution is or how it would work.  The problem is, how do you create
a code object that defines all the annotations on a module, when the
module never finishes being defined because it's the interactive shell?


Cheers,


//arry/
Re: PEP 649: Deferred Evaluation Of Annotations Using Descriptors, round 2 [ In reply to ]
On 4/13/21 1:52 PM, Guido van Rossum wrote:
> On Tue, Apr 13, 2021 at 12:32 PM Larry Hastings <larry@hastings.org
> <mailto:larry@hastings.org>> wrote:
>
>
> On 4/12/21 7:24 PM, Guido van Rossum wrote:
>> I've been thinking about this a bit, and I think that the way
>> forward is for Python to ignore the text of annotations ("relaxed
>> annotation syntax"), not to try and make it available as an
>> expression.
>>
>> To be honest, the most pressing issue with annotations is the
>> clumsy way that type variables have to be introduced. The current
>> convention, `T = TypeVar('T')`, is both verbose (why do I have to
>> repeat the name?) and widely misunderstood (many help request for
>> mypy and pyright follow from users making a mistaken association
>> between two type variables that are unrelated but share the same
>> TypeVar definition). And relaxed annotation syntax alone doesn't
>> solve this.
>>
>> Nevertheless I think that it's time to accept that annotations
>> are for types -- the intention of PEP 3107 was to experiment with
>> different syntax and semantics for types, and that experiment has
>> resulted in the successful adoption of a specific syntax for
>> types that is wildly successful.
>
>
> I don't follow your reasoning.  I'm glad that type hints have
> found success, but I don't see why that implies "and therefore we
> should restrict the use of annotations solely for type hints". 
> Annotations are a useful, general-purpose feature of Python, with
> legitimate uses besides type hints.  Why would it make Python
> better to restrict their use now?
>
>
> Because typing is, to many folks, a Really Important Concept, and it's
> confusing to use the same syntax ("x: blah blah") for different
> purposes, in a way that makes it hard to tell whether a particular
> "blah blah" is meant as a type or as something else -- because you
> have to know what's introspecting the annotations before you can tell.
> And that introspection could be signalled by a magical decorator, but
> it could also be implicit: maybe you have a driver that calls a
> function based on a CLI entry point name, and introspects that
> function even if it's not decorated.


I'm not sure I understand your point.  Are you saying that we need to
take away the general-purpose functionality of annotations, that's been
in the language since 3.0, and restrict annotations to just type
hints... because otherwise an annotation might not be used for a type
hint, and then the programmer would have to figure out what it means? 
We need to take away the functionality from all other use cases in order
to lend /clarity/ to one use case?

Also, if you're stating that programmers get confused reading source
code because annotations get used for different things at different
places--surely that confirms that annotations are /useful/ for more than
just type hints, in real-world code, today.  I genuinely have no sense
of how important static type analysis is in Python--personally I have no
need for it--but I find it hard to believe that type hints are so
overwhelmingly important that they should become the sole use case for
annotations, and we need to take away this long-standing functionality,
that you suggest is being successfully used side-by-side with type hints
today, merely to make type hints clearer.


Cheers,


//arry/

1 2 3  View All