Mailing List Archive

I like! I really appreciate the work you've put into this to get it
this far.

Questions and comments:


> PEP 563 also requires using ``eval()`` or ``typing.get_type_hints()``
> to examine annotations. Code updated to work with PEP 563 that calls
> ``eval()`` directly would have to be updated simply to remove the
> ``eval()`` call. Code using ``typing.get_type_hints()`` would
> continue to work unchanged, though future use of that function
> would become optional in most cases.

I think it is worth noting somewhere that string annotations are still
valid, and should still be evaluated if so.


> Because this PEP makes semantic changes to how annotations are
> evaluated, this PEP will be initially gated with a per-module
> ``from __future__ import co_annotations`` before it eventually
> becomes the default behavior.

Is it safe to assume that a module that does not import co_annotations,
but imports a module that does, will exhibit PEP 649 behavior when the
former accesses an annotation defined in the latter?


> * *Code that sets annotations on module or class attributes
> from inside any kind of flow control statement.* It's
> currently possible to set module and class attributes with
> annotations inside an ``if`` or ``try`` statement, and it works
> as one would expect. It's untenable to support this behavior
> when this PEP is active.

Is the following an example of the above?

@dataclass
class Foo:
if some_condition:
x: int
else:
x: float

If so, would the following still be valid?

if some_condition:
type_ = int
else:
type_ = float

@dataclass
class Foo:
x: type_


> * *Code in module or class scope that references or modifies the
> local* ``__annotations__`` *dict directly.* Currently, when
> setting annotations on module or class attributes, the generated
> code simply creates a local ``__annotations__`` dict, then sets
> mappings in it as needed. It's also possible for user code
> to directly modify this dict, though this doesn't seem like it's
> an intentional feature. Although it would be possible to support
> this after a fashion when this PEP was active, the semantics
> would likely be surprising and wouldn't make anyone happy.

I recognize the point you make later about its impact on static type
checkers. Setting that aside, I'm wondering about caes where
annotations can be dynamically generated, such as
dataclasses.make_dataclass(...). And, I could see reasons for
overwriting values in __annotations__, especially in the case where it
may be stored as a string and one wants to later affix its evaluated
value. These are considerations specific to runtime (dynamic) type
checking.

I wonder if it would make sense for each item in __annotations__ to be
evaluated separately on first access of each key, rather than all
__annotations__ on first access to the dict. Basically the dict would
act as a LazyDict. It could also provide the benefit of lessening the
expense of evaluating complex but otherwise unused annotations.


Paul


On Sun, 2021-04-11 at 18:55 -0700, Larry Hastings wrote:
>
> Attached is my second draft of PEP 649.  The PEP and the prototype
> have both seen a marked improvement since round 1 in January; PEP 649
> now allows annotations to refer to any variable they could see under
> stock semantics:
> * Local variables in the current function scope or in enclosing
> function scopes become closures and use LOAD_DEFER.
> * Class variables in the current class scope are made available using
> a new mechanism, in which the class dict is attached to the bound
> annotation function, then loaded into f_locals when the annotation
> function is run.  Thus permitting LOAD_NAME opcodes to function
> normally.
>
> I look forward to your comments,
>
> /arry
> _______________________________________________
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-leave@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-dev@python.org/message/QSASX6PZ3LIIFIANHQQFS752BJYFUFPY/
> Code of Conduct: http://python.org/psf/codeofconduct/

On 4/11/21 7:55 PM, Paul Bryan wrote:
>> PEP 563 also requires using ``eval()`` or ``typing.get_type_hints()``
>> to examine annotations. Code updated to work with PEP 563 that calls
>> ``eval()`` directly would have to be updated simply to remove the
>> ``eval()`` call. Code using ``typing.get_type_hints()`` would
>> continue to work unchanged, though future use of that function
>> would become optional in most cases.
>
> I think it is worth noting somewhere that string annotations are still
> valid, and should still be evaluated if so.


That's not up to me, it's up to the static type checkers who created
that idiom.  But I assume they'll continue to support stringized
annotations, whether manually or automatically created.


>> Because this PEP makes semantic changes to how annotations are
>> evaluated, this PEP will be initially gated with a per-module
>> ``from __future__ import co_annotations`` before it eventually
>> becomes the default behavior.
>
> Is it safe to assume that a module that does not import
> co_annotations, but imports a module that does, will exhibit PEP 649
> behavior when the former accesses an annotation defined in the latter?

Yes.


>> * *Code that sets annotations on module or class attributes
>> from inside any kind of flow control statement.* It's
>> currently possible to set module and class attributes with
>> annotations inside an ``if`` or ``try`` statement, and it works
>> as one would expect. It's untenable to support this behavior
>> when this PEP is active.
>
> Is the following an example of the above?
>
> @dataclass
> class Foo:
> if some_condition:
> x: int
> else:
> x: float
> If so, would the following still be valid?
>
> if some_condition:
> type_ = int
> else:
> type_ = float
> @dataclass
> class Foo:
> x: type_

Your example was valid, and I think your workaround should be fine.  Do
you have a use case for this, or is this question motivated purely by
curiosity?


>> * *Code in module or class scope that references or modifies the
>> local* ``__annotations__`` *dict directly.* Currently, when
>> setting annotations on module or class attributes, the generated
>> code simply creates a local ``__annotations__`` dict, then sets
>> mappings in it as needed. It's also possible for user code
>> to directly modify this dict, though this doesn't seem like it's
>> an intentional feature. Although it would be possible to support
>> this after a fashion when this PEP was active, the semantics
>> would likely be surprising and wouldn't make anyone happy.
>
> I recognize the point you make later about its impact on static type
> checkers. Setting that aside, I'm wondering about caes where
> annotations can be dynamically generated, such as
> dataclasses.make_dataclass(...). And, I could see reasons for
> overwriting values in __annotations__, especially in the case where it
> may be stored as a string and one wants to later affix its evaluated
> value. These are considerations specific to runtime (dynamic) type
> checking.
It's fine to modify the __annotations__ dict after the creation of the
class or module.  It's code that modifies "__annotations__" from within
the class or module that is disallowed here.  Similarly for dataclasses;
once it creates a class object, it can explicitly set and / or modify
the annotations dict on that class.


> I wonder if it would make sense for each item in __annotations__ to be
> evaluated separately on first access /of each key/, rather than all
> __annotations__ on first access to the dict. Basically the dict would
> act as a LazyDict. It could also provide the benefit of lessening the
> expense of evaluating complex but otherwise unused annotations.

This would cause an immense proliferation of code objects (with some
pre-bound to function objects).  Rather than one code object per
annotation dict, it would create one code object per annotation key. 
Also, we don't have a "lazy dict" object built in to Python, so we'd
have to create one.

I don't have any problems that this would solve, so I'm not super
interested in it.  Personally I'd want to see a real compelling use case
for this feature before I'd consider adding it to Python.  Of course,
I'm not on the steering committee, so my opinion is only worth so much.


//arry/

On Sun, 2021-04-11 at 23:34 -0700, Larry Hastings wrote:

> Your example was valid, and I think your workaround should be fine. 
> Do you have a use case for this, or is this question motivated purely
> by curiosity?

It took a few readings for me to understand the limitations in the PEP.
My example and workaround were mostly for me to confirm I had read it
correctly. 

> It's fine to modify the __annotations__ dict after the creation of
> the class or module.  It's code that modifies "__annotations__" from
> within the class or module that is disallowed here.  Similarly for
> dataclasses; once it creates a class object, it can explicitly set
> and / or modify the annotations dict on that class.

Thanks. I think this clarification should be added to the PEP.

Paul

I'm a big fan of this PEP, for many reasons. But the fact that it
addresses some of the issues with get_type_hints() is very important.
dataclasses avoids calling get_type_hints() for performance reasons and
because it doesn't always succeed, see
https://github.com/python/typing/issues/508. I believe this issue is
fixed by PEP 649.

On 4/12/2021 2:34 AM, Larry Hastings wrote:
>
>
> On 4/11/21 7:55 PM, Paul Bryan wrote:
>> I recognize the point you make later about its impact on static type
>> checkers. Setting that aside, I'm wondering about caes where
>> annotations can be dynamically generated, such as
>> dataclasses.make_dataclass(...). And, I could see reasons for
>> overwriting values in __annotations__, especially in the case where
>> it may be stored as a string and one wants to later affix its
>> evaluated value. These are considerations specific to runtime
>> (dynamic) type checking.
> It's fine to modify the __annotations__ dict after the creation of the
> class or module.  It's code that modifies "__annotations__" from
> within the class or module that is disallowed here. Similarly for
> dataclasses; once it creates a class object, it can explicitly set and
> / or modify the annotations dict on that class.

There won't be any direct impact to make_dataclass(). It doesn't do
anything tricky here: it just builds up the annotations dictionary and
passes it as __annotations__ to the class namespace in
types.new_class(). After creating the class, it just applies the normal
dataclass() decorator.

Eric

_______________________________________________
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-leave@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/T4KPZKA57SPV2JQPEVZLYMPF45F2TFSG/
Code of Conduct: http://python.org/psf/codeofconduct/

I still prefer PEP 563.
I will describe what we lost if PEP 597 is accepted and PEP 563 is rejected.

### Types not accessible in runtime

First of all, PEP 563 solves not only forward references.

Note that PEP 563 says: "we'll call any name imported or defined
within a `if TYPE_CHECKING: block` a forward reference, too."
https://www.python.org/dev/peps/pep-0563/#forward-references

PEP 563 solves all problems relating to types not accessible in runtime.
There are many reasons users can not get types used in annotations at runtime:

* To avoid circular import
* Types defined only in pyi files
* Optional dependency that is slow to import or hard to install

This is the most clear point where PEP 563 is better for some users.
See this example:

```
from dataclasses import dataclass

if 0:
from fakemod import FakeType

@dataclass
class C:
a : FakeType = 0
```

This works on PEP 563 semantics (Python 3.10a7). User can get
stringified annotation.

With stock semantics, it cause NameError when importing so author can
notice they need to quote "FakeType".

With PEP 649 semantics, author may not notice this annotation cause
error. User can not get any type hints at runtime.


### Type alias

Another PEP 563 benefit is user can see simple type alias.
Consider this example.

```
from typing import *

AliasType = Union[List[Dict[Tuple[int, str], Set[int]]], Tuple[str, List[str]]]

def f() -> AliasType:
pass

help(f)
```

Currently, help() calls `typing.get_type_hints()`. So it shows:

```
f() -> Union[List[Dict[Tuple[int, str], Set[int]]], Tuple[str, List[str]]]
```

But with PEP 563 semantics, we can stop evaluating annotations and
user can see more readable alias type.

```
f() -> AliasType
```

As PEP 597 says, eval() is slow. But it can avoidable in many cases
with PEP 563 semantics.
I am not sure but I expect dataclass can avoid eval() too in PEP 563 semantics.

Sphinx uses this feature already.
See https://www.sphinx-doc.org/en/master/usage/extensions/autodoc.html#confval-autodoc_type_aliases


### Relaxing annotation syntax

As discussed in PEP 647 thread, we can consider having different
syntax for annotation with PEP 597 semantics.


Regards,

On Mon, Apr 12, 2021 at 10:58 AM Larry Hastings <larry@hastings.org> wrote:
>
>
> Attached is my second draft of PEP 649. The PEP and the prototype have both seen a marked improvement since round 1 in January; PEP 649 now allows annotations to refer to any variable they could see under stock semantics:
>
> Local variables in the current function scope or in enclosing function scopes become closures and use LOAD_DEFER.
> Class variables in the current class scope are made available using a new mechanism, in which the class dict is attached to the bound annotation function, then loaded into f_locals when the annotation function is run. Thus permitting LOAD_NAME opcodes to function normally.
>
>
> I look forward to your comments,
>
>
> /arry
>
> _______________________________________________
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-leave@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/QSASX6PZ3LIIFIANHQQFS752BJYFUFPY/
> Code of Conduct: http://python.org/psf/codeofconduct/



--
Inada Naoki <songofacandy@gmail.com>
_______________________________________________
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-leave@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/KK5QG75RWSDBU4E36XAVSDPY5OUERA73/
Code of Conduct: http://python.org/psf/codeofconduct/

On 4/12/21 4:50 PM, Inada Naoki wrote:
> As PEP 597 says, eval() is slow. But it can avoidable in many cases
> with PEP 563 semantics.

PEP 597 is "Add optional EncodingWarning".  You said PEP 597 in one
other place too.  Did you mean PEP 649 in both places?


Cheers,


//arry/

On Tue, Apr 13, 2021 at 8:58 AM Larry Hastings <larry@hastings.org> wrote:
>
> On 4/12/21 4:50 PM, Inada Naoki wrote:
>
> As PEP 597 says, eval() is slow. But it can avoidable in many cases
> with PEP 563 semantics.
>
> PEP 597 is "Add optional EncodingWarning". You said PEP 597 in one other place too. Did you mean PEP 649 in both places?
>

You're right. I meant PEP 649 vs PEP 563. I'm sorry.

>
> Cheers,
>
>
> /arry



--
Inada Naoki <songofacandy@gmail.com>
_______________________________________________
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-leave@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/CLTMVQS7PMLEM237IAY4WCXC7M5DL7T6/
Code of Conduct: http://python.org/psf/codeofconduct/

On 4/12/21 4:50 PM, Inada Naoki wrote:
> PEP 563 solves all problems relating to types not accessible in runtime.
> There are many reasons users can not get types used in annotations at runtime:
>
> * To avoid circular import
> * Types defined only in pyi files
> * Optional dependency that is slow to import or hard to install

It only "solves" these problems if you leave the annotation as a
string.  If PEP 563 is active, but you then use typing.get_type_hints()
to examine the actual Python value of the annotation, all of these
examples will fail with a NameError.  So, in this case, "solves the
problem" is a positive way of saying "hides a runtime error".

I don't know what the use cases are for examining type hints at runtime,
so I can't speak as to how convenient or inconvenient it is to deal with
them strictly as strings.  But it seems to me that examining annotations
as their actual Python values would be preferable.


> This is the most clear point where PEP 563 is better for some users.
> See this example:
>
> ```
> from dataclasses import dataclass
>
> if 0:
> from fakemod import FakeType
>
> @dataclass
> class C:
> a : FakeType = 0
> ```
>
> This works on PEP 563 semantics (Python 3.10a7). User can get
> stringified annotation.
>
> With stock semantics, it cause NameError when importing so author can
> notice they need to quote "FakeType".
>
> With PEP 649 semantics, author may not notice this annotation cause
> error. User can not get any type hints at runtime.

Again, by "works on PEP 563 semantics", you mean "doesn't raise an
error".  But the code /has/ an error.  It's just that it has been hidden
by PEP 563 semantics.

I don't agree that changing Python to automatically hide errors is an
improvement.  As the Zen says: "Errors should never pass silently."

This is really the heart of the debate over PEP 649 vs PEP 563. If you
examine an annotation, and it references an undefined symbol, should
that throw an error?  There is definitely a contingent of people who say
"no, that's inconvenient for us".  I think it should raise an error. 
Again from the Zen: "Special cases aren't special enough to break the
rules."  Annotations are expressions, and if evaluating an expression
fails because of an undefined name, it should raise a NameError.


> ### Type alias
>
> Another PEP 563 benefit is user can see simple type alias.
> Consider this example.
>
> ```
> from typing import *
>
> AliasType = Union[List[Dict[Tuple[int, str], Set[int]]], Tuple[str, List[str]]]
>
> def f() -> AliasType:
> pass
>
> help(f)
> ```
>
> Currently, help() calls `typing.get_type_hints()`. So it shows:
>
> ```
> f() -> Union[List[Dict[Tuple[int, str], Set[int]]], Tuple[str, List[str]]]
> ```
>
> But with PEP 563 semantics, we can stop evaluating annotations and
> user can see more readable alias type.
>
> ```
> f() -> AliasType
> ```

It's a matter of personal opinion whether "AliasType" or the full
definition is better here.  And it could lead to ambiguity, if the
programmer assigns to "AliasType" more than once.

Iif the programmer has a strong opinion that "AliasType" is better, they
could use an annotation of 'AliasType'--in quotes. Although I haven't
seen the topic discussed specifically, I assume that the static typing
analysis tools will continue to support manually stringized annotations
even if PEP 649 is accepted.

Either way, this hypothetical feature might be "nice-to-have", but I
don't think it's very important.  I would certainly forego this behavior
in favor of accepting PEP 649.


Cheers,


//arry/

On Tue, Apr 13, 2021 at 9:57 AM Larry Hastings <larry@hastings.org> wrote:
>
>
> On 4/12/21 4:50 PM, Inada Naoki wrote:
>
> PEP 563 solves all problems relating to types not accessible in runtime.
> There are many reasons users can not get types used in annotations at runtime:
>
> * To avoid circular import
> * Types defined only in pyi files
> * Optional dependency that is slow to import or hard to install
>
> It only "solves" these problems if you leave the annotation as a string. If PEP 563 is active, but you then use typing.get_type_hints() to examine the actual Python value of the annotation, all of these examples will fail with a NameError. So, in this case, "solves the problem" is a positive way of saying "hides a runtime error".
>

Of course, "get type which is unavailable in runtime" is unsolvable
problem. PEP 597 doesn't solve it too. Author needs to quote the hint
manually, and `typing.get_type_hints()` raises NameError too.
And if author forget to quote, user can not get any type hints.


> I don't know what the use cases are for examining type hints at runtime, so I can't speak as to how convenient or inconvenient it is to deal with them strictly as strings. But it seems to me that examining annotations as their actual Python values would be preferable.
>

This is use cases for examining type hints at runtime and stringified
hints are OK.

* Sphinx autodoc
* help()
* IPython and other REPLS showing type hint in popup.


>
> ```
> from dataclasses import dataclass
>
> if 0:
> from fakemod import FakeType
>
> @dataclass
> class C:
> a : FakeType = 0
> ```
>
> This works on PEP 563 semantics (Python 3.10a7). User can get
> stringified annotation.
>
> With stock semantics, it cause NameError when importing so author can
> notice they need to quote "FakeType".
>
> With PEP 649 semantics, author may not notice this annotation cause
> error. User can not get any type hints at runtime.
>
> Again, by "works on PEP 563 semantics", you mean "doesn't raise an error". But the code has an error. It's just that it has been hidden by PEP 563 semantics.
>
> I don't agree that changing Python to automatically hide errors is an improvement. As the Zen says: "Errors should never pass silently."
>
> This is really the heart of the debate over PEP 649 vs PEP 563. If you examine an annotation, and it references an undefined symbol, should that throw an error? There is definitely a contingent of people who say "no, that's inconvenient for us". I think it should raise an error. Again from the Zen: "Special cases aren't special enough to break the rules." Annotations are expressions, and if evaluating an expression fails because of an undefined name, it should raise a NameError.
>

I agree that this is the heart of the debate. I think "annotations are
for type hitns". They are for:

* Static type checkers
* document.

So I don't think `if TYPE_CHECKING` idiom is violating Python Zen.


Regards,

--
Inada Naoki <songofacandy@gmail.com>
_______________________________________________
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-leave@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/DAMYYG5CY6MGRCAKIWRA4IKXW75DYXW6/
Code of Conduct: http://python.org/psf/codeofconduct/

On Tue, 2021-04-13 at 10:47 +0900, Inada Naoki wrote:

> On Tue, Apr 13, 2021 at 9:57 AM Larry Hastings <larry@hastings.org>
> wrote:

> > This is really the heart of the debate over PEP 649 vs PEP 563.  If
> > you examine an annotation, and it references an undefined symbol,
> > should that throw an error?  There is definitely a contingent of
> > people who say "no, that's inconvenient for us".  I think it should
> > raise an error.  Again from the Zen: "Special cases aren't special
> > enough to break the rules."  Annotations are expressions, and if
> > evaluating an expression fails because of an undefined name, it
> > should raise a NameError.

> I agree that this is the heart of the debate. I think "annotations
> are
> for type hitns". They are for:
>
> * Static type checkers
> * document.

+ dynamic type validation, encoding and decoding (Pydantic, FastAPI,
Fondat, et al.)

Paul

I've been thinking about this a bit, and I think that the way forward is
for Python to ignore the text of annotations ("relaxed annotation syntax"),
not to try and make it available as an expression.

To be honest, the most pressing issue with annotations is the clumsy way
that type variables have to be introduced. The current convention, `T =
TypeVar('T')`, is both verbose (why do I have to repeat the name?) and
widely misunderstood (many help request for mypy and pyright follow from
users making a mistaken association between two type variables that are
unrelated but share the same TypeVar definition). And relaxed annotation
syntax alone doesn't solve this.

Nevertheless I think that it's time to accept that annotations are for
types -- the intention of PEP 3107 was to experiment with different syntax
and semantics for types, and that experiment has resulted in the successful
adoption of a specific syntax for types that is wildly successful.





On Sun, Apr 11, 2021 at 7:02 PM Larry Hastings <larry@hastings.org> wrote:

>
> Attached is my second draft of PEP 649. The PEP and the prototype have
> both seen a marked improvement since round 1 in January; PEP 649 now allows
> annotations to refer to any variable they could see under stock semantics:
>
> - Local variables in the current function scope or in enclosing
> function scopes become closures and use LOAD_DEFER.
> - Class variables in the current class scope are made available using
> a new mechanism, in which the class dict is attached to the bound
> annotation function, then loaded into f_locals when the annotation function
> is run. Thus permitting LOAD_NAME opcodes to function normally.
>
>
> I look forward to your comments,
>
>
> */arry*
> _______________________________________________
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-leave@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-dev@python.org/message/QSASX6PZ3LIIFIANHQQFS752BJYFUFPY/
> Code of Conduct: http://python.org/psf/codeofconduct/
>


--
--Guido van Rossum (python.org/~guido)
*Pronouns: he/him **(why is my pronoun here?)*
<http://feministing.com/2015/02/03/how-using-they-as-a-singular-pronoun-can-change-the-world/>

On Tue, Apr 13, 2021 at 11:18 AM Paul Bryan <pbryan@anode.ca> wrote:
>
> On Tue, 2021-04-13 at 10:47 +0900, Inada Naoki wrote:
>
> On Tue, Apr 13, 2021 at 9:57 AM Larry Hastings <larry@hastings.org> wrote:
>
>
> This is really the heart of the debate over PEP 649 vs PEP 563. If you examine an annotation, and it references an undefined symbol, should that throw an error? There is definitely a contingent of people who say "no, that's inconvenient for us". I think it should raise an error. Again from the Zen: "Special cases aren't special enough to break the rules." Annotations are expressions, and if evaluating an expression fails because of an undefined name, it should raise a NameError.
>
>
> I agree that this is the heart of the debate. I think "annotations are
> for type hitns". They are for:
>
> * Static type checkers
> * document.
>
>
> + dynamic type validation, encoding and decoding (Pydantic, FastAPI, Fondat, et al.)
>
> Paul
>

OK. It is important use case too.

Such use cases doesn't justify raising NameError instead of getting
stringified type hints for documents for document use cases.

On the other hand, if "dynamic type" is used heavily, eval()
performance can be a problem.

--
Inada Naoki <songofacandy@gmail.com>
_______________________________________________
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-leave@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/ICKGBL674MUOYBOGNGKKJDPHYD3TYGER/
Code of Conduct: http://python.org/psf/codeofconduct/

On Tue, 2021-04-13 at 11:33 +0900, Inada Naoki wrote:
> On Tue, Apr 13, 2021 at 11:18 AM Paul Bryan <pbryan@anode.ca> wrote:
> >
> > On Tue, 2021-04-13 at 10:47 +0900, Inada Naoki wrote:
> >
> > On Tue, Apr 13, 2021 at 9:57 AM Larry Hastings <larry@hastings.org>
> > wrote:
> >
> >
> > This is really the heart of the debate over PEP 649 vs PEP 563.  If
> > you examine an annotation, and it references an undefined symbol,
> > should that throw an error?  There is definitely a contingent of
> > people who say "no, that's inconvenient for us".  I think it should
> > raise an error.  Again from the Zen: "Special cases aren't special
> > enough to break the rules."  Annotations are expressions, and if
> > evaluating an expression fails because of an undefined name, it
> > should raise a NameError.
> >
> >
> > I agree that this is the heart of the debate. I think "annotations
> > are
> > for type hitns". They are for:
> >
> > * Static type checkers
> > * document.
> >
> >
> > + dynamic type validation, encoding and decoding (Pydantic,
> > FastAPI, Fondat, et al.)
> >
> > Paul
> >
>
> OK. It is important use case too.
>
> Such use cases doesn't justify raising NameError instead of getting
> stringified type hints for documents for document use cases.
>
> On the other hand, if "dynamic type" is used heavily, eval()
> performance can be a problem.


In 3.9 this cost is paid once when a type is defined. However, in 3.10,
it gets expensive, because when the string is evaluated by
get_type_hints, its result is not stored/cached anywhere (repeated
calls to get_type_hints results in repeated evaluation). As a
workaround, I have code to "affix" the evaluated expression in
__annotations__ value. PEP 649 would resolve this and eliminate the
need for such a hack.

Paul

On Mon, Apr 12, 2021 at 7:47 PM Paul Bryan <pbryan@anode.ca> wrote:

> In 3.9 this cost is paid once when a type is defined. However, in 3.10, it
> gets expensive, because when the string is evaluated by get_type_hints, its
> result is not stored/cached anywhere (repeated calls to get_type_hints
> results in repeated evaluation). As a workaround, I have code to "affix"
> the evaluated expression in __annotations__ value. PEP 649 would resolve
> this and eliminate the need for such a hack.
>

Why not submit a PR that adds caching to get_type_hints(), rather than
promote a paradigm shift?

--
--Guido van Rossum (python.org/~guido)
*Pronouns: he/him **(why is my pronoun here?)*
<http://feministing.com/2015/02/03/how-using-they-as-a-singular-pronoun-can-change-the-world/>

On Mon, 2021-04-12 at 19:52 -0700, Guido van Rossum wrote:

> Why not submit a PR that adds caching to get_type_hints(), rather
> than promote a paradigm shift?

A couple of reasons:

1. In reviewing the code, I didn't find an obvious way to store cached
values. Anything but a non-trivial change would suggest the need for a
PEP of its own to document new behavior.

2. I've been hoping that PEP 649 would be adopted, making such a hack
or any plan to cache type hints moot.

Paul

Hi,

Le 13/04/2021 à 04:24, Guido van Rossum a écrit :
> I've been thinking about this a bit, and I think that the way forward is
> for Python to ignore the text of annotations ("relaxed annotation
> syntax"), not to try and make it available as an expression.

Then, what's wrong with quoting? It's just 2 characters, and prevent the
user (or their IDE) from trying to parse them as Python syntax.

As a comparison: docstrings do get quoting, even though they also have
special semantics in the language.

Cheers,
Baptiste
_______________________________________________
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-leave@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/V66L56SNJRUSM7CIJVCJDIPNHNZTGMWJ/
Code of Conduct: http://python.org/psf/codeofconduct/

Hi,

Le 12/04/2021 à 03:55, Larry Hastings a écrit :
>
> I look forward to your comments,

2 reading notes:

* in section "Annotations That Refer To Class Variables":

> If it's possible that an annotation function refers
> to class variables--if all these conditions are true:
>
> * The annotation function is being defined inside
> a class scope.
> * The generated code for the annotation function
> has at least one ``LOAD_NAME`` instruction.

I'm afraid I don't really understand the second condition. Would it be
possible to rephrase it in a less technical way, i.e. some condition on
the user code itself, not on what the implementation does with it.

* in section "Interactive REPL Shell":

> For the sake of simplicity, in this case we forego delayed evaluation.

This has the unpleasant consequence that any code using forward
references cannot be copy-pasted into the REPL. While such copy-pasting
is a very casual practice and does already often break, it is sometimes
useful in quick'n dirty prototyping. Would it be possible to specify
that in this case, a possible NameError in evaluation is caught, and the
annotation is set to None or some sentinel value?

Cheers,
Baptiste
_______________________________________________
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-leave@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/IPIJKJ6F4D634XT5ZXIKLD574QNWS2YB/
Code of Conduct: http://python.org/psf/codeofconduct/

Hi Larry,

?On 4/12/21, 6:57 PM, "Larry Hastings" <larry@midwinter.com on behalf of larry@hastings.org> wrote:
Again, by "works on PEP 563 semantics", you mean "doesn't raise an error". But the code has an error. It's just that it has been hidden by PEP 563 semantics.
I don't agree that changing Python to automatically hide errors is an improvement. As the Zen says: "Errors should never pass silently."

This is really the heart of the debate over PEP 649 vs PEP 563. If you examine an annotation, and it references an undefined symbol, should that throw an error? There is definitely a contingent of people who say "no, that's inconvenient for us". I think it should raise an error. Again from the Zen: "Special cases aren't special enough to break the rules." Annotations are expressions, and if evaluating an expression fails because of an undefined name, it should raise a NameError.

Normally in Python, if you reference a symbol in a function definition line, the symbol must be defined at that point in module execution. Forward references are not permitted, and will raise `NameError`.

And yet you have implemented PEP 649, whose entire raison d'être is to implement a "special case" to "break the rules" by delaying evaluation of annotations such that a type annotation, unlike any other expression in the function definition line, may include forward reference names which will not be defined until later in the module.

The use case for `if TYPE_CHECKING` imports is effectively the same. They are just forward references to names in other modules which can't be imported eagerly, because e.g. it would cause a cycle. Those who have used type annotations in earnest are likely to confirm that such inter-module forward references are just as necessary as intra-module forward references for the usability of type annotations.

So it doesn't seem that we have here is a firm stand on principle of the Zen, it appears to rather be a disagreement about exactly where to draw the line on the "special case" that we all already seem to agree is needed.

The Zen argument seems to be a bit of a circular one: I have defined PEP 649 semantics in precisely this way, therefore code that works with PEP 649 does not have an error, and code that does not work with PEP 649 "has an error" which must be surfaced!

With PEP 563, although `get_type_hints()` cannot natively resolve inter-module forward references and raises `NameError`, it is possible to work around this by supplying a globals dict to `get_type_hints()` that has been augmented with those forward-referenced names. Under the current version of PEP 649, it becomes impossible to get access to such type annotations at runtime at all, without reverting to manually stringifying the annotation and then using something like `get_type_hints()`. So for users of type annotations who need `if TYPE_CHECKING` (which I think is most users of type annotations), the best-case overall effect of PEP 649 will be that a) some type annotations have to go back to being ugly strings in the source, and b) if type annotation values are needed at runtime, `get_type_hints()` will still be as necessary as it ever was.

It is possible for PEP 649 to draw the line differently and support both intra-module and inter-module forward references in annotations, by doing something like https://github.com/larryhastings/co_annotations/pull/3 and replacing unknown names with forward-reference markers, so the annotation values are still accessible at runtime. This meets the real needs of users of type annotations better, and gives up none of the benefits of PEP 649.

Carl



_______________________________________________
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-leave@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/DSZFE7XTRK2ESRJDPQPZIDP2I67E76WH/
Code of Conduct: http://python.org/psf/codeofconduct/

On Tue, Apr 13, 2021 at 9:39 AM Baptiste Carvello <
devel2021@baptiste-carvello.net> wrote:

> Le 13/04/2021 à 04:24, Guido van Rossum a écrit :
> > I've been thinking about this a bit, and I think that the way forward is
> > for Python to ignore the text of annotations ("relaxed annotation
> > syntax"), not to try and make it available as an expression.
>
> Then, what's wrong with quoting? It's just 2 characters, and prevent the
> user (or their IDE) from trying to parse them as Python syntax.
>

Informal user research has shown high resistance to quoting.


> As a comparison: docstrings do get quoting, even though they also have
> special semantics in the language.
>

Not the same thing. Docstrings use English, which has no formal (enough)
syntax. The idea for annotations is that they *do* have a formal syntax, it
just evolves separately from that of Python itself.

--
--Guido van Rossum (python.org/~guido)
*Pronouns: he/him **(why is my pronoun here?)*
<http://feministing.com/2015/02/03/how-using-they-as-a-singular-pronoun-can-change-the-world/>

On 4/12/21 7:24 PM, Guido van Rossum wrote:
> I've been thinking about this a bit, and I think that the way forward
> is for Python to ignore the text of annotations ("relaxed annotation
> syntax"), not to try and make it available as an expression.
>
> To be honest, the most pressing issue with annotations is the clumsy
> way that type variables have to be introduced. The current convention,
> `T = TypeVar('T')`, is both verbose (why do I have to repeat the
> name?) and widely misunderstood (many help request for mypy and
> pyright follow from users making a mistaken association between two
> type variables that are unrelated but share the same TypeVar
> definition). And relaxed annotation syntax alone doesn't solve this.
>
> Nevertheless I think that it's time to accept that annotations are for
> types -- the intention of PEP 3107 was to experiment with different
> syntax and semantics for types, and that experiment has resulted in
> the successful adoption of a specific syntax for types that is wildly
> successful.


I don't follow your reasoning.  I'm glad that type hints have found
success, but I don't see why that implies "and therefore we should
restrict the use of annotations solely for type hints". Annotations are
a useful, general-purpose feature of Python, with legitimate uses
besides type hints.  Why would it make Python better to restrict their
use now?


//arry/

On Tue, Apr 13, 2021 at 12:32 PM Larry Hastings <larry@hastings.org> wrote:

>
> On 4/12/21 7:24 PM, Guido van Rossum wrote:
>
> I've been thinking about this a bit, and I think that the way forward is
> for Python to ignore the text of annotations ("relaxed annotation syntax"),
> not to try and make it available as an expression.
>
> To be honest, the most pressing issue with annotations is the clumsy way
> that type variables have to be introduced. The current convention, `T =
> TypeVar('T')`, is both verbose (why do I have to repeat the name?) and
> widely misunderstood (many help request for mypy and pyright follow from
> users making a mistaken association between two type variables that are
> unrelated but share the same TypeVar definition). And relaxed annotation
> syntax alone doesn't solve this.
>
> Nevertheless I think that it's time to accept that annotations are for
> types -- the intention of PEP 3107 was to experiment with different syntax
> and semantics for types, and that experiment has resulted in the successful
> adoption of a specific syntax for types that is wildly successful.
>
>
> I don't follow your reasoning. I'm glad that type hints have found
> success, but I don't see why that implies "and therefore we should restrict
> the use of annotations solely for type hints". Annotations are a useful,
> general-purpose feature of Python, with legitimate uses besides type
> hints. Why would it make Python better to restrict their use now?
>

Because typing is, to many folks, a Really Important Concept, and it's
confusing to use the same syntax ("x: blah blah") for different purposes,
in a way that makes it hard to tell whether a particular "blah blah" is
meant as a type or as something else -- because you have to know what's
introspecting the annotations before you can tell. And that introspection
could be signalled by a magical decorator, but it could also be implicit:
maybe you have a driver that calls a function based on a CLI entry point
name, and introspects that function even if it's not decorated.

OTOH, not requiring that annotations are syntactically valid expressions
might liberate such a CLI library too: you could write things like

def foo(prec: --precision int):
...


--
--Guido van Rossum (python.org/~guido)
*Pronouns: he/him **(why is my pronoun here?)*
<http://feministing.com/2015/02/03/how-using-they-as-a-singular-pronoun-can-change-the-world/>

On 4/13/2021 4:21 AM, Baptiste Carvello wrote:
> Le 12/04/2021 à 03:55, Larry Hastings a écrit :

> * in section "Interactive REPL Shell":
>
>> For the sake of simplicity, in this case we forego delayed evaluation.

The intention of the code + codeop modules is that people should be able
to write interactive consoles that simulate the standard REPL. For example:

Python 3.10.0a7+ (heads/master-dirty:a9cf69df2e, Apr 12 2021, 15:36:39)
[MSC v.1900 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import code
>>> code.interact()
Python 3.10.0a7+ (heads/master-dirty:a9cf69df2e, Apr 12 2021, 15:36:39)
[MSC v.1900 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
(InteractiveConsole)
>>> # Call has not returned. Prompt is from code.InteractiveConsole.
>>> def f(x:int): -> float

>>> f.__annotations__ # should match REPL result

>>> ^Z

now exiting InteractiveConsole...
>>> Now back to repl

If the REPL compiles with "mode='single' and spec is changes to "when
mode is 'single'", then above should work. Larry, please test with your
proposed implementation.

--
Terry Jan Reedy


_______________________________________________
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-leave@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/KDFTZQMJRBRDBSUSVTMBDXTLCXIZXBFP/
Code of Conduct: http://python.org/psf/codeofconduct/

On 4/13/21 3:28 PM, Terry Reedy wrote:
> On 4/13/2021 4:21 AM, Baptiste Carvello wrote:
>> Le 12/04/2021 à 03:55, Larry Hastings a écrit :
>
>> * in section "Interactive REPL Shell":
>>
>>> For the sake of simplicity, in this case we forego delayed evaluation.
>
> The intention of the code + codeop modules is that people should be
> able to write interactive consoles that simulate the standard REPL. 
> For example:
>
> Python 3.10.0a7+ (heads/master-dirty:a9cf69df2e, Apr 12 2021,
> 15:36:39) [MSC v.1900 64 bit (AMD64)] on win32
> Type "help", "copyright", "credits" or "license" for more information.
> >>> import code
> >>> code.interact()
> Python 3.10.0a7+ (heads/master-dirty:a9cf69df2e, Apr 12 2021,
> 15:36:39) [MSC v.1900 64 bit (AMD64)] on win32
> Type "help", "copyright", "credits" or "license" for more information.
> (InteractiveConsole)
> >>> # Call has not returned.  Prompt is from code.InteractiveConsole.
> >>> def f(x:int): -> float
>
> >>> f.__annotations__ # should match REPL result
>
> >>> ^Z
>
> now exiting InteractiveConsole...
> >>> Now back to repl
>
> If the REPL compiles with "mode='single' and spec is changes to "when
> mode is 'single'", then above should work.  Larry, please test with
> your proposed implementation.


A couple things!

1. I apologize if the PEP wasn't clear, but this section was talking
about the problem of /module/ annotations in the implicit __main__
module when using the interactive REPL. Annotations on other objects
(classes, functions, etc) defined in the interactive REPL work as
expected.
2. The above example has a minor bug: when defining a return annotation
on a function, the colon ending the function declaration goes
/after/ the return annotation.  It should have been "def f(x:int) ->
float:".
3. The above example works fine when run in my branch.
4. You need to "from __future__ import co_annotations" in order to
activate delayed evaluation of annotations using code objects in my
branch.  I added that (inside the code.interact() shell!) and it
still works fine.

So I'm not sure what problem you're proposing to solve with this "mode
is single" stuff.

* If you thought there was a problem with defining annotations on
functions and classes defined in the REPL, good news!, it was never
a problem.
* If you're solving the problem of defining annotations on the
interactive module /itself,/ I don't understand what your proposed
solution is or how it would work.  The problem is, how do you create
a code object that defines all the annotations on a module, when the
module never finishes being defined because it's the interactive shell?


Cheers,


//arry/

On 4/13/21 1:52 PM, Guido van Rossum wrote:
> On Tue, Apr 13, 2021 at 12:32 PM Larry Hastings <larry@hastings.org
> <mailto:larry@hastings.org>> wrote:
>
>
> On 4/12/21 7:24 PM, Guido van Rossum wrote:
>> I've been thinking about this a bit, and I think that the way
>> forward is for Python to ignore the text of annotations ("relaxed
>> annotation syntax"), not to try and make it available as an
>> expression.
>>
>> To be honest, the most pressing issue with annotations is the
>> clumsy way that type variables have to be introduced. The current
>> convention, `T = TypeVar('T')`, is both verbose (why do I have to
>> repeat the name?) and widely misunderstood (many help request for
>> mypy and pyright follow from users making a mistaken association
>> between two type variables that are unrelated but share the same
>> TypeVar definition). And relaxed annotation syntax alone doesn't
>> solve this.
>>
>> Nevertheless I think that it's time to accept that annotations
>> are for types -- the intention of PEP 3107 was to experiment with
>> different syntax and semantics for types, and that experiment has
>> resulted in the successful adoption of a specific syntax for
>> types that is wildly successful.
>
>
> I don't follow your reasoning.  I'm glad that type hints have
> found success, but I don't see why that implies "and therefore we
> should restrict the use of annotations solely for type hints". 
> Annotations are a useful, general-purpose feature of Python, with
> legitimate uses besides type hints.  Why would it make Python
> better to restrict their use now?
>
>
> Because typing is, to many folks, a Really Important Concept, and it's
> confusing to use the same syntax ("x: blah blah") for different
> purposes, in a way that makes it hard to tell whether a particular
> "blah blah" is meant as a type or as something else -- because you
> have to know what's introspecting the annotations before you can tell.
> And that introspection could be signalled by a magical decorator, but
> it could also be implicit: maybe you have a driver that calls a
> function based on a CLI entry point name, and introspects that
> function even if it's not decorated.


I'm not sure I understand your point.  Are you saying that we need to
take away the general-purpose functionality of annotations, that's been
in the language since 3.0, and restrict annotations to just type
hints... because otherwise an annotation might not be used for a type
hint, and then the programmer would have to figure out what it means? 
We need to take away the functionality from all other use cases in order
to lend /clarity/ to one use case?

Also, if you're stating that programmers get confused reading source
code because annotations get used for different things at different
places--surely that confirms that annotations are /useful/ for more than
just type hints, in real-world code, today.  I genuinely have no sense
of how important static type analysis is in Python--personally I have no
need for it--but I find it hard to believe that type hints are so
overwhelmingly important that they should become the sole use case for
annotations, and we need to take away this long-standing functionality,
that you suggest is being successfully used side-by-side with type hints
today, merely to make type hints clearer.


Cheers,


//arry/

On Wed, Apr 14, 2021 at 10:44 AM Larry Hastings <larry@hastings.org> wrote:
>
>
> On 4/13/21 1:52 PM, Guido van Rossum wrote:
>
>
> Because typing is, to many folks, a Really Important Concept, and it's confusing to use the same syntax ("x: blah blah") for different purposes, in a way that makes it hard to tell whether a particular "blah blah" is meant as a type or as something else -- because you have to know what's introspecting the annotations before you can tell. And that introspection could be signalled by a magical decorator, but it could also be implicit: maybe you have a driver that calls a function based on a CLI entry point name, and introspects that function even if it's not decorated.
>
>
> I'm not sure I understand your point. Are you saying that we need to take away the general-purpose functionality of annotations, that's been in the language since 3.0, and restrict annotations to just type hints... because otherwise an annotation might not be used for a type hint, and then the programmer would have to figure out what it means? We need to take away the functionality from all other use cases in order to lend clarity to one use case?
>

I don't think we need to take away "general purpose functionality".
But if we define type hinting is 1st class use case of annotations,
annotations should be optimized for type hinting. General purpose use
case should accept some limitation and overhead.

On the other hand, if we decide general purpose functionality is 1st
class too, we shouldn't annotation syntax different from Python
syntax.

But annotations should be optimized for type hinting anyway. General
purpose use case used only is a limited part of application. On the
other hand, type hint can be used almost everywhere in application
code base. It must cheap enough.

Regards,

--
Inada Naoki <songofacandy@gmail.com>
_______________________________________________
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-leave@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/UGFWTZUGH6QZRHF3FKTQHZLYG2ZNX5EG/
Code of Conduct: http://python.org/psf/codeofconduct/

For the record, Cython allows using annotations for typing:
https://cython.readthedocs.io/en/latest/src/tutorial/pure.html#pep-484-type-annotations

I don't know if they are fully compatible with the type hints we're
talking about here.

Regards

Antoine.


On Wed, 14 Apr 2021 10:58:07 +0900
Inada Naoki <songofacandy@gmail.com> wrote:
> On Wed, Apr 14, 2021 at 10:44 AM Larry Hastings <larry@hastings.org> wrote:
> >
> >
> > On 4/13/21 1:52 PM, Guido van Rossum wrote:
> >
> >
> > Because typing is, to many folks, a Really Important Concept, and it's confusing to use the same syntax ("x: blah blah") for different purposes, in a way that makes it hard to tell whether a particular "blah blah" is meant as a type or as something else -- because you have to know what's introspecting the annotations before you can tell. And that introspection could be signalled by a magical decorator, but it could also be implicit: maybe you have a driver that calls a function based on a CLI entry point name, and introspects that function even if it's not decorated.
> >
> >
> > I'm not sure I understand your point. Are you saying that we need to take away the general-purpose functionality of annotations, that's been in the language since 3.0, and restrict annotations to just type hints... because otherwise an annotation might not be used for a type hint, and then the programmer would have to figure out what it means? We need to take away the functionality from all other use cases in order to lend clarity to one use case?
> >
>
> I don't think we need to take away "general purpose functionality".
> But if we define type hinting is 1st class use case of annotations,
> annotations should be optimized for type hinting. General purpose use
> case should accept some limitation and overhead.
>
> On the other hand, if we decide general purpose functionality is 1st
> class too, we shouldn't annotation syntax different from Python
> syntax.
>
> But annotations should be optimized for type hinting anyway. General
> purpose use case used only is a limited part of application. On the
> other hand, type hint can be used almost everywhere in application
> code base. It must cheap enough.
>
> Regards,
>



_______________________________________________
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-leave@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/VJWVKMQFAOMYNORX3F4557KWINO4Z7GG/
Code of Conduct: http://python.org/psf/codeofconduct/

Hi,

tl;dr: imho the like or dislike of PEP 563 is related to whether people
intend to learn a second syntax for typing, or would rather ignore it;
both groups should be taken into account.

Le 13/04/2021 à 19:30, Guido van Rossum a écrit :
> On Tue, Apr 13, 2021 at 9:39 AM Baptiste Carvello
> <devel2021@baptiste-carvello.net
> <mailto:devel2021@baptiste-carvello.net>> wrote:
>
> Then, what's wrong with quoting? It's just 2 characters, and prevent the
> user (or their IDE) from trying to parse them as Python syntax.
>
>
> Informal user research has shown high resistance to quoting.

OK, but why? I'd bet it's due to an "aesthetic" concern: for typing
users, type hints are code, not textual data. So it irks them to see
them quoted and syntax-highlighted as text strings.

> As a comparison: docstrings do get quoting, even though they also have
> special semantics in the language.
>
>
> Not the same thing. Docstrings use English, which has no formal (enough)
> syntax. The idea for annotations is that they *do* have a formal syntax,
> it just evolves separately from that of Python itself.

If I may say it in my words: to both the parser and (more importantly)
typing-savvy developers, type hints are code. I now see the point.

But what about developers who won't learn this (future) incompatible
typing syntax, and only encounter it in the wild? To them, those
annotations are akin to docstrings: pieces of textual data that Python
manages specially because of their role in the greater ecosystem, but
that they can ignore because the program behavior is not modified.

So it will irk them if annotations in this new syntax are not quoted or
otherwise made distinguishable from code written in the normal Python
syntax they understand. Again the "aesthetic" concern, and imho it
explains in large part why some people dislike PEP 563.

Can the needs of both groups of developers be addressed? Could code in
the new typing syntax be marked with a specific syntactic marker,
distinguishing it from both normal Python syntax and text strings? Then
this new marker could also be used outside of annotations, to mark
analysis-time-only imports or statements?

Or is this all not worth the expense, and typing syntax can manage to
stay compatible with normal Python syntax, in which case PEP 649 is the
way to go?

Cheers,
Baptiste
_______________________________________________
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-leave@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/YI3DPTJXKTKZRVVWBJ6UYTUH3PPAJM6Y/
Code of Conduct: http://python.org/psf/codeofconduct/

It looks like a small subset of PEP 484, syntactically. So it should be
fine. Possibly cython might be interested in using a relaxed notation if it
is ever introduced, e.g. ‘long long’ or ‘static int’ (for a return type)?

On Wed, Apr 14, 2021 at 02:27 Antoine Pitrou <antoine@python.org> wrote:

>
> For the record, Cython allows using annotations for typing:
>
> https://cython.readthedocs.io/en/latest/src/tutorial/pure.html#pep-484-type-annotations
>
> I don't know if they are fully compatible with the type hints we're
> talking about here.
>
> Regards
>
> Antoine.
>
>
> On Wed, 14 Apr 2021 10:58:07 +0900
> Inada Naoki <songofacandy@gmail.com> wrote:
> > On Wed, Apr 14, 2021 at 10:44 AM Larry Hastings <larry@hastings.org>
> wrote:
> > >
> > >
> > > On 4/13/21 1:52 PM, Guido van Rossum wrote:
> > >
> > >
> > > Because typing is, to many folks, a Really Important Concept, and it's
> confusing to use the same syntax ("x: blah blah") for different purposes,
> in a way that makes it hard to tell whether a particular "blah blah" is
> meant as a type or as something else -- because you have to know what's
> introspecting the annotations before you can tell. And that introspection
> could be signalled by a magical decorator, but it could also be implicit:
> maybe you have a driver that calls a function based on a CLI entry point
> name, and introspects that function even if it's not decorated.
> > >
> > >
> > > I'm not sure I understand your point. Are you saying that we need to
> take away the general-purpose functionality of annotations, that's been in
> the language since 3.0, and restrict annotations to just type hints...
> because otherwise an annotation might not be used for a type hint, and then
> the programmer would have to figure out what it means? We need to take
> away the functionality from all other use cases in order to lend clarity to
> one use case?
> > >
> >
> > I don't think we need to take away "general purpose functionality".
> > But if we define type hinting is 1st class use case of annotations,
> > annotations should be optimized for type hinting. General purpose use
> > case should accept some limitation and overhead.
> >
> > On the other hand, if we decide general purpose functionality is 1st
> > class too, we shouldn't annotation syntax different from Python
> > syntax.
> >
> > But annotations should be optimized for type hinting anyway. General
> > purpose use case used only is a limited part of application. On the
> > other hand, type hint can be used almost everywhere in application
> > code base. It must cheap enough.
> >
> > Regards,
> >
>
>
>
> _______________________________________________
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-leave@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-dev@python.org/message/VJWVKMQFAOMYNORX3F4557KWINO4Z7GG/
> Code of Conduct: http://python.org/psf/codeofconduct/
>
--
--Guido (mobile)

On 4/12/21 7:24 PM, Guido van Rossum wrote:
> To be honest, the most pressing issue with annotations is the clumsy
> way that type variables have to be introduced. The current convention,
> `T = TypeVar('T')`, is both verbose (why do I have to repeat the
> name?) and widely misunderstood (many help request for mypy and
> pyright follow from users making a mistaken association between two
> type variables that are unrelated but share the same TypeVar definition).


This repeat-the-name behavior has been in Python for a long time, e.g.

Point = namedtuple('Point', ['x', 'y'])

namedtuple() shipped with Python 2.6 in 2008.  So if that's the most
pressing issue with annotations, annotations must be going quite well,
because we've known about this for at least 13 years without attempting
to solve it.

I've always assumed that this repetition was worth the minor
inconvenience.  You only have to retype the name once, and the resulting
code is clear and readable, with predictable behavior. A small price to
pay to preserve Python's famous readability.


For what it's worth--and forgive me for straying slightly into
python-ideas territory--/if/ we wanted to eliminate the need to repeat
the name, I'd prefer a general-purpose solution rather than something
tailored specifically for type hints.  In a recent private email
conversation on a different topic, I proposed this syntax:

bind <id> <expression>

This statement would be equivalent to

id = expression('id')


Cheers,


//arry/

On Tue, Apr 13, 2021 at 6:48 PM Larry Hastings <larry@hastings.org> wrote:

>
> On 4/13/21 1:52 PM, Guido van Rossum wrote:
>
> On Tue, Apr 13, 2021 at 12:32 PM Larry Hastings <larry@hastings.org>
> wrote:
>
>>
>> On 4/12/21 7:24 PM, Guido van Rossum wrote:
>>
>> I've been thinking about this a bit, and I think that the way forward is
>> for Python to ignore the text of annotations ("relaxed annotation syntax"),
>> not to try and make it available as an expression.
>>
>> To be honest, the most pressing issue with annotations is the clumsy way
>> that type variables have to be introduced. The current convention, `T =
>> TypeVar('T')`, is both verbose (why do I have to repeat the name?) and
>> widely misunderstood (many help request for mypy and pyright follow from
>> users making a mistaken association between two type variables that are
>> unrelated but share the same TypeVar definition). And relaxed annotation
>> syntax alone doesn't solve this.
>>
>> Nevertheless I think that it's time to accept that annotations are for
>> types -- the intention of PEP 3107 was to experiment with different syntax
>> and semantics for types, and that experiment has resulted in the successful
>> adoption of a specific syntax for types that is wildly successful.
>>
>>
>> I don't follow your reasoning. I'm glad that type hints have found
>> success, but I don't see why that implies "and therefore we should restrict
>> the use of annotations solely for type hints". Annotations are a useful,
>> general-purpose feature of Python, with legitimate uses besides type
>> hints. Why would it make Python better to restrict their use now?
>>
>
> Because typing is, to many folks, a Really Important Concept, and it's
> confusing to use the same syntax ("x: blah blah") for different purposes,
> in a way that makes it hard to tell whether a particular "blah blah" is
> meant as a type or as something else -- because you have to know what's
> introspecting the annotations before you can tell. And that introspection
> could be signalled by a magical decorator, but it could also be implicit:
> maybe you have a driver that calls a function based on a CLI entry point
> name, and introspects that function even if it's not decorated.
>
>
> I'm not sure I understand your point. Are you saying that we need to take
> away the general-purpose functionality of annotations, that's been in the
> language since 3.0, and restrict annotations to just type hints... because
> otherwise an annotation might not be used for a type hint, and then the
> programmer would have to figure out what it means? We need to take away
> the functionality from all other use cases in order to lend *clarity* to
> one use case?
>

Yes, that's how I see it.

And before you get too dramatic about it, the stringification of
annotations has been in the making a long time, with the community's and
the SC's support. You came up with a last-minute attempt to change it,
using the PEP process to propose to *revert* the decision already codified
in PEP 563 and implemented in the master branch. But you've waited until
the last minute (feature freeze is in three weeks) and IMO you're making
things awkward for the SC (who can and will speak for themselves).


> Also, if you're stating that programmers get confused reading source code
> because annotations get used for different things at different
> places--surely that confirms that annotations are *useful* for more than
> just type hints, in real-world code, today.
>

No, it doesn't, it's just a hypothetical that they *would* be confused if
there *were* other uses. Personally I haven't used any libraries that use
non-type-hint annotations, but I've been told they exist.

I genuinely have no sense of how important static type analysis is in
> Python--personally I have no need for it--but I find it hard to believe
> that type hints are so overwhelmingly important that they should become the
> sole use case for annotations, and we need to take away this long-standing
> functionality, that you suggest is being successfully used side-by-side
> with type hints today, merely to make type hints clearer.
>

For projects and teams that use type hints, they are *very* important. For
example, they are so important to the Instagram team at Facebook that they
wrote their own static type checker when they found mypy wasn't fast enough
for their million-line codebase. And of course they were so important to
Dropbox that they sponsored a multi-year, multi-person effort to create
mypy in the first place. The amount of feedback we've received for mypy
indicates that it's not just those two companies that are using type hints.

--
--Guido van Rossum (python.org/~guido)
*Pronouns: he/him **(why is my pronoun here?)*
<http://feministing.com/2015/02/03/how-using-they-as-a-singular-pronoun-can-change-the-world/>

On Tue, Apr 13, 2021 at 6:58 PM Inada Naoki <songofacandy@gmail.com> wrote:

> On Wed, Apr 14, 2021 at 10:44 AM Larry Hastings <larry@hastings.org>
> wrote:
> >
> >
> > On 4/13/21 1:52 PM, Guido van Rossum wrote:
> >
> >
> > Because typing is, to many folks, a Really Important Concept, and it's
> confusing to use the same syntax ("x: blah blah") for different purposes,
> in a way that makes it hard to tell whether a particular "blah blah" is
> meant as a type or as something else -- because you have to know what's
> introspecting the annotations before you can tell. And that introspection
> could be signalled by a magical decorator, but it could also be implicit:
> maybe you have a driver that calls a function based on a CLI entry point
> name, and introspects that function even if it's not decorated.
> >
> >
> > I'm not sure I understand your point. Are you saying that we need to
> take away the general-purpose functionality of annotations, that's been in
> the language since 3.0, and restrict annotations to just type hints...
> because otherwise an annotation might not be used for a type hint, and then
> the programmer would have to figure out what it means? We need to take
> away the functionality from all other use cases in order to lend clarity to
> one use case?
> >
>
> I don't think we need to take away "general purpose functionality".
> But if we define type hinting is 1st class use case of annotations,
> annotations should be optimized for type hinting. General purpose use
> case should accept some limitation and overhead.
>
> On the other hand, if we decide general purpose functionality is 1st
> class too, we shouldn't annotation syntax different from Python
> syntax.
>

Has anyone reached out to people like Pydantic, FastAPI, typer, etc. to see
what they think of this PEP? For instance, are they having issues with the
way things are today enough that this is a very clear win for them?

-Brett


>
> But annotations should be optimized for type hinting anyway. General
> purpose use case used only is a limited part of application. On the
> other hand, type hint can be used almost everywhere in application
> code base. It must cheap enough.
>
> Regards,
>
> --
> Inada Naoki <songofacandy@gmail.com>
> _______________________________________________
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-leave@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-dev@python.org/message/UGFWTZUGH6QZRHF3FKTQHZLYG2ZNX5EG/
> Code of Conduct: http://python.org/psf/codeofconduct/
>

On Wed, Apr 14, 2021 at 9:42 AM Baptiste Carvello <
devel2021@baptiste-carvello.net> wrote:

> Hi,
>
> tl;dr: imho the like or dislike of PEP 563 is related to whether people
> intend to learn a second syntax for typing, or would rather ignore it;
> both groups should be taken into account.
>
> Le 13/04/2021 à 19:30, Guido van Rossum a écrit :
> > On Tue, Apr 13, 2021 at 9:39 AM Baptiste Carvello
> > <devel2021@baptiste-carvello.net
> > <mailto:devel2021@baptiste-carvello.net>> wrote:
> >
> > Then, what's wrong with quoting? It's just 2 characters, and prevent
> the
> > user (or their IDE) from trying to parse them as Python syntax.
> >
> >
> > Informal user research has shown high resistance to quoting.
>
> OK, but why? I'd bet it's due to an "aesthetic" concern: for typing
> users, type hints are code, not textual data. So it irks them to see
> them quoted and syntax-highlighted as text strings.
>

No, what I heard is that, since in *most* cases the string quotes are not
needed, people are surprised and annoyed when they encounter cases where
they are needed. And if you have a large code base it takes an expensive
run of the static type checker to find out that you've forgotten the quotes.


> > As a comparison: docstrings do get quoting, even though they also
> have
> > special semantics in the language.
> >
> >
> > Not the same thing. Docstrings use English, which has no formal (enough)
> > syntax. The idea for annotations is that they *do* have a formal syntax,
> > it just evolves separately from that of Python itself.
>
> If I may say it in my words: to both the parser and (more importantly)
> typing-savvy developers, type hints are code. I now see the point.
>
> But what about developers who won't learn this (future) incompatible
> typing syntax, and only encounter it in the wild? To them, those
> annotations are akin to docstrings: pieces of textual data that Python
> manages specially because of their role in the greater ecosystem, but
> that they can ignore because the program behavior is not modified.
>
> So it will irk them if annotations in this new syntax are not quoted or
> otherwise made distinguishable from code written in the normal Python
> syntax they understand. Again the "aesthetic" concern, and imho it
> explains in large part why some people dislike PEP 563.
>

They will treat it as anything else they don't quite understand -- they
will ignore it unless it bites them. And the rule for finding the end of an
annotation would be very simple -- just skip words until the next comma,
close paren or colon, skipping matching brackets etc.

Certainly I use this strategy all the time for quickly skimming code (in
any language) that I don't need to completely understand -- "oh, this is
where the parameters are processed, this is where the defaults are sorted
out, and this is where the work is being done; and since I'm investigating
why the default is weird, let me look at that part of the code in more
detail."


> Can the needs of both groups of developers be addressed? Could code in
> the new typing syntax be marked with a specific syntactic marker,
> distinguishing it from both normal Python syntax and text strings? Then
> this new marker could also be used outside of annotations, to mark
> analysis-time-only imports or statements?
>

There already is a special marker for annotations in function definitions
-- for arguments, it's the colon following the parameter name, and for
return types, it's the arrow after the parameter list. And for variable
declarations ("x: int") the same colon also suffices.

The idea to use the same marker for other analysis-time code is
interesting, but the syntactic requirements are somewhat different --
annotations live at the "expression" level (informally speaking) and are
already in a clearly indicated syntactic position -- analysis-time code
looks just like other code and can occur in any position where statements
can appear. So an appropriate marker would probably be an if-statement with
a special condition, like "if TYPE_CHECKING" (I am all for making that a
built-in constant, BTW).

If you were thinking of backticks, sorry, I'm not biting. (There are folks
who have other plans for those -- presumably because it's one of the few
ASCII characters that currently has no meaning.)


> Or is this all not worth the expense, and typing syntax can manage to
> stay compatible with normal Python syntax, in which case PEP 649 is the
> way to go?
>

I don't see much expense in the proposal to relax the syntax, and I see
benefits for new kinds of type annotations (e.g. PEP 647 would have
benefited).

And certainly PEP 649 has considerable cost as well -- besides the cost of
closing the door to relaxed annotation syntax, there's the engineering work
of undoing the work that was done to make `from __future__ import
annotations` the default (doing this was a significant effort spread over
many commits, and undoing will be just as hard). And we would still have to
support stringification when that import is explicitly given for several
more releases.

--
--Guido van Rossum (python.org/~guido)
*Pronouns: he/him **(why is my pronoun here?)*
<http://feministing.com/2015/02/03/how-using-they-as-a-singular-pronoun-can-change-the-world/>

I favour annotations for type hints; the writing's been on the wall for
some time. I think the necessary escape hatch for those using it for
other purposes should be Annotated[Any, ...] (or a similar, nicer
alternative).

Guido, one of the difficulties I'm having is understanding the
direction you're going with "relaxed syntax". PEP 649 is concrete; it's
hard to weigh its merits against the usability—even feasibility—of
incorporating an as yet undefined relaxed syntax.

At the end of the day, such syntax is going to have to be represented
in some structure. If one were to accept that annotations are for type
hints only, is the debate then the difference between a Python type
(which PEP 649 would yield) and some other as yet undefined structure?
 

Paul


On Wed, 2021-04-14 at 10:24 -0700, Guido van Rossum wrote:
> On Tue, Apr 13, 2021 at 6:48 PM Larry Hastings <larry@hastings.org>
> wrote:
> >
> > On 4/13/21 1:52 PM, Guido van Rossum wrote:
> >
> > > On Tue, Apr 13, 2021 at 12:32 PM Larry Hastings
> > > <larry@hastings.org> wrote:
> > >
> > > >
> > > > On 4/12/21 7:24 PM, Guido van Rossum wrote:
> > > >
> > > > > I've been thinking about this a bit, and I think that the way
> > > > > forward is for Python to ignore the text of annotations
> > > > > ("relaxed annotation syntax"), not to try and make it
> > > > > available as an expression.
> > > > >
> > > > > To be honest, the most pressing issue with annotations is the
> > > > > clumsy way that type variables have to be introduced. The
> > > > > current convention, `T = TypeVar('T')`, is both verbose (why
> > > > > do I have to repeat the name?) and widely misunderstood (many
> > > > > help request for mypy and pyright follow from users making a
> > > > > mistaken association between two type variables that are
> > > > > unrelated but share the same TypeVar definition). And relaxed
> > > > > annotation syntax alone doesn't solve this.
> > > > >
> > > > > Nevertheless I think that it's time to accept that
> > > > > annotations are for types -- the intention of PEP 3107 was to
> > > > > experiment with different syntax and semantics for types, and
> > > > > that experiment has resulted in the successful adoption of a
> > > > > specific syntax for types that is wildly successful.
> > > >
> > > > I don't follow your reasoning.  I'm glad that type hints have
> > > > found success, but I don't see why that implies "and therefore
> > > > we should restrict the use of annotations solely for type
> > > > hints".  Annotations are a useful, general-purpose feature of
> > > > Python, with legitimate uses besides type hints.  Why would it
> > > > make Python better to restrict their use now?
> > > >
> > >
> > > Because typing is, to many folks, a Really Important Concept, and
> > > it's confusing to use the same syntax ("x: blah blah") for
> > > different purposes, in a way that makes it hard to tell whether a
> > > particular "blah blah" is meant as a type or as something else --
> > > because you have to know what's introspecting the annotations
> > > before you can tell. And that introspection could be signalled by
> > > a magical decorator, but it could also be implicit: maybe you
> > > have a driver that calls a function based on a CLI entry point
> > > name, and introspects that function even if it's not decorated.
> >
> > I'm not sure I understand your point.  Are you saying that we need
> > to take away the general-purpose functionality of annotations,
> > that's been in the language since 3.0, and restrict annotations to
> > just type hints... because otherwise an annotation might not be
> > used for a type hint, and then the programmer would have to figure
> > out what it means?  We need to take away the functionality from all
> > other use cases in order to lend clarity to one use case?
> >
>
>
> Yes, that's how I see it.
>
> And before you get too dramatic about it, the stringification of
> annotations has been in the making a long time, with the community's
> and the SC's support. You came up with a last-minute attempt to
> change it, using the PEP process to propose to *revert* the decision
> already codified in PEP 563 and implemented in the master branch. But
> you've waited until the last minute (feature freeze is in three
> weeks) and IMO you're making things awkward for the SC (who can and
> will speak for themselves).
>  
> > Also, if you're stating that programmers get confused reading
> > source code because annotations get used for different things at
> > different places--surely that confirms that annotations are useful
> > for more than just type hints, in real-world code, today.
> >
>
> No, it doesn't, it's just a hypothetical that they *would* be
> confused if there *were* other uses. Personally I haven't used any
> libraries that use non-type-hint annotations, but I've been told they
> exist.
>
> > I genuinely have no sense of how important static type analysis is
> > in Python--personally I have no need for it--but I find it hard to
> > believe that type hints are so overwhelmingly important that they
> > should become the sole use case for annotations, and we need to
> > take away this long-standing functionality, that you suggest is
> > being successfully used side-by-side with type hints today, merely
> > to make type hints clearer.
> >
>
>
> For projects and teams that use type hints, they are *very*
> important. For example, they are so important to the Instagram team
> at Facebook that they wrote their own static type checker when they
> found mypy wasn't fast enough for their million-line codebase. And of
> course they were so important to Dropbox that they sponsored a multi-
> year, multi-person effort to create mypy in the first place. The
> amount of feedback we've received for mypy indicates that it's not
> just those two companies that are using type hints.
>
> _______________________________________________
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-leave@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-dev@python.org/message/UONAPCPQQLMI755NULYS3F67A6MGZKJY/
> Code of Conduct: http://python.org/psf/codeofconduct/

On Wed, Apr 14, 2021 at 10:47 AM Paul Bryan <pbryan@anode.ca> wrote:

> I favour annotations for type hints; the writing's been on the wall for
> some time. I think the necessary escape hatch for those using it for other
> purposes should be Annotated[Any, ...] (or a similar, nicer alternative).
>
> Guido, one of the difficulties I'm having is understanding the direction
> you're going with "relaxed syntax". PEP 649 is concrete; it's hard to weigh
> its merits against the usability—even feasibility—of incorporating an as
> yet undefined relaxed syntax.
>
> At the end of the day, such syntax is going to have to be represented in
> some structure. If one were to accept that annotations are for type hints
> only, is the debate then the difference between a Python type (which PEP
> 649 would yield) and some other as yet undefined structure?
>

In `__annotations__` it would be a string, as currently implemented in the
3.10 alpha code. The string just might not be parsable as an expression.

In the AST, it will have to be a new node that just collects tokens and
bracketed things; that could be an array of low-level tokens.

--
--Guido van Rossum (python.org/~guido)
*Pronouns: he/him **(why is my pronoun here?)*
<http://feministing.com/2015/02/03/how-using-they-as-a-singular-pronoun-can-change-the-world/>

What would you expect get_type_hints(...) to return with relaxed
syntax? Today, for type hint annotations, it returns a type, which I'd
argue is an important feature to preserve (in it or some successor).

On Wed, 2021-04-14 at 10:54 -0700, Guido van Rossum wrote:
> On Wed, Apr 14, 2021 at 10:47 AM Paul Bryan <pbryan@anode.ca> wrote:
> > I favour annotations for type hints; the writing's been on the wall
> > for some time. I think the necessary escape hatch for those using
> > it for other purposes should be Annotated[Any, ...] (or a similar,
> > nicer alternative).
> >
> > Guido, one of the difficulties I'm having is understanding the
> > direction you're going with "relaxed syntax". PEP 649 is concrete;
> > it's hard to weigh its merits against the usability—even
> > feasibility—of incorporating an as yet undefined relaxed syntax.
> >
> > At the end of the day, such syntax is going to have to be
> > represented in some structure. If one were to accept that
> > annotations are for type hints only, is the debate then the
> > difference between a Python type (which PEP 649 would yield) and
> > some other as yet undefined structure? 
> >
>
>
> In `__annotations__` it would be a string, as currently implemented
> in the 3.10 alpha code. The string just might not be parsable as an
> expression.
>
> In the AST, it will have to be a new node that just collects tokens
> and bracketed things; that could be an array of low-level tokens.
>

On 4/14/21 10:44 AM, Guido van Rossum wrote:
> besides the cost of closing the door to relaxed annotation syntax,
> there's the engineering work of undoing the work that was done to make
> `from __future__ import annotations` the default (doing this was a
> significant effort spread over many commits, and undoing will be just
> as hard).


I'm not sure either of those statements is true.

Accepting PEP 649 as written would deprecate stringized annotations,
it's true.  But the SC can make any decision it wants here, including
only accepting the new semantics of 649 without deprecating stringized
annotations.  They could remain in the language for another release (or
two? or three?) while we "kick the can down the road".  This is not
without its costs too but it might be the best approach for now.

As for undoing the effort to make stringized annotations the default,
git should do most of the heavy lifting here.  There's a technique where
you check out the revision that made the change, generate a reverse
patch, apply it, and check that in.  This creates a new head which you
then merge.  That's what I did when I created my co_annotations branch,
and at the time it was literally the work of ten minutes.  I gather the
list of changes is more substantial now, so this would have to be done
multiple times, and it may be more involved.  Still, if PEP 649 is
accepted, I would happily volunteer to undertake this part of the workload.


Cheers,


//arry/

On Wed, Apr 14, 2021 at 11:03 AM Paul Bryan <pbryan@anode.ca> wrote:

> What would you expect get_type_hints(...) to return with relaxed syntax?
> Today, for type hint annotations, it returns a type, which I'd argue is an
> important feature to preserve (in it or some successor).
>

It would have to return some other representation. Presumably the (purely
hypothetical) new syntax would be syntactic sugar for something that can be
expressed as an object, just like (as of PEP 604) X | Y is syntactic sugar
for Union[X, Y].

--
--Guido van Rossum (python.org/~guido)
*Pronouns: he/him **(why is my pronoun here?)*
<http://feministing.com/2015/02/03/how-using-they-as-a-singular-pronoun-can-change-the-world/>

Let's just wait for the SC to join the discussion. I'm sure they will,
eventually.

On Wed, Apr 14, 2021 at 11:12 AM Larry Hastings <larry@hastings.org> wrote:

> On 4/14/21 10:44 AM, Guido van Rossum wrote:
>
> besides the cost of closing the door to relaxed annotation syntax, there's
> the engineering work of undoing the work that was done to make `from
> __future__ import annotations` the default (doing this was a significant
> effort spread over many commits, and undoing will be just as hard).
>
>
> I'm not sure either of those statements is true.
>
> Accepting PEP 649 as written would deprecate stringized annotations, it's
> true. But the SC can make any decision it wants here, including only
> accepting the new semantics of 649 without deprecating stringized
> annotations. They could remain in the language for another release (or
> two? or three?) while we "kick the can down the road". This is not without
> its costs too but it might be the best approach for now.
>
> As for undoing the effort to make stringized annotations the default, git
> should do most of the heavy lifting here. There's a technique where you
> check out the revision that made the change, generate a reverse patch,
> apply it, and check that in. This creates a new head which you then
> merge. That's what I did when I created my co_annotations branch, and at
> the time it was literally the work of ten minutes. I gather the list of
> changes is more substantial now, so this would have to be done multiple
> times, and it may be more involved. Still, if PEP 649 is accepted, I would
> happily volunteer to undertake this part of the workload.
>
>
> Cheers,
>
>
> */arry*
> _______________________________________________
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-leave@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-dev@python.org/message/LRVFVLH4AHF7SX5MOEUBPPII7UNINAMJ/
> Code of Conduct: http://python.org/psf/codeofconduct/
>


--
--Guido van Rossum (python.org/~guido)
*Pronouns: he/him **(why is my pronoun here?)*
<http://feministing.com/2015/02/03/how-using-they-as-a-singular-pronoun-can-change-the-world/>

On Wed, Apr 14, 2021 at 12:08 PM Guido van Rossum <guido@python.org> wrote:

> Let's just wait for the SC to join the discussion. I'm sure they will,
> eventually.
>

FYI the PEP has not been sent to us via
https://github.com/python/steering-council/issues as ready for
pronouncement, so we have not started officially discussing this PEP yet.

-Brett


>
> On Wed, Apr 14, 2021 at 11:12 AM Larry Hastings <larry@hastings.org>
> wrote:
>
>> On 4/14/21 10:44 AM, Guido van Rossum wrote:
>>
>> besides the cost of closing the door to relaxed annotation syntax,
>> there's the engineering work of undoing the work that was done to make
>> `from __future__ import annotations` the default (doing this was a
>> significant effort spread over many commits, and undoing will be just as
>> hard).
>>
>>
>> I'm not sure either of those statements is true.
>>
>> Accepting PEP 649 as written would deprecate stringized annotations, it's
>> true. But the SC can make any decision it wants here, including only
>> accepting the new semantics of 649 without deprecating stringized
>> annotations. They could remain in the language for another release (or
>> two? or three?) while we "kick the can down the road". This is not without
>> its costs too but it might be the best approach for now.
>>
>> As for undoing the effort to make stringized annotations the default, git
>> should do most of the heavy lifting here. There's a technique where you
>> check out the revision that made the change, generate a reverse patch,
>> apply it, and check that in. This creates a new head which you then
>> merge. That's what I did when I created my co_annotations branch, and at
>> the time it was literally the work of ten minutes. I gather the list of
>> changes is more substantial now, so this would have to be done multiple
>> times, and it may be more involved. Still, if PEP 649 is accepted, I would
>> happily volunteer to undertake this part of the workload.
>>
>>
>> Cheers,
>>
>>
>> */arry*
>> _______________________________________________
>> Python-Dev mailing list -- python-dev@python.org
>> To unsubscribe send an email to python-dev-leave@python.org
>> https://mail.python.org/mailman3/lists/python-dev.python.org/
>> Message archived at
>> https://mail.python.org/archives/list/python-dev@python.org/message/LRVFVLH4AHF7SX5MOEUBPPII7UNINAMJ/
>> Code of Conduct: http://python.org/psf/codeofconduct/
>>
>
>
> --
> --Guido van Rossum (python.org/~guido)
> *Pronouns: he/him **(why is my pronoun here?)*
> <http://feministing.com/2015/02/03/how-using-they-as-a-singular-pronoun-can-change-the-world/>
> _______________________________________________
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-leave@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-dev@python.org/message/V5ASSMVVAP4RZX3DOGJIDS52OEJ6LP7C/
> Code of Conduct: http://python.org/psf/codeofconduct/
>

My plan was to post it here and see what the response was first. Back in
January, when I posted the first draft, I got some very useful feedback
that resulted in some dramatic changes.  This time around, so far,
nobody has suggested even minor changes.  Folks have just expressed
their opinions about it (which is fine).

Still left to do: ping the project leads of some other static type
analysis projects and see if they have any feedback to contribute.  Once
the dust completely settles around the conversation here, I expect to
formally submit the PEP, hopefully later this week.

Cheers,


//arry/

On 4/14/21 12:22 PM, Brett Cannon wrote:
>
>
> On Wed, Apr 14, 2021 at 12:08 PM Guido van Rossum <guido@python.org
> <mailto:guido@python.org>> wrote:
>
> Let's just wait for the SC to join the discussion. I'm sure they
> will, eventually.
>
>
> FYI the PEP has not been sent to us via
> https://github.com/python/steering-council/issues
> <https://github.com/python/steering-council/issues> as ready for
> pronouncement, so we have not started officially discussing this PEP yet.
>
> -Brett
>
>
> On Wed, Apr 14, 2021 at 11:12 AM Larry Hastings
> <larry@hastings.org <mailto:larry@hastings.org>> wrote:
>
> On 4/14/21 10:44 AM, Guido van Rossum wrote:
>> besides the cost of closing the door to relaxed annotation
>> syntax, there's the engineering work of undoing the work that
>> was done to make `from __future__ import annotations` the
>> default (doing this was a significant effort spread over many
>> commits, and undoing will be just as hard).
>
>
> I'm not sure either of those statements is true.
>
> Accepting PEP 649 as written would deprecate stringized
> annotations, it's true.  But the SC can make any decision it
> wants here, including only accepting the new semantics of 649
> without deprecating stringized annotations.  They could remain
> in the language for another release (or two? or three?) while
> we "kick the can down the road". This is not without its costs
> too but it might be the best approach for now.
>
> As for undoing the effort to make stringized annotations the
> default, git should do most of the heavy lifting here. 
> There's a technique where you check out the revision that made
> the change, generate a reverse patch, apply it, and check that
> in.  This creates a new head which you then merge. That's what
> I did when I created my co_annotations branch, and at the time
> it was literally the work of ten minutes.  I gather the list
> of changes is more substantial now, so this would have to be
> done multiple times, and it may be more involved.  Still, if
> PEP 649 is accepted, I would happily volunteer to undertake
> this part of the workload.
>
>
> Cheers,
>
>
> //arry/
>
> _______________________________________________
> Python-Dev mailing list -- python-dev@python.org
> <mailto:python-dev@python.org>
> To unsubscribe send an email to python-dev-leave@python.org
> <mailto:python-dev-leave@python.org>
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> <https://mail.python.org/mailman3/lists/python-dev.python.org/>
> Message archived at
> https://mail.python.org/archives/list/python-dev@python.org/message/LRVFVLH4AHF7SX5MOEUBPPII7UNINAMJ/
> <https://mail.python.org/archives/list/python-dev@python.org/message/LRVFVLH4AHF7SX5MOEUBPPII7UNINAMJ/>
> Code of Conduct: http://python.org/psf/codeofconduct/
> <http://python.org/psf/codeofconduct/>
>
>
>
> --
> --Guido van Rossum (python.org/~guido <http://python.org/~guido>)
> /Pronouns: he/him //(why is my pronoun here?)/
> <http://feministing.com/2015/02/03/how-using-they-as-a-singular-pronoun-can-change-the-world/>
> _______________________________________________
> Python-Dev mailing list -- python-dev@python.org
> <mailto:python-dev@python.org>
> To unsubscribe send an email to python-dev-leave@python.org
> <mailto:python-dev-leave@python.org>
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> <https://mail.python.org/mailman3/lists/python-dev.python.org/>
> Message archived at
> https://mail.python.org/archives/list/python-dev@python.org/message/V5ASSMVVAP4RZX3DOGJIDS52OEJ6LP7C/
> <https://mail.python.org/archives/list/python-dev@python.org/message/V5ASSMVVAP4RZX3DOGJIDS52OEJ6LP7C/>
> Code of Conduct: http://python.org/psf/codeofconduct/
> <http://python.org/psf/codeofconduct/>
>

Hi Larry,

?On 4/14/21, 1:56 PM, "Larry Hastings" <larry@midwinter.com on behalf of larry@hastings.org> wrote:

> My plan was to post it here and see what the response was first. Back in January, when I posted the first draft, I got some very useful feedback that resulted in some dramatic changes. This time around, so far, nobody has suggested even minor changes. Folks have just expressed their opinions about it (which is fine).

This is not true. I suggested yesterday (in https://mail.python.org/archives/list/python-dev@python.org/message/DSZFE7XTRK2ESRJDPQPZIDP2I67E76WH/ ) that PEP 649 could avoid making life worse for users of type annotations (relative to PEP 563) if it replaced runtime-undefined names with forward reference markers, as implemented in https://github.com/larryhastings/co_annotations/pull/3

Perhaps you've chosen to ignore the suggestion, but that's not the same as nobody suggesting any changes ;)

Carl

_______________________________________________
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-leave@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/4LXOED3ABKDSNUDJ3JTNEGTXD3R7TEWT/
Code of Conduct: http://python.org/psf/codeofconduct/

Hi,

Le 14/04/2021 à 19:44, Guido van Rossum a écrit :
>
> No, what I heard is that, since in *most* cases the string quotes are
> not needed, people are surprised and annoyed when they encounter cases
> where they are needed. And if you have a large code base it takes an
> expensive run of the static type checker to find out that you've
> forgotten the quotes.

Well, I had assumed quotes would be used in all cases for consistency.
Indeed, using them only if needed leads to surprises. Are there specific
annoyances associated with quoting always, apart from the 2 more characters?

>  
> [...]
>
> They will treat it as anything else they don't quite understand -- they
> will ignore it unless it bites them. And the rule for finding the end of
> an annotation would be very simple -- just skip words until the next
> comma, close paren or colon, skipping matching brackets etc.

That's assuming the syntax in the annotations doesn't diverge too much
from the Python syntax as far as brackets etc are concerned. I must say
I'm not too worried about typing. But the hypothetic "def foo(prec:
--precision int):" is already less readable. Will finding the closing
comma or colon always be obvious to the human reader?

Cheers,
Baptiste
_______________________________________________
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-leave@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/E654LKI36TFURBOSBMFUPEMLJQ32CZNN/
Code of Conduct: http://python.org/psf/codeofconduct/

On 4/14/21 1:42 PM, Baptiste Carvello wrote:
> Are there specific annoyances associated with quoting always, apart
> from the 2 more characters?


Yes.  Since the quoted strings aren't parsed by Python, syntax errors in
these strings go undetected until somebody does parse them (e.g. your
static type analyzer).  Having the Python compiler de-compile them back
into strings means they got successfully parsed.  Though this doesn't
rule out other errors, e.g. NameError.

I thought this was discussed in PEP 563, but now I can't find it, so
unfortunately I can't steer you towards any more info on the subject.


Cheers,


//arry/

On Wed, 2021-04-14 at 22:42 +0200, Baptiste Carvello wrote:

> That's assuming the syntax in the annotations doesn't diverge too
> much
> from the Python syntax as far as brackets etc are concerned. I must
> say
> I'm not too worried about typing. But the hypothetic "def foo(prec:
> --precision int):" is already less readable. Will finding the closing
> comma or colon always be obvious to the human reader?

To push the limit, let's add some default value:

def foo(prec: --precision int = 123):
    ...

vs.

def foo(prec: "--precision int" = 123):
    ...

And if a "type parameter" becomes numeric. For example:

def foo(prec: --max bar 3000 = 123):
    ...

vs.

def foo(prec: "--max bar 3000" = 123):
    ...

Now, the quotes seem even more readable as a delimiter.

Paul

Baptiste Carvello wrote:
> Le 14/04/2021 à 19:44, Guido van Rossum a écrit :

> > No, what I heard is that, since in *most* cases the string quotes are
> > not needed, people are surprised and annoyed when they encounter cases
> > where they are needed.

> Well, I had assumed quotes would be used in all cases for consistency.

That does seem like a reasonable solution. Redundant, ugly, and annoying, but safe and consistent. Sort of like using type constraints in the first place. :D

> > ... the rule for finding the end of
> > an annotation would be very simple -- just skip words until the next
> > comma, close paren or colon, skipping matching brackets etc.

> ... But the hypothetic "def foo(prec:
> --precision int):" is already less readable. Will finding the closing
> comma or colon always be obvious to the human reader?

Nope. "--" sometimes means "ignore the rest of the line, including the ")". At the moment, I can't remember where I've seen this outside of SQL, but I can guarantee that if I read it late enough at night, the *best* case would be that I notice the ambiguity, guess correctly and am only annoyed.

-jJ
_______________________________________________
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-leave@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/MV3HQASUSAEKC332UFUDWQNONLCXZDRZ/
Code of Conduct: http://python.org/psf/codeofconduct/

Larry Hastings wrote:
> On 4/14/21 1:42 PM, Baptiste Carvello wrote:
> > Are there specific annoyances associated with quoting always, apart
> > from the 2 more characters?

> Yes.  Since the quoted strings aren't parsed by Python, syntax errors in
> these strings go undetected until somebody does parse them (e.g. your
> static type analyzer).

This is a real problem. But in theory, your code editor (and python) *could* parse the strings. They generally don't, but I'm not sure asking them to do that is much harder than asking them to deal with new syntax.

-jJ
_______________________________________________
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-leave@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/JR2RKEEUI3WSIUBZOJ4J3EVRTVYGLXSU/
Code of Conduct: http://python.org/psf/codeofconduct/

I created simple benchmark:
https://gist.github.com/methane/abb509e5f781cc4a103cc450e1e7925d

This benchmark creates 1000 annotated functions and measure time to
load and exec.
And here is the result. All interpreters are built without --pydebug,
--enable-optimization, and --with-lto.

```
# Python 3.9 w/ stock semantics

$ python3 ~/ann_test.py 1
code size: 121011
unmarshal: avg: 0.33605549649801103 +/- 0.007382938279889738
exec: avg: 0.395090194279328 +/- 0.001004608380122509

# Python 3.9 w/ PEP 563 semantics

$ python3 ~/ann_test.py 2
code size: 121070
unmarshal: avg: 0.3407619891455397 +/- 0.0011833618746421965
exec: avg: 0.24590165729168803 +/- 0.0003123404336687428

# master branch w/ PEP 563 semantics

$ ./python ~/ann_test.py 2
code size: 149086
unmarshal: avg: 0.45410854648798704 +/- 0.00107521956753799
exec: avg: 0.11281821667216718 +/- 0.00011939747308270317

# master branch + optimization (*) w/ PEP 563 semantics
$ ./python ~/ann_test.py 2
code size: 110488
unmarshal: avg: 0.3184352931333706 +/- 0.0015278719180908732
exec: avg: 0.11042822999879717 +/- 0.00018108884723599264

# co_annotatins reference implementation w/ PEP 649 semantics

$ ./python ~/ann_test.py 3
code size: 229679
unmarshal: avg: 0.6402394526172429 +/- 0.0006400500128250688
exec: avg: 0.09774857209995388 +/- 9.275466265195788e-05

# co_annotatins reference implementation + optimization (*) w/ PEP 649 semantics

$ ./python ~/ann_test.py 3
code size: 204963
unmarshal: avg: 0.5824743471574039 +/- 0.007219086642131638
exec: avg: 0.09641968684736639 +/- 0.0001416784753249878
```

(*) I found constant folding creates new tuple every time even though
same tuple is in constant table.
See https://github.com/python/cpython/pull/25419
For co_annotations, I cherry-pick
https://github.com/python/cpython/pull/23056 too.


--
Inada Naoki <songofacandy@gmail.com>
_______________________________________________
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-leave@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/JEP4MXAHCVGQF7AI5OUUSGOENXAOR43O/
Code of Conduct: http://python.org/psf/codeofconduct/

I added memory usage data by tracemalloc.

```
# Python 3.9 w/ old semantics
$ python3 ann_test.py 1
code size: 121011
memory: (385200, 385200)
unmarshal: avg: 0.3341682574478909 +/- 3.700437551781949e-05
exec: avg: 0.4067857594229281 +/- 0.0006858555167675445

# Python 3.9 w/ PEP 563 semantics
$ python3 ann_test.py 2
code size: 121070
memory: (398675, 398675)
unmarshal: avg: 0.3352349083404988 +/- 7.749102039824168e-05
exec: avg: 0.24610224328935146 +/- 0.0008628035427956459

# master + optimization w/ PEP 563 semantics
$ ./python ~/ann_test.py 2
code size: 110488
memory: (193572, 193572)
unmarshal: avg: 0.31316645480692384 +/- 0.00011766086337841035
exec: avg: 0.11456295938696712 +/- 0.0017481202239372398

# co_annotations + optimization w/ PEP 649 semantics
$ ./python ~/ann_test.py 3
code size: 204963
memory: (208273, 208273)
unmarshal: avg: 0.597023528907448 +/- 0.00016614519056599577
exec: avg: 0.09546191191766411 +/- 0.00018099485135812695
```

Summary:

* Both of PEP 563 and PEP 649 has low memory consumption than Python 3.9.
* Importing time (unmarshal+exec) is about 0.7sec on old semantics and
PEP 649, 0.43sec on PEP 563.

On Thu, Apr 15, 2021 at 10:31 AM Inada Naoki <songofacandy@gmail.com> wrote:
>
> I created simple benchmark:
> https://gist.github.com/methane/abb509e5f781cc4a103cc450e1e7925d
>
> This benchmark creates 1000 annotated functions and measure time to
> load and exec.
> And here is the result. All interpreters are built without --pydebug,
> --enable-optimization, and --with-lto.
>
> ```
> # Python 3.9 w/ stock semantics
>
> $ python3 ~/ann_test.py 1
> code size: 121011
> unmarshal: avg: 0.33605549649801103 +/- 0.007382938279889738
> exec: avg: 0.395090194279328 +/- 0.001004608380122509
>
> # Python 3.9 w/ PEP 563 semantics
>
> $ python3 ~/ann_test.py 2
> code size: 121070
> unmarshal: avg: 0.3407619891455397 +/- 0.0011833618746421965
> exec: avg: 0.24590165729168803 +/- 0.0003123404336687428
>
> # master branch w/ PEP 563 semantics
>
> $ ./python ~/ann_test.py 2
> code size: 149086
> unmarshal: avg: 0.45410854648798704 +/- 0.00107521956753799
> exec: avg: 0.11281821667216718 +/- 0.00011939747308270317
>
> # master branch + optimization (*) w/ PEP 563 semantics
> $ ./python ~/ann_test.py 2
> code size: 110488
> unmarshal: avg: 0.3184352931333706 +/- 0.0015278719180908732
> exec: avg: 0.11042822999879717 +/- 0.00018108884723599264
>
> # co_annotatins reference implementation w/ PEP 649 semantics
>
> $ ./python ~/ann_test.py 3
> code size: 229679
> unmarshal: avg: 0.6402394526172429 +/- 0.0006400500128250688
> exec: avg: 0.09774857209995388 +/- 9.275466265195788e-05
>
> # co_annotatins reference implementation + optimization (*) w/ PEP 649 semantics
>
> $ ./python ~/ann_test.py 3
> code size: 204963
> unmarshal: avg: 0.5824743471574039 +/- 0.007219086642131638
> exec: avg: 0.09641968684736639 +/- 0.0001416784753249878
> ```
>
> (*) I found constant folding creates new tuple every time even though
> same tuple is in constant table.
> See https://github.com/python/cpython/pull/25419
> For co_annotations, I cherry-pick
> https://github.com/python/cpython/pull/23056 too.
>
>
> --
> Inada Naoki <songofacandy@gmail.com>



--
Inada Naoki <songofacandy@gmail.com>
_______________________________________________
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-leave@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/JIZ6PCV5SSIL7BUKZUCVF45OFNO4H26I/
Code of Conduct: http://python.org/psf/codeofconduct/

Thanks for doing this!  I don't think PEP 649 is going to be accepted or
rejected based on either performance or memory usage, but it's nice to
see you confirmed that its performance and memory impact is acceptable.


If I run "ann_test.py 1", the annotations are already turned into
strings.  Why do you do it that way?  It makes stock semantics look
better, because manually stringized annotations are much faster than
evaluating real expressions.

It seems to me that the test would be more fair if test 1 used real
annotations.  So I added this to "lines":

from types import SimpleNamespace
foo = SimpleNamespace()
foo.bar = SimpleNamespace()
foo.bar.baz = float

I also changed quote(t) so it always returned t unchanged.  When I ran
it that way, stock semantics "exec" time got larger.


Cheers,


//arry/

On 4/14/21 6:44 PM, Inada Naoki wrote:
> I added memory usage data by tracemalloc.
>
> ```
> # Python 3.9 w/ old semantics
> $ python3 ann_test.py 1
> code size: 121011
> memory: (385200, 385200)
> unmarshal: avg: 0.3341682574478909 +/- 3.700437551781949e-05
> exec: avg: 0.4067857594229281 +/- 0.0006858555167675445
>
> # Python 3.9 w/ PEP 563 semantics
> $ python3 ann_test.py 2
> code size: 121070
> memory: (398675, 398675)
> unmarshal: avg: 0.3352349083404988 +/- 7.749102039824168e-05
> exec: avg: 0.24610224328935146 +/- 0.0008628035427956459
>
> # master + optimization w/ PEP 563 semantics
> $ ./python ~/ann_test.py 2
> code size: 110488
> memory: (193572, 193572)
> unmarshal: avg: 0.31316645480692384 +/- 0.00011766086337841035
> exec: avg: 0.11456295938696712 +/- 0.0017481202239372398
>
> # co_annotations + optimization w/ PEP 649 semantics
> $ ./python ~/ann_test.py 3
> code size: 204963
> memory: (208273, 208273)
> unmarshal: avg: 0.597023528907448 +/- 0.00016614519056599577
> exec: avg: 0.09546191191766411 +/- 0.00018099485135812695
> ```
>
> Summary:
>
> * Both of PEP 563 and PEP 649 has low memory consumption than Python 3.9.
> * Importing time (unmarshal+exec) is about 0.7sec on old semantics and
> PEP 649, 0.43sec on PEP 563.
>
> On Thu, Apr 15, 2021 at 10:31 AM Inada Naoki <songofacandy@gmail.com> wrote:
>> I created simple benchmark:
>> https://gist.github.com/methane/abb509e5f781cc4a103cc450e1e7925d
>>
>> This benchmark creates 1000 annotated functions and measure time to
>> load and exec.
>> And here is the result. All interpreters are built without --pydebug,
>> --enable-optimization, and --with-lto.
>>
>> ```
>> # Python 3.9 w/ stock semantics
>>
>> $ python3 ~/ann_test.py 1
>> code size: 121011
>> unmarshal: avg: 0.33605549649801103 +/- 0.007382938279889738
>> exec: avg: 0.395090194279328 +/- 0.001004608380122509
>>
>> # Python 3.9 w/ PEP 563 semantics
>>
>> $ python3 ~/ann_test.py 2
>> code size: 121070
>> unmarshal: avg: 0.3407619891455397 +/- 0.0011833618746421965
>> exec: avg: 0.24590165729168803 +/- 0.0003123404336687428
>>
>> # master branch w/ PEP 563 semantics
>>
>> $ ./python ~/ann_test.py 2
>> code size: 149086
>> unmarshal: avg: 0.45410854648798704 +/- 0.00107521956753799
>> exec: avg: 0.11281821667216718 +/- 0.00011939747308270317
>>
>> # master branch + optimization (*) w/ PEP 563 semantics
>> $ ./python ~/ann_test.py 2
>> code size: 110488
>> unmarshal: avg: 0.3184352931333706 +/- 0.0015278719180908732
>> exec: avg: 0.11042822999879717 +/- 0.00018108884723599264
>>
>> # co_annotatins reference implementation w/ PEP 649 semantics
>>
>> $ ./python ~/ann_test.py 3
>> code size: 229679
>> unmarshal: avg: 0.6402394526172429 +/- 0.0006400500128250688
>> exec: avg: 0.09774857209995388 +/- 9.275466265195788e-05
>>
>> # co_annotatins reference implementation + optimization (*) w/ PEP 649 semantics
>>
>> $ ./python ~/ann_test.py 3
>> code size: 204963
>> unmarshal: avg: 0.5824743471574039 +/- 0.007219086642131638
>> exec: avg: 0.09641968684736639 +/- 0.0001416784753249878
>> ```
>>
>> (*) I found constant folding creates new tuple every time even though
>> same tuple is in constant table.
>> See https://github.com/python/cpython/pull/25419
>> For co_annotations, I cherry-pick
>> https://github.com/python/cpython/pull/23056 too.
>>
>>
>> --
>> Inada Naoki <songofacandy@gmail.com>
>
>

On Thu, Apr 15, 2021 at 11:09 AM Larry Hastings <larry@hastings.org> wrote:
>
> Thanks for doing this! I don't think PEP 649 is going to be accepted or rejected based on either performance or memory usage, but it's nice to see you confirmed that its performance and memory impact is acceptable.
>
>
> If I run "ann_test.py 1", the annotations are already turned into strings. Why do you do it that way? It makes stock semantics look better, because manually stringized annotations are much faster than evaluating real expressions.
>

Because `if TYPE_CHECKING` and manually stringified annotation is used
in real world applications.
I want to mix both use cases.

--
Inada Naoki <songofacandy@gmail.com>
_______________________________________________
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-leave@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/MKAGFBTVZ4LXYJC5X6P3LXXXRD7P2WH5/
Code of Conduct: http://python.org/psf/codeofconduct/

I updated the benchmark little:

* Added no annotation mode for baseline performance.
* Better stats output.

https://gist.github.com/methane/abb509e5f781cc4a103cc450e1e7925d

```
# No annotation (master + GH-25419)
$ ./python ~/ann_test.py 0
code size: 102967 bytes
memory: 181288 bytes
unmarshal: avg: 299.301ms +/-1.257ms
exec: avg: 104.019ms +/-0.038ms

# PEP 563 (master + GH-25419)
$ ./python ~/ann_test.py 2
code size: 110488 bytes
memory: 193572 bytes
unmarshal: avg: 313.032ms +/-0.068ms
exec: avg: 108.456ms +/-0.048ms

# PEP 649 (co_annotations + GH-25419 + GH-23056)
$ ./python ~/ann_test.py 3
code size: 204963 bytes
memory: 209257 bytes
unmarshal: avg: 587.336ms +/-2.073ms
exec: avg: 97.056ms +/-0.046ms

# Python 3.9
$ python3 ann_test.py 0
code size: 108955 bytes
memory: 173296 bytes
unmarshal: avg: 333.527ms +/-1.750ms
exec: avg: 90.810ms +/-0.347ms

$ python3 ann_test.py 1
code size: 121011 bytes
memory: 385200 bytes
unmarshal: avg: 334.929ms +/-0.055ms
exec: avg: 400.260ms +/-0.249ms
```

## Rough estimation of annotation overhead

Python 3.9 w/o PEP 563
code (pyc) size: +11%
memory usage: +122% (211bytes / function)
import time: +73% (*)

PEP 563
code (pyc) size: +7.3%
memory usage: +0.68% (13.3bytes / function)
import time: +4.5%

PEP 649
code (pyc) size: +99%
memory usage: +15% (28 bytes / function)
import time: +70%

(*) import time can be much more slower for complex annotations.

## Conclusion

* PEP 563 is close to "zero overhead" in memory consumption. And
import time overhead is ~5%. Users can write type annotations without
caring overhead.

* PEP 649 is much better than old semantics for memory usage and
import time. But import time is still longer than no annotation code.

* The import time overhead is coming from unmarshal, not from
eval(). If we implement a "lazy load" mechanizm for docstrings and
annotations, overhead will become cheaper.
* pyc file become bigger (but who cares?)

I will read PEP 649 implementation to find missing optimizations other
than GH-25419 and GH-23056.

--
Inada Naoki <songofacandy@gmail.com>
_______________________________________________
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-leave@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/K5PFGS6DQUHUG63UWRXYNLLMXAVELP32/
Code of Conduct: http://python.org/psf/codeofconduct/

Thanks Brett Cannon for suggesting to get Samuel Colvin (Pydantic) and me, Sebastián Ramírez (FastAPI and Typer) involved in this.

TL;DR: it seems to me PEP 649 would be incredibly important/useful for Pydantic, FastAPI, Typer, and similar tools, and their communities.

## About FastAPI, Pydantic, Typer

Some of you probably don't know what these tools are, so, in short, FastAPI is a web API framework based on Pydantic. It has had massive growth and adoption, and very positive feedback.

FastAPI was included for the first time in the last Python developers survey and it's already the third most used web framework, and apparently the fastest growing one: https://www.jetbrains.com/lp/python-developers-survey-2020/.

It was also recently recommended by ThoughtWorks for the enterprises: https://www.thoughtworks.com/radar/languages-and-frameworks?blipid=202104087

And it's currently used in lots of organizations, some widely known, chances are your orgs already use it in some way.

Pydantic, in very short, is a library that looks a lot like dataclasses (and also supports them), but it uses the same type annotations not only for type hints, but also for data validation, serialization (e.g. to JSON) and documentation (JSON Schema).

The key feature of FastAPI (thanks to Pydantic) is using the same type annotations for _more_ than just type hinting: data validation, serialization, and documentation. All those features are provided by default when building an API with FastAPI and Pydantic.

Typer is a library for building CLIs, based on Click, but using the same ideas from type annotations, from FastAPI and Pydantic.

## Why PEP 649

You can read Samuel's message to this mailing list here: https://mail.python.org/archives/list/python-dev@python.org/thread/7VMJWFGHVXDSRQFHMXVJKDDOVT47B54T/

And a longer discussion of how PEP 563 affects Pydantic here: https://github.com/samuelcolvin/pydantic/issues/2678

He has done most of the work to support these additional features from type annotations. So he would have the deepest insight into the tradeoffs/issues.

>From my point of view, just being able to use local variables in Pydantic models would be enough to justify PEP 649. With PEP 563, if a developer decides to create a Pydantic model inside a function (say a factory function) they would probably get an error. And it would probably not be obvious that they have to declare the model in the top level of the module.

The main feature of FastAPI and Pydantic is that they are very easy/intuitive to use. People from many backgrounds are now quickly and efficiently building APIs with best practices.

I've read some isolated comments of people that were against type annotations in general, saying that these tools justify adopting them. And I've also seen comments from people coming from other languages and fields, and adopting Python just to be able to use these tools.

Many of these developers are not Python experts, and supporting them and their intuition as much as possible when using these tools would help towards the PSF goal to:

> [...] support and facilitate the growth of a diverse and international community of Python programmers.

## Community support

To avoid asking people to spam here, Samuel and I are collecting "likes" in:

* This tweet: https://twitter.com/tiangolo/status/1382800928982642692
* This issue: https://github.com/samuelcolvin/pydantic/issues/2678

I just sent that tweet, I expect/hope it will collect some likes in support by the time you see it.

## Questions

I'm not very familiar with the internals of Python, and I'm not sure how the new syntax for `Union`s using the vertical bar character ("pipe", "|") work.

But would PEP 649 still support things like this?:

def run(arg: int | str = 0): pass

And would it be inspectable at runtime?

## Additional comments

The original author of PEP 563 was ?ukasz Langa.

I was recently chatting with him about Typer and annotations. And he showed interest, support, and help.

I think he probably knows the benefits of the way these libraries use type annotations and I would like/hope to read his opinion on all this.

Or alternatively, any possible ideas for how to handle these things in tools like Pydantic.
_______________________________________________
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-leave@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/J7YKCCCV2GL7HUDKG7DX4INAT65SXYRF/
Code of Conduct: http://python.org/psf/codeofconduct/

On 4/15/21 2:02 PM, Sebastián Ramírez wrote:
> ## Questions
>
> I'm not very familiar with the internals of Python, and I'm not sure how the new syntax for `Union`s using the vertical bar character ("pipe", "|") work.
>
> But would PEP 649 still support things like this?:
>
> def run(arg: int | str = 0): pass
>
> And would it be inspectable at runtime?


As far as I can tell, absolutely PEP 649 would support this feature. 
Under the covers, all PEP 649 is really doing is changing the
destination that annotation expressions get compiled to.  So anything
that works in an annotation with "stock" semantics would work fine with
PEP 649 semantics too, barring the exceptions specifically listed in the
PEP (e.g. annotations defined in conditionals, walrus operator, etc).


Cheers,


//arry/

>
> I will read PEP 649 implementation to find missing optimizations other
> than GH-25419 and GH-23056.
>

I found each "__co_annotation__" has own name like "func0.__co_annotation__".
It increased pyc size a little.
I created a draft pull request for cherry-picking GH-25419 and
GH-23056 and using just "__co_annotation__" as a name.
https://github.com/larryhastings/co_annotations/pull/9/commits/48a99e0aafa2dd006d72194bc1d7d47443900502

```
# previous result
$ ./python ~/ann_test.py 3
code size: 204963 bytes
memory: 209257 bytes
unmarshal: avg: 587.336ms +/-2.073ms
exec: avg: 97.056ms +/-0.046ms

# Use single name
$ ./python ~/ann_test.py 3
code size: 182088 bytes
memory: 209257 bytes
unmarshal: avg: 539.841ms +/-0.227ms
exec: avg: 98.351ms +/-0.064ms
```

It reduced code size and unmarshal time by 10%.
I confirmed GH-25419 and GH-23056 works very well. All same constants
are shared.

Unmarshal time is still slow. It is caused by unmarshaling code object.
But I don't know where is the bottleneck: parsing marshal file, or
creating code object.

---

Then, I tried to measure method annotation overhead.
Code: https://gist.github.com/methane/abb509e5f781cc4a103cc450e1e7925d#file-ann_test_method-py
Result:

```
# No annotation
$ ./python ~/ann_test_method.py 0
code size: 113019 bytes
memory: 256008 bytes
unmarshal: avg: 336.665ms +/-6.185ms
exec: avg: 176.791ms +/-3.067ms

# PEP 563
$ ./python ~/ann_test_method.py 2
code size: 120532 bytes
memory: 269004 bytes
unmarshal: avg: 348.285ms +/-0.102ms
exec: avg: 176.933ms +/-4.343ms

# PEP 649 (all optimization included)
$ ./python ~/ann_test_method.py 3
code size: 196135 bytes
memory: 436565 bytes
unmarshal: avg: 579.680ms +/-0.147ms
exec: avg: 259.781ms +/-7.087ms
```

PEP 563 vs 649
* code size: +63%
* memory: +62%
* import time: +60%

PEP 649 annotation overhead (compared with no annotation):
* code size: +83 byte/method
* memory: +180 byte/method
* import time: +326 us/method

It is disappointing because having thousands methods is very common
for web applications.

Unlike simple function case, PEP 649 creates function object instead
of code object for __co_annotation__ of methods.
It cause this overhead. Can we avoid creating functions for each annotation?

--
Inada Naoki <songofacandy@gmail.com>
_______________________________________________
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-leave@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/4KUXC373ZOP7YHZI6N4NKOOOB3DCL7NW/
Code of Conduct: http://python.org/psf/codeofconduct/

On 4/15/21 9:24 PM, Inada Naoki wrote:
> Unlike simple function case, PEP 649 creates function object instead
> of code object for __co_annotation__ of methods.
> It cause this overhead. Can we avoid creating functions for each annotation?


As the implementation of PEP 649 currently stands, there are two reasons
why the compiler might pre-bind the __co_annotations__ code object to a
function, instead of simply storing the code object:

* If the annotations refer to a closure ("freevars" is nonzero), or
* If the annotations /possibly/ refer to a class variable (the
annotations code object contains either LOAD_NAME or LOAD_CLASSDEREF).

If the annotations refer to a closure, then the code object also needs
to be bound with the "closure" tuple.  If the annotations possibly refer
to a class variable, then the code object also needs to be bound with
the current "f_locals" dict.  (Both could be true.)

Unfortunately, when generating annotations on a method, references to
builtins (e.g. "int", "str") seem to generate LOAD_NAME instructions
instead of LOAD_GLOBAL.  Which means pre-binding the function happens
pretty often for methods.  I believe in your benchmark it will happen
every time.  There's a lot of code, and a lot of runtime data
structures, inside compile.c and symtable.c behind the compiler's
decision about whether something is NAME vs GLOBAL vs DEREF etc, and I
wasn't comfortable with seeing if I could fix it.

Anyway I assume it wasn't "fixable".  The compiler would presumably
already prefer to generate LOAD_GLOBAL vs LOAD_NAME, because LOAD_GLOBAL
would be cheaper every time for a global or builtin.  The fact that it
already doesn't do so implies that it can't.


At the moment I have only one idea for a possible optimization, as
follows.  Instead of binding the function object immediately, it /might/
be cheaper to write the needed values into a tuple, then only actually
bind the function object on demand (like normal).

I haven't tried this because I assumed the difference at runtime would
be negligible.  On one hand, you're creating a function object; on the
other you're creating a tuple.  Either way you're creating an object at
runtime, and I assumed that bound functions weren't /that/ much more
expensive than tuples.  Of course I could be very wrong about that.

The other thing is, it would be a lot of work to even try the
experiment.  Also, it's an optimization, and I was more concerned with
correctness... and getting it done and getting this discussion underway.


What follows are my specific thoughts about how to implement this
optimization.

In this scenario, the logic in the compiler that generates the code
object would change to something like this:

has_closure = co.co_freevars != 0
has_load_name = co.co_code does not contain LOAD_NAME or
LOAD_CLASSDEREF bytecodes
if not (has_closure or has_load_name):
    co_ann = co
elif has_closure and (not has_load_name):
    co_ann = (co, freevars)
elif (not has_closure) and has_load_name:
    co_ann = (co, f_locals)
else:
    co_ann = (co, freevars, f_locals)
setattr(o, "__co_annotations__", co_ann)

(The compiler would have to generate instructions creating the tuple and
setting its members, then storing the resulting object on the object
with the annotations.)

Sadly, we can't pre-create this "co_ann" tuple as a constant and store
it in the .pyc file, because the whole point of the tuple is to contain
one or more objects only created at runtime.


The code implementing __co_annotations__ in the three objects (function,
class, module) would examine the object it got.  If it was a code
object, it would bind it; if it was a tuple, it would unpack the tuple
and use the values based on their type:

// co_ann = internal storage for __co_annotations__
if isinstance(co_ann, FunctionType) or (co_ann == None):
    return co_ann
co, freevars, locals = None
if isinstance(co_ann, CodeType):
    co = co_ann
else:
    assert isinstance(co_ann, tuple)
    assert 1 <= len(co_ann) <= 3
    for o in co_ann:
        if isinstance(o, CodeObject):
            assert not co
            co = o
        elif isinstance(o, tuple):
            assert not freevars
            freevars = o
        elif isinstance(o, dict):
            assert not locals
            locals = o
        else:
            raise ValueError(f"illegal value in co_annotations
tuple: {o!r}")
co_ann = make_function(co, freevars=freevars, locals=locals)
return co_ann


If you experiment with this approach, I'd be glad to answer questions
about it, either here or on Github, etc.


Cheers,


//arry/

On Fri, 16 Apr 2021, 3:14 pm Larry Hastings, <larry@hastings.org> wrote:

>
> Anyway I assume it wasn't "fixable". The compiler would presumably
> already prefer to generate LOAD_GLOBAL vs LOAD_NAME, because LOAD_GLOBAL
> would be cheaper every time for a global or builtin. The fact that it
> already doesn't do so implies that it can't.
>

Metaclass __prepare__ methods can inject names into the class namespace
that the compiler doesn't know about, so yeah, it unfortunately has to be
conservative and use LOAD_NAME in class level code.

Cheers,
Nick.

>
>

El sáb, 17 abr 2021 a las 8:30, Nick Coghlan (<ncoghlan@gmail.com>)
escribió:

>
>
> On Fri, 16 Apr 2021, 3:14 pm Larry Hastings, <larry@hastings.org> wrote:
>
>>
>> Anyway I assume it wasn't "fixable". The compiler would presumably
>> already prefer to generate LOAD_GLOBAL vs LOAD_NAME, because LOAD_GLOBAL
>> would be cheaper every time for a global or builtin. The fact that it
>> already doesn't do so implies that it can't.
>>
>
> Metaclass __prepare__ methods can inject names into the class namespace
> that the compiler doesn't know about, so yeah, it unfortunately has to be
> conservative and use LOAD_NAME in class level code.
>
> But of course, most metaclasses don't. I wonder if there are cases where
the compiler can statically figure out that there are no metaclass
shenanigans going on, and emit LOAD_GLOBAL anyway. It seems safe at least
when the class has no base classes and no metaclass=.


> Cheers,
> Nick.
>
>>
>> _______________________________________________
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-leave@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-dev@python.org/message/IZJYDHWJNMMMICUE32M3O7DGMSMVIOQ3/
> Code of Conduct: http://python.org/psf/codeofconduct/
>

On Sun, 18 Apr 2021, 1:59 am Jelle Zijlstra, <jelle.zijlstra@gmail.com>
wrote:

> El sáb, 17 abr 2021 a las 8:30, Nick Coghlan (<ncoghlan@gmail.com>)
> escribió:.
>
>>
>> Metaclass __prepare__ methods can inject names into the class namespace
>> that the compiler doesn't know about, so yeah, it unfortunately has to be
>> conservative and use LOAD_NAME in class level code.
>>
>> But of course, most metaclasses don't. I wonder if there are cases where
> the compiler can statically figure out that there are no metaclass
> shenanigans going on, and emit LOAD_GLOBAL anyway. It seems safe at least
> when the class has no base classes and no metaclass=.
>

Aye, that particular case is one the symtable pass could at least
theoretically identify.

As soon as there is a name to resolve in the class header, though, it's no
longer safe for the compiler to make assumptions :(

Cheers,
Nick.

>


>>