Mailing List Archive

Paul Moore wrote:
> More hazy memories here, but I think the original proposal left open
> the possibility of annotations not being types at all - for example,
> being docstrings for the arguments, or option names for a "function
> call to CLI" tool, etc.

Absolutely.

While it was clear that Guido's own use cases were about typing, annotations were explicitly not limited to typing, which is one reason why some of the later changes have felt to some people like bait and switch. Maybe it is already too late to avoid that.

> ... the expectation was that annotations
> would be *types*,

Even from the start, it was assumed that they would be objects. (Specifically types was expected to be common, but not universal.) The particular way strings are being substituted for evaluated objects has sometimes reminded me of raising a string instead of an exception class/object. It will work, but it can seem sloppy, and it can be annoying if you were assuming otherwise and suddenly have to add a bunch of evals. (That said, I haven't yet been sufficiently motivated to even tease out exactly what the problems are, let alone to propose an alternative that also satisfies the typing fans -- in part because it feels like the obvious optimization is to just not run typing, and it isn't clear what middle grounds are generally worthwhile.)

>... personally, I have the same discomfort
> about using explicit string annotations for forward references, it
> feels like I'm not declaring a "proper type".
> If what I say above is right, the debate here isn't about whether
> annotations "are for types", but rather about whether reading the
> types in annotations and using them to affect behaviour *at runtime*
> is a legitimate use of annotations.

I see that as a second dispute, which I had previously missed. I think you're right, though. On the other hand, I'm not sure the solution to both isn't just a helper function that does the 2nd-pass resolution -- preferably without requiring that all the rest of typing be imported, since even the people who want to use the typing package agree that importing it is not lightweight.

> ... I lurk on the typing-sig, and from an outsider's perspective, the
> participants seem to be almost entirely designers or heavy users of
> static type checkers. That gives a certain emphasis to the proposals
> coming from that group.

At times, it sort of reminds me of OWL and "Semantic Web". There are plenty of people who will want to use annotations as a tool, but won't be willing to wade through what can feel like "How many angels can dance on the head of a pin?" discussions. That said, I'm not sure how to best reach people who just want a rough-and-ready usually-good-enough tool.

-jJ
_______________________________________________
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-leave@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/JL3AQ4UVEVWXYTRFAVGVHNT23W2NCUDI/
Code of Conduct: http://python.org/psf/codeofconduct/

Steven D'Aprano wrote:
> On Sat, Nov 20, 2021 at 11:46:56PM -0800, Christopher Barker wrote:

> Maybe PEP 563 could include a decorator in the typing module to
> destringify all the annotations in a class or function?

If it were in an annotations module, that would probably be sufficient.

If it is in typing, then it is a very heavyweight dependency -- heavy enough that even the people actually using that module for development (and not for production runs) are worried about the costs. If the costs of the typing module are that high, it is not acceptable to impose them on people not otherwise using the module.

-jJ
_______________________________________________
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-leave@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/ZQHP24T2PKRTDBGZ36KLBHLLOKITP5ON/
Code of Conduct: http://python.org/psf/codeofconduct/

On Fri, Nov 26, 2021 at 5:47 PM Jim J. Jewett <jimjjewett@gmail.com> wrote:

> Steven D'Aprano wrote:
> > Maybe PEP 563 could include a decorator in the typing module to
> > destringify all the annotations in a class or function?
>
> If it were in an annotations module, that would probably be sufficient.
>
> If it is in typing, then it is a very heavyweight dependency --


As of Py 3.10 there is:

inspect.get_annotations

one could write a decorator around that, but I think the real question is
when do you want the annotations to be "finalized"?

-CHB



heavy enough that even the people actually using that module for
> development (and not for production runs) are worried about the costs. If
> the costs of the typing module are that high, it is not acceptable to
> impose them on people not otherwise using the module.
>
> -jJ
> _______________________________________________
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-leave@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-dev@python.org/message/ZQHP24T2PKRTDBGZ36KLBHLLOKITP5ON/
> Code of Conduct: http://python.org/psf/codeofconduct/
>

I hope my understanding of where the SC’s debate about this currently sits isn’t a misrepresentation, but since I sent the email on behalf of the SC, let me try to clarify what I was trying to say.

Aside: A little insight into how the SC works. For communications like this, after whatever debate we have in our live weekly meetings (and sometimes off-line discussions in Slack and email), one of us volunteers to write a draft of the email. We then all comment and wordsmith until we’re in unanimous agreement that it’s ready to go out. Then usually the person who wrote it will send the email. We all have slightly different styles, but I usually sign it with my name “on behalf of the Steering Council”. That’s my way of saying that the communication comes from all of us and represents the SC’s opinion, but it generally has the written style of the original drafter.

> On Nov 25, 2021, at 13:41, Christopher Barker <pythonchb@gmail.com> wrote:
> From Barry's email:
> """
> This is also a call to you, the folks who both care deeply about typing in Python, and have a solid understanding of the subject matter (or willingness to learn) to help us! There are use cases we don’t know about, and unclear requirements from both the static and dynamic typing users.
> """
> I don't care deeply about typing, and I'm not a static or dynamic typing user. So this implies that a decision has been made that annotations are for typing, and typing only.

Practically speaking, I think that’s current reality: annotations are for typing, be they static type checking or dynamic runtime uses. I was inelegantly saying that even with that narrower interpretation of annotations, there are use cases, interactions, side-effects, semantics, and implications that we do not fully understand. PEP 563 and 649 have visible effects that even within that domain can have important side effects. For example, PEP 563’s loss of local scope, which even “de-stringify-ing” can’t recover. This is what we need help with.

> (not that you're not allowed to use for anything else, of course you are, but that other uses won;t be taken into account when designing the new interface)
>
> But I have never seen that clearly stated anywhere. The closest is from PEP 563, where it says:
>
> """
> With this in mind, uses for annotations incompatible with the aforementioned PEPs should be considered deprecated.
> """
>
> Which pretty much makes the point, but it's a bit subtle -- what does "incompatible' mean?

You make a good point. I agree that while all the signs are there for “annotations are for typing”, this has never been explicitly or sufficiently codified, and I’ve been proposing that Someone write a PEP that makes this an official pronouncement. That someone may very well be a person on the SC, but given the timing of elections, it’s likely to fall to the next SC, if they still agree with that position! :-D

My recollection of the history of annotations falls somewhere between Greg’s and Guido’s. Annotations as a feature were inspired by the typing use case (with no decision at the time whether those were to be static or runtime checks), but at the same time allowing for experimentation for other use cases. Over time, annotations-for-typing clearly won the mindset and became the predominant use case. Personally, I was strongly against type annotations because of my experience in other languages, but over time I was also won over. From library documentation, to complex code bases, to the transient nature of contributors, type annotations are pretty compelling, and most (but not all) of my worries really didn’t come to fruition.

We can lament the non-typing use of annotations, but I think that horse is out of the barn and I don’t know how you would resolve conflicts of use for typing and non-typing annotations. It’s been a slow boil, with no definitive pronouncement, and that needs to be fixed, but I think it's just acknowledging reality. That’s my personal opinion.

> well, yes. The issue is that, intended or not, typing is making it's way into Python culture. As an instructor of beginning python users, I am unsure at this point when to introduce type annotations.
>
> What is their role? Up to today, I have treated them as an advanced feature, useful for "complex codebases". But there are any number of examples springing up on the internet, to the point where many students now think they are "best practice", if not actually required.

This is an important observation. As much as I’m in the "type annotations are good” crowd now, I still think they should always be optional. Python’s use is so broad these days, I for one don’t want to have to add annotations to every bit of Python I write.

-Barry

On Nov 26, 2021, at 01:13, Paul Moore <p.f.moore@gmail.com> wrote:

> I'd therefore interpret Barry's plea as being for *anyone* with a use
> for annotations to provide their feedback (at least, anyone who
> accepts that annotations are types), with particular emphasis on
> people who want to use the types declared in annotations to affect
> runtime behaviour, as that's the most under-represented group at the
> moment (and it's not clear whether it's under-represented because
> there aren't many such uses, or because the users aren't being heard
> from).

Spot on.

-Barry

On 11/29/21 2:56 PM, Barry Warsaw wrote:
> PEP 563 and 649 have visible effects that even within that domain can have important side effects. For example, PEP 563’s loss of local scope, which even “de-stringify-ing” can’t recover. This is what we need help with.

Well, sure.  If PEP 563 and 649 didn't have visible effects, there'd be
no point in doing them.  That said, I suggest 649 does a lovely job of
avoiding /undesirable/ side-effects.

Sure, 649 has observable side effects.  For example, you can detect
whether or not 649 is active by rebinding a name after it's used in an
annotation but before examining that annotation at runtime.  This seems
harmless--unlikely to happen in production code, and easily remedied if
someone did trip over it.

A more credible side effect: if you use an undefined name in an
annotation, you won't notice at compile-time.  Now, this is actually
649's major feature!  But there are also scenarios where this feature
could cover up a bug, like if you misspell a name--you won't notice
until you examine the annotation at runtime (or, more likely, until you
run your static analyzer).  563 has this same behavior--and it wasn't
enough to prevent 563 being accepted.  So I assume this wouldn't be
enough to prevent accepting 649 either.

649 has effects on memory usage and performance, but honestly I'm not
worried about these.  I don't think the memory usage and performance of
the prototype were particularly bad.  Anyway, as I've said many times:
we should figure out the semantics we want first, and then we can worry
about optimization.  The Python core dev community has no end of smart
people who love optimizing things--I'm sure if 649 was accepted and
merged, the optimizations would start rolling in.

Then of course there are also things 649 simply doesn't do, e.g. resolve
the "if TYPE_CHECKING" situation.  But it's not appropriate to call that
a "side effect" per se.

And that's my list.  If anybody knows of other visible side effects from
649, naturally you should contact the SC.  And/or me, if you think we
could change 649 to mitigate it without losing its major features.


Happy holidays,


//arry/

Here's a use case for runtime type annotations: dataclasses use
annotations to define fields. With the exception of ClassVar, the actual
type is ignored. There's code in dataclasses.py to deal with stringized
annotations, specifically just looking for ClassVar. I'd like to see
this special case go away, assuming 649 is accepted.

And while dataclasses itself doesn't care about the types, apparently
users do, and want to be able to call get_type_hints on the annotations.
Many people actually want dataclasses to call this for them, but I've
resisted. See https://bugs.python.org/issue36643

There are plenty of bugs around this issue, some of which have been
fixed. But there are also at least a couple that haven't been fixed, and
I'm not sure they will be. I'm hoping that 649 gets accepted and I don't
have to worry about the problem at all. Here's an example of an open bug
around get_type_hints with dataclasses: https://bugs.python.org/issue45524

Eric

On 11/29/2021 6:00 PM, Barry Warsaw wrote:
> On Nov 26, 2021, at 01:13, Paul Moore <p.f.moore@gmail.com> wrote:
>
>> I'd therefore interpret Barry's plea as being for *anyone* with a use
>> for annotations to provide their feedback (at least, anyone who
>> accepts that annotations are types), with particular emphasis on
>> people who want to use the types declared in annotations to affect
>> runtime behaviour, as that's the most under-represented group at the
>> moment (and it's not clear whether it's under-represented because
>> there aren't many such uses, or because the users aren't being heard
>> from).
> Spot on.
>
> -Barry
>
>
> _______________________________________________
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-leave@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/4522JDS2ZOYXXFGPVXRGJ67IB3YLCEFM/
> Code of Conduct: http://python.org/psf/codeofconduct/
_______________________________________________
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-leave@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/ENVEVQUEMEDKT2YUI2LQOBZDJC2YMNQ5/
Code of Conduct: http://python.org/psf/codeofconduct/

On 11/29/2021 5:56 PM, Barry Warsaw wrote:

> On Nov 25, 2021, at 13:41, Christopher Barker <pythonchb@gmail.com> wrote:
>> What is their role? Up to today, I have treated them as an advanced feature, useful for "complex codebases". But there are any number of examples springing up on the internet, to the point where many students now think they are "best practice", if not actually required.

> This is an important observation. As much as I’m in the "type annotations are good” crowd now, I still think they should always be optional. Python’s use is so broad these days, I for one don’t want to have to add annotations to every bit of Python I write.

Maybe it should be reiterated with whatever decision comes forth that

>>> def muladd(x, y, z):
... return x * (y+z)
...
>>> muladd(3.1459, 87.33, 2.7e2)
1124.124447
>>> muladd(3, 5, 7)
36
>>> from fractions import Fraction as Fr
>>> muladd(Fr(22, 7), 87, Fr(2714, 100))
Fraction(62777, 175)
>>> muladd(3, 'hel', 'lo ')
'hello hello hello '

and other duck-typed code will always be legal, idiomatic, and even
expected as good practice for beginner, informal, exploratory, and
similar python code.

--
Terry Jan Reedy
_______________________________________________
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-leave@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/R2HLTW5DZQGKCMJTTFTAKHX7EYQWNROG/
Code of Conduct: http://python.org/psf/codeofconduct/

On Mon, Nov 29, 2021 at 07:56:16PM -0500, Terry Reedy wrote:

> Maybe it should be reiterated with whatever decision comes forth that
>
> >>> def muladd(x, y, z):
> ... return x * (y+z)
> ...

[...]

> and other duck-typed code will always be legal, idiomatic, and even
> expected as good practice for beginner, informal, exploratory, and
> similar python code.

Many of the typing-related PEP comes with such a disclaimer, listed as
"Non-goals". For example:

https://www.python.org/dev/peps/pep-0484/#non-goals

?ukasz's stringified annotations PEP has a non-goals section:

https://www.python.org/dev/peps/pep-0563/#non-goals

Larry's deferred evaluation PEP does not:

https://www.python.org/dev/peps/pep-0649/

but I don't think we should hold it against PEP-649. It's not trying to
sneakily sneak mandatory static typechecking in by the back door like
some sort of sneaking sneak :-)

--
Steve
_______________________________________________
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-leave@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/XRBNYBEK6ZBZUC2KIDBGEYKYXZ2CBF4C/
Code of Conduct: http://python.org/psf/codeofconduct/

On Mon, Nov 29, 2021 at 5:01 PM Terry Reedy <tjreedy@udel.edu> wrote:

> On 11/29/2021 5:56 PM, Barry Warsaw wrote:
>
> > On Nov 25, 2021, at 13:41, Christopher Barker <pythonchb@gmail.com>
> wrote:
> >> What is their role? Up to today, I have treated them as an advanced
> feature, useful for "complex codebases". But there are any number of
> examples springing up on the internet, to the point where many students now
> think they are "best practice", if not actually required.
>
> > This is an important observation. As much as I’m in the "type
> annotations are good” crowd now, I still think they should always be
> optional. Python’s use is so broad these days, I for one don’t want to
> have to add annotations to every bit of Python I write.
>
> Maybe it should be reiterated with whatever decision comes forth that
>
> >>> def muladd(x, y, z):
> ... return x * (y+z)
> ...
> >>> muladd(3.1459, 87.33, 2.7e2)
> 1124.124447
> >>> muladd(3, 5, 7)
> 36
> >>> from fractions import Fraction as Fr
> >>> muladd(Fr(22, 7), 87, Fr(2714, 100))
> Fraction(62777, 175)
> >>> muladd(3, 'hel', 'lo ')
> 'hello hello hello '
>
> and other duck-typed code will always be legal, idiomatic, and even
> expected as good practice for beginner, informal, exploratory, and
> similar python code.
>

Why would it need to be reiterated? Are there really people who believe
that such code would become invalid? AFAIK *everybody* here agrees that
this should stay valid. So who would we be reiterating it for?

(Yes, several static type checkers have options that cause the checker to
complain about unannotated code. But that's not the default behavior and
running a static checker is a choice, like running a linter. Three-space
indents or capitalized function names are never going to be disallowed
either, even though PEP 8 says that's not how you ought to code.)

--
--Guido van Rossum (python.org/~guido)
*Pronouns: he/him **(why is my pronoun here?)*
<http://feministing.com/2015/02/03/how-using-they-as-a-singular-pronoun-can-change-the-world/>

On 11/29/2021 8:16 PM, Guido van Rossum wrote:
> On Mon, Nov 29, 2021 at 5:01 PM Terry Reedy <tjreedy@udel.edu
> <mailto:tjreedy@udel.edu>> wrote:
>
> On 11/29/2021 5:56 PM, Barry Warsaw wrote:
>
> > On Nov 25, 2021, at 13:41, Christopher Barker
> <pythonchb@gmail.com <mailto:pythonchb@gmail.com>> wrote:
> >> What is their role? Up to today, I have treated them as an
> advanced feature, useful for "complex codebases".


> But there are any
> number of examples springing up on the internet, to the point where
> many students now think they are "best practice", if not actually
> required.

Chris, can you say anything more about why people would get such a
mis-impression?

> > This is an important observation.  As much as I’m in the "type
> annotations are good” crowd now, I still think they should always be
> optional.  Python’s use is so broad these days, I for one don’t want
> to have to add annotations to every bit of Python I write.
>
> Maybe it should be reiterated with whatever decision comes forth that
>
>  >>> def muladd(x, y, z):
> ...     return x * (y+z)
> ...
>  >>> muladd(3.1459, 87.33, 2.7e2)
> 1124.124447
>  >>> muladd(3, 5, 7)
> 36
>  >>> from fractions import Fraction as Fr
>  >>> muladd(Fr(22, 7), 87, Fr(2714, 100))
> Fraction(62777, 175)
>  >>> muladd(3, 'hel', 'lo ')
> 'hello hello hello '
>
> and other duck-typed code will always be legal, idiomatic, and even
> expected as good practice for beginner, informal, exploratory, and
> similar python code.
>

> Why would it need to be reiterated? Are there really people who believe
> that such code would become invalid? AFAIK *everybody* here agrees that
> this should stay valid. So who would we be reiterating it for?

The people that Chris B. referred to, and their 'sources'. But we need
more info to know how to best counter such wrong beliefs.

--
Terry Jan Reedy
_______________________________________________
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-leave@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/KLJ7UCGAWQ7X5EW6WPHBIH2XC3DIVEFZ/
Code of Conduct: http://python.org/psf/codeofconduct/

On Nov 29, 2021, at 15:57, Larry Hastings <larry@hastings.org> wrote:
>
> On 11/29/21 2:56 PM, Barry Warsaw wrote:
>> PEP 563 and 649 have visible effects that even within that domain can have important side effects. For example, PEP 563’s loss of local scope, which even “de-stringify-ing” can’t recover. This is what we need help with.
> Well, sure. If PEP 563 and 649 didn't have visible effects, there'd be no point in doing them. That said, I suggest 649 does a lovely job of avoiding undesirable side-effects.
>
> Sure, 649 has observable side effects. For example, you can detect whether or not 649 is active by rebinding a name after it's used in an annotation but before examining that annotation at runtime. This seems harmless--unlikely to happen in production code, and easily remedied if someone did trip over it.

PEP 563 also has visible side effects, but the real implication of them wasn’t well understood at the time the PEP was approved. Or maybe it was and we underestimated the impact on certain important users and use cases. I’m particularly wary about getting into that same (um) pickle again.

> A more credible side effect: if you use an undefined name in an annotation, you won't notice at compile-time. Now, this is actually 649's major feature! But there are also scenarios where this feature could cover up a bug, like if you misspell a name--you won't notice until you examine the annotation at runtime (or, more likely, until you run your static analyzer). 563 has this same behavior--and it wasn't enough to prevent 563 being accepted. So I assume this wouldn't be enough to prevent accepting 649 either.

How common is it for code to examine annotations at run time, outside of specialized libraries that have to be more defensive anyway (especially given they already have to be defensive in light of 563)?

> 649 has effects on memory usage and performance, but honestly I'm not worried about these. I don't think the memory usage and performance of the prototype were particularly bad. Anyway, as I've said many times: we should figure out the semantics we want first, and then we can worry about optimization. The Python core dev community has no end of smart people who love optimizing things--I'm sure if 649 was accepted and merged, the optimizations would start rolling in.

I agree about that priority (semantics first, optimization later), as long as those semantics don’t make such future optimizations impossible, or severely degrade important performance constraints such as import times.

> Then of course there are also things 649 simply doesn't do, e.g. resolve the "if TYPE_CHECKING" situation. But it's not appropriate to call that a "side effect" per se.

Although, it would be nice to have a comprehensive solution to this and other situations too, even if they happen outside of PEP 563/649 discussion…

> And that's my list. If anybody knows of other visible side effects from 649, naturally you should contact the SC. And/or me, if you think we could change 649 to mitigate it without losing its major features.

That list is exactly what I'm (we’re?) looking for.

-Barry

On 11/30/2021 1:16 AM, Guido van Rossum wrote:
> Why would it need to be reiterated? Are there really people who believe
> that such code would become invalid? AFAIK *everybody* here agrees that
> this should stay valid. So who would we be reiterating it for?
>
> (Yes, several static type checkers have options that cause the checker
> to complain about unannotated code. But that's not the default behavior
> and running a static checker is a choice, like running a linter.
> Three-space indents or capitalized function names are never going to be
> disallowed either, even though PEP 8 says that's not how you ought to code.)

I know it sounds like a straw-man argument, but I don't think it is. I
just think the non-straw-men making the argument aren't "important"
enough to call out specifically - they are "random individual
programmers who have gotten the wrong impression from another language
and applied it to Python" rather than "influencers trying to change how
we code".[1]

And we can always say that the *language* does not require running the
type checker, but the reality for many developers is that the projects
and teams they are part of will require it. And often, they require it
because it's some kind of global "Best Practice" rather than because it
makes sense for their particular situation.

For example, I saw a snippet fly by on Twitter the other day (again, not
the fault of the individual who posted it, so I'm not
naming-and-shaming) showing how to use Black to format a big directory
of Jupyter notebooks. Convenient, perhaps, but the vast majority of
people with "big directories of Jupyter notebooks" are going to be much
happier if you just leave them alone! And yet, "The Community" is
suggesting we should format all of them in bulk.

THAT'S the kind of thing that also has been happening with typing, and
why some of us feel the need to publicly re-state things that are all
agreed upon within this group, but are struggling to be heard over the
public discourse on the topics. And this kind of reiteration is easier
when our official documents (PEPs, etc.) state it explicitly.

Cheers,
Steve


[1]: Okay, I will call out the Python Bytes podcast - which I've been on
a couple of times - as one that regularly opines on topics based on very
little information and says things more definitively than they've been
said anywhere else. They have a much bigger (and more easily influenced)
audience than python-dev.

I'll also say that what I see from LWN is generally very good in this
respect, their writers handle things well. Though I don't tend to read
or listen to either of these that often.
_______________________________________________
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-leave@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/SFLYLQO4YVQGV3BUKQO7KS6N367HNG62/
Code of Conduct: http://python.org/psf/codeofconduct/

On Thu, Nov 18, 2021 at 8:00 AM Barry Warsaw <barry@python.org> wrote:
>
> Does PEP 563 or 649 satisfy static and dynamic typing needs?
>
> In the interest of full transparency, we want to let the Python community know that the Steering Council continues to discuss PEP 563 (Postponed Evaluation of Annotations) and PEP 649 (Deferred Evaluation Of Annotations Using Descriptors). We have concluded that we lack enough detailed information to make a decision in favor of either PEP. As we see it, adopting either PEP 563 or PEP 649 as the default would be insufficient. They will not fully resolve the existing problems these PEPs intend to fix, will break some existing code, and likely don’t address all the use cases and requirements across the static and dynamic typing constituents. We are also uncertain as to the best migration path from the current state of affairs.
>
> Defer decision on PEP 563 and 649 in 3.11
>
> As such, at this time, the only reasonable path forward that the SC sees is to defer the decision in Python 3.11 again, essentially keeping the 3.10 status quo. We know that this is far from ideal, but it’s also the safest path since we can’t clearly make the situation better, and we don’t have confidence that either PEP solves the problems once and for all. Pragmatically, we don’t want to make the situation worse, and we really don’t want to find ourselves back here again in a couple of releases because we overlooked an important requirement for a set of users.
>

In my opinion, we can agree that we can not make PEP 563 default in
the future because it will break too many use cases.
Anyone against making a statement that "PEP 563 will never be the
default behavior"?

Then, we do not need to decide "PEP 563 or 649".
We can focus on whether we can replace "stock semantics + opt-in PEP
563" with PEP 649 or not.

Regards,
--
Inada Naoki <songofacandy@gmail.com>
_______________________________________________
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-leave@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/KCZKREGT66VE67MPT7NNJ5UPCACY6X7A/
Code of Conduct: http://python.org/psf/codeofconduct/

On Mon, Nov 29, 2021 at 8:17 PM Guido van Rossum <guido@python.org> wrote:

> Why would it need to be reiterated? Are there really people who believe
> that such code would become invalid? AFAIK *everybody* here agrees that
> this should stay valid. So who would we be reiterating it for?
>

I'm certainly not alone, among people on this list, in regard to the
following. But I teach a lot of people who are coders, but not necessarily
senior (in Python, or otherwise). I also mentor/lead such junior
programmers.

I find it quite common when people who haven't known Python for 20 years
see type annotations, they have trouble getting the optional and gradual
nature of them. Explaining that is certainly not impossible, nor even all
that difficult. However, it DOES need to be explained. Especially when
less experienced Pythonistas have worked with statically typed languages,
they often see code with type annotations and overgeneralize to their
requirement.

I don't know exactly what that means about *where* this needs to be
explained. Maybe just by trainers and authors. But maybe things in
official Python documentation as well, which seem more definitive.

--
Keeping medicines from the bloodstreams of the sick; food
from the bellies of the hungry; books from the hands of the
uneducated; technology from the underdeveloped; and putting
advocates of freedom in prisons. Intellectual property is
to the 21st century what the slave trade was to the 16th.

On 11/29/21 7:10 PM, Inada Naoki wrote:
> Anyone against making a statement that "PEP 563 will never be the
> default behavior"?

I think only the SC is empowered to make such a statement.


> Then, we do not need to decide "PEP 563 or 649".
> We can focus on whether we can replace "stock semantics + opt-in PEP
> 563" with PEP 649 or not.

I doubt the current status quo--keeping PEP 563 as optional behavior,
permanently--is viable long-term.  It causes problems for most runtime
uses of annotations, which we were collectively basically ignoring until
now.  As runtime use of annotations /and/ typing both become more
pervasive, these problems are only going to grow.


//arry/

I don't think the issue of how typing is making its way intot he community
is particular relevant to this thread, but I was directly asked a question,
so I'll answer it:

I don't know exactly where the impression is coming from that typing is a
best practice, but it's certainly creeping into example code, etc. So folks
will see examples of code that have nothing to do with typing, and probably
have no need for it, and there will be type annotations. Many (most?) folks
learn from examples more than targeted instruction, so as those examples
grow, so will the impression that it's an inherent part of Python.

David Mertz' experience is similar to mine as an instructor.

Another anecdotal example: I've been teaching an intro to Python class for
years, based on the same material, that we try to keep up to date. But
typing is not yet included. However, about a year ago, one of the other
instructors (that had not previously developed materials for the class)
added a new example / exercise to the "intro to classes" lesson. And that
example had type annotations. I ended up removing them, as I thought it
could be pretty confusing to students to see them for the first time with
no explanation (would they think it was something specific to classes?). I
wish I'd discussed it with the other instructor, I don't know why that was
done -- was it standard practice in their workplace? Seemed like a best
practice? I have no idea.

Another observation is that more and more library authors are being asked
to add type annotations -- maybe that IS a best practice for widely used
libraries, but that's one more piece of data.

Also: I know this isn't intentional, but Guido has an enormous influence on
the Python community -- so I suspect that the fact that he's taken an
active role in developing static typing has influenced how it's being
perceived.

I do think this is a topic for the community to grapple with, but not
really a Python-dev responsibility.

-CHB


--
Christopher Barker, PhD (Chris)

Python Language Consulting
- Teaching
- Scientific Software Development
- Desktop GUI and Web Development
- wxPython, numpy, scipy, Cython

On Tue, 30 Nov 2021 at 02:52, Steve Dower <steve.dower@python.org> wrote:
>
> THAT'S the kind of thing that also has been happening with typing, and
> why some of us feel the need to publicly re-state things that are all
> agreed upon within this group, but are struggling to be heard over the
> public discourse on the topics. And this kind of reiteration is easier
> when our official documents (PEPs, etc.) state it explicitly.

This is very definitely the case. There's a subtle (maybe not so
subtle, actually) and increasing pressure on projects to add typing.
Often with little or no justification beyond "you should", as if
having typing is a sort of "obvious best practice". Sometimes "because
it will make it easier for your users who use typing" is given as a
justification, but while that's fair, it's also a disturbing gradual
pressure for typing to extend everywhere, manifesting by making it
feel like not adding typing is somehow "not caring about your users".

Also, related to the question Terry raised, IMO it would be useful to
have a clear statement on code that *does* use type annotations, but
violates them at runtime. To be specific, is the following considered
as an error?

>>> def muladd(x: int, y: int, z: int) -> int:
... return x * (y+z)
...
>>> muladd(3.1459, 87.33, 2.7e2)
1124.124447

If (as in the current interpreter) typing is optional, this isn't an
error. Certainly, running mypy over it fails, but the point here is
that mypy is an external tool - the Python interpreter itself (and the
language definition) allows it. I think it would be useful to have a
clear message that this is intended, and won't be altered lightly.
Remember that the "correct" annotation here is either muladd(x: Any,
y: Any, z: Any) -> Any, or muladd(x: SomeMultiplyProtocol, y:
SomeAddProtocol, z: SomeAddProtocol) -> Any, neither of which is
particularly helpful...

Paul
_______________________________________________
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-leave@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/E36ILTCKFVW3MNFU753PKKWIFY2FWQXU/
Code of Conduct: http://python.org/psf/codeofconduct/

On Tue, Nov 30, 2021 at 8:19 PM Paul Moore <p.f.moore@gmail.com> wrote:
> Also, related to the question Terry raised, IMO it would be useful to
> have a clear statement on code that *does* use type annotations, but
> violates them at runtime. To be specific, is the following considered
> as an error?
>
> >>> def muladd(x: int, y: int, z: int) -> int:
> ... return x * (y+z)
> ...
> >>> muladd(3.1459, 87.33, 2.7e2)
> 1124.124447
>

My understanding is that it's precisely as wrong, or not wrong, as this:

def add(x, y):
"""Multiply two numbers"""
return x / y

To my mind, annotations are machine-readable metadata, with no
inherent "correctness" to them (from the language's point of view),
other than syntactically.

ChrisA
_______________________________________________
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-leave@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/54VRD5SJ3RQT6W4AX5WPIW2OUWZNWPZO/
Code of Conduct: http://python.org/psf/codeofconduct/

On Tue, 30 Nov 2021 at 09:37, Chris Angelico <rosuav@gmail.com> wrote:
>
> On Tue, Nov 30, 2021 at 8:19 PM Paul Moore <p.f.moore@gmail.com> wrote:
> > Also, related to the question Terry raised, IMO it would be useful to
> > have a clear statement on code that *does* use type annotations, but
> > violates them at runtime. To be specific, is the following considered
> > as an error?
> >
> > >>> def muladd(x: int, y: int, z: int) -> int:
> > ... return x * (y+z)
> > ...
> > >>> muladd(3.1459, 87.33, 2.7e2)
> > 1124.124447
> >
>
> My understanding is that it's precisely as wrong, or not wrong, as this:
>
> def add(x, y):
> """Multiply two numbers"""
> return x / y
>
> To my mind, annotations are machine-readable metadata, with no
> inherent "correctness" to them (from the language's point of view),
> other than syntactically.

That makes sense to me, and I'd support that being clearly stated as
the "official position".

Paul
_______________________________________________
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-leave@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/LUY2SXBGX4TQKTX6PE2FHYHABDCYY2IN/
Code of Conduct: http://python.org/psf/codeofconduct/

On Fri, 26 Nov 2021 at 09:14, Paul Moore <p.f.moore@gmail.com> wrote:

> I'd therefore interpret Barry's plea as being for *anyone* with a use
> for annotations to provide their feedback (at least, anyone who
> accepts that annotations are types), with particular emphasis on
> people who want to use the types declared in annotations to affect
> runtime behaviour, as that's the most under-represented group at the
> moment (and it's not clear whether it's under-represented because
> there aren't many such uses, or because the users aren't being heard
> from).


I don't know if this helps, but I'm a maintainer/creator of a Python library
that uses type annotations at runtime. The library is used to create GraphQL
APIs[1]. We took a lot of inspiration from dataclasses, our API looks like
this:

```python
@strawberry.type
class User:
name: str
```

The `strawberry.type` decorator reads the annotations and creates an object
with a list
of fields and types, but it doesn't try to resolve them (that happens later
to prevent issues
with circular dependencies and so on).

I remember the discussion around PEP 563 and how it would break libraries
using type annotations,
in our case things are fine since the types we define should be defined in
the global namespace[2]
anyway. I think the only issue we have with type annotations is when you
have circular references
inside your python code (where you'd use `if TYPE_CHECKING: import X`), we
"solved" that with
a custom class called `LazyType[type, module]`, that allows to specify a
type and where it comes
from, so our library can resolve the proper type, even if it was not
imported in the namespace of the
class using that type.

Sorry for the digression into some of the details of how the library works,
I hope it helps a bit
understanding the use case.

I think there's definitely use cases for type hints being used at runtime.
I personally love
the fact that we are allowed to do this, in other languages (like
TypeScript) you don't have
access to the types at runtime and it does prevent the creation of
libraries like dataclasses
or mine, which is a bit of a shame (you'd need to use code generation to
get a similar experience).

There's also other libraries that use type hints at runtime (not for
runtime type validation[3]):

- Typer - http://typer.tiangolo.com
- SQLModel - http://sqlmodel.tiangolo.com
- Beanie - https://github.com/roman-right/beanie/
- Pydantic - http://pydantic-docs.helpmanual.io
- odmantic - https://github.com/art049/odmantic/
- singledispatch -
https://docs.python.org/3/library/functools.html#functools.singledispatch


[1] If you're not familiar with GraphQL there's some info here:
https://graphql.org
[2] I did ask if there was a way to resolve types when having them
encapsulated into functions,
but that was a specific use case with tests and I was/am fine with
using global there

https://mail.python.org/archives/list/typing-sig@python.org/thread/SNKJB2U5S74TWGDWVD6FMXOP63WVIGDR/#A5JFR7RDE7NNKESM2DWYM3ZT5ZYGOLCF

[3] while I see runtime validation being something useful, I think it is
worth taking a look at
different use cases for type hints :)

--
Patrick Arminio

On 29 Nov 2021, at 23:56, Barry Warsaw wrote:

> [...]
>> (not that you're not allowed to use for anything else, of course you
>> are, but that other uses won;t be taken into account when designing
>> the new interface)
>>
>> But I have never seen that clearly stated anywhere. The closest is
>> from PEP 563, where it says:
>>
>> """
>> With this in mind, uses for annotations incompatible with the
>> aforementioned PEPs should be considered deprecated.
>> """
>>
>> Which pretty much makes the point, but it's a bit subtle -- what does
>> "incompatible' mean?
>
> You make a good point. I agree that while all the signs are there for
> “annotations are for typing”, this has never been explicitly or
> sufficiently codified, and I’ve been proposing that Someone write a
> PEP that makes this an official pronouncement. That someone may very
> well be a person on the SC, but given the timing of elections, it’s
> likely to fall to the next SC, if they still agree with that position!
> :-D
>
> My recollection of the history of annotations falls somewhere between
> Greg’s and Guido’s. Annotations as a feature were inspired by the
> typing use case (with no decision at the time whether those were to be
> static or runtime checks), but at the same time allowing for
> experimentation for other use cases. Over time,
> annotations-for-typing clearly won the mindset and became the
> predominant use case. Personally, I was strongly against type
> annotations because of my experience in other languages, but over time
> I was also won over. From library documentation, to complex code
> bases, to the transient nature of contributors, type annotations are
> pretty compelling, and most (but not all) of my worries really
> didn’t come to fruition.
>
> We can lament the non-typing use of annotations, but I think that
> horse is out of the barn and I don’t know how you would resolve
> conflicts of use for typing and non-typing annotations. It’s been a
> slow boil, with no definitive pronouncement, and that needs to be
> fixed, but I think it's just acknowledging reality. That’s my
> personal opinion.

But isn't that the reason why we have `typing.Annotated`, so that
annotations used for typing and annotations used for other purposes can
coexist?

An annotation used for typing only is:

```
>>> def f(x: int):
... return x
...
>>> f.__annotations__['x']
<class 'int'>
```

An annotation used for something else is:

```
>>> def f(x: 'something not typing related'):
... return x
...
>>> f.__annotations__['x']
'something not typing related'
```

`typing.Annotated` gives us both:

```
>>> from typing import *
>>> def f(x: Annotated[int, 'something not typing related']):
... return x
...
>>> f.__annotations__['x'].__args__[0]
<class 'int'>
>>> f.__annotations__['x'].__metadata__[0]
'something not typing related'
```

Granted, for someone who only wants to use annotations for their own
purpose, digging out the original not typing related stuff from
`typing.Annotated` is more work, but doable.

Or am I missing something here?

> [...]

Servus,
Walter

On Tue, Nov 30, 2021 at 09:17:13AM +0000, Paul Moore wrote:

> Also, related to the question Terry raised, IMO it would be useful to
> have a clear statement on code that *does* use type annotations, but
> violates them at runtime. To be specific, is the following considered
> as an error?
>
> >>> def muladd(x: int, y: int, z: int) -> int:
> ... return x * (y+z)
> ...
> >>> muladd(3.1459, 87.33, 2.7e2)
> 1124.124447

Isn't that just duck-typing? You've got a function that is documented as
taking arguments of one type (int), but actually will work with any
numeric type (and some non-numeric types) that quacks like an int.

muladd(2, 'ab', 'cd') # returns 'abcdabcd'

If you replaced the type annotations with a docstring that listed the
parameters x, y, z as int, would duck-typing be wrong? Maybe.

I don't think we can make any concrete claims here. Sometimes
duck-typing is perfectly fine. Sometimes its not. It depends on the
function's implementation and its semantics. Sometimes calling a
function with a duck-typed value seems to work fine but the result is
meaningless junk. (Garbage In, Garbage Out.)

I guess all we can say is that the Python language is agnostic and
neutral on this matter.


--
Steve
_______________________________________________
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-leave@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/W24VGDUH6F43C6VT4OUGERNEZFW223MP/
Code of Conduct: http://python.org/psf/codeofconduct/

On Tue, 30 Nov 2021 at 09:23, Paul Moore <p.f.moore@gmail.com> wrote:
>
> On Tue, 30 Nov 2021 at 02:52, Steve Dower <steve.dower@python.org> wrote:
> >
> > THAT'S the kind of thing that also has been happening with typing, and
> > why some of us feel the need to publicly re-state things that are all
> > agreed upon within this group, but are struggling to be heard over the
> > public discourse on the topics. And this kind of reiteration is easier
> > when our official documents (PEPs, etc.) state it explicitly.
>
> This is very definitely the case. There's a subtle (maybe not so
> subtle, actually) and increasing pressure on projects to add typing.
> Often with little or no justification beyond "you should", as if
> having typing is a sort of "obvious best practice". Sometimes "because
> it will make it easier for your users who use typing" is given as a
> justification, but while that's fair, it's also a disturbing gradual
> pressure for typing to extend everywhere, manifesting by making it
> feel like not adding typing is somehow "not caring about your users".

I suspect that a big source of this is editors that use typing for
autocompletion etc. Speaking as a vim user this is something that has
passed me by except that I have students who I see adding type hints
even though I deliberately haven't taught them anything about type
hints. I can only presume that some editor is doing this for them or
telling them that they need to do this (students often can't tell the
difference between editor warnings and actual errors).

Others have mentioned the pressure on libraries to adopt typing and
I've certainly noticed this with SymPy. I think type hints could be
good for SymPy for internal use but it seems that a lot of users want
it for external reasons that I don't always understand but that also
seems to come partly from editors as well.

Some people apparently want to add type hints that look completely
useless to me like
def f() -> Union[MyClass, Any]
As I understand it this does not give any meaningful information to a
type checker but it apparently makes vscode work better:
https://github.com/sympy/sympy/pull/22180#discussion_r718917577

Here in SymPy you can see a suggestion that type hints are needed
because otherwise vscode is slow and uses 4GB of memory just to load
documentation for SymPy functions:
https://github.com/sympy/sympy/issues/17945#issuecomment-928798977

Not to do with editors but also indicative is this PR whose title
implies that SymPy is "incompatible" with PEP 561 (a PEP that I had
never heard of before seeing the PR):
https://github.com/sympy/sympy/pull/22337

There are other open "issues" like this for SymPy where the
presumption is that not having type hints is now to be considered a
deficiency of the library regardless of whether the hints have any
benefit for internal use. I don't object to adding the hints but it's
a huge amount of work that someone would have to do and I don't want
to add useless/inaccurate hints temporarily (as some have suggested to
do).

--
Oscar
_______________________________________________
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-leave@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/YQ773X24AICPLPLIN742B25WNVZJHMRO/
Code of Conduct: http://python.org/psf/codeofconduct/

On Tue, 30 Nov 2021 at 12:39, Oscar Benjamin <oscar.j.benjamin@gmail.com> wrote:

> There are other open "issues" like this for SymPy where the
> presumption is that not having type hints is now to be considered a
> deficiency of the library regardless of whether the hints have any
> benefit for internal use. I don't object to adding the hints but it's
> a huge amount of work that someone would have to do and I don't want
> to add useless/inaccurate hints temporarily (as some have suggested to
> do).

This is precisely the sort of concern I have. Although it is awfully
tempting to passive-aggressively annotate everything as Any, just to
shut people up :-(

And to be clear, it's often very non-obvious how to annotate something
- in https://github.com/pfmoore/editables I basically gave up because
I couldn't work out how to write a maintainable annotation for an
argument that is "a Path, or something that can be passed to the Path
constructor to create a Path" (it's essentially impossible without
copy/pasting the argument annotation for the Path constructor). I also
spent a lot of time trying to deal with a typeshed bug. And even
worse, if I put the types in a .pyi file (which I want to do, because
I don't want the library to import typing at runtime because it's a
significant overhead for something that's supposed to be lightweight),
apparently mypy won't check them so I gain no benefit for the project
itself.

Anyway, we're *way* off topic now, and I doubt there's much that the
SC or python-dev can do to change the community view on typing,
anyway.

Paul
_______________________________________________
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-leave@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/4DUJM7MNJSJ3TOYYFILQCG4RT4CU6DME/
Code of Conduct: http://python.org/psf/codeofconduct/