Hi all,
I got pinged to voice my opinion on PEP 649 as the instigator of PEP 563. I'm sorry, this is long, and a separate thread, because it deals with three things:
- Goals set for PEP 563 and how it did in practice;
- PEP 649 and how it addresses those same goals;
- can we cleanly adopt PEP 649?
First off, it looks like this isn't worded clearly enough in the PEP itself so let me summarize what the goals of PEP 563 were:
Goal 1. To get rid of the forward reference problem, e.g. when a type is declared lower in the file than its use. A cute special case of this is when a class has a method that accepts or returns objects of its own type.
Goal 2. To somewhat decouple the syntax of type annotations from the runtime requirements, allowing for better expressibility.
Goal 3. To make annotations affect runtime characteristics of typed modules less, namely import time and memory usage.
Now, did PEP 563 succeed in its goals? Well, partially at best. Let's see.
In terms of Goal 1, it turned out that `typing.get_type_hints()` has limits that make its use in general costly at runtime, and more importantly insufficient to resolve all types. The most common example deals with non-global context in which types are generated (e.g. inner classes, classes within functions, etc.). But one of the crown examples of forward references: classes with methods accepting or returning objects of their own type, also isn't properly handled by `typing.get_type_hints()` if a class generator is used. There's some trickery we can do to connect the dots but in general it's not great.
As for Goal 2, it became painfully obvious that a number of types used for static typing purposes live outside of the type annotation context. So while PEP 563 tried to enable a backdoor for more usable static typing syntax, it ultimately couldn't. This is where PEP 585 and later PEP 604 came in, filling the gap by doing the sad but necessary work of enabling this extended typing syntax in proper runtime Python context. This is what should have been done all along and it makes PEP 563 in this context irrelevant as of Python 3.9 (for PEP 585) and 3.10 (for PEP 604). However, to the extent of types used within annotations, the PEP 563 future-import allows using the new cute typing syntax already for Python 3.7+ compatible code. So library authors can already adopt the lowercase type syntax of PEP 585 and the handy pipe syntax for unions of PEP 604. And even for non-type annotation uses that can be successfully barred by a `if TYPE_CHECKING` block, like type aliases, type variables, and such. Of course that has no chance of working with `typing.get_type_hints()`.
Now, Goal 3 is a different matter. As Inada Naoki demonstrated somewhere in the preceding discussion here, PEP 563 made fully type-annotationed codebase import pretty much as fast as non-annotated code, and through the joys of string interning, use relatively little extra memory. At the time PEP 563, a popular concern around static typing in Python was that it slows down runtime while it's only useful for static analysis. While we (the typing crowd) were always sure the "only useful as a better linter" is dismissive, the performance argument had to go.
So where does this leave us today?
Runtime use of types was somewhat overly optimistically treated as solvable with `typing.get_type_hints()`. Now Pydantic and other similar tools show that this isn't sadly the case. Without the future-import, they could ignore the problem until Python 3.10 but no longer. I was somewhat surprised this was the case because forward references as strings could always be used. So I guess the answer there was to just not use them if you want your runtime tool to work. Fair enough.
PEP 649 addresses this runtime usage of type annotations head on in a way that eluded me when I first set out to solve this problem. Back then, Larry and Mark Shannon did voice their opinion that through some clever frame object storage, "implicit lambdas", we can address the issue of forward references. This seemed unattractive to me at the time because it didn't deal with our Goal 2 and our understanding was that it actually makes Goal 3 worse by holding on to all frames in memory where type annotations appear, and by creating massive duplication of equivalent annotations in memory due to lack of a mechanism similar to string interning. Now those issues are somewhat solved in the final PEP 649 and this makes for an interesting compromise for us to make. I say "compromise" because as Inada Naoki measured, there's still a non-zero performance cost of PEP 649 versus PEP 563:
- code size: +63%
- memory: +62%
- import time: +60%
Will this hurt some current users of typing? Yes, I can name you multiple past employers of mine where this will be the case. Is it worth it for Pydantic? I tend to think that yes, it is, since it is a significant community, and the operations on type annotations it performs are in the sensible set for which `typing.get_type_hints()` was proposed.
However, there are some big open questions about how to adopt PEP 649.
Question 1. What should happen to code that already adopted `from __future__ import annotations`? Future imports were never thought of as feature toggles so if PEP 563 isn't going to become the default, it should get removed. However, removing it needs a deprecation period, otherwise code that already adopted the future-import will fail to execute (even if you leave a dummy future-import there -- forward references, remember?). So deprecation, and with that, a rather tricky situation where PEP 649 will have to support files with the future-import. Now PEP 649 says that an object cannot both have __annotations__ and __co_annotations__ set at the same time though, so I guess files with the future-import would necessarily be treated as some deprecated code that might or might not be translatable to PEP 649 co-annotations.
Question 2. If PEP 649 deprecates PEP 563, will the use of the future-import for early adoption of PEP 585 and PEP 604 syntax become disallowed? Ultimately this is my biggest worry -- that there doesn't seem to be a clear adoption path for this change. If we just take it wholesale and declare PEP 563 a failure, that bars library authors from using PEP 585/604 syntax until Python 3.10 becomes their lowest supported version. That's October 2025 at the earliest if we're looking at 3.9 lifespan. That would be a significant wrench thrown in typing adoption as PEP 585 and 604 provide significant usability advantages, to the point where often modules with non-trivial annotations don't even have to import the typing module at all.
Question 3. Additionally, it's unclear to me whether PEP 649 allows for any relaxed syntax in type annotations that might not be immediately valid in a given version of Python. Say, hypothetically (please don't bikeshed this!), we adopt PEP 649 in Python 3.10 and in Python 3.11 we come up with a shorthand for optional types using a question mark. Can I put the question mark there in Python 3.10 at all? Second made up example: let's say in Python 3.12 we allow passing a function as a type (meaning "a callable just LIKE this one"). Can I put this callable there now in Python 3.10?
Now, should we postpone with until Python 3.11? We should not, at least not in terms of figuring out a clear migration path from here to there, ideally such that existing end users of the PEP 563 future-import can just live their lives as if nothing ever happened. Given that the goal of the future-import was largely to say "I don't care about those annotations at runtime", maybe PEP 649 could somehow adopt the existing future-import? `typing.get_type_hints()` would do what it does today anyway. The only users affected would be those directly using the strings in PEP 563 annotations, and it's unclear whether this is a real problem. There are a hundred pages of results on GitHub [1]_ that include `__annotations__` so this would have to be sought through.
If we don't make it in time for including this in Python 3.10, then so be it. But let's not wait until April 2022 ;-)
i-regret-nothing'ly yours,
?ukasz
_[1] https://github.com/search?l=Python&q=__annotations__&type=Code
I got pinged to voice my opinion on PEP 649 as the instigator of PEP 563. I'm sorry, this is long, and a separate thread, because it deals with three things:
- Goals set for PEP 563 and how it did in practice;
- PEP 649 and how it addresses those same goals;
- can we cleanly adopt PEP 649?
First off, it looks like this isn't worded clearly enough in the PEP itself so let me summarize what the goals of PEP 563 were:
Goal 1. To get rid of the forward reference problem, e.g. when a type is declared lower in the file than its use. A cute special case of this is when a class has a method that accepts or returns objects of its own type.
Goal 2. To somewhat decouple the syntax of type annotations from the runtime requirements, allowing for better expressibility.
Goal 3. To make annotations affect runtime characteristics of typed modules less, namely import time and memory usage.
Now, did PEP 563 succeed in its goals? Well, partially at best. Let's see.
In terms of Goal 1, it turned out that `typing.get_type_hints()` has limits that make its use in general costly at runtime, and more importantly insufficient to resolve all types. The most common example deals with non-global context in which types are generated (e.g. inner classes, classes within functions, etc.). But one of the crown examples of forward references: classes with methods accepting or returning objects of their own type, also isn't properly handled by `typing.get_type_hints()` if a class generator is used. There's some trickery we can do to connect the dots but in general it's not great.
As for Goal 2, it became painfully obvious that a number of types used for static typing purposes live outside of the type annotation context. So while PEP 563 tried to enable a backdoor for more usable static typing syntax, it ultimately couldn't. This is where PEP 585 and later PEP 604 came in, filling the gap by doing the sad but necessary work of enabling this extended typing syntax in proper runtime Python context. This is what should have been done all along and it makes PEP 563 in this context irrelevant as of Python 3.9 (for PEP 585) and 3.10 (for PEP 604). However, to the extent of types used within annotations, the PEP 563 future-import allows using the new cute typing syntax already for Python 3.7+ compatible code. So library authors can already adopt the lowercase type syntax of PEP 585 and the handy pipe syntax for unions of PEP 604. And even for non-type annotation uses that can be successfully barred by a `if TYPE_CHECKING` block, like type aliases, type variables, and such. Of course that has no chance of working with `typing.get_type_hints()`.
Now, Goal 3 is a different matter. As Inada Naoki demonstrated somewhere in the preceding discussion here, PEP 563 made fully type-annotationed codebase import pretty much as fast as non-annotated code, and through the joys of string interning, use relatively little extra memory. At the time PEP 563, a popular concern around static typing in Python was that it slows down runtime while it's only useful for static analysis. While we (the typing crowd) were always sure the "only useful as a better linter" is dismissive, the performance argument had to go.
So where does this leave us today?
Runtime use of types was somewhat overly optimistically treated as solvable with `typing.get_type_hints()`. Now Pydantic and other similar tools show that this isn't sadly the case. Without the future-import, they could ignore the problem until Python 3.10 but no longer. I was somewhat surprised this was the case because forward references as strings could always be used. So I guess the answer there was to just not use them if you want your runtime tool to work. Fair enough.
PEP 649 addresses this runtime usage of type annotations head on in a way that eluded me when I first set out to solve this problem. Back then, Larry and Mark Shannon did voice their opinion that through some clever frame object storage, "implicit lambdas", we can address the issue of forward references. This seemed unattractive to me at the time because it didn't deal with our Goal 2 and our understanding was that it actually makes Goal 3 worse by holding on to all frames in memory where type annotations appear, and by creating massive duplication of equivalent annotations in memory due to lack of a mechanism similar to string interning. Now those issues are somewhat solved in the final PEP 649 and this makes for an interesting compromise for us to make. I say "compromise" because as Inada Naoki measured, there's still a non-zero performance cost of PEP 649 versus PEP 563:
- code size: +63%
- memory: +62%
- import time: +60%
Will this hurt some current users of typing? Yes, I can name you multiple past employers of mine where this will be the case. Is it worth it for Pydantic? I tend to think that yes, it is, since it is a significant community, and the operations on type annotations it performs are in the sensible set for which `typing.get_type_hints()` was proposed.
However, there are some big open questions about how to adopt PEP 649.
Question 1. What should happen to code that already adopted `from __future__ import annotations`? Future imports were never thought of as feature toggles so if PEP 563 isn't going to become the default, it should get removed. However, removing it needs a deprecation period, otherwise code that already adopted the future-import will fail to execute (even if you leave a dummy future-import there -- forward references, remember?). So deprecation, and with that, a rather tricky situation where PEP 649 will have to support files with the future-import. Now PEP 649 says that an object cannot both have __annotations__ and __co_annotations__ set at the same time though, so I guess files with the future-import would necessarily be treated as some deprecated code that might or might not be translatable to PEP 649 co-annotations.
Question 2. If PEP 649 deprecates PEP 563, will the use of the future-import for early adoption of PEP 585 and PEP 604 syntax become disallowed? Ultimately this is my biggest worry -- that there doesn't seem to be a clear adoption path for this change. If we just take it wholesale and declare PEP 563 a failure, that bars library authors from using PEP 585/604 syntax until Python 3.10 becomes their lowest supported version. That's October 2025 at the earliest if we're looking at 3.9 lifespan. That would be a significant wrench thrown in typing adoption as PEP 585 and 604 provide significant usability advantages, to the point where often modules with non-trivial annotations don't even have to import the typing module at all.
Question 3. Additionally, it's unclear to me whether PEP 649 allows for any relaxed syntax in type annotations that might not be immediately valid in a given version of Python. Say, hypothetically (please don't bikeshed this!), we adopt PEP 649 in Python 3.10 and in Python 3.11 we come up with a shorthand for optional types using a question mark. Can I put the question mark there in Python 3.10 at all? Second made up example: let's say in Python 3.12 we allow passing a function as a type (meaning "a callable just LIKE this one"). Can I put this callable there now in Python 3.10?
Now, should we postpone with until Python 3.11? We should not, at least not in terms of figuring out a clear migration path from here to there, ideally such that existing end users of the PEP 563 future-import can just live their lives as if nothing ever happened. Given that the goal of the future-import was largely to say "I don't care about those annotations at runtime", maybe PEP 649 could somehow adopt the existing future-import? `typing.get_type_hints()` would do what it does today anyway. The only users affected would be those directly using the strings in PEP 563 annotations, and it's unclear whether this is a real problem. There are a hundred pages of results on GitHub [1]_ that include `__annotations__` so this would have to be sought through.
If we don't make it in time for including this in Python 3.10, then so be it. But let's not wait until April 2022 ;-)
i-regret-nothing'ly yours,
?ukasz
_[1] https://github.com/search?l=Python&q=__annotations__&type=Code