Mailing List Archive

I haven't commented on these, as I don't use type annotations in the
"usual" way.,
but:

On Thu, Apr 15, 2021 at 12:06 PM Samuel Colvin <samcolvin@gmail.com> wrote:

> I maintain pydantic <https://pydantic-docs.helpmanual.io/> which uses
> type annotations to provide runtime data validation.
>

I maintain in in-house system kinda similar to pydantic that doesn't use
type annotations for validation, but does use them for other things (mostly
serializing/deserializing to/from JSON), and other kinds of data validation.

> using `typing.get_type_hints()` is not a good replacement for type
annotations that are accessible as python objects.

Absolutely. It is very, very, handy to simply have the object itself easily
accessible.

-CHB

--
Christopher Barker, PhD (Chris)

Python Language Consulting
- Teaching
- Scientific Software Development
- Desktop GUI and Web Development
- wxPython, numpy, scipy, Cython

Please accept PEP 649!

Python's type hinting has become so much more useful than originally thought, and without this change much of that will be hindered. For example (you already know about Pydantic and FastAPI) [discord.py](https://github.com/Rapptz/discord.py)'s commands system allows you to use typehinting to specify how arguments should be converted. Take the following code:

```py
import discord
from discord.ext import commands

bot = commands.Bot(command_prefix='>')

@bot.command()
# discord.py reads the typehints and converts the arguments accordingly
async def reply(ctx, member: discord.Member, *, text: str): # ctx is always passed
await ctx.send(f'{member.mention}! {text}')

bot.run('token')
```

I must say, this is extremely ergonomic design! Don't break it :)
_______________________________________________
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-leave@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/S2O7SE4QZQARAYSCOT7PQUEPNENHDJTQ/
Code of Conduct: http://python.org/psf/codeofconduct/

I should point out that "accept PEP 649" and "break pydantic" are not
the only options here. The thing that will break pydantic is the end of
PEP 563's deprecation period, not a failure to implement PEP 649.

Other viable options include:

- Leave PEP 563 opt-in until we can agree on a solution to the problem.
- Leave PEP 563 opt-in forever.
- Deprecate PEP 563 and go back to status quo ante.

I haven't followed this closely enough — if PEP 649 were accepted today,
would it even be ready for use before the 3.10 code freeze (which is in
a few weeks)?

Assuming this is a real problem (and based in part on how long it took
for attrs to get what support it has for PEP 563
<https://github.com/python-attrs/attrs/issues/288>, I wouldn't be
surprised if PEP 563 is quietly throwing a spanner in the works in
several other places as well), my vote is to leave PEP 563 opt-in until
at least 3.11 rather than try to rush through a discussion on and
implementation of PEP 649.

Best,
Paul

On 4/15/21 4:20 PM, Bluenix wrote:
> Please accept PEP 649!
>
> Python's type hinting has become so much more useful than originally thought, and without this change much of that will be hindered. For example (you already know about Pydantic and FastAPI) [discord.py](https://github.com/Rapptz/discord.py)'s commands system allows you to use typehinting to specify how arguments should be converted. Take the following code:
>
> ```py
> import discord
> from discord.ext import commands
>
> bot = commands.Bot(command_prefix='>')
>
> @bot.command()
> # discord.py reads the typehints and converts the arguments accordingly
> async def reply(ctx, member: discord.Member, *, text: str): # ctx is always passed
> await ctx.send(f'{member.mention}! {text}')
>
> bot.run('token')
> ```
>
> I must say, this is extremely ergonomic design! Don't break it :)
> _______________________________________________
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-leave@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/S2O7SE4QZQARAYSCOT7PQUEPNENHDJTQ/
> Code of Conduct: http://python.org/psf/codeofconduct/

I've been working on a similar framework, Fondat [1], that shares
similar objectives as Pydantic and FastAPI, albeit implementing in a
different manner. 

I believe other frameworks will have to "run a gauntlet" (as I already
have)  to deal with the affects of PEP 563. If anyone on these teams
are interested in how I've dealt with such issues, please feel free to
contact me directly.

I also recommend members of these teams monitor and/or participate in
the typing-sig (and maybe the python-dev) mailing list, as discussions
in these lists are likely to affect their work in the future.

Paul 

[1] https://github.com/fondat/fondat-core


On Thu, 2021-04-15 at 17:49 -0400, Paul Ganssle wrote:
> I should point out that "accept PEP 649" and "break pydantic" are not
> the only options here. The thing that will break pydantic is the end
> of PEP 563's deprecation period, not a failure to implement PEP 649.
> Other viable options include:
> - Leave PEP 563 opt-in until we can agree on a solution to the
> problem.
> - Leave PEP 563 opt-in forever.
> - Deprecate PEP 563 and go back to status quo ante.
>
> I haven't followed this closely enough — if PEP 649 were accepted
> today, would it even be ready for use before the 3.10 code freeze
> (which is in a few weeks)?
>
> Assuming this is a real problem (and based in part on how long it
> took for attrs to get what support it has for PEP 563, I wouldn't be
> surprised if PEP 563 is quietly throwing a spanner in the works in
> several other places as well), my vote is to leave PEP 563 opt-in
> until at least 3.11 rather than try to rush through a discussion on
> and implementation of PEP 649.
>
> Best,
> Paul
> On 4/15/21 4:20 PM, Bluenix wrote:
>
> > Please accept PEP 649!
> >
> > Python's type hinting has become so much more useful than originally thought, and without this change much of that will be hindered. For example (you already know about Pydantic and FastAPI) [discord.py](https://github.com/Rapptz/discord.py)'s commands system allows you to use typehinting to specify how arguments should be converted. Take the following code:
> >
> > ```py
> > import discord
> > from discord.ext import commands
> >
> > bot = commands.Bot(command_prefix='>')
> >
> > @bot.command()
> > # discord.py reads the typehints and converts the arguments accordingly
> > async def reply(ctx, member: discord.Member, *, text: str): # ctx is always passed
> > await ctx.send(f'{member.mention}! {text}')
> >
> > bot.run('token')
> > ```
> >
> > I must say, this is extremely ergonomic design! Don't break it :)
> > _______________________________________________
> > Python-Dev mailing list -- python-dev@python.org
> > To unsubscribe send an email to python-dev-leave@python.org
> > https://mail.python.org/mailman3/lists/python-dev.python.org/
> > Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/S2O7SE4QZQARAYSCOT7PQUEPNENHDJTQ/
> > Code of Conduct: http://python.org/psf/codeofconduct/
> _______________________________________________
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-leave@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-dev@python.org/message/GT2HQDH2HOZFSOTA5LHTFL5OV46UPKTB/
> Code of Conduct: http://python.org/psf/codeofconduct/

On 4/15/21 2:49 PM, Paul Ganssle wrote:
> I haven't followed this closely enough — if PEP 649 were accepted
> today, would it even be ready for use before the 3.10 code freeze
> (which is in a few weeks)?


Perhaps I'm a poor judge of the quality of my own code.   But I'd say I
have a working prototype and it seems... fine.  It didn't require a lot
of changes, it just needed the right changes in the right spots. 
Getting my interactions with symtable and compile were the hardest parts
and I think those are all sorted out now.


Cheers,


//arry/

Hello,

I'm a bit surprised that this topic is brought up just days before the
feature freeze of Python 3.10. How did tools work until now with from
future import annotations? This feature has been in the language for years
(since 3.7), and I haven't heard until today anyone complaining about this
feature gap. Did everyone just told users not to use lazy evaluation of
type hints until now? Also there's been advertised for at least a year that
next python will make it default ????

Thanks,

Bernat

On Thu, Apr 15, 2021, 23:13 Paul Bryan <pbryan@anode.ca> wrote:

> I've been working on a similar framework, Fondat [1], that shares similar
> objectives as Pydantic and FastAPI, albeit implementing in a different
> manner.
>
> I believe other frameworks will have to "run a gauntlet" (as I already
> have) to deal with the affects of PEP 563. If anyone on these teams are
> interested in how I've dealt with such issues, please feel free to contact
> me directly.
>
> I also recommend members of these teams monitor and/or participate in the
> typing-sig (and maybe the python-dev) mailing list, as discussions in these
> lists are likely to affect their work in the future.
>
> Paul
>
> [1] https://github.com/fondat/fondat-core
>
>
> On Thu, 2021-04-15 at 17:49 -0400, Paul Ganssle wrote:
>
> I should point out that "accept PEP 649" and "break pydantic" are not the
> only options here. The thing that will break pydantic is the end of PEP
> 563's deprecation period, not a failure to implement PEP 649.
>
> Other viable options include:
>
> - Leave PEP 563 opt-in until we can agree on a solution to the problem.
> - Leave PEP 563 opt-in forever.
> - Deprecate PEP 563 and go back to status quo ante.
>
> I haven't followed this closely enough — if PEP 649 were accepted today,
> would it even be ready for use before the 3.10 code freeze (which is in a
> few weeks)?
>
> Assuming this is a real problem (and based in part on how long it took
> for attrs to get what support it has for PEP 563
> <https://github.com/python-attrs/attrs/issues/288>, I wouldn't be
> surprised if PEP 563 is quietly throwing a spanner in the works in several
> other places as well), my vote is to leave PEP 563 opt-in until at least
> 3.11 rather than try to rush through a discussion on and implementation of
> PEP 649.
>
> Best,
> Paul
> On 4/15/21 4:20 PM, Bluenix wrote:
>
> Please accept PEP 649!
>
>
> Python's type hinting has become so much more useful than originally thought, and without this change much of that will be hindered. For example (you already know about Pydantic and FastAPI) [discord.py](https://github.com/Rapptz/discord.py)'s commands system allows you to use typehinting to specify how arguments should be converted. Take the following code:
>
>
> ```py
>
> import discord
>
> from discord.ext import commands
>
>
> bot = commands.Bot(command_prefix='>')
>
>
> @bot.command()
>
> # discord.py reads the typehints and converts the arguments accordingly
>
> async def reply(ctx, member: discord.Member, *, text: str): # ctx is always passed
>
> await ctx.send(f'{member.mention}! {text}')
>
>
> bot.run('token')
>
> ```
>
>
> I must say, this is extremely ergonomic design! Don't break it :)
>
> _______________________________________________
>
> Python-Dev mailing list -- python-dev@python.org
>
> To unsubscribe send an email to python-dev-leave@python.org
>
> https://mail.python.org/mailman3/lists/python-dev.python.org/
>
> Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/S2O7SE4QZQARAYSCOT7PQUEPNENHDJTQ/
>
> Code of Conduct: http://python.org/psf/codeofconduct/
>
> _______________________________________________
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-leave@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-dev@python.org/message/GT2HQDH2HOZFSOTA5LHTFL5OV46UPKTB/
> Code of Conduct: http://python.org/psf/codeofconduct/
>
>
> _______________________________________________
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-leave@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-dev@python.org/message/PBGFXPC7D3NFLJSFQBDAQQCW6JCMVCFC/
> Code of Conduct: http://python.org/psf/codeofconduct/
>

On Thu, Apr 15, 2021 at 3:29 PM Bernat Gabor <gaborjbernat@gmail.com> wrote:
> I'm a bit surprised that this topic is brought up just days before the
feature freeze of Python 3.10.

I have not idea about the technical details, but I think there is a bit of
a discomment in the community:

Annotations have been around for years.

More recently, there was an official specification of how to use them for
type hinting (was that PEP 484? a bit confused as that appears to still be
provisional -- but anyway)

Since the specification for type hinting, a lot of work has been done
on/with static type checkers (e.g. MyPy).

That work has informed the changes / improvements / proposals to the
annotation and type system in Python.

However, those of us that are not actively involved (and maybe not that
interested) in static typing have been not paying a great deal of attention
to those efforts.

But some of us may, in fact, be using annotations for other reasons, but
since we haven't been involved in the discussion, some issues may have been
missed.

-CHB


--
Christopher Barker, PhD (Chris)

Python Language Consulting
- Teaching
- Scientific Software Development
- Desktop GUI and Web Development
- wxPython, numpy, scipy, Cython

On Thu, Apr 15, 2021 at 12:01 PM Samuel Colvin <samcolvin@gmail.com> wrote:

> I've read the recent discussions
> <https://mail.python.org/archives/list/python-dev@python.org/thread/QSASX6PZ3LIIFIANHQQFS752BJYFUFPY/#UITB2A657TAINAGWGRD6GCKWFC5PEBIZ>
> regarding PEP 649 and PEP 563 with interest, Larry Hastings recently
> contacted me when prompted
> <https://mail.python.org/archives/list/python-dev@python.org/message/YKVYJMLUWUVT4KMLUNEQYVBZWNAPR4GV/>
> to do so in a related discussion.
>
> I maintain pydantic <https://pydantic-docs.helpmanual.io/> which uses
> type annotations to provide runtime data validation.
>
> I'm extremely worried that if PEP 649 gets rejected, pydantic will not be
> able to fully support python 3.10 and may even end up getting abandoned, at
> best it will be much slower and more brittle.
>
> Please, please put pragmatism over principle and accept PEP 649.
>

Please don't phrase the decision in these terms. While for Pydantic PEP 649
is more pragmatic, that does not mean PEP 563 isn't pragmatic for other
people for other reasons. Making this an "us versus them" discussion just
makes the whole situation feel confrontational when instead everyone is
trying to figure out the best thing for everybody when there's no perfect
answer.

It seems evident that the problems with PEP 563 have been well-known at
least to pydantic for several years, as you can see in the issue Samuel
Colvin linked: https://github.com/samuelcolvin/pydantic/issues/2678

That said, while I do think that "please contact upstream when you see a
problem developing, not just before a major deadline" is a good lesson
to take away from this, it is probably not worth litigating the question
of the particular manner of the notification. As much as I think it
would have been good for this discussion to happen 6 months, 1 year, 2
years or 3 years ago, "before the code freeze" is a heck of a lot better
than "after the release" (which is often when the notification comes in,
and why we have been encouraging people to test against alphas and betas).

Hopefully we can let it rest by saying that the timing of learning about
this apparently urgent situation could have been much better, but it
could have been worse as well.

Best,
Paul

On 4/15/21 7:12 PM, Christopher Barker wrote:
> On Thu, Apr 15, 2021 at 3:29 PM Bernat Gabor <gaborjbernat@gmail.com
> <mailto:gaborjbernat@gmail.com>> wrote:
> > I'm a bit surprised that this topic is brought up just days before
> the feature freeze of Python 3.10.
>
> I have not idea about the technical details, but I think there is a
> bit of a discomment in the community:
>
> Annotations have been around for years.
>
> More recently, there was an official specification of how to use them
> for type hinting (was that PEP 484? a bit confused as that appears to
> still be provisional -- but anyway)
>
> Since the specification for type hinting, a lot of work has been done
> on/with static type checkers (e.g. MyPy).
>
> That work has informed the changes / improvements / proposals to the
> annotation and type system in Python.
>
> However, those of us that are not actively involved (and maybe not
> that interested) in static typing have been not paying a great deal of
> attention to those efforts.
>
> But some of us may, in fact, be using annotations for other reasons,
> but since we haven't been involved in the discussion, some issues may
> have been missed.
>
> -CHB
>
>
> --
> Christopher Barker, PhD (Chris)
>
> Python Language Consulting
>   - Teaching
>   - Scientific Software Development
>   - Desktop GUI and Web Development
>   - wxPython, numpy, scipy, Cython
>
> _______________________________________________
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-leave@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/5TGKKRZXGQROQKS2WX6WFGCHTMNUJYBF/
> Code of Conduct: http://python.org/psf/codeofconduct/

On Thu, Apr 15, 2021 at 4:20 PM Brett Cannon <brett@python.org> wrote:

>
> Making this an "us versus them" discussion just makes the whole situation
> feel confrontational when instead everyone is trying to figure out the best
> thing for everybody when there's no perfect answer.
>

I agree that that was strong language, but from the perspective of folks
not interested in Static Typing, there has been a lot of activity, or at
least discussion, that makes it seem like this optional feature is driving
the evolution of the language.

So folks are going to get concerned, and maybe upset, if it looks like
changes might be made that may break other features in order to make static
typing work better.

And as I noted in my last post — many folks have not been paying attention
to the typing discussions because they didn’t realize it concerned them.

-CHB

--
Christopher Barker, PhD (Chris)

Python Language Consulting
- Teaching
- Scientific Software Development
- Desktop GUI and Web Development
- wxPython, numpy, scipy, Cython

On Thu, Apr 15, 2021 at 16:48 Christopher Barker <pythonchb@gmail.com>
wrote:

> And as I noted in my last post — many folks have not been paying attention
> to the typing discussions because they didn’t realize it concerned them.
>

It seems a little disingenuous to claim discussions about annotations don’t
concern you when you’re actively using them (for typing, no less, in the
case of pydantic). And I am sure a project as popular (by their own
description) as pydantic will find a way forward if PEP 649 is rejected,
despite overdramatized claims.

That said, I agree it is better that this came up before the feature freeze
than after the release. And I am willing to accept that the hypothetical
future where annotations are not always syntactically expressions (which
did not even exist before this week) is less important than backwards
compatibility.


--
--Guido (mobile)

On Fri, 16 Apr 2021 at 01:13, Guido van Rossum <guido@python.org> wrote:
>
> On Thu, Apr 15, 2021 at 16:48 Christopher Barker <pythonchb@gmail.com> wrote:
>>
>> And as I noted in my last post — many folks have not been paying attention to the typing discussions because they didn’t realize it concerned them.
>
> It seems a little disingenuous to claim discussions about annotations don’t concern you when you’re actively using them (for typing, no less, in the case of pydantic). And I am sure a project as popular (by their own description) as pydantic will find a way forward if PEP 649 is rejected, despite overdramatized claims.

It looks to me like pydantic were well aware that this affected them
and have been working over time to try and support this. They probably
also had their eye on PEP 649 or hoped that something else would come
along to fix this but they can now see the crunch point approaching.

As Chris says though other libraries/developers have been using
annotations for some time and might not have realised that typing
considerations could undermine their other uses for annotations since
annotations predate typing and typing is an optional feature.

> That said, I agree it is better that this came up before the feature freeze than after the release. And I am willing to accept that the hypothetical future where annotations are not always syntactically expressions (which did not even exist before this week) is less important than backwards compatibility.

Would it be problematic to postpone making __future__.annotations the default?


Oscar
_______________________________________________
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-leave@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/VCFHDQFOBQXZDKKPSTU5FNKP2AZIJMX2/
Code of Conduct: http://python.org/psf/codeofconduct/

On Apr 15, 2021, at 17:47, Oscar Benjamin <oscar.j.benjamin@gmail.com> wrote:
>
> Would it be problematic to postpone making __future__.annotations the default?

This is a good question, and I think the SC would really like to know if anybody has objections to postponing this to 3.11. We haven’t discussed it yet (this whole topic is on our agenda for next Monday), but it might be the best thing to do given where we are in the 3.10 release cycle. It would give everyone a chance to breathe and come up with the right long term solution.

Cheers,
-Barry

On Fri, Apr 16, 2021 at 9:49 AM Oscar Benjamin
<oscar.j.benjamin@gmail.com> wrote:
>
>
> > That said, I agree it is better that this came up before the feature freeze than after the release. And I am willing to accept that the hypothetical future where annotations are not always syntactically expressions (which did not even exist before this week) is less important than backwards compatibility.
>
> Would it be problematic to postpone making __future__.annotations the default?
>

__future__.annotation is the default since 2020-10-04.
https://github.com/python/cpython/commit/044a1048ca93d466965afc027b91a5a9eb9ce23c#diff-ebc983d9f91e5bcf73500e377ac65e85863c4f77fd5b6b6caf4fcdf7c0f0b057

After that, many changes are made on compiler and other places.
So reverting the change is not so simple.

And personally, I love static typing but I don't use type hint for
performance/memory usage reason.
I spend much effort to optimize PEP 563 to minimize type hinting overhead.
So it's very sad that if I can not use type hinting when I can drop
Python 3.9 support.

So if PEP 649 is accepted, I want to use it since Python 3.10.
Otherwise, I can not use type hinting even after I dropped Python 3.9
support.

But it is up to release manager and steering council.

--
Inada Naoki <songofacandy@gmail.com>
_______________________________________________
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-leave@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/6ZYCD63KBP2EPDZIYJD2IPHCVRV4LQGP/
Code of Conduct: http://python.org/psf/codeofconduct/

On Thu, Apr 15, 2021 at 18:20 Barry Warsaw <barry@python.org> wrote:

> On Apr 15, 2021, at 17:47, Oscar Benjamin <oscar.j.benjamin@gmail.com>
> wrote:
> >
> > Would it be problematic to postpone making __future__.annotations the
> default?
>
> This is a good question, and I think the SC would really like to know if
> anybody has objections to postponing this to 3.11. We haven’t discussed
> it yet (this whole topic is on our agenda for next Monday), but it might be
> the best thing to do given where we are in the 3.10 release cycle. It
> would give everyone a chance to breathe and come up with the right long
> term solution.


As long as you know that’s not zero cost either. Undoing the work will be
complex (merge conflicts will abound) and and all testing and
implementation people have done assuming PEP 563 semantics will still have
to be redone.

And note that there are some corner cases where the “PEP 563 semantics” in
3.10 could not be made fully compatible with the behavior under `from
__future__ Import annotations` in 3.9 (in truth, because not enough testing
was done under those semantics in 3.7-3.9).

But it could be done, and that’s the way PEP 649 is currently written
(using `from __future__ import co_annotations`).



--
--Guido (mobile)

On 4/15/21 6:09 PM, Barry Warsaw wrote:
> On Apr 15, 2021, at 17:47, Oscar Benjamin <oscar.j.benjamin@gmail.com> wrote:
>> Would it be problematic to postpone making __future__.annotations the default?
> This is a good question, and I think the SC would really like to know if anybody has objections to postponing this to 3.11. We haven’t discussed it yet (this whole topic is on our agenda for next Monday), but it might be the best thing to do given where we are in the 3.10 release cycle. It would give everyone a chance to breathe and come up with the right long term solution.


I don't have any objections, and I certainly see the wisdom in such a
decision.


Best wishes,


//arry/

On Fri, Apr 16, 2021 at 6:03 AM Bluenix <bluenixdev@gmail.com> wrote:
>
> Please accept PEP 649!
>
> Python's type hinting has become so much more useful than originally thought, and without this change much of that will be hindered. For example (you already know about Pydantic and FastAPI) [discord.py](https://github.com/Rapptz/discord.py)'s commands system allows you to use typehinting to specify how arguments should be converted. Take the following code:
>
> ```py
> import discord
> from discord.ext import commands
>
> bot = commands.Bot(command_prefix='>')
>
> @bot.command()
> # discord.py reads the typehints and converts the arguments accordingly
> async def reply(ctx, member: discord.Member, *, text: str): # ctx is always passed
> await ctx.send(f'{member.mention}! {text}')
>
> bot.run('token')
> ```
>
> I must say, this is extremely ergonomic design! Don't break it :)

Are you sure about PEP 563 break it and discord.py can not fix it?

As far as my understanding, PEP 563 don't hurt this use case so much:

* This use case evaluates runtime type information only once. So
eval() overhead is not a problem.
* Currently, annotation is very very complex and varies. For example,
List[int] will be `List[int]`, `List['int']`, `'List[int]'`,
`'List["int"]'`, `List[ForwardRef('int')]` etc...
After PEP 563, only `'List[int]'` is practical so we can stop
supporting `List["int"]` and others at some point.
So playing with runtime type will become easier in the future.

Am I wrong?

--
Inada Naoki <songofacandy@gmail.com>
_______________________________________________
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-leave@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/NSZGWCABWFYWZZTNCE5VE5ZVC3OUJCNU/
Code of Conduct: http://python.org/psf/codeofconduct/

On Fri, 16 Apr 2021 10:35:19 +0900
Inada Naoki <songofacandy@gmail.com> wrote:
>
> And personally, I love static typing but I don't use type hint for
> performance/memory usage reason.
> I spend much effort to optimize PEP 563 to minimize type hinting overhead.
> So it's very sad that if I can not use type hinting when I can drop
> Python 3.9 support.

AFAIU, we're not talking about dropping PEP 563 (yet), but about making
it opt-in again.

Regards

Antoine.


_______________________________________________
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-leave@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/NVJOKSXRFEUKWHF4USVOB2U3YTCL3PZL/
Code of Conduct: http://python.org/psf/codeofconduct/

> Are you sure about PEP 563 break it and discord.py can not fix it?

PEP 563 has been supported by discord.py since the PEP was introduced -- the change was not particularly hard to support on my end. Newer versions of the library are even more supportive of things like `ForwardRef` and the like.
_______________________________________________
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-leave@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/Z4FDBM2NKDSY4LKAJ7JY2ZOXJQVL22YD/
Code of Conduct: http://python.org/psf/codeofconduct/

Guido:
> It seems a little disingenuous to claim discussions about annotations
don’t
> concern you when you’re actively using them (for typing, no less, in the
> case of pydantic). And I am sure a project as popular (by their own
> description) as pydantic will find a way forward if PEP 649 is rejected,
> despite overdramatized claims.

I also maintain a library that uses type annotations in a runtime context
(cattrs), but in a slightly different way than Pydantic. My project is much
less popular than Pydantic (to be expected, since it only deals with
de/serialization and the class definition layer is attrs/dataclasses), and
the first issue filed on my bug tracker about future annotations was
created at the end of 2019, so that's when I started thinking about it.

To be honest, supporting stringified annotations was a lot of work to do in
an efficient way (and that's the main reason cattrs has two converter
classes now), but there was plenty of time and I simply did the work. The
new design is even faster than the old one.

On Fri, Apr 16, 2021 at 8:27 AM Inada Naoki <songofacandy@gmail.com> wrote:

> After PEP 563, only `'List[int]'` is practical so we can stop
> supporting `List["int"]` and others at some point.
>

There's a lot of code written before PEP 563 was available (and code that
needs to support earlier Python versions) that uses string literal quoting
in various places. If Python stops supporting List["X"], it will be a
significant backward compatibility break. Some of these string literal
escapes are made unnecessary by PEP 563, but there are still cases where
string literals are needed that neither PEP 563 nor PEP 649 address.
Examples include type aliases, type variable definitions and base classes,
but there are others [1]. These aren't evaluated in a type annotation
context, so manual string literal escaping would still be needed.


> So playing with runtime type will become easier in the future.
>
> Am I wrong?
>

In some cases things will become easier, but other common use cases still
seem to be unresolved. I haven't seen any proposal that can completely
replace all uses of string literal escapes. I guess one option would be to
only allow string literal escaping to be used outside type annotations in
the future. I think that this could be feasible with a suitable deprecation
period.

Also, the use of "if TYPE_CHECKING" breaks the runtime use of annotations.
PEP 563 and PEP 649 reduce the need for string literal escaping in this use
case, but it mostly helps static type checkers. This is pretty common in
codebases that use static type checking. My main issue with PEP 649 is that
it only addresses a subset of remaining issues, i.e. it doesn't go far
enough. It's also not clear to me if PEP 649 (or PEP 563) can be extended
to cover the remaining runtime issues, or if we'd need a *third* approach
to solve them.

Jukka

[1] https://www.python.org/dev/peps/pep-0563/#forward-references

Thank you everyone for your responses.

I entirely accept that I should have brought this up earlier, perhaps much earlier.

In my defence, when PEP 563 first came on my radar I assumed that get_type_hint() would be improved before it became the default behaviour, AFAIK it hasn't really changed. In particular the subtle changes to scope described in detail in PEP 649 [1] are still there and are still a big headache for pydantic.

As Paul Ganssle says, my timing was not great, but it could also have been worse.

I also never meant to create an "us versus them" style discussion, I was just worried about the narrow time window I had and the likelihood that my voice would not be heard, hence sounding as impassioned as I did.

For a fairly comprehensive description of why PEP 563 represents challenges to pydantic, I defer to ?ukasz Langa (the instigator of PEP 563) who has explained it very clearly in another thread "[Python-Dev] PEP 563 in light of PEP 649" [2].

Samuel

[1] - https://www.python.org/dev/peps/pep-0649/#id9
[2] - https://mail.python.org/archives/list/python-dev@python.org/message/ZBJ7MD6CSGM6LZAOTET7GXAVBZB7O77O/
_______________________________________________
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-leave@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/CJTAJJEXZXM5WKJ47POLK3YWWNPKK5NB/
Code of Conduct: http://python.org/psf/codeofconduct/

I wonder if anyone has considered the impact of PEP 563 on dataclasses ?

I did find this SO post:

https://stackoverflow.com/questions/52171982/new-annotations-break-inspection-of-dataclasses

which is related, but not quite the same -- THAT issue was fixed.

My issue does show up in this BPO:

https://bugs.python.org/issue39442

Somewhere in the middle of that thread, Eric wrote: "At this point, this
seems more like fodder for python-ideas."

Then there's a bit more to the thread, which peters out without resolution.

On to the issue:

dataclasses may be the only part of the stdlib that uses annotations.

dataclasses store the annotations in Field objects type attribute, in
the __dataclass_fields__ attribute.

We can see that with this code:

@dataclass
class Simple:
x : int
y : float
name : str

s = Simple(3, 4.0, "fred")

print("The field types:")
for f in s.__dataclass_fields__.values():
print(f"name: {f.name}, type: {f.type}, type of type: {type(f.type)}")

Which results in:

The field types:
name: x, type: <class 'int'>, type of type: <class 'type'>
name: y, type: <class 'float'>, type of type: <class 'type'>
name: name, type: <class 'str'>, type of type: <class 'type'>

with:

from __future__ import annotations

The result is:

The field types:
name: x, type: int, type of type: <class 'str'>
name: y, type: float, type of type: <class 'str'>
name: name, type: str, type of type: <class 'str'>

This of course is completely as expected.

I have no idea how dataclasses uses the Field type attribute -- as far as I
can tell, for nothing at all. However, it is there, and it is called
"type", rather than say, "annotation".

And I have a whole pile of code that fully expects the Fields' type
attribute to be an actual type object that I can call to crate an instance
of that type (or call a method on it, which is what I am actually doing)

So my code will very much break with this change.

I fully understand that the __dataclass_fields__ attribute was probably
never intended to be part of the public API, so I get what I deserve.

However, the Field object is documented, as such:

"""
class dataclasses.Field

Field objects describe each defined field. These objects are created
internally, and are returned by the fields() module-level method (see
below). Users should never instantiate a Field object directly. Its
documented attributes are:

name: The name of the field.
type: The type of the field.
default, default_factory, init, repr, hash, compare, and metadata have the
identical meaning and values as they do in the field() declaration.

Other attributes may exist, but they are private and must not be inspected
or relied on.
"""

That last sentence implies that the type attribute CAN be inspected and
relied upon, which is what I am doing.

And I haven't tried to solve this problem in my use case, but I'm not sure
it can be done -- when I get around to inspecting the type of the Field
objects, I have no idea what namespace they are in -- so I can't
reconstruct them from the string. I really need the type object itself.

So I'll probably need to re-write much of the dataclasses decorator, to
call eval() early -- though even then I'm not sure I'll have access to the
proper namespace.

Anyway -- one more data point: PEP 563 changes the (semi-public?) API of
dataclasses.

Though *maybe* that could be addressed with a dataclasses update -- again,
I've only started to think about it -- there was some discussion of that in
the BPO, though Eric didn't seem particularly interested.

-CHB

--
Christopher Barker, PhD (Chris)

Python Language Consulting
- Teaching
- Scientific Software Development
- Desktop GUI and Web Development
- wxPython, numpy, scipy, Cython

On 4/17/2021 12:28 AM, Christopher Barker wrote:
> I wonder if anyone has considered the impact of PEP 563 on dataclasses ?
I have!
>
> I did find this SO post:
>
> https://stackoverflow.com/questions/52171982/new-annotations-break-inspection-of-dataclasses
> <https://stackoverflow.com/questions/52171982/new-annotations-break-inspection-of-dataclasses>
>
> which is related, but not quite the same -- THAT issue was fixed.
>
> My issue does show up in this BPO:
>
> https://bugs.python.org/issue39442 <https://bugs.python.org/issue39442>
>
> Somewhere in the middle of that thread, Eric wrote: "At this point,
> this seems more like fodder for python-ideas."
>
> Then there's a bit more to the thread, which peters out without
> resolution.
>
> On to the issue:
>
> dataclasses may be the only part of the stdlib that uses annotations.
>
> dataclasses store the annotations in Field objects type attribute, in
> the __dataclass_fields__ attribute.
>
> We can see that with this code:
>
> @dataclass
> class Simple:
>     x : int
>     y : float
>     name : str
>
> s = Simple(3, 4.0, "fred")
>
> print("The field types:")
> for f in s.__dataclass_fields__.values():
>     print(f"name: {f.name <http://f.name>}, type: {f.type}, type of
> type: {type(f.type)}")
>
> Which results in:
>
> The field types:
> name: x, type: <class 'int'>, type of type: <class 'type'>
> name: y, type: <class 'float'>, type of type: <class 'type'>
> name: name, type: <class 'str'>, type of type: <class 'type'>
>
> with:
>
> from __future__ import annotations
>
> The result is:
>
> The field types:
> name: x, type: int, type of type: <class 'str'>
> name: y, type: float, type of type: <class 'str'>
> name: name, type: str, type of type: <class 'str'>
>
> This of course is completely as expected.
>
> I have no idea how dataclasses uses the Field type attribute -- as far
> as I can tell, for nothing at all. However, it is there, and it is
> called "type", rather than say, "annotation".
In retrospect, that field probably should have been named "annotation".
Regardless, the intent was always "store what's in
__annotations__[field_name]", or what the user specified in field(...,
type=whatever, ...).
>
> And I have a whole pile of code that fully expects the Fields' type
> attribute to be an actual type object that I can call to crate an
> instance of that type (or call a method on it, which is what I am
> actually doing)
>
> So my code will very much break with this change.
True, unfortunately. To be clear to everyone not paying close attention,
"this change" is PEP 563.
>
> I fully understand that the __dataclass_fields__ attribute was
> probably never intended to be part of the public API, so I get what I
> deserve.
>
> However, the Field object is documented, as such:
>
> """
> class dataclasses.Field
>
> Field objects describe each defined field. These objects are created
> internally, and are returned by the fields() module-level method (see
> below). Users should never instantiate a Field object directly. Its
> documented attributes are:
>
> name: The name of the field.
> type: The type of the field.
> default, default_factory, init, repr, hash, compare, and metadata have
> the identical meaning and values as they do in the field() declaration.
>
> Other attributes may exist, but they are private and must not be
> inspected or relied on.
> """
>
> That last sentence implies that the type attribute CAN be inspected
> and relied upon, which is what I am doing.
Yes, Field.type is very much part of the public dataclasses API as
available through dataclasses.fields(), not through
cls.__dataclass_fields__.
>
> And I haven't tried to solve this problem in my use case, but I'm not
> sure it can be done -- when I get around to inspecting the type of the
> Field objects, I have no idea what namespace they are in -- so I can't
> reconstruct them from the string. I really need the type object itself.
@dataclass pretty much has the same problem with respect to calling
typing.get_type_hints().
>
> So I'll probably need to re-write much of the dataclasses decorator,
> to call eval() early -- though even then I'm not sure I'll have access
> to the proper namespace.
>
> Anyway -- one more data point:  PEP 563 changes the (semi-public?) API
> of dataclasses.
>
> Though *maybe* that could be addressed with a dataclasses update --
> again, I've only started to think about it -- there was some
> discussion of that in the BPO, though Eric didn't seem particularly
> interested.

I still think it's working as intended: it uses what's in
__annotations__ (or passed in to field()). As everyone who has tried to
call typing.get_type_hints() knows, it's hard to get right, if it's even
possible, because, as you say "when I get around to inspecting the type
of the Field objects, I have no idea what namespace they are in". My
opinion is that the person who wants a real type object probably has a
better idea of that namespace than @dataclass does, but there's a very
good chance they don't know, either.

@dataclass goes out of its way to not call typing.get_type_hints(). The
original reason for this is not wanting to force typing to be imported,
if it wasn't already being used. That may have been addressed with PEP
560, but I've never really checked on the impact.

Another reason for not calling typing.get_type_hints(): there's really
only one thing [*] dataclasses wants to know, with regard to the
type/annotation of the field: is the type of this field typing.ClassVar?
It doesn't seem that the performance issues and possible failures make
it worth calling typing.get_type_hints() just for this case. @dataclass
uses other tricks (not described here).

In any event, all of this mess is one reason I'd like to see PEP 649 get
accepted: there would never be a reason to call typing.get_type_hints(),
and the values in the Field object would again be type objects.

Back to my original point: If you ignore the test for ClassVar, then
dataclasses completely ignores the values in __annotations__ or
Field.type. It's no different from typing.NamedTuple in that regard.

I do have sympathy for users looking at Field.type and getting a string
instead of a type object: but that's really no different from
non-dataclass users looking at any occurrence of __annotations__ and now
getting a string: that's a result of PEP 563 across the board, not just
with dataclasses. As I said in bpo-39442:

Isn't that the entire point of "from __future__ import annotations"?

Eric

[*]: Actually two things: the other being "is the field of type
dataclasses.InitVar?". It has some of the same problems as ClassVar, but
we know that dataclasses has been imported, so it's less of a big deal.