Mailing List Archive

On Tue, 30 Nov 2021 at 19:07, Brett Cannon <brett@python.org> wrote:
>
> On Tue, Nov 30, 2021 at 9:09 AM Steven D'Aprano <steve@pearwood.info> wrote:
>>
>> On Tue, Nov 30, 2021 at 02:30:18PM +0000, Paul Moore wrote:
>>
>> > And to be clear, it's often very non-obvious how to annotate something
>> > - in https://github.com/pfmoore/editables I basically gave up because
>> > I couldn't work out how to write a maintainable annotation for an
>> > argument that is "a Path, or something that can be passed to the Path
>> > constructor to create a Path" (it's essentially impossible without
>> > copy/pasting the argument annotation for the Path constructor).
>
> You're after https://docs.python.org/3/library/os.html?highlight=pathlike#os.PathLike: `str | PathLike[str]` (if you're only accepting string paths).

Well, it's not really. What I'm after, as I stated, is "anything that
can be passed to the Path constructor". Yes, str | PathLike[str] is
probably close enough (although why is it OK for me to prohibit bytes
paths?) but that's what I mean about copying the Path constructor's
annotations. If Path changes, I have to change my code.

This is a very common idiom:

def f(p: ???):
p = Path(p)
...

Why isn't it correspondingly straightforward to annotate?

If PathLike[str] included str, then it would be a lot easier. It's not
at all obvious to me why it doesn't (well, that's not entirely true -
it's because PathLike is an ABC, not a protocol, and it's not intended
to define "the type of objects that the Path constructor takes"). It
would still not be documented anywhere, though.

>> I thought that type inference was supposed to solve that sort of
>> problem? If the typechecker can see that an argument is passed to the
>> Path constructor, it should be able to infer that it must be the same
>> types as accepted by Path.
>
> I would change that "should" to "may". Python's dynamism makes inferencing really hard.

That's fair. That's why I think it should be straightforward for the
user to explicitly say "this argument should accept the same types as
pathlib.Path does". If inference can't do it automatically, and the
user can't (easily) let the checker know that it's OK to do it, then
we're left with no easy way to express a very common pattern.

>> Aside: I'm a little disappointed in the way the typing ecosystem has
>> developed. What I understood was that we'd get type inference like ML or
>> Haskell use, so we wouldn't need to annotate *everything*, only the bits
>> needed to resolve ambiguity. But what we seem to have got is typing like
>> C, Pascal and Java, except gradual. Am I being unreasonable to be
>> disappointed? I'm not a heavy mypy user, I just dabble with it
>> occasionally, so maybe I've missed something.
>
>
> It really depends on the code base. Type checkers can make guesses based on the code they have available to them, but that only works if the usage is really clear and the dynamic nature of the code doesn't make things murky. For instance, look at open() and how whether you opened a file with `b` or not influences whether the object's methods return strings or bytes. What would you expect to be inferred in that case if you didn't annotate open() with overrides to specify how its arguments influence the returned object?

Personally, I'd be quite happy leaving open() as duck typed. I see
this as what Steven was getting at - the "typing ecosystem" has moved
into a situation where it's acknowledged that some typing problems are
really hard, due to Python's dynamism, and yet there's still a drive
to try to express such highly dynamic type constraints statically. And
worse still, to insist that doing so is somehow necessary.

Whatever happened to "practicality beats purity", and typing being
"gradual"? Surely annotating everything except open() is practical and
gradual?

Paul
_______________________________________________
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-leave@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/YXVXN22TIHBT454CKHKKOCELANF73FZA/
Code of Conduct: http://python.org/psf/codeofconduct/

Another concern I have is the over specification of types.

I have seen many examples of, e.g.

func(x: int, y: float, stuff: List(int]):

but very few of:

func(x: SupportsInt, y: SupportsFloat, stuff: Sequence[SupportsInt]):

(or even Iterable[int])

Is that even the right thing to do to get generic number types? How do I
specify a type that could be any number type (float, int, Fraction,
Decimal, numpy.float32) ... I just spent a few minutes perusing the typing
module and MyPy docs, and didn't find a discussion of that.

But I'd really love to see the community as a whole start with more
generic types in examples, and be clear that the concrete types should be
used only when necessary -- even the typing module docs use a lot of
concrete types in the examples.

I think this is an issue for two reasons:

1) Folks learn from examples more than instruction -- people are very
likely to follow the first example they see that seems to do the job.

2) as mentioned in this thread, library authors are being asked to type
hint their libraries. I think it will be all too common for that type
hinting to be more specific than required. For the most part, that won't
cause any real issues right away, but as soon as someone uses that lib in a
large project that enforces static type checking, it's could get ugly.

-CHB






On Tue, Nov 30, 2021 at 1:37 PM Paul Moore <p.f.moore@gmail.com> wrote:

> On Tue, 30 Nov 2021 at 19:07, Brett Cannon <brett@python.org> wrote:
> >
> > On Tue, Nov 30, 2021 at 9:09 AM Steven D'Aprano <steve@pearwood.info>
> wrote:
> >>
> >> On Tue, Nov 30, 2021 at 02:30:18PM +0000, Paul Moore wrote:
> >>
> >> > And to be clear, it's often very non-obvious how to annotate something
> >> > - in https://github.com/pfmoore/editables I basically gave up because
> >> > I couldn't work out how to write a maintainable annotation for an
> >> > argument that is "a Path, or something that can be passed to the Path
> >> > constructor to create a Path" (it's essentially impossible without
> >> > copy/pasting the argument annotation for the Path constructor).
> >
> > You're after
> https://docs.python.org/3/library/os.html?highlight=pathlike#os.PathLike:
> `str | PathLike[str]` (if you're only accepting string paths).
>
> Well, it's not really. What I'm after, as I stated, is "anything that
> can be passed to the Path constructor". Yes, str | PathLike[str] is
> probably close enough (although why is it OK for me to prohibit bytes
> paths?) but that's what I mean about copying the Path constructor's
> annotations. If Path changes, I have to change my code.
>
> This is a very common idiom:
>
> def f(p: ???):
> p = Path(p)
> ...
>
> Why isn't it correspondingly straightforward to annotate?
>
> If PathLike[str] included str, then it would be a lot easier. It's not
> at all obvious to me why it doesn't (well, that's not entirely true -
> it's because PathLike is an ABC, not a protocol, and it's not intended
> to define "the type of objects that the Path constructor takes"). It
> would still not be documented anywhere, though.
>
> >> I thought that type inference was supposed to solve that sort of
> >> problem? If the typechecker can see that an argument is passed to the
> >> Path constructor, it should be able to infer that it must be the same
> >> types as accepted by Path.
> >
> > I would change that "should" to "may". Python's dynamism makes
> inferencing really hard.
>
> That's fair. That's why I think it should be straightforward for the
> user to explicitly say "this argument should accept the same types as
> pathlib.Path does". If inference can't do it automatically, and the
> user can't (easily) let the checker know that it's OK to do it, then
> we're left with no easy way to express a very common pattern.
>
> >> Aside: I'm a little disappointed in the way the typing ecosystem has
> >> developed. What I understood was that we'd get type inference like ML or
> >> Haskell use, so we wouldn't need to annotate *everything*, only the bits
> >> needed to resolve ambiguity. But what we seem to have got is typing like
> >> C, Pascal and Java, except gradual. Am I being unreasonable to be
> >> disappointed? I'm not a heavy mypy user, I just dabble with it
> >> occasionally, so maybe I've missed something.
> >
> >
> > It really depends on the code base. Type checkers can make guesses based
> on the code they have available to them, but that only works if the usage
> is really clear and the dynamic nature of the code doesn't make things
> murky. For instance, look at open() and how whether you opened a file with
> `b` or not influences whether the object's methods return strings or bytes.
> What would you expect to be inferred in that case if you didn't annotate
> open() with overrides to specify how its arguments influence the returned
> object?
>
> Personally, I'd be quite happy leaving open() as duck typed. I see
> this as what Steven was getting at - the "typing ecosystem" has moved
> into a situation where it's acknowledged that some typing problems are
> really hard, due to Python's dynamism, and yet there's still a drive
> to try to express such highly dynamic type constraints statically. And
> worse still, to insist that doing so is somehow necessary.
>
> Whatever happened to "practicality beats purity", and typing being
> "gradual"? Surely annotating everything except open() is practical and
> gradual?
>
> Paul
> _______________________________________________
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-leave@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-dev@python.org/message/YXVXN22TIHBT454CKHKKOCELANF73FZA/
> Code of Conduct: http://python.org/psf/codeofconduct/
>

On Tue, Nov 30, 2021 at 2:52 PM Christopher Barker <pythonchb@gmail.com>
wrote:

> Another concern I have is the over specification of types.
>
> I have seen many examples of, e.g.
>
> func(x: int, y: float, stuff: List(int]):
>
> but very few of:
>
> func(x: SupportsInt, y: SupportsFloat, stuff: Sequence[SupportsInt]):
>
> (or even Iterable[int])
>
> Is that even the right thing to do to get generic number types? How do I
> specify a type that could be any number type (float, int, Fraction,
> Decimal, numpy.float32) ... I just spent a few minutes perusing the typing
> module and MyPy docs, and didn't find a discussion of that.
>

There is some discussion of the numeric tower in PEP 484 but the PEP says
you should just use 'int', 'float' and be happy. *If* we were to change our
mind on that we would let you spell it using numbers.Integer, numbers.Real
etc., not SupportsInt, SupportsFloat. But the usability of any of those
generic solutions is just terrible (too much typing for too little benefit)
and honestly in many cases the runtime is not very good at these either
(e.g. Decimal refuses to play). In practice it is just fiction that you can
write your own numeric type.

Regarding Iterable[int] vs. Sequence[int], there are subtle semantic
differences so it depends, but these have much better support and *are*
encouraged. (What helps is that List is invariant but Sequence is
covariant, and people often want covariance, so there is a real incentive
to use Sequence, Iterable etc.)


> But I'd really love to see the community as a whole start with more
> generic types in examples, and be clear that the concrete types should be
> used only when necessary -- even the typing module docs use a lot of
> concrete types in the examples.
>

Maybe you are speaking with a lot of enthusiasm but not a lot of
experience? Generics aren't always better, and in many cases they just
aren't worth the effort to get them right (just like not every bit of code
you write needs to be published on PyPI).

I also have a feeling that most frameworks that use *runtime* introspection
of types strongly prefer concrete types, i.e. list[int] rather than
Sequence[T].


> I think this is an issue for two reasons:
>
> 1) Folks learn from examples more than instruction -- people are very
> likely to follow the first example they see that seems to do the job.
>

It would be nice if someone did some work and collected a list of tutorials
about type annotations that exist (especially the ones that are
discoverable with a simple Bing query) and ranked them by quality.


> 2) as mentioned in this thread, library authors are being asked to type
> hint their libraries. I think it will be all too common for that type
> hinting to be more specific than required. For the most part, that won't
> cause any real issues right away, but as soon as someone uses that lib in a
> large project that enforces static type checking, it's could get ugly.
>

That's just the tip of the iceberg. In practice, there are quite a few
reasons why the first few iterations of type hints for many popular
packages fall short of expectations. Often the issue isn't just
sufficiently wide types (like you speculate here) but missing types (for
example, stubs containing functions with arguments but no annotations),
missing methods/attributes, missing classes, and missing submodules. Not to
mention signatures that are impossible to type with the current type system
(you can't actually type zip() or filter(), for example).

We should definitely push back on zealous new converts to typing who insist
that everything should be annotated. But we should also recognize that even
in their current, far from perfect state, type annotations can provide a
lot of value, when used right. (Have you run into VS Code yet? It gets
tremendous value from typing stubs, in the form of improved auto-complete
and hover-doc functionality.)

--
--Guido van Rossum (python.org/~guido)
*Pronouns: he/him **(why is my pronoun here?)*
<http://feministing.com/2015/02/03/how-using-they-as-a-singular-pronoun-can-change-the-world/>

>
>
> Heh. We could update PEP 8 to ban type annotations, then watch as the
> people who over-zealously apply PEP 8 to everything AND
> over-zealously
> insist on adding type annotations to everything have their heads
> explode.
>
>
> --
> Steve
>
I love it!
"Surtout, pas trop de zèle".  [Charles-Maurice de Talleyrand]
Rob Cliffe

On Tue, 30 Nov 2021 at 23:37, Guido van Rossum <guido@python.org> wrote:
>
> We should definitely push back on zealous new converts to typing who insist that everything should be annotated. But we should also recognize that even in their current, far from perfect state, type annotations can provide a lot of value, when used right. (Have you run into VS Code yet? It gets tremendous value from typing stubs, in the form of improved auto-complete and hover-doc functionality.)

I might be misunderstanding but if "hover-doc" is the same as
"mouse-over" then it apparently needs many gigabytes of memory and a
lot of CPU time when you put the mouse over a sympy function that you
are using in your code. It has been suggested that SymPy should fix
this by adding hints like Any but I don't see the point of that: SymPy
has plenty of bugs but this particular bug is in VS Code.

I can see the potential value that type hints can bring for an editor
like VS Code but is the editor also pushing an agenda that all code
should be *explicitly* typed? Does VS Code pressurise people into
using type hints in their own code? I presume it does because I see
the effect in my students (although I might be misattributing this to
VS Code rather than something else).

--
Oscar
_______________________________________________
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-leave@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/C2EEXBT6BEWGOUHBDZ4ILWN6MQZUGQAW/
Code of Conduct: http://python.org/psf/codeofconduct/

On Tue, Nov 30, 2021 at 4:48 PM Oscar Benjamin <oscar.j.benjamin@gmail.com>
wrote:

> On Tue, 30 Nov 2021 at 23:37, Guido van Rossum <guido@python.org> wrote:
> >
> > We should definitely push back on zealous new converts to typing who
> insist that everything should be annotated. But we should also recognize
> that even in their current, far from perfect state, type annotations can
> provide a lot of value, when used right. (Have you run into VS Code yet? It
> gets tremendous value from typing stubs, in the form of improved
> auto-complete and hover-doc functionality.)
>
> I might be misunderstanding but if "hover-doc" is the same as
> "mouse-over" then it apparently needs many gigabytes of memory and a
> lot of CPU time when you put the mouse over a sympy function that you
> are using in your code. It has been suggested that SymPy should fix
> this by adding hints like Any but I don't see the point of that: SymPy
> has plenty of bugs but this particular bug is in VS Code.
>

(Yes, I meant mouse-over. I had a senior moment. :-) I cannot help you much
here, there's a tracker and generally speaking the team does pay attention
to tracker issues, so I recommend starting a discussion there. Feel free to
CC me on the issue if you decide to create one.


> I can see the potential value that type hints can bring for an editor
> like VS Code but is the editor also pushing an agenda that all code
> should be *explicitly* typed? Does VS Code pressurise people into
> using type hints in their own code? I presume it does because I see
> the effect in my students (although I might be misattributing this to
> VS Code rather than something else).
>

There are two modes in VS Code, a lenient and a strict mode. The strict
mode does insist on typing, but you have to opt in to it first. Once you've
opted in, it does indeed encourage you to type everything -- but that's
pretty much the meaning of strict type checking (mypy has a --strict flag
that also requires this).

That said, your students probably copied some configuration that turns on
strict mode from someone else who is thereby pushing this agenda
unwittingly. Sadly most people forget where they got their initial
configuration (I know I forget it as soon as it works :-). And the strict
mode also provides useful error messages in certain cases.

--
--Guido van Rossum (python.org/~guido)
*Pronouns: he/him **(why is my pronoun here?)*
<http://feministing.com/2015/02/03/how-using-they-as-a-singular-pronoun-can-change-the-world/>

On 12/1/2021 12:47 AM, Oscar Benjamin wrote:
> On Tue, 30 Nov 2021 at 23:37, Guido van Rossum <guido@python.org> wrote:
>>
>> We should definitely push back on zealous new converts to typing who insist that everything should be annotated. But we should also recognize that even in their current, far from perfect state, type annotations can provide a lot of value, when used right. (Have you run into VS Code yet? It gets tremendous value from typing stubs, in the form of improved auto-complete and hover-doc functionality.)
>
> I might be misunderstanding but if "hover-doc" is the same as
> "mouse-over" then it apparently needs many gigabytes of memory and a
> lot of CPU time when you put the mouse over a sympy function that you
> are using in your code. It has been suggested that SymPy should fix
> this by adding hints like Any but I don't see the point of that: SymPy
> has plenty of bugs but this particular bug is in VS Code.

For the record, this is the *old* support that has since been replaced
(and was based on full program analysis in the Python of ~2010 that had
no annotations at all, so always struggled with programs that could not
be thoroughly inferred and particularly with those that generated
runtime types - Sympy was a special case in it since about 2012 to avoid
those issues, but that may have been lost when it was migrated from
Visual Studio to VS Code).

The latest versions of VS Code have a completely new system, that does
rely very heavily on programs and libraries being annotated, in exchange
for being much lighter on memory use and preprocessing. But it should
stay out of your way on unannotated code if you haven't enabled its more
strict modes.

Cheers,
Steve
_______________________________________________
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-leave@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/LHNW4CDJVJKNWTCJSTLVMBOBFUYMSWGZ/
Code of Conduct: http://python.org/psf/codeofconduct/

On Tue, Nov 30, 2021 at 3:31 PM Guido van Rossum <guido@python.org> wrote:

>
>> There is some discussion of the numeric tower in PEP 484 but the PEP says
> you should just use 'int', 'float' and be happy.
>

Thanks -- I didn't think to look there. And this: "when an argument is
annotated as having type float, an argument of type int is acceptable"
makes it pretty workable.

I do wonder what happens with oddball numbers, like, say numpy.float32.
But maybe that's Ok, as for though most part one will be dealing with
arrays of them -- the scalars are fairly rare in real code.

however, IIRC, __index__ was introduced at least partially so that numpy
integer types could be used as indexes. Which would mean SupportsIndex for
a type that's going to be used as an index, yes?


> solutions is just terrible (too much typing for too little benefit) and
> honestly in many cases the runtime is not very good at these either (e.g.
> Decimal refuses to play). In practice it is just fiction that you can write
> your own numeric type.
>

Yes, Decimal and Fraction are pretty special purpose, really.


> Regarding Iterable[int] vs. Sequence[int], there are subtle semantic
> differences so it depends, but these have much better support and *are*
> encouraged. (What helps is that List is invariant but Sequence is
> covariant, and people often want covariance, so there is a real incentive
> to use Sequence, Iterable etc.)
>
>
>> But I'd really love to see the community as a whole start with more
>> generic types in examples, and be clear that the concrete types should be
>> used only when necessary -- even the typing module docs use a lot of
>> concrete types in the examples.
>>
>
> Maybe you are speaking with a lot of enthusiasm but not a lot of
> experience?
>

well, yes, for sure.


> Generics aren't always better, and in many cases they just aren't worth
> the effort to get them right
>

I suspect that depends a bit on the use case. For something internal to one
system, probably not worth the effort at all. But for a general purpose
library, I'm (perhaps needlessly) concerned about types getting locked down
more than they need to be, just because its easier, and "it works for me"


> (just like not every bit of code you write needs to be published on PyPI).
>

Someone should tell that to all the folks publishing 0.0.0.1 version of
packages :-)


> I also have a feeling that most frameworks that use *runtime*
> introspection of types strongly prefer concrete types, i.e. list[int]
> rather than Sequence[T].
>

This is getting into what it means to use Types at runtime -- if it's
runtime type checking, then more abstract types might be fine. But if it's
using the type itself (like my use case), then absolutely -- my system will
only work with real concrete type objects. Though that may not play that
well with type checkers :-(


> It would be nice if someone did some work and collected a list of
> tutorials about type annotations that exist (especially the ones that are
> discoverable with a simple Bing query) and ranked them by quality.
>

yes, it would.


> We should definitely push back on zealous new converts to typing who
> insist that everything should be annotated.
>

well, we got folks wanting to change PEP 8 becuase they don't want their
linter to complain -- so it will be a battle.


> (Have you run into VS Code yet? It gets tremendous value from typing
> stubs, in the form of improved auto-complete and hover-doc functionality.)
>

Some of the folks on my team use it -- I've been meaning to check it out --
one more reason now.

-CHB

--
Christopher Barker, PhD (Chris)

Python Language Consulting
- Teaching
- Scientific Software Development
- Desktop GUI and Web Development
- wxPython, numpy, scipy, Cython

On Tue, Nov 30, 2021 at 1:34 PM Paul Moore <p.f.moore@gmail.com> wrote:

> On Tue, 30 Nov 2021 at 19:07, Brett Cannon <brett@python.org> wrote:
> >
> > On Tue, Nov 30, 2021 at 9:09 AM Steven D'Aprano <steve@pearwood.info>
> wrote:
> >>
> >> On Tue, Nov 30, 2021 at 02:30:18PM +0000, Paul Moore wrote:
> >>
> >> > And to be clear, it's often very non-obvious how to annotate something
> >> > - in https://github.com/pfmoore/editables I basically gave up because
> >> > I couldn't work out how to write a maintainable annotation for an
> >> > argument that is "a Path, or something that can be passed to the Path
> >> > constructor to create a Path" (it's essentially impossible without
> >> > copy/pasting the argument annotation for the Path constructor).
> >
> > You're after
> https://docs.python.org/3/library/os.html?highlight=pathlike#os.PathLike:
> `str | PathLike[str]` (if you're only accepting string paths).
>
> Well, it's not really. What I'm after, as I stated, is "anything that
> can be passed to the Path constructor". Yes, str | PathLike[str] is
> probably close enough (although why is it OK for me to prohibit bytes
> paths?)


It's your code; do what you want. ???? If you want to accept bytes, then you
can accept bytes. I personally just don't care about the bytes edge case,
so I leave it out. Plus I don't think pathlib supports bytes paths to begin
with, so if you are after just the pathlib.Path case then you don't want
bytes unless you're going to handle the decoding.

But if you want bytes then you're after
https://github.com/python/typeshed/blob/8542b3518a36313af57b13227996168e592fe868/stdlib/_typeshed/__init__.pyi#L84
which is what `open()` takes (plus ints): `str | bytes | os.PathLike[str] |
os.PathLike[bytes]`.


> but that's what I mean about copying the Path constructor's
> annotations. If Path changes, I have to change my code.
>

Yep. You are writing down your expectations of what is acceptable to pass
in and you're choosing to be very flexible, and so you have to write down
more. I don't think anyone is claiming that typing your code takes no
effort.

But practically speaking the constructor to pathlib.Path isn't going to
change.


>
> This is a very common idiom:
>
> def f(p: ???):
> p = Path(p)
> ...
>
> Why isn't it correspondingly straightforward to annotate?
>

If we added an object to pathlib that represented the type annotation for
the constructor of pathlib then it wouldn't be. It's probably not an
unreasonable idea since this idiom is common enough, I just don't know if
anyone has bothered to suggest it for the module.


>
> If PathLike[str] included str, then it would be a lot easier. It's not
> at all obvious to me why it doesn't (well, that's not entirely true -
> it's because PathLike is an ABC, not a protocol, and it's not intended
> to define "the type of objects that the Path constructor takes"). It
> would still not be documented anywhere, though.
>

You're essentially wrapping the constructor to `pathlib.Path` as broadly as
possible, which means you want types as broad as possible while still being
accurate. You could also tell users, "give me pathlib.Path objects" and
your troubles go away or "an object that implements the `__fspath__`
protocol" (which is what `os.PathLike` represents, hence why `str` isn't
covered by it); it all depends on what sort of assumptions you want to be
able to make about what you're given. But because you're wanting to accept
multiple types to support both the "old" way of string paths and the "new"
way of `__fspath__` you then need to put the work in to accept those types.

Personally, I treat string paths vs pathlib object paths like encoding and
decoding strings; get it converted immediately at the boundary of your code
and then just assume pathlib everywhere else. Pathlib was added in Python
3.4 and is well-known enough at this point that I don't worry about people
not being aware of it, so I let users do the glue from any string paths
they get to my APIs.

But to be clear, path representations != pathlib.Path type parameters !=
`__fspath__` protocol; there's subtlety here regardless of the types.


>
> >> I thought that type inference was supposed to solve that sort of
> >> problem? If the typechecker can see that an argument is passed to the
> >> Path constructor, it should be able to infer that it must be the same
> >> types as accepted by Path.
> >
> > I would change that "should" to "may". Python's dynamism makes
> inferencing really hard.
>
> That's fair. That's why I think it should be straightforward for the
> user to explicitly say "this argument should accept the same types as
> pathlib.Path does". If inference can't do it automatically, and the
> user can't (easily) let the checker know that it's OK to do it, then
> we're left with no easy way to express a very common pattern.
>

But how would you specify that? And what if the thing you're wrapping isn't
typed itself? And what if the object you're wrapping takes multiple
arguments? You could talk to the typing-sig and see if they have discussed
some sort of `Parameter[pathlib.Path][0]` object to specify the type of the
first argument to `pathlib.Path`.


>
> >> Aside: I'm a little disappointed in the way the typing ecosystem has
> >> developed. What I understood was that we'd get type inference like ML or
> >> Haskell use, so we wouldn't need to annotate *everything*, only the bits
> >> needed to resolve ambiguity. But what we seem to have got is typing like
> >> C, Pascal and Java, except gradual. Am I being unreasonable to be
> >> disappointed? I'm not a heavy mypy user, I just dabble with it
> >> occasionally, so maybe I've missed something.
> >
> >
> > It really depends on the code base. Type checkers can make guesses based
> on the code they have available to them, but that only works if the usage
> is really clear and the dynamic nature of the code doesn't make things
> murky. For instance, look at open() and how whether you opened a file with
> `b` or not influences whether the object's methods return strings or bytes.
> What would you expect to be inferred in that case if you didn't annotate
> open() with overrides to specify how its arguments influence the returned
> object?
>
> Personally, I'd be quite happy leaving open() as duck typed. I see
> this as what Steven was getting at - the "typing ecosystem" has moved
> into a situation where it's acknowledged that some typing problems are
> really hard, due to Python's dynamism, and yet there's still a drive
> to try to express such highly dynamic type constraints statically. And
> worse still, to insist that doing so is somehow necessary.
>

Sure, but how would you have expected old code that isn't about to change
to be typed?


>
> Whatever happened to "practicality beats purity", and typing being
> "gradual"? Surely annotating everything except open() is practical and
> gradual?
>

Yep, hence why people can still use code you didn't type and get *some*
benefit from typing still. But there are limits as flowing types through
untyped code is hard, hence why your users are asking for types; t limits
what *they* can check for without writing type annotations for your code
(at which point they are now the ones worrying about a drifting of types,
much like you are with pathlib.Path). Code like pathlib.Path that can take
various types or other things that can return different types become
something of a blackhole for typing because the type checkers just can't
figure out what you want or it figures it out to be so broad as to be
useless.


>
> Paul
>

I know this isn't really the place for this conversation, but:


> which is what `os.PathLike` represents, hence why `str` isn't covered by
> it);
>

wait, what? It seems so clear to me that "PathLike" (as a type specifier)
would mean: anything that can be passed into os.fspath to give me a path.
(or, of course to the stdlib functions that take paths)

Isn't the entire purpose of os.fspath that you can write code like:

def fun(some_kind_of_path):
some_kind_of_path = os.fspath(some_kind_of_path)

(or just pass it to a function you takes PathLIke)

and go on your merry way -- e.g. duck typing, baby!

Is there really no way to annotate that simply now?

-CHB

--
Christopher Barker, PhD (Chris)

Python Language Consulting
- Teaching
- Scientific Software Development
- Desktop GUI and Web Development
- wxPython, numpy, scipy, Cython

On Wed, Dec 1, 2021 at 10:50 PM Christopher Barker <pythonchb@gmail.com>
wrote:

> I know this isn't really the place for this conversation, but:
>
>
>> which is what `os.PathLike` represents, hence why `str` isn't covered by
>> it);
>>
>
> wait, what? It seems so clear to me that "PathLike" (as a type specifier)
> would mean: anything that can be passed into os.fspath to give me a path.
> (or, of course to the stdlib functions that take paths)
>
> Isn't the entire purpose of os.fspath that you can write code like:
>
> def fun(some_kind_of_path):
> some_kind_of_path = os.fspath(some_kind_of_path)
>
> (or just pass it to a function you takes PathLIke)
>
> and go on your merry way -- e.g. duck typing, baby!
>
> Is there really no way to annotate that simply now?
>

Assuming you want the return value of 'fun' to be covariant with the path
input, I believe you would say this:

def fun(some_kind_of_path: str) -> str: ...
def fun(some_kind_of_path: bytes) -> bytes: ...
def fun(some_kind_of_path: os.PathLike[AnyStr]) -> AnyStr:
some_kind_of_path = os.fspath(some_kind_of_path)
# transform it
return some_kind_of_path

I would love to be shown how to do this with just a one-line declaration of
'fun', but I've given up trying to figure it out.

On Wed, Dec 1, 2021 at 10:40 PM Christopher Barker <pythonchb@gmail.com>
wrote:

> I know this isn't really the place for this conversation, but:
>
>
>> which is what `os.PathLike` represents, hence why `str` isn't covered by
>> it);
>>
>
> wait, what? It seems so clear to me that "PathLike" (as a type specifier)
> would mean: anything that can be passed into os.fspath to give me a path.
> (or, of course to the stdlib functions that take paths)
>

That is not what the docs say:
https://docs.python.org/3/library/os.html#os.PathLike. And as the creator
of that ABC it's very much on purpose (see
https://www.python.org/dev/peps/pep-0519/ for details).


> Isn't the entire purpose of os.fspath that you can write code like:
>
> def fun(some_kind_of_path):
> some_kind_of_path = os.fspath(some_kind_of_path)
>
> (or just pass it to a function you takes PathLIke)
>
> and go on your merry way -- e.g. duck typing, baby!
>

Depends on what "your merry way" is. As the docs say, os.fspath() is about
getting the file system representation. That's not something to directly
manipulate in a pathlib world unless you're using os.path to manipulate
that string encoding (but which PEP 519 was specifically created to avoid
such encoding headaches):
https://docs.python.org/3/library/os.html#os.fspath .


>
> Is there really no way to annotate that simply now?
>

Once again, depends on what "simply" means to you. The examples I gave I
don't find complicated, especially when we are talking about APIs that are
trying to accept a broad set of types to convert into a single type. If you
want to create a type variable that represents anything that is valid to
`os.fspath()` or `pathlib.Path` to avoid typing out 2 or 4 types, then that
can be considered for inclusion in the stdlib somewhere.

But as I said previously, in my code I just take pathlib.Path or
pathlib.PurePath and I'm done as I don't want to be dealing with encoded
file paths and instead with objects that represent file paths.


>
> -CHB
>
> --
> Christopher Barker, PhD (Chris)
>
> Python Language Consulting
> - Teaching
> - Scientific Software Development
> - Desktop GUI and Web Development
> - wxPython, numpy, scipy, Cython
>

On Thu, Dec 2, 2021 at 9:48 AM Eric Fahlgren <ericfahlgren@gmail.com> wrote:

> wait, what? It seems so clear to me that "PathLike" (as a type specifier)
> would mean: anything that can be passed into os.fspath to give me a path.
> (or, of course to the stdlib functions that take paths)
>>
>>
>> Isn't the entire purpose of os.fspath that you can write code like:
>>
>> def fun(some_kind_of_path):
>> some_kind_of_path = os.fspath(some_kind_of_path)
>>
>> (or just pass it to a function you takes PathLIke)
>>
>> and go on your merry way -- e.g. duck typing, baby!
>>
>> Is there really no way to annotate that simply now?
>>
>
> Assuming you want the return value of 'fun' to be covariant with the path
> input, I believe you would say this:
>

I just read the wikipedia page on "Covariant return type", and I either
misunderstood what it means, or it's not relevant to my intended example.


> def fun(some_kind_of_path: str) -> str: ...
> def fun(some_kind_of_path: bytes) -> bytes: ...
> def fun(some_kind_of_path: os.PathLike[AnyStr]) -> AnyStr:
> some_kind_of_path = os.fspath(some_kind_of_path)
> # transform it
> return some_kind_of_path
>

Ahh I see, I wasn't intending, in my example, to return the path -- my
intent was a function that would need to use the path, to, e.g. open a file.

This Strikes me as a very common use case -- you want your users to be able
to pass multiple different types in, as long as they can be used as a path.
I have a LOT of code that does this -- for years it accepted only string
paths, and now I'm updating it to take PathLike as well.

My usual code is one of:

def fun_that_opens_a_file(the_path):
# if I'm working with an "old" library, or one that, e.g. wraps a C lib:
the_path = os.fspath(the_path)
# if I'm working with things I know will take a Path object:
the_path = Path(the_path)

>
> I would love to be shown how to do this with just a one-line declaration
> of 'fun', but I've given up trying to figure it out.
>

Darn. could you at least use a Union type:

def fun_that_opens_a_file(the_path: Union[str, Path]):
...

which isn't too bad. Personally, I think accepting bytes is the way of
madness, but you could add that too.

On Thu, Dec 2, 2021 at 12:32 PM Brett Cannon <brett@python.org> wrote:

>
> wait, what? It seems so clear to me that "PathLike" (as a type specifier)
> would mean: anything that can be passed into os.fspath to give me a path.
> (or, of course to the stdlib functions that take paths)
>
> That is not what the docs say:
> https://docs.python.org/3/library/os.html#os.PathLike. And as the creator
> of that ABC it's very much on purpose (see
> https://www.python.org/dev/peps/pep-0519/ for details).
>

Sorry, I wasn't clear. By "PathLike" (as a type specifier)" I meant using
it as, well, a type specifier, not as an ABC -- often we can use an ABC as
a type specifier, but not always. But you've prompted me to go re-read the
PEP and I see this:

"""
Provide specific type hinting support

There was some consideration to providing a generic typing.PathLike class
which would allow for e.g. typing.PathLike[str] to specify a type hint for
a path object which returned a string representation. While potentially
beneficial, the usefulness was deemed too small to bother adding the type
hint class.

This also removed any desire to have a class in the typing module which
represented the union of all acceptable path-representing types as that can
be represented with typing.Union[str, bytes, os.PathLike] easily enough and
the hope is users will slowly gravitate to path objects only.
"""

I didn't pay any attention to that at the time, as I wasn't paying
attention to typing. But personally, I think I do have that desire :-)

But I can see that "the hope is users will slowly gravitate to path objects
only".

would lead to:

> in my code I just take pathlib.Path or pathlib.PurePath and I'm done as
I don't want to be dealing with encoded file paths and instead with objects
that represent file paths.

For my part, I'm not interested in encoded paths (e.g. bytes) -- that is
absolutely not the right boundary to deal with file system encoding.
(that's mentioned in the PEP) But I also don't want my users (or me, with
old code, scripts, etc) to have to suddenly go in and wrap a call to Path()
everywhere in order to continue to use my library. String paths are
ubiquitous, and I think here to stay.

I really like PEP 519 -- I think it not only provides a transition, but
also a future in which Path objects and string paths can continue to play
well together.

While Union[PathLike, str] is a pretty light lift, this is one tiny example
of what we "typing skeptics" are concerned about: a transition away from
full on dynamic typing to a more static style.

Maybe that won't come to pass -- we'll see.

-CHB


--
Christopher Barker, PhD (Chris)

Python Language Consulting
- Teaching
- Scientific Software Development
- Desktop GUI and Web Development
- wxPython, numpy, scipy, Cython

> It would be nice if someone did some work and collected a list of tutorials about type annotations that exist (especially the ones that are discoverable with a simple Bing query) and ranked them by quality.

I went with Google rather than Bing but here's what I found:
https://gist.github.com/stroxler/ade7977ed07e27448222a468796bc467

I did check a few Bing queries, the results are mostly a bit less helpful, I get more discussions of the `type` function and fewer about type annotations. But the Real Python tutorial still shows up on the first page most of the time.
_______________________________________________
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-leave@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/AHJCF57LISD7G7DHLSQLKDKGN6FFHT3M/
Code of Conduct: http://python.org/psf/codeofconduct/