Mailing List Archive

1 2  View All
Re: PEP 654: Exception Groups and except* [REPOST] [ In reply to ]
Hi Irit,

reading this subthread specifically, I just got a wild idea and I couldn‘t find any related information in the PEP:

Why not extending BaseException by __group__ among __cause__ and __context__?

Would this reduce some of the added complexity and thus increase broader acceptance?

Cheers,
Sven


> On 7. Apr 2021, at 20:26, Irit Katriel via Python-Dev <python-dev@python.org> wrote:
> ?
>
> On Mon, Apr 5, 2021 at 2:59 PM Chris Jerdonek <chris.jerdonek@gmail.com> wrote:
>> This point reminded me again of this issue in the tracker ("Problems with recursive automatic exception chaining" from 2013): https://bugs.python.org/issue18861
>> I'm not sure if it's exactly the same, but you can see that a couple of the later comments there talk about "exception trees" and other types of annotations.
>>
>> If that issue were addressed after ExceptionGroups were introduced, does that mean there would then be two types of exception-related trees layered over each other (e.g. groups of trees, trees of groups, etc)? It makes me wonder if there's a more general tree structure that could accommodate both use cases...
>>
>> --Chris
>
> Interesting, I commented on that issue - I think we may be able to solve it without adding more trees.
>
> That said, we will have groups-of-trees/trees-of-groups. Already today, an exception plus its chained __cause__s and __context__s is the root of a binary tree of exceptions. The nodes of this tree represent the times that the exceptions were caught.
>
> An exception group is a tree where the nodes represent the times when exceptions were grouped together and raised.
>
> Irit
> _______________________________________________
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-leave@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/BZWELKDUAKCOXSH5KQRFGQJRQWJ2OHKW/
> Code of Conduct: http://python.org/psf/codeofconduct/
Re: PEP 654: Exception Groups and except* [REPOST] [ In reply to ]
Hi Sven,

I don’t follow. What would the value of __group__ be and how would it work?

Irit

> On 20 Apr 2021, at 20:44, srkunze@mail.de wrote:
>
> ?
> Hi Irit,
>
> reading this subthread specifically, I just got a wild idea and I couldn‘t find any related information in the PEP:
>
> Why not extending BaseException by __group__ among __cause__ and __context__?
>
> Would this reduce some of the added complexity and thus increase broader acceptance?
>
> Cheers,
> Sven

_______________________________________________
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-leave@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/NYAXQQ7EVXUNX7MTQJDLAXVKGRHAUSEF/
Code of Conduct: http://python.org/psf/codeofconduct/
Re: PEP 654: Exception Groups and except* [REPOST] [ In reply to ]
So, forgive me my relatively simple mental model about ExceptionGroup. I still try to create one for daily use.

As noted in the discussion, an EG provides a way to collect exceptions from different sources and raise them as a bundle. They have no apparent relation up until this point in time (for whatever reason they have been separate and for whatever reason they are bundled now). The result would be a tree graph in any case.

A usual datastructure for a tree is to store all child nodes at the parent node.

That was the idea behind the content of BaseException.__group__: it’s the list of child exceptions bundled at a specific point in time and raise as such a bundle. So all exceptions could become EGs with the additional semantics you‘ve described in the PEP.

Illustrative Example:
>>> bundle_exc.__group__
[IOError(123), RuntimerError(‘issue somewhere’)]

I was wondering what of the PEP could be removed to make it simpler and more acceptable/less confusing (also looking at reactions from Twitter etc.) and I found these additional classes to be a part of it. Additionally, I fail to see how to access these bundled exceptions in an easy manner like __cause__ and __context__. (As the PEP also referring to them). So, I removed the classes and added a regular attribute.

The reason I brought this up what the section “rejected ideas” didn’t showed anything in this direction (or I managed to missed that).

Sven

>
> On 20. Apr 2021, at 22:05, Irit Katriel <iritkatriel@yahoo.com> wrote:
>
> ?Hi Sven,
>
> I don’t follow. What would the value of __group__ be and how would it work?
>
> Irit
>
>> On 20 Apr 2021, at 20:44, srkunze@mail.de wrote:
>> ?
>> Hi Irit,
>> reading this subthread specifically, I just got a wild idea and I couldn‘t find any related information in the PEP:
>> Why not extending BaseException by __group__ among __cause__ and __context__?
>> Would this reduce some of the added complexity and thus increase broader acceptance?
>> Cheers,
>> Sven

_______________________________________________
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-leave@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/4NUG6R36HN557AY36SPINNRFRYBW4QM7/
Code of Conduct: http://python.org/psf/codeofconduct/
Re: PEP 654: Exception Groups and except* [REPOST] [ In reply to ]
I don’t see what this simplifies. We still need to implement split, and to worry about wrapping or not wrapping BaseExceptions and we still need to define exception handling semantics (except/except*).


> On 20 Apr 2021, at 22:12, srkunze@mail.de wrote:
>
> ?So, forgive me my relatively simple mental model about ExceptionGroup. I still try to create one for daily use.
>
> As noted in the discussion, an EG provides a way to collect exceptions from different sources and raise them as a bundle. They have no apparent relation up until this point in time (for whatever reason they have been separate and for whatever reason they are bundled now). The result would be a tree graph in any case.
>
> A usual datastructure for a tree is to store all child nodes at the parent node.
>
> That was the idea behind the content of BaseException.__group__: it’s the list of child exceptions bundled at a specific point in time and raise as such a bundle. So all exceptions could become EGs with the additional semantics you‘ve described in the PEP.
>
> Illustrative Example:
>>>> bundle_exc.__group__
> [IOError(123), RuntimerError(‘issue somewhere’)]
>
> I was wondering what of the PEP could be removed to make it simpler and more acceptable/less confusing (also looking at reactions from Twitter etc.) and I found these additional classes to be a part of it. Additionally, I fail to see how to access these bundled exceptions in an easy manner like __cause__ and __context__. (As the PEP also referring to them). So, I removed the classes and added a regular attribute.
>
> The reason I brought this up what the section “rejected ideas” didn’t showed anything in this direction (or I managed to missed that).
>
> Sven
>
>>
>> On 20. Apr 2021, at 22:05, Irit Katriel <iritkatriel@yahoo.com> wrote:
>>
>> ?Hi Sven,
>>
>> I don’t follow. What would the value of __group__ be and how would it work?
>>
>> Irit
>>
>>>> On 20 Apr 2021, at 20:44, srkunze@mail.de wrote:
>>> ?
>>> Hi Irit,
>>> reading this subthread specifically, I just got a wild idea and I couldn‘t find any related information in the PEP:
>>> Why not extending BaseException by __group__ among __cause__ and __context__?
>>> Would this reduce some of the added complexity and thus increase broader acceptance?
>>> Cheers,
>>> Sven
>
_______________________________________________
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-leave@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/OJJXMPBD4ZUZOUZ2ULWT4Q6ULI75HLDE/
Code of Conduct: http://python.org/psf/codeofconduct/
Re: PEP 654: Exception Groups and except* [REPOST] [ In reply to ]
Removing two concepts and preserving semantics simplifies the matter for users. People need less to memorize and less to learn.

Or am I missing something here? Couldn’t we achieve our goal without these two new classes?

Splitting, wrapping and exception handling semantics are perfectly fine and serve their purposes. So, of course this still needs to be implemented.

> On 20. Apr 2021, at 23:26, Irit Katriel <iritkatriel@googlemail.com> wrote:
>
> ?I don’t see what this simplifies. We still need to implement split, and to worry about wrapping or not wrapping BaseExceptions and we still need to define exception handling semantics (except/except*).
>
>
>> On 20 Apr 2021, at 22:12, srkunze@mail.de wrote:
>>
>> ?So, forgive me my relatively simple mental model about ExceptionGroup. I still try to create one for daily use.
>>
>> As noted in the discussion, an EG provides a way to collect exceptions from different sources and raise them as a bundle. They have no apparent relation up until this point in time (for whatever reason they have been separate and for whatever reason they are bundled now). The result would be a tree graph in any case.
>>
>> A usual datastructure for a tree is to store all child nodes at the parent node.
>>
>> That was the idea behind the content of BaseException.__group__: it’s the list of child exceptions bundled at a specific point in time and raise as such a bundle. So all exceptions could become EGs with the additional semantics you‘ve described in the PEP.
>>
>> Illustrative Example:
>>>>> bundle_exc.__group__
>> [IOError(123), RuntimerError(‘issue somewhere’)]
>>
>> I was wondering what of the PEP could be removed to make it simpler and more acceptable/less confusing (also looking at reactions from Twitter etc.) and I found these additional classes to be a part of it. Additionally, I fail to see how to access these bundled exceptions in an easy manner like __cause__ and __context__. (As the PEP also referring to them). So, I removed the classes and added a regular attribute.
>>
>> The reason I brought this up what the section “rejected ideas” didn’t showed anything in this direction (or I managed to missed that).
>>
>> Sven
>>
>>>
>>>> On 20. Apr 2021, at 22:05, Irit Katriel <iritkatriel@yahoo.com> wrote:
>>>
>>> ?Hi Sven,
>>>
>>> I don’t follow. What would the value of __group__ be and how would it work?
>>>
>>> Irit
>>>
>>>>> On 20 Apr 2021, at 20:44, srkunze@mail.de wrote:
>>>> ?
>>>> Hi Irit,
>>>> reading this subthread specifically, I just got a wild idea and I couldn‘t find any related information in the PEP:
>>>> Why not extending BaseException by __group__ among __cause__ and __context__?
>>>> Would this reduce some of the added complexity and thus increase broader acceptance?
>>>> Cheers,
>>>> Sven
>>

_______________________________________________
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-leave@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/OD6IDT5HGDULGLVVQ2HF7WJXNQM7RA3K/
Code of Conduct: http://python.org/psf/codeofconduct/
Re: PEP 654: Exception Groups and except* [REPOST] [ In reply to ]
On Wed, Apr 21, 2021 at 11:50 AM srkunze@mail.de <srkunze@mail.de> wrote:
>
> Removing two concepts and preserving semantics simplifies the matter for users. People need less to memorize and less to learn.
>
> Or am I missing something here? Couldn’t we achieve our goal without these two new classes?

No, we can't. What you are proposing would make it very hard for users
to understand at a glance if what you have in an innocently looking
`except Exception` is correct or not. In my async/await code I'd have
to always check the `__group__` attribute to make sure it's not an
exception group in disguise.

So while you're "simplifying" the proposal by removing a couple of
types, you're complicating it in all other places. Besides, I don't
think that adding the ExceptionGroup type is a controversial idea that
needs any simplification.

Yury
_______________________________________________
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-leave@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/IXHLXQYTEFPRC2GLFGAM6GHCYAAQG4DQ/
Code of Conduct: http://python.org/psf/codeofconduct/
Re: PEP 654: Exception Groups and except* [REPOST] [ In reply to ]
On Tue, Apr 20, 2021 at 2:15 PM srkunze@mail.de <srkunze@mail.de> wrote:
>
> So, forgive me my relatively simple mental model about ExceptionGroup. I still try to create one for daily use.
>
> As noted in the discussion, an EG provides a way to collect exceptions from different sources and raise them as a bundle. They have no apparent relation up until this point in time (for whatever reason they have been separate and for whatever reason they are bundled now). The result would be a tree graph in any case.
>
> A usual datastructure for a tree is to store all child nodes at the parent node.
>
> That was the idea behind the content of BaseException.__group__: it’s the list of child exceptions bundled at a specific point in time and raise as such a bundle. So all exceptions could become EGs with the additional semantics you‘ve described in the PEP.
>
> Illustrative Example:
> >>> bundle_exc.__group__
> [IOError(123), RuntimerError(‘issue somewhere’)]
>
> I was wondering what of the PEP could be removed to make it simpler and more acceptable/less confusing (also looking at reactions from Twitter etc.) and I found these additional classes to be a part of it. Additionally, I fail to see how to access these bundled exceptions in an easy manner like __cause__ and __context__. (As the PEP also referring to them). So, I removed the classes and added a regular attribute.

This seems more confusing to me too. Instead of having a single
ExceptionGroup class, you're suggesting that all exceptions should
effectively become ExceptionGroups. I know what
ExceptionGroup([ValueError]) means -- it means there was a ValueError,
and it got put in a group, so it should probably be handled the same
way as a ValueError. I have no idea what a KeyError([ValueError])
would mean. Is that a KeyError or a ValueError? Adding flexibility
doesn't necessarily make things simpler :-)

-n

--
Nathaniel J. Smith -- https://vorpus.org
_______________________________________________
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-leave@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/F6LVP7CLQGIAZ2PI2SL6FN3VXF2MVY2M/
Code of Conduct: http://python.org/psf/codeofconduct/
Re: PEP 654: Exception Groups and except* [REPOST] [ In reply to ]
On Tue, Apr 20, 2021 at 3:15 AM Irit Katriel <iritkatriel@googlemail.com> wrote:
> On Tue, Apr 20, 2021 at 2:48 AM Nathaniel Smith <njs@pobox.com> wrote:
>>
>>
>> The problem is that most of the time, even if you're using concurrency
>> internally so multiple things *could* go wrong at once, only one thing
>> actually *does* go wrong. So it's unfortunate if some existing code is
>> prepared for a specific exception that it knows can be raised, that
>> exact exception is raised... and the existing code fails to catch it
>> because now it's wrapped in an EG.
>
> Yes, this was discussed at length on this list. Raising an exception group is an API-breaking change. If a function starts raising exception groups its callers need to be prepared for that. Realistically we think exception groups will be raised by new APIs. We tried and were unable to define exception group semantics for except that would be reasonable and backwards compatible. That's why we added except*.

Sure. This was in my list of reasons why the backwards compatibility
tradeoffs are forcing us into awkward compromises. I only elaborated
on it b/c in your last email you said you didn't understand why this
was a problem :-). And except* is definitely useful. But I think there
are options for 'except' that haven't been considered fully.

Saying that you have to make a new API every time you start using
concurrency inside a function is extremely restrictive. The whole
point of structured concurrency (and structured programming in
general) is that function callers don't need to know about the control
flow decisions inside a function. So right now, the EG proposal is
like saying that if you every have a function that doesn't contain a
'for' loop, and then you want to add a 'for' loop, then you have to
define a whole new API instead of modifying the existing one.

I absolutely get why the proposal looks like that. I'm just making the
point that we should make sure we've exhausted all our options before
settling for that as a compromise.

>> > It is easy enough to write a denormalize() function in traceback.py that constructs this from the current EG structure, if you need it (use the leaf_generator from the PEP). I'm not sure I see why we should trouble the interpreter with this.
>>
>> In the current design, once an exception is wrapped in an EG, then it
>> can never be unwrapped, because its traceback information is spread
>> across the individual exception + the EG tree around it. This is
>> confusing to users ("how do I check errno?"), and makes the design
>> more complicated (the need for topology-preserving .split(), the
>> inability to define a sensible EG.__iter__, ...). The advantage of
>> making the denormalized form the native form is that now the leaf
>> exceptions would be self-contained objects like they are now, so you
>> don't need EG nesting at all, and users can write intuitive code like:
>>
>> except OSError as *excs:
>> remainder = [exc for exc in excs if exc.errno != ...]
>> if remainder:
>> raise ExceptionGroup(remainder)
>
>
> We have this precise example in the PEP:
> match, rest = excs.split(lambda e: e.errno != ...)
>
> You use split() instead of iteration for that. split() preserves all __context__, __cause__ and __traceback__ information, on all leaf and non-leaf exceptions.

Well, yeah, I know, I'm the one who invented split() :-). My point was
to compare these two options: in the flat EG example, most Python
users could write and read that code without knowing anything except
"there can be multiple exceptions now". It's all old, well-known
constructs used in the obvious way.

For the .split() version, you have to write a lambda (which is allowed
to access parts of the exception object, but not all of it!), and use
this idiosyncratic method that only makes sense if you know about EG
tree structures. That's a lot more stuff that users have to understand
and hold in their head.

Or here's another example. Earlier you suggested:

> If you are using concurrency internally and don't want to raise EGs externally, then surely you will catch EGs, select one of the exceptions to raise and throw away all the others

But with nested EGs, this is difficult, because you *can't* just pull
out one of the exceptions to raise. The leaf exceptions aren't
standalone objects; you need some obscure traceback manipulation to do
this. I guarantee that users will get this wrong, even if we provide
the tools, because the explanation about when and why you need the
tools is complicated and people won't internalize it.

With flat EGs, this is trivial: it's just `raise
ExceptionGroup[the_selected_exception_index]`.

>> > For display purposes, it is probably nicer to look at a normalized traceback where common parts are not repeated.
>>
>> Yeah, I agree; display code would want to re-normalize before
>> printing. But now it's only the display code that needs to care about
>> figuring out shared parts of the traceback, rather than something that
>>
>> has to be maintained as an invariant everywhere.
>
> If you *do* want iteration, we show in the PEP how to write a leaf_generator() that gives you the leaf exceptions with their tracebacks (as a list of chunks). It is easy to copy the chunks into a single flat traceback. We didn't propose to add it to traceback.py yet because the use case is unclear but if people need it we're talking about 10-15 lines in traceback.py.
>
> So for your suggestion:
>
> Pros:
> 1. Those who want a denormalized traceback (if they exist) won't need to call traceback.denormalize().
>
> Cons:
> 1. A significant change in the interpreter that will make it less efficient (both time and space).

In terms of interpreter complexity, the straightforward implementation
would be pretty simple. Whenever the exception handling code goes to
write back a traceback to an exception's __traceback__ attribute,
check if the exception is an ExceptionGroup; if so, then loop over
each exception in the group and push a clone of the traceback onto
that exception's __traceback__. It takes code of course, everything
does, but I'd be surprised if it was more than, say, 20 lines of C. I
think that's a pretty good tradeoff to get a simpler user-level
language semantics. Python is supposed to fit in your head :-).

Time/space efficiency is interesting, because yeah, with the nested EG
approach then common parts of the traceback are stored once, while the
flat EG approach needs to store a separate copy for each exception.
But:

- The extra cost is 1 traceback object per common stack frame, per
exception. So the worst case is if you have, like, a 1000 exceptions
in a single group, and then that group travels together for 1000 stack
frames. But... is that realistic? Very few programs recurse down 1000
levels and *then* spawn a bunch of tasks. And very few programs have
1000 tasks that all fail simultaneously with different exceptions. And
even if that did happen, 1000*1000 = 1 million traceback objects take
~56 megabytes of RAM and ~90 ms to instantiate on my laptop, which is
not a big deal. [2]

- This is assuming the most simple/naive implementation, where we
simply duplicate every traceback entry onto every exception. If it
turns out to be unacceptable, there are lots of tricks we could use to
speed it up, e.g. by updating __traceback__ lazily or letting
traceback entry objects be shared between tracebacks.

[2] Estimated based on 'sys.getsizeof(tb_object)' and '%timeit
types.TracebackType(None, frame, 0, 0)'.

> 2. Display code will need to normalize the traceback, which is much more complicated than denormalizing because you need to discover the shared parts.
>
> Am I missing something?

I think re-normalizing is very cheap and simple? It's just a common
prefix problem. Given a tree-in-progress and a traceback, walk down
each of them in parallel until you run out of matching entries, then
insert a branch node and append the rest of the traceback.

>> >
>> > It sounds like you want some way to enrich exceptions. This should be optional (we wouldn't want EG to require additional metadata for exceptions) so yeah, I agree it should sit on the leaf exception and not on the group. In that sense it's orthogonal to this PEP.
>>
>> Well, the extra metadata would specifically be at "join" points in the
>> traceback, which are a thing that this PEP is creating :-). And it's
>> useful for every user of EGs, since by definition, an EG is
>> multiplexing exceptions from multiple sources, so it's nice to tell
>> the user which sources those are.
>>
>> That said, you're right, if we want to handle this by defining a new
>> kind of traceback entry that code like Trio/asyncio/hypothesis can
>> manually attach to exceptions, then that could be written as a
>> separate-but-complementary PEP. In my original design, instead of
>> defining a new kind of traceback entry, I was storing this on the EG
>> itself, so that's why I was thinking about it needing to be part of
>> this PEP.
>
>
> You can also create an ExceptionGroup subclass with whatever extra data you want to include.

Unfortunately, that doesn't work either, because this is data that
should be included in traceback displays, similar to __cause__ and
__context__...

-n

--
Nathaniel J. Smith -- https://vorpus.org
_______________________________________________
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-leave@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/TMCA44SYKOJ2FTQ4LUL466LONAZJW4AD/
Code of Conduct: http://python.org/psf/codeofconduct/
Re: PEP 654: Exception Groups and except* [REPOST] [ In reply to ]
On Wed, Apr 21, 2021 at 11:22 PM Nathaniel Smith <njs@pobox.com> wrote:

>
>
> Saying that you have to make a new API every time you start using
> concurrency inside a function is extremely restrictive.


You don't need a new API when you start using concurrency inside a
function. You need a new API when you start raising exception groups from a
function because that changes the function's external behaviour.

-1 from me on the denormalized traceback idea. I've said what I had to say
about it in my previous email.

Cheers
Irit
Re: PEP 654: Exception Groups and except* [REPOST] [ In reply to ]
On Wed, Apr 21, 2021 at 3:26 PM Nathaniel Smith <njs@pobox.com> wrote:

> On Tue, Apr 20, 2021 at 3:15 AM Irit Katriel <iritkatriel@googlemail.com>
> wrote:
> > On Tue, Apr 20, 2021 at 2:48 AM Nathaniel Smith <njs@pobox.com> wrote:
> >>
> >>
> >> The problem is that most of the time, even if you're using concurrency
> >> internally so multiple things *could* go wrong at once, only one thing
> >> actually *does* go wrong. So it's unfortunate if some existing code is
> >> prepared for a specific exception that it knows can be raised, that
> >> exact exception is raised... and the existing code fails to catch it
> >> because now it's wrapped in an EG.
> >
> > Yes, this was discussed at length on this list. Raising an exception
> group is an API-breaking change. If a function starts raising exception
> groups its callers need to be prepared for that. Realistically we think
> exception groups will be raised by new APIs. We tried and were unable to
> define exception group semantics for except that would be reasonable and
> backwards compatible. That's why we added except*.
>
> Sure. This was in my list of reasons why the backwards compatibility
> tradeoffs are forcing us into awkward compromises. I only elaborated
> on it b/c in your last email you said you didn't understand why this
> was a problem :-). And except* is definitely useful. But I think there
> are options for 'except' that haven't been considered fully.
>

Do you have any suggestions, or are you just telling us to think harder?
Because we've already thought as hard as we could and within all the
constraints (backwards compatibility and otherwise) we just couldn't think
of a better one.


> Saying that you have to make a new API every time you start using
> concurrency inside a function is extremely restrictive. The whole
> point of structured concurrency (and structured programming in
> general) is that function callers don't need to know about the control
> flow decisions inside a function. So right now, the EG proposal is
> like saying that if you every have a function that doesn't contain a
> 'for' loop, and then you want to add a 'for' loop, then you have to
> define a whole new API instead of modifying the existing one.
>

That analogy doesn't fly. If "what exceptions can this raise" is not part
of the function spec (so users who catch exceptions are already flying
blind or guessing based on what they've seen happen in the past), then you
can certainly switch to using EGs, and this *is* like adding a for-loop.
But if the exceptions that a function raises *are* part of its spec, and
users are catching them, then switching to concurrency (without catching
and converting EGs in the function) *is* an API change, akin to taking a
function that returns an int and changing it to (sometimes?) returning a
list of ints.


> I absolutely get why the proposal looks like that. I'm just making the
> point that we should make sure we've exhausted all our options before
> settling for that as a compromise.
>

But if you're not able to make constructive proposals I have to conclude
that we *have* exhausted our options, and we should move on.


> >> > It is easy enough to write a denormalize() function in traceback.py
> that constructs this from the current EG structure, if you need it (use the
> leaf_generator from the PEP). I'm not sure I see why we should trouble the
> interpreter with this.
> >>
> >> In the current design, once an exception is wrapped in an EG, then it
> >> can never be unwrapped, because its traceback information is spread
> >> across the individual exception + the EG tree around it. This is
> >> confusing to users ("how do I check errno?"), and makes the design
> >> more complicated (the need for topology-preserving .split(), the
> >> inability to define a sensible EG.__iter__, ...). The advantage of
> >> making the denormalized form the native form is that now the leaf
> >> exceptions would be self-contained objects like they are now, so you
> >> don't need EG nesting at all, and users can write intuitive code like:
> >>
> >> except OSError as *excs:
> >> remainder = [exc for exc in excs if exc.errno != ...]
> >> if remainder:
> >> raise ExceptionGroup(remainder)
> >
> >
> > We have this precise example in the PEP:
> > match, rest = excs.split(lambda e: e.errno != ...)
> >
> > You use split() instead of iteration for that. split() preserves all
> __context__, __cause__ and __traceback__ information, on all leaf and
> non-leaf exceptions.
>
> Well, yeah, I know, I'm the one who invented split() :-). My point was
> to compare these two options: in the flat EG example, most Python
> users could write and read that code without knowing anything except
> "there can be multiple exceptions now". It's all old, well-known
> constructs used in the obvious way.
>
> For the .split() version, you have to write a lambda (which is allowed
> to access parts of the exception object, but not all of it!), and use
> this idiosyncratic method that only makes sense if you know about EG
> tree structures. That's a lot more stuff that users have to understand
> and hold in their head.
>

But it *is* a tree structure. The interior nodes of the tree have a
meaning: they represent join points where exceptions were grouped together,
and the selection process needs to traverse the tree.


> Or here's another example. Earlier you suggested:
>
> > If you are using concurrency internally and don't want to raise EGs
> externally, then surely you will catch EGs, select one of the exceptions to
> raise and throw away all the others
>
> But with nested EGs, this is difficult, because you *can't* just pull
> out one of the exceptions to raise. The leaf exceptions aren't
> standalone objects; you need some obscure traceback manipulation to do
> this. I guarantee that users will get this wrong, even if we provide
> the tools, because the explanation about when and why you need the
> tools is complicated and people won't internalize it.
>
> With flat EGs, this is trivial: it's just `raise
> ExceptionGroup[the_selected_exception_index]`.
>

But it's generally wrong to raise just one of the exceptions anyway, so you
want to make it easier to get a wrong result?

I'd rather make this an option for asyncio.gather() or its TaskGroup
equivalent, then the user doesn't have to do anything. gather() already has
as its default behavior to raise the first exception it encounters, which
seems good enough.


> >> > For display purposes, it is probably nicer to look at a normalized
> traceback where common parts are not repeated.
> >>
> >> Yeah, I agree; display code would want to re-normalize before
> >> printing. But now it's only the display code that needs to care about
> >> figuring out shared parts of the traceback, rather than something that
> >>
> >> has to be maintained as an invariant everywhere.
> >
> > If you *do* want iteration, we show in the PEP how to write a
> leaf_generator() that gives you the leaf exceptions with their tracebacks
> (as a list of chunks). It is easy to copy the chunks into a single flat
> traceback. We didn't propose to add it to traceback.py yet because the use
> case is unclear but if people need it we're talking about 10-15 lines in
> traceback.py.
> >
> > So for your suggestion:
> >
> > Pros:
> > 1. Those who want a denormalized traceback (if they exist) won't need to
> call traceback.denormalize().
> >
> > Cons:
> > 1. A significant change in the interpreter that will make it less
> efficient (both time and space).
>
> In terms of interpreter complexity, the straightforward implementation
> would be pretty simple. Whenever the exception handling code goes to
> write back a traceback to an exception's __traceback__ attribute,
> check if the exception is an ExceptionGroup; if so, then loop over
> each exception in the group and push a clone of the traceback onto
> that exception's __traceback__. It takes code of course, everything
> does, but I'd be surprised if it was more than, say, 20 lines of C. I
> think that's a pretty good tradeoff to get a simpler user-level
> language semantics. Python is supposed to fit in your head :-).
>
> Time/space efficiency is interesting, because yeah, with the nested EG
> approach then common parts of the traceback are stored once, while the
> flat EG approach needs to store a separate copy for each exception.
> But:
>
> - The extra cost is 1 traceback object per common stack frame, per
> exception. So the worst case is if you have, like, a 1000 exceptions
> in a single group, and then that group travels together for 1000 stack
> frames. But... is that realistic? Very few programs recurse down 1000
> levels and *then* spawn a bunch of tasks. And very few programs have
> 1000 tasks that all fail simultaneously with different exceptions. And
> even if that did happen, 1000*1000 = 1 million traceback objects take
> ~56 megabytes of RAM and ~90 ms to instantiate on my laptop, which is
> not a big deal. [2]
>
> - This is assuming the most simple/naive implementation, where we
> simply duplicate every traceback entry onto every exception. If it
> turns out to be unacceptable, there are lots of tricks we could use to
> speed it up, e.g. by updating __traceback__ lazily or letting
> traceback entry objects be shared between tracebacks.
>
> [2] Estimated based on 'sys.getsizeof(tb_object)' and '%timeit
> types.TracebackType(None, frame, 0, 0)'.
>

The real cost here is that we would need a new "TracebackGroup" concept,
since the internal data structures and APIs keep the traceback chain and
the exception object separated until the exception is caught. In our early
design stages we actually explored this and the complexity of the data
structures was painful. We eventually realized that we didn't need this
concept at all, and the result is much clearer, despite what you seem to
think.


> > 2. Display code will need to normalize the traceback, which is much more
> complicated than denormalizing because you need to discover the shared
> parts.
> >
> > Am I missing something?
>
> I think re-normalizing is very cheap and simple? It's just a common
> prefix problem. Given a tree-in-progress and a traceback, walk down
> each of them in parallel until you run out of matching entries, then
> insert a branch node and append the rest of the traceback.
>
> >> >
> >> > It sounds like you want some way to enrich exceptions. This should be
> optional (we wouldn't want EG to require additional metadata for
> exceptions) so yeah, I agree it should sit on the leaf exception and not on
> the group. In that sense it's orthogonal to this PEP.
> >>
> >> Well, the extra metadata would specifically be at "join" points in the
> >> traceback, which are a thing that this PEP is creating :-). And it's
> >> useful for every user of EGs, since by definition, an EG is
> >> multiplexing exceptions from multiple sources, so it's nice to tell
> >> the user which sources those are.
> >>
> >> That said, you're right, if we want to handle this by defining a new
> >> kind of traceback entry that code like Trio/asyncio/hypothesis can
> >> manually attach to exceptions, then that could be written as a
> >> separate-but-complementary PEP. In my original design, instead of
> >> defining a new kind of traceback entry, I was storing this on the EG
> >> itself, so that's why I was thinking about it needing to be part of
> >> this PEP.
> >
> >
> > You can also create an ExceptionGroup subclass with whatever extra data
> you want to include.
>
> Unfortunately, that doesn't work either, because this is data that
> should be included in traceback displays, similar to __cause__ and
> __context__...
>

Fortunately in the flat design there wouldn't be a need for such metadata
anyway...

--
--Guido van Rossum (python.org/~guido)
*Pronouns: he/him **(why is my pronoun here?)*
<http://feministing.com/2015/02/03/how-using-they-as-a-singular-pronoun-can-change-the-world/>
Re: PEP 654: Exception Groups and except* [REPOST] [ In reply to ]
On Wed, Apr 21, 2021 at 4:50 PM Guido van Rossum <guido@python.org> wrote:
> On Wed, Apr 21, 2021 at 3:26 PM Nathaniel Smith <njs@pobox.com> wrote:
>> Sure. This was in my list of reasons why the backwards compatibility
>> tradeoffs are forcing us into awkward compromises. I only elaborated
>> on it b/c in your last email you said you didn't understand why this
>> was a problem :-). And except* is definitely useful. But I think there
>> are options for 'except' that haven't been considered fully.
>
> Do you have any suggestions, or are you just telling us to think harder? Because we've already thought as hard as we could and within all the constraints (backwards compatibility and otherwise) we just couldn't think of a better one.

The main possibility that I don't think we've examined fully is to
make 'except' blocks fire multiple times when there are multiple
exceptions. We ruled it out early b/c it's incompatible with nested
EGs, but if flat EGs are better anyway, then the balance shifts around
and it might land somewhere different. it's a tricky discussion
though, b/c both the current proposal and the alternative have very
complex implications and downsides. So we probably shouldn't get too
distracted by that until after the flat vs nested discussion has
settled down more.

I'm not trying to filibuster here -- I really want some form of EGs to
land. I think python has the potential to be the most elegant and
accessible language around for writing concurrent programs, and EGs
are a key part of that. I don't want to fight about anything; I just
want to work together to make sure we have a full picture of our
options, so we can be confident we're making the best choice.

> The real cost here is that we would need a new "TracebackGroup" concept, since the internal data structures and APIs keep the traceback chain and the exception object separated until the exception is caught. In our early design stages we actually explored this and the complexity of the data structures was painful. We eventually realized that we didn't need this concept at all, and the result is much clearer, despite what you seem to think.

I'm not talking about TracebackGroups (at least, I think I'm not?). I
think it can be done with exactly our current data structures, nothing
new.

- When an EG is raised, build the traceback for just that EG while
it's unwinding. This means if any C code peeks at exc_info while it's
in flight, it'll only see the current branch of the traceback tree,
but that seems fine.
- When the exception is caught and we go to write back the traceback
to its __traceback__ attribute, instead "peek through" the EG and
append the built-up traceback entries onto each of the constituent
exceptions.

You could get cleverer for efficiency, but that basic concept seems
pretty simple and viable to me. What am I missing?

-n

--
Nathaniel J. Smith -- https://vorpus.org
_______________________________________________
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-leave@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/VOBOWZGW44GNMW6IUZU6P5OO2A5YKB53/
Code of Conduct: http://python.org/psf/codeofconduct/
Re: PEP 654: Exception Groups and except* [REPOST] [ In reply to ]
On Fri, Apr 23, 2021 at 6:25 PM Nathaniel Smith <njs@pobox.com> wrote:
>
> On Wed, Apr 21, 2021 at 4:50 PM Guido van Rossum <guido@python.org> wrote:
> > On Wed, Apr 21, 2021 at 3:26 PM Nathaniel Smith <njs@pobox.com> wrote:
> >> Sure. This was in my list of reasons why the backwards compatibility
> >> tradeoffs are forcing us into awkward compromises. I only elaborated
> >> on it b/c in your last email you said you didn't understand why this
> >> was a problem :-). And except* is definitely useful. But I think there
> >> are options for 'except' that haven't been considered fully.
> >
> > Do you have any suggestions, or are you just telling us to think harder? Because we've already thought as hard as we could and within all the constraints (backwards compatibility and otherwise) we just couldn't think of a better one.
>
> The main possibility that I don't think we've examined fully is to
> make 'except' blocks fire multiple times when there are multiple
> exceptions.

Vanilla except blocks? Not sure if I'm misunderstanding, but that
could cause some problems. Consider:

trn.begin()
try:
...
except BaseException:
trn.rollback()
raise
else:
trn.commit()

What happens if multiple exceptions get raised? Will the transaction
be rolled back more than once? What gets reraised?

If I'm completely misunderstanding you here, my apologies.

ChrisA
_______________________________________________
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-leave@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/SGFE3XG4COLBC747U7IBVWRKWAULY43C/
Code of Conduct: http://python.org/psf/codeofconduct/
Re: PEP 654: Exception Groups and except* [REPOST] [ In reply to ]
On Fri, Apr 23, 2021 at 9:22 AM Nathaniel Smith <njs@pobox.com> wrote:

> On Wed, Apr 21, 2021 at 4:50 PM Guido van Rossum <guido@python.org> wrote:
> > On Wed, Apr 21, 2021 at 3:26 PM Nathaniel Smith <njs@pobox.com> wrote:
> >> Sure. This was in my list of reasons why the backwards compatibility
> >> tradeoffs are forcing us into awkward compromises. I only elaborated
> >> on it b/c in your last email you said you didn't understand why this
> >> was a problem :-). And except* is definitely useful. But I think there
> >> are options for 'except' that haven't been considered fully.
> >
> > Do you have any suggestions, or are you just telling us to think harder?
> Because we've already thought as hard as we could and within all the
> constraints (backwards compatibility and otherwise) we just couldn't think
> of a better one.
>
> The main possibility that I don't think we've examined fully is to
> make 'except' blocks fire multiple times when there are multiple
> exceptions. We ruled it out early b/c it's incompatible with nested
> EGs, but if flat EGs are better anyway, then the balance shifts around
> and it might land somewhere different. it's a tricky discussion
> though, b/c both the current proposal and the alternative have very
> complex implications and downsides. So we probably shouldn't get too
> distracted by that until after the flat vs nested discussion has
> settled down more.
>
> I'm not trying to filibuster here -- I really want some form of EGs to
> land.


I'm very glad to hear that. It's been hard to know where you stand, because
you didn't even decline our invitation in October to work together on this,
and several later invitations to look at the implementation and try it with
Trio -- you just didn't reply. The only responses I saw were public, on
this list, taking us right back to the drawing board (for example - the
suggestion you mention in the previous paragraph as
not-sufficiently-explored, actually appears in the rejected ideas section
of the PEP, and the reason for rejection is not that it's incompatible with
nesting). So thank you for clarifying that you are, ultimately, supportive
of our efforts.

We do realize that we didn't adequately credit you for the contributions of
your 2018 work to this PEP, and have now added an acknowledgements section
for that purpose. Apologies for that.

I'm confused about the flattening suggestion - above you talk about "flat
EG", but below about tracebacks. It's not clear to me whether you want EG
to be flat (ie no nesting of EGs) or just the traceback to be flat (but you
can still have a nested EG). I also don't know what problem you are trying
to solve with this.

Fortunately we're not at the whiteboard stage anymore, we have a fully
working implementation and you can use it to demonstrate any problems you
see with what we did.

Irit



> I think python has the potential to be the most elegant and
> accessible language around for writing concurrent programs, and EGs
> are a key part of that. I don't want to fight about anything; I just
> want to work together to make sure we have a full picture of our
> options, so we can be confident we're making the best choice.
>
> > The real cost here is that we would need a new "TracebackGroup" concept,
> since the internal data structures and APIs keep the traceback chain and
> the exception object separated until the exception is caught. In our early
> design stages we actually explored this and the complexity of the data
> structures was painful. We eventually realized that we didn't need this
> concept at all, and the result is much clearer, despite what you seem to
> think.
>
> I'm not talking about TracebackGroups (at least, I think I'm not?). I
> think it can be done with exactly our current data structures, nothing
> new.
>
> - When an EG is raised, build the traceback for just that EG while
> it's unwinding. This means if any C code peeks at exc_info while it's
> in flight, it'll only see the current branch of the traceback tree,
> but that seems fine.
> - When the exception is caught and we go to write back the traceback
> to its __traceback__ attribute, instead "peek through" the EG and
> append the built-up traceback entries onto each of the constituent
> exceptions.
>
> You could get cleverer for efficiency, but that basic concept seems
> pretty simple and viable to me. What am I missing?
>
> -n
>
> --
> Nathaniel J. Smith -- https://vorpus.org
>
Re: PEP 654: Exception Groups and except* [REPOST] [ In reply to ]
On Fri, Apr 23, 2021 at 2:45 AM Chris Angelico <rosuav@gmail.com> wrote:
>
> On Fri, Apr 23, 2021 at 6:25 PM Nathaniel Smith <njs@pobox.com> wrote:
> > The main possibility that I don't think we've examined fully is to
> > make 'except' blocks fire multiple times when there are multiple
> > exceptions.
>
> Vanilla except blocks? Not sure if I'm misunderstanding, but that
> could cause some problems. Consider:
>
> trn.begin()
> try:
> ...
> except BaseException:
> trn.rollback()
> raise
> else:
> trn.commit()
>
> What happens if multiple exceptions get raised? Will the transaction
> be rolled back more than once? What gets reraised?
>
> If I'm completely misunderstanding you here, my apologies.

Yeah, you've understood correctly, and you see why I wrote "both the
current proposal and the alternative have very complex implications
and downsides" :-)

A lot depends on the details, too. One possible design is "in general,
vanilla except blocks will trigger multiple times, but as a special
case, except: and except BaseException: will only fire once, receiving
the whole ExceptionGroup as a single exception". (Rationale for making
a special case: if you're saying "catch anything, I don't care what",
then you obviously didn't intend to do anything in particular with
those exceptions, plus this is the only case where the types work.) In
that design, your example becomes correct again.

Some other interesting cases:

# Filters out all OSError's that match the condition, while letting
all other OSError's continue
try:
...
except OSError as exc:
if exc.errno != errno.EWHATEVER:
raise

# If this gets an ExceptionGroup([ValueError, KeyboardInterrupt]),
then it logs the ValueError and
# then exits with the selected error code
try:
...
except Exception as exc:
logger.log_exception(exc)
except KeyboardInterrupt:
sys.exit(17)

OTOH you can still come up with cases that aren't handled correctly.
If you have old 'except' code mixed with new code raising
ExceptionGroup, then there's no way to make that do the right thing in
*all* cases. By definition, when the 'except' code was written, the
author wasn't thinking about ExceptionGroups! So the question is which
combination of semantics ends up doing the least harm on average
across all sorts of different real-world cases...

-n

--
Nathaniel J. Smith -- https://vorpus.org
_______________________________________________
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-leave@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/3V6YEKHOD5MYLSCPVDB7IYGX4EOO4KWB/
Code of Conduct: http://python.org/psf/codeofconduct/
Re: PEP 654: Exception Groups and except* [REPOST] [ In reply to ]
On Mon, Apr 26, 2021 at 2:27 PM Nathaniel Smith <njs@pobox.com> wrote:
> Yeah, you've understood correctly, and you see why I wrote "both the
> current proposal and the alternative have very complex implications
> and downsides" :-)
>
> [chomp lots of very helpful summarizing]

Gotcha, thanks!

ChrisA
_______________________________________________
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-leave@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/LCQLKVQMBLHFX74WQD3Z5BIMYT5ARQFP/
Code of Conduct: http://python.org/psf/codeofconduct/
Re: PEP 654: Exception Groups and except* [REPOST] [ In reply to ]
On Fri, Apr 23, 2021 at 4:08 AM Irit Katriel <iritkatriel@googlemail.com> wrote:
>
> On Fri, Apr 23, 2021 at 9:22 AM Nathaniel Smith <njs@pobox.com> wrote:
>> I'm not trying to filibuster here -- I really want some form of EGs to
>> land.
>
> I'm very glad to hear that. It's been hard to know where you stand, because you didn't even decline our invitation in October to work together on this, and several later invitations to look at the implementation and try it with Trio -- you just didn't reply. The only responses I saw were public, on this list, taking us right back to the drawing board (for example - the suggestion you mention in the previous paragraph as not-sufficiently-explored, actually appears in the rejected ideas section of the PEP, and the reason for rejection is not that it's incompatible with nesting). So thank you for clarifying that you are, ultimately, supportive of our efforts.

Yes, I apologize again for the radio silence there -- I had real world
stuff that left me with no cope for open-source. I, uh, was feeling
guilty about not getting back to you the whole time, if that helps?
Probably that doesn't help.

My memory is that in our initial discussions, I suggested having each
'except Blah as exc' clause be executed once, receiving an
ExceptionGroup containing all the Blah exceptions. Guido pointed out
that this broke typing -- 'exc' would not longer have type 'Blah' --
and I was like... okay yeah that's a fatal flaw, never mind. And I
never seriously raised the 'execute a single clause multiple times'
option, because of the issue where in the nested design, taking
individual exceptions out of an ExceptionGroup breaks tracebacks.

Looking at the relevant section of the PEP again [1], it notes the
same fatal flaw with my first suggestion, and then says that the
multiple-except-executions option should be rejected because users
have written code like 'except SomeError: ...' with the expectation
that the 'except' clause would run exactly once. That's definitely
true, and it's a downside of the multiple-except-executions approach,
but I don't think it's convincing enough to rule this out on its own.
The problem is, *all* our options for how 'except' should interact
with ExceptionGroups will somehow break previous expectations.

Concretely: imagine you have a pre-existing 'except SomeError', and
some new code inside the 'try' block raises some number of
'SomeError's wrapped in an ExceptionGroup. There are three options:

- Execute the 'except' block multiple times. This breaks the
expectation that it should be executed at most once.
- Execute the 'except' block exactly once. But if there are multiple
SomeError's, this require they be grouped and delivered as a single
exception, which breaks typing.
- Execute the 'except' block zero times. This is what the current PEP
chooses, and breaks the expectation that 'except SomeError' should
catch 'SomeError'.

So we have to pick our poison.

[1] https://www.python.org/dev/peps/pep-0654/#extend-except-to-handle-exception-groups

> We do realize that we didn't adequately credit you for the contributions of your 2018 work to this PEP, and have now added an acknowledgements section for that purpose. Apologies for that.

Oh, that didn't bother me at all, but thanks :-). And I'm sorry if I
was denying you credit for things that you had actually independently
re-invented.

> I'm confused about the flattening suggestion - above you talk about "flat EG", but below about tracebacks. It's not clear to me whether you want EG to be flat (ie no nesting of EGs) or just the traceback to be flat (but you can still have a nested EG).

Hmm, I was thinking about making both of them flat, so no nested EGs.
In all my designs, the only reason I ever had nesting was because I
couldn't figure out any other way to make the tracebacks work. Do you
have some other motivation for wanting nesting? If so that would be
interesting, because it might point to why we're talking past each
other and help us understand the problem better...

> I also don't know what problem you are trying to solve with this.

I'm not saying that there's some fatal problem with the current PEP.
(In my first message I explicitly said that it would be an improvement
over the status quo :-).) But I think that nesting will be really
counterintuitive/confusing for users in some ways. And concurrency
APIs will be offputting if they force you to use a different special
form of 'except' all the time. Basically the 'flat' version might be a
lot more ergonomic, and that's important for a language like Python.

-n

--
Nathaniel J. Smith -- https://vorpus.org
_______________________________________________
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-leave@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/D6G7Z4QVC5H6UMHOFR33FMVCRV4HL6SM/
Code of Conduct: http://python.org/psf/codeofconduct/
Re: PEP 654: Exception Groups and except* [REPOST] [ In reply to ]
On Wed, Apr 28, 2021 at 8:53 PM Nathaniel Smith <njs@pobox.com> wrote:
> Looking at the relevant section of the PEP again [1], it notes the
> same fatal flaw with my first suggestion, and then says that the
> multiple-except-executions option should be rejected because users
> have written code like 'except SomeError: ...' with the expectation
> that the 'except' clause would run exactly once. That's definitely
> true, and it's a downside of the multiple-except-executions approach,
> but I don't think it's convincing enough to rule this out on its own.
> The problem is, *all* our options for how 'except' should interact
> with ExceptionGroups will somehow break previous expectations.

Well, this is where we respectfully disagree. The case of changing how
the regular try..except works in a backwards incompatible way is a
very convincing blocker to us.

>
> Concretely: imagine you have a pre-existing 'except SomeError', and
> some new code inside the 'try' block raises some number of
> 'SomeError's wrapped in an ExceptionGroup. There are three options:
>
> - Execute the 'except' block multiple times. This breaks the
> expectation that it should be executed at most once.
> - Execute the 'except' block exactly once. But if there are multiple
> SomeError's, this require they be grouped and delivered as a single
> exception, which breaks typing.
> - Execute the 'except' block zero times. This is what the current PEP
> chooses, and breaks the expectation that 'except SomeError' should
> catch 'SomeError'.
>
> So we have to pick our poison.

We did. The PEP talks at length as to why this isn't a problem.

> > I'm confused about the flattening suggestion - above you talk about "flat EG", but below about tracebacks. It's not clear to me whether you want EG to be flat (ie no nesting of EGs) or just the traceback to be flat (but you can still have a nested EG).
>
> Hmm, I was thinking about making both of them flat, so no nested EGs.
> In all my designs, the only reason I ever had nesting was because I
> couldn't figure out any other way to make the tracebacks work. Do you
> have some other motivation for wanting nesting? If so that would be
> interesting, because it might point to why we're talking past each
> other and help us understand the problem better...
>
> > I also don't know what problem you are trying to solve with this.
>
> I'm not saying that there's some fatal problem with the current PEP.
> (In my first message I explicitly said that it would be an improvement
> over the status quo :-).) But I think that nesting will be really
> counterintuitive/confusing for users in some ways. And concurrency
> APIs will be offputting if they force you to use a different special
> form of 'except' all the time. Basically the 'flat' version might be a
> lot more ergonomic, and that's important for a language like Python.

You keep saying that your idea of flat EGs is "ergonomic" and
"important for a language like Python". The problem is that after all
these emails I still have absolutely no idea about:

- what exactly are you trying to propose?
- what specific problem do you want to address? (that the current PEP,
in your opinion, does not address)
- why do you think that what you propose would be more ergonomic or simple?
- what "flat EGs" or "flat tracebacks" even mean; I can't figure out
not only the high level API, I don't even understand what you're
talking about at the data structures level.

Nathaniel, at this point it's clear that this thread somehow does not
help us understand what you want. Could you please just write your own
PEP clearly outlining your proposal, its upsides and downsides?
Without a PEP from you this thread is just a distraction.


Yury
_______________________________________________
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-leave@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/AR737RYK2KCTATCBW3QSDK5SSF7CH75M/
Code of Conduct: http://python.org/psf/codeofconduct/
Re: PEP 654: Exception Groups and except* [REPOST] [ In reply to ]
Nathaniel Smith (in a conversation with Irit) wrote:

> ... suggested having each
> 'except Blah as exc' clause be executed once, receiving an
> ExceptionGroup containing all the Blah exceptions. Guido pointed out
> that this broke typing -- 'exc' would not longer have type 'Blah' --
> and I was like... okay yeah that's a fatal flaw, never mind.

The fact that "Blah as exc" could return either a single Blah or a collection of Blahs would be annoying, but not fatal -- that seems well within the level of ambiguity typing can deal with.

> never seriously raised the 'execute a single clause multiple times'
> option, because of the issue where in the nested design, taking
> individual exceptions out of an ExceptionGroup breaks tracebacks.

This also seems like something that isn't hard to work around. The obvious way would involve shared (parts of) tracebacks, but even that isn't required.

> multiple-except-executions option should be rejected because users
> have written code like 'except SomeError: ...' with the expectation
> that the 'except' clause would run exactly once.

That is indeed a problem, because some error handlers would do bad things if rerun.

> The problem is, *all* our options for how 'except' should interact
> with ExceptionGroups will somehow break previous expectations.

agreed.

> Concretely: imagine you have a pre-existing 'except SomeError', and
> some new code inside the 'try' block raises some number of
> 'SomeError's wrapped in an ExceptionGroup. There are three options:
> - Execute the 'except' block multiple times. This breaks the
> expectation that it should be executed at most once.

Do we know how bad this really is? A second rollback or cancel or endConnection won't do what is expected, but they normally won't do much harm either. A second logError does the right thing.

> - Execute the 'except' block exactly once. But if there are multiple
> SomeError's, this require they be grouped and delivered as a single
> exception, which breaks typing.

I'm not worried about typing, but I am worried about silently dropping all but the "first" SomeError.

On the other hand, in practice were they all (including the "first") dropped before, in favor of some sort of superseding asyncCancelledError?

> - Execute the 'except' block zero times. This is what the current PEP
> chooses, and breaks the expectation that 'except SomeError' should
> catch 'SomeError'.

To me, this seems the worst of the three options, but ... maybe it is effectively the status quo often enough that it becomes the least bad?

> > I'm confused about the flattening suggestion - above you talk about "flat EG", but below about tracebacks. It's not clear to me whether you want EG to be flat (ie no nesting of EGs) or just the traceback to be flat (but you can still have a nested EG).
> > Hmm, I was thinking about making both of them flat, so no nested EGs.
> In all my designs, the only reason I ever had nesting was because I
> couldn't figure out any other way to make the tracebacks work. Do you
> have some other motivation for wanting nesting? If so that would be
> interesting, because it might point to why we're talking past each
> other and help us understand the problem better...

When I view this as strictly a typing problem, it matters which exceptions got joined at which junction; the shape of the tree is part of the type.

When I view this as a production support programmer, I really don't care about that ... I only care what triggered each SomeException so that I can account for all the bad data instead of just one piece. Having to dig through multiple layers of grouping to get to each original SomeException separately is just an annoyance that *will* cause juniors to sometimes miss part of the bad data. ("Open the door and check for problems" is a lot easier to learn and remember than a recursive "Open the door and check for problems or other doors, and then repeat on each door you find.")

-jJ
_______________________________________________
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-leave@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/VO5I7DMRYAB7EPUYSYEHHQFLGZAYWUB4/
Code of Conduct: http://python.org/psf/codeofconduct/
Re: PEP 654: Exception Groups and except* [REPOST] [ In reply to ]
On Thu, Apr 29, 2021 at 9:14 AM Yury Selivanov <yselivanov.ml@gmail.com> wrote:
> Nathaniel, at this point it's clear that this thread somehow does not
> help us understand what you want. Could you please just write your own
> PEP clearly outlining your proposal, its upsides and downsides?
> Without a PEP from you this thread is just a distraction.

If that's the best way to move forward, then ok. My main thing is just
that I don't want to make this some antagonistic me-vs-you thing.
After all, we all want the best design to be chosen, and none of us
know what that is yet, so there's no need for conflict :-).

Irit, Yury, would you be interested in co-authoring a PEP for the
"flat EG" approach? Basically trying to set down the best possible
version of each approach, so that we can put them next to each other?

-n

--
Nathaniel J. Smith -- https://vorpus.org
_______________________________________________
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-leave@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/LZBM5F6SAULEN7KEYNHBCLUPTB4JHBGO/
Code of Conduct: http://python.org/psf/codeofconduct/
Re: PEP 654: Exception Groups and except* [REPOST] [ In reply to ]
On Thu, May 6, 2021 at 2:17 AM Nathaniel Smith <njs@pobox.com> wrote:
>
> On Thu, Apr 29, 2021 at 9:14 AM Yury Selivanov <yselivanov.ml@gmail.com> wrote:
> > Nathaniel, at this point it's clear that this thread somehow does not
> > help us understand what you want. Could you please just write your own
> > PEP clearly outlining your proposal, its upsides and downsides?
> > Without a PEP from you this thread is just a distraction.
>
> If that's the best way to move forward, then ok. My main thing is just
> that I don't want to make this some antagonistic me-vs-you thing.
> After all, we all want the best design to be chosen, and none of us
> know what that is yet, so there's no need for conflict :-).
>
> Irit, Yury, would you be interested in co-authoring a PEP for the
> "flat EG" approach? Basically trying to set down the best possible
> version of each approach, so that we can put them next to each other?

Uh, probably this is obvious, but I mean "co-authoring with me". I'm
not suggesting you two go off to do it without me :-)

--
Nathaniel J. Smith -- https://vorpus.org
_______________________________________________
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-leave@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/5TFK5LEMDQ7GNYFKNS7QO7GGT33M6KAP/
Code of Conduct: http://python.org/psf/codeofconduct/
Re: PEP 654: Exception Groups and except* [REPOST] [ In reply to ]
Hi Nathaniel,

Philosophy of Science 101: nobody knows what TheBestSolution is and nobody
ever will.

That said, when we decided to submit PEP 654, that was a point in time when
we came to believe that we found the right solution. We made significant
changes later following feedback we agreed with, but in the case of "flat
EG" I see several disadvantages and don't quite understand what the win
is. I appreciate the invitation, but I don't want to work on that idea (nor
am I the right person to push it forward).

However, I think it's fantastic if you and others do pursue alternative
options (independently of my biases). You will either find something better
or validate PEP 654, and in either case the world wins. Feel free to adapt
our implementation if that helps. You have a year before we have our next
chance to commit to PEP 654.

Irit


On Thu, May 6, 2021 at 10:18 AM Nathaniel Smith <njs@pobox.com> wrote:

> On Thu, Apr 29, 2021 at 9:14 AM Yury Selivanov <yselivanov.ml@gmail.com>
> wrote:
> > Nathaniel, at this point it's clear that this thread somehow does not
> > help us understand what you want. Could you please just write your own
> > PEP clearly outlining your proposal, its upsides and downsides?
> > Without a PEP from you this thread is just a distraction.
>
> If that's the best way to move forward, then ok. My main thing is just
> that I don't want to make this some antagonistic me-vs-you thing.
> After all, we all want the best design to be chosen, and none of us
> know what that is yet, so there's no need for conflict :-).


> Irit, Yury, would you be interested in co-authoring a PEP for the
> "flat EG" approach? Basically trying to set down the best possible
> version of each approach, so that we can put them next to each other?
>
> -n
>
> --
> Nathaniel J. Smith -- https://vorpus.org
>

1 2  View All