OK, better late than never... here's a much-delayed review of the PEP.
Thank you Irit and Guido for carrying this forward while I've been AWOL!
It's fantastic to see my old design sketches turned into something like,
actually real.
== Overall feelings ==
Honestly, I have somewhat mixed feelings ExceptionGroups. I don't see any
way around adding ExceptionGroups in some form, because it's just a fact of
life that in a concurrent program, multiple things can go wrong at once,
and we want Python to be usable for writing concurrent programs. Right now
the state of the art is "exceptions in background threads/tasks get dropped
on the floor", and almost anything is better than that. The current PEP is
definitely better than that. But at the same time, there are a lot of
compromises needed to retrofit this onto Python's existing system, and the
current proposal feels like a bunch of awkward hacks with hacks on top.
That's largely my fault for not thinking of something better, and maybe
there is nothing better. But I still wish we could come up with something
more elegant, and I do see why this proposal has made people uncomfortable.
For example:
- I'm uncomfortable with how in some contexts we treat EG's as placeholders
for the contained exceptions, and other places we treat them like a single
first-class exceptions. (Witness all the feedback about "why not just catch
the ExceptionGroup and handle it by hand?", and imagine writing the docs
explaining all the situations where that is or isn't a good idea and the
pitfalls involved...) If we could somehow pick one and stick to it then I
think that would make it easier for users to grasp. (My gut feeling is that
making them pure containers is better, which to me is why it makes sense
for them to be @final and why I keep hoping we can figure out some better
way for plain 'except' and EGs to interact.)
- If a function wants to start using concurrency internally, then now *all*
its exceptions have to get wrapped in EGs and callers have to change *all*
their exception handling code to use except* or similar. You would think
this was an internal implementation detail that the caller shouldn't have
to care about, but instead it forces a major change on the function's
public API. And this is because regular 'except' can't do anything useful
with EGs.
- We have a special-case hack to keep 'except Exception' working, but it
has tricky edge cases (Exceptions can still sneak past if they're paired up
with a BaseException), and it really is specific to 'except Exception'; it
doesn't work for any other 'except SomeError' code. This smells funny.
Anyway, that's just abstract context to give an idea where I'm coming from.
Maybe we just have to accept these trade-offs, but if anyone has any ideas,
speak up...
== Most important comment ==
Flat ExceptionGroups: there were two basic design approaches we discussed
last year, which I'll call "flat" vs "nested". The current PEP uses the
nested design, where ExceptionGroups form a tree, and traceback information
is distributed in pieces over this tree. This is the source of a lot of the
complexity in the current PEP: for example, it's why EG's don't have one
obvious iteration semantics, and it's why once an exception is wrapped in
an EG, it can never be unwrapped again (because it would lose traceback
information).
The idea of the "flat" design is to instead store all the traceback info
directly on the leaf exceptions, so the EG itself can be just a pure
container holding a list of exceptions, that's it, with no nesting. The
downside is that it requires changes to the interpreter's code for updating
__traceback__ attributes, which is currently hard-coded to only update one
__traceback__ at a time.
For a third-party library like Trio, changing the interpreter is obviously
impossible, so we never considered it seriously. But in a PEP, changing the
interpreter is possible. And now I'm worried that we ruled out a better
solution early on for reasons that no longer apply. The more I think about
it, the more I suspect that flat EGs would end up being substantially
simpler all around? So I think we should at least think through what that
would look like (and Irit, I'd love your thoughts here now that you're the
expert on the CPython details!), and document an explicit decision one way
or another. (Maybe we should do a call or something to go over the details?
I'm trying to keep this email from ballooning out of control...)
== Smaller points ==
- In my original proposal, EGs didn't just hold a list of exceptions, but
also a list of "origins" for each exception. The idea being that if, say,
you attempt to connect to a host with an IPv4 address and an IPv6 address,
and they raised two different OSErrors that got bundled together into one
EG, then it would be nice to know which OSError came from which attempt. Or
in asyncio/trio it would be nice if tracebacks could show which task each
exception came from. It seems like this got dropped at some point?
On further consideration, I think this might be better handled as a
special kind of traceback entry that we can attach to each exception, that
just holds some arbitrary text that's inserted into the traceback at the
appropriate place? But either way, I think it would be good to be able to
attach this kind of information to tracebacks somehow.
- Recording pre-empted exceptions: This is another type of metadata that
would be useful to print along with the traceback. It's non-obvious and a
bit hard to explain, but multiple trio users have complained about this, so
I assume it will bite asyncio users too as soon as TaskGroups are added.
The situation is, you have a parent task P and two child tasks C1 and C2:
P
/ \
C1 C2
C1 terminates with an unhandled exception E1, so in order to continue
unwinding, the nursery/taskgroup in P cancels C2. But, C2 was itself in the
middle of unwinding another, different exception E2 (so e.g. the
cancellation arrived during a `finally` block). E2 gets replaced with a
`Cancelled` exception whose __context__=E2, and that exception unwinds out
of C2 and the nursery/taskgroup in P catches the `Cancelled` and discards
it, then re-raises E1 so it can continue unwinding.
The problem here is that E2 gets "lost" -- there's no record of it in the
final output. Basically E1 replaced it. And that can be bad: for example,
if the two children are interacting with each other, then E2 might be the
actual error that broke the program, and E1 is some exception complaining
that the connection to C2 was lost. If you have two exceptions that are
triggered from the same underlying event, it's a race which one survives.
This is conceptually similar to the way an exception in an 'except' block
used to cause exceptions to be lost, so we added __context__ to avoid that.
And just like for __context__, it would be nice if we could attach some
info to E1 recording that E2 had happened and then got preempted. But I
don't see how we can reuse __context__ itself for this, because it's a
somewhat different relationship: __context__ means that an exception
happened in the handler for another exception, while in this case you might
have multiple preempted exceptions, and they're associated with particular
points in the stack trace where the preemption occurred.
This is a complex issue and maybe we should call it out-of-scope for the
first version of ExceptionGroups. But I mention it because it's a second
place where adding some extra annotations to the traceback info would be
useful, and maybe we can keep it simple by adding some minimal hooks in the
core traceback machinery and let libraries like trio/asyncio handle the
complicated parts?
- There are a number of places where the Python VM itself catches
exceptions and has hard-coded handling for certain exception types. For
example:
- Unhandled exceptions that reach the top of the main thread generally
cause a traceback to be printed, but if the exception is SystemExit then
the interpreter instead exits silently with status exc.args[0].
- 'for' loops call iter.__next__, and catch StopIteration while allowing
other exceptions to escape.
- Generators catch StopIteration from their bodies and replace it with
RuntimeError (PEP 479)
With this PEP, it's now possible for the main thread to terminate with
ExceptionGroup(SystemExit), __next__ to raise
ExceptionGroup(StopIteration), a generator to raise
ExceptionGroup(StopIteration), either alone or mixed with other exceptions.
How should the VM handle these new cases? Should they be using except* or
except?
I don't think there's an obvious answer here, and possibly the answer is
just "don't do that". But I feel like the PEP should say what the language
semantics are in these cases, one way or another.
- traceback module API changes: The PEP notes that traceback.print_tb and
traceback.print_exception will be updated to handle ExceptionGroups. The
traceback module also has some special data structures for representing
"pre-processed" stack traces, via the traceback.StackSummary type. This is
used to capture tracebacks in a structured way but without holding onto the
full frame objects. Maybe this API should also be extended somehow so it
can also represent traceback trees?
On Sat, Mar 20, 2021 at 10:06 AM Irit Katriel via Python-Dev <
python-dev@python.org> wrote:
>
> We would like to present for feedback a new version of PEP 654, which
> incorporates the feedback we received in the discussions so far:
> https://www.python.org/dev/peps/pep-0654/
> The reference implementation has also been updated along with the PEP.
>
> The changes we made since the first post are:
>
> 1. Instead of ExceptionGroup(BaseException), we will have two new builtin
> types: BaseExceptionGroup(BaseException) and
> ExceptionGroup(BaseExceptionGroup, Exception).
> This is so that "except Exception" catches ExceptionGroups (but not
> BaseExceptionGroups). BaseExceptionGroup.__new__ inspects the wrapped
> exceptions, and if they are all Exception subclasses, it creates an
> ExceptionGroup instead of a BaseExceptionGroup.
>
> 2. The exception group classes are not final - they can be subclassed and
> split()/subgroup() work correctly if the subclass overrides the derive()
> instance method as described here:
> https://www.python.org/dev/peps/pep-0654/#subclassing-exception-groups
>
> 3. We had some good suggestions on formatting exception groups, which we
> have implemented as you can see in the output shown for the examples in the
> PEP.
>
> 4. We expanded the section on handling Exception Groups, to show how
> subgroup can be used (with side effects) to do something for each leaf
> exception, and how to iterate correctly when the tracebacks of leaf
> exceptions are needed:
> https://www.python.org/dev/peps/pep-0654/#handling-exception-groups
>
> 5. We expanded the sections on rationale and backwards compatibility to
> explain our premise and expectations regarding how exception groups will be
> used and how the transition to using them will be managed.
>
> 6. We added several items to the rejected ideas section.
>
> We did not receive any comments (or make any changes) to the proposed
> semantics of except*, hopefully this is because everyone thought they are
> sensible.
>
> Irit, Yury and Guido
>
> _______________________________________________
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-leave@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-dev@python.org/message/MQ2UCSQ2ZC4FIGT7KSVI6BJA4FCXSOCL/
> Code of Conduct: http://python.org/psf/codeofconduct/
>
--
Nathaniel J. Smith --
https://vorpus.org <
http://vorpus.org>