I, too, think I ran into basically the same situation as Paul: An outer
async generator calls an inner async generator. The outer doesn't fully
consume the inner. The inner uses an async context manager. The inner CM
isn't exited when expected and resources aren't released when
required/expected.
FWIW, the analysis of PEP 533 seems spot on -- for synchronous code, while
still possible, it's harder to trigger thanks to the GC tricks. For
asynchronous code, it easily (and subtly) results in resources not being
released when expected. Or perhaps never at all? IIUC, a long-lived server
might just accumulate unfinished async generators indefinitely, or at least
until some OS resource was exhausted.
In my particular case, I have thousands of network requests to make, each
of which is potentially large and/or slow to complete in its entirety, so
I'm taking extra care to break early to minimize resource usage to both
client and servers. I spent about an entire day trying to figure why the
synchronous version of my code Just Worked, while the asynchronous version
simply refused to close things when I was done with them. It was only after
I started looking at third party libraries and stumbled upon a reference to
PEP 533 and a detailed blog post from 5 years ago
<
https://async-generator.readthedocs.io/en/latest/reference.html#garbage-collection-hooks>
that
I had a clue what was going on. My point being, the feature PEP 533
describes would have made things Just Work in line with my expectations.
On Tue, Jan 5, 2021 at 8:35 AM Stestagg <stestagg@gmail.com> wrote:
> For what it's worth, this recent issue: https://bugs.python.org/issue42762
> is somewhat related to the non-async aspect of 533. (generator func getting
> closed during __del__ and causing odd side-effects at seemingly random
> points in the execution flow)
>
> The example code provided in that issue is contrived, and involves
> catching unconditional exceptions and recursing, but following the code
> flow in that situation feels unreasonably hard, and deciding if it's a
> python bug or not is even harder.
>
> I'm not sure how easy an implementation of 533 would be that doesn't break
> a lot of existing code, but if one could be made, it might help make
> generator lifecycles more predictable.
>
>
>
>
> On Tue, Jan 5, 2021 at 1:08 PM Senthil Kumaran <senthil@uthcode.com>
> wrote:
>
>> Hi Paul,
>>
>> > Per PEP 525, I can call aclose coroutine method to cleanup the
>> generator, but
>> > it requires the code iterating to be aware that that closing the
>> generator is
>> > necessary.
>>
>> How about treating this as a bug for the specific use case that you
>> mentioned,
>> rather than a complete addition of " PEP 533 -- Deterministic cleanup for
>> iterators". I am not certain why PEP 533 was deffered.
>>
>> I'd also suggest a documentation update if the details of aclose
>> requirement is
>> mentioned only in a PEP and not in the documentation.
>>
>> Thank you,
>> Senthil
>> _______________________________________________
>> Python-Dev mailing list -- python-dev@python.org
>> To unsubscribe send an email to python-dev-leave@python.org
>> https://mail.python.org/mailman3/lists/python-dev.python.org/
>> Message archived at
>> https://mail.python.org/archives/list/python-dev@python.org/message/EPKMYH7DH5G6YR3AQHZTQKHBQF46YXLC/
>> Code of Conduct: http://python.org/psf/codeofconduct/
>>
> _______________________________________________
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-leave@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-dev@python.org/message/VB64QJAYU2CCLOHRCBUIQOHMS2DIQQCC/
> Code of Conduct: http://python.org/psf/codeofconduct/
>