Mailing List Archive: [issue45262] crash if asyncio is used before and after re-initialization if using python embedded in an application

[issue45262] crash if asyncio is used before and after re-initialization if using python embedded in an application

Sep 22, 2021, 1:32 AM

Post #1 of 11 (144 views)

New submission from Benjamin Schiller <benjamin.schiller@vector.com>:

We have embedded Python in our application and we deinitialize/initialize the interpreter at some point of time. If a simple script with a thread that sleeps with asyncio.sleep is loaded before and after the re-initialization, then we get the following assertion in the second run of the python module:

"Assertion failed: Py_IS_TYPE(rl, &PyRunningLoopHolder_Type), file D:\a\1\s\Modules_asynciomodule.c, line 261"

Example to reproduce this crash: https://github.com/benjamin-sch/asyncio_crash_in_second_run

----------
components: asyncio
messages: 402412
nosy: asvetlov, benjamin-sch, yselivanov
priority: normal
severity: normal
status: open
title: crash if asyncio is used before and after re-initialization if using python embedded in an application
type: crash
versions: Python 3.9

_______________________________________
Python tracker <report@bugs.python.org>
<https://bugs.python.org/issue45262>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com

[issue45262] crash if asyncio is used before and after re-initialization if using python embedded in an application [ In reply to ]

Sep 22, 2021, 12:57 PM

Post #2 of 11 (144 views)

Andrew Svetlov <andrew.svetlov@gmail.com> added the comment:

I guess the fix requires switching C Extension types from static to heap for _asyncio module.
It is possible for sure but requires a non-trivial amount of work.

We need a champion for the issue.

----------

_______________________________________
Python tracker <report@bugs.python.org>
<https://bugs.python.org/issue45262>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com

[issue45262] crash if asyncio is used before and after re-initialization if using python embedded in an application [ In reply to ]

Sep 29, 2021, 1:41 PM

Post #3 of 11 (142 views)

STINNER Victor <vstinner@python.org> added the comment:

> "Assertion failed: Py_IS_TYPE(rl, &PyRunningLoopHolder_Type), file D:\a\1\s\Modules_asynciomodule.c, line 261"

The _asyncio extension uses static types and doesn't implement the multiphase initialization API. It doesn't work well with your "deinitialize/initialize the interpreter" use case.

The _asyncio extension should use heap types and the multiphase init API.

----------
nosy: +vstinner

_______________________________________
Python tracker <report@bugs.python.org>
<https://bugs.python.org/issue45262>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com

[issue45262] crash if asyncio is used before and after re-initialization if using python embedded in an application [ In reply to ]

Oct 7, 2021, 6:28 AM

Post #4 of 11 (139 views)

Change by Matthias Reichl <hias@horus.com>:

----------
keywords: +patch
nosy: +HiassofT
nosy_count: 4.0 -> 5.0
pull_requests: +27124
stage: -> patch review
pull_request: https://github.com/python/cpython/pull/28796

_______________________________________
Python tracker <report@bugs.python.org>
<https://bugs.python.org/issue45262>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com

[issue45262] crash if asyncio is used before and after re-initialization if using python embedded in an application [ In reply to ]

Oct 7, 2021, 6:37 AM

Post #5 of 11 (139 views)

Matthias Reichl <hias@horus.com> added the comment:

We were hitting the same issue in kodi, which uses embedded sub-interpreters to run python addons, after one of the addons was switched to asyncio.

The immediate cause of the assertion failure is a use-after-free issue from the running loop holder cache:

When the running loop holder is deallocated (which happens eg on interpreter shutdown) cached_running_holder holds a dangling pointer.

A subsequent call to get_running_loop() may then pick that up and boom.

While probably not a full fix for this issue I think it would be good to fix the use-after-free problem - there could be other code paths that lead to this situation. I've created a github pull request for that

----------

_______________________________________
Python tracker <report@bugs.python.org>
<https://bugs.python.org/issue45262>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com

[issue45262] crash if asyncio is used before and after re-initialization if using python embedded in an application [ In reply to ]

Oct 7, 2021, 3:47 PM

Post #6 of 11 (139 views)

STINNER Victor <vstinner@python.org> added the comment:

New changeset 392a89835371baa0fc4bf79ae479abb80661f57d by Matthias Reichl in branch 'main':
bpo-45262, asyncio: Fix cache of the running loop holder (GH-28796)
https://github.com/python/cpython/commit/392a89835371baa0fc4bf79ae479abb80661f57d

----------

_______________________________________
Python tracker <report@bugs.python.org>
<https://bugs.python.org/issue45262>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com

[issue45262] crash if asyncio is used before and after re-initialization if using python embedded in an application [ In reply to ]

Oct 7, 2021, 3:47 PM

Post #7 of 11 (139 views)

Change by miss-islington <mariatta.wijaya+miss-islington@gmail.com>:

----------
nosy: +miss-islington
nosy_count: 5.0 -> 6.0
pull_requests: +27133
pull_request: https://github.com/python/cpython/pull/28815

_______________________________________
Python tracker <report@bugs.python.org>
<https://bugs.python.org/issue45262>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com

[issue45262] crash if asyncio is used before and after re-initialization if using python embedded in an application [ In reply to ]

Oct 7, 2021, 3:47 PM

Post #8 of 11 (139 views)

Change by miss-islington <mariatta.wijaya+miss-islington@gmail.com>:

----------
pull_requests: +27134
pull_request: https://github.com/python/cpython/pull/28816

_______________________________________
Python tracker <report@bugs.python.org>
<https://bugs.python.org/issue45262>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com

[issue45262] crash if asyncio is used before and after re-initialization if using python embedded in an application [ In reply to ]

Oct 7, 2021, 4:14 PM

Post #9 of 11 (139 views)

miss-islington <mariatta.wijaya+miss-islington@gmail.com> added the comment:

New changeset 87f0156a229e4cda92ad8e50645c5a71030caf7c by Miss Islington (bot) in branch '3.9':
bpo-45262, asyncio: Fix cache of the running loop holder (GH-28796)
https://github.com/python/cpython/commit/87f0156a229e4cda92ad8e50645c5a71030caf7c

----------

_______________________________________
Python tracker <report@bugs.python.org>
<https://bugs.python.org/issue45262>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com

[issue45262] crash if asyncio is used before and after re-initialization if using python embedded in an application [ In reply to ]

Oct 8, 2021, 1:55 AM

Post #10 of 11 (139 views)

STINNER Victor <vstinner@python.org> added the comment:

Is Matthias Reichl's fix enough to use asyncio in a subinterpreter?

----------

_______________________________________
Python tracker <report@bugs.python.org>
<https://bugs.python.org/issue45262>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com

[issue45262] crash if asyncio is used before and after re-initialization if using python embedded in an application [ In reply to ]

Oct 8, 2021, 1:55 AM

Post #11 of 11 (139 views)

STINNER Victor <vstinner@python.org> added the comment:

New changeset 6846d6712a0894f8e1a91716c11dd79f42864216 by Miss Islington (bot) in branch '3.10':
bpo-45262, asyncio: Fix cache of the running loop holder (GH-28796) (GH-28816)
https://github.com/python/cpython/commit/6846d6712a0894f8e1a91716c11dd79f42864216

----------

_______________________________________
Python tracker <report@bugs.python.org>
<https://bugs.python.org/issue45262>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com