Mailing List Archive

1 2  View All
[issue43760] The DISPATCH() macro is not as efficient as it could be (move PyThreadState.use_tracing) [ In reply to ]
Change by STINNER Victor <vstinner@python.org>:


----------
pull_requests: +26926
pull_request: https://github.com/python/cpython/pull/28542

_______________________________________
Python tracker <report@bugs.python.org>
<https://bugs.python.org/issue43760>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com
[issue43760] The DISPATCH() macro is not as efficient as it could be (move PyThreadState.use_tracing) [ In reply to ]
STINNER Victor <vstinner@python.org> added the comment:

Analysis use use_tracing usage in 3rd part code.

I see two main ways to add C API functions covering these use cases:

* Provide high-level functions like "call a trace function" (disable tracing, call trace function, reenable tracing, increment/decrement tstate->tracing)
* Provide low-level functions just to control use_tracing: make PyThreadState structure opaque, but stil make the assumption that it is possible to disable temporarily tracing and profiling (in practice, it's implemented as use_tracing=0).



(*) greenlet

greenlet disables temporarily tracing in g_calltrace(), and then restore it, to call a "tracing" function:
---
tstate->tracing++;
TSTATE_USE_TRACING(tstate) = 0;
retval = PyObject_CallFunction(tracefunc, "O(OO)", event, origin, target);
tstate->tracing--;
TSTATE_USE_TRACING(tstate) =
(tstate->tracing <= 0 &&
((tstate->c_tracefunc != NULL) || (tstate->c_profilefunc != NULL)));
---

It also saves and then restores use_tracing value:
---
ts__g_switchstack_use_tracing = tstate->cframe->use_tracing;
(...)
tstate->cframe->use_tracing = ts__g_switchstack_use_tracing;
---

=> it can use PyThreadState_IsTracing(), PyThreadState_DisableTracing() and PyThreadState_ResetTracing().

These functions don't handle "tstate->tracing++;" and "tstate->tracing--;" which is also used by greenlet.

greenlet also saves and restores tstate->cframe:
https://github.com/python-greenlet/greenlet/blob/master/src/greenlet/greenlet.c


(*) dipy

Code generated by Cython.


(*) smartcols

Code generated by Cython.


(*) yappi

yappi is Python profiler.

yappi sets use_tracing to 1 when it sets its profile function: "ts->c_profilefunc = _yapp_callback;".

It sets use_tracing to 0 when it clears the profile function: "ts->c_profilefunc = NULL;". That's wrong, it ignores the trace function.

PyEval_SetProfile() cannot be used because yappi works on a PyThreadState (ts).

Code: https://github.com/sumerc/yappi/blob/master/yappi/_yappi.c

It can use PyThreadState_DisableTracing() and PyThreadState_ResetTracing(). Maybe a PyThreadState_SetProfile(tstate, func) function would fit better yappi's use case.


(*) Cython

Cython defines 2 compatibility functions:

* __Pyx_IsTracing(tstate, check_tracing, check_funcs): it can check c_profilefunc and c_tracefunc
* __Pyx_SetTracing(tstate, enable)

Code: https://github.com/cython/cython/blob/0.29.x/Cython/Utility/Profile.c

The code is quite complicated. In short, it checks if tracing and/or profiling is enabled. If it's enabled, it disables temporarily tracing (use_tracing=0) while calling trace and profile functions.

=> it requires PyThreadState_IsTracing(), PyThreadState_DisableTracing() and PyThreadState_ResetTracing().

----------

_______________________________________
Python tracker <report@bugs.python.org>
<https://bugs.python.org/issue43760>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com
[issue43760] The DISPATCH() macro is not as efficient as it could be (move PyThreadState.use_tracing) [ In reply to ]
Guido van Rossum <guido@python.org> added the comment:

Ah, I think the docs need to be clarified a bit. Here's what I was missing:

The key thing to know here is that there are *three* state variables; `c_tracefunc`, `c_profilefunc` (on the thread state), and `use_tracing` (on the C frame).

Normally `use_tracing` is initialized to false if both functions are NULL, and true otherwise (if at least one of the functions is set).

*Disabling* means setting `use_tracing` to false regardless. *Resetting* means setting `use_tracing` to the value computed above.

There's also a fourth variable, `tstate->tracing`, which indicates whether a tracing function is active (i.e., it has been called and hasn't exited yet). This can be incremented and decremented. But none of the proposed APIs affect it.

Would it be reasonable to just put these APIs in pythoncapi_compat, instead of in the stdlib? (It would be yet one more selling point for people to start using that. :-)

----------

_______________________________________
Python tracker <report@bugs.python.org>
<https://bugs.python.org/issue43760>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com
[issue43760] The DISPATCH() macro is not as efficient as it could be (move PyThreadState.use_tracing) [ In reply to ]
Change by Mark Shannon <mark@hotpy.org>:


----------
pull_requests: +27070
pull_request: https://github.com/python/cpython/pull/28723

_______________________________________
Python tracker <report@bugs.python.org>
<https://bugs.python.org/issue43760>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com
[issue43760] The DISPATCH() macro is not as efficient as it could be (move PyThreadState.use_tracing) [ In reply to ]
Pablo Galindo Salgado <pablogsal@gmail.com> added the comment:


New changeset 78184fa6b0e6129203673e425718e08f5edb6e2e by Pablo Galindo (Miss Islington (bot)) in branch '3.10':
bpo-43760: Document PyThreadState.use_tracing removal (GH-28527) (GH-28529)
https://github.com/python/cpython/commit/78184fa6b0e6129203673e425718e08f5edb6e2e


----------

_______________________________________
Python tracker <report@bugs.python.org>
<https://bugs.python.org/issue43760>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com
[issue43760] The DISPATCH() macro is not as efficient as it could be (move PyThreadState.use_tracing) [ In reply to ]
Mark Shannon <mark@hotpy.org> added the comment:


New changeset bd627eb7ed08a891dd1356756feb1ce2600358e4 by Mark Shannon in branch 'main':
bpo-43760: Check for tracing using 'bitwise or' instead of branch in dispatch. (GH-28723)
https://github.com/python/cpython/commit/bd627eb7ed08a891dd1356756feb1ce2600358e4


----------

_______________________________________
Python tracker <report@bugs.python.org>
<https://bugs.python.org/issue43760>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com
[issue43760] The DISPATCH() macro is not as efficient as it could be (move PyThreadState.use_tracing) [ In reply to ]
STINNER Victor <vstinner@python.org> added the comment:


New changeset 547d26aa08aa5e4ec6e4f8a5587b30b39064a5ba by Victor Stinner in branch 'main':
bpo-43760: Add PyThreadState_EnterTracing() (GH-28542)
https://github.com/python/cpython/commit/547d26aa08aa5e4ec6e4f8a5587b30b39064a5ba


----------

_______________________________________
Python tracker <report@bugs.python.org>
<https://bugs.python.org/issue43760>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com
[issue43760] The DISPATCH() macro is not as efficient as it could be (move PyThreadState.use_tracing) [ In reply to ]
STINNER Victor <vstinner@python.org> added the comment:

> bpo-43760: Add PyThreadState_EnterTracing() (GH-28542)

I created changes to use it:

* pythoncapi_compat: https://github.com/pythoncapi/pythoncapi_compat/commit/10fde24739cab4547e9c27c31c8804a25e23e8a0
* Cython: https://github.com/cython/cython/pull/4411
* greenlet: https://github.com/python-greenlet/greenlet/pull/267

----------

_______________________________________
Python tracker <report@bugs.python.org>
<https://bugs.python.org/issue43760>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com
[issue43760] The DISPATCH() macro is not as efficient as it could be (move PyThreadState.use_tracing) [ In reply to ]
STINNER Victor <vstinner@python.org> added the comment:

PyThreadState.cframe.use_tracing format changed again: set value set to 0 or 255.
https://github.com/python/cpython/commit/bd627eb7ed08a891dd1356756feb1ce2600358e4

----------

_______________________________________
Python tracker <report@bugs.python.org>
<https://bugs.python.org/issue43760>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com
[issue43760] The DISPATCH() macro is not as efficient as it could be (move PyThreadState.use_tracing) [ In reply to ]
STINNER Victor <vstinner@python.org> added the comment:

> * Cython: https://github.com/cython/cython/pull/4411

Merged:

* 0.29.x: https://github.com/cython/cython/commit/cbddad23e30ea6d31e0178a4c623f1f6d75452c3
* master: https://github.com/cython/cython/commit/4df1103bd30143ce022b07f98a2f62678d417e92

----------

_______________________________________
Python tracker <report@bugs.python.org>
<https://bugs.python.org/issue43760>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com
[issue43760] The DISPATCH() macro is not as efficient as it could be (move PyThreadState.use_tracing) [ In reply to ]
Change by STINNER Victor <vstinner@python.org>:


----------
pull_requests: +27304
pull_request: https://github.com/python/cpython/pull/29032

_______________________________________
Python tracker <report@bugs.python.org>
<https://bugs.python.org/issue43760>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com
[issue43760] The DISPATCH() macro is not as efficient as it could be (move PyThreadState.use_tracing) [ In reply to ]
STINNER Victor <vstinner@python.org> added the comment:


New changeset 034f607906de6e375cc9a98cc3b09f6d56f8be10 by Victor Stinner in branch 'main':
bpo-43760: Rename _PyThreadState_DisableTracing() (GH-29032)
https://github.com/python/cpython/commit/034f607906de6e375cc9a98cc3b09f6d56f8be10


----------

_______________________________________
Python tracker <report@bugs.python.org>
<https://bugs.python.org/issue43760>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com
[issue43760] The DISPATCH() macro is not as efficient as it could be (move PyThreadState.use_tracing) [ In reply to ]
STINNER Victor <vstinner@python.org> added the comment:

I created https://github.com/python/cpython/pull/29121 to add PyThreadState_SetProfile() and PyThreadState_SetTrace() functions.

----------

_______________________________________
Python tracker <report@bugs.python.org>
<https://bugs.python.org/issue43760>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com

1 2  View All