Mailing List Archive

PEP 651 -- Robust Overflow Handling
Hi everyone,

It's time for yet another PEP :)

Fortunately, this one is a small one that doesn't change much.
It's aim is to make the VM more robust.

Abstract
========

This PEP proposes that machine stack overflow is treated differently
from runaway recursion. This would allow programs to set the maximum
recursion depth to fit their needs and provide additional safety guarantees.

The following program will run safely to completion:

sys.setrecursionlimit(1_000_000)

def f(n):
if n:
f(n-1)

f(500_000)

The following program will raise a StackOverflow, without causing a VM
crash:

sys.setrecursionlimit(1_000_000)

class X:
def __add__(self, other):
return self + other

X() + 1

-----------

The full PEP can be found here:
https://www.python.org/dev/peps/pep-0651

As always, comments are welcome.

Cheers,
Mark.
_______________________________________________
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-leave@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/ZY32N43YZJM3WYXSVD7OCGVNDGPR6DUM/
Code of Conduct: http://python.org/psf/codeofconduct/
Re: PEP 651 -- Robust Overflow Handling [ In reply to ]
On 1/19/2021 8:31 AM, Mark Shannon wrote:
> Hi everyone,
>
> It's time for yet another PEP :)
>
> Fortunately, this one is a small one that doesn't change much.
> It's aim is to make the VM more robust.
>
> Abstract
> ========
>
> This PEP proposes that machine stack overflow is treated differently
> from runaway recursion. This would allow programs to set the maximum
> recursion depth to fit their needs and provide additional safety
> guarantees.
>
> The following program will run safely to completion:
>
>     sys.setrecursionlimit(1_000_000)
>
>     def f(n):
>         if n:
>             f(n-1)
>
>     f(500_000)

Are you sure? On Windows, after adding the import
and a line at the top of f
if not n % 1000: print(n)
I get with Command Prompt

C:\Users\Terry>py -m a.tem4
500000
499000
498000

C:\Users\Terry>

with a pause of after 1 to multiple seconds. Clearly did not run to
completion, but no exception or Windows crash box to indicate such
without the print.

In IDLE, I get nearly the same:
========================= RESTART: F:\Python\a\tem4.py
500000
499000
498000

================================ RESTART: Shell
>>>
The Shell restart indicates that the user code subprocess crashed and
was restarted. I checked that sys.getrecursionlimit() really returns
1_000_000.

To show completion, do something like add global m and m+=1 in f and m=0
and print(m) after the f call.


--
Terry Jan Reedy

_______________________________________________
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-leave@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/KNWNOKK3QJVS3DCHNY3FOFSVMVU3EVMA/
Code of Conduct: http://python.org/psf/codeofconduct/
Re: PEP 651 -- Robust Overflow Handling [ In reply to ]
On 1/19/2021 8:31 AM, Mark Shannon wrote:

> It's time for yet another PEP :)
>
> Fortunately, this one is a small one that doesn't change much.
> It's aim is to make the VM more robust.
>
> Abstract
> ========
>
> This PEP proposes that machine stack overflow is treated differently
> from runaway recursion.

My impression is that this is already the case in the sense given below.

https://bugs.python.org/issue42887 is about segfaults with this code
with no Python-level recursion.

mystr = "hello123"
for x in range(1000000):
mystr = mystr.__sizeof__()

Christian Heimes reproduced and said "The stack trace is several hundred
thousand (!) levels deep."
Ronald Oussoren said "The dealloc of "mystr" will cause recursive calls
to tp_dealloc along the entire chain and that can exhaust the C stack."

Since hundreds of thousands is a lot bigger than 1000, I assume that the
C recursion is not tracked. Whick is to say, recursive C calls are not
counted, unlike recursive Python calls. There have been several issues
about highly nested code and recursive structures causes segfaults,
presumably due to C level recursion. These are usually closed as 'Won't
fix' and the suggestion 'don't do that'.

I strongly agree that a fix would be great.

--
Terry Jan Reedy
_______________________________________________
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-leave@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/4NA42RO7E5XV75WHWBEFVSJMFHJ7FCUJ/
Code of Conduct: http://python.org/psf/codeofconduct/
Re: PEP 651 -- Robust Overflow Handling [ In reply to ]
On 19/01/2021 2:38 pm, Terry Reedy wrote:
> On 1/19/2021 8:31 AM, Mark Shannon wrote:
>> Hi everyone,
>>
>> It's time for yet another PEP :)
>>
>> Fortunately, this one is a small one that doesn't change much.
>> It's aim is to make the VM more robust.
>>
>> Abstract
>> ========
>>
>> This PEP proposes that machine stack overflow is treated differently
>> from runaway recursion. This would allow programs to set the maximum
>> recursion depth to fit their needs and provide additional safety
>> guarantees.
>>
>> The following program will run safely to completion:
>>
>>      sys.setrecursionlimit(1_000_000)
>>
>>      def f(n):
>>          if n:
>>              f(n-1)
>>
>>      f(500_000)
>
> Are you sure?  On Windows, after adding the import
> and a line at the top of f
>     if not n % 1000: print(n)
> I get with Command Prompt
>
> C:\Users\Terry>py -m a.tem4
> 500000
> 499000
> 498000
>
> C:\Users\Terry>
>
> with a pause of after 1 to multiple seconds.  Clearly did not run to
> completion, but no exception or Windows crash box to indicate such
> without the print.
>
> In IDLE, I get nearly the same:
> ========================= RESTART: F:\Python\a\tem4.py
> 500000
> 499000
> 498000
>
> ================================ RESTART: Shell
> >>>
> The Shell restart indicates that the user code subprocess crashed and
> was restarted.  I checked that sys.getrecursionlimit() really returns
> 1_000_000.
>
> To show completion, do something like add global m and m+=1 in f and m=0
> and print(m) after the f call.
>
>

I'm not sure whether you are saying that this doesn't work now, that it
can't work, or that it shouldn't work.

If that it doesn't work now, then I agree. That's why I've written the
PEP; it should work.

If either of the other two, why?

Cheers,
Mark.
_______________________________________________
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-leave@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/3F5S7G57GXRZ2C4E7OI5LJSTVGC6NZOI/
Code of Conduct: http://python.org/psf/codeofconduct/
Re: PEP 651 -- Robust Overflow Handling [ In reply to ]
On Tue, 19 Jan 2021 13:31:45 +0000
Mark Shannon <mark@hotpy.org> wrote:
> Hi everyone,
>
> It's time for yet another PEP :)
>
> Fortunately, this one is a small one that doesn't change much.
> It's aim is to make the VM more robust.

On the principle, no objection.

In practice, can you show how an implementation of Py_CheckStackDepth()
would look like?

Also, what is the `headroom` argument in
Py_CheckStackDepthWithHeadroom() supposed to represent? Bytes? Stack
frames?

Regards

Antoine.

_______________________________________________
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-leave@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/XG6IU6A7ZGXVMF2TXZXOZ32SIKMAHB5X/
Code of Conduct: http://python.org/psf/codeofconduct/
Re: PEP 651 -- Robust Overflow Handling [ In reply to ]
On Tue, 2021-01-19 at 13:31 +0000, Mark Shannon wrote:
> Hi everyone,
>
> It's time for yet another PEP :)
>
> Fortunately, this one is a small one that doesn't change much.
> It's aim is to make the VM more robust.
>
> Abstract
> ========
>
> This PEP proposes that machine stack overflow is treated differently
> from runaway recursion. This would allow programs to set the maximum
> recursion depth to fit their needs and provide additional safety
> guarantees.
>
> The following program will run safely to completion:
>
>      sys.setrecursionlimit(1_000_000)
>
>      def f(n):
>          if n:
>              f(n-1)
>
>      f(500_000)
>
> The following program will raise a StackOverflow, without causing a
> VM
> crash:
>
>      sys.setrecursionlimit(1_000_000)
>
>      class X:
>          def __add__(self, other):
>              return self + other
>
>      X() + 1
>


This is appreciated! I recently spend quite a bit of time trying to
solve a StackOverflow like this in NumPy (and was unable to fully
resolve it). Of course the code triggering it was bordering on
malicious, but it would be nice if it was clear how to not segfault.

Just some questions/notes:

* We currently mostly use `Py_EnterRecursiveCall()` in situations where
we need to safe-guard against "almost python" recursions. For example
an attribute lookup that returns `self`, or a list containing itself.
In those cases the python recursion limit seems a bit nicer (lower and
easier to understand).
I am not sure it actually matters much, but my question is: Are we sure
we want to replace all (or even many) C recursion checks?

* Assuming we swap `Py_EnterRecursiveCall()` logic, I am wondering if a
new `StackOverflow` exception name is useful. It may create two names
for almost identical Python code: If you unpack a list containing
itself compared to a mapping implementing `__getitem__` in Python you
would get different exceptions.

* `Py_CheckStackDepthWithHeadRoom()` is usually not necessary, because
`Py_CheckStackDepth()` would leave plenty of headroom for typical
clean-up?
Can we assume that DECREF's (i.e. list, tuple), will never check the
depth, so head-room is usually not necessary? This is all good, but I
am not immediately sure when `Py_CheckStackDepthWithHeadRoom()` would
be necessary (There are probably many cases where it clearly is, but is
it ever for fairly simple code?).
What happens if the maximum stack depth is reached while a
`StackOverflow` exception is already set? Will the current "watermark"
mechanism remain, or could there be a simple rule that an uncleared
`StackOverflow` exception ensures some additional head-room?

Cheers,

Sebastian



> -----------
>
> The full PEP can be found here:
> https://www.python.org/dev/peps/pep-0651
>
> As always, comments are welcome.
>
> Cheers,
> Mark.
> _______________________________________________
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-leave@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-dev@python.org/message/ZY32N43YZJM3WYXSVD7OCGVNDGPR6DUM/
> Code of Conduct: http://python.org/psf/codeofconduct/
>
Re: PEP 651 -- Robust Overflow Handling [ In reply to ]
On 19/01/2021 3:40 pm, Antoine Pitrou wrote:
> On Tue, 19 Jan 2021 13:31:45 +0000
> Mark Shannon <mark@hotpy.org> wrote:
>> Hi everyone,
>>
>> It's time for yet another PEP :)
>>
>> Fortunately, this one is a small one that doesn't change much.
>> It's aim is to make the VM more robust.
>
> On the principle, no objection.
>
> In practice, can you show how an implementation of Py_CheckStackDepth()
> would look like?

It would depend on the platform, but a portable-ish implementation is here:

https://github.com/markshannon/cpython/blob/pep-overflow-implementation/Include/internal/pycore_ceval.h#L71

>
> Also, what is the `headroom` argument in
> Py_CheckStackDepthWithHeadroom() supposed to represent? Bytes? Stack
> frames?

Bytes. I'll update the PEP.

>
> Regards
>
> Antoine.
>
> _______________________________________________
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-leave@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/XG6IU6A7ZGXVMF2TXZXOZ32SIKMAHB5X/
> Code of Conduct: http://python.org/psf/codeofconduct/
>
_______________________________________________
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-leave@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/G5HQG6NMIU3XSI5TMPDBMHM623WY2YPV/
Code of Conduct: http://python.org/psf/codeofconduct/
Re: PEP 651 -- Robust Overflow Handling [ In reply to ]
On Tue, 19 Jan 2021 15:54:39 +0000
Mark Shannon <mark@hotpy.org> wrote:
> On 19/01/2021 3:40 pm, Antoine Pitrou wrote:
> > On Tue, 19 Jan 2021 13:31:45 +0000
> > Mark Shannon <mark@hotpy.org> wrote:
> >> Hi everyone,
> >>
> >> It's time for yet another PEP :)
> >>
> >> Fortunately, this one is a small one that doesn't change much.
> >> It's aim is to make the VM more robust.
> >
> > On the principle, no objection.
> >
> > In practice, can you show how an implementation of Py_CheckStackDepth()
> > would look like?
>
> It would depend on the platform, but a portable-ish implementation is here:
>
> https://github.com/markshannon/cpython/blob/pep-overflow-implementation/Include/internal/pycore_ceval.h#L71

This doesn't tell me how `stack_limit_pointer` is computed or estimated
:-)


_______________________________________________
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-leave@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/5HELIHBQATVIDWT53MJZTPFUEG5CKSOQ/
Code of Conduct: http://python.org/psf/codeofconduct/
Re: PEP 651 -- Robust Overflow Handling [ In reply to ]
On 19/01/2021 3:43 pm, Sebastian Berg wrote:
> On Tue, 2021-01-19 at 13:31 +0000, Mark Shannon wrote:
>> Hi everyone,
>>
>> It's time for yet another PEP :)
>>
>> Fortunately, this one is a small one that doesn't change much.
>> It's aim is to make the VM more robust.
>>
>> Abstract
>> ========
>>
>> This PEP proposes that machine stack overflow is treated differently
>> from runaway recursion. This would allow programs to set the maximum
>> recursion depth to fit their needs and provide additional safety
>> guarantees.
>>
>> The following program will run safely to completion:
>>
>>      sys.setrecursionlimit(1_000_000)
>>
>>      def f(n):
>>          if n:
>>              f(n-1)
>>
>>      f(500_000)
>>
>> The following program will raise a StackOverflow, without causing a
>> VM
>> crash:
>>
>>      sys.setrecursionlimit(1_000_000)
>>
>>      class X:
>>          def __add__(self, other):
>>              return self + other
>>
>>      X() + 1
>>
>
>
> This is appreciated! I recently spend quite a bit of time trying to
> solve a StackOverflow like this in NumPy (and was unable to fully
> resolve it). Of course the code triggering it was bordering on
> malicious, but it would be nice if it was clear how to not segfault.
>
> Just some questions/notes:
>
> * We currently mostly use `Py_EnterRecursiveCall()` in situations where
> we need to safe-guard against "almost python" recursions. For example
> an attribute lookup that returns `self`, or a list containing itself.
> In those cases the python recursion limit seems a bit nicer (lower and
> easier to understand).
> I am not sure it actually matters much, but my question is: Are we sure
> we want to replace all (or even many) C recursion checks?

Would it help if you had the ability to increase and decrease the
recursion depth, as `Py_EnterRecursiveCall()` currently does?

I'm reluctant to expose it, as it might encourage C code authors to use
it, rather than `Py_CheckStackDepth()` resulting in crashes.

To be robust, C code must make a call to `Py_CheckStackDepth()`.
To check the recursion limit as well would be extra overhead.

>
> * Assuming we swap `Py_EnterRecursiveCall()` logic, I am wondering if a
> new `StackOverflow` exception name is useful. It may create two names
> for almost identical Python code: If you unpack a list containing
> itself compared to a mapping implementing `__getitem__` in Python you
> would get different exceptions.

True, but they are different. One is a soft limit that can be increased,
the other is a hard limit that cannot (at least not easily).

>
> * `Py_CheckStackDepthWithHeadRoom()` is usually not necessary, because
> `Py_CheckStackDepth()` would leave plenty of headroom for typical
> clean-up?

What is "typical" clean up? I would hope that typical cleanup is to
return immediately.

> Can we assume that DECREF's (i.e. list, tuple), will never check the
> depth, so head-room is usually not necessary? This is all good, but I
> am not immediately sure when `Py_CheckStackDepthWithHeadRoom()` would
> be necessary (There are probably many cases where it clearly is, but is
> it ever for fairly simple code?).

Ideally, Dealloc should call `Py_CheckStackDepth()`, but it will need
to be very cheap for that to be practical.

If C code is consuming the stack, its responsibility is to not overflow.
We can't make you call `Py_CheckStackDepth()`, but we can provide it, so
you that will have no excuse for blowing the stack :)

> What happens if the maximum stack depth is reached while a
> `StackOverflow` exception is already set? Will the current "watermark"
> mechanism remain, or could there be a simple rule that an uncleared
> `StackOverflow` exception ensures some additional head-room?

When an exception is "set", the C code should be unwinding stack,
so those states shouldn't be possible.

We can't give you extra headroom. The C stack is a fixed size.
That's why `Py_CheckStackDepthWithHeadRoom()` is provided, if
`Py_CheckStackDepth()` fails then it is too late to do much.

Cheers,
Mark.

>
> Cheers,
>
> Sebastian
>
>
>
>> -----------
>>
>> The full PEP can be found here:
>> https://www.python.org/dev/peps/pep-0651
>>
>> As always, comments are welcome.
>>
>> Cheers,
>> Mark.
>> _______________________________________________
>> Python-Dev mailing list -- python-dev@python.org
>> To unsubscribe send an email to python-dev-leave@python.org
>> https://mail.python.org/mailman3/lists/python-dev.python.org/
>> Message archived at
>> https://mail.python.org/archives/list/python-dev@python.org/message/ZY32N43YZJM3WYXSVD7OCGVNDGPR6DUM/
>> Code of Conduct: http://python.org/psf/codeofconduct/
>>
>
>
> _______________________________________________
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-leave@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/N456CVKWZ3E3VKPOE2DZMFLVSMOK5BSF/
> Code of Conduct: http://python.org/psf/codeofconduct/
>
_______________________________________________
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-leave@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/GSR6DCTXK4WWP2U5X65A3HBVY4UWQUPR/
Code of Conduct: http://python.org/psf/codeofconduct/
Re: PEP 651 -- Robust Overflow Handling [ In reply to ]
On 19/01/2021 4:15 pm, Antoine Pitrou wrote:
> On Tue, 19 Jan 2021 15:54:39 +0000
> Mark Shannon <mark@hotpy.org> wrote:
>> On 19/01/2021 3:40 pm, Antoine Pitrou wrote:
>>> On Tue, 19 Jan 2021 13:31:45 +0000
>>> Mark Shannon <mark@hotpy.org> wrote:
>>>> Hi everyone,
>>>>
>>>> It's time for yet another PEP :)
>>>>
>>>> Fortunately, this one is a small one that doesn't change much.
>>>> It's aim is to make the VM more robust.
>>>
>>> On the principle, no objection.
>>>
>>> In practice, can you show how an implementation of Py_CheckStackDepth()
>>> would look like?
>>
>> It would depend on the platform, but a portable-ish implementation is here:
>>
>> https://github.com/markshannon/cpython/blob/pep-overflow-implementation/Include/internal/pycore_ceval.h#L71
>
> This doesn't tell me how `stack_limit_pointer` is computed or estimated
> :-)

It's nothing clever, and the numbers I've chosen are just off the top of
my head.

https://github.com/markshannon/cpython/blob/pep-overflow-implementation/Modules/_threadmodule.c#L1071



>
>
> _______________________________________________
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-leave@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/5HELIHBQATVIDWT53MJZTPFUEG5CKSOQ/
> Code of Conduct: http://python.org/psf/codeofconduct/
>
_______________________________________________
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-leave@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/PXVS2I3UWTG2CDQUZ57IQ6NNCJ2JAT23/
Code of Conduct: http://python.org/psf/codeofconduct/
Re: PEP 651 -- Robust Overflow Handling [ In reply to ]
On Tue, 19 Jan 2021 16:28:46 +0000
Mark Shannon <mark@hotpy.org> wrote:

> On 19/01/2021 4:15 pm, Antoine Pitrou wrote:
> > On Tue, 19 Jan 2021 15:54:39 +0000
> > Mark Shannon <mark@hotpy.org> wrote:
> >> On 19/01/2021 3:40 pm, Antoine Pitrou wrote:
> >>> On Tue, 19 Jan 2021 13:31:45 +0000
> >>> Mark Shannon <mark@hotpy.org> wrote:
> >>>> Hi everyone,
> >>>>
> >>>> It's time for yet another PEP :)
> >>>>
> >>>> Fortunately, this one is a small one that doesn't change much.
> >>>> It's aim is to make the VM more robust.
> >>>
> >>> On the principle, no objection.
> >>>
> >>> In practice, can you show how an implementation of Py_CheckStackDepth()
> >>> would look like?
> >>
> >> It would depend on the platform, but a portable-ish implementation is here:
> >>
> >> https://github.com/markshannon/cpython/blob/pep-overflow-implementation/Include/internal/pycore_ceval.h#L71
> >
> > This doesn't tell me how `stack_limit_pointer` is computed or estimated
> > :-)
>
> It's nothing clever, and the numbers I've chosen are just off the top of
> my head.
>
> https://github.com/markshannon/cpython/blob/pep-overflow-implementation/Modules/_threadmodule.c#L1071

What about the main thread?

Is `stack_limit_pointer` some kind of thread-local? I suppose nothing
stops an OS to use different virtual addresses for the stacks of
different threads (especially if randomization of stack addresses is
desired).

Regards

Antoine.

_______________________________________________
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-leave@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/SKJMONH55RMKJNZVI4K7AVOIUOK77WZ7/
Code of Conduct: http://python.org/psf/codeofconduct/
Re: PEP 651 -- Robust Overflow Handling [ In reply to ]
I'm not clear on how you plan to implement this in CPython.

I can totally see that if a Python function calls another Python function,
you can avoid the C stack frame and hence you can have as many Python call
levels as you want.

However, there are many scenarios where a Python function calls a C
function (e.g. `filter()`, or `dict.__setitem__()`) and that C function at
some point calls a Python function (e.g. the `__hash__()` method of the
key, or even the `__del__()` method of the value being replaced). Then that
Python function can recursively do a similar thing.

Are you proposing to also support that kind of thing to go on for a million
levels of C stack frames?

(Do we even have a cross-platform way of avoiding segfaults due to C stack
overflow?)

On Tue, Jan 19, 2021 at 5:38 AM Mark Shannon <mark@hotpy.org> wrote:

> Hi everyone,
>
> It's time for yet another PEP :)
>
> Fortunately, this one is a small one that doesn't change much.
> It's aim is to make the VM more robust.
>
> Abstract
> ========
>
> This PEP proposes that machine stack overflow is treated differently
> from runaway recursion. This would allow programs to set the maximum
> recursion depth to fit their needs and provide additional safety
> guarantees.
>
> The following program will run safely to completion:
>
> sys.setrecursionlimit(1_000_000)
>
> def f(n):
> if n:
> f(n-1)
>
> f(500_000)
>
> The following program will raise a StackOverflow, without causing a VM
> crash:
>
> sys.setrecursionlimit(1_000_000)
>
> class X:
> def __add__(self, other):
> return self + other
>
> X() + 1
>
> -----------
>
> The full PEP can be found here:
> https://www.python.org/dev/peps/pep-0651
>
> As always, comments are welcome.
>
> Cheers,
> Mark.
> _______________________________________________
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-leave@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-dev@python.org/message/ZY32N43YZJM3WYXSVD7OCGVNDGPR6DUM/
> Code of Conduct: http://python.org/psf/codeofconduct/
>


--
--Guido van Rossum (python.org/~guido)
*Pronouns: he/him **(why is my pronoun here?)*
<http://feministing.com/2015/02/03/how-using-they-as-a-singular-pronoun-can-change-the-world/>
Re: PEP 651 -- Robust Overflow Handling [ In reply to ]
On Tue, 2021-01-19 at 16:22 +0000, Mark Shannon wrote:
>
>
> On 19/01/2021 3:43 pm, Sebastian Berg wrote:
> > On Tue, 2021-01-19 at 13:31 +0000, Mark Shannon wrote:
> > > Hi everyone,
> > >
> > > It's time for yet another PEP :)
> > >
> > > Fortunately, this one is a small one that doesn't change much.
> > > It's aim is to make the VM more robust.
> > >
> > > Abstract
> > > ========
> > >
> > > This PEP proposes that machine stack overflow is treated
> > > differently
> > > from runaway recursion. This would allow programs to set the
> > > maximum
> > > recursion depth to fit their needs and provide additional safety
> > > guarantees.
> > >
> > > The following program will run safely to completion:
> > >
> > >       sys.setrecursionlimit(1_000_000)
> > >
> > >       def f(n):
> > >           if n:
> > >               f(n-1)
> > >
> > >       f(500_000)
> > >
> > > The following program will raise a StackOverflow, without causing
> > > a
> > > VM
> > > crash:
> > >
> > >       sys.setrecursionlimit(1_000_000)
> > >
> > >       class X:
> > >           def __add__(self, other):
> > >               return self + other
> > >
> > >       X() + 1
> > >
> >
> >
> > This is appreciated! I recently spend quite a bit of time trying to
> > solve a StackOverflow like this in NumPy (and was unable to fully
> > resolve it).  Of course the code triggering it was bordering on
> > malicious, but it would be nice if it was clear how to not
> > segfault.
> >
> > Just some questions/notes:
> >
> > * We currently mostly use `Py_EnterRecursiveCall()` in situations
> > where
> > we need to safe-guard against "almost python" recursions. For
> > example
> > an attribute lookup that returns `self`, or a list containing
> > itself.
> > In those cases the python recursion limit seems a bit nicer (lower
> > and
> > easier to understand).
> > I am not sure it actually matters much, but my question is: Are we
> > sure
> > we want to replace all (or even many) C recursion checks?
>
> Would it help if you had the ability to increase and decrease the
> recursion depth, as `Py_EnterRecursiveCall()` currently does?
>
> I'm reluctant to expose it, as it might encourage C code authors to
> use
> it, rather than `Py_CheckStackDepth()` resulting in crashes.
>
> To be robust, C code must make a call to `Py_CheckStackDepth()`.
> To check the recursion limit as well would be extra overhead.
>
> >
> > * Assuming we swap `Py_EnterRecursiveCall()` logic, I am wondering
> > if a
> > new `StackOverflow` exception name is useful. It may create two
> > names
> > for almost identical Python code:  If you unpack a list containing
> > itself compared to a mapping implementing `__getitem__` in Python
> > you
> > would get different exceptions.
>
> True, but they are different. One is a soft limit that can be
> increased,
> the other is a hard limit that cannot (at least not easily).


Right. I think my confusion completely resolves around your proposed
change of `Py_EnterRecursiveCall()`.

A simple example (written in C):

def depth(obj, current=0):
Py_EnterRecursiveCall()

if isinstance(obj, sequence): # has the sequence slots
return depth(obj[0], current+1)
return current

will never hit the "depth" limit for a self containing list or even
sequence (as long as `GetItem` can use the C-level slot).

But `obj[0]` could nevertheless return a non-trivial object (one with
`__del__`, definitely a container with unrelated objects that could use
deleting).

As the author of the function, I have no knowledge over how much stack
space cleaning those up may require?
And say someone adds a check for `Py_CheckStackDepth()` inside a
dealloc, then this might have to cause a fatal error?

Maybe it should even be a fatal error by default in some cases?

Also, if the code is slow, the previous recursion may guard against
hanging (arguably, if that is the case I probably add an interrupt
check, I admit).


Long story short, I will trust you guys on it of course, but I am not
yet convinced that replacing the check will actually do any good (as
opposed to adding and/or providing the additional check) or even be a
service to users (since I assume that the vast majority do not crank up
the recursion limit to huge values).

Cheers,

Sebastian



>
> >
> > * `Py_CheckStackDepthWithHeadRoom()` is usually not necessary,
> > because
> > `Py_CheckStackDepth()` would leave plenty of headroom for typical
> > clean-up?
>
> What is "typical" clean up? I would hope that typical cleanup is to
> return immediately.
>
> > Can we assume that DECREF's (i.e. list, tuple), will never check
> > the
> > depth, so head-room is usually not necessary?  This is all good,
> > but I
> > am not immediately sure when `Py_CheckStackDepthWithHeadRoom()`
> > would
> > be necessary (There are probably many cases where it clearly is,
> > but is
> > it ever for fairly simple code?).
>
> Ideally, Dealloc should call `Py_CheckStackDepth()`, but it will need
> to be very cheap for that to be practical.
>
> If C code is consuming the stack, its responsibility is to not
> overflow.
> We can't make you call `Py_CheckStackDepth()`, but we can provide it,
> so
> you that will have no excuse for blowing the stack :)
>
> > What happens if the maximum stack depth is reached while a
> > `StackOverflow` exception is already set?  Will the current
> > "watermark"
> > mechanism remain, or could there be a simple rule that an uncleared
> > `StackOverflow` exception ensures some additional head-room?
>
> When an exception is "set", the C code should be unwinding stack,
> so those states shouldn't be possible.
>
> We can't give you extra headroom. The C stack is a fixed size.
> That's why `Py_CheckStackDepthWithHeadRoom()` is provided, if
> `Py_CheckStackDepth()` fails then it is too late to do much.
>
> Cheers,
> Mark.
>
> >
> > Cheers,
> >
> > Sebastian
> >
> >
> >
> > > -----------
> > >
> > > The full PEP can be found here:
> > > https://www.python.org/dev/peps/pep-0651
> > >
> > > As always, comments are welcome.
> > >
> > > Cheers,
> > > Mark.
> > > _______________________________________________
> > > Python-Dev mailing list -- python-dev@python.org
> > > To unsubscribe send an email to python-dev-leave@python.org
> > > https://mail.python.org/mailman3/lists/python-dev.python.org/
> > > Message archived at
> > > https://mail.python.org/archives/list/python-dev@python.org/message/ZY32N43YZJM3WYXSVD7OCGVNDGPR6DUM/
> > > Code of Conduct: http://python.org/psf/codeofconduct/
> > >
> >
> >
> > _______________________________________________
> > Python-Dev mailing list -- python-dev@python.org
> > To unsubscribe send an email to python-dev-leave@python.org
> > https://mail.python.org/mailman3/lists/python-dev.python.org/
> > Message archived at
> > https://mail.python.org/archives/list/python-dev@python.org/message/N456CVKWZ3E3VKPOE2DZMFLVSMOK5BSF/
> > Code of Conduct: http://python.org/psf/codeofconduct/
> >
> _______________________________________________
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-leave@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-dev@python.org/message/GSR6DCTXK4WWP2U5X65A3HBVY4UWQUPR/
> Code of Conduct: http://python.org/psf/codeofconduct/
Re: PEP 651 -- Robust Overflow Handling [ In reply to ]
On 19/01/2021 5:48 pm, Guido van Rossum wrote:
> I'm not clear on how you plan to implement this in CPython.
>
> I can totally see that if a Python function calls another Python
> function, you can avoid the C stack frame and hence you can have as many
> Python call levels as you want.
>
> However, there are many scenarios where a Python function calls a C
> function (e.g. `filter()`, or `dict.__setitem__()`) and that C function
> at some point calls a Python function (e.g. the `__hash__()` method of
> the key, or even the `__del__()` method of the value being replaced).
> Then that Python function can recursively do a similar thing.

Indeed, that is the second case below, where a Python __add__
function recursively performs addition. Most likely, the C stack will
get exhausted before the recursion limit is hit, so you'll get a
StackOverflow exception.

>
> Are you proposing to also support that kind of thing to go on for a
> million levels of C stack frames?

No, most likely 10k to 20k calls before a StackOverflow exception.

>
> (Do we even have a cross-platform way of avoiding segfaults due to C
> stack overflow?)

Arithmetic and comparisons on pointers from within the C stack may not
strictly conform to the C standard, but it works just fine.
It's pretty standard VM implementation stuff. All the JVMs do this sort
of thing.

IMO practically beats purity in this case, and crashing less has to be a
good thing.


A rather hacky proof of concept for the stack overflow handling is here:

https://github.com/python/cpython/compare/master...markshannon:pep-overflow-implementation


Cheers,
Mark.

>
> On Tue, Jan 19, 2021 at 5:38 AM Mark Shannon <mark@hotpy.org
> <mailto:mark@hotpy.org>> wrote:
>
> Hi everyone,
>
> It's time for yet another PEP :)
>
> Fortunately, this one is a small one that doesn't change much.
> It's aim is to make the VM more robust.
>
> Abstract
> ========
>
> This PEP proposes that machine stack overflow is treated differently
> from runaway recursion. This would allow programs to set the maximum
> recursion depth to fit their needs and provide additional safety
> guarantees.
>
> The following program will run safely to completion:
>
>      sys.setrecursionlimit(1_000_000)
>
>      def f(n):
>          if n:
>              f(n-1)
>
>      f(500_000)
>
> The following program will raise a StackOverflow, without causing a VM
> crash:
>
>      sys.setrecursionlimit(1_000_000)
>
>      class X:
>          def __add__(self, other):
>              return self + other
>
>      X() + 1
>
> -----------
>
> The full PEP can be found here:
> https://www.python.org/dev/peps/pep-0651
>
> As always, comments are welcome.
>
> Cheers,
> Mark.
> _______________________________________________
> Python-Dev mailing list -- python-dev@python.org
> <mailto:python-dev@python.org>
> To unsubscribe send an email to python-dev-leave@python.org
> <mailto:python-dev-leave@python.org>
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-dev@python.org/message/ZY32N43YZJM3WYXSVD7OCGVNDGPR6DUM/
> Code of Conduct: http://python.org/psf/codeofconduct/
>
>
>
> --
> --Guido van Rossum (python.org/~guido <http://python.org/~guido>)
> /Pronouns: he/him //(why is my pronoun here?)/
> <http://feministing.com/2015/02/03/how-using-they-as-a-singular-pronoun-can-change-the-world/>
_______________________________________________
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-leave@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/PYJHVAA63AWOL72OJMNFDY7VODIT5KM7/
Code of Conduct: http://python.org/psf/codeofconduct/
Re: PEP 651 -- Robust Overflow Handling [ In reply to ]
On Tue, Jan 19, 2021 at 10:08 AM Mark Shannon <mark@hotpy.org> wrote:

>
>
> On 19/01/2021 5:48 pm, Guido van Rossum wrote:
> > I'm not clear on how you plan to implement this in CPython.
> >
> > I can totally see that if a Python function calls another Python
> > function, you can avoid the C stack frame and hence you can have as many
> > Python call levels as you want.
> >
> > However, there are many scenarios where a Python function calls a C
> > function (e.g. `filter()`, or `dict.__setitem__()`) and that C function
> > at some point calls a Python function (e.g. the `__hash__()` method of
> > the key, or even the `__del__()` method of the value being replaced).
> > Then that Python function can recursively do a similar thing.
>
> Indeed, that is the second case below, where a Python __add__
> function recursively performs addition. Most likely, the C stack will
> get exhausted before the recursion limit is hit, so you'll get a
> StackOverflow exception.
>
> >
> > Are you proposing to also support that kind of thing to go on for a
> > million levels of C stack frames?
>
> No, most likely 10k to 20k calls before a StackOverflow exception.
>

Okay, that helps my understanding of the proposal. So StackOverflow really
means "C stack overflow"? And the plan seems to be

- no intervening C frame for Python-to-Python calls
- fix all C code to explicitly check for stack overflow (even in `__del__`)
- decouple RecursionError (Python) from StackOverflow (C)

If you can pull that off I am all for it.

--
--Guido van Rossum (python.org/~guido)
*Pronouns: he/him **(why is my pronoun here?)*
<http://feministing.com/2015/02/03/how-using-they-as-a-singular-pronoun-can-change-the-world/>
Re: PEP 651 -- Robust Overflow Handling [ In reply to ]
Hi Mark,

Thanks for gathering this proposal! Looks very interesting. I have some preliminary questions: how is this going to affect
the "py-bt" command of the gdb helpers (https://github.com/python/cpython/blob/master/Tools/gdb/libpython.py#L1876-L1897)
and other similar tools that produce a unified Python-C backtrace? There are several debuggers that
rely on the fact that they can merge the C stack and the Python stack by substituting every call to "_PyEval_EvalFrameDefault"
and friends for the respective Python frame obtained separately or by inspecting the frame object in the stack (like gdb does).

Is this change going to affect these tools or they will continue working as before? In case this change will affect this tools,
is there any workaround to produce the unified C/Python call stack given the Python stack and the C stack?

Kind regards,
Pablo Galindo Salgado
_______________________________________________
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-leave@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/27DON4D7Y3WMFMOT7OV6D4LD6QUXXQRB/
Code of Conduct: http://python.org/psf/codeofconduct/
Re: PEP 651 -- Robust Overflow Handling [ In reply to ]
On 1/19/2021 10:01 AM, Mark Shannon wrote:

>>> The following program will run safely to completion:

I interpreted this to mean 'works now', on whatever system you tested
this on. You question suggests that you meant "fails now but will work
with a successful patch for the PEP".

>>>      sys.setrecursionlimit(1_000_000)

On Windows, this recursion limit increase lets the function run about
2160 loops (when run with IDLE) instead of 1000, but makes the behavior
WORSE by replacing the exception with a silent failure. This could be
considered worse than a crash.

>>>      def f(n):
>>>          if n:
>>>              f(n-1)
>>>
>>>      f(500_000)

> I'm not sure whether you are saying that this doesn't work now, that it
> can't work, or that it shouldn't work.

> If that it doesn't work now, then I agree.

On my Win 10, setting the recursion limit to 2160 (2135 in IDLE, which
bumps it up to account for additional stack space used by IDLE) results
in RecursionError. Much higher and the failure is silent. Always
getting a RecursionError would itself be an improvement.

So I am saying that current recursion limit handling on Windows is
terrible and perhaps worse than on Linux.

--
Terry Jan Reedy

_______________________________________________
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-leave@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/XSPGPFHIP5VWDBEWQHKSU4A55DHFWAGF/
Code of Conduct: http://python.org/psf/codeofconduct/
Re: PEP 651 -- Robust Overflow Handling [ In reply to ]
> On 19 Jan 2021, at 17:15, Antoine Pitrou <antoine@python.org> wrote:
>
> On Tue, 19 Jan 2021 15:54:39 +0000
> Mark Shannon <mark@hotpy.org <mailto:mark@hotpy.org>> wrote:
>> On 19/01/2021 3:40 pm, Antoine Pitrou wrote:
>>> On Tue, 19 Jan 2021 13:31:45 +0000
>>> Mark Shannon <mark@hotpy.org> wrote:
>>>> Hi everyone,
>>>>
>>>> It's time for yet another PEP :)
>>>>
>>>> Fortunately, this one is a small one that doesn't change much.
>>>> It's aim is to make the VM more robust.
>>>
>>> On the principle, no objection.
>>>
>>> In practice, can you show how an implementation of Py_CheckStackDepth()
>>> would look like?
>>
>> It would depend on the platform, but a portable-ish implementation is here:
>>
>> https://github.com/markshannon/cpython/blob/pep-overflow-implementation/Include/internal/pycore_ceval.h#L71
>
> This doesn't tell me how `stack_limit_pointer` is computed or estimated
> :-)

There already is an implementation of this for Windows (``PyOS_CheckStack``). For other platforms there will
have to be a different implementation of this function. I’ve looked into this in the past for macOS, and that platform
has an API to retrieve the size of the stack as well as a pointer to the start of the stack (AFAIK stacks aren’t auto-growing
on macOS)

Ronald


Twitter / micro.blog: @ronaldoussoren
Blog: https://blog.ronaldoussoren.net/
Re: PEP 651 -- Robust Overflow Handling [ In reply to ]
Hi Pablo,

On 19/01/2021 6:46 pm, Pablo Galindo Salgado wrote:
> Hi Mark,
>
> Thanks for gathering this proposal! Looks very interesting. I have some preliminary questions: how is this going to affect
> the "py-bt" command of the gdb helpers (https://github.com/python/cpython/blob/master/Tools/gdb/libpython.py#L1876-L1897)
> and other similar tools that produce a unified Python-C backtrace? There are several debuggers that
> rely on the fact that they can merge the C stack and the Python stack by substituting every call to "_PyEval_EvalFrameDefault"
> and friends for the respective Python frame obtained separately or by inspecting the frame object in the stack (like gdb does).

I don't see any problem fixing up the gdb helpers. The code will need to
be aware that C frames will be one-to-many to Python frames, but there's
nothing tricky there.

>
> Is this change going to affect these tools or they will continue working as before? In case this change will affect this tools,
> is there any workaround to produce the unified C/Python call stack given the Python stack and the C stack?

It depends on what you mean by "similar tools".
For in-process tools, then the API will continue to work.
For out-of-process debuggers, then the author's are on their own.
But that's always been the case.
The source code is public, and it won't be any more opaque than it is at
the moment :)

Cheers,
Mark.

>
> Kind regards,
> Pablo Galindo Salgado
> _______________________________________________
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-leave@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/27DON4D7Y3WMFMOT7OV6D4LD6QUXXQRB/
> Code of Conduct: http://python.org/psf/codeofconduct/
>
_______________________________________________
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-leave@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/Y3NKLDE4ZL7SQQ6UWPHR73KA5RUY4BQI/
Code of Conduct: http://python.org/psf/codeofconduct/
Re: PEP 651 -- Robust Overflow Handling [ In reply to ]
>It depends on what you mean by "similar tools".

Any 3rd party tool or debugger that is printing merged stacks. (There are
many: gdb helpers, lldb helpers, TotalView debugger, py-spy, ...)

> For in-process tools, then the API will continue to work.
> For out-of-process debuggers, then the author's are on their own.
> But that's always been the case.
> The source code is public, and it won't be any more opaque than it is at
> the moment :)

I agree with that, but we need to have in mind that this will break
these tools and therefore we need to be aware on how difficult will
be to overcome the new situation. Is not the same that a small or
medium change is required than the case where is virtually impossible
to overcome. If we were designing the interpreter from scratch that would
not matter, but with an existing ecosystem of tools, it does. Breaking tools
in an established ecosystem has always been a major complaint of users when
we do major releases.

Notice as well that the gdb helpers *are* an out-of-process tool.

Of course, I don't imply that this should be a show stopper or similar, but
it should be taken into consideration along all the other pros and cons.

Regards from cloudy London,
Pablo Galindo Salgado

On Wed, 20 Jan 2021 at 15:12, Mark Shannon <mark@hotpy.org> wrote:

> Hi Pablo,
>
> On 19/01/2021 6:46 pm, Pablo Galindo Salgado wrote:
> > Hi Mark,
> >
> > Thanks for gathering this proposal! Looks very interesting. I have some
> preliminary questions: how is this going to affect
> > the "py-bt" command of the gdb helpers (
> https://github.com/python/cpython/blob/master/Tools/gdb/libpython.py#L1876-L1897
> )
> > and other similar tools that produce a unified Python-C backtrace? There
> are several debuggers that
> > rely on the fact that they can merge the C stack and the Python stack by
> substituting every call to "_PyEval_EvalFrameDefault"
> > and friends for the respective Python frame obtained separately or by
> inspecting the frame object in the stack (like gdb does).
>
> I don't see any problem fixing up the gdb helpers. The code will need to
> be aware that C frames will be one-to-many to Python frames, but there's
> nothing tricky there.
>
> >
> > Is this change going to affect these tools or they will continue working
> as before? In case this change will affect this tools,
> > is there any workaround to produce the unified C/Python call stack given
> the Python stack and the C stack?
>
> It depends on what you mean by "similar tools".
> For in-process tools, then the API will continue to work.
> For out-of-process debuggers, then the author's are on their own.
> But that's always been the case.
> The source code is public, and it won't be any more opaque than it is at
> the moment :)
>
> Cheers,
> Mark.
>
> >
> > Kind regards,
> > Pablo Galindo Salgado
> > _______________________________________________
> > Python-Dev mailing list -- python-dev@python.org
> > To unsubscribe send an email to python-dev-leave@python.org
> > https://mail.python.org/mailman3/lists/python-dev.python.org/
> > Message archived at
> https://mail.python.org/archives/list/python-dev@python.org/message/27DON4D7Y3WMFMOT7OV6D4LD6QUXXQRB/
> > Code of Conduct: http://python.org/psf/codeofconduct/
> >
>
Re: PEP 651 -- Robust Overflow Handling [ In reply to ]
Hi Pablo,

On 20/01/2021 3:18 pm, Pablo Galindo Salgado wrote:
> >It depends on what you mean by "similar tools".
>
> Any 3rd party tool or debugger that is printing merged stacks. (There
> are many: gdb helpers, lldb helpers, TotalView debugger, py-spy, ...)
>
>> For in-process tools, then the API will continue to work.
>> For out-of-process debuggers, then the author's are on their own.
>> But that's always been the case.
>> The source code is public, and it won't be any more opaque than it is at
>> the moment :)
>
> I agree with that, but we need to have in mind that this will break
> these tools and therefore we need to be aware on how difficult will
> be to overcome the new situation. Is not the same that a small or
> medium change is required than the case where is virtually impossible
> to overcome. If we were designing the interpreter from scratch that would
> not matter, but with an existing ecosystem of tools, it does. Breaking tools
> in an established ecosystem has always been a major complaint of users
> when we do major releases.

Do you have any specific examples?

I feel the responsibility for this belongs with the tool authors.
If they aren't using the C-API, or requesting an alternative interface
should the API be insufficient, then they can't really complain.

>
> Notice as well that the gdb helpers *are* an out-of-process tool.

True, but they are in the standard library, so they are our
responsibility to fix.

Cheers,
Mark.

>
> Of course, I don't imply that this should be a show stopper or similar,
> but it should be taken into consideration along all the other pros and cons.
>
> Regards from cloudy London,
> Pablo Galindo Salgado
>
> On Wed, 20 Jan 2021 at 15:12, Mark Shannon <mark@hotpy.org
> <mailto:mark@hotpy.org>> wrote:
>
> Hi Pablo,
>
> On 19/01/2021 6:46 pm, Pablo Galindo Salgado wrote:
> > Hi Mark,
> >
> > Thanks for gathering this proposal! Looks very interesting. I
> have some preliminary questions: how is this going to affect
> > the "py-bt" command of the gdb helpers
> (https://github.com/python/cpython/blob/master/Tools/gdb/libpython.py#L1876-L1897)
> > and other similar tools that produce a unified Python-C
> backtrace? There are several debuggers that
> > rely on the fact that they can merge the C stack and the Python
> stack by substituting every call to "_PyEval_EvalFrameDefault"
> > and friends for the respective Python frame obtained separately
> or by inspecting the frame object in the stack (like gdb does).
>
> I don't see any problem fixing up the gdb helpers. The code will
> need to
> be aware that C frames will be one-to-many to Python frames, but
> there's
> nothing tricky there.
>
> >
> > Is this change going to affect these tools or they will continue
> working as before? In case this change will affect this tools,
> > is there any workaround to produce the unified C/Python call
> stack given the Python stack and the C stack?
>
> It depends on what you mean by "similar tools".
> For in-process tools, then the API will continue to work.
> For out-of-process debuggers, then the author's are on their own.
> But that's always been the case.
> The source code is public, and it won't be any more opaque than it
> is at
> the moment :)
>
> Cheers,
> Mark.
>
> >
> > Kind regards,
> > Pablo Galindo Salgado
> > _______________________________________________
> > Python-Dev mailing list -- python-dev@python.org
> <mailto:python-dev@python.org>
> > To unsubscribe send an email to python-dev-leave@python.org
> <mailto:python-dev-leave@python.org>
> > https://mail.python.org/mailman3/lists/python-dev.python.org/
> > Message archived at
> https://mail.python.org/archives/list/python-dev@python.org/message/27DON4D7Y3WMFMOT7OV6D4LD6QUXXQRB/
> > Code of Conduct: http://python.org/psf/codeofconduct/
> >
>
_______________________________________________
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-leave@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/HHEKGU66A5LZRTQDZWO5627MK5CL37NC/
Code of Conduct: http://python.org/psf/codeofconduct/
Re: PEP 651 -- Robust Overflow Handling [ In reply to ]
> Do you have any specific examples?

From my previous message: gdb helpers, lldb helpers, TotalView debugger,
py-spy and I know of at least 3 closed source tools.

>I feel the responsibility for this belongs with the tool authors.
> If they aren't using the C-API, or requesting an alternative interface
> should the API be insufficient, then they can't really complain.

The responsibility to fix it is on the authors, yes, but is not that back
and white. In the worst case, what users will see is that tools that worked
now don't and in the worst-worst case users will see that those tools will
not work again because is virtually impossible with the new system. And
that, as I said is not a show stopper, but it matters.

In any case, if we are able to fix the gdb helpers in the stdlib, then I am
not afraid because it means that everyone could fix their own thing.

Cheers,
Pablo Galindo Salgado


On Wed, 20 Jan 2021 at 16:06, Mark Shannon <mark@hotpy.org> wrote:

> Hi Pablo,
>
> On 20/01/2021 3:18 pm, Pablo Galindo Salgado wrote:
> > >It depends on what you mean by "similar tools".
> >
> > Any 3rd party tool or debugger that is printing merged stacks. (There
> > are many: gdb helpers, lldb helpers, TotalView debugger, py-spy, ...)
> >
> >> For in-process tools, then the API will continue to work.
> >> For out-of-process debuggers, then the author's are on their own.
> >> But that's always been the case.
> >> The source code is public, and it won't be any more opaque than it is at
> >> the moment :)
> >
> > I agree with that, but we need to have in mind that this will break
> > these tools and therefore we need to be aware on how difficult will
> > be to overcome the new situation. Is not the same that a small or
> > medium change is required than the case where is virtually impossible
> > to overcome. If we were designing the interpreter from scratch that would
> > not matter, but with an existing ecosystem of tools, it does. Breaking
> tools
> > in an established ecosystem has always been a major complaint of users
> > when we do major releases.
>
> Do you have any specific examples?
>
> I feel the responsibility for this belongs with the tool authors.
> If they aren't using the C-API, or requesting an alternative interface
> should the API be insufficient, then they can't really complain.
>
> >
> > Notice as well that the gdb helpers *are* an out-of-process tool.
>
> True, but they are in the standard library, so they are our
> responsibility to fix.
>
> Cheers,
> Mark.
>
> >
> > Of course, I don't imply that this should be a show stopper or similar,
> > but it should be taken into consideration along all the other pros and
> cons.
> >
> > Regards from cloudy London,
> > Pablo Galindo Salgado
> >
> > On Wed, 20 Jan 2021 at 15:12, Mark Shannon <mark@hotpy.org
> > <mailto:mark@hotpy.org>> wrote:
> >
> > Hi Pablo,
> >
> > On 19/01/2021 6:46 pm, Pablo Galindo Salgado wrote:
> > > Hi Mark,
> > >
> > > Thanks for gathering this proposal! Looks very interesting. I
> > have some preliminary questions: how is this going to affect
> > > the "py-bt" command of the gdb helpers
> > (
> https://github.com/python/cpython/blob/master/Tools/gdb/libpython.py#L1876-L1897
> )
> > > and other similar tools that produce a unified Python-C
> > backtrace? There are several debuggers that
> > > rely on the fact that they can merge the C stack and the Python
> > stack by substituting every call to "_PyEval_EvalFrameDefault"
> > > and friends for the respective Python frame obtained separately
> > or by inspecting the frame object in the stack (like gdb does).
> >
> > I don't see any problem fixing up the gdb helpers. The code will
> > need to
> > be aware that C frames will be one-to-many to Python frames, but
> > there's
> > nothing tricky there.
> >
> > >
> > > Is this change going to affect these tools or they will continue
> > working as before? In case this change will affect this tools,
> > > is there any workaround to produce the unified C/Python call
> > stack given the Python stack and the C stack?
> >
> > It depends on what you mean by "similar tools".
> > For in-process tools, then the API will continue to work.
> > For out-of-process debuggers, then the author's are on their own.
> > But that's always been the case.
> > The source code is public, and it won't be any more opaque than it
> > is at
> > the moment :)
> >
> > Cheers,
> > Mark.
> >
> > >
> > > Kind regards,
> > > Pablo Galindo Salgado
> > > _______________________________________________
> > > Python-Dev mailing list -- python-dev@python.org
> > <mailto:python-dev@python.org>
> > > To unsubscribe send an email to python-dev-leave@python.org
> > <mailto:python-dev-leave@python.org>
> > > https://mail.python.org/mailman3/lists/python-dev.python.org/
> > > Message archived at
> >
> https://mail.python.org/archives/list/python-dev@python.org/message/27DON4D7Y3WMFMOT7OV6D4LD6QUXXQRB/
> > > Code of Conduct: http://python.org/psf/codeofconduct/
> > >
> >
>