Mailing List Archive

No longer enable Py_TRACE_REFS by default in debug build
Hi,

When Python is built in debug mode, PyObject gets 2 new fields:
_ob_prev and _ob_next. These fields change the offset of following
fields in the PyObject structure and so breaks the ABI.

I propose to modify the debug build (Py_DEBUG) to no longer imply
Py_TRACE_REFS. Antoine Pitrou proposed this idea when the C API was
discussed to get a stable ABI.

https://bugs.python.org/issue36465
https://github.com/python/cpython/pull/12615

This change makes the debug build ABI closer to the release build ABI,
but I am not sure how to compare these two ABI. Technically, C
extensions still need to be recompiled.

What do you think?

--

I also wrote a PR to remove all code related to Py_TRACE_REFS:
https://github.com/python/cpython/pull/12614

I don't think that it's right approach. I prefer to keep this special
build around to see if anyone needs it, and wait one or two Python
releases to decide what to do with it.

Victor
--
Night gathers, and now my watch begins. It shall not end until my death.
_______________________________________________
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: https://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com
Re: No longer enable Py_TRACE_REFS by default in debug build [ In reply to ]
On 09Apr2019 0925, Victor Stinner wrote:
> This change makes the debug build ABI closer to the release build ABI,
> but I am not sure how to compare these two ABI. Technically, C
> extensions still need to be recompiled.
>
> What do you think?

What are the other changes that would be required? And is there another
way to get the same functionality without ABI modifications?

I think it's worthwhile if we can really get to debug and non-debug
builds being ABI compatible. Getting partway there in this case doesn't
seem to offer any benefits.

Cheers,
Steve
_______________________________________________
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: https://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com
Re: No longer enable Py_TRACE_REFS by default in debug build [ In reply to ]
Le mar. 9 avr. 2019 à 22:16, Steve Dower <steve.dower@python.org> a écrit :
> What are the other changes that would be required?

I don't know.

> And is there another
> way to get the same functionality without ABI modifications?

Py_TRACE_REFS is a double linked list of *all* Python objects. To get
this functionality, you need to store the list somewhere. I don't know
how to maintain such list outside the PyObject structure.

One solution would be to enable Py_TRACE_REFS in release mode. Does
anyone want to add 16 bytes to every PyObject? I don't want that :-)


> I think it's worthwhile if we can really get to debug and non-debug
> builds being ABI compatible. Getting partway there in this case doesn't
> seem to offer any benefits.

Disabling Py_TRACE_REFS by default in debug mode reduces the Python
memory footprint. Py_TRACE_REFS costs 2 pointers per PyObject: 16
bytes on 64-bit platforms.

I don't think that I ever used sys.getobjects(), whereas many projects
use gc.get_objects() which is also available in release builds (not
only in debug builds).

I'm quite sure that almost nobody uses debug builds because the ABI is
incompatible.

The main question is if anyone ever used Py_TRACE_REFS? Does someone
use sys.getobjects() or PYTHONDUMPREFS environment variable?

Using PYTHONDUMPREFS=1 on a debug build (with Py_TRACE_REFS) does
simply crash Python 3.7 at exit. So I don't think that anyone use it
:-)


I wrote PR 12614 to remove all code related to Py_TRACE_REFS. I wrote
it to see which code depends on it:

commit 63509498761a0e7f72585a8cd7df325ea2abd1b2 (HEAD ->
remove_trace_refs, origin/remove_trace_refs)
Author: Victor Stinner <vstinner@redhat.com>
Date: Thu Mar 28 23:26:58 2019 +0100

WIP: bpo-36465: Remove Py_TRACE_REFS special build

Remove _ob_prev and _ob_next fields of PyObject when Python is
compiled in debug mode to make debug ABI closer to the release ABI.

Remove:

* sys.getobjects()
* PYTHONDUMPREFS environment variable
* _PyCoreConfig.dump_refs
* PyObject._ob_prev and PyObject._ob_next fields
* _PyObject_HEAD_EXTRA and _PyObject_EXTRA_INIT macros
* _Py_AddToAllObjects()
* _Py_PrintReferenceAddresses()
* _Py_PrintReferences()

Victor
--
Night gathers, and now my watch begins. It shall not end until my death.
_______________________________________________
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: https://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com
Re: No longer enable Py_TRACE_REFS by default in debug build [ In reply to ]
On 10Apr2019 0401, Victor Stinner wrote:
> Le mar. 9 avr. 2019 à 22:16, Steve Dower <steve.dower@python.org> a écrit :
>> What are the other changes that would be required?
>
> I don't know.
>
>> And is there another
>> way to get the same functionality without ABI modifications?
>
> Py_TRACE_REFS is a double linked list of *all* Python objects. To get
> this functionality, you need to store the list somewhere. I don't know
> how to maintain such list outside the PyObject structure.

There's certainly no more convenient way to do it. Maybe if we had
detached reference counts it would be easier, but it would likely still
result in ABI compatibility issues between debug builds of extensions
and release builds of Python (the most common scenario, in my experience).

> One solution would be to enable Py_TRACE_REFS in release mode. Does
> anyone want to add 16 bytes to every PyObject? I don't want that :-)

Yeah, nobody suggested that anyway :)

>> I think it's worthwhile if we can really get to debug and non-debug
>> builds being ABI compatible. Getting partway there in this case doesn't
>> seem to offer any benefits.
>
> Disabling Py_TRACE_REFS by default in debug mode reduces the Python
> memory footprint. Py_TRACE_REFS costs 2 pointers per PyObject: 16
> bytes on 64-bit platforms.

Right, except it's debug mode.

> I don't think that I ever used sys.getobjects(), whereas many projects
> use gc.get_objects() which is also available in release builds (not
> only in debug builds).
>
> I'm quite sure that almost nobody uses debug builds because the ABI is
> incompatible.

There were just over 250,000 downloads of the prebuilt debug binaries
for Windows (which are optional in the installer and turned off by
default) in March. Whether they are being used is another question, but
I know for sure at least a few people who use them.

When you want to use a debug build of your extension module, using a
debug build of CPython is the only way to do it. So unless we can get
rid of *all* the ABI incompatibilities, a debug build of CPython is
still going to be necessary and disabling/removing reference tracking
doesn't provide any benefit.

> The main question is if anyone ever used Py_TRACE_REFS? Does someone
> use sys.getobjects() or PYTHONDUMPREFS environment variable?
>
> Using PYTHONDUMPREFS=1 on a debug build (with Py_TRACE_REFS) does
> simply crash Python 3.7 at exit. So I don't think that anyone use it
> :-)

How do we track reference leaks in the buildbots? Can/should we be using
this?

It doesn't crash on Python 3.8, so I suspect fixing the bug is a better
option than using it as an excuse to remove the feature. From a quick
test, it seems that a tuple element is being freed but not removed from
the tuple, so it's probably a double-decref bug somewhere in 3.7.

Cheers,
Steve
_______________________________________________
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: https://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com
Re: No longer enable Py_TRACE_REFS by default in debug build [ In reply to ]
On 10Apr2019 1109, Steve Dower wrote:
> On 10Apr2019 0401, Victor Stinner wrote:
>>> I think it's worthwhile if we can really get to debug and non-debug
>>> builds being ABI compatible. Getting partway there in this case doesn't
>>> seem to offer any benefits.
>>
>> Disabling Py_TRACE_REFS by default in debug mode reduces the Python
>> memory footprint. Py_TRACE_REFS costs 2 pointers per PyObject: 16
>> bytes on 64-bit platforms.
>
> Right, except it's debug mode.

I left this comment unfinished :)

It's debug mode, and so you should expect less efficient memory and CPU
usage. That's why we have two modes - so that it's easier to debug issues.

Now, if debug mode was unusably slow or had way too much overhead, we'd
want to fix that. But it isn't unusable, so reducing memory usage at the
cost of making debugging harder is not compelling.

Cheers,
Steve
_______________________________________________
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: https://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com
Re: No longer enable Py_TRACE_REFS by default in debug build [ In reply to ]
I recall finding memory leaks using this. (E.g. I remember a leak in Zope
due to a cache that was never pruned.)

But presumably gc.get_objects() would have been sufficient. (IIRC it didn't
exist at the time.)

On Wed, Apr 10, 2019 at 11:48 AM Steve Dower <steve.dower@python.org> wrote:

> On 10Apr2019 1109, Steve Dower wrote:
> > On 10Apr2019 0401, Victor Stinner wrote:
> >>> I think it's worthwhile if we can really get to debug and non-debug
> >>> builds being ABI compatible. Getting partway there in this case doesn't
> >>> seem to offer any benefits.
> >>
> >> Disabling Py_TRACE_REFS by default in debug mode reduces the Python
> >> memory footprint. Py_TRACE_REFS costs 2 pointers per PyObject: 16
> >> bytes on 64-bit platforms.
> >
> > Right, except it's debug mode.
>
> I left this comment unfinished :)
>
> It's debug mode, and so you should expect less efficient memory and CPU
> usage. That's why we have two modes - so that it's easier to debug issues.
>
> Now, if debug mode was unusably slow or had way too much overhead, we'd
> want to fix that. But it isn't unusable, so reducing memory usage at the
> cost of making debugging harder is not compelling.
>
> Cheers,
> Steve
> _______________________________________________
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/guido%40python.org
>


--
--Guido van Rossum (python.org/~guido)
*Pronouns: he/him/his **(why is my pronoun here?)*
<http://feministing.com/2015/02/03/how-using-they-as-a-singular-pronoun-can-change-the-world/>
Re: No longer enable Py_TRACE_REFS by default in debug build [ In reply to ]
On Wed, Apr 10, 2019, 04:04 Victor Stinner <vstinner@redhat.com> wrote:

> Le mar. 9 avr. 2019 à 22:16, Steve Dower <steve.dower@python.org> a écrit
> :
> > What are the other changes that would be required?
>
> I don't know.
>
> > And is there another
> > way to get the same functionality without ABI modifications?
>
> Py_TRACE_REFS is a double linked list of *all* Python objects. To get
> this functionality, you need to store the list somewhere. I don't know
> how to maintain such list outside the PyObject structure.
>

I assume these pointers get updated from some generic allocation/free code.
Could that code instead overallocate by 16 bytes, use the first 16 bytes to
hold the pointers, and then return the PyObject* as (actual allocated
pointer + 16)? Basically the "container_of" trick.

I don't think that I ever used sys.getobjects(), whereas many projects
> use gc.get_objects() which is also available in release builds (not
> only in debug builds).


Can anyone explain what pydebug builds are... for? Confession: I've never
used them myself, and don't know why I would want to.

(I have to assume that most of Steve's Windows downloads are from folks who
thought they were downloading a python debugger.)

-n
Re: No longer enable Py_TRACE_REFS by default in debug build [ In reply to ]
On Wed, Apr 10, 2019 at 12:30 PM Nathaniel Smith <njs@pobox.com> wrote:

> On Wed, Apr 10, 2019, 04:04 Victor Stinner <vstinner@redhat.com> wrote:
>
>> Le mar. 9 avr. 2019 à 22:16, Steve Dower <steve.dower@python.org> a
>> écrit :
>> > What are the other changes that would be required?
>>
>> I don't know.
>>
>> > And is there another
>> > way to get the same functionality without ABI modifications?
>>
>> Py_TRACE_REFS is a double linked list of *all* Python objects. To get
>> this functionality, you need to store the list somewhere. I don't know
>> how to maintain such list outside the PyObject structure.
>>
>
> I assume these pointers get updated from some generic allocation/free
> code. Could that code instead overallocate by 16 bytes, use the first 16
> bytes to hold the pointers, and then return the PyObject* as (actual
> allocated pointer + 16)? Basically the "container_of" trick.
>
> I don't think that I ever used sys.getobjects(), whereas many projects
>> use gc.get_objects() which is also available in release builds (not
>> only in debug builds).
>
>
> Can anyone explain what pydebug builds are... for? Confession: I've never
> used them myself, and don't know why I would want to.
>

There is a bunch of extra things done in a debug build, e.g. all freed
memory is blanked out with a known pattern so it's easy to tell when you're
reading from freed memory (and thus probably messed up your refcounts). And
then various extras are tossed on to the sys module to help with things.
Basically anything people have found useful and require being compiled in
typically get clumped in under the debug build.

-Brett


>
> (I have to assume that most of Steve's Windows downloads are from folks
> who thought they were downloading a python debugger.)
>
> -n
> _______________________________________________
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/brett%40python.org
>
Re: No longer enable Py_TRACE_REFS by default in debug build [ In reply to ]
On 10Apr2019 1227, Nathaniel Smith wrote:
> On Wed, Apr 10, 2019, 04:04 Victor Stinner <vstinner@redhat.com
> <mailto:vstinner@redhat.com>> wrote:
> I don't think that I ever used sys.getobjects(), whereas many projects
> use gc.get_objects() which is also available in release builds (not
> only in debug builds).
>
>
> Can anyone explain what pydebug builds are... for? Confession: I've
> never used them myself, and don't know why I would want to.
>
> (I have to assume that most of Steve's Windows downloads are from folks
> who thought they were downloading a python debugger.)

They're for debugging :)

In general, debug builds are meant for faster inner-loop development.
They generally do incremental builds properly and much faster by
omitting most optimisations, which also enables source mapping to be
more accurate when debugging. Assertions are typically enabled so that
you are notified when a precondition is first identified rather than
when it causes the crash (compiling these out later means you don't pay
a runtime cost once you've got the inputs correct - generally these are
used for developer-controlled values, rather than user-provided ones).

So the idea is that you can quickly edit, build, debug, fix your code in
a debug configuration, and then use a release configuration for the
actual released build. Full release builds may take 2-3x longer than
full debug builds, given the extra effort they make at optimisation, and
very often can't do minimal incremental builds at all (so they may be
10-100x slower if you only modified one source file). But because the
builds behave functionally equivalently, you can iterate with the faster
configuration and get more done.

(Disclaimer: I do most of my work on Windows where this has been
properly developed. What I hear from non-Windows developers is that
other tools can't actually handle this kind of workflow properly. Sorry.)

The reason we ship debug Python binaries is because debug builds use a
different C Runtime, so if you do a debug build of an extension module
you're working on it won't actually work with a non-debug build of CPython.

While it's possible that people misread "Download debug binaries" (the
text in the installer) and think that it's an actual debugger, I'd
suggest that your total lack of context here means you should avoid
making assumptions about users you know nothing about.

Cheers,
Steve
_______________________________________________
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: https://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com
Re: No longer enable Py_TRACE_REFS by default in debug build [ In reply to ]
On 4/10/2019 2:45 PM, Steve Dower wrote:

> It's debug mode, and so you should expect less efficient memory and CPU
> usage.

On my Windows machine, 'python -m test -ugui' takes about twice as long.

> That's why we have two modes - so that it's easier to debug issues.

--
Terry Jan Reedy

_______________________________________________
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: https://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com
Re: No longer enable Py_TRACE_REFS by default in debug build [ In reply to ]
Le mer. 10 avr. 2019 à 20:09, Steve Dower <steve.dower@python.org> a écrit :
> > The main question is if anyone ever used Py_TRACE_REFS? Does someone
> > use sys.getobjects() or PYTHONDUMPREFS environment variable?
> >
> > Using PYTHONDUMPREFS=1 on a debug build (with Py_TRACE_REFS) does
> > simply crash Python 3.7 at exit. So I don't think that anyone use it
> > :-)
>
> How do we track reference leaks in the buildbots? Can/should we be using
> this?

Ah, maybe there is a misunderstanding. You don't need Py_TRACE_REFS to
track memory leaks: "python3 -m test -R 3:3" works without that.
test_regrtest contains an unit test for reference leaks (I know it
that I wrote the test :-)), and you can see that the test pass on my
PR. I also checked manually by adding a memory leak into a test: it is
still detected :-)

regrtest uses sys.gettotalrefcount(), sys.getallocatedblocks() and
support.fd_count() to track reference, memory and file descriptor
leaks. None of these functions are related to Py_TRACE_REFS.

Again, the question is who rely on Py_TRACE_REFS. If nobody rely on
it, I don't see the point of keeping this expensive feature (at least,
not by default).

> It doesn't crash on Python 3.8, so I suspect fixing the bug is a better
> option than using it as an excuse to remove the feature.

It's not what I said. I only said that it seems that nobody uses
PYTHONDUMPREFS, since it's broken for a long time. It's just a hint
about the usage of Py_TRACE_REFS.

I don't propose to remove the feature, but to disable it by default.

Victor
--
Night gathers, and now my watch begins. It shall not end until my death.
_______________________________________________
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: https://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com
Re: No longer enable Py_TRACE_REFS by default in debug build [ In reply to ]
Le mer. 10 avr. 2019 à 21:45, Brett Cannon <brett@python.org> a écrit :
>> Can anyone explain what pydebug builds are... for? Confession: I've never used them myself, and don't know why I would want to.
>
> There is a bunch of extra things done in a debug build, e.g. all freed memory is blanked out with a known pattern so it's easy to tell when you're reading from freed memory (and thus probably messed up your refcounts).

Since the debug build ABI is incompatible, it's not easy to use a
debug build. For that reasons, I'm working for a few years to add such
debugging features into regular release build. For example, you can
now get this debugger on memory allocations using PYTHONMALLOC=debug
environment variable since Python 3.6.

Since such debug feature is not easy to discover (especially if you
don't read closely What's New In Python 3.x), I added a generic "-X
dev" command line option to enable a "development mode". It enables
various similar features to debug code:
https://docs.python.org/dev/using/cmdline.html#id5

Effect of the developer mode:

* Add default warning filter, as -W default.
* Install debug hooks on memory allocators: see the
PyMem_SetupDebugHooks() C function.
* Enable the faulthandler module to dump the Python traceback on a crash.
* Enable asyncio debug mode.
* Set the dev_mode attribute of sys.flags to True

See also https://pythondev.readthedocs.io/debug_tools.html where I
started to document these debug tools and how to use them.


> And then various extras are tossed on to the sys module to help with things. Basically anything people have found useful and require being compiled in typically get clumped in under the debug build.

The debug build still contains many features which are useful to debug
C extensions. For example, it adds sys.gettotalrefcnt() which is a
convenient way to detect reference leaks. This funtion require
Py_REF_DEBUG which modifies Py_INCREF() to add "_Py_RefTotal++;". Iit
is has an impact on overall Python performance and should not be
enabled in release build.

Victor
--
Night gathers, and now my watch begins. It shall not end until my death.
_______________________________________________
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: https://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com
Re: No longer enable Py_TRACE_REFS by default in debug build [ In reply to ]
On Wed, Apr 10, 2019 at 1:50 PM Steve Dower <steve.dower@python.org> wrote:
>
> On 10Apr2019 1227, Nathaniel Smith wrote:
> > On Wed, Apr 10, 2019, 04:04 Victor Stinner <vstinner@redhat.com
> > <mailto:vstinner@redhat.com>> wrote:
> > I don't think that I ever used sys.getobjects(), whereas many projects
> > use gc.get_objects() which is also available in release builds (not
> > only in debug builds).
> >
> >
> > Can anyone explain what pydebug builds are... for? Confession: I've
> > never used them myself, and don't know why I would want to.
> >
> > (I have to assume that most of Steve's Windows downloads are from folks
> > who thought they were downloading a python debugger.)
>
> They're for debugging :)
>
> In general, debug builds are meant for faster inner-loop development.
> They generally do incremental builds properly and much faster by
> omitting most optimisations, which also enables source mapping to be
> more accurate when debugging. Assertions are typically enabled so that
> you are notified when a precondition is first identified rather than
> when it causes the crash (compiling these out later means you don't pay
> a runtime cost once you've got the inputs correct - generally these are
> used for developer-controlled values, rather than user-provided ones).
>
> So the idea is that you can quickly edit, build, debug, fix your code in
> a debug configuration, and then use a release configuration for the
> actual released build. Full release builds may take 2-3x longer than
> full debug builds, given the extra effort they make at optimisation, and
> very often can't do minimal incremental builds at all (so they may be
> 10-100x slower if you only modified one source file). But because the
> builds behave functionally equivalently, you can iterate with the faster
> configuration and get more done.

Sure, I'm familiar with the idea of debug and optimization settings in
compilers. I build python with custom -g and -O flags all the time. (I
do it by setting OPT when running configure.) It's also handy that
many Linux distros these days let you install debug metadata for all
the binaries they ship – I've used that when debugging third-party
extension modules, to get a better idea of what was happening when a
backtrace passes through libpython. But --with-pydebug is a whole
other thing beyond that, that changes the ABI, has its own wheel tags,
requires special cases in packages that use ctypes to access PyObject*
internals, and appears to be almost entirely undocumented.

It sounds like --with-pydebug has accumulated a big grab bag of
unrelated features, mostly stuff that was useful at some point for
some CPython dev trying to debug CPython itself? It's clearly not
designed with end users as the primary audience, given that no-one
knows what it actually does and that it makes third-party extensions
really awkward to run. If that's right then I think Victor's plan of
to sort through what it's actually doing makes a lot of sense,
especially if we can remove the ABI breaking stuff, since that causes
a disproportionate amount of trouble.

> The reason we ship debug Python binaries is because debug builds use a
> different C Runtime, so if you do a debug build of an extension module
> you're working on it won't actually work with a non-debug build of CPython.

...But this is an important point. I'd forgotten that MSVC has a habit
of changing the entire C runtime when you turn on the compiler's
debugging mode. (On Linux, we generally don't bother rebuilding the C
runtime unless you're actually debugging the C runtime, and anyway if
you do want to switch to a debug version of the C runtime, it's ABI
compatible so your program binaries don't have to be rebuilt.)

Is it true that if the interpreter is built against ucrtd.lib, and an
extension module is built against ucrt.lib, then they'll have
incompatible ABIs and not work together? And that this detail is part
of what's been glommed together into the "d" flag in the soabi tag on
Windows?

Is it possible for the Windows installer to include PDB files (/Zi
/DEBUG) to allow debuggers to understand the regular release
executable? (That's what I would have expected to get if I checked a
box labeled "Download debug binaries".)

-n

--
Nathaniel J. Smith -- https://vorpus.org
_______________________________________________
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: https://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com
Re: No longer enable Py_TRACE_REFS by default in debug build [ In reply to ]
10.04.19 14:01, Victor Stinner ????:
> Disabling Py_TRACE_REFS by default in debug mode reduces the Python
> memory footprint. Py_TRACE_REFS costs 2 pointers per PyObject: 16
> bytes on 64-bit platforms.

Does not the memory allocator in debug mode have even larger cost per
allocated block?

_______________________________________________
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: https://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com
Re: No longer enable Py_TRACE_REFS by default in debug build [ In reply to ]
Le jeu. 11 avr. 2019 à 07:49, Serhiy Storchaka <storchaka@gmail.com> a écrit :
> 10.04.19 14:01, Victor Stinner ????:
> > Disabling Py_TRACE_REFS by default in debug mode reduces the Python
> > memory footprint. Py_TRACE_REFS costs 2 pointers per PyObject: 16
> > bytes on 64-bit platforms.
>
> Does not the memory allocator in debug mode have even larger cost per
> allocated block?

What do you mean? That a debug build already waste too much memory and
so doesn't deserve to have a smaller memory footprint? I'm not sure
that I understand your point.

A smaller footprint can mean that more people may be able to use debug
build. Disabling Py_TRACE_REFS should make Python a little bit faster.

My question stands: is it worth to keep a feature which "waste"
resources (memory footprint and CPU) and nobody uses it?

Debug hooks add 4 x sizeof(size_t) bytes to every memory allocation to
detect buffer underflow and buffer overflow. That's 32 bytes per
memory allocation. By the way, IMHO the "serial number" is not really
useful and could be removed to only add 3 x sizeof(size_t) (24 bytes).
But the debug hook is very useful, it's common that it helps me to
find real bugs in the code. Whereas I don't recall that Py_TRACE_REFS
helped me even once.

Victor
--
Night gathers, and now my watch begins. It shall not end until my death.
_______________________________________________
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: https://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com
Re: No longer enable Py_TRACE_REFS by default in debug build [ In reply to ]
On 11Apr2019 0228, Victor Stinner wrote:
> Le jeu. 11 avr. 2019 à 07:49, Serhiy Storchaka <storchaka@gmail.com> a écrit :
>> 10.04.19 14:01, Victor Stinner ????:
>>> Disabling Py_TRACE_REFS by default in debug mode reduces the Python
>>> memory footprint. Py_TRACE_REFS costs 2 pointers per PyObject: 16
>>> bytes on 64-bit platforms.
>>
>> Does not the memory allocator in debug mode have even larger cost per
>> allocated block?
>
> What do you mean? That a debug build already waste too much memory and
> so doesn't deserve to have a smaller memory footprint? I'm not sure
> that I understand your point.

He means you're micro-optimising something that doesn't matter. If you
really wanted to reduce memory usage in debug builds, you'd go after one
of the bigger "problems".

> A smaller footprint can mean that more people may be able to use debug
> build. Disabling Py_TRACE_REFS should make Python a little bit faster.

This isn't one of the goals of a debug build though, and you haven't
pointed at any examples of people not being able to use the debug build
because of memory pressure. (Which is because most people who are not
working on CPython itself should not be using the debug build.)

> My question stands: is it worth to keep a feature which "waste"
> resources (memory footprint and CPU) and nobody uses it?

You haven't even tried to show that nobody uses it, other than pointing
out that it exposes a crash due to a refcounting bug (which is kind of
the point ;) ).

Cheers,
Steve
_______________________________________________
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: https://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com
Re: No longer enable Py_TRACE_REFS by default in debug build [ In reply to ]
On 10Apr2019 1917, Nathaniel Smith wrote:
> It sounds like --with-pydebug has accumulated a big grab bag of
> unrelated features, mostly stuff that was useful at some point for
> some CPython dev trying to debug CPython itself? It's clearly not
> designed with end users as the primary audience, given that no-one
> knows what it actually does and that it makes third-party extensions
> really awkward to run. If that's right then I think Victor's plan of
> to sort through what it's actually doing makes a lot of sense,
> especially if we can remove the ABI breaking stuff, since that causes
> a disproportionate amount of trouble.

Does it really cause a "disproportionate" amount of trouble? It's
definitely not meant for anyone who isn't working on C code, whether in
CPython, an extension or a host application. If you want to use
third-party extensions and are not able to rebuild them, that's a very
good sign that you probably shouldn't be on the debug build at all.

Perhaps the "--with-pydebug" option is too attractive? (Is it the
default?) That's easily fixed.

>> The reason we ship debug Python binaries is because debug builds use a
>> different C Runtime, so if you do a debug build of an extension module
>> you're working on it won't actually work with a non-debug build of CPython.
>
> ...But this is an important point. I'd forgotten that MSVC has a habit
> of changing the entire C runtime when you turn on the compiler's
> debugging mode.

Technically they are separate options, but most project files are
configured such that *their* Debug/Release switch affects both the
compiler options (optimization) and the linker options (C runtime linkage).

> Is it true that if the interpreter is built against ucrtd.lib, and an
> extension module is built against ucrt.lib, then they'll have
> incompatible ABIs and not work together? And that this detail is part
> of what's been glommed together into the "d" flag in the soabi tag on
> Windows?

Yep, except it's not actually in the soabi tag, but it's the "_d" suffix
on module/executable names.

> Is it possible for the Windows installer to include PDB files (/Zi
> /DEBUG) to allow debuggers to understand the regular release
> executable? (That's what I would have expected to get if I checked a
> box labeled "Download debug binaries".)

That box is immediately below one labelled "Download debug symbols", so
hopefully seeing it in context would have set the right expectation.
(And since I have them, there were 1.3 million downloads of the symbol
packages via this option in March, but we also enable it by default via
Visual Studio and that's responsible for about 1 million of those.)

Cheers,
Steve
_______________________________________________
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: https://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com
Re: No longer enable Py_TRACE_REFS by default in debug build [ In reply to ]
11.04.19 12:28, Victor Stinner ????:
> Le jeu. 11 avr. 2019 à 07:49, Serhiy Storchaka <storchaka@gmail.com> a écrit :
>> 10.04.19 14:01, Victor Stinner ????:
>>> Disabling Py_TRACE_REFS by default in debug mode reduces the Python
>>> memory footprint. Py_TRACE_REFS costs 2 pointers per PyObject: 16
>>> bytes on 64-bit platforms.
>>
>> Does not the memory allocator in debug mode have even larger cost per
>> allocated block?
>
> What do you mean? That a debug build already waste too much memory and
> so doesn't deserve to have a smaller memory footprint? I'm not sure
> that I understand your point.

If reducing the Python memory footprint is an argument for disabling
Py_TRACE_REFS, it is a weak argument because there is larger overhead in
the debug build.

On other hand, since using the debug allocator doesn't cause problems
with compatibility, it may be possible to use similar technique for the
objects double list. Although this is not easy because of objects placed
at static memory.

_______________________________________________
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: https://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com
Re: No longer enable Py_TRACE_REFS by default in debug build [ In reply to ]
On Thu, Apr 11, 2019 at 8:32 AM Serhiy Storchaka <storchaka@gmail.com> wrote:
> On other hand, since using the debug allocator doesn't cause problems
> with compatibility, it may be possible to use similar technique for the
> objects double list. Although this is not easy because of objects placed
> at static memory.

I guess one could track static objects separately, e.g. keep a simple
global PyList containing all statically allocated objects. (This is
easy since we know they're all immortal.) And then sys.getobjects()
could walk the heap objects and statically allocated objects
separately.

-n

--
Nathaniel J. Smith -- https://vorpus.org
_______________________________________________
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: https://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com
Re: No longer enable Py_TRACE_REFS by default in debug build [ In reply to ]
Le jeu. 11 avr. 2019 à 17:33, Serhiy Storchaka <storchaka@gmail.com> a écrit :
> If reducing the Python memory footprint is an argument for disabling
> Py_TRACE_REFS, it is a weak argument because there is larger overhead in
> the debug build.

The "serialno" field of debug memory allocators is documented as: "an
excellent way to set a breakpoint on the next run, to capture the
instant at which this block was passed out." I'm debugging crashes and
memory leaks in CPython for 10 years, and I simply never had to use
"serialno". I wrote https://bugs.python.org/issue36611 to remove the
serialno field of debug hooks on Python memory allocators: it reduces
the memory footprint by 5% (ex: 1.2 MiB on 33.0 MiB when running
test_asyncio).

Python is used on devices with low memory (ex: 256 MiB for the whole
system). Allowing developers to use a debug build on such devices seem
to be a legit rationale for such change. The debug build is very
useful to identify bugs in C extensions.


> On other hand, since using the debug allocator doesn't cause problems
> with compatibility, it may be possible to use similar technique for the
> objects double list. Although this is not easy because of objects placed
> at static memory.

I'm not sure of what you means by "objects placed at static memory":
the double linked list of all Python objects is created at runtime.
_ob_next and _ob_prev are initialized statically to NULL.

I would be interested if Py_TRACE_REFS could be reimplemented in a
more dynamic fashion. Even if it would still require a debug build, it
would be nice to be able to "opt-in" for this feature (have it
disabled by default, again, to reduce the overhead and reduce the
memory footprint), as tracemalloc which plugs itself into memory
allocators to attach traces to memory blocks.

Except Guido who wrote "I recall finding memory leaks using this.
(E.g. I remember a leak in Zope due to a cache that was never pruned.)
But presumably gc.get_objects() would have been sufficient. (IIRC it
didn't exist at the time.)", at this point, nobody said that they use
Py_TRACE_REFS. So I'm not sure that it's worth it to invest time on a
feature if nobody uses it?

Victor
--
Night gathers, and now my watch begins. It shall not end until my death.
_______________________________________________
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: https://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com
Re: No longer enable Py_TRACE_REFS by default in debug build [ In reply to ]
Le ven. 12 avr. 2019 à 12:57, Victor Stinner <vstinner@redhat.com> a écrit :
> I wrote https://bugs.python.org/issue36611 to remove the serialno field
> of debug hooks on Python memory allocators: it reduces
> the memory footprint by 5% (ex: 1.2 MiB on 33.0 MiB when running
> test_asyncio).

I measured the memory footprint when I combine my two changes:

* disable Py_TRACE_REFS: https://bugs.python.org/issue36465
* disable/remove serialno field: https://bugs.python.org/issue36611

python3 -m test test_asyncio, without => with the change: 34,038.0 kB
=> 30,612.2 kB (-3,425.8 kiB, -10%)

A reduction of 3.4 MiB on 34.0 MiB is quite significant, no?

Victor
--
Night gathers, and now my watch begins. It shall not end until my death.
_______________________________________________
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: https://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com
Re: No longer enable Py_TRACE_REFS by default in debug build [ In reply to ]
Serhiy Storchaka schrieb am 11.04.19 um 17:30:
> If reducing the Python memory footprint is an argument for disabling
> Py_TRACE_REFS, it is a weak argument because there is larger overhead in
> the debug build.

I think what Victor is argueing is rather that we have better ways to debug
memory problems these days, so we might be able to get rid of a relict that
no-one is using (or should be using) anymore and that has its drawbacks
(such as a very different ABI and higher memory load).

I don't really have an opinion here, but I can at least say that I never
found a use case for Py_TRACE_REFS myself and therefore certainly wouldn't
miss it.

Stefan

_______________________________________________
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: https://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com
Re: No longer enable Py_TRACE_REFS by default in debug build [ In reply to ]
On Fri, Apr 12, 2019 at 5:51 AM Stefan Behnel <stefan_ml@behnel.de> wrote:

> Serhiy Storchaka schrieb am 11.04.19 um 17:30:
> > If reducing the Python memory footprint is an argument for disabling
> > Py_TRACE_REFS, it is a weak argument because there is larger overhead in
> > the debug build.
>
> I think what Victor is argueing is rather that we have better ways to debug
> memory problems these days, so we might be able to get rid of a relict that
> no-one is using (or should be using) anymore and that has its drawbacks
> (such as a very different ABI and higher memory load).
>
> I don't really have an opinion here, but I can at least say that I never
> found a use case for Py_TRACE_REFS myself and therefore certainly wouldn't
> miss it.
>

I have a feeling that at some point someone might want to use this to debug
some leak (presumably caused by C code) beyond what gc.get_objects() can
report. But I agree that it isn't useful to the vast majority of users of a
regular debug build. So let's leave it off by default even in debug builds.
But let's not delete the macros.

--
--Guido van Rossum (python.org/~guido)
*Pronouns: he/him/his **(why is my pronoun here?)*
<http://feministing.com/2015/02/03/how-using-they-as-a-singular-pronoun-can-change-the-world/>
Re: No longer enable Py_TRACE_REFS by default in debug build [ In reply to ]
12.04.19 17:40, Guido van Rossum ????:
> So let's leave it off
> by default even in debug builds. But let's not delete the macros.

Maybe switch it on (together with other disabled by default options) on
some fast buildbot?

_______________________________________________
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: https://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com
Re: No longer enable Py_TRACE_REFS by default in debug build [ In reply to ]
Victor Stinner wrote:
> Python is used on devices with low memory (ex: 256 MiB for the whole
> system). Allowing developers to use a debug build on such devices seem
> to be a legit rationale for such change.

Rather than removing features altogether, maybe the debug
build could be split into a number of separate features
that can be enabled individually?

--
Greg
_______________________________________________
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: https://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com

1 2  View All