Mailing List Archive

Victor Stinner wrote:
> I'm not sure of what you means by "objects placed at static memory":
> the double linked list of all Python objects is created at runtime.
> _ob_next and _ob_prev are initialized statically to NULL.

The trick of allocating extra memory in front of the object
would be harder to pull off for statically allocated objects,
although probably not impossible.

--
Greg
_______________________________________________
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: https://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com

Le sam. 13 avr. 2019 à 00:38, Greg Ewing <greg.ewing@canterbury.ac.nz> a écrit :
> Rather than removing features altogether, maybe the debug
> build could be split into a number of separate features
> that can be enabled individually?

I don't propose to *remove* a feature, but just to *disable* it *by
default* (when Python is compiled in debug mode):

"[WIP] bpo-36465: Py_DEBUG no longer implies Py_TRACE_REFS #12615"
https://github.com/python/cpython/pull/12615/files

In short, my change just removes:

/* Py_DEBUG implies Py_TRACE_REFS. */
#if defined(Py_DEBUG) && !defined(Py_TRACE_REFS)
#define Py_TRACE_REFS
#endif

The feature will still be accessible if you compile Python with
Py_TRACE_REFS defined.

In practice, I understood that the debug build of Python is not known
by all core developers, and it seems like it's mostly used by core
developers. Maybe it's even only used by core developers? It's hard to
say. If it's only used by core developers, I hope that all core devs
know to compile Python :-)

Victor
--
Night gathers, and now my watch begins. It shall not end until my death.
_______________________________________________
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: https://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com

On Thu, Apr 11, 2019 at 8:26 AM Steve Dower <steve.dower@python.org> wrote:
>
> On 10Apr2019 1917, Nathaniel Smith wrote:
> > It sounds like --with-pydebug has accumulated a big grab bag of
> > unrelated features, mostly stuff that was useful at some point for
> > some CPython dev trying to debug CPython itself? It's clearly not
> > designed with end users as the primary audience, given that no-one
> > knows what it actually does and that it makes third-party extensions
> > really awkward to run. If that's right then I think Victor's plan of
> > to sort through what it's actually doing makes a lot of sense,
> > especially if we can remove the ABI breaking stuff, since that causes
> > a disproportionate amount of trouble.
>
> Does it really cause a "disproportionate" amount of trouble? It's
> definitely not meant for anyone who isn't working on C code, whether in
> CPython, an extension or a host application. If you want to use
> third-party extensions and are not able to rebuild them, that's a very
> good sign that you probably shouldn't be on the debug build at all.

Well, here's what I mean by "disproportionate". Some of the costs of
the ABI divergence are:

- The first time I had to debug a C extension, I wasted a bunch of
time trying to figure out how I was supposed to use Debian's
'python-dbg' package (the --with-pydebug build), before eventually
figuring out that it was a red herring and what I actually wanted was
the -dbgsym package (their equivalent of MSVC's /Zi /DEBUG files).

- The extension loading machinery has extra code and complexity to
track the two different ABIs. The package ecosystem does too, e.g.
distutils needs to name extensions appropriately, and we need special
wheel tags, and pip needs code to handle these tags:
https://github.com/pypa/pip/blob/54b6a91405adc79cdb8a2954e9614d6860799ccb/src/pip/_internal/pep425tags.py#L106-L109

- If you want some of the features of --with-pydebug that don't change
the ABI, then you still have to rebuild third-party extensions to get
at them, and that's a significant hassle. (I could do it if I had to,
but my time has value.)

- Everyone who uses ctypes to access a PyObject* has to include some
extra hacks to handle the difference between the regular and debug
ABIs. There are a few different versions that get copy/pasted around
as folklore, and they're all pretty obscure. For example:
https://github.com/pallets/jinja/blob/fd89fed7456e755e33ba70674c41be5ab222e193/jinja2/debug.py#L317-L334
https://github.com/johndpope/sims4-ai-engine/blob/865212e841c716dc4364e0dba286f02af8d716e8/core/framewrapper.py#L12-L41
https://github.com/python-trio/trio/blob/862ced04e1f19287e098380ed8a0635004c36dd1/trio/_core/_multierror.py#L282
And then if you want to test this code, it means you have to add a
--with-pydebug build to your CI infrastructure...

I don't know how many people use Py_TRACE_REFS, but if we can't find
anyone on python-dev who uses it then it must be pretty rare. If
dropping Py_TRACE_REFS would let us converge the ABIs and get rid of
all the stuff above, then that seems like a pretty good trade! But
maybe the Windows C runtime issue will foil this...

> >> The reason we ship debug Python binaries is because debug builds use a
> >> different C Runtime, so if you do a debug build of an extension module
> >> you're working on it won't actually work with a non-debug build of CPython.
> >
> > ...But this is an important point. I'd forgotten that MSVC has a habit
> > of changing the entire C runtime when you turn on the compiler's
> > debugging mode.
>
> Technically they are separate options, but most project files are
> configured such that *their* Debug/Release switch affects both the
> compiler options (optimization) and the linker options (C runtime linkage).

So how do other projects handle this? I guess historically the main
target audience for Visual Studio was folks building monolithic apps,
where you can just rebuild everything with whatever options you want,
and compared to that Python extensions are messier. But Python isn't
the only project in this boat. Do ruby, nodejs, R, etc., all provide
separate debug builds with incompatible ABIs on Windows, and propagate
that information throughout their module/package ecosystem?

-n

--
Nathaniel J. Smith -- https://vorpus.org
_______________________________________________
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: https://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com

On 12Apr.2019 1643, Nathaniel Smith wrote:
> On Thu, Apr 11, 2019 at 8:26 AM Steve Dower <steve.dower@python.org> wrote:
>>
>> On 10Apr2019 1917, Nathaniel Smith wrote:
> I don't know how many people use Py_TRACE_REFS, but if we can't find
> anyone on python-dev who uses it then it must be pretty rare. If
> dropping Py_TRACE_REFS would let us converge the ABIs and get rid of
> all the stuff above, then that seems like a pretty good trade! But
> maybe the Windows C runtime issue will foil this...

The very first question I asked was whether this would let us converge
the ABIs, and the answer was "no".

Otherwise I'd have said go for it, despite the C runtime issues.

>>>> The reason we ship debug Python binaries is because debug builds use a
>>>> different C Runtime, so if you do a debug build of an extension module
>>>> you're working on it won't actually work with a non-debug build of CPython.
>>>
>>> ...But this is an important point. I'd forgotten that MSVC has a habit
>>> of changing the entire C runtime when you turn on the compiler's
>>> debugging mode.
>>
>> Technically they are separate options, but most project files are
>> configured such that *their* Debug/Release switch affects both the
>> compiler options (optimization) and the linker options (C runtime linkage).
>
> So how do other projects handle this? I guess historically the main
> target audience for Visual Studio was folks building monolithic apps,
> where you can just rebuild everything with whatever options you want,
> and compared to that Python extensions are messier. But Python isn't
> the only project in this boat. Do ruby, nodejs, R, etc., all provide
> separate debug builds with incompatible ABIs on Windows, and propagate
> that information throughout their module/package ecosystem?

Mostly I hear complaints about those languages *not* providing any help
here. Python is renowned for having significantly better Windows support
than any of them, so they're the wrong comparison to make in my opinion.
Arguing that we should regress because other languages haven't caught up
to us yet makes no sense.

The tools that are better than Python typically don't ship debug builds
either, unless you specifically request them. But they also don't leak
their implementation details all over the place. If we had a better C
API, we wouldn't have users who needed to match ABIs.

For the most part, disabling optimizations in your own extension but
using the non-debug ABI is sufficient, and if you're having to deal with
other people's packages then maybe you don't have any choice (though I
do know of people who have built debug versions of numpy before - turns
out Windows developers are often just as capable as non-Windows
developers when it comes to building things ;) ). And yes, they could
also build CPython from source as well to get the debug ABI, or get the
debug symbols, but I saw enough need that I decided it was worth the
effort to just solve that problem. 250k downloads a month is enough to
justify it for me.

Not to bring the packaging discussions to another venue, but maybe this
is yet another area we need to stop pretending that we're able to solve
every single problem with just the tools we already have available?
People who want debug builds of packages can build them themselves, even
numpy and scipy, they don't need us to preemptively do all their work
for them. But we can (and should) help short-cut unnecessary effort or
research by providing helpful tools and instruction.

Cheers,
Steve
_______________________________________________
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: https://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com

On Fri, Apr 12, 2019 at 5:05 PM Steve Dower <steve.dower@python.org> wrote:
>
> On 12Apr.2019 1643, Nathaniel Smith wrote:
> > On Thu, Apr 11, 2019 at 8:26 AM Steve Dower <steve.dower@python.org> wrote:
> >>
> >> On 10Apr2019 1917, Nathaniel Smith wrote:
> > I don't know how many people use Py_TRACE_REFS, but if we can't find
> > anyone on python-dev who uses it then it must be pretty rare. If
> > dropping Py_TRACE_REFS would let us converge the ABIs and get rid of
> > all the stuff above, then that seems like a pretty good trade! But
> > maybe the Windows C runtime issue will foil this...
>
> The very first question I asked was whether this would let us converge
> the ABIs, and the answer was "no".
>
> Otherwise I'd have said go for it, despite the C runtime issues.

I don't see that in the thread... just Victor saying he isn't sure
whether there might be other ABI incompatibilities lurking that he
hasn't found yet. Did I miss something?

I'm mostly interested in this because of the possibility of converging
the ABIs. If you think that the C runtime thing isn't a blocker for
that, then that's useful information. Though obviously we still need
to figure out whether there are any other blockers :-).

> >>>> The reason we ship debug Python binaries is because debug builds use a
> >>>> different C Runtime, so if you do a debug build of an extension module
> >>>> you're working on it won't actually work with a non-debug build of CPython.
> >>>
> >>> ...But this is an important point. I'd forgotten that MSVC has a habit
> >>> of changing the entire C runtime when you turn on the compiler's
> >>> debugging mode.
> >>
> >> Technically they are separate options, but most project files are
> >> configured such that *their* Debug/Release switch affects both the
> >> compiler options (optimization) and the linker options (C runtime linkage).
> >
> > So how do other projects handle this? I guess historically the main
> > target audience for Visual Studio was folks building monolithic apps,
> > where you can just rebuild everything with whatever options you want,
> > and compared to that Python extensions are messier. But Python isn't
> > the only project in this boat. Do ruby, nodejs, R, etc., all provide
> > separate debug builds with incompatible ABIs on Windows, and propagate
> > that information throughout their module/package ecosystem?
>
> Mostly I hear complaints about those languages *not* providing any help
> here. Python is renowned for having significantly better Windows support
> than any of them, so they're the wrong comparison to make in my opinion.
> Arguing that we should regress because other languages haven't caught up
> to us yet makes no sense.
>
> The tools that are better than Python typically don't ship debug builds
> either, unless you specifically request them. But they also don't leak
> their implementation details all over the place. If we had a better C
> API, we wouldn't have users who needed to match ABIs.

Do you happen to have a list of places where the C API leaks details
of the underlying CRT?

(I'm mostly curious because whenever I've looked my conclusion was
essentially: "Well....... I don't see any places that are *definitely*
broken, so maybe mixing CRTs is fine? but I have zero confidence that
I caught everything, so probably better to play it safe?". At least on
py3 – I know the py2 C API was definitely broken if you mixed CRTs,
because of the exposed FILE*.)

> For the most part, disabling optimizations in your own extension but
> using the non-debug ABI is sufficient, and if you're having to deal with
> other people's packages then maybe you don't have any choice (though I
> do know of people who have built debug versions of numpy before - turns
> out Windows developers are often just as capable as non-Windows
> developers when it comes to building things ;)

I'm not sure why you think I was implying otherwise? I'm sorry if you
thought I was attacking your users or something. I did say that I
thought most users downloading the debug builds were probably confused
about what they were actually getting, but I didn't mean because they
were stupid Windows users, I meant because the debug builds are so
confusing that even folks on the Python core team are confused about
what they're actually getting.

-n

--
Nathaniel J. Smith -- https://vorpus.org
_______________________________________________
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: https://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com

>> The very first question I asked was whether this would let us converge
>> the ABIs, and the answer was "no".

The answer is yes and it's my primary goal.

See my first email: "This change makes the debug build ABI closer to the
release build ABI".

To be honest, I am now lost in this long thread :-) I don't recall why I
started to argue so much about the memory footprint, it's not really the
main point here.

Victor


>> Otherwise I'd have said go for it, despite the C runtime issues.
>
> I don't see that in the thread... just Victor saying he isn't sure
> whether there might be other ABI incompatibilities lurking that he
> hasn't found yet. Did I miss something?
>
> I'm mostly interested in this because of the possibility of converging
> the ABIs. If you think that the C runtime thing isn't a blocker for
> that, then that's useful information. Though obviously we still need
> to figure out whether there are any other blockers :-).
>
>> >>>> The reason we ship debug Python binaries is because debug builds
use a
>> >>>> different C Runtime, so if you do a debug build of an extension
module
>> >>>> you're working on it won't actually work with a non-debug build of
CPython.
>> >>>
>> >>> ...But this is an important point. I'd forgotten that MSVC has a
habit
>> >>> of changing the entire C runtime when you turn on the compiler's
>> >>> debugging mode.
>> >>
>> >> Technically they are separate options, but most project files are
>> >> configured such that *their* Debug/Release switch affects both the
>> >> compiler options (optimization) and the linker options (C runtime
linkage).
>> >
>> > So how do other projects handle this? I guess historically the main
>> > target audience for Visual Studio was folks building monolithic apps,
>> > where you can just rebuild everything with whatever options you want,
>> > and compared to that Python extensions are messier. But Python isn't
>> > the only project in this boat. Do ruby, nodejs, R, etc., all provide
>> > separate debug builds with incompatible ABIs on Windows, and propagate
>> > that information throughout their module/package ecosystem?
>>
>> Mostly I hear complaints about those languages *not* providing any help
>> here. Python is renowned for having significantly better Windows support
>> than any of them, so they're the wrong comparison to make in my opinion.
>> Arguing that we should regress because other languages haven't caught up
>> to us yet makes no sense.
>>
>> The tools that are better than Python typically don't ship debug builds
>> either, unless you specifically request them. But they also don't leak
>> their implementation details all over the place. If we had a better C
>> API, we wouldn't have users who needed to match ABIs.
>
> Do you happen to have a list of places where the C API leaks details
> of the underlying CRT?
>
> (I'm mostly curious because whenever I've looked my conclusion was
> essentially: "Well....... I don't see any places that are *definitely*
> broken, so maybe mixing CRTs is fine? but I have zero confidence that
> I caught everything, so probably better to play it safe?". At least on
> py3 – I know the py2 C API was definitely broken if you mixed CRTs,
> because of the exposed FILE*.)
>
>> For the most part, disabling optimizations in your own extension but
>> using the non-debug ABI is sufficient, and if you're having to deal with
>> other people's packages then maybe you don't have any choice (though I
>> do know of people who have built debug versions of numpy before - turns
>> out Windows developers are often just as capable as non-Windows
>> developers when it comes to building things ;)
>
> I'm not sure why you think I was implying otherwise? I'm sorry if you
> thought I was attacking your users or something. I did say that I
> thought most users downloading the debug builds were probably confused
> about what they were actually getting, but I didn't mean because they
> were stupid Windows users, I meant because the debug builds are so
> confusing that even folks on the Python core team are confused about
> what they're actually getting.
>
> -n
>
> --
> Nathaniel J. Smith -- https://vorpus.org
> _______________________________________________
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
https://mail.python.org/mailman/options/python-dev/vstinner%40redhat.com
>

--
Night gathers, and now my watch begins. It shall not end until my death.

On Thu, 11 Apr 2019 08:26:47 -0700
Steve Dower <steve.dower@python.org> wrote:
> On 10Apr2019 1917, Nathaniel Smith wrote:
> > It sounds like --with-pydebug has accumulated a big grab bag of
> > unrelated features, mostly stuff that was useful at some point for
> > some CPython dev trying to debug CPython itself? It's clearly not
> > designed with end users as the primary audience, given that no-one
> > knows what it actually does and that it makes third-party extensions
> > really awkward to run. If that's right then I think Victor's plan of
> > to sort through what it's actually doing makes a lot of sense,
> > especially if we can remove the ABI breaking stuff, since that causes
> > a disproportionate amount of trouble.
>
> Does it really cause a "disproportionate" amount of trouble? It's
> definitely not meant for anyone who isn't working on C code, whether in
> CPython, an extension or a host application. If you want to use
> third-party extensions and are not able to rebuild them, that's a very
> good sign that you probably shouldn't be on the debug build at all.

I can't really agree with that. There are third-party extensions that
have non-trivial build requirements. The fact that you have to rebuild
third-party dependencies is a strong deterrent against using pydebug
builds even when they may be actually useful (for example when
debugging an extension module of your own).

If you could just install mainstream binary packages (e.g. from Anaconda
or PyPI) on a debug build interpreter, the pain would go away.

Regards

Antoine.


_______________________________________________
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: https://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com

On 12Apr2019 1819, Nathaniel Smith wrote:
> On Fri, Apr 12, 2019 at 5:05 PM Steve Dower <steve.dower@python.org> wrote:
>>
>> On 12Apr.2019 1643, Nathaniel Smith wrote:
>>> On Thu, Apr 11, 2019 at 8:26 AM Steve Dower <steve.dower@python.org> wrote:
>> The very first question I asked was whether this would let us converge
>> the ABIs, and the answer was "no".
>>
>> Otherwise I'd have said go for it, despite the C runtime issues.
>
> I don't see that in the thread... just Victor saying he isn't sure
> whether there might be other ABI incompatibilities lurking that he
> hasn't found yet. Did I miss something?

"I don't know" means we can't say the APIs are converged, which is a no.

I don't think you missed anything, but just read it through a different
filter.

> I'm mostly interested in this because of the possibility of converging
> the ABIs. If you think that the C runtime thing isn't a blocker for
> that, then that's useful information. Though obviously we still need
> to figure out whether there are any other blockers :-).
> [SNIP]
> Do you happen to have a list of places where the C API leaks details
> of the underlying CRT?
>
> (I'm mostly curious because whenever I've looked my conclusion was
> essentially: "Well....... I don't see any places that are *definitely*
> broken, so maybe mixing CRTs is fine? but I have zero confidence that
> I caught everything, so probably better to play it safe?". At least on
> py3 – I know the py2 C API was definitely broken if you mixed CRTs,
> because of the exposed FILE*.)

Not since the discussions about migrating to VS 2015, but a few off the
top of my head:
* locale
* file descriptors
* stream buffers
* thread locals
* exception [handler] state (yes, there are exceptions used within the
CRT, and they occasionally intentionally leak out past the C code)
* atexit handlers
* internal callbacks (mostly debug handlers, but since we're talking
about debugging...)

I'm pretty sure if I did some digging I'd be able to figure out which of
these come from vcruntime140.dll vs ucrtbase.dll, and then come up with
some far-too-clever linker options to make some of these more
consistent, but there's no complete solution other than making sure
you've got a complete debug or complete release build.

>> For the most part, disabling optimizations in your own extension but
>> using the non-debug ABI is sufficient, and if you're having to deal with
>> other people's packages then maybe you don't have any choice (though I
>> do know of people who have built debug versions of numpy before - turns
>> out Windows developers are often just as capable as non-Windows
>> developers when it comes to building things ;)
>
> I'm not sure why you think I was implying otherwise? I'm sorry if you
> thought I was attacking your users or something. I did say that I
> thought most users downloading the debug builds were probably confused
> about what they were actually getting, but I didn't mean because they
> were stupid Windows users, I meant because the debug builds are so
> confusing that even folks on the Python core team are confused about
> what they're actually getting.

"Our users", please :)

In my experience, Windows developers just treat debug and release builds
as part of the normal development process. The only confusion I've seen
has been related to CPython's not-quite-Windows-ish approach to debug
builds, and in practically every case it's been enough to explain
"release CPython uses a different CRT to your debug extension, but once
you align those it'll be fine".

I definitely *do not* want to force or encourage package developers to
release debug ABI versions of their prebuilt packages. But at the same
time I don't want to remove the benefits that debug builds currently
include.

Basically, I'm happy with the status quo, and the users I talk to are
happy with it. So I'd rather not worry about optimising debug builds for
speed or memory usage. (It's a question of direction more than anything
else, and until we get some official statement of direction then I'll
keep advocating a direction based on my experiences ;) )

Cheers,
Steve
_______________________________________________
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: https://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com

On Mon, 15 Apr 2019 12:50:00 +0200
Antoine Pitrou <solipsis@pitrou.net> wrote:
> On Thu, 11 Apr 2019 08:26:47 -0700
> Steve Dower <steve.dower@python.org> wrote:
> > On 10Apr2019 1917, Nathaniel Smith wrote:
> > > It sounds like --with-pydebug has accumulated a big grab bag of
> > > unrelated features, mostly stuff that was useful at some point for
> > > some CPython dev trying to debug CPython itself? It's clearly not
> > > designed with end users as the primary audience, given that no-one
> > > knows what it actually does and that it makes third-party extensions
> > > really awkward to run. If that's right then I think Victor's plan of
> > > to sort through what it's actually doing makes a lot of sense,
> > > especially if we can remove the ABI breaking stuff, since that causes
> > > a disproportionate amount of trouble.
> >
> > Does it really cause a "disproportionate" amount of trouble? It's
> > definitely not meant for anyone who isn't working on C code, whether in
> > CPython, an extension or a host application. If you want to use
> > third-party extensions and are not able to rebuild them, that's a very
> > good sign that you probably shouldn't be on the debug build at all.
>
> I can't really agree with that. There are third-party extensions that
> have non-trivial build requirements. The fact that you have to rebuild
> third-party dependencies is a strong deterrent against using pydebug
> builds even when they may be actually useful (for example when
> debugging an extension module of your own).

Oh, and as a datapoint, there are user requests for pydebug builds in
Anaconda and conda-forge:
https://github.com/ContinuumIO/anaconda-issues/issues/80
https://github.com/conda-forge/staged-recipes/issues/1593

The problem is, while it's technically relatively easy to build and
distribute a special build of Python, to make it useful implies also
building a whole separate distribution of Python libraries as well. I
suspect the latter is why those issues were never acted upon.

So, there's actual demand from people who would (probably) benefit from
it, but are blocked by burden of recompiling all dependencies.

Regards

Antoine.


_______________________________________________
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: https://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com

> The main question is if anyone ever used Py_TRACE_REFS? Does someone
> use sys.getobjects() or PYTHONDUMPREFS environment variable?

I used sys.getobjects() today to track down a memory leak in the
mypyc-compiled version of mypy.

We were leaking memory badly but no sign of the leak was showing up in
mypy's gc.get_objects() based profiler. Using a debug build and switching
to sys.getobjects() showed that we were badly leaking int objects. A quick
inspection of the values in question (large and random looking) suggested
we were leaking hash values, and that quickly pointed me to
https://github.com/mypyc/mypyc/pull/562.

I don't have any strong feelings about whether to keep it in the "default"
debug build, though. I was using a debug build that I built myself with
every debug feature that seemed potentially useful.

-sully

On Mon, Apr 15, 2019, 15:27 Michael Sullivan <sully@msully.net> wrote:

> > The main question is if anyone ever used Py_TRACE_REFS? Does someone
> > use sys.getobjects() or PYTHONDUMPREFS environment variable?
>
> I used sys.getobjects() today to track down a memory leak in the
> mypyc-compiled version of mypy.
>
> We were leaking memory badly but no sign of the leak was showing up in
> mypy's gc.get_objects() based profiler. Using a debug build and switching
> to sys.getobjects() showed that we were badly leaking int objects. A quick
> inspection of the values in question (large and random looking) suggested
> we were leaking hash values, and that quickly pointed me to
> https://github.com/mypyc/mypyc/pull/562.
>
> I don't have any strong feelings about whether to keep it in the "default"
> debug build, though. I was using a debug build that I built myself with
> every debug feature that seemed potentially useful.
>

This is mostly to satisfy my curiosity, so feel free to ignore: did you try
using address sanitizer or valgrind?

-n

>

On Mon, Apr 15, 2019 at 4:06 PM Nathaniel Smith <njs@pobox.com> wrote:

> On Mon, Apr 15, 2019, 15:27 Michael Sullivan <sully@msully.net> wrote:
>
>> > The main question is if anyone ever used Py_TRACE_REFS? Does someone
>> > use sys.getobjects() or PYTHONDUMPREFS environment variable?
>>
>> I used sys.getobjects() today to track down a memory leak in the
>> mypyc-compiled version of mypy.
>>
>> We were leaking memory badly but no sign of the leak was showing up in
>> mypy's gc.get_objects() based profiler. Using a debug build and switching
>> to sys.getobjects() showed that we were badly leaking int objects. A quick
>> inspection of the values in question (large and random looking) suggested
>> we were leaking hash values, and that quickly pointed me to
>> https://github.com/mypyc/mypyc/pull/562.
>>
>> I don't have any strong feelings about whether to keep it in the
>> "default" debug build, though. I was using a debug build that I built
>> myself with every debug feature that seemed potentially useful.
>>
>
> This is mostly to satisfy my curiosity, so feel free to ignore: did you
> try using address sanitizer or valgrind?
>
> I didn't, mostly because I assume that valgrind wouldn't play well with
cpython. (I've never used address sanitizer.)

I was curious, so I went back and tried it out.
It turned out to not seem to need that much fiddling to get to work. It
slows things down a *lot* and produced 17,000 "loss records", though, so
maybe I don't have it working right. At a glance the records did not shed
any light.

I'd definitely believe that valgrind is up to the task of debugging this,
but my initial take with it shed much less light than my sys.getobjects()
approach. (Though note that my sys.getobjects() approach was slotting it
into an existing python memory profiler we had hacked up, so...)

-sully


> -n
>
>>

Hi Michael,

Do you know the tracemalloc module? Did you try it? It works on a
regular Python (compiled in debug mode).

I would be curious to know if tracemalloc also allows you to track the
memory leak.

sys.getobjects() is just a list of objects. Do you have a tool written
on top of it to track memory leaks? If yes, how?

Victor

Le mar. 16 avr. 2019 à 00:28, Michael Sullivan <sully@msully.net> a écrit :
>
> > The main question is if anyone ever used Py_TRACE_REFS? Does someone
> > use sys.getobjects() or PYTHONDUMPREFS environment variable?
>
> I used sys.getobjects() today to track down a memory leak in the mypyc-compiled version of mypy.
>
> We were leaking memory badly but no sign of the leak was showing up in mypy's gc.get_objects() based profiler. Using a debug build and switching to sys.getobjects() showed that we were badly leaking int objects. A quick inspection of the values in question (large and random looking) suggested we were leaking hash values, and that quickly pointed me to https://github.com/mypyc/mypyc/pull/562.
>
> I don't have any strong feelings about whether to keep it in the "default" debug build, though. I was using a debug build that I built myself with every debug feature that seemed potentially useful.
>
> -sully
> _______________________________________________
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: https://mail.python.org/mailman/options/python-dev/vstinner%40redhat.com



--
Night gathers, and now my watch begins. It shall not end until my death.
_______________________________________________
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: https://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com

On Mon, Apr 15, 2019 at 8:58 PM Michael Sullivan <sully@msully.net> wrote:
>
> On Mon, Apr 15, 2019 at 4:06 PM Nathaniel Smith <njs@pobox.com> wrote:
>>
>> On Mon, Apr 15, 2019, 15:27 Michael Sullivan <sully@msully.net> wrote:
>>>
>>> > The main question is if anyone ever used Py_TRACE_REFS? Does someone
>>> > use sys.getobjects() or PYTHONDUMPREFS environment variable?
>>>
>>> I used sys.getobjects() today to track down a memory leak in the mypyc-compiled version of mypy.
>>>
>>> We were leaking memory badly but no sign of the leak was showing up in mypy's gc.get_objects() based profiler. Using a debug build and switching to sys.getobjects() showed that we were badly leaking int objects. A quick inspection of the values in question (large and random looking) suggested we were leaking hash values, and that quickly pointed me to https://github.com/mypyc/mypyc/pull/562.
>>>
>>> I don't have any strong feelings about whether to keep it in the "default" debug build, though. I was using a debug build that I built myself with every debug feature that seemed potentially useful.
>>
>>
>> This is mostly to satisfy my curiosity, so feel free to ignore: did you try using address sanitizer or valgrind?
>>
> I didn't, mostly because I assume that valgrind wouldn't play well with cpython. (I've never used address sanitizer.)
>
> I was curious, so I went back and tried it out.
> It turned out to not seem to need that much fiddling to get to work. It slows things down a *lot* and produced 17,000 "loss records", though, so maybe I don't have it working right. At a glance the records did not shed any light.
>
> I'd definitely believe that valgrind is up to the task of debugging this, but my initial take with it shed much less light than my sys.getobjects() approach. (Though note that my sys.getobjects() approach was slotting it into an existing python memory profiler we had hacked up, so...)

valgrind on CPython is definitely a bit fiddly – if you need it again
you might check out Misc/README.valgrind.

Supposedly memory sanitizer is just './configure
--with-memory-sanitizer', but I haven't tried it either :-)

-n

--
Nathaniel J. Smith -- https://vorpus.org
_______________________________________________
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: https://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com

Since Python 3.6, you can use PYTHONMALLOC=malloc for Valgrind: it
avoids false alarms produced by the pymalloc allocator.

Victor

Le mar. 16 avr. 2019 à 12:09, Nathaniel Smith <njs@pobox.com> a écrit :
>
> On Mon, Apr 15, 2019 at 8:58 PM Michael Sullivan <sully@msully.net> wrote:
> >
> > On Mon, Apr 15, 2019 at 4:06 PM Nathaniel Smith <njs@pobox.com> wrote:
> >>
> >> On Mon, Apr 15, 2019, 15:27 Michael Sullivan <sully@msully.net> wrote:
> >>>
> >>> > The main question is if anyone ever used Py_TRACE_REFS? Does someone
> >>> > use sys.getobjects() or PYTHONDUMPREFS environment variable?
> >>>
> >>> I used sys.getobjects() today to track down a memory leak in the mypyc-compiled version of mypy.
> >>>
> >>> We were leaking memory badly but no sign of the leak was showing up in mypy's gc.get_objects() based profiler. Using a debug build and switching to sys.getobjects() showed that we were badly leaking int objects. A quick inspection of the values in question (large and random looking) suggested we were leaking hash values, and that quickly pointed me to https://github.com/mypyc/mypyc/pull/562.
> >>>
> >>> I don't have any strong feelings about whether to keep it in the "default" debug build, though. I was using a debug build that I built myself with every debug feature that seemed potentially useful.
> >>
> >>
> >> This is mostly to satisfy my curiosity, so feel free to ignore: did you try using address sanitizer or valgrind?
> >>
> > I didn't, mostly because I assume that valgrind wouldn't play well with cpython. (I've never used address sanitizer.)
> >
> > I was curious, so I went back and tried it out.
> > It turned out to not seem to need that much fiddling to get to work. It slows things down a *lot* and produced 17,000 "loss records", though, so maybe I don't have it working right. At a glance the records did not shed any light.
> >
> > I'd definitely believe that valgrind is up to the task of debugging this, but my initial take with it shed much less light than my sys.getobjects() approach. (Though note that my sys.getobjects() approach was slotting it into an existing python memory profiler we had hacked up, so...)
>
> valgrind on CPython is definitely a bit fiddly – if you need it again
> you might check out Misc/README.valgrind.
>
> Supposedly memory sanitizer is just './configure
> --with-memory-sanitizer', but I haven't tried it either :-)
>
> -n
>
> --
> Nathaniel J. Smith -- https://vorpus.org
> _______________________________________________
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: https://mail.python.org/mailman/options/python-dev/vstinner%40redhat.com



--
Night gathers, and now my watch begins. It shall not end until my death.
_______________________________________________
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: https://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com

On Tue, Apr 16, 2019 at 2:11 AM Victor Stinner <vstinner@redhat.com> wrote:

> Hi Michael,
>
> Do you know the tracemalloc module? Did you try it? It works on a
> regular Python (compiled in debug mode).
>
> I would be curious to know if tracemalloc also allows you to track the
> memory leak.
>
> Playing around with it a little it does not seem super helpful here
(unless I am missing something): it tracks the allocations based on the
python call stack, which doesn't help here, in a C extension module
generated from python code.

Though, in the the mypyc case, we could implement a debug option for
creating dummy frames so that we always have a useful call stack. That
seems like less of an option for actual hand-written extension modules,
though. (Though on the flip side, the python call stacks might be more
useful there.)

sys.getobjects() is just a list of objects. Do you have a tool written
> on top of it to track memory leaks? If yes, how?
>
> Not really.

We have a very simple memory profiler built on top of gc.get_objects() that
just reports how many of different types of objects there are and how much
memory they are using:
https://github.com/python/mypy/blob/master/mypy/memprofile.py.
I swapped out gc.get_objects() for sys.getobjects(), observed that we were
leaking int objects, and inspected the live int objects, which gave a
pretty good clue where the leak was.


> Victor
>
> Le mar. 16 avr. 2019 à 00:28, Michael Sullivan <sully@msully.net> a écrit
> :
> >
> > > The main question is if anyone ever used Py_TRACE_REFS? Does someone
> > > use sys.getobjects() or PYTHONDUMPREFS environment variable?
> >
> > I used sys.getobjects() today to track down a memory leak in the
> mypyc-compiled version of mypy.
> >
> > We were leaking memory badly but no sign of the leak was showing up in
> mypy's gc.get_objects() based profiler. Using a debug build and switching
> to sys.getobjects() showed that we were badly leaking int objects. A quick
> inspection of the values in question (large and random looking) suggested
> we were leaking hash values, and that quickly pointed me to
> https://github.com/mypyc/mypyc/pull/562.
> >
> > I don't have any strong feelings about whether to keep it in the
> "default" debug build, though. I was using a debug build that I built
> myself with every debug feature that seemed potentially useful.
> >
> > -sully
> > _______________________________________________
> > Python-Dev mailing list
> > Python-Dev@python.org
> > https://mail.python.org/mailman/listinfo/python-dev
> > Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/vstinner%40redhat.com
>
>
>
> --
> Night gathers, and now my watch begins. It shall not end until my death.
>