Mailing List Archive

Marco Sulla wrote:

> Error LNK2001: unresolved external symbol PyErr_SetObject
>
> and so on.
>
> I post the part of my setup.py about the C Extension:
>
> extra_compile_args = ["-DPY_SSIZE_T_CLEAN", "-DPy_BUILD_CORE"]

Shouldn't this be "-DPy_BUILD_CORE_MODULE"?


--
--gv
--
https://mail.python.org/mailman/listinfo/python-list

On Fri, 12 Nov 2021 at 15:55, Gisle Vanem <gisle.vanem@gmail.com> wrote:
> Marco Sulla wrote:
> > Error LNK2001: unresolved external symbol PyErr_SetObject
> >
> > and so on.
> >
> > I post the part of my setup.py about the C Extension:
> >
> > extra_compile_args = ["-DPY_SSIZE_T_CLEAN", "-DPy_BUILD_CORE"]
>
> Shouldn't this be "-DPy_BUILD_CORE_MODULE"?

I tried it, but now I get three

error C2099: initializer is not a constant

when I try to compile dictobject.c. Yes, my extension needs dictobject.
--
https://mail.python.org/mailman/listinfo/python-list

On Sat, Nov 13, 2021 at 3:28 AM Marco Sulla
<Marco.Sulla.Python@gmail.com> wrote:
>
> On Fri, 12 Nov 2021 at 15:55, Gisle Vanem <gisle.vanem@gmail.com> wrote:
> > Marco Sulla wrote:
> > > Error LNK2001: unresolved external symbol PyErr_SetObject
> > >
> > > and so on.
> > >
> > > I post the part of my setup.py about the C Extension:
> > >
> > > extra_compile_args = ["-DPY_SSIZE_T_CLEAN", "-DPy_BUILD_CORE"]
> >
> > Shouldn't this be "-DPy_BUILD_CORE_MODULE"?
>
> I tried it, but now I get three
>
> error C2099: initializer is not a constant
>
> when I try to compile dictobject.c. Yes, my extension needs dictobject.

Are you sure that you really need Py_BUILD_CORE? Here's what the code
has to say about it:

https://github.com/python/cpython/blob/main/Include/pyport.h#L17

Py_BUILD_CORE: Build Python core. Give access to Python internals, but
should not be used by third-party modules.

If your extension truly needs to reach into the internals of CPython,
then you're going to have to figure a lot of things out for yourself,
including how to make sure it works on every platform. A more
productive approach would be to stick to the actual API:

https://docs.python.org/3/c-api/stable.html

What is it that doesn't work without Py_BUILD_CORE?

ChrisA
--
https://mail.python.org/mailman/listinfo/python-list

Chris? Maybe I'm dreaming X-D

On Fri, 12 Nov 2021 at 17:38, Chris Angelico <rosuav@gmail.com> wrote:
> Are you sure that you really need Py_BUILD_CORE?

Yes, because I need the internal functions of `dict`. So I need to
compile also dictobject.c and include it. So I need that flag.

This is the code:

https://github.com/Marco-Sulla/python-frozendict.git

On Linux and MacOS it works like a charme. On Windows, it seems it does not find
python3.lib. I also added its path to the PATH variable
--
https://mail.python.org/mailman/listinfo/python-list

On Sat, Nov 13, 2021 at 7:01 AM Marco Sulla
<Marco.Sulla.Python@gmail.com> wrote:
> On Fri, 12 Nov 2021 at 17:38, Chris Angelico <rosuav@gmail.com> wrote:
> > Are you sure that you really need Py_BUILD_CORE?
>
> Yes, because I need the internal functions of `dict`. So I need to
> compile also dictobject.c and include it. So I need that flag.
>
> This is the code:
>
> https://github.com/Marco-Sulla/python-frozendict.git
>

Ah, gotcha.

Unfortunately that does mean you're delving deep into internals, and a
lot of stuff that isn't designed for extensions to use. So my best
recommendation is: dig even deeper into internals, and duplicate how
the core is doing things (maybe including another header or
something). It may be that, by declaring Py_BUILD_CORE, you're getting
a macro version of that instead of the normal exported function.

ChrisA
--
https://mail.python.org/mailman/listinfo/python-list

On Fri, 12 Nov 2021 at 21:09, Chris Angelico <rosuav@gmail.com> wrote:
>
> On Sat, Nov 13, 2021 at 7:01 AM Marco Sulla
> <Marco.Sulla.Python@gmail.com> wrote:
> > On Fri, 12 Nov 2021 at 17:38, Chris Angelico <rosuav@gmail.com> wrote:
> > > Are you sure that you really need Py_BUILD_CORE?
> >
> > Yes, because I need the internal functions of `dict`. So I need to
> > compile also dictobject.c and include it. So I need that flag.
> >
> > This is the code:
> >
> > https://github.com/Marco-Sulla/python-frozendict.git
> >
>
> Ah, gotcha.
>
> Unfortunately that does mean you're delving deep into internals, and a
> lot of stuff that isn't designed for extensions to use. So my best
> recommendation is: dig even deeper into internals, and duplicate how
> the core is doing things (maybe including another header or
> something). It may be that, by declaring Py_BUILD_CORE, you're getting
> a macro version of that instead of the normal exported function.

I've not understood what I have to do in practice.... but anyway, as I
said, Py_BUILD_CORE works on Linux and MacOS. And it works also on
Windows. Indeed dictobject.c is compiled. The only problem is in the
linking phase, when the two objects should be linked in one library,
_the_ library. It seems that on Windows it doesn't find python3.lib,
even if I put it in the path. So I get the `unresolved external link`
errors.
--
https://mail.python.org/mailman/listinfo/python-list

> On 12 Nov 2021, at 22:53, Marco Sulla <Marco.Sulla.Python@gmail.com> wrote:
>
> It seems that on Windows it doesn't find python3.lib,
> even if I put it in the path. So I get the `unresolved external link`
> errors.

I think you need the python310.lib (not sure of file name) to get to the internal symbols.

You can use the objdump(?) utility to check that the symbols are in the lib.

Barry


--
https://mail.python.org/mailman/listinfo/python-list

> On 13 Nov 2021, at 09:00, Barry <barry@barrys-emacs.org> wrote:
>
>
>
>> On 12 Nov 2021, at 22:53, Marco Sulla <Marco.Sulla.Python@gmail.com> wrote:
>>
>> It seems that on Windows it doesn't find python3.lib,
>> even if I put it in the path. So I get the `unresolved external link`
>> errors.
>
> I think you need the python310.lib (not sure of file name) to get to the internal symbols.

Another thing that you will need to check is that the symbols you are after have been
exposed in the DLL at all. Being external in the source is not enough they also have to
listed in the .DLL's def file ( is that the right term?) as well.

If its not clear yet, you are going to have to read a lot or source code and understand
the tool chain used on Windows to solve this.


>
> You can use the objdump(?) utility to check that the symbols are in the lib.
>
> Barry

Barry

--
https://mail.python.org/mailman/listinfo/python-list

..... Sorry, the problem is I downloaded the 32 bit version of VS
compiler and 64 bit version of Python......

On Sat, 13 Nov 2021 at 11:10, Barry Scott <barry@barrys-emacs.org> wrote:
>
>
>
> > On 13 Nov 2021, at 09:00, Barry <barry@barrys-emacs.org> wrote:
> >
> >
> >
> >> On 12 Nov 2021, at 22:53, Marco Sulla <Marco.Sulla.Python@gmail.com> wrote:
> >>
> >> It seems that on Windows it doesn't find python3.lib,
> >> even if I put it in the path. So I get the `unresolved external link`
> >> errors.
> >
> > I think you need the python310.lib (not sure of file name) to get to the internal symbols.
>
> Another thing that you will need to check is that the symbols you are after have been
> exposed in the DLL at all. Being external in the source is not enough they also have to
> listed in the .DLL's def file ( is that the right term?) as well.
>
> If its not clear yet, you are going to have to read a lot or source code and understand
> the tool chain used on Windows to solve this.
>
>
> >
> > You can use the objdump(?) utility to check that the symbols are in the lib.
> >
> > Barry
>
> Barry
>
--
https://mail.python.org/mailman/listinfo/python-list

Okay, now the problem seems to be another: I get the same "unresolved
external link" errors, but only for internal functions.

This seems quite normal. The public .lib does not expose the internals
of Python.
The strange fact is: why can I compile it on Linux and MacOS? Their
external libraries expose the internal functions?

Anyway, is there a way to compile Python on Windows in such a way that
I get a shared library that exposes all the functions?

On Sat, 13 Nov 2021 at 12:17, Marco Sulla <Marco.Sulla.Python@gmail.com> wrote:
>
> ..... Sorry, the problem is I downloaded the 32 bit version of VS
> compiler and 64 bit version of Python......
>
> On Sat, 13 Nov 2021 at 11:10, Barry Scott <barry@barrys-emacs.org> wrote:
> >
> >
> >
> > > On 13 Nov 2021, at 09:00, Barry <barry@barrys-emacs.org> wrote:
> > >
> > >
> > >
> > >> On 12 Nov 2021, at 22:53, Marco Sulla <Marco.Sulla.Python@gmail.com> wrote:
> > >>
> > >> It seems that on Windows it doesn't find python3.lib,
> > >> even if I put it in the path. So I get the `unresolved external link`
> > >> errors.
> > >
> > > I think you need the python310.lib (not sure of file name) to get to the internal symbols.
> >
> > Another thing that you will need to check is that the symbols you are after have been
> > exposed in the DLL at all. Being external in the source is not enough they also have to
> > listed in the .DLL's def file ( is that the right term?) as well.
> >
> > If its not clear yet, you are going to have to read a lot or source code and understand
> > the tool chain used on Windows to solve this.
> >
> >
> > >
> > > You can use the objdump(?) utility to check that the symbols are in the lib.
> > >
> > > Barry
> >
> > Barry
> >
--
https://mail.python.org/mailman/listinfo/python-list

Sorry iPad sent the message before it was complete...

> On 14 Nov 2021, at 10:38, Marco Sulla <Marco.Sulla.Python@gmail.com> wrote:
>
> Okay, now the problem seems to be another: I get the same "unresolved
> external link" errors, but only for internal functions.
>
> This seems quite normal. The public .lib does not expose the internals
> of Python.
> The strange fact is: why can I compile it on Linux and MacOS? Their
> external libraries expose the internal functions?

Windows is not Linux is not macOS,
The toolchain on each OS has its own strengths, weaknesses and quirks.

On Windows DLLs only allow access to the symbols that are explicitly listed to be access.
On macOS .dynlib and Unix .so its being extern that does this.

>
> Anyway, is there a way to compile Python on Windows in such a way that
> I get a shared library that exposes all the functions?

Yes you can do your own build of python that exposes the symbols you want.
But that build will be private to you and will not allow others to use your work
(on the assumption that they will not use your private build of python).

Maybe you could copy the code that you want and add it to your code?
Change any conflicting symbols of course.

Barry

>
> On Sat, 13 Nov 2021 at 12:17, Marco Sulla <Marco.Sulla.Python@gmail.com> wrote:
>>
>> ..... Sorry, the problem is I downloaded the 32 bit version of VS
>> compiler and 64 bit version of Python......
>>
>> On Sat, 13 Nov 2021 at 11:10, Barry Scott <barry@barrys-emacs.org> wrote:
>>>
>>>
>>>
>>>> On 13 Nov 2021, at 09:00, Barry <barry@barrys-emacs.org> wrote:
>>>>
>>>>
>>>>
>>>>> On 12 Nov 2021, at 22:53, Marco Sulla <Marco.Sulla.Python@gmail.com> wrote:
>>>>>
>>>>> It seems that on Windows it doesn't find python3.lib,
>>>>> even if I put it in the path. So I get the `unresolved external link`
>>>>> errors.
>>>>
>>>> I think you need the python310.lib (not sure of file name) to get to the internal symbols.
>>>
>>> Another thing that you will need to check is that the symbols you are after have been
>>> exposed in the DLL at all. Being external in the source is not enough they also have to
>>> listed in the .DLL's def file ( is that the right term?) as well.
>>>
>>> If its not clear yet, you are going to have to read a lot or source code and understand
>>> the tool chain used on Windows to solve this.
>>>
>>>
>>>>
>>>> You can use the objdump(?) utility to check that the symbols are in the lib.
>>>>
>>>> Barry
>>>
>>> Barry
>>>
>

--
https://mail.python.org/mailman/listinfo/python-list

On Sun, 14 Nov 2021 at 16:42, Barry Scott <barry@barrys-emacs.org> wrote:
>
> Sorry iPad sent the message before it was complete...
>
> > On 14 Nov 2021, at 10:38, Marco Sulla <Marco.Sulla.Python@gmail.com> wrote:
> >
> > Okay, now the problem seems to be another: I get the same "unresolved
> > external link" errors, but only for internal functions.
> >
> > This seems quite normal. The public .lib does not expose the internals
> > of Python.
> > The strange fact is: why can I compile it on Linux and MacOS? Their
> > external libraries expose the internal functions?
>
> Windows is not Linux is not macOS,
> The toolchain on each OS has its own strengths, weaknesses and quirks.
>
> On Windows DLLs only allow access to the symbols that are explicitly listed to be access.

Where are those symbols listed?

> On macOS .dynlib and Unix .so its being extern that does this.

And extern is the default. I understand now.

> Maybe you could copy the code that you want and add it to your code?
> Change any conflicting symbols of course.

It's quite hard. I have to compile dictobject.c, which needs a lot of
internal functions. And I suppose that every internal function may
require 1 or more other internal functions.

I have other two other solutions:
* compile a whole python DLL with the symbols I need and link against
it. I have to put this DLL in my code, which is ugly.
* drop the support of the C Extension for Windows users and make for
them the slow, pure py version only.

Since my interest in Windows now is near to zero, I think I'll opt for
the third for now.
--
https://mail.python.org/mailman/listinfo/python-list

On 11/14/21, Marco Sulla <Marco.Sulla.Python@gmail.com> wrote:
> On Sun, 14 Nov 2021 at 16:42, Barry Scott <barry@barrys-emacs.org> wrote:
>
>> On macOS .dynlib and Unix .so its being extern that does this.
>
> And extern is the default. I understand now.

Per Include/exports.h and Include/pyport.h, Python should be built in
Unix with "default" visibility (per global/local binding) for the API
PyAPI_FUNC(RTYPE) and PyAPI_DATA(RTYPE) symbols. Everything else with
global binding will be hidden via the "-fvisibility=hidden" compiler
option that's configured in the makefile. For example:

$ readelf -s Objects/dictobject.o | grep HIDDEN | cut -b 40-
GLOBAL HIDDEN 4 _pydict_global_version
GLOBAL HIDDEN 1 _PyDict_ClearFreeList
GLOBAL HIDDEN 1 _PyDict_Fini
GLOBAL HIDDEN 1 _PyDictKeys_StringLookup
GLOBAL HIDDEN 9 _Py_dict_lookup
GLOBAL HIDDEN 1 _PyDict_GetItemHint
GLOBAL HIDDEN 1 _PyDict_LoadGlobal
GLOBAL HIDDEN 1 _PyDict_Pop_KnownHash
GLOBAL HIDDEN 1 _PyDict_FromKeys
GLOBAL HIDDEN 1 _PyDict_KeysSize
GLOBAL HIDDEN 1 _PyDict_NewKeysForClass
GLOBAL HIDDEN 1 _PyObject_InitializeDict
GLOBAL HIDDEN 1 _PyObject_MakeDictFromIns
GLOBAL HIDDEN 1 _PyObject_StoreInstanceAt
GLOBAL HIDDEN 1 _PyObject_GetInstanceAttr
GLOBAL HIDDEN 1 _PyObject_IsInstanceDictE
GLOBAL HIDDEN 1 _PyObject_VisitInstanceAt
GLOBAL HIDDEN 1 _PyObject_ClearInstanceAt
GLOBAL HIDDEN 1 _PyObject_FreeInstanceAtt
GLOBAL HIDDEN 1 _PyObjectDict_SetItem
GLOBAL HIDDEN 1 _PyDictKeys_DecRef
GLOBAL HIDDEN 1 _PyDictKeys_GetVersionFor

These hidden symbols get linked in the executable or shared-object
with local binding. For example:

$ readelf -s python | grep _PyDict_FromKeys | cut -b 40-
LOCAL DEFAULT 16 _PyDict_FromKeys

I suggest testing your project with Python built as a shared library,
i.e. --enable-shared. The local binding on internal symbols may be a
problem in this case.
--
https://mail.python.org/mailman/listinfo/python-list