Mailing List Archive

Making 'compiled' modules work with multiple python versions on Linux
I have a python module that includes some C++ code that links with the
Python C API

I have now modified the c++ code so that it only uses the Limited API,
and linked with python3.lib instead of python311.lib.

I can now use that python module with different python versions on Windows

But on Linux, it seems that linking to libpython3.so instead of
libpython3.11.so.1.0 does not have the same effect, and results in
many unresolved python symbols at link time

Is this functionality only available on Windows?
--
https://mail.python.org/mailman/listinfo/python-list
Re: Making 'compiled' modules work with multiple python versions on Linux [ In reply to ]
> On 28 Mar 2024, at 16:13, Olivier B. via Python-list <python-list@python.org> wrote:
>
> But on Linux, it seems that linking to libpython3.so instead of
> libpython3.11.so.1.0 does not have the same effect, and results in
> many unresolved python symbols at link time
>
> Is this functionality only available on Windows?

Python limited API works on linux, but you do not link against the .so on linux I recall.

You will have missed that libpython3.so is a symlink to libpython3.11.so.10.

Windows build practices do not translate one-to-one to linux, or macOS.

Barry


--
https://mail.python.org/mailman/listinfo/python-list
Re: Making 'compiled' modules work with multiple python versions on Linux [ In reply to ]
It sounds weird that symbols from Limited API are _missing_ (I'd
expect them to be there no matter what library version you link with).
But, I haven't done this myself, so, what do I know? It would help
though to see the actual error.

That aside: why do you want to do this? One side effect of doing what
you want will be the "weird" name of your wheel archive. Weird in a
sense that virtually nobody does that. And when virtually nobody does
something, you are almost guaranteed to be the first to find bugs, and
then be the one whose bug reports are shoved into the backlog and
never looked at again.

You, kind of, are already walking into the world of pain trying to
make Python binary packages, and then you also want them to be
cross-platform, and then you want them to be usable by different
versions of Python... Unless it's for your own amusement, I'd just
have a package per version of Python. Maintenance-wise it's going to
be a lot easier.

On Fri, Mar 29, 2024 at 10:13?AM Barry via Python-list
<python-list@python.org> wrote:
>
>
>
> > On 28 Mar 2024, at 16:13, Olivier B. via Python-list <python-list@python.org> wrote:
> >
> > But on Linux, it seems that linking to libpython3.so instead of
> > libpython3.11.so.1.0 does not have the same effect, and results in
> > many unresolved python symbols at link time
> >
> > Is this functionality only available on Windows?
>
> Python limited API works on linux, but you do not link against the .so on linux I recall.
>
> You will have missed that libpython3.so is a symlink to libpython3.11.so.10.
>
> Windows build practices do not translate one-to-one to linux, or macOS.
>
> Barry
>
>
> --
> https://mail.python.org/mailman/listinfo/python-list
--
https://mail.python.org/mailman/listinfo/python-list
Re: Making 'compiled' modules work with multiple python versions on Linux [ In reply to ]
It is not a symlink on my system, where i built python myself, but a
15KB so file. But it seems to lack lots of python symbols.

Maybe what i should do is actually make libpython.so a physical copy
of libpyton311.so before linking to it, so now on any system the
module would look to load libpython.so, which could be pointing to any
version. I'll try that next

Le ven. 29 mars 2024 à 10:10, Barry <barry@barrys-emacs.org> a écrit :
>
>
>
> > On 28 Mar 2024, at 16:13, Olivier B. via Python-list <python-list@python.org> wrote:
> >
> > But on Linux, it seems that linking to libpython3.so instead of
> > libpython3.11.so.1.0 does not have the same effect, and results in
> > many unresolved python symbols at link time
> >
> > Is this functionality only available on Windows?
>
> Python limited API works on linux, but you do not link against the .so on linux I recall.
>
> You will have missed that libpython3.so is a symlink to libpython3.11.so.10.
>
> Windows build practices do not translate one-to-one to linux, or macOS.
>
> Barry
>
>
--
https://mail.python.org/mailman/listinfo/python-list
Re: Making 'compiled' modules work with multiple python versions on Linux [ In reply to ]
> On 29 Mar 2024, at 16:09, Olivier B. <perso.olivier.barthelemy@gmail.com> wrote:
>
> It is not a symlink on my system, where i built python myself, but a
> 15KB so file. But it seems to lack lots of python symbols.
>
> Maybe what i should do is actually make libpython.so a physical copy
> of libpyton311.so before linking to it, so now on any system the
> module would look to load libpython.so, which could be pointing to any
> version. I'll try that next

You do not link against the .so at all. All the symbols you need are defined in the
python process that loads the extension. Try without the -lpython and it should
just work.

Barry


>
> Le ven. 29 mars 2024 à 10:10, Barry <barry@barrys-emacs.org> a écrit :
>>
>>
>>
>>> On 28 Mar 2024, at 16:13, Olivier B. via Python-list <python-list@python.org> wrote:
>>>
>>> But on Linux, it seems that linking to libpython3.so instead of
>>> libpython3.11.so.1.0 does not have the same effect, and results in
>>> many unresolved python symbols at link time
>>>
>>> Is this functionality only available on Windows?
>>
>> Python limited API works on linux, but you do not link against the .so on linux I recall.
>>
>> You will have missed that libpython3.so is a symlink to libpython3.11.so.10.
>>
>> Windows build practices do not translate one-to-one to linux, or macOS.
>>
>> Barry
>>
>>
>

--
https://mail.python.org/mailman/listinfo/python-list
Re: Making 'compiled' modules work with multiple python versions on Linux [ In reply to ]
> On 1 Apr 2024, at 18:14, Left Right via Python-list <python-list@python.org> wrote:
>
> It sounds weird that symbols from Limited API are _missing_ (I'd
> expect them to be there no matter what library version you link with).

You have to specify the version of the limited API that you want to use.
Each release adds more symbols to the limited API.
So if you compile against 3.10 version it works with 3.10, 3.11, 3.12 etc,
but not with 3.9.

My pycxx project has test code that I use to verify pycxx’s use of python API.
You could check out the source and run my tests to see how things work.
It is on https://sourceforge.net/projects/cxx/

Barry


--
https://mail.python.org/mailman/listinfo/python-list