Greetings,
I'm working on embedding a Python interpreter into a C++ application. My
embedding example program is here, largely taken from Python docs:
https://gist.github.com/jmccardle/f3f19d3753ae023aa52b927f0d181c43
I'm simply not interested in writing in Lua, so regardless of any
particular downsides like `sys` in the standard library or performance
issues, I'm committed to Python itself as what I want to hack in. This
is for fun.
I started by compiling Python:
`./configure --enable-shared --enable-optimizations`
Then I can compile my example embedded program:
`g++ -I Python-3.11.1/Include -I Python-3.11.1 -L Python-3.11.1 -pthread
scripting_engine.cpp libpython3.11.a -o scripteng -lm -ldl -lutil`
This is working not so bad! I can control what C++ functionality is
exposed and I seemingly don't need anything but the Python shared object
to execute. But the finer details of making this work truly correctly
are eluding me.
1) To get the compiled Python to run independently, I have to hack
LD_LIBRARY_PATH to get it to execute. `LD_LIBRARY_PATH=./Python-3.11.1
./Python-3.11.1/python` . Even when trying to execute from the same
directory as the binary & executable, I get an error, `/python: error
while loading shared libraries: libpython3.11.so.1.0: cannot open shared
object file: No such file or directory`.
2) When running the C++ program that embeds Python, I see these messages
after initializing:
`Could not find platform independent libraries <prefix>
Could not find platform dependent libraries <exec_prefix>`
This is seemingly connected to some issues regarding libraries: When I
run the Python interpreter directly, I can get some of the way through
the process of creating a virtual environment, but it doesn't seem to
leave me with a working pip:
`$ LD_LIBRARY_PATH=./Python-3.11.1 ./Python-3.11.1/python
>>> import venv
>>> venv.create("./venv", with_pip=True)
subprocess.CalledProcessError: Command
'['/home/john/Development/7DRL/cpp_embedded_python/venv/bin/python',
'-m', 'ensurepip', '--upgrade', '--default-pip']' returned non-zero exit
status 127.`
Meanwhile, if I try to run a script from the C++ program that includes
`import venv`, I get a traceback about a platform library:
`Traceback (most recent call last):
File "engine_user.py", line 7, in <module>
import venv
File
"/home/john/Development/7DRL/cpp_embedded_python/Python-3.11.1/Lib/venv/__init__.py",
line 10, in <module>
import subprocess
File
"/home/john/Development/7DRL/cpp_embedded_python/Python-3.11.1/Lib/subprocess.py",
line 104, in <module>
from _posixsubprocess import fork_exec as _fork_exec
ModuleNotFoundError: No module named '_posixsubprocess'
`
3) I'm not sure I even need to be statically linking the interpreter.
My desired end state is this:
* Deploy a C++ program that doesn't rely on a system Python. I'm not
sure if I need just the shared object / DLLs, or a Python executable in
a subdirectory - I'd like to "do it the right way".
* C++ program can run a script to create a virtual environment, which
the embedded Python environment will use. Users can activate the venv
like any other Python environment and install packages with pip.
* ideally, some sort of "inside-out" runnable mode, where the API
exposed by the C++ executable is available in that venv, so that I can
test a script in Thonny or other IDE. I think I'd do this by providing a
separate test-mode library in the venv, and when C++ executes
`PyImport_AppendInittab("scriptable", &PyInit_scriptable);` then the
module of the same name should be overwritten with the C++ program's
functionality.
I've been through the embedded programming docs a bit, and they seem
quite good as a reference, but I don't know what I'm doing well enough
to solve my problems using them. Thanks for reading.
My ultimate goal is to expose features written in C++ for a game engine
using SFML, and run .py files in a subdirectory to generate maps,
control NPC dialogue and actions, etc. I'm hoping to have something
usable for this year's 7DRL, which starts March 4th. I'd like to spend
the time until then getting this engine working smoothly and porting it
to Windows, so I can focus exclusively on "game content" for the
timeboxed 7-day portion.
Kind Regards,
-John McCardle
--
https://mail.python.org/mailman/listinfo/python-list
I'm working on embedding a Python interpreter into a C++ application. My
embedding example program is here, largely taken from Python docs:
https://gist.github.com/jmccardle/f3f19d3753ae023aa52b927f0d181c43
I'm simply not interested in writing in Lua, so regardless of any
particular downsides like `sys` in the standard library or performance
issues, I'm committed to Python itself as what I want to hack in. This
is for fun.
I started by compiling Python:
`./configure --enable-shared --enable-optimizations`
Then I can compile my example embedded program:
`g++ -I Python-3.11.1/Include -I Python-3.11.1 -L Python-3.11.1 -pthread
scripting_engine.cpp libpython3.11.a -o scripteng -lm -ldl -lutil`
This is working not so bad! I can control what C++ functionality is
exposed and I seemingly don't need anything but the Python shared object
to execute. But the finer details of making this work truly correctly
are eluding me.
1) To get the compiled Python to run independently, I have to hack
LD_LIBRARY_PATH to get it to execute. `LD_LIBRARY_PATH=./Python-3.11.1
./Python-3.11.1/python` . Even when trying to execute from the same
directory as the binary & executable, I get an error, `/python: error
while loading shared libraries: libpython3.11.so.1.0: cannot open shared
object file: No such file or directory`.
2) When running the C++ program that embeds Python, I see these messages
after initializing:
`Could not find platform independent libraries <prefix>
Could not find platform dependent libraries <exec_prefix>`
This is seemingly connected to some issues regarding libraries: When I
run the Python interpreter directly, I can get some of the way through
the process of creating a virtual environment, but it doesn't seem to
leave me with a working pip:
`$ LD_LIBRARY_PATH=./Python-3.11.1 ./Python-3.11.1/python
>>> import venv
>>> venv.create("./venv", with_pip=True)
subprocess.CalledProcessError: Command
'['/home/john/Development/7DRL/cpp_embedded_python/venv/bin/python',
'-m', 'ensurepip', '--upgrade', '--default-pip']' returned non-zero exit
status 127.`
Meanwhile, if I try to run a script from the C++ program that includes
`import venv`, I get a traceback about a platform library:
`Traceback (most recent call last):
File "engine_user.py", line 7, in <module>
import venv
File
"/home/john/Development/7DRL/cpp_embedded_python/Python-3.11.1/Lib/venv/__init__.py",
line 10, in <module>
import subprocess
File
"/home/john/Development/7DRL/cpp_embedded_python/Python-3.11.1/Lib/subprocess.py",
line 104, in <module>
from _posixsubprocess import fork_exec as _fork_exec
ModuleNotFoundError: No module named '_posixsubprocess'
`
3) I'm not sure I even need to be statically linking the interpreter.
My desired end state is this:
* Deploy a C++ program that doesn't rely on a system Python. I'm not
sure if I need just the shared object / DLLs, or a Python executable in
a subdirectory - I'd like to "do it the right way".
* C++ program can run a script to create a virtual environment, which
the embedded Python environment will use. Users can activate the venv
like any other Python environment and install packages with pip.
* ideally, some sort of "inside-out" runnable mode, where the API
exposed by the C++ executable is available in that venv, so that I can
test a script in Thonny or other IDE. I think I'd do this by providing a
separate test-mode library in the venv, and when C++ executes
`PyImport_AppendInittab("scriptable", &PyInit_scriptable);` then the
module of the same name should be overwritten with the C++ program's
functionality.
I've been through the embedded programming docs a bit, and they seem
quite good as a reference, but I don't know what I'm doing well enough
to solve my problems using them. Thanks for reading.
My ultimate goal is to expose features written in C++ for a game engine
using SFML, and run .py files in a subdirectory to generate maps,
control NPC dialogue and actions, etc. I'm hoping to have something
usable for this year's 7DRL, which starts March 4th. I'd like to spend
the time until then getting this engine working smoothly and porting it
to Windows, so I can focus exclusively on "game content" for the
timeboxed 7-day portion.
Kind Regards,
-John McCardle
--
https://mail.python.org/mailman/listinfo/python-list