Mailing List Archive: [issue43022] Unable to dynamically load functions from python3.dll

[issue43022] Unable to dynamically load functions from python3.dll

Jan 25, 2021, 4:14 AM

Post #1 of 14 (73 views)

New submission from Paul Moore <p.f.moore@gmail.com>:

I am writing a small application using the embedded distribution to run a script supplied by the user. The requirements are very simple, so all I need to do is set up argv and call Py_Main.

I'm trying to load the Py_Main function dynamically (for flexibility - see below) but GetProcAddress returns 0 when loading the function from python3.dll (the stable ABI). This seems to be because the symbols in python3.dll are special "fowarding" symbols, that GetProcAddress can't handle.

Is there a way to dynamically load the Python API from the stable ABI? If there isn't currently, could one be added?

To explain my requirements in a bit more detail, I don't want to statically link, because I want to put the Python distribution in a subdirectory, so that it isn't visible on PATH along with my executable, but I'd rather avoid the complexities involved in adding a dedicated SxS manifest to point the loader to the correct subdirectory for the python3.dll.

Furthermore, I want to provide a graceful fallback if the Python distribution is missing (initially, just a friendly error message, but in future maybe locating an alternative Python installation to use) and static linking won't allow that as I can't recover control if the expected Python DLL isn't present.

The reason I want to use the stable ABI is so that I can upgrade the embedded distribution without needing to rebuild the C code. I could search for python3X.dll, and just ignore the stable ABI. But that's more filesystem code than I really want to write in C... And honestly, I feel that dynamically linking is a perfect use case for the stable ABI, so it really should be supported.

----------
components: Windows
messages: 385621
nosy: paul.moore, steve.dower, tim.golden, zach.ware
priority: normal
severity: normal
status: open
title: Unable to dynamically load functions from python3.dll
type: behavior
versions: Python 3.9

_______________________________________
Python tracker <report@bugs.python.org>
<https://bugs.python.org/issue43022>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com

[issue43022] Unable to dynamically load functions from python3.dll [ In reply to ]

report at bugs

Jan 25, 2021, 4:29 AM

Post #2 of 14 (73 views)

Permalink

Change by Jeremy Kloth <jeremy.kloth+python-tracker@gmail.com>:

----------
nosy: +jkloth

_______________________________________
Python tracker <report@bugs.python.org>
<https://bugs.python.org/issue43022>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com

[issue43022] Unable to dynamically load functions from python3.dll [ In reply to ]

report at bugs

Jan 25, 2021, 9:51 AM

Post #3 of 14 (73 views)

Permalink

Eryk Sun <eryksun@gmail.com> added the comment:

"python3.dll" doesn't directly depend on "python39.dll", so the loader waits to load it until first accessed via GetProcAddress(). It should remember the activation context of "python3.dll", such as whether it was loaded with a fully-qualified path and LOAD_LIBRARY_SEARCH_DLL_LOAD_DIR | LOAD_LIBRARY_SEARCH_DEFAULT_DIRS -- or LOAD_WITH_ALTERED_SEARCH_PATH. It works for me.

----------
nosy: +eryksun

_______________________________________
Python tracker <report@bugs.python.org>
<https://bugs.python.org/issue43022>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com

[issue43022] Unable to dynamically load functions from python3.dll [ In reply to ]

report at bugs

Jan 25, 2021, 10:50 AM

Post #4 of 14 (73 views)

Permalink

Paul Moore <p.f.moore@gmail.com> added the comment:

So I need to dynamically load *both* python3.dll and python39.dll, but if I do that I can get the functions from python3.dll? What's the point in doing that? Surely I might as well just load python39.dll and get the functions from there, in that case.

I hoped that by using python3.dll, I'd be able to avoid needing to deal with the version-specific DLL at all, so making my code portable to any version of Python 3. I'm pretty sure that's how it works if I statically link to python3.dll...

----------

_______________________________________
Python tracker <report@bugs.python.org>
<https://bugs.python.org/issue43022>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com

[issue43022] Unable to dynamically load functions from python3.dll [ In reply to ]

report at bugs

Jan 25, 2021, 11:02 AM

Post #5 of 14 (73 views)

Permalink

Paul Moore <p.f.moore@gmail.com> added the comment:

Thinking about what you said, "the loader waits to load it until first accessed via GetProcAddress()" did you mean by that, that the following code should work:

h = LoadLibraryW(L"some/path/to/python3.dll")
py_main = GetProcAddress(h, "Py_Main")

(with some/path/to/python39.dll being loaded on the second line)? Because if so, then that's what doesn't work for me.

Or are you suggesting that I do

AddDllDirectoryW(L"some/path/to");
h = LoadLibraryW(L"python3.dll");
py_main = GetProcAddress(h, "Py_Main");

Because I didn't think to try that - and I'm not 100% sure how I'd have to do stuff like LOAD_LIBRARY_SEARCH_USER_DIRS and/or SetDefaultDllDirectories to make that work. I find the Microsoft docs on all of this really complex and hard to follow if you're a non-expert :-(

----------

_______________________________________
Python tracker <report@bugs.python.org>
<https://bugs.python.org/issue43022>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com

[issue43022] Unable to dynamically load functions from python3.dll [ In reply to ]

report at bugs

Jan 25, 2021, 11:07 AM

Post #6 of 14 (73 views)

Permalink

Eryk Sun <eryksun@gmail.com> added the comment:

> So I need to dynamically load *both* python3.dll and python39.dll,
> but if I do that I can get the functions from python3.dll?

No, just load the fully-qualified name of "python3.dll", with the loader flags LOAD_LIBRARY_SEARCH_DLL_LOAD_DIR | LOAD_LIBRARY_SEARCH_DEFAULT_DIRS -- or LOAD_WITH_ALTERED_SEARCH_PATH.

----------

_______________________________________
Python tracker <report@bugs.python.org>
<https://bugs.python.org/issue43022>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com

[issue43022] Unable to dynamically load functions from python3.dll [ In reply to ]

report at bugs

Jan 25, 2021, 11:09 AM

Post #7 of 14 (73 views)

Permalink

Eryk Sun <eryksun@gmail.com> added the comment:

> h = LoadLibraryW(L"some/path/to/python3.dll")

You should be using LoadLibraryExW(). It's a one-time call. You do not need to set the default flags via SetDefaultDllDirectories().

----------

_______________________________________
Python tracker <report@bugs.python.org>
<https://bugs.python.org/issue43022>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com

[issue43022] Unable to dynamically load functions from python3.dll [ In reply to ]

report at bugs

Jan 25, 2021, 11:36 AM

Post #8 of 14 (73 views)

Permalink

Eryk Sun <eryksun@gmail.com> added the comment:

I noted that the path to "python3.dll" must be fully qualified. But let me stress that point. You cannot use the relative path "path/to/python3.dll" with LOAD_LIBRARY_SEARCH_DLL_LOAD_DIR. The loader will fail the call as an invalid parameter.

Unlike POSIX, a Windows path that contains slashes may be handled as a relative path in search contexts, if it lacks a drive or leading slash. In the case of LoadLibraryW(L"path/to/python3.dll"), the loader will try to resolve "path/to/python3.dll" against every directory in the DLL search path, which may not even include the current working directory.

----------

_______________________________________
Python tracker <report@bugs.python.org>
<https://bugs.python.org/issue43022>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com

[issue43022] Unable to dynamically load functions from python3.dll [ In reply to ]

report at bugs

Jan 25, 2021, 12:07 PM

Post #9 of 14 (73 views)

Permalink

Paul Moore <p.f.moore@gmail.com> added the comment:

Thanks, I'll give that a try. It sounds like I'm just using the APIs incorrectly, which would be good (in the sense that I can do what I wanted, I just didn't know how ;-))

I wonder whether it would be worth having a section in the docs somewhere explaining how to do this, or more generally what the "best practices" are for embedding? If I were to try to put something together, would that be worthwhile?

> I noted that the path to "python3.dll" must be fully qualified.

Just to note, my actual code does use an absolute path, I was over-simplifying here. But it's worth being clear on it, particularly if I do write something for the docs.

----------

_______________________________________
Python tracker <report@bugs.python.org>
<https://bugs.python.org/issue43022>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com

[issue43022] Unable to dynamically load functions from python3.dll [ In reply to ]

report at bugs

Jan 25, 2021, 12:49 PM

Post #10 of 14 (73 views)

Permalink

Steve Dower <steve.dower@python.org> added the comment:

> I wonder whether it would be worth having a section in the docs somewhere explaining how to do this, or more generally what the "best practices" are for embedding?

Yes, it would be great. I think there are a few pieces in Doc/using/windows.rst already, and maybe a few bits in the embedding/extending section, but more would be valuable here.

> If I were to try to put something together, would that be worthwhile?

It would give us something concrete to argue about, at least ;) Speaking of arguing...

(Earlier post)
> The reason I want to use the stable ABI is so that I can upgrade the embedded distribution without needing to rebuild the C code.

I think here you're in a very small minority who could get away with this, and so I'd hesitate to make it sound like the recommended approach.

What I'd actually recommend (on Windows) is to bundle a copy of Python with your application. The embeddable distro is specifically for this purpose, and once you've done that then there's no need to worry about the version changing at all (unless you do it yourself).

What's missing *here* is any kind of guide or walkthrough on how to do it. I think I imagined that I'd have the time and inclination to do some myself, but obviously I never have. Ultimately, I'd definitely be suggesting "just fix your Python version and load it directly" for embedders, so probably wouldn't have helped with what you're trying to do here anyway, Paul.

----------

_______________________________________
Python tracker <report@bugs.python.org>
<https://bugs.python.org/issue43022>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com

[issue43022] Unable to dynamically load functions from python3.dll [ In reply to ]

report at bugs

Jan 25, 2021, 1:02 PM

Post #11 of 14 (73 views)

Permalink

Paul Moore <p.f.moore@gmail.com> added the comment:

Confirmed. The following code works as I want:

Py_Main_t get_pymain(wchar_t *base_dir) {
wchar_t *dll_path;
HRESULT hr = PathAllocCombine(
base_dir, L"python\\python3.dll",
PATHCCH_ALLOW_LONG_PATHS, &dll_path
);
if (hr != S_OK) {
error(L"Could not construct Python DLL path");
}

HMODULE py_dll = LoadLibraryExW(dll_path, 0, LOAD_LIBRARY_SEARCH_DLL_LOAD_DIR | LOAD_LIBRARY_SEARCH_DEFAULT_DIRS);
if (!py_dll) {
error(L"Could not load Python DLL %ls", dll_path);
}
LocalFree(dll_path);

Py_Main_t py_main = (Py_Main_t)GetProcAddress(py_dll, "Py_Main");
if (!py_main) {
error(L"Could not locate Py_Main function");
}

return py_main;
}

If people think it's worthwhile, I can put together a change to https://docs.python.org/3/extending/embedding.html#embedding-python-in-another-application (maybe a new section, "Embedding on Windows by dynamically loading the stable ABI"?) that uses a code snippet like this.

----------

_______________________________________
Python tracker <report@bugs.python.org>
<https://bugs.python.org/issue43022>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com

[issue43022] Unable to dynamically load functions from python3.dll [ In reply to ]

report at bugs

Jan 25, 2021, 1:23 PM

Post #12 of 14 (73 views)

Permalink

Paul Moore <p.f.moore@gmail.com> added the comment:

> I think here you're in a very small minority who could get away with this, and so I'd hesitate to make it sound like the recommended approach.

Well, the evidence here is that maybe even I shouldn't be doing this :-)

> What I'd actually recommend (on Windows) is to bundle a copy of Python with your application.

That's actually precisely what I'm doing. But I don't exactly have an "application", in the sense that I suspect you mean, which is the difficulty.

I'm trying to address the problem that it's a real pain to ship Python scripts as "utilities" on a Windows system. You want such scripts to still be plain text, because you typically hack on them a lot. You can't rely on shebangs and the launcher, because I continually find that PATHEXT doesn't include ".py" so they don't run properly, and "py name_of_script.py" doesn't do path searches. And anyway, I don't want to run the scripts with my system Python, because I don't want to dump all my dependencies in there.

So what I have is a small stub, that looks for a .py file of the same name, alongside the executable (so myutil.exe looks for myutil.py). Rather than using the system Python, it looks for a copy of Python in a subdirectory next to the script, and uses that. I can then install dependencies in the dedicated interpreter.

(And yes, PEP 582 would help make things easier in this area, too, but it never gained enough traction.)

I use the embedded distribution as that interpreter. I may be able to use a virtualenv instead, but I've not tried that yet.

Multiple copies of the launcher, one per script, and you're done :-)

To be honest, it really sucks that this is the most reliable way of managing small utility scripts in Python on Windows. But every other solution I've tried has had its own issues. I'd *much* prefer something standard, but I don't know how to bundle this in a way that makes it straightforward for "ordinary" users, so I've decided just to solve my own problem and leave it at that :-)

----------

_______________________________________
Python tracker <report@bugs.python.org>
<https://bugs.python.org/issue43022>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com

[issue43022] Unable to dynamically load functions from python3.dll [ In reply to ]

report at bugs

Jan 25, 2021, 1:43 PM

Post #13 of 14 (73 views)

Permalink

Eryk Sun <eryksun@gmail.com> added the comment:

> uses a code snippet like this.

PathAllocCombine() is Windows 8+, so sample code that uses it would only be for Python 3.9+.

I'd prefer the function pointer to be returned as an out parameter, and return an HRESULT status as the result. If LoadLibraryExW() or GetProcAddress() fails, map the error code via HRESULT_FROM_WIN32(GetLastError()). And log to stderr instead of exiting with an error() call.

----------

_______________________________________
Python tracker <report@bugs.python.org>
<https://bugs.python.org/issue43022>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com

[issue43022] Unable to dynamically load functions from python3.dll [ In reply to ]

report at bugs

Jan 25, 2021, 2:26 PM

Post #14 of 14 (73 views)

Permalink

Paul Moore <p.f.moore@gmail.com> added the comment:

> PathAllocCombine() is Windows 8+, so sample code that uses it would only be for Python 3.9+.

Yeah, I'm probably going to remove that. I mainly used it because I'm *so* spoiled by Python, writing code in C where I have to actually implement stuff for myself rather than just using a stdlib function, just feels so tiresome these days :-)

Thanks for the other suggestions.

----------

_______________________________________
Python tracker <report@bugs.python.org>
<https://bugs.python.org/issue43022>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com