Mailing List Archive: Incomplete sys.path with embeddable python (Windows)!?

Incomplete sys.path with embeddable python (Windows)!?

Apr 20, 2023, 2:47 PM

Post #1 of 11 (330 views)

Hello,

when I run a script with a "normally" installed python, the directory
the script resides in is automatically added as first element to
sys.path, so that "import my_local_module" finds my_local_module.py in
the directory of the script.

However, when I run the same script with embeddable python ("Windows
embeddable package (64-bit)", download link
https://www.python.org/ftp/python/3.11.3/python-3.11.3-embed-amd64.zip)
the script directory is *not* prepended to the path, thus "import
my_local_module" gives an ImportError.

I couldn't find an option to get the "normal" behaviour. Any ideas how
to do that?

What I tried so far:

* The start-up value for sys.path seems to be defined in python311._pth.
It looks that I can add further static paths to it, but I don't know how
to make it add the script path (which can be different for different
scripts).

* Uncommenting "import site" in python311._pth doesn't help.

* It seems that I could import something else in python311._pth, but I
don't know how something imported from there could find out the path of
the script that is about to be started.

* I read the (rather short) documentation of the embeddable package and
of the site module several times but couldn't recognize a hint as to how
to solve the issue.

* I can add the following lines to every script:
import sys
script_path = __file__.rsplit("\\", 1)[0]
if script_path not in sys.path:
sys.path[0:0] = [script_path]
import my_local_modul
That works, but it's ugly, executing code between imports is frowned
upon, and it needs to be added to every script.

Does anybody have a better idea?
Any help is appreciated.

Ralf M.
--
https://mail.python.org/mailman/listinfo/python-list

Re: Incomplete sys.path with embeddable python (Windows)!? [ In reply to ]

mats at wichmann

Apr 21, 2023, 8:31 AM

Post #2 of 11 (330 views)

Permalink

On 4/20/23 15:47, Ralf M. wrote:
> Hello,
>
> when I run a script with a "normally" installed python, the directory
> the script resides in is automatically added as first element to
> sys.path, so that "import my_local_module" finds my_local_module.py in
> the directory of the script.
>
> However, when I run the same script with embeddable python ("Windows
> embeddable package (64-bit)", download link
> https://www.python.org/ftp/python/3.11.3/python-3.11.3-embed-amd64.zip)
> the script directory is *not* prepended to the path, thus "import
> my_local_module" gives an ImportError.

This is intended behavior - the question comes up from time to time. The
embeddable distribution is intended to be part of an application, not a
general-purpose Python you can call for just anything.

There are a bunch of details here, for example:

https://github.com/python/cpython/issues/79022

--
https://mail.python.org/mailman/listinfo/python-list

Re: Incomplete sys.path with embeddable python (Windows)!? [ In reply to ]

list1 at tompassin

Apr 21, 2023, 9:07 AM

Post #3 of 11 (330 views)

Permalink

On 4/20/2023 5:47 PM, Ralf M. wrote:
> Hello,
>
> when I run a script with a "normally" installed python, the directory
> the script resides in is automatically added as first element to
> sys.path, so that "import my_local_module" finds my_local_module.py in
> the directory of the script.
>
> However, when I run the same script with embeddable python ("Windows
> embeddable package (64-bit)", download link
> https://www.python.org/ftp/python/3.11.3/python-3.11.3-embed-amd64.zip)
> the script directory is *not* prepended to the path, thus "import
> my_local_module" gives an ImportError.
>
> I couldn't find an option to get the "normal" behaviour. Any ideas how
> to do that?
>
> What I tried so far:
>
> * The start-up value for sys.path seems to be defined in python311._pth.
> It looks that I can add further static paths to it, but I don't know how
> to make it add the script path (which can be different for different
> scripts).
>
> * Uncommenting "import site" in python311._pth doesn't help.
>
> * It seems that I could import something else in python311._pth, but I
> don't know how something imported from there could find out the path of
> the script that is about to be started.
>
> * I read the (rather short) documentation of the embeddable package and
> of the site module several times but couldn't recognize a hint as to how
> to solve the issue.
>
> * I can add the following lines to every script:
>     import sys
>     script_path = __file__.rsplit("\\", 1)[0]
>     if script_path not in sys.path:
>         sys.path[0:0] = [script_path]
>     import my_local_modul
> That works, but it's ugly, executing code between imports is frowned
> upon, and it needs to be added to every script.
>
> Does anybody have a better idea?
> Any help is appreciated.

I haven't worked with embeddable python, but here are some possibilities
that came to mind, depending on how your system works -

1. If your script is started from the command line, sys.argv[0] gives
the path to the script; You could use os.path.dirname() to get its
directory. This will end up the same place as your code fragment, but
looks nicer and handles different path separators (e.g., Linux vs Windows);

2. You could write a little module that figures out the script's path
and import that first in all your scripts.

3. If you know all the directories that your scripts will be in, you
could add them all to a xx.pth file (do a search to make sure where to
put .pth files for an embeddable case).

Not executing code between imports is a guideline, but guidelines are
not laws and you can certainly deviate from them if there is a good reason.

My preference would be for 1) if at all possible. Something like this
(untested) -

import sys
import os.path

script_dir = os.path.dirname(sys.argv[0])
sys.path.insert(0, script_dir) # Use built-in method to insert

Inserting the path entry again in case it's already there is not harmful
so there's no need to check. If for some reason sys.argv is not
available, just substitute __file__. That still won't be too clunky.
--
https://mail.python.org/mailman/listinfo/python-list

Re: Incomplete sys.path with embeddable python (Windows)!? [ In reply to ]

python-list at python

Apr 21, 2023, 6:27 PM

Post #4 of 11 (330 views)

Permalink

How are you invoking your script? Presumably you have some code
in your embedding application that takes a script path and runs
it. Instead of putting the code to update sys.path into every
script, the embedding application could do it before running
the script.

--
Greg
--
https://mail.python.org/mailman/listinfo/python-list

Re: Incomplete sys.path with embeddable python (Windows)!? [ In reply to ]

Ralf_M at t-online

Apr 22, 2023, 2:35 PM

Post #5 of 11 (330 views)

Permalink

Am 21.04.2023 um 17:31 schrieb Mats Wichmann:
> On 4/20/23 15:47, Ralf M. wrote:
>> Hello,
>>
>> when I run a script with a "normally" installed python, the directory
>> the script resides in is automatically added as first element to
>> sys.path, so that "import my_local_module" finds my_local_module.py in
>> the directory of the script.
>>
>> However, when I run the same script with embeddable python ("Windows
>> embeddable package (64-bit)", download link
>> https://www.python.org/ftp/python/3.11.3/python-3.11.3-embed-amd64.zip) the script directory is *not* prepended to the path, thus "import my_local_module" gives an ImportError.
>
> This is intended behavior - the question comes up from time to time. The
> embeddable distribution is intended to be part of an application, not a
> general-purpose Python you can call for just anything.
>
> There are a bunch of details here, for example:
>
> https://github.com/python/cpython/issues/79022
>
Thank you for the pointer to the issue. I'll try to remove the ._pth
completely (and see whether that breaks anything) and may have a look at
the nuget.org package.

I can see that for many cases the behaviour is appropriate, but I had
hoped that there is a configuration option for the cases where it is not.

About my use case:
There is a complex application package, consisting of some commercial
and some freeware software, tied together with scripts in at least four
different scripting languages. Now I intend to add further functionality
in a fifth language, Python. The idea is to make the embeddedable
package part of the application package and have scripts for the new
functions. Several independent functions are to be added, each
consisting of a script plus some local modules, and all of them should
use the same python embedded into the application package.
--
https://mail.python.org/mailman/listinfo/python-list

Re: Incomplete sys.path with embeddable python (Windows)!? [ In reply to ]

Ralf_M at t-online

Apr 22, 2023, 2:45 PM

Post #6 of 11 (330 views)

Permalink

Am 22.04.2023 um 03:27 schrieb Greg Ewing via Python-list:
> How are you invoking your script? Presumably you have some code
> in your embedding application that takes a script path and runs
> it. Instead of putting the code to update sys.path into every
> script, the embedding application could do it before running
> the script.
>
In principle a good idea, but I don't know how to do that:
The script is currently invoked by a .cmd file, but that may change to a
shortcut (.lnk). This is what the embeddable package documentation calls
"Python Application - simple approach".
To update sys.path on start up I would need to do something like
C:\path\to\python.exe --add-path C:\s-path C:\s-path\script.py
but I couldn't find an option like --add-path.
--
https://mail.python.org/mailman/listinfo/python-list

Re: Incomplete sys.path with embeddable python (Windows)!? [ In reply to ]

Ralf_M at t-online

Apr 22, 2023, 3:04 PM

Post #7 of 11 (330 views)

Permalink

Am 21.04.2023 um 18:07 schrieb Thomas Passin:
> On 4/20/2023 5:47 PM, Ralf M. wrote:
>> Hello,
>>
>> when I run a script with a "normally" installed python, the directory
>> the script resides in is automatically added as first element to
>> sys.path, so that "import my_local_module" finds my_local_module.py in
>> the directory of the script.
>>
>> However, when I run the same script with embeddable python ("Windows
>> embeddable package (64-bit)", download link
>> https://www.python.org/ftp/python/3.11.3/python-3.11.3-embed-amd64.zip) the script directory is *not* prepended to the path, thus "import my_local_module" gives an ImportError.
>>
>> I couldn't find an option to get the "normal" behaviour. Any ideas how
>> to do that?
>>
>> What I tried so far:
>> [...]
>> * I can add the following lines to every script:
>>      import sys
>>      script_path = __file__.rsplit("\\", 1)[0]
>>      if script_path not in sys.path:
>>          sys.path[0:0] = [script_path]
>>      import my_local_modul
>> [...]

Thank your for your hints.

> I haven't worked with embeddable python, but here are some possibilities
> that came to mind, depending on how your system works -
>
> 1. If your script is started from the command line, sys.argv[0] gives
> the path to the script;
I didn't think of sys.argv[0] to get at the path; this might be quite
useful, I'll try it out next week.

> You could use os.path.dirname() to get its
> directory. This will end up the same place as your code fragment, but
> looks nicer and handles different path separators (e.g., Linux vs Windows);
Yes, but it requires another import and the embedded package is only
available for windows anyway, I think. I'll consider the idea, though.

> 2. You could write a little module that figures out the script's path
> and import that first in all your scripts.
>
> 3. If you know all the directories that your scripts will be in, you
> could add them all to a xx.pth file (do a search to make sure where to
> put .pth files for an embeddable case).
I thought about that, but for that to work all local modules across all
script locations must have unique names, otherwise import might get hold
of a module from the wrong directory. Certainly doable for a few
scripts, but might become a source of hard to track errors when the
number of scripts increases and later maintainers are not aware of the
naming restriction.

> [..}

--
https://mail.python.org/mailman/listinfo/python-list

Re: Incomplete sys.path with embeddable python (Windows)!? [ In reply to ]

list1 at tompassin

Apr 22, 2023, 3:17 PM

Post #8 of 11 (330 views)

Permalink

On 4/22/2023 5:45 PM, Ralf M. wrote:
> Am 22.04.2023 um 03:27 schrieb Greg Ewing via Python-list:
>> How are you invoking your script? Presumably you have some code
>> in your embedding application that takes a script path and runs
>> it. Instead of putting the code to update sys.path into every
>> script, the embedding application could do it before running
>> the script.
>>
> In principle a good idea, but I don't know how to do that:
> The script is currently invoked by a .cmd file, but that may change to a
> shortcut (.lnk). This is what the embeddable package documentation calls
> "Python Application - simple approach".
> To update sys.path on start up I would need to do something like
> C:\path\to\python.exe --add-path C:\s-path C:\s-path\script.py
> but I couldn't find an option like --add-path.

You can get the .cmd file's path and put it into an environmental
variable (or as a command line argument). If you haven't encountered
that trick of Windows batch file procesing, here it is:

%~dp0 (returns the directory a batch file is running from)

--
https://mail.python.org/mailman/listinfo/python-list

Re: Incomplete sys.path with embeddable python (Windows)!? [ In reply to ]

python-list at python

Apr 22, 2023, 4:49 PM

Post #9 of 11 (330 views)

Permalink

On 23/04/23 10:04 am, Ralf M. wrote:
> I thought about that, but for that to work all local modules across all
> script locations must have unique names, otherwise import might get hold
> of a module from the wrong directory.

You could put all the local modules belonging to a particular
script into a package named after the script, e.g. put the local
modules used by foo.py into a package called foolib.

--
Greg
--
https://mail.python.org/mailman/listinfo/python-list

Re: Incomplete sys.path with embeddable python (Windows)!? [ In reply to ]

nospam at please

Apr 22, 2023, 11:38 PM

Post #10 of 11 (319 views)

Permalink

Ralf M. ha scritto:
> Am 22.04.2023 um 03:27 schrieb Greg Ewing via Python-list:
>> How are you invoking your script? Presumably you have some code
>> in your embedding application that takes a script path and runs
>> it. Instead of putting the code to update sys.path into every
>> script, the embedding application could do it before running
>> the script.
>>
> In principle a good idea, but I don't know how to do that:
> The script is currently invoked by a .cmd file, but that may change to a
> shortcut (.lnk). This is what the embeddable package documentation calls
> "Python Application - simple approach".
> To update sys.path on start up I would need to do something like
> C:\path\to\python.exe --add-path C:\s-path C:\s-path\script.py
> but I couldn't find an option like --add-path.

what about:

foo.py:
from sys import path
print('path:', path)
#end foo.py

from prompt:

$ python -c "import sys;sys.path=['/user/foopath']+sys.path;import foo;foo"

output:

path: ['/user/foopath', '', '/usr/lib/python39.zip',
'/usr/lib/python3.9', '/usr/lib/python3.9/lib-dynload',
'/usr/lib/python3.9/site-packages']
--
https://mail.python.org/mailman/listinfo/python-list

Re: Incomplete sys.path with embeddable python (Windows)!? [ In reply to ]

mats at wichmann

Apr 23, 2023, 9:17 AM

Post #11 of 11 (330 views)

Permalink

On 4/22/23 16:04, Ralf M. wrote:
> Am 21.04.2023 um 18:07 schrieb Thomas Passin:
>> On 4/20/2023 5:47 PM, Ralf M. wrote:
>>> Hello,
>>>
>>> when I run a script with a "normally" installed python, the directory
>>> the script resides in is automatically added as first element to
>>> sys.path, so that "import my_local_module" finds my_local_module.py
>>> in the directory of the script.
>>>
>>> However, when I run the same script with embeddable python ("Windows
>>> embeddable package (64-bit)", download link
>>> https://www.python.org/ftp/python/3.11.3/python-3.11.3-embed-amd64.zip) the script directory is *not* prepended to the path, thus "import my_local_module" gives an ImportError.
>>>
>>> I couldn't find an option to get the "normal" behaviour. Any ideas
>>> how to do that?
>>>
>>> What I tried so far:
>>> [...]
>>> * I can add the following lines to every script:
>>>      import sys
>>>      script_path = __file__.rsplit("\\", 1)[0]
>>>      if script_path not in sys.path:
>>>          sys.path[0:0] = [script_path]
>>>      import my_local_modul
>>> [...]

Have used this stanza (actually, for the inverse purpose, to remove that
script's dir from sys.path, but whatever...):

script_dir = os.path.dirname(os.path.realpath(__file__))

you can then insert it:

sys.path.insert(0, script_dir)

or just add:

sys.path = [script_dir] + sys.path

assuming you want it at the front...

but it doesn't really solve your problem of needing it *everywhere*...

--
https://mail.python.org/mailman/listinfo/python-list