Mailing List Archive: PEP668 / pipx and "--editable" installs

PEP668 / pipx and "--editable" installs

Sep 15, 2023, 7:15 AM

Post #1 of 14 (686 views)

Hello,
I wonder that today was the first day I stumbled over PEP668 / pipx and
the "externally-managed-environment" problem. I migrated from Debian 11
to 12.

I'm developing some Python packages; applications and libraries. I never
used virtual environments and wouldn't like to start with it.
But I do use "--editable" for my packages. I wonder how I should go on
with using "pipx" and "--editable" mode because I had some problems with
it I couldn't fix following PEP668 but only with using
"--break-system-packages". And PEP668 do not mention editable-mode.

The two example packages I use here are public available and clonable
(is this a word?) via git.

Let's start with a command line application named "hyperorg"
(https://codeberg.org/buhtz/hyperorg).
I tried to install it via "pipx install -e .[develop]". It's
pyproject.toml has a bug: A missing dependency "dateutil". But
"dateutil" is not available from PyPi for Python 3.11 (the default in
Debian 12). But thanks to great Debian they have a "python3-dateutil"
package. I installed it.
But "hyperorg" do ignore it and can't import it. Of course I
re-installed hyperorg via pipx.
Installing it via "pip" and "--break-system-packages" it works well and
do see the "python3-dateutil" package.

The second example is a "library" named "buhtzology"
(https://codeberg.org/buhtz/buhtzology).
Here it seems that "pipx" do refuse to install it in the first place
("pipx install -e .[develop]") and gave me this message.

"No apps associated with package buhtzology. Try again with
'--include-deps' to include apps of dependent packages, which are listed
above. If you are attempting to install a library, pipx should not be
used. Consider using pip or a similar tool instead."

What is the difference between an "app" (it is a google/apple marketing
term used for software with user interaction. Just name it application.)
and a "library" in context of pip(x)? Why does it count?
I tried again with "pipx install --include-deps -e .[develop]" and got
an output I'm not sure if this is an error or a success message. I
assume it is an error because I'm not able to "import buhtzology". It
seems like that the package was not installed.

?? Note: normalizer was already on your PATH at /usr/bin/normalizer
?? Note: f2py was already on your PATH at /usr/bin/f2py
?? Note: f2py3 was already on your PATH at /usr/bin/f2py3
?? Note: f2py3.11 was already on your PATH at /usr/bin/f2py3.11
?? Note: py.test was already on your PATH at /usr/bin/py.test
?? Note: pytest was already on your PATH at /usr/bin/pytest
?? File exists at /home/user/.local/bin/pycodestyle and points to
/home/user/.local/bin/pycodestyle, not
/home/user/.local/pipx/venvs/buhtzology/bin/pycodestyle. Not modifying.
?? File exists at /home/user/.local/bin/ruff and points to
/home/user/.local/bin/ruff, not
/home/user/.local/pipx/venvs/buhtzology/bin/ruff. Not modifying.
installed package buhtzology 0.2.0.dev0, installed using Python 3.11.2
These apps are now globally available
- f2py
- f2py3
- f2py3.11
- fonttools
- normalizer
- py.test
- pyftmerge
- pyftsubset
- pytest
- tabulate
- ttx
- pycodestyle (symlink missing or pointing to unexpected location)
- ruff (symlink missing or pointing to unexpected location)
done! ? ???? ?

BUT!

$ python3 -c "import buhtzology"
Traceback (most recent call last):
File "<string>", line 1, in <module>
ModuleNotFoundError: No module named 'buhtzology'
--
https://mail.python.org/mailman/listinfo/python-list

Re: PEP668 / pipx and "--editable" installs [ In reply to ]

python-list at python

Sep 15, 2023, 7:17 PM

Post #2 of 14 (684 views)

Permalink

>
> I never
> used virtual environments and wouldn't like to start with it.

There's your problem - everything else is a result of this. There is
just no nice way to work with a combination of pypi, apt-get and
system-wide python.

Everyone uses virtual environments.

Sorry.
--
https://mail.python.org/mailman/listinfo/python-list

Re: PEP668 / pipx and "--editable" installs [ In reply to ]

python-list at python

Sep 15, 2023, 11:44 PM

Post #3 of 14 (682 views)

Permalink

Dear Rimu,

thanks for your reply. I'm willing to learn for sure.

On 2023-09-16 14:17 Rimu Atkinson via Python-list
<python-list@python.org> wrote:
> There's your problem - everything else is a result of this. There is
> just no nice way to work with a combination of pypi, apt-get and
> system-wide python.
>
> Everyone uses virtual environments.

It is nothing bad about using virtual environments but also not about
not using them. In my own work I haven't see a use case where I needed
them. And I expect that some day I'll encounter a use case for it. This
here is not about pro and cons of virtual environments.

Please explain how the two problems I explained are influenced by not
using virtual environments.

As I explained I tried to use "pipx" (the X at the end!) because it
seems to be the recommended way today because of PEP668. And to my
knowledge pipx do use a virtual environment in the back. Nice.

But it did not work the way I expected.

I don't say that pipx is wrong or buggy. I assume I misunderstood
something in context of PEP668 and in the use of pipx. That is why I'm
asking here.
--
https://mail.python.org/mailman/listinfo/python-list

Re: PEP668 / pipx and "--editable" installs [ In reply to ]

python-list at python

Sep 16, 2023, 3:09 AM

Post #4 of 14 (682 views)

Permalink

Am Sat, Sep 16, 2023 at 02:17:19PM +1200 schrieb Rimu Atkinson via Python-list:

> Everyone uses virtual environments.

Umm, like, no.

Karsten
--
GPG 40BE 5B0E C98E 1713 AFA6 5BC0 3BEA AC80 7D4F C89B
--
https://mail.python.org/mailman/listinfo/python-list

Re: PEP668 / pipx and "--editable" installs [ In reply to ]

python-list at python

Sep 16, 2023, 4:57 PM

Post #5 of 14 (680 views)

Permalink

> It is nothing bad about using virtual environments but also not about
> not using them. In my own work I haven't see a use case where I needed
> them. And I expect that some day I'll encounter a use case for it. This
> here is not about pro and cons of virtual environments.
>
You are in a use case where you need them, right now :) When you
understand the benefits of virtual environments you will understand what
I meant by that.

> Please explain how the two problems I explained are influenced by not
> using virtual environments.

The first problem can be avoided because virtual environments can use a
different version of python than the system one. If you need an earlier
version of python then you can use it instead.

The second problem can be avoided because virtual environments exist in
a part of the file system that you have write access to, so you don't
need to use sudo to install packages. Your main user account does not
have write access to /usr/bin.

Also when a virtual environment is activated the path to it's packages
is a part of that environment so your code will always be able to import
the packages you want.

It's much easier to understand if you try it for yourself. Google has
many excellent resources, here is one
https://www.freecodecamp.org/news/how-to-setup-virtual-environments-in-python/

Best of luck :)

R
--
https://mail.python.org/mailman/listinfo/python-list

Re: PEP668 / pipx and "--editable" installs [ In reply to ]

python-list at python

Sep 17, 2023, 9:12 PM

Post #6 of 14 (680 views)

Permalink

On 9/16/2023 7:57 PM, Rimu Atkinson via Python-list wrote:
>
>> It is nothing bad about using virtual environments but also not about
>> not using them. In my own work I haven't see a use case where I needed
>> them. And I expect that some day I'll encounter a use case for it. This
>> here is not about pro and cons of virtual environments.
>>
> You are in a use case where you need them, right now :) When you
> understand the benefits of virtual environments you will understand what
> I meant by that.
>
>> Please explain how the two problems I explained are influenced by not
>> using virtual environments.
>
> The first problem can be avoided because virtual environments can use a
> different version of python than the system one. If you need an earlier
> version of python then you can use it instead.

I have multiple versions of Python on both Windows and Linux machines.
I don't have to use the system version. On Windows the "py" launcher
can launch any Python version on your system.

> The second problem can be avoided because virtual environments exist in
> a part of the file system that you have write access to, so you don't
> need to use sudo to install packages. Your main user account does not
> have write access to /usr/bin.

And therefore on Linux pip will install packages as --user even if the
option isn't specified. On Windows, one would just use --user routinely.

> Also when a virtual environment is activated the path to it's packages
> is a part of that environment so your code will always be able to import
> the packages you want.

Whenever a version of Python is launched its system path is set up so
that its own packages are used. The one thing that can cause problems
is programs in the Python Scripts directory, which may not be on the
path for a particular version of Python. A virtual environment will
take case of that, but I almost never run scripts in that directory so
it's not an issue for me.

I think there are situations where a venv is useful, but not because of
the points you have asserted here. For myself, I find that after a
while, I tend to forget what I set up the various venvs for, and what
state they are in. So I have stopped using them. I just put up with
having more packages in /site-packages than a particular applications
needs. If I want to, say, work on code in a git clone's directory tree,
I launch Python using a batch file that sets the PYTHONPATH to point there.

> It's much easier to understand if you try it for yourself. Google has
> many excellent resources, here is one
> https://www.freecodecamp.org/news/how-to-setup-virtual-environments-in-python/
>
> Best of luck :)
>
> R

--
https://mail.python.org/mailman/listinfo/python-list

Re: PEP668 / pipx and "--editable" installs [ In reply to ]

python-list at python

Sep 17, 2023, 10:52 PM

Post #7 of 14 (680 views)

Permalink

Thanks for your reply. btw: I do know what a virtual environment is and
how its works. No need to explain. And again: I do expect "pipx" to use
virtual environments. So this thread is not about pro and cons of using
virtual environments.

On 2023-09-17 11:57 Rimu Atkinson via Python-list
<python-list@python.org> wrote:
> The first problem can be avoided because virtual environments can use
> a different version of python than the system one. If you need an
> earlier version of python then you can use it instead.

But as I explained I used pipx to install it. So virtual environments
are used in that case.
Maybe I made not the problem clear enough: The problem is that the
installed application ("hyperorg") did not see the system package
"python3-dateutils".

How can I solve this without(!) using another Python version?
The behavior that the pipx-installed application is not able to see
system python packages (installed via "apt install python3-dateutils")
is unexpected to me.

Maybe there is a good reason for that? Explain it please?
Maybe I did something wrong? How can I make it right?
Maybe it is a bug or a use case that is not covered by current version
of pipx or Debian 12?

> The second problem can be avoided because virtual environments exist
> in a part of the file system

Also here I assume I didn't make my problem clear enough.
I tried to install a library (from source) with "pipx". But it did not
work somehow (no clear error message). The global python interpreter
did not see that library.

Why?
--
https://mail.python.org/mailman/listinfo/python-list

Re: PEP668 / pipx and "--editable" installs [ In reply to ]

python-list at python

Sep 18, 2023, 1:16 AM

Post #8 of 14 (680 views)

Permalink

On 2023-09-15 14:15:23 +0000, c.buhtz--- via Python-list wrote:
> I tried to install it via "pipx install -e .[develop]". It's pyproject.toml
> has a bug: A missing dependency "dateutil". But "dateutil" is not available
> from PyPi for Python 3.11 (the default in Debian 12). But thanks to great
> Debian they have a "python3-dateutil" package. I installed it.

Sidenote:
PyPI does have several packages with "dateutil" in their name. From the
version number (2.8.2) I guess that "python-dateutil" is the one
packaged in Debian 12.

This can be installed via pip:

% lsb_release -a
No LSB modules are available.
Distributor ID: Debian
Description: Debian GNU/Linux 12 (bookworm)
Release: 12
Codename: bookworm

(dateutil) % pip install python-dateutil
Collecting python-dateutil
Downloading python_dateutil-2.8.2-py2.py3-none-any.whl (247 kB)
???????????????????????????????????????? 247.7/247.7 kB 3.1 MB/s eta 0:00:00
Collecting six>=1.5
Downloading six-1.16.0-py2.py3-none-any.whl (11 kB)
Installing collected packages: six, python-dateutil
Successfully installed python-dateutil-2.8.2 six-1.16.0

hp

--
_ | Peter J. Holzer | Story must make more sense than reality.
|_|_) | |
| | | hjp@hjp.at | -- Charles Stross, "Creative writing
__/ | http://www.hjp.at/ | challenge!"

Re: PEP668 / pipx and "--editable" installs [ In reply to ]

python-list at python

Sep 18, 2023, 8:22 AM

Post #9 of 14 (680 views)

Permalink

On 9/18/23 02:16, Peter J. Holzer via Python-list wrote:
> On 2023-09-15 14:15:23 +0000, c.buhtz--- via Python-list wrote:
>> I tried to install it via "pipx install -e .[develop]". It's pyproject.toml
>> has a bug: A missing dependency "dateutil". But "dateutil" is not available
>> from PyPi for Python 3.11 (the default in Debian 12). But thanks to great
>> Debian they have a "python3-dateutil" package. I installed it.
>
> Sidenote:
> PyPI does have several packages with "dateutil" in their name. From the
> version number (2.8.2) I guess that "python-dateutil" is the one
> packaged in Debian 12.
>
> This can be installed via pip:

It *is* the case that package name is not always equal to importable
name. That certainly occurs in the universe of Python packages on PyPI;
it's if anything much more likely on Linux distributions which have to
share the package name namespace with a lot more than just Python
packages (just for starters, what seems like billions of Perl packages),
so you're even more likely to see names like python-foo or python3-foo
when the thing you import is foo. That has nothing to do virtualenvs,
of course.

The use of a virtualenv for a project actually makes it more likely that
you discover unstated dependencies, which is generally a good thing.

--
https://mail.python.org/mailman/listinfo/python-list

Re: PEP668 / pipx and "--editable" installs [ In reply to ]

python-list at python

Sep 18, 2023, 11:56 AM

Post #10 of 14 (680 views)

Permalink

On 2023-09-18 10:16 "Peter J. Holzer via Python-list"
<python-list@python.org> wrote:
> On 2023-09-15 14:15:23 +0000, c.buhtz--- via Python-list wrote:
> > I tried to install it via "pipx install -e .[develop]". It's
> > pyproject.toml has a bug: A missing dependency "dateutil". But
> > "dateutil" is not available from PyPi for Python 3.11 (the default
> > in Debian 12). But thanks to great Debian they have a
> > "python3-dateutil" package. I installed it.
>
> This can be installed via pip:

I'm aware of this. But this is not the question.

I would like to know and understand why my via "pipx" installed package
"hyperorg" is not able to see the systems packages installed via "apt
install python3-dateutils"?

Is this the usual behavior? Is this correct?
What is the design decision behind it?

Is it really the intention of PEP668 that pipx-installed packages are
not allowed to use system packages?
--
https://mail.python.org/mailman/listinfo/python-list

Re: PEP668 / pipx and "--editable" installs [ In reply to ]

python-list at python

Sep 18, 2023, 12:09 PM

Post #11 of 14 (680 views)

Permalink

On 9/18/23 12:56, c.buhtz--- via Python-list wrote:
> On 2023-09-18 10:16 "Peter J. Holzer via Python-list"
> <python-list@python.org> wrote:
>> On 2023-09-15 14:15:23 +0000, c.buhtz--- via Python-list wrote:
>>> I tried to install it via "pipx install -e .[develop]". It's
>>> pyproject.toml has a bug: A missing dependency "dateutil". But
>>> "dateutil" is not available from PyPi for Python 3.11 (the default
>>> in Debian 12). But thanks to great Debian they have a
>>> "python3-dateutil" package. I installed it.
>>
>> This can be installed via pip:
>
> I'm aware of this. But this is not the question.
>
> I would like to know and understand why my via "pipx" installed package
> "hyperorg" is not able to see the systems packages installed via "apt
> install python3-dateutils"?
>
> Is this the usual behavior? Is this correct?

Yes. By default, the virtualenv contains just what you've installed.
It's designed to give you tight control over what's installed, so you
can track dependencies, avoid accidental inclusions, etc. As usual, you
don't have to accept the default. For example, for the venv module:

usage: venv [-h] [--system-site-packages] [--symlinks | --copies] [--clear]
[--upgrade] [--without-pip] [--prompt PROMPT] [--upgrade-deps]
ENV_DIR [ENV_DIR ...]

Creates virtual Python environments in one or more target directories.

positional arguments:
ENV_DIR A directory to create the environment in.

options:
-h, --help show this help message and exit
--system-site-packages
Give the virtual environment access to the system
site-packages dir.
...

--
https://mail.python.org/mailman/listinfo/python-list

Re: PEP668 / pipx and "--editable" installs [ In reply to ]

python-list at python

Sep 18, 2023, 12:38 PM

Post #12 of 14 (680 views)

Permalink

On 9/18/2023 2:56 PM, c.buhtz--- via Python-list wrote:
> On 2023-09-18 10:16 "Peter J. Holzer via Python-list"
> <python-list@python.org> wrote:
>> On 2023-09-15 14:15:23 +0000, c.buhtz--- via Python-list wrote:
>>> I tried to install it via "pipx install -e .[develop]". It's
>>> pyproject.toml has a bug: A missing dependency "dateutil". But
>>> "dateutil" is not available from PyPi for Python 3.11 (the default
>>> in Debian 12). But thanks to great Debian they have a
>>> "python3-dateutil" package. I installed it.
>>
>> This can be installed via pip:
>
> I'm aware of this. But this is not the question.
>
> I would like to know and understand why my via "pipx" installed package
> "hyperorg" is not able to see the systems packages installed via "apt
> install python3-dateutils"?
>
> Is this the usual behavior? Is this correct?
> What is the design decision behind it?
>
> Is it really the intention of PEP668 that pipx-installed packages are
> not allowed to use system packages?

One way this could happen is if the hyperorg package got installed by a
different version of Python than the system version. I've lost track of
just how your installation is set up so this may not apply. But, for
example, if the system is using Python 3.10.x and you installed hyperorg
using Python 3.11, that would be such a situation. The apt-installed
packages will get installed into the system python directories, rather
than to the Python 3.11 directories.

--
https://mail.python.org/mailman/listinfo/python-list

Re: PEP668 / pipx and "--editable" installs [ In reply to ]

python-list at python

Sep 20, 2023, 5:43 AM

Post #13 of 14 (669 views)

Permalink

On 2023-09-18 18:56:35 +0000, c.buhtz--- via Python-list wrote:
> On 2023-09-18 10:16 "Peter J. Holzer via Python-list"
> <python-list@python.org> wrote:
> > On 2023-09-15 14:15:23 +0000, c.buhtz--- via Python-list wrote:
> > > I tried to install it via "pipx install -e .[develop]". It's
> > > pyproject.toml has a bug: A missing dependency "dateutil". But
> > > "dateutil" is not available from PyPi for Python 3.11 (the default
> > > in Debian 12). But thanks to great Debian they have a
> > > "python3-dateutil" package. I installed it.
> >
> > This can be installed via pip:
>
> I'm aware of this.

You wrote:

> > > "dateutil" is not available from PyPi for Python 3.11

That's quite a curious thing to write if you are aware that dateutil is
in fact available from PyPi for Python 3.11.

> But this is not the question.

I know. That's why I labeled my comment as a "Sidenote".

hp

--
_ | Peter J. Holzer | Story must make more sense than reality.
|_|_) | |
| | | hjp@hjp.at | -- Charles Stross, "Creative writing
__/ | http://www.hjp.at/ | challenge!"

Re: PEP668 / pipx and "--editable" installs [ In reply to ]

python-list at python

Sep 20, 2023, 6:31 AM

Post #14 of 14 (669 views)

Permalink

Dear Peter,

maybe we have a missunderstanding.

Am 20.09.2023 14:43 schrieb Peter J. Holzer via Python-list:
>> > > "dateutil" is not available from PyPi for Python 3.11
>
> That's quite a curious thing to write if you are aware that dateutil is
> in fact available from PyPi for Python 3.11.

Do I miss something here?

See https://pypi.org/project/dateutils/ and also the open Issue about
the missing support for Python 3.11
https://github.com/dateutil/dateutil/issues/1233 ?
--
https://mail.python.org/mailman/listinfo/python-list

Mailing List Archive

Attached Files:

Attached Files: