Mailing List Archive

Hello all,

I don't know if this suggestion is missing some point, or it's part of something already proposed.

In a professional environment, we've came to a point in which most people use virtual environments or conda environments to avoid "polluting a global environment".

However, I think that's a problem with the default behaviour of the module management in Python. A nice default behaviour would be to search for a requirements.txt file in the same directory as __file__, and use the newest version of every module that matches the constraints. If no requirements where given, the newest version already installed could be used. That would require allowing multiple versions of the same module to be downloaded.

I already anticipate some problems: increased disk usage for people that are not using virtual environments, the possibility of breaking changes for scripts with no constraints over a module (so that if a different module download a newer version, both would be using it), and of course the hassle of a completely new default behaviour that would require a transition in many codebases.

That there are other solutions to the problem, such as forcing the usage of semantic versioning, but that's a bit utopic. But still, I believe it would pay off in terms of time saved in environment installing and switching. Also, it's a good step in the path to integrating pip as something closer to the Python core.

What's your opinion, is the effort required too big for the returns? Do you think other problems may arise?
--
https://mail.python.org/mailman/listinfo/python-list

On Wed, Nov 11, 2020 at 10:06 PM j c <jucaranlu@gmail.com> wrote:
>
> Hello all,
>
> I don't know if this suggestion is missing some point, or it's part of something already proposed.
>
> In a professional environment, we've came to a point in which most people use virtual environments or conda environments to avoid "polluting a global environment".
>
> However, I think that's a problem with the default behaviour of the module management in Python. A nice default behaviour would be to search for a requirements.txt file in the same directory as __file__, and use the newest version of every module that matches the constraints. If no requirements where given, the newest version already installed could be used. That would require allowing multiple versions of the same module to be downloaded.
>

This would stop venvs from providing the isolation that they are
supposed to, and instead would just create yet another way to invoke
dependency hell. No thank you.

A virtual environment isn't just a way to install different versions
of modules. It's way WAY more than that, and if you need to have too
many different versions around, you have bigger problems to deal with.

(As a simple thought experiment to prove the problem with your
proposal: what happens with your dependencies' dependencies, and what
if they conflict? At what point would that be detected?)

ChrisA
--
https://mail.python.org/mailman/listinfo/python-list

On Wed, Nov 11, 2020 at 3:00 AM j c <jucaranlu@gmail.com> wrote:

> Hello all,
>
> I don't know if this suggestion is missing some point, or it's part of
> something already proposed before.
>
> In a professional environment, we've came to a point in which most people
> use virtual environments or code environments to avoid "polluting a global
> environment".
>
I think it'd be a good idea to have a directory (hierarchy) for each python
application, and make pip (or similar tool) download to that directory -
and then modify the _application's_ sys.path to include that directory at
the beginning.

This is what I've done with backshift (
https://stromberg.dnsalias.org/~strombrg/backshift/). It works well,
without a need for a virtual environment, while still giving dependency
isolation. But it's not as automatic as I'd like - I've had to manually
specify what dependencies to put in the directory.

I use virtual environments sometimes, but I always cringe slightly when
doing so. There should be a better way. This is one of the chief benefits
of languages like C, C++ and Rust over Python - they don't require you to
source something before you can run an app written in them.
--
https://mail.python.org/mailman/listinfo/python-list

On Thu, Nov 12, 2020 at 4:35 AM Dan Stromberg <drsalists@gmail.com> wrote:
>
> On Wed, Nov 11, 2020 at 3:00 AM j c <jucaranlu@gmail.com> wrote:
>
> > Hello all,
> >
> > I don't know if this suggestion is missing some point, or it's part of
> > something already proposed before.
> >
> > In a professional environment, we've came to a point in which most people
> > use virtual environments or code environments to avoid "polluting a global
> > environment".
> >
> I think it'd be a good idea to have a directory (hierarchy) for each python
> application, and make pip (or similar tool) download to that directory -
> and then modify the _application's_ sys.path to include that directory at
> the beginning.
>
> This is what I've done with backshift (
> https://stromberg.dnsalias.org/~strombrg/backshift/). It works well,
> without a need for a virtual environment, while still giving dependency
> isolation. But it's not as automatic as I'd like - I've had to manually
> specify what dependencies to put in the directory.
>

Can you elaborate on exactly how this is different? When you create a
venv, it creates some symlinks and such that basically mean you get
that - a directory for the application, pip installs into it, and then
when you run that Python binary, it'll automatically have sys.path
contain the appropriate directory.

ChrisA
--
https://mail.python.org/mailman/listinfo/python-list

On Wed, Nov 11, 2020 at 10:38 AM Chris Angelico <rosuav@gmail.com> wrote:

> On Thu, Nov 12, 2020 at 4:35 AM Dan Stromberg <drsalists@gmail.com> wrote:
> >
> > On Wed, Nov 11, 2020 at 3:00 AM j c <jucaranlu@gmail.com> wrote:
> >
> > > Hello all,
> > >
> > > I don't know if this suggestion is missing some point, or it's part of
> > > something already proposed before.
> > >
> > > In a professional environment, we've came to a point in which most
> people
> > > use virtual environments or code environments to avoid "polluting a
> global
> > > environment".
> > >
> > I think it'd be a good idea to have a directory (hierarchy) for each
> python
> > application, and make pip (or similar tool) download to that directory -
> > and then modify the _application's_ sys.path to include that directory at
> > the beginning.
> >
> > This is what I've done with backshift (
> > https://stromberg.dnsalias.org/~strombrg/backshift/). It works well,
> > without a need for a virtual environment, while still giving dependency
> > isolation. But it's not as automatic as I'd like - I've had to manually
> > specify what dependencies to put in the directory.
> >
>
> Can you elaborate on exactly how this is different? When you create a
> venv, it creates some symlinks and such that basically mean you get
> that - a directory for the application, pip installs into it, and then
> when you run that Python binary, it'll automatically have sys.path
> contain the appropriate directory.
>

Maybe it's not that different.

I do get a .../bin/backshift though, which is a bash script that knows how
to start up python on the main module. So the user need not source
something at install time or at run time.

The install process is ./configure and make install, where ./configure is
Not autoconf, but acts sort of like an autoconf script.
--
https://mail.python.org/mailman/listinfo/python-list

On Wednesday, 11 November 2020 at 12:22:24 UTC+1, Chris Angelico wrote:
> On Wed, Nov 11, 2020 at 10:06 PM j c wrote:
> >
> > Hello all,
> >
> > I don't know if this suggestion is missing some point, or it's part of something already proposed.
> >
> > In a professional environment, we've came to a point in which most people use virtual environments or conda environments to avoid "polluting a global environment".
> >
> > However, I think that's a problem with the default behaviour of the module management in Python. A nice default behaviour would be to search for a requirements.txt file in the same directory as __file__, and use the newest version of every module that matches the constraints. If no requirements where given, the newest version already installed could be used. That would require allowing multiple versions of the same module to be downloaded.
> >
> This would stop venvs from providing the isolation that they are
> supposed to, and instead would just create yet another way to invoke
> dependency hell. No thank you.
>
> A virtual environment isn't just a way to install different versions
> of modules. It's way WAY more than that, and if you need to have too
> many different versions around, you have bigger problems to deal with.
>
> (As a simple thought experiment to prove the problem with your
> proposal: what happens with your dependencies' dependencies, and what
> if they conflict? At what point would that be detected?)
>
> ChrisA

How can this behaviour turn into dependency hell? Every dependency use the specified version if any, otherwise the most recent one, which also applies to second order dependencies. In case of conflict, the first version would be imported, which is currently the default behaviour. The main difference is that this approach would be able to generate a warning before running the script with no need for pipdeptree.
--
https://mail.python.org/mailman/listinfo/python-list

On Thu, Nov 12, 2020 at 10:44 AM Dan Stromberg <drsalists@gmail.com> wrote:
>
>
> On Wed, Nov 11, 2020 at 10:38 AM Chris Angelico <rosuav@gmail.com> wrote:
>>
>> On Thu, Nov 12, 2020 at 4:35 AM Dan Stromberg <drsalists@gmail.com> wrote:
>> >
>> > On Wed, Nov 11, 2020 at 3:00 AM j c <jucaranlu@gmail.com> wrote:
>> >
>> > > Hello all,
>> > >
>> > > I don't know if this suggestion is missing some point, or it's part of
>> > > something already proposed before.
>> > >
>> > > In a professional environment, we've came to a point in which most people
>> > > use virtual environments or code environments to avoid "polluting a global
>> > > environment".
>> > >
>> > I think it'd be a good idea to have a directory (hierarchy) for each python
>> > application, and make pip (or similar tool) download to that directory -
>> > and then modify the _application's_ sys.path to include that directory at
>> > the beginning.
>> >
>> > This is what I've done with backshift (
>> > https://stromberg.dnsalias.org/~strombrg/backshift/). It works well,
>> > without a need for a virtual environment, while still giving dependency
>> > isolation. But it's not as automatic as I'd like - I've had to manually
>> > specify what dependencies to put in the directory.
>> >
>>
>> Can you elaborate on exactly how this is different? When you create a
>> venv, it creates some symlinks and such that basically mean you get
>> that - a directory for the application, pip installs into it, and then
>> when you run that Python binary, it'll automatically have sys.path
>> contain the appropriate directory.
>
>
> Maybe it's not that different.
>
> I do get a .../bin/backshift though, which is a bash script that knows how to start up python on the main module. So the user need not source something at install time or at run time.
>
> The install process is ./configure and make install, where ./configure is Not autoconf, but acts sort of like an autoconf script.
>

Ah, fair enough.

Did you know that, with a vanilla venv, you can actually just run a
Python interpreter from within there, without sourcing anything? Very
handy for automated scripts.

rosuav@sikorsky:~$ mustard-mine/env/bin/python3 -c 'import sys; print(sys.path)'
['', '/usr/local/lib/python38.zip', '/usr/local/lib/python3.8',
'/usr/local/lib/python3.8/lib-dynload',
'/home/rosuav/mustard-mine/env/lib/python3.8/site-packages']

ChrisA
--
https://mail.python.org/mailman/listinfo/python-list

On 2020-11-12 10:56:45 +1100, Chris Angelico wrote:
> On Thu, Nov 12, 2020 at 10:44 AM Dan Stromberg <drsalists@gmail.com> wrote:
> > I do get a .../bin/backshift though, which is a bash script that
> > knows how to start up python on the main module. So the user need
> > not source something at install time or at run time.
> >
> > The install process is ./configure and make install, where
> > ./configure is Not autoconf, but acts sort of like an autoconf
> > script.
> >
>
> Ah, fair enough.
>
> Did you know that, with a vanilla venv, you can actually just run a
> Python interpreter from within there, without sourcing anything? Very
> handy for automated scripts.

This also works with the Python interpreter in the shebang line.

So if your "make install" for "mytool" creates a venv in
/usr/local/lib/mytool/venv and then replaces the shebang in
/usr/local/bin/mytool with "#!/usr/local/lib/mytool/venv/python", the
user can just invoke "mytool" without caring about the virtual
environment.

If you do that for every Python script, size may be a problem, though: I
just checked a few of my venvs, and they are between 13 and 418 MB:
Average 81 MB.

hp

--
_ | Peter J. Holzer | Story must make more sense than reality.
|_|_) | |
| | | hjp@hjp.at | -- Charles Stross, "Creative writing
__/ | http://www.hjp.at/ | challenge!"

On Fri, Nov 13, 2020 at 9:41 AM Peter J. Holzer <hjp-python@hjp.at> wrote:
>
> On 2020-11-12 10:56:45 +1100, Chris Angelico wrote:
> > On Thu, Nov 12, 2020 at 10:44 AM Dan Stromberg <drsalists@gmail.com> wrote:
> > > I do get a .../bin/backshift though, which is a bash script that
> > > knows how to start up python on the main module. So the user need
> > > not source something at install time or at run time.
> > >
> > > The install process is ./configure and make install, where
> > > ./configure is Not autoconf, but acts sort of like an autoconf
> > > script.
> > >
> >
> > Ah, fair enough.
> >
> > Did you know that, with a vanilla venv, you can actually just run a
> > Python interpreter from within there, without sourcing anything? Very
> > handy for automated scripts.
>
> This also works with the Python interpreter in the shebang line.
>
> So if your "make install" for "mytool" creates a venv in
> /usr/local/lib/mytool/venv and then replaces the shebang in
> /usr/local/bin/mytool with "#!/usr/local/lib/mytool/venv/python", the
> user can just invoke "mytool" without caring about the virtual
> environment.
>
> If you do that for every Python script, size may be a problem, though: I
> just checked a few of my venvs, and they are between 13 and 418 MB:
> Average 81 MB.
>

Indeed. I would divide Python scripts into a few categories:

1) System scripts. If the script came from a Debian repository, it
runs in the Debian-provided Python and uses libraries installed with
apt/dpkg.

2) Scripts that don't really much care about their environment. These
will generally run in the latest master branch build of Python
(currently 3.10), and if they need anything installed, I install it
("globally") into that instance of Python. They can't interfere with
system scripts, but can interfere with each other.

3) Scripts that get their own venv. That could be because they
conflict with something else, or something else conflicts with them,
or they need specific versions of things, or anything.

At the moment, I can find a grand total of nine virtual environments
for various scripts. Some of them are because Python 3.10 can't run
the scripts in question; others probably could run, but I want a
consistent environment for my dev/staging so it's easier to push to
production. The largest env directory is 250ish MB, and they average
94MB, so my figures are fairly comparable to yours.

ChrisA
--
https://mail.python.org/mailman/listinfo/python-list