Mailing List Archive

change of behaviour for '.' in sys.path between 3.10.0a7 and 3.10.0b1
I have a failing test case in 3.10.0b1 for code that uses changes to sys.path and attempts to load unpackaged modules by
changing directory and placing '.' into sys.path; the code works as expected in 2.7.18, 3.6-3.10.0a7.

I printed out related variables in the a7 and b1 runs and find that sys.path is identical (modulo version name changes)
and that os.getcwd() is as expected and that os.path.isfile('./%s.py' % modulename) is True.

In a7 exec('%s as m', NS) works, but in b1 I get ModuleNotFoundError.

I can fix the problem by changing

sys.path.insert(0,'.')

into

sys.path.insert(0,os.getcwd())

The behaviour of '.' in sys.path has always been slightly mysterious, but is this a real bug and have we just been lucky
for 20 years?

I have placed some details here

https://www.reportlab.com/ftp/import-bug-310b1.txt

ReportLab has quite a large codebase and I think it would be hard to get a concise test of this behaviour change. I
would be glad if this is an expected change a7-->b1 and if the use of '.' in this way has become somehow wrong.

I do know that during the tests the errant code is executed with different values for the target directory; the first
usage seems to succeed.
--
Robin Becker
_______________________________________________
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-leave@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/DE3MDGB2JGOJ3X4NWEGJS26BK6PJUPKW/
Code of Conduct: http://python.org/psf/codeofconduct/
Re: change of behaviour for '.' in sys.path between 3.10.0a7 and 3.10.0b1 [ In reply to ]
Might be out of context here, but IMHO "." shouldn't be assumed to be the current directory anyway.

As someone who has ported python to a system where it isn't, these assumptions tend to cause problems.

Cheers

Chris

Sent from my iPhone

> On 3 Jun 2021, at 09:31, Robin Becker <robin@reportlab.com> wrote:
>
> ?I have a failing test case in 3.10.0b1 for code that uses changes to sys.path and attempts to load unpackaged modules by changing directory and placing '.' into sys.path; the code works as expected in 2.7.18, 3.6-3.10.0a7.
>
> I printed out related variables in the a7 and b1 runs and find that sys.path is identical (modulo version name changes) and that os.getcwd() is as expected and that os.path.isfile('./%s.py' % modulename) is True.
>
> In a7 exec('%s as m', NS) works, but in b1 I get ModuleNotFoundError.
>
> I can fix the problem by changing
>
> sys.path.insert(0,'.')
>
> into
>
> sys.path.insert(0,os.getcwd())
>
> The behaviour of '.' in sys.path has always been slightly mysterious, but is this a real bug and have we just been lucky for 20 years?
>
> I have placed some details here
>
> https://www.reportlab.com/ftp/import-bug-310b1.txt
>
> ReportLab has quite a large codebase and I think it would be hard to get a concise test of this behaviour change. I would be glad if this is an expected change a7-->b1 and if the use of '.' in this way has become somehow wrong.
>
> I do know that during the tests the errant code is executed with different values for the target directory; the first usage seems to succeed.
> --
> Robin Becker
> _______________________________________________
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-leave@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/DE3MDGB2JGOJ3X4NWEGJS26BK6PJUPKW/
> Code of Conduct: http://python.org/psf/codeofconduct/

_______________________________________________
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-leave@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/664GGOGFJZ3SQ7PWLA6BYP4A27TSFZ63/
Code of Conduct: http://python.org/psf/codeofconduct/
Re: change of behaviour for '.' in sys.path between 3.10.0a7 and 3.10.0b1 [ In reply to ]
On 6/3/21 4:20 AM, Chris Johns wrote:
> Might be out of context here, but IMHO "." shouldn't be assumed to be the current directory anyway.
>
> As someone who has ported python to a system where it isn't, these assumptions tend to cause problems.


That sounds miserable.  What does "." signify on such a system, if not
the current directory?


//arry/
Re: change of behaviour for '.' in sys.path between 3.10.0a7 and 3.10.0b1 [ In reply to ]
The only other thing that would make sense is a folder in the current
working directory named ".", not?

On Thu, Jun 3, 2021 at 5:05 PM Larry Hastings <larry@hastings.org> wrote:

>
> On 6/3/21 4:20 AM, Chris Johns wrote:
>
> Might be out of context here, but IMHO "." shouldn't be assumed to be the current directory anyway.
>
> As someone who has ported python to a system where it isn't, these assumptions tend to cause problems.
>
>
> That sounds miserable. What does "." signify on such a system, if not the
> current directory?
>
>
> */arry*
> _______________________________________________
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-leave@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-dev@python.org/message/LVOYZGSOJ3OGDX7OUTDXRQVXZ7YGGPS7/
> Code of Conduct: http://python.org/psf/codeofconduct/
>
Re: change of behaviour for '.' in sys.path between 3.10.0a7 and 3.10.0b1 [ In reply to ]
On 3 Jun 2021, at 09:31, Robin Becker <robin@reportlab.com> wrote:

> ReportLab has quite a large codebase and I think it would be hard to
> get a concise test of this behaviour change. I would be glad if this
> is an expected change a7-->b1 and if the use of '.' in this way has
> become somehow wrong.

To me, this sounds like an excellent candidate for git bisect to help
and figure out which exact commit caused this behavior change.

Was it possible for you to detect that?

Thank you,
Senthil
_______________________________________________
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-leave@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/CW2SAOGIO5EIXU7GJZIN26MLF6AC2HD5/
Code of Conduct: http://python.org/psf/codeofconduct/
Re: change of behaviour for '.' in sys.path between 3.10.0a7 and 3.10.0b1 [ In reply to ]
Maybe this?

04732ca993 bpo-43105: Importlib now resolves relative paths when creating
module spec objects from file locations (GH-25121)

On Thu, Jun 3, 2021 at 9:49 AM Senthil Kumaran <senthil@python.org> wrote:

> On 3 Jun 2021, at 09:31, Robin Becker <robin@reportlab.com> wrote:
>
> > ReportLab has quite a large codebase and I think it would be hard to
> > get a concise test of this behaviour change. I would be glad if this
> > is an expected change a7-->b1 and if the use of '.' in this way has
> > become somehow wrong.
>
> To me, this sounds like an excellent candidate for git bisect to help
> and figure out which exact commit caused this behavior change.
>
> Was it possible for you to detect that?
>
> Thank you,
> Senthil
> _______________________________________________
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-leave@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-dev@python.org/message/CW2SAOGIO5EIXU7GJZIN26MLF6AC2HD5/
> Code of Conduct: http://python.org/psf/codeofconduct/
>


--
--Guido van Rossum (python.org/~guido)
*Pronouns: he/him **(why is my pronoun here?)*
<http://feministing.com/2015/02/03/how-using-they-as-a-singular-pronoun-can-change-the-world/>
Re: change of behaviour for '.' in sys.path between 3.10.0a7 and 3.10.0b1 [ In reply to ]
On Thu, Jun 03, 2021 at 09:55:57AM -0700, Guido van Rossum wrote:
> Maybe this?
>
> 04732ca993 bpo-43105: Importlib now resolves relative paths when creating
> module spec objects from file locations (GH-25121)

Likely!. But
https://github.com/python/cpython/commit/04732ca993fa077a8b9640cc77fb2f152339585a
was supposed to a platform specific bug fix.

I didn't a sense that reportlab regression detected was platform
specific one. The behavior change is an interesting one though.

More interesting thing for me was developers running large test suites
between alpha and beta releases, and reporting bugs! :-)

--
Senthil

_______________________________________________
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-leave@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/GJWGQQGZNUWYIX3UX6T4H5VN7IE4YGN6/
Code of Conduct: http://python.org/psf/codeofconduct/
Re: change of behaviour for '.' in sys.path between 3.10.0a7 and 3.10.0b1 [ In reply to ]
On 03/06/2021 18:14, Senthil Kumaran wrote:
> On Thu, Jun 03, 2021 at 09:55:57AM -0700, Guido van Rossum wrote:
>> Maybe this?
>>
>> 04732ca993 bpo-43105: Importlib now resolves relative paths when creating
>> module spec objects from file locations (GH-25121)
>
> Likely!. But
> https://github.com/python/cpython/commit/04732ca993fa077a8b9640cc77fb2f152339585a
> was supposed to a platform specific bug fix.
>
> I didn't a sense that reportlab regression detected was platform
> specific one. The behavior change is an interesting one though.
>
The regression may well be a platform issue. I am by no means an expert at building python; I followed a recipe from the
ARCH PKGBUILD of some time past so perhaps there are problems from my incompetence.

> More interesting thing for me was developers running large test suites
> between alpha and beta releases, and reporting bugs! :-)
>
You guys can certainly develop language changes faster than us maintainers can use them; the dead hand of past usage is
a real handicap.

I have managed a smaller example which breaks for me in arch linux latest.

The self contained code is here

https://www.reportlab.com/ftp/timport-310b1.py

I switched to using importlib.import_module as I think that can now always be used in the versions we support.

To generate a test base I used an empty directory and then run (in bash)

for d in A B;do rm -rf $d; mkdir $d;echo "print('+++++ in module %s' % __name__)" > $d/$(echo $d | tr [:upper:]
[:lower:]).py; done


or
$ for d in A B;do
> rm -rf $d
> mkdir $d
> echo "print('+++++ in module %s' % __name__)" \
> > $d/$(echo $d | tr [:upper:] [:lower:]).py
> done

to create two simple modules in dirs A & B.

If I run python timport-310b1.py in 3.10.0a7 I occasionally see a problem, but in 3.10.0b1 or b2 I see the problem every
time; the TB ends like this

> print(recursiveImport(mn, debug=1))
> File "/home/robin/devel/reportlab/REPOS/timpbug/tests/timport-310b1.py", line 36, in recursiveImport
> return importlib.import_module(modulename)
> File "/home/robin/devel/reportlab/.py310/lib/python3.10/importlib/__init__.py", line 126, in import_module
> return _bootstrap._gcd_import(name[level:], package, level)
> File "<frozen importlib._bootstrap>", line 1050, in _gcd_import
> File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
> File "<frozen importlib._bootstrap>", line 1004, in _find_and_load_unlocked
> ModuleNotFoundError: No module named 'a'

The occasional problem with 3.10.0a7 make me think some random element is at work perhaps hashing stuff.
--
Robin Becker
_______________________________________________
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-leave@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/UJKVOAR6BUDOIFRSUGQ3THFA2DYG47PQ/
Code of Conduct: http://python.org/psf/codeofconduct/
Re: change of behaviour for '.' in sys.path between 3.10.0a7 and 3.10.0b1 [ In reply to ]
On Thu, Jun 03, 2021 at 07:08:06PM +0100, Robin Becker wrote:
> The regression may well be a platform issue. I am by no means an expert at
> building python; I followed a recipe from the ARCH PKGBUILD of some time

I meant the change in the diff we were suspecting was supposed to be
"Windows" specific. But interesting that it was observed in ARCH. The
non-windows piece of changes were probably responsible

> You guys can certainly develop language changes faster than us maintainers
> can use them; the dead hand of past usage is a real handicap.

I was thinking in terms of Bug-Fix (https://bugs.python.org/issue43105)
that is causing an interesting behavior change (still not certain to
call this a regression).

> I have managed a smaller example which breaks for me in arch linux latest.
>
> The self contained code is here
>
> https://www.reportlab.com/ftp/timport-310b1.py
>
> I switched to using importlib.import_module as I think that can now always be used in the versions we support.

Thanks for this report, Robin.

I will reference this in the issue43105, which has probably has the
context to this.

--
Senthil
_______________________________________________
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-leave@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/DGR6DEWZJRN5IDOZPC4LAHP5J3EIQ55X/
Code of Conduct: http://python.org/psf/codeofconduct/
Re: change of behaviour for '.' in sys.path between 3.10.0a7 and 3.10.0b1 [ In reply to ]
On 03Jun2021 17:06, Bernat Gabor <jokerjokerer@gmail.com> wrote:
>On Thu, Jun 3, 2021 at 5:05 PM Larry Hastings <larry@hastings.org>
>wrote:
>> On 6/3/21 4:20 AM, Chris Johns wrote:
>> Might be out of context here, but IMHO "." shouldn't be assumed to be
>> the current directory anyway.
>>
>> As someone who has ported python to a system where it isn't, these assumptions tend to cause problems.
>>
>> That sounds miserable. What does "." signify on such a system, if not the
>> current directory?
>
>The only other thing that would make sense is a folder in the current
>working directory named ".", not?

I can't speak for Chris Johns, and we're off topic for this thread.

That said, I dislike "." in sys.path, particularly near the front. In
fact my own "invoke this module's main" wrapper scripts strip it out. I
remain personally of the opinion that it is a security nightmare, making
imports _subject_ to the current directory (which might be anywhere,
include a directory owned by a hostile user). So i dislike it as a
default.

I know Robin becker is testing specific behaviour - I don't oppose being
_able_ to put "." in sys.path (though I think a concrete absolute path
is a saner choice).

So for Bernat and Larry: not systems where "." doesn't mean the working
directory, but definitely in situations where you want a more secure
loading of modules (i.e. only from where I expect, not destabilised by a
cd).

Cheers,
Cameron Simpson <cs@cskk.id.au>
_______________________________________________
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-leave@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/YMV4LUW5LYFKJYEIHVM5GTCC6POM6BYF/
Code of Conduct: http://python.org/psf/codeofconduct/
Re: change of behaviour for '.' in sys.path between 3.10.0a7 and 3.10.0b1 [ In reply to ]
On 6/3/2021 7:42 PM, Senthil Kumaran wrote:
> On Thu, Jun 03, 2021 at 07:08:06PM +0100, Robin Becker wrote:
>> The regression may well be a platform issue. I am by no means an expert at
>> building python; I followed a recipe from the ARCH PKGBUILD of some time
>
> I meant the change in the diff we were suspecting was supposed to be
> "Windows" specific. But interesting that it was observed in ARCH. The
> non-windows piece of changes were probably responsible

While we were investigating the reliably broken behaviour, we figured
that related behaviour was *unreliably* broken on all platforms.

Specifically, if you imported a module through a relative path (resolved
to CWD), changed the CWD, cleared the module cache, and reimported, you
would still get the original module, contrary to the intent of getting
the newly resolved module. ("Correct" code would have also cleared the
import path caches, not just sys.modules.) This was because the module
info was cached using the relative path, and so would be used later even
though its absolute path had changed.

Someone reported this change in 3.8 and we decided to revert it in
existing released, because the bugfix was for an obscure enough
situation that it really wasn't worth breaking any existing versions.
For unreleased versions, it's better to fix it so that imports will be
more reliable in the future.

Based on Robin's original post, it looks like this is exactly the
scenario we wanted to fix, and it's probably exposed an underlying issue
in that test suite.

Unfortunately, import caching has changed a lot since Python 2.7, so
it's very unlikely that one approach will be able to do reliable
reloading across all of those versions. Your best bet since 3.3 is to
use the importlib module's methods (but your *best* bet overall is to
run separate test suites in their own process so you can avoid the CWD
changes).

Cheers,
Steve
_______________________________________________
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-leave@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/RDIMZA2KVWZSYUVLWQUZT4TE5BH6QWOV/
Code of Conduct: http://python.org/psf/codeofconduct/
Re: change of behaviour for '.' in sys.path between 3.10.0a7 and 3.10.0b1 [ In reply to ]
On 04/06/2021 19:29, Steve Dower wrote:
...........
>
> While we were investigating the reliably broken behaviour, we figured that related behaviour was
> *unreliably* broken on all platforms.
>
> Specifically, if you imported a module through a relative path (resolved to CWD), changed the CWD, cleared the module
> cache, and reimported, you would still get the original module, contrary to the intent of getting the newly resolved
> module. ("Correct" code would have also cleared the import path caches, not just sys.modules.) This was because the
> module info was cached using the relative path, and so would be used later even though its absolute path had changed.
>
> Someone reported this change in 3.8 and we decided to revert it in existing released, because the bugfix was for an
> obscure enough situation that it really wasn't worth breaking any existing versions. For unreleased versions, it's
> better to fix it so that imports will be more reliable in the future.


Unfortunately we are using these relative imports to implement plugins for an xml language processor so this is likely
not just an accident of ordering the failing tests. I agree we have to get a plugin module a bit more directly using the
import internals. I still think we might need to control sys.path as presumably the plugin code could have imports of
parallel modules.

I'm beginning to understand that messing about with sys.path/modules/meta_path etc etc is a lot harder than it used to be.

Incidentally I find that I am able to import a namespace module directly; this code doesn't crash (at least in python >= 3.6

###################################################
#timpA.py don't run with any meaningful directory A
import importlib
def main():
module = importlib.import_module('A')
print('module=%r' % module)

if __name__=='__main__':
main()
###################################################

in python 3.6-3.9 this prints out "module=<module 'A' (namespace)>" in 3.10.0b1 it prints
"module=<module 'A' (<_frozen_importlib_external._NamespaceLoader object at 0x7f1c8a22fe20>)>".
I thought I had to use pkgutils or pkg_resources to mess with these new-fangled packages.

Likewise I find that the python statement 'import A' also doesn't fail and A is a namespace. I feel quite stupid now :(
--
Robin Becker
_______________________________________________
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-leave@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/KMJAQUGSXINVBLQOMREURUFN4LVCEIVX/
Code of Conduct: http://python.org/psf/codeofconduct/
Re: change of behaviour for '.' in sys.path between 3.10.0a7 and 3.10.0b1 [ In reply to ]
On Sat., Jun. 5, 2021, 07:38 Robin Becker, <robin@reportlab.com> wrote:

> On 04/06/2021 19:29, Steve Dower wrote:
> ...........
> >
> > While we were investigating the reliably broken behaviour, we figured
> that related behaviour was
> > *unreliably* broken on all platforms.
> >
> > Specifically, if you imported a module through a relative path (resolved
> to CWD), changed the CWD, cleared the module
> > cache, and reimported, you would still get the original module, contrary
> to the intent of getting the newly resolved
> > module. ("Correct" code would have also cleared the import path caches,
> not just sys.modules.) This was because the
> > module info was cached using the relative path, and so would be used
> later even though its absolute path had changed.
> >
> > Someone reported this change in 3.8 and we decided to revert it in
> existing released, because the bugfix was for an
> > obscure enough situation that it really wasn't worth breaking any
> existing versions. For unreleased versions, it's
> > better to fix it so that imports will be more reliable in the future.
>
>
> Unfortunately we are using these relative imports to implement plugins for
> an xml language processor so this is likely
> not just an accident of ordering the failing tests. I agree we have to get
> a plugin module a bit more directly using the
> import internals. I still think we might need to control sys.path as
> presumably the plugin code could have imports of
> parallel modules.
>
> I'm beginning to understand that messing about with
> sys.path/modules/meta_path etc etc is a lot harder than it used to be.
>

How so? The key details have not changed since Python 3.4 when module specs
were introduced. And meta path, etc. are 2.7 concepts.


> Incidentally I find that I am able to import a namespace module directly;
> this code doesn't crash (at least in python >= 3.6
>
> ###################################################
> #timpA.py don't run with any meaningful directory A
> import importlib
> def main():
> module = importlib.import_module('A')
> print('module=%r' % module)
>
> if __name__=='__main__':
> main()
> ###################################################
>
> in python 3.6-3.9 this prints out "module=<module 'A' (namespace)>" in
> 3.10.0b1 it prints
> "module=<module 'A' (<_frozen_importlib_external._NamespaceLoader object
> at 0x7f1c8a22fe20>)>".
> I thought I had to use pkgutils or pkg_resources to mess with these
> new-fangled packages.
>

Nope, that has never been required since the concept of namespace packages
were introduced to import.


> Likewise I find that the python statement 'import A' also doesn't fail and
> A is a namespace.


Yep, that's expected.

I feel quite stupid now :(
>

As soon as you step outside the norm there are a lot of details to
understand; import is very much following the "Complex is better than
complicated" zen. So no need to feel stupid, there's a lot to keep track of
when you start to dive into the details.

-Brett

--
> Robin Becker
> _______________________________________________
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-leave@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-dev@python.org/message/KMJAQUGSXINVBLQOMREURUFN4LVCEIVX/
> Code of Conduct: http://python.org/psf/codeofconduct/
>