Mailing List Archive

1 2  View All
Re: New sys.module_names attribute in Python 3.10: list of all stdlib modules [ In reply to ]
On Tue, Jan 26, 2021 at 12:08:03PM -0800, Brett Cannon wrote:
> On Tue, Jan 26, 2021 at 1:26 AM Antoine Pitrou <antoine@python.org> wrote:

[...]
> > Disagreed. This is niche enough that it warrants a long but explicit
> > name, rather than some ambiguous shortcut.
> >
>
> I agree w/ Antoine that a more descriptive name for such a niche (but
> useful!) attribute makes sense.

This descriptive name is *literally incorrect*. By design, it doesn't
list modules. It only lists sub-packages and not sub-modules, to keep
the number of entries more managable.

(Personally, I don't think an extra hundred or two names makes that much
difference. Its going to be a big list one way or the other.)

So by the current rules, many stdlib modules are not included and the
name is inaccurate.

If you're not going to list all the dotted modules of a package, why
distinguish sub-modules from sub-packages? It is confusing and ackward
to have only some dotted modules listed based on the **implementation**.

(We need a good term for "things you can import that create a module
*object* regardless of whether they are a *module file* or a *package*.
I'm calling them a dotted module for lack of a better name.)

By the current rules, many stdlib modules are not listed, and you can't
see why unless you know their implementation:


* urllib - listed
* urllib.parse - not listed

* collections - listed
* collections.abc - not listed

* email - listed
* email.parser - not listed
* email.mime - listed # Surprise!


So we have this weird situation where an implementation detail of the
dotted module (whether it is a file `package/module.py` or
`package/module/__init__.py`) determines whether it shows up or not.

And because the file system structure of a module is not part of its
API, that implementation detail could change without warning.

I think that either of:

1. list *all* package dotted modules regardless of whether they are
implemented as a sub-module or sub-package;

2. list *no* package dotted modules, only the top-level package;

would be better than this inconsistent hybrid of only listing some
dotted modules.

(Excluding the test modules is fine.)



--
Steve
_______________________________________________
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-leave@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/GNOAP4TVTHIUKE2GUGZWV6HNVE37KU4Q/
Code of Conduct: http://python.org/psf/codeofconduct/
Re: New sys.module_names attribute in Python 3.10: list of all stdlib modules [ In reply to ]
On Tue, Jan 26, 2021 at 11:19:13PM +0100, Victor Stinner wrote:
> On Tue, Jan 26, 2021 at 10:04 PM Steve Dower <steve.dower@python.org> wrote:
> >
> > On 1/26/2021 8:32 PM, Steve Holden wrote:
> > > If the length of the name is any kind of issue, since the stdlib
> > > only contains modules (and packages), why not just sys.stdlib_names?
> >
> > And since the modules can vary between platforms and builds, why
> > wouldn't this be sysconfig.stdlib_names rather than sys.stdlib_names?
>
> The list is the same on all platforms on purpose ;-) Example:
>
> >>> 'winsound' in sys.stdlib_module_names
> True

Right. This is (I think) Steve's point: the list is inaccurate, because
the existence of 'winsound' in the stdlib_module_names doesn't mean that
the module 'winsound' exists.



--
Steve
_______________________________________________
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-leave@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/BHKOH3N2FGMTUVXT4YCJMPCOI7SWOTEU/
Code of Conduct: http://python.org/psf/codeofconduct/
Re: New sys.module_names attribute in Python 3.10: list of all stdlib modules [ In reply to ]
On Tue, Jan 26, 2021 at 10:56:57AM +0100, Victor Stinner wrote:
> On Tue, Jan 26, 2021 at 12:44 AM Steven D'Aprano <steve@pearwood.info> wrote:

[...]
> > Your first instinct that it is too long is correct. Just call it
> > "stdlib" or "stdlib_names". The fact that it is a frozen set of module
> > names will be obvious from just looking at it, and there is no need for
> > the name to explain everything about it. We have:
>
> The sys module already has a sys.modules attribute, and so
> sys.module_names sounds like "give me the name of all imported
> modules": sys.module.keys().

Then its a good thing I didn't propose calling it "module_names" :-)



--
Steve
_______________________________________________
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-leave@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/K7FP7PSFULJYL2GPZDSOGKDPKZZ5GYKP/
Code of Conduct: http://python.org/psf/codeofconduct/
Re: New sys.module_names attribute in Python 3.10: list of all stdlib modules [ In reply to ]
On Wed, 27 Jan 2021 11:05:28 +1100
Steven D'Aprano <steve@pearwood.info> wrote:
> On Tue, Jan 26, 2021 at 12:08:03PM -0800, Brett Cannon wrote:
> > On Tue, Jan 26, 2021 at 1:26 AM Antoine Pitrou <antoine@python.org> wrote:
>
> [...]
> > > Disagreed. This is niche enough that it warrants a long but explicit
> > > name, rather than some ambiguous shortcut.
> > >
> >
> > I agree w/ Antoine that a more descriptive name for such a niche (but
> > useful!) attribute makes sense.
>
> This descriptive name is *literally incorrect*. By design, it doesn't
> list modules. It only lists sub-packages and not sub-modules, to keep
> the number of entries more managable.
>
> (Personally, I don't think an extra hundred or two names makes that much
> difference. Its going to be a big list one way or the other.)
>
> So by the current rules, many stdlib modules are not included and the
> name is inaccurate.

Ok, then "stdlib_package_names"? :-)

Regards

Antoine.

_______________________________________________
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-leave@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/NMDFLQDCXRQNUBMHTHTH37OMDPCCQYRZ/
Code of Conduct: http://python.org/psf/codeofconduct/
Re: New sys.module_names attribute in Python 3.10: list of all stdlib modules [ In reply to ]
On Wed, Jan 27, 2021 at 10:44:00AM +0100, Antoine Pitrou wrote:

> Ok, then "stdlib_package_names"? :-)

Heh :-)

I see your smiley, and I'm not going to argue about the name any
further. I have my preference, but if the consensus is
stdlib_module_names, so be it.

But I think the inconsistency between sub-modules and sub-packages is
important. We should either list all sub-whatever or none of them,
rather than only some of them.

--
Steve
_______________________________________________
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-leave@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/4YYPKUGD4HCFXUNBYXFNNTDDOUF7ZRR2/
Code of Conduct: http://python.org/psf/codeofconduct/
Re: New sys.module_names attribute in Python 3.10: list of all stdlib modules [ In reply to ]
Hi Steven,

That makes sense to me: I wrote
https://github.com/python/cpython/pull/24353 to exclude sub-package.

The change removes 12 sub-packages from sys.stdlib_module_names and
len(sys.stdlib_module_names) becomes 300 :-)

-"concurrent.futures",
-"ctypes.macholib",
-"distutils.command",
-"email.mime",
-"ensurepip._bundled",
-"lib2to3.fixes",
-"lib2to3.pgen2",
-"multiprocessing.dummy",
-"xml.dom",
-"xml.etree",
-"xml.parsers",
-"xml.sax",

With that name, names of sys.stdlib_module_names don't contain "." anymore.

So to check if "email.message" is a stdlib module name, exclude the
part after the first dot, and check if "email" is in
sys.stdlib_module_names. In practice, it is not possible to add a
sub-package or a sub-module to a stdlib module, so this limitation
(excluding sub-packages and sub-modules) sounds reasonable to me.

Victor


On Wed, Jan 27, 2021 at 1:09 AM Steven D'Aprano <steve@pearwood.info> wrote:
> This descriptive name is *literally incorrect*. By design, it doesn't
> list modules. It only lists sub-packages and not sub-modules, to keep
> the number of entries more managable.
>
> (Personally, I don't think an extra hundred or two names makes that much
> difference. Its going to be a big list one way or the other.)
>
> So by the current rules, many stdlib modules are not included and the
> name is inaccurate.
>
> If you're not going to list all the dotted modules of a package, why
> distinguish sub-modules from sub-packages? It is confusing and ackward
> to have only some dotted modules listed based on the **implementation**.
>
> (We need a good term for "things you can import that create a module
> *object* regardless of whether they are a *module file* or a *package*.
> I'm calling them a dotted module for lack of a better name.)
>
> By the current rules, many stdlib modules are not listed, and you can't
> see why unless you know their implementation:
>
>
> * urllib - listed
> * urllib.parse - not listed
>
> * collections - listed
> * collections.abc - not listed
>
> * email - listed
> * email.parser - not listed
> * email.mime - listed # Surprise!
>
>
> So we have this weird situation where an implementation detail of the
> dotted module (whether it is a file `package/module.py` or
> `package/module/__init__.py`) determines whether it shows up or not.
>
> And because the file system structure of a module is not part of its
> API, that implementation detail could change without warning.
>
> I think that either of:
>
> 1. list *all* package dotted modules regardless of whether they are
> implemented as a sub-module or sub-package;
>
> 2. list *no* package dotted modules, only the top-level package;
>
> would be better than this inconsistent hybrid of only listing some
> dotted modules.
>
> (Excluding the test modules is fine.)

--
Night gathers, and now my watch begins. It shall not end until my death.
_______________________________________________
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-leave@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/UW6F2MRC5RNOLEJJI64BALENK7R7UYA2/
Code of Conduct: http://python.org/psf/codeofconduct/
Re: New sys.module_names attribute in Python 3.10: list of all stdlib modules [ In reply to ]
On Wed, Jan 27, 2021 at 1:16 AM Steven D'Aprano <steve@pearwood.info> wrote:
> Right. This is (I think) Steve's point: the list is inaccurate, because
> the existence of 'winsound' in the stdlib_module_names doesn't mean that
> the module 'winsound' exists.

This point is addressed by the definition of the list:
sys.stdlib_module_names documentation.

"It is the same on all platforms. Modules which are not available on
some platforms and modules disabled at Python build are also listed.
All module kinds are listed: pure Python, built-in, frozen and
extension modules. Test modules are excluded."

https://docs.python.org/dev/library/sys.html#sys.stdlib_module_names

As I wrote previously, there are use cases which *require* the list
being the same on all platforms. Moreover, in practice, it's quite
hard to build a list of available stdlib module names. You need to
build extension modules, try to implement them, then rebuild the list
of module which requires to rebuild Python. It's not convenient. Also,
there are different definition of "available". For example, "import
multiprocessing" can fail on some platforms if there is no lock
implementation available. It's not because it's installed on the
system that the import will work for sure.

IMO the only reliable way to check if a module can be imported... is
to import it. And then you hit again the issue of import side effects.

There are different ways to filter sys.stdlib_module_names list to
only list "available" modules. Try import, pkgutil.iter_modules() or
pkgutil.walk_packages(). IMO it should remain out of the scope of
sys.stdlib_module_names.

Victor
--
Night gathers, and now my watch begins. It shall not end until my death.
_______________________________________________
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-leave@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/IKFK6CTTYTWD2VFH36AIN5IGS66KSMFA/
Code of Conduct: http://python.org/psf/codeofconduct/
Re: New sys.module_names attribute in Python 3.10: list of all stdlib modules [ In reply to ]
I probably wouldn't think of that on my own, but the need is rare enough that having the recipe in the documentation (preferably including the docstring) might be enough. (Or it might not.)
_______________________________________________
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-leave@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/BFSS7QXGT3PA6TKSC55JLLUFO5AXUTOC/
Code of Conduct: http://python.org/psf/codeofconduct/
Re: New sys.module_names attribute in Python 3.10: list of all stdlib modules [ In reply to ]
I see a bunch of similar -- but not quite the same -- use cases.

I feel like instead of a set, it should be a dict pointing to an object with attributes that describe the module in various ways (top-level vs subpackage, installed on this machine or not, test module or not, etc). I'll understand if this seems like scope creep, but try not to rule it out as a future enhancement. (e.g., don't promise it will be precisely a set., as opposed to the keys of a map.)
_______________________________________________
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-leave@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/VZ5NSCXXHRE63477ANQXJHD3U2YDFU3J/
Code of Conduct: http://python.org/psf/codeofconduct/

1 2  View All