Mailing List Archive: PROPOSAL: Packages in Python lib

PROPOSAL: Packages in Python lib

May 1, 1999, 12:57 PM

Post #1 of 13 (1970 views)

I would like to suggest a package organization for the standard modules.
The package directories should all be in sys.path so that older modules
would continue to work. Import syntax could become more consistent:

from sys import types
from sys import path

This would make the shift between using the standard libraries and using
packages easier. Consistency!

Other benefits: responsibility for packages would be easier. Migration of
packages into and out of the Python standard library would be easier.
Platform unportable and implementation unportable code could be more
cleanly segmented: ("compiler" is implementation unportable, "win" is
platform unportable).

Over time, (or in the move to Python 2) these packages would migrate to
real, cohesive packages with consistent internal interfaces (i.e. among
formats and protocols) and some consistent interfaces at the package level
(e.g. __all__)

Main benefit: It would be easier to find things in the standard library
and to document the library!

Downsides:

* slightly more typing on import.

Here is a rough example of what I am thinking of:

sys:
except
traceback
types

string:
string
re

compiler:
token.
tokenize
compileall

database:
*dbm
*SQL

formats:
xdrlib
xmllib
rfc822
zlib
...

Math:
whrandom
time
cmath

Data Structures:
UserDict
UserList
array

Unix:

SGI:
al
...

SunOS:
...

Windows:
...

Reflect:
code
commands

Protocols:
urllib
urlparse
gopherlib
htmllib
httplib
telnetlib

Server:
BaseHTTPServer
cgi

Misc:
Bastion
fileinput

Thread:
thread
threading

OS:
tempfile
cmd

Serial:
pickle
cpickle
...

Crypt:
...
Old:
regex

--
Paul Prescod - ISOGEN Consulting Engineer speaking for only himself
http://itrc.uwaterloo.ca/~papresco

"Microsoft spokesman Ian Hatton admits that the Linux system would have
performed better had it been tuned."
"Future press releases on the issue will clearly state that the research
was sponsored by Microsoft."
http://www.itweb.co.za/sections/enterprise/1999/9904221410.asp

PROPOSAL: Packages in Python lib [ In reply to ]

ivnowa at hvision

May 2, 1999, 2:20 AM

Post #2 of 13 (1954 views)

Permalink

On 1 May 99, Ce'Nedra took her magical amulet and heard Paul Prescod say:

>I would like to suggest a package organization for the standard modules.
>The package directories should all be in sys.path so that older modules
>would continue to work. Import syntax could become more consistent:
>
>from sys import types
>from sys import path
>
>This would make the shift between using the standard libraries and using
>packages easier. Consistency!
>
>Other benefits: responsibility for packages would be easier. Migration of
>packages into and out of the Python standard library would be easier.
>Platform unportable and implementation unportable code could be more
>cleanly segmented: ("compiler" is implementation unportable, "win" is
>platform unportable).
>
>Over time, (or in the move to Python 2) these packages would migrate to
>real, cohesive packages with consistent internal interfaces (i.e. among
>formats and protocols) and some consistent interfaces at the package level
>(e.g. __all__)
>
>Main benefit: It would be easier to find things in the standard library
>and to document the library!
>
>Downsides:
>
> * slightly more typing on import.
>
>Here is a rough example of what I am thinking of:
>
>sys:
> except
> traceback
> types

Wouldn't a package sys clash with the builtin module sys? A package with the
same name could never be imported...?

I guess this is a trivial problem, because you could call it 'system' or
something instead. What bothers me more is the following. I have been trying
myself to organize all kinds of modules in a package collection, and a
problem that occurs quite often is, that a module can logically belong in
more than one package. For instance, modules dealing with HTML parsing, do
they belong in the Internet package, or in the Text package? (Yes, you don't
have those packages here... but I do. :^)

I think it will be very hard to make a package hierarchy that is logical to
all users (or most of them). That does not mean your proposal isn't good...
I'm just wondering how these problems will be solved. It makes me wish that
there was a different approach to grouping of modules... something non-
hierarchical.

Veel liefs,

+ Hans Nowak (Zephyr Falcon)
+ Homepage (under construction): http://www.cuci.nl/~hnowak/
+ You call me a masterless man. You are wrong. I am my own master.
+ May a dead president sit on your hard body!

PROPOSAL: Packages in Python lib [ In reply to ]

paul at prescod

May 2, 1999, 7:42 AM

Post #3 of 13 (1952 views)

Permalink

Hans Nowak wrote:
>
> I think it will be very hard to make a package hierarchy that is logical to
> all users (or most of them). That does not mean your proposal isn't good...
> I'm just wondering how these problems will be solved. It makes me wish that
> there was a different approach to grouping of modules... something non-
> hierarchical.

For human beings, hierarchy is the simplest organization after a flat
list. A hierarchy doesn't have to be perfect to be usable. Looking for
HTML? Check out the Internet package and then the formats package. I
already have to do that with the Python library anyhow -- sometimes I
don't know if a thing is in os or os.path, re or string, glob or fnmatch,
zlib or gzip, etc. You just have to look around.

Anyhow, the existence of hierarchy in the standard library does not
preclude a flat representation of it. The Python library documentation is
already hierarchical but you can find HTML in the table of contents, the
module list or the index.

Overall, I don't see it as a big problem. I mean if it became a problem we
could easily add a concept of symlinks to the package mechanism but I
don't think that we need that.

--
Paul Prescod - ISOGEN Consulting Engineer speaking for only himself
http://itrc.uwaterloo.ca/~papresco

"Microsoft spokesman Ian Hatton admits that the Linux system would have
performed better had it been tuned."
"Future press releases on the issue will clearly state that the research
was sponsored by Microsoft."
http://www.itweb.co.za/sections/enterprise/1999/9904221410.asp

PROPOSAL: Packages in Python lib [ In reply to ]

pkoren at hex

May 2, 1999, 8:15 AM

Post #4 of 13 (1952 views)

Permalink

Hans Nowak wrote:
>
I have been trying
> myself to organize all kinds of modules in a package collection, and a
> problem that occurs quite often is, that a module can logically belong in
> more than one package. For instance, modules dealing with HTML parsing, do
> they belong in the Internet package, or in the Text package? (Yes, you don't
> have those packages here... but I do. :^)

Hmmmmm, it sounds like we need some conditional import mechanism. C uses
the preprocessor:

#ifndef themodule
#define themodule
...
#endif

I am too new to Python to hack the solution, but a conditional import
approach might work.

Regards,

Peter Koren

--
Remove the '.zap-this' from the email address to reach me.

"One may proceed from absurd premise to ridiculous conclusion with
impeccable
logic." -- Santayana

PROPOSAL: Packages in Python lib [ In reply to ]

petrilli at trump

May 2, 1999, 8:45 AM

Post #5 of 13 (1957 views)

Permalink

Peter A. Koren <pkoren@hex.net> wrote:
> Hans Nowak wrote:
>>
> I have been trying
>> myself to organize all kinds of modules in a package collection, and a
>> problem that occurs quite often is, that a module can logically belong in
>> more than one package. For instance, modules dealing with HTML parsing, do
>> they belong in the Internet package, or in the Text package? (Yes, you don't
>> have those packages here... but I do. :^)

> Hmmmmm, it sounds like we need some conditional import mechanism. C uses
> the preprocessor:

> #ifndef themodule
> #define themodule
> ...
> #endif

> I am too new to Python to hack the solution, but a conditional import
> approach might work.

From billions of modules I've written:

try:
import cPickle
pickle=cPickle
except ImportError:
import pickle

Is this what you meant? :-)

This gives me a 'pickle' object which refers to the C implementation if
available, and the regular python one if not.

Chris
--
| Christopher Petrilli ``Television is bubble-gum for
| petrilli@amber.org the mind.''-Frank Lloyd Wright

PROPOSAL: Packages in Python lib [ In reply to ]

paul at prescod

May 2, 1999, 2:58 PM

Post #6 of 13 (1953 views)

Permalink

"Peter A. Koren" wrote:
>
> Hans Nowak wrote:
> >
> I have been trying
> > myself to organize all kinds of modules in a package collection, and a
> > problem that occurs quite often is, that a module can logically belong in
> > more than one package. For instance, modules dealing with HTML parsing, do
> > they belong in the Internet package, or in the Text package? (Yes, you don't
> > have those packages here... but I do. :^)
>
> Hmmmmm, it sounds like we need some conditional import mechanism.

I interpreted Hans' message as reporting the usual problem with building
ontologies -- the world isn't strictly hierarchical and people wouldn't
agree on the hierarchy even if it was. I think you interpred his message
differently, because I don't see how a conditional import mechanism would
help this problem.

--
Paul Prescod - ISOGEN Consulting Engineer speaking for only himself
http://itrc.uwaterloo.ca/~papresco

"Microsoft spokesman Ian Hatton admits that the Linux system would have
performed better had it been tuned."
"Future press releases on the issue will clearly state that the research
was sponsored by Microsoft."
http://www.itweb.co.za/sections/enterprise/1999/9904221410.asp

PROPOSAL: Packages in Python lib [ In reply to ]

news at helen

May 3, 1999, 4:24 AM

Post #7 of 13 (1949 views)

Permalink

Hans Nowak wrote in message <199905020817.KAA29168@axil.hvision.nl>...

>I think it will be very hard to make a package hierarchy that is logical to
>all users (or most of them). That does not mean your proposal isn't good...
>I'm just wondering how these problems will be solved. It makes me wish
that
>there was a different approach to grouping of modules... something non-
>hierarchical.

I love Paul's Idea. It worked for Java, so why not for Python? But please...
no Poldermodels
here. Just let Guido pickup the idea and implement it.
Hierarchy not perfect? Ok, but it's better than linear organisation.

Ilja

PROPOSAL: Packages in Python lib [ In reply to ]

phd at sun

May 3, 1999, 4:25 AM

Post #8 of 13 (1952 views)

Permalink

Hello!

While I love the idea very much, this is what I do not love in
particular:

On Sat, 1 May 1999, Paul Prescod wrote:
> Serial:
> pickle
> cpickle

I am expecting to find Rs-232 modules in Serial, but instead found
modules that serialize objects.
Rename Serial to... may be Pickling? Marshalling? SerialObjects?

Any objections?

Oleg.
----
Oleg Broytmann National Research Surgery Centre http://sun.med.ru/~phd/
Programmers don't die, they just GOSUB without RETURN.

PROPOSAL: Packages in Python lib [ In reply to ]

spamfranke at bigfoot

May 3, 1999, 4:28 AM

Post #9 of 13 (1956 views)

Permalink

On Sat, 1 May 1999 19:57:43 GMT, Paul Prescod <paul@prescod.net> wrote:

>I would like to suggest a package organization for the standard modules.
>The package directories should all be in sys.path so that older modules
>would continue to work.

I think it's about time to "packagize" the library every time I try to find
something inside the lib/ dir.
However, could this approach possibly result in increased startup times?
sys.path gets pretty long with your suggested approach.
On an OS with symlinks there should be no problem to maintain both structures,
but I see no easy way without them. I guess the hierarchical structure could
could be faked by customizing the import right now, but I'm not clever enough to
figure it out.
Generally, it would be nice to have a simple way in Python to allow different
views on the same module set (I could imagine locally interpreted .pth files as
placeholders inside a package), but maybe I just don't know it is possible
already.

Stefan

PROPOSAL: Packages in Python lib [ In reply to ]

paul at prescod

May 3, 1999, 5:36 AM

Post #10 of 13 (1953 views)

Permalink

Oleg Broytmann wrote:
>
> Hello!
>
> While I love the idea very much, this is what I do not love in
> particular:
>
> On Sat, 1 May 1999, Paul Prescod wrote:
> > Serial:

It was just an example hierarchy. I'm sure that Guido would have his own
ideas about the right organization and naming.

Someone wrote:
> I love Paul's Idea. It worked for Java, so why not for Python?

Comparing it to Java is a good way to kill the idea!

>But please... no Poldermodels here.

I'm not familar with that word.

Stefan Franke wrote:
> Generally, it would be nice to have a simple way in Python to
> allow different views on the same module set (I could imagine
> locally interpreted .pth files as placeholders inside a package), but
> maybe I just don't know it is possible already.

I don't think such a thing exists but it is a good idea. Overall it would
be nice if there were ways through text files (and not environment
variables) to configure the mapping between files and packages.

--
Paul Prescod - ISOGEN Consulting Engineer speaking for only himself
http://itrc.uwaterloo.ca/~papresco

Diplomatic term: "We had a frank exchange of views."
Translation: Negotiations stopped just short of shouting and
table-banging.
(Brills Content, Apr. 1999)

PROPOSAL: Packages in Python lib [ In reply to ]

M.Faassen at vet

May 3, 1999, 6:36 AM

Post #11 of 13 (1953 views)

Permalink

Paul Prescod wrote:

Ilja Heitlager wrote:
> >But please... no Poldermodels here.
>
> I'm not familar with that word.

A couple of years ago the Netherlands was apparently doing exceptionally
well economically compared to most of the rest of Europe (it still seems
to do rather well); this was attributed to the Dutch 'polder model' of
economics (a polder is land reclaimed from the sea, though the model has
nothing to do with that as far as I know). The polder model is based
mainly on consensus between the various parties in the Dutch economy;
employers, employees, unions, government, etc. Wages were kept
relatively low for instance without massive social unrest. The idea is
that this was a positive impulse for the Dutch economy.

Of course, before all that there were the Swedish model that was very
popular, and I've heard of the Rineland model and the Anglo-Saxon model
too, and I don't know enough about economics to judge if there's much
truth or difference in any of these models.

So I think what Ilja meant to say 'no endless debates to reach consensus
here, let's just let Guido do his thing'.

Of course Guido can and should do his thing, but I think Guido in
general appreciates at least some debates about the possibilities and
consequencse of proposed changes, so that he can take the best ones and
go back in time to put them in Python. :) In addition Guido isn't an
expert on everything (gasp!), so that's another way the newsgroup (and
the SIGs etc) can be useful to him.

After all, Usenet is in a large part about endless debates anyway, even
though consensus is sometimes far away. comp.lang.python isn't that bad
consensuswise anyway -- we manage to avoid quite a bit of the flaming
and rudeness that occurs in many other newsgroups, has been my
experience.

comp.lang.python-the-poldermodelgroup-of-usenet?-ly yours,

Martijn

PROPOSAL: Packages in Python lib [ In reply to ]

greg.ewing at compaq

May 4, 1999, 4:59 PM

Post #12 of 13 (1950 views)

Permalink

Oleg Broytmann wrote:
>
> Rename Serial to... may be Pickling? Marshalling? SerialObjects?

Persistence?

Greg

PROPOSAL: Packages in Python lib [ In reply to ]

aahz at netcom

May 5, 1999, 3:38 PM

Post #13 of 13 (1952 views)

Permalink

In article <372B5CB7.66DBB930@prescod.net>,
Paul Prescod <paul@prescod.net> wrote:
>
> [[ liberal sniparoo ]]
>
>Unix:
>
>SGI:
> al
> ...
>
>SunOS:
> ...
>
>Windows:
> ...
>
>OS:
> tempfile
> cmd

I think it makes more sense for the individual OS pieces to be grouped
under OS:

OS:
tempfile
cmd
Unix:
...
SunOS:
...
Windows:
...
--
--- Aahz (@netcom.com)

Hugs and backrubs -- I break Rule 6 <*> http://www.rahul.net/aahz/
Androgynous poly kinky vanilla queer het

"In the end, outside of spy agencies, people are far too trusting and
willing to help." -- Ira Winkler