Mailing List Archive

Christian> Well, the (bad) effect of this patch is that you cannot run
Christian> PythonWin any longer unless Mark either supplies an updated
Christian> distribution, or one corrects the two barfing Scintilla
Christian> support scripts by hand.

Zope has the multi-arg append problem as well.

It wasn't clear to me where all the problems resided. I've been thinking
mostly about the problem in Python code. Thankfully Tim released
checkappend last night! Even after getting rid of all the cases it found I
was still getting some flaky tracebacks which suggest to me that some C code
makes the same sort of assumptions.

I've never labored under the assumption that append would take multiple
arguments. I was mildly surprised to find an instance of this crop up in my
own code...

Skip Montanaro | http://www.mojam.com/
skip@mojam.com | http://www.musi-cal.com/

On Tue, 29 Feb 2000, Christian Tismer wrote:
> Greg Stein wrote:
> > +1 on breaking it now, rather than deferring it Yet Again.
> >
> > IMO, there has been plenty of warning, and there is plenty of time to
> > correct the software.
> >
> > I'm +0 on adding a warning architecture to Python to support issuing a
> > warning/error when .append is called with multiple arguments.
>
> Well, the (bad) effect of this patch is that you cannot run
> PythonWin any longer unless Mark either supplies an updated
> distribution, or one corrects the two barfing Scintilla
> support scripts by hand.

Yes, but there is no reason to assume this won't happen.

Why don't we simply move forward with the assumption that PythonWin and
Scintilla will be updated? If we stand around pointing at all the uses of
append that are incorrect and claim that is why we can't move forward,
then we won't get anywhere. Instead, let's just *MOVE* and see that
software authors update accordingly. It isn't like it is a difficult
change to make. Heck, PythonWin and Scintilla could be updated within the
week and re-released. *WAY* ahead of the 1.6 release.

> Bad for me, since I'm building Stackless Python against 1.5.2+,
> and that means the users will see PythonWin barf when installing SLP.

If you're building a system using an interim release of Python, then I
think you need to take responsibility for that. If you don't want those
people to have problems, then you can back out the list.append change. Or
you can release patches to PythonWin. I don't think the Python world at
large should be hampered because somebody is using an unstable/interim
version of Python. Again: we couldn't move forward.

> Adding a warning instead of raising an exception would be nice IMHO,
> since the warning could probably contain the file name and line
> number to change, and I would leave my users with this easy task.

Yes, this would be nice. But somebody has to take the time to code it up.
The warning won't appear out of nowhere...

Cheers,
-g

--
Greg Stein, http://www.lyra.org/

> Why don't we simply move forward with the assumption that PythonWin and
> Scintilla will be updated?

Done :-)

However, I think dropping it now _is_ a little heavy handed. I decided to
do a wider search and found a few in, eg, Sam Rushings calldll based ODBC
package.

Personally, I would much prefer a warning now, and drop it later. _Then_ we
can say we have made enough noise about it. It would only be 2 years ago
that I became aware that this "feature" of append was not a feature at all -
up until then I used it purposely, and habits are sometimes hard to change
:-)

MArk.

On Wed, 1 Mar 2000, Mark Hammond wrote:
> > Why don't we simply move forward with the assumption that PythonWin and
> > Scintilla will be updated?
>
> Done :-)

hehe...

> However, I think dropping it now _is_ a little heavy handed. I decided to
> do a wider search and found a few in, eg, Sam Rushings calldll based ODBC
> package.
>
> Personally, I would much prefer a warning now, and drop it later. _Then_ we
> can say we have made enough noise about it. It would only be 2 years ago
> that I became aware that this "feature" of append was not a feature at all -
> up until then I used it purposely, and habits are sometimes hard to change
> :-)

What's the difference between a warning and an error? If you're running a
program and it suddenly spits out a warning about a misuse of list.append,
I'd certainly see that as "the program did something unexpected; that is
an error."

But this is all moot. Guido has already said that we would be amenable to
a warning/error infrastructure which list.append could use. His
description used some awkward sentences, so I'm not sure (without spending
some brain cycles to parse the email) exactly what his desired defaults
and behavior are. But hey... the possibility is there, and is just waiting
for somebody to code it.

IMO, Guido has left an out for people that are upset with the current
hard-line approach. One of those people just needs to spend a bit of time
coming up with a patch :-)

And yes, Guido is also the Benevolent Dictator and can certainly have his
mind changed, so people can definitely continue pestering him to back away
from the hard-line approach...

Cheers,
-g

--
Greg Stein, http://www.lyra.org/

On Tue, 29 Feb 2000, Greg Stein wrote:
>
> What's the difference between a warning and an error? If you're running a
> program and it suddenly spits out a warning about a misuse of list.append,
> I'd certainly see that as "the program did something unexpected; that is
> an error."

A big, big difference. Perhaps to one of us, it's the minor inconvenience
of reading the error message and inserting a couple of parentheses in the
appropriate file -- but to the end user, it's the difference between the
program working (albeit noisily) and *not* working. When the program throws
an exception and stops, it is safe to say most users will declare it broken
and give up.

We can't assume that they're going to be able to figure out what to edit
(or be brave enough to try) just by reading the error message... or even
what interpreter flag to give, if errors (rather than warnings) are the
default behaviour.


-- ?!ng

On Wed, 1 Mar 2000, Mark Hammond wrote:

> > Why don't we simply move forward with the assumption that PythonWin and
> > Scintilla will be updated?
>
> Done :-)
>
> However, I think dropping it now _is_ a little heavy handed. I decided to
> do a wider search and found a few in, eg, Sam Rushings calldll based ODBC
> package.
>
> Personally, I would much prefer a warning now, and drop it later. _Then_ we
> can say we have made enough noise about it. It would only be 2 years ago
> that I became aware that this "feature" of append was not a feature at all -
> up until then I used it purposely, and habits are sometimes hard to change
> :-)

I agree with mark. Why the sudden rush?? It seems to me to be unfair to
make such a change - one that will break peoples code - without advanced
warning, which typically is handled by a deprecation period. There *are*
going to be people who won't be informed of the change in the short span
of less than a single release. Just because it won't cause you pain isn't
a good reason to disregard the pain of those that will suffer,
particularly when you can do something relatively low-cost to avoid it.

Ken
klm@digicool.com

On Tue, 29 Feb 2000, Ken Manheimer wrote:
>...
> I agree with mark. Why the sudden rush?? It seems to me to be unfair to
> make such a change - one that will break peoples code - without advanced
> warning, which typically is handled by a deprecation period. There *are*
> going to be people who won't be informed of the change in the short span
> of less than a single release. Just because it won't cause you pain isn't
> a good reason to disregard the pain of those that will suffer,
> particularly when you can do something relatively low-cost to avoid it.

Sudden rush?!?

Mark said he knew about it for a couple years. Same here. It was a long
while ago that .append()'s semantics were specified to "no longer" accept
multiple arguments.

I see in the HISTORY file, that changes were made to Python 1.4 (October,
1996) to avoid calling append() with multiple arguments.

So, that is over three years that append() has had multiple-args
deprecated. There was probably discussion even before that, but I can't
seem to find something to quote. Seems like plenty of time -- far from
rushed.

Cheers,
-g

--
Greg Stein, http://www.lyra.org/

On Tue, 29 Feb 2000, Greg Stein wrote:

> On Tue, 29 Feb 2000, Ken Manheimer wrote:
> >...
> > I agree with mark. Why the sudden rush?? It seems to me to be unfair to
> > make such a change - one that will break peoples code - without advanced
> > warning, which typically is handled by a deprecation period. There *are*
> > going to be people who won't be informed of the change in the short span
> > of less than a single release. Just because it won't cause you pain isn't
> > a good reason to disregard the pain of those that will suffer,
> > particularly when you can do something relatively low-cost to avoid it.
>
> Sudden rush?!?
>
> Mark said he knew about it for a couple years. Same here. It was a long
> while ago that .append()'s semantics were specified to "no longer" accept
> multiple arguments.
>
> I see in the HISTORY file, that changes were made to Python 1.4 (October,
> 1996) to avoid calling append() with multiple arguments.
>
> So, that is over three years that append() has had multiple-args
> deprecated. There was probably discussion even before that, but I can't
> seem to find something to quote. Seems like plenty of time -- far from
> rushed.

None the less, for those practicing it, the incorrectness of it will be
fresh news. I would be less sympathetic with them if there was recent
warning, eg, the schedule for changing it in the next release was part of
the current release. But if you tell somebody you're going to change
something, and then don't for a few years, you probably need to renew the
warning before you make the change. Don't you think so? Why not?

Ken
klm@digicool.com

Software configuration management is HARD. Every sudden backwards
incompatible change (warranted or not) makes it harder. Mutli-arg append
is not hurting anyone as much as a sudden change to it would. It would
be better to leave append() alone and publicize its near-term removal
rather than cause random, part-time supported modules to stop working
because their programmers may be too busy to update them right now.

So no, I'm not stepping up to do it. But I'm also saying that the better
"lazy" option is to put something in a prominent place in the
documentation and otherwise leave it alone.

<aside>
As far as I am concerned, a formal warning-based deprecation mechanism
is necessary for Python's continued evolution. Perhaps we can even
expose the deprecation flag to the programmer so we can say:

if deprecation:
print "This module isn't supported anymore."

if deprecation:
print "Use method FooEx instead."

If we had a deprecation mechanism, maybe introducing new keywords would
not be quite so painful. Version x deprecates, version y adds the
keyword. Mayhap we should also deprecate implicit truncating integral
division while we are at it...
</aside>

--
Paul Prescod - ISOGEN Consulting Engineer speaking for himself
"The calculus and the rich body of mathematical analysis to which it
gave rise made modern science possible, but it was the algorithm that
made possible the modern world."
- from "Advent of the Algorithm" David Berlinski
http://www.opengroup.com/mabooks/015/0151003386.shtml

> Software configuration management is HARD. Every sudden backwards
> incompatible change (warranted or not) makes it harder. Mutli-arg append
> is not hurting anyone as much as a sudden change to it would. It would
> be better to leave append() alone and publicize its near-term removal
> rather than cause random, part-time supported modules to stop working
> because their programmers may be too busy to update them right now.

I'm tired of this rhetoric. It's not like I'm changing existing
Python installations retroactively. I'm planning to release a new
version of Python which no longer supports certain long-obsolete and
undocumented behavior. If you maintain a non-core Python module, you
should test it against the new release and fix anything that comes up.
This is why we have an alpha and beta test cycle and even before that
the CVS version. If you are a Python user who depends on a 3rd party
module, you need to find out whether the new version is compatible
with the 3rd party code you are using, or whether there's a newer
version available that solves the incompatibility.

There are people who still run Python 1.4 (really!) because they
haven't upgraded. I don't have a problem with that -- they don't get
much support, but it's their choice, and they may not need the new
features introduced since then. I expect that lots of people won't
upgrade their Python 1.5.2 to 1.6 right away -- they'll wait until the
other modules/packages they need are compatible with 1.6. Multi-arg
append probably won't be the only reason why e.g. Digital Creations
may need to release an update to Zope for Python 1.6. Zope comes with
its own version of Python anyway, so they have control over when they
make the switch.

--Guido van Rossum (home page: http://www.python.org/~guido/)

On Tue, 29 Feb 2000, Ken Manheimer wrote:
>...
> None the less, for those practicing it, the incorrectness of it will be
> fresh news. I would be less sympathetic with them if there was recent
> warning, eg, the schedule for changing it in the next release was part of
> the current release. But if you tell somebody you're going to change
> something, and then don't for a few years, you probably need to renew the
> warning before you make the change. Don't you think so? Why not?

I agree.

Note that Guido posted a note to c.l.py on Monday. I believe that meets
your notification criteria.

Cheers,
-g

--
Greg Stein, http://www.lyra.org/

On Tue, 29 Feb 2000, Guido van Rossum wrote:
> I'm tired of this rhetoric. It's not like I'm changing existing
> Python installations retroactively. I'm planning to release a new
> version of Python which no longer supports certain long-obsolete and
> undocumented behavior. If you maintain a non-core Python module, you
> should test it against the new release and fix anything that comes up.
> This is why we have an alpha and beta test cycle and even before that
> the CVS version. If you are a Python user who depends on a 3rd party
> module, you need to find out whether the new version is compatible
> with the 3rd party code you are using, or whether there's a newer
> version available that solves the incompatibility.
>
> There are people who still run Python 1.4 (really!) because they
> haven't upgraded. I don't have a problem with that -- they don't get
> much support, but it's their choice, and they may not need the new
> features introduced since then. I expect that lots of people won't
> upgrade their Python 1.5.2 to 1.6 right away -- they'll wait until the
> other modules/packages they need are compatible with 1.6. Multi-arg
> append probably won't be the only reason why e.g. Digital Creations
> may need to release an update to Zope for Python 1.6. Zope comes with
> its own version of Python anyway, so they have control over when they
> make the switch.

I wholeheartedly support his approach. Just ask Mark Hammond :-) how many
times I've said "let's change the code to make it Right; people aren't
required to upgrade [and break their code]."

Of course, his counter is that people need to upgrade to fix other,
unrelated problems. So I relax and try again later :-). But I still
maintain that they can independently grab the specific fixes and leave the
other changes we make.

Maybe it is grey, but I think this change is quite fine. Especially given
Tim's tool.

Cheers,
-g

--
Greg Stein, http://www.lyra.org/

[Greg Stein]
> ...
> Maybe it is grey, but I think this change is quite fine. Especially given
> Tim's tool.

What the heck does Tim's one-eyed trouser snake have to do with this? I
know *it* likes to think it's the measure of all things, but, frankly, my
tool barely affects the world at all a mere two feet beyond its base <wink>.

tim-and-his-tool-think-the-change-is-a-mixed-thing-but-on-balance-
the-best-thing-ly y'rs - tim

Greg Stein wrote:
> Note that Guido posted a note to c.l.py on Monday. I believe that meets
> your notification criteria.

ahem. do you seriously believe that everyone in the
Python universe reads comp.lang.python?

afaik, most Python programmers don't.

...

so as far as I'm concerned, this was officially deprecated
with Guido's post. afaik, no official python documentation
has explicitly mentioned this (and the fact that it doesn't
explicitly allow it doesn't really matter, since the docs don't
explicitly allow the x[a, b, c] syntax either. both work in
1.5.2).

has anyone checked the recent crop of Python books,
btw? the eff-bot guide uses old syntax in two examples
out of 320. how about the others?

...

sigh. running checkappend over a 50k LOC application, I
just realized that it doesn't catch a very common append
pydiom.

how fun. even though 99% of all append calls are "legal",
this "minor" change will break every single application and
library we have :-(

oh, wait. xmlrpclib isn't affected. always something!

</F>

On Wed, 1 Mar 2000, Fredrik Lundh wrote:
> Greg Stein wrote:
> > Note that Guido posted a note to c.l.py on Monday. I believe that meets
> > your notification criteria.
>
> ahem. do you seriously believe that everyone in the
> Python universe reads comp.lang.python?
>
> afaik, most Python programmers don't.

Now you're simply taking my comments out of context. Not a proper thing to
do. Ken said that he wanted notification along certain guidelines. I said
that I believed Guido's post did just that. Period.

Personally, I think it is fine. I also think that a CHANGES file that
arrives with 1.6 that points out the incompatibility is also fine.

>...
> sigh. running checkappend over a 50k LOC application, I
> just realized that it doesn't catch a very common append
> pydiom.

And which is that? Care to help out? Maybe just a little bit? Or do you
just want to talk about how bad this change is? :-(

Cheers,
-g

--
Greg Stein, http://www.lyra.org/

On Wed, 1 Mar 2000, Tim Peters wrote:
> [Greg Stein]
> > ...
> > Maybe it is grey, but I think this change is quite fine. Especially given
> > Tim's tool.
>
> What the heck does Tim's one-eyed trouser snake have to do with this? I
> know *it* likes to think it's the measure of all things, but, frankly, my
> tool barely affects the world at all a mere two feet beyond its base <wink>.
>
> tim-and-his-tool-think-the-change-is-a-mixed-thing-but-on-balance-
> the-best-thing-ly y'rs - tim

Heh. Now how is one supposed to respond to *that* ??!

All right. Fine. +3 cool points go to Tim.

:-)

--
Greg Stein, http://www.lyra.org/

Greg Stein <gstein@lyra.org> wrote:
> On Wed, 1 Mar 2000, Fredrik Lundh wrote:
> > Greg Stein wrote:
> > > Note that Guido posted a note to c.l.py on Monday. I believe that meets
> > > your notification criteria.
> >
> > ahem. do you seriously believe that everyone in the
> > Python universe reads comp.lang.python?
> >
> > afaik, most Python programmers don't.
>
> Now you're simply taking my comments out of context. Not a proper thing to
> do. Ken said that he wanted notification along certain guidelines. I said
> that I believed Guido's post did just that. Period.

my point was that most Python programmers won't
see that notification. when these people download
1.6 final and find that all theirs apps just broke, they
probably won't be happy with a pointer to dejanews.

> And which is that? Care to help out? Maybe just a little bit?

this rather common pydiom:

append = list.append
for x in something:
append(...)

it's used a lot where performance matters.

> Or do you just want to talk about how bad this change is? :-(

yes, I think it's bad. I've been using Python since 1.2,
and no other change has had the same consequences
(wrt. time/money required to fix it)

call me a crappy programmer if you want, but I'm sure
there are others out there who are nearly as bad. and
lots of them won't be aware of this change until some-
one upgrades the python interpreter on their server.

</F>

[/F]
> ...
> so as far as I'm concerned, this was officially deprecated
> with Guido's post. afaik, no official python documentation
> has explicitly mentioned this (and the fact that it doesn't
> explicitly allow it doesn't really matter, since the docs don't
> explicitly allow the x[a, b, c] syntax either. both work in
> 1.5.2).

The "Subscriptions" section of the Reference Manual explicitly allows for

dict[a, b, c]

and explicitly does not allow for

sequence[a, b, c]

The "Mapping Types" section of the Library Ref does not explicitly allow for
it, though, and if you read it as implicitly allowing for it (based on the
Reference Manual's clarification of "key" syntax), you would also have to
read the Library Ref as allowing for

dict.has_key(a, b, c)

Which 1.5.2 does allow, but which Guido very recently patched to treat as a
syntax error.

> ...
> sigh. running checkappend over a 50k LOC application, I
> just realized that it doesn't catch a very common append
> pydiom.

[And, later, after prodding by GregS]

> this rather common pydiom:
>
> append = list.append
> for x in something:
> append(...)

This limitation was pointed out in checkappend's module docstring. Doesn't
make it any easier for you to swallow, but I needed to point out that you
didn't *have* to stumble into this the hard way <wink>.

> how fun. even though 99% of all append calls are "legal",
> this "minor" change will break every single application and
> library we have :-(
>
> oh, wait. xmlrpclib isn't affected. always something!

What would you like to do, then? The code will be at least as broken a year
from now, and probably more so -- unless you fix it. So this sounds like an
indirect argument for never changing Python's behavior here. Frankly, I
expect you could fix the 50K LOC in less time than it took me to write this
naggy response <0.50K wink>.

embrace-change-ly y'rs - tim

[Tim. needing sleep]
> dict.has_key(a, b, c)
>
> Which 1.5.2 does allow, but which Guido very recently patched to
> treat as a syntax error.

No, a runtime error. haskeynanny.py, anyone?

not-me-ly y'rs - tim

Fredrik Lundh wrote:
>
> Greg Stein <gstein@lyra.org> wrote:
> > On Wed, 1 Mar 2000, Fredrik Lundh wrote:
> > > Greg Stein wrote:
> > > > Note that Guido posted a note to c.l.py on Monday. I believe that meets
> > > > your notification criteria.
> > >
> > > ahem. do you seriously believe that everyone in the
> > > Python universe reads comp.lang.python?
> > >
> > > afaik, most Python programmers don't.
> >
> > Now you're simply taking my comments out of context. Not a proper thing to
> > do. Ken said that he wanted notification along certain guidelines. I said
> > that I believed Guido's post did just that. Period.
>
> my point was that most Python programmers won't
> see that notification. when these people download
> 1.6 final and find that all theirs apps just broke, they
> probably won't be happy with a pointer to dejanews.

Dito. Anyone remember the str(2L) == '2' change, BTW ?
That one will cost lots of money in case someone implemented
an eShop using the common str(2L)[:-1] idiom...

There will need to be a big warning sign somewhere that
people see *before* finding the download link. (IMHO, anyways.)

> > And which is that? Care to help out? Maybe just a little bit?
>
> this rather common pydiom:
>
> append = list.append
> for x in something:
> append(...)
>
> it's used a lot where performance matters.

Same here. checkappend.py doesn't find these (a great tool BTW,
thanks Tim; I noticed that it leaks memory badly though).

> > Or do you just want to talk about how bad this change is? :-(
>
> yes, I think it's bad. I've been using Python since 1.2,
> and no other change has had the same consequences
> (wrt. time/money required to fix it)
>
> call me a crappy programmer if you want, but I'm sure
> there are others out there who are nearly as bad. and
> lots of them won't be aware of this change until some-
> one upgrades the python interpreter on their server.

--
Marc-Andre Lemburg
______________________________________________________________________
Business: http://www.lemburg.com/
Python Pages: http://www.lemburg.com/python/

Tim Peters wrote:
> The "Subscriptions" section of the Reference Manual explicitly allows for
>
> dict[a, b, c]
>
> and explicitly does not allow for
>
> sequence[a, b, c]

I'd thought we'd agreed that nobody reads the
reference manual ;-)

> What would you like to do, then?

more time to fix it, perhaps? it's surely a minor
code change, but fixing it can be harder than
you think (just witness Gerrit's bogus patches)

after all, python might be free, but more and more
people are investing lots of money in using it [1].

> The code will be at least as broken a year
> from now, and probably more so -- unless you fix it.

sure. we've already started. but it's a lot of work,
and it's quite likely that it will take a while until we
can be 100% confident that all the changes are pro-
perly done.

(not all software have a 100% complete test suite that
simply says "yes, this works" or "no, it doesn't")

</F>

1) fwiw, some poor soul over here posted a short note
to the pythonworks mailing, mentioning that we've now
fixed the price. a major flamewar erupted, and my mail-
box is now full of mail from unknowns telling me that I
must be a complete moron that doesn't understand that
Python is just a toy system, which everyone uses just be-
cause they cannot afford anything better...

Fredrik Lundh writes:
>yes, I think it's bad. I've been using Python since 1.2,
>and no other change has had the same consequences
>(wrt. time/money required to fix it)

There are more things in 1.6 that might require fixing existing code:
str(2L) returning '2', the int/long changes, the Unicode changes, and
if it gets added, garbage collection -- and bugs caused by those
changes might not be catchable by a nanny. IMHO it's too early to
point at the .append() change as breaking too much existing code;
there may be changes that break a lot more. I'd wait and see what
happens once the 1.6 alphas become available; if c.l.p is filled with
shrieks and groans, GvR might decide to back the offending change out.
(Or he might not...)

--
A.M. Kuchling http://starship.python.net/crew/amk/
I have no skills with machines. I fear them, and because I cannot help
attributing human qualities to them, I suspect that they hate me and will kill
me if they can.
-- Robertson Davies, "Reading"

On Tue, 29 Feb 2000, Greg Stein wrote:

> On Tue, 29 Feb 2000, Ken Manheimer wrote:
> >...
> > None the less, for those practicing it, the incorrectness of it will be
> > fresh news. I would be less sympathetic with them if there was recent
> > warning, eg, the schedule for changing it in the next release was part of
> > the current release. But if you tell somebody you're going to change
> > something, and then don't for a few years, you probably need to renew the
> > warning before you make the change. Don't you think so? Why not?
>
> I agree.
>
> Note that Guido posted a note to c.l.py on Monday. I believe that meets
> your notification criteria.

Actually, by "part of the current release", i meant having the
deprecation/impending-deletion warning in the release notes for the
release before the one where the deletion happens - saying it's being
deprecated now, will be deleted next time around.

Ken
klm@digicool.com

I mean, you tell one guy it's blue. He tells his guy it's brown, and it
lands on the page sorta purple. Wavy Gravy/Hugh Romney

Andrew M. Kuchling wrote:
> There are more things in 1.6 that might require fixing existing code:
> str(2L) returning '2', the int/long changes, the Unicode changes, and
> if it gets added, garbage collection -- and bugs caused by those
> changes might not be catchable by a nanny.

hey, you make it sound like "1.6" should really be "2.0" ;-)

</F>