Mailing List Archive

LOAD_NAME & classes
There's a bug report on SF that notes there is a difference between:

def f(x):
class Foo:
x = x

and

x = 1
class Foo:
x = x

The latter works because the class namespace uses LOAD_NAME and finds
a global binding for x. The fact that x is also a local is invisible
to LOAD_NAME.

The former fails because x is none of locals, globals, or builtins,
although it is bound in an enclosing scope. LOAD_NAME knows nothing
about free variables, however, so it just blows up.

Do we want to do anything about this apparent inconsistency?

LOAD_NAME is obviously necessary to make stuff like exec work at all,
and after a recent bug fix, it evens works as documented for nested
scopes. But the docs for a class namespace (there aren't any, right?)
don't suggest there is anything special going on.

I imagine it would be possible to stop using LOAD_NAME for classes,
but I'm not sure if that's a good thing. It could presumably break
code that relies on LOAD_NAME's old fashioned search. It also seems
like a non trivial amount of work because we'd need a new LOAD/STORE
combo that only searched a locals dict. (Maybe that's not so hard.)

I think module namespaces also use LOAD_NAME, but it's not clear why.
Isn't a module's locals the same as its globals? If so, LOAD_GLOBAL
could be used for all names.

Any opinion on whether this is worth fixing? And is it a bug fix or a
new feature?

Jeremy
Re: LOAD_NAME & classes [ In reply to ]
> There's a bug report on SF that notes there is a difference between:
>
> def f(x):
> class Foo:
> x = x
>
> and
>
> x = 1
> class Foo:
> x = x
>
> The latter works because the class namespace uses LOAD_NAME and finds
> a global binding for x. The fact that x is also a local is invisible
> to LOAD_NAME.
>
> The former fails because x is none of locals, globals, or builtins,
> although it is bound in an enclosing scope. LOAD_NAME knows nothing
> about free variables, however, so it just blows up.
>
> Do we want to do anything about this apparent inconsistency?
>
> LOAD_NAME is obviously necessary to make stuff like exec work at all,
> and after a recent bug fix, it evens works as documented for nested
> scopes. But the docs for a class namespace (there aren't any, right?)
> don't suggest there is anything special going on.
>
> I imagine it would be possible to stop using LOAD_NAME for classes,
> but I'm not sure if that's a good thing. It could presumably break
> code that relies on LOAD_NAME's old fashioned search. It also seems
> like a non trivial amount of work because we'd need a new LOAD/STORE
> combo that only searched a locals dict. (Maybe that's not so hard.)
>
> I think module namespaces also use LOAD_NAME, but it's not clear why.
> Isn't a module's locals the same as its globals? If so, LOAD_GLOBAL
> could be used for all names.
>
> Any opinion on whether this is worth fixing? And is it a bug fix or a
> new feature?

I just tried this in 2.1 (without nested scopes enabled) and there the
first example fails too. While it's slightly confusing, it's
consistent with the rule that class bodies don't play the nested
scopes game, and I think it shouldn't be fixed. Otherwise you'd have
the confusing issue that a function inside a class can't see the class
scope, but a class inside a function can see the function scope.
Better if neither can see the other.

--Guido van Rossum (home page: http://www.python.org/~guido/)
Re: LOAD_NAME & classes [ In reply to ]
>>>>> "GvR" == Guido van Rossum <guido@python.org> writes:

> def f(x):
> class Foo:
> x = x
>
> and
>
> x = 1
> class Foo:
> x = x
>

GvR> I just tried this in 2.1 (without nested scopes enabled) and
GvR> there the first example fails too.

I think you misunderstood my original explanation. With the current
implementations, the first example fails and the second example
works. I think the bug is that the second example works.

GvR> While it's slightly confusing, it's consistent with the rule
GvR> that class bodies don't play the nested scopes game, and I
GvR> think it shouldn't be fixed. Otherwise you'd have the
GvR> confusing issue that a function inside a class can't see the
GvR> class scope, but a class inside a function can see the function
GvR> scope. Better if neither can see the other.

None of the documentation suggests that class and functions don't see
each other. Rather, it says the free variables are resolved in the
nearest enclosing function scope. The current implementation supports
resolution of free variables in class scope, e.g.

def f(x):
class Foo:
print x

>>> f(3)
3

I think the problem is with x = x, which ought to be an error if x is
a local and x is unbound. The code will succeed, nested or otherwise,
if x happens to be a module global. The confusion is that the code
will not succeed if x is defined in an enclosing, non-top-level
scope.

Jeremy
Re: LOAD_NAME & classes [ In reply to ]
>>>>> "GvR" == Guido van Rossum <guido@python.org> writes:

>> I think you misunderstood my original explanation. With the
>> current implementations, the first example fails and the second
>> example works. I think the bug is that the second example works.

GvR> But it has always worked, and I definitely don't want to break
GvR> this now.

I don't want to break this now either (although I don't think either
of my messages betrayed that intent). I think it's a wart we live
with, but I'd love to fix it when we get to Py3K. (Will we ever?)

GvR> I don't understand why you want to break it? That would
GvR> definitely break working code!

And there's the rub.

>> I think the problem is with x = x, which ought to be an error if
>> x is a local and x is unbound. The code will succeed, nested or
>> otherwise, if x happens to be a module global. The confusion is
>> that the code will not succeed if x is defined in an enclosing,
>> non-top-level scope.

GvR> Nested scopes in Python will never be completely intuitive,
GvR> because they break the illusion that "there's only runtime" (as
GvR> Samuele summarized it a while ago).

Wasn't that about macros?

GvR> That said, I can't decide whether it's better to make the first
GvR> example work, or reject the bug report. I'm tempted to say
GvR> "the ice is thin here, don't play with fire" or something
GvR> cryptic like that, and leave it.

In the bug tracker (532860), I suggested "The result of a reference to
an unbound local name in a class block is undefined." With a note
that says this is for backwards compatibility and another that
explains how CPython is currently implemented.

I have a separate bug report about ref man documentation for all
this. The ref man documents that world the way I'd like it to be,
nicely glossing over all the messy details like LOAD_NAME. I think we
need an extra section (4.1.1) that explains some of the messy details.

Jeremy
Re: LOAD_NAME & classes [ In reply to ]
> >>>>> "GvR" == Guido van Rossum <guido@python.org> writes:
>
> > def f(x):
> > class Foo:
> > x = x
> >
> > and
> >
> > x = 1
> > class Foo:
> > x = x
>
> GvR> I just tried this in 2.1 (without nested scopes enabled) and
> GvR> there the first example fails too.
>
> I think you misunderstood my original explanation. With the current
> implementations, the first example fails and the second example
> works. I think the bug is that the second example works.

But it has always worked, and I definitely don't want to break this
now.

I don't understand why you want to break it? That would definitely
break working code!

> GvR> While it's slightly confusing, it's consistent with the rule
> GvR> that class bodies don't play the nested scopes game, and I
> GvR> think it shouldn't be fixed. Otherwise you'd have the
> GvR> confusing issue that a function inside a class can't see the
> GvR> class scope, but a class inside a function can see the function
> GvR> scope. Better if neither can see the other.
>
> None of the documentation suggests that class and functions don't see
> each other. Rather, it says the free variables are resolved in the
> nearest enclosing function scope.

Oops, I misremembered how this works.

> The current implementation supports resolution of free variables in
> class scope, e.g.
>
> def f(x):
> class Foo:
> print x
>
> >>> f(3)
> 3
>
> I think the problem is with x = x, which ought to be an error if x is
> a local and x is unbound. The code will succeed, nested or otherwise,
> if x happens to be a module global. The confusion is that the code
> will not succeed if x is defined in an enclosing, non-top-level
> scope.

Nested scopes in Python will never be completely intuitive, because
they break the illusion that "there's only runtime" (as Samuele
summarized it a while ago).

That said, I can't decide whether it's better to make the first
example work, or reject the bug report. I'm tempted to say "the ice
is thin here, don't play with fire" or something cryptic like that,
and leave it.

--Guido van Rossum (home page: http://www.python.org/~guido/)
Re: LOAD_NAME & classes [ In reply to ]
On Mon, Apr 22, 2002, Guido van Rossum wrote:
>
> That said, I can't decide whether it's better to make the first
> example work, or reject the bug report. I'm tempted to say "the ice
> is thin here, don't play with fire" or something cryptic like that,
> and leave it.

"On thin ice, play with fire you should not."
--
Aahz (aahz@pythoncraft.com) <*> http://www.pythoncraft.com/

What if there were no rhetorical questions?
Re: LOAD_NAME & classes [ In reply to ]
>>>>> "A" == aahz <aahz@pythoncraft.com> writes:

A> "On thin ice, play with fire you should not."

A> What if there were no rhetorical questions?

I think I should read this in a Frank Oz voice, right?

Somehow, though, it makes me think: "but sometimes my arms bend back."

can-guido-pronounce-his-own-name-ly y'rs,
Jeremy
Re: LOAD_NAME & classes [ In reply to ]
> >>>>> "GvR" == Guido van Rossum <guido@python.org> writes:
>
> >> I think you misunderstood my original explanation. With the
> >> current implementations, the first example fails and the second
> >> example works. I think the bug is that the second example works.
>
> GvR> But it has always worked, and I definitely don't want to break
> GvR> this now.
>
> I don't want to break this now either (although I don't think either
> of my messages betrayed that intent).

The fact that you called it a bug (rather than a problem) made me
think this.

> I think it's a wart we live
> with, but I'd love to fix it when we get to Py3K. (Will we ever?)

There probably won't ever be an opportunity to start over and be
truly incompatible. I don't think I'm up for a herculean effort like
the Perl 6 folks are making.

> GvR> I don't understand why you want to break it? That would
> GvR> definitely break working code!
>
> And there's the rub.
>
> >> I think the problem is with x = x, which ought to be an error if
> >> x is a local and x is unbound. The code will succeed, nested or
> >> otherwise, if x happens to be a module global. The confusion is
> >> that the code will not succeed if x is defined in an enclosing,
> >> non-top-level scope.
>
> GvR> Nested scopes in Python will never be completely intuitive,
> GvR> because they break the illusion that "there's only runtime" (as
> GvR> Samuele summarized it a while ago).
>
> Wasn't that about macros?

That was the context, but it nicely summarizes a lot about Python, and
about scripting languages in general. It may even be an explanation
of the success of scripting languages: typical programmers probably
have a much better mental model of runtime than of compile time.

Variable scope already violates the "only runtime" rule, and this has
caused endless complaints and misunderstandings. Before nested
scopes, I've toyed with the idea of making LOAD_FAST fall through
(conceptually) to LOAD_GLOBAL when the local slot is NULL, just to
shut up the complaints. Then this example (whose failure always
surprises newcomers) would work:

x = 10
def f():
print x
x = 12
f()

But now that we have nested scopes, I don't think this would be
feasible -- the fallback would have to dynamically inspect all
surrounding scopes.

> GvR> That said, I can't decide whether it's better to make the first
> GvR> example work, or reject the bug report. I'm tempted to say
> GvR> "the ice is thin here, don't play with fire" or something
> GvR> cryptic like that, and leave it.
>
> In the bug tracker (532860), I suggested "The result of a reference to
> an unbound local name in a class block is undefined." With a note
> that says this is for backwards compatibility and another that
> explains how CPython is currently implemented.

At least this leaves the door open to fixing it my way. :-)

> I have a separate bug report about ref man documentation for all
> this. The ref man documents that world the way I'd like it to be,
> nicely glossing over all the messy details like LOAD_NAME. I think we
> need an extra section (4.1.1) that explains some of the messy details.

You can never go wrong by documenting something. (Famous last words,
I expect. :-)

--Guido van Rossum (home page: http://www.python.org/~guido/)
RE: LOAD_NAME & classes [ In reply to ]
[Guido]
> ...
> Variable scope already violates the "only runtime" rule, and this has
> caused endless complaints and misunderstandings.

Any scope rules do: you can't win here. You can only choose which specific
rules cause endless complaints and misunderstandings. Pure dynamic scoping
is the easiest for newcomers to understand, and it's a Bad Idea despite
that.

> Before nested scopes, I've toyed with the idea of making LOAD_FAST
> fall through (conceptually) to LOAD_GLOBAL when the local slot is NULL,
> just to shut up the complaints. Then this example (whose failure
> always surprises newcomers)

It surprises some newcomers, plus Andrew Kuchling <wink>.

> would work:
>
> x = 10
> def f():
> print x
> x = 12
> f()

But this still wouldn't:

x = 10
def f():
print x

def g():
x = 12
f()

g()

You can't explain why that doesn't print 12 without insisting that different
textual regions have different scopes. Newbie complaints and
misunderstandings follow. Under an "only runtime" conception, the only
obvious behavior is that x is bound to 12 at the time f() is called, so of
course it should print 12. It takes a certain sophistication to understand
that the name "x" means different things in different parts of the file.
Once that's understood, the behavior of your example is no longer a mystery;
but before it's understood, your example is only one of countless
confusions. Indeed, the single most frequent newbie confusion I've heard
over the last year is variations on "OK, I did 'import math' like you said,
but 'sin(x)' *still* gives a NameError!".

> But now that we have nested scopes, I don't think this would be
> feasible -- the fallback would have to dynamically inspect all
> surrounding scopes.

As above, dynamic scoping is what newbies expect -- but it would be a great
disservice to give it to them.

pons-asinorum-ly y'rs - tim
Re: LOAD_NAME & classes [ In reply to ]
[Guido]
> > would work:
> >
> > x = 10
> > def f():
> > print x
> > x = 12
> > f()

[Tim]
> But this still wouldn't:
>
> x = 10
> def f():
> print x
>
> def g():
> x = 12
> f()
>
> g()
>
> You can't explain why that doesn't print 12 without insisting that
> different textual regions have different scopes. Newbie complaints
> and misunderstandings follow. Under an "only runtime" conception,
> the only obvious behavior is that x is bound to 12 at the time f()
> is called, so of course it should print 12. It takes a certain
> sophistication to understand that the name "x" means different
> things in different parts of the file. Once that's understood, the
> behavior of your example is no longer a mystery; but before it's
> understood, your example is only one of countless confusions.

I think this is a red herring, and not an argument against what I
proposed. The "only runtime" rules doesn't require dynamic scopes (I
agree that dynamic scopes would be bad). Dynamic scopes, and your
example, mix up the call context with the definition context. My
example takes the definition context, and applies the "is x defined in
this scope?" test at runtime instead of at compile time. Very
different!

--Guido van Rossum (home page: http://www.python.org/~guido/)
RE: LOAD_NAME & classes [ In reply to ]
[Guido]
> I think this is a red herring, and not an argument against what I
> proposed.

Sure it does <wink>. It's one example of the larger point that *any* scope
rules confuse newbies. Lexical scoping doesn't come naturally except to
born Schemers, and indeed didn't come naturally to language designers either
(e.g., early LISPs all had dynamic scoping). So it goes.

> The "only runtime" rules doesn't require dynamic scopes (I agree that
> dynamic scopes would be bad). Dynamic scopes, and your example, mix up
> the call context with the definition context. My example takes the
> definition context, and applies the "is x defined in this scope?" test
> at runtime instead of at compile time. Very different!

Rewrite it trivially:

def f():
print x
x = 12

x = 10 # moved the line down
f()

I don't know what you mean by "definition context" (and neither will a
newbie), but any guess I'm likely to come up with wouldn't include the idea
that "x = 10" is now in f's definition context. You would have to spell out
that there are three namespaces at work here, and that what "x" means in f
depends on the dynamic state of two of them, and simply can't be known in
general without running the program. If that's not dynamic scoping, it's
too close for me to believe a distinction is worth making.

BTW, I consider Python's treatment of global vs builtin namespaces dynamic
scoping too, and it's nothing but trouble that globals can mask and unmask
builtins dynamically. I'd rather make globals and builtins act more like
locals now than make locals act more like they're dynamically scoped.

BTW2, I see plenty of UnboundLocalErrors in my own code, and some of those
have occurred when the same name is also in use as a global. It's always
been a logic error due to forgetting to initialize a local, and usually due
to "moving code up" in an editor. It sure wouldn't be doing me any favor to
let such code silently pick up whatever crap happened to be bound to the
same-named global; UnboundLocalError is a fine bug-catcher.
Re: LOAD_NAME & classes [ In reply to ]
> > The "only runtime" rules doesn't require dynamic scopes (I agree
> > that dynamic scopes would be bad). Dynamic scopes, and your
> > example, mix up the call context with the definition context. My
> > example takes the definition context, and applies the "is x
> > defined in this scope?" test at runtime instead of at compile
> > time. Very different!
>
> Rewrite it trivially:
>
> def f():
> print x
> x = 12
>
> x = 10 # moved the line down
> f()
>
> I don't know what you mean by "definition context" (and neither will
> a newbie), but any guess I'm likely to come up with wouldn't include
> the idea that "x = 10" is now in f's definition context. You would
> have to spell out that there are three namespaces at work here, and
> that what "x" means in f depends on the dynamic state of two of
> them, and simply can't be known in general without running the
> program. If that's not dynamic scoping, it's too close for me to
> believe a distinction is worth making.

I seem to have trouble explaining what I meant.

Long ago, before I introduced LOAD_FAST and friends, Python had
something that for want of a better term I'll call "lexical scoping
with dynamic lookup". It did a dynamic lookup in a (max 3 deep: local
/ global / builtin) stack of namespaces, but the set of namespaces was
determined by the compiler. This does not have the problems of
dynamic scoping (the caller's stack frame can't cause trouble). But
it also doesn't have the problem of the current strict static scoping.

I like the older model better than the current model (apart from
nested scopes) and I believe that the "only runtime" rule explains why
the old model is more attractive: it doesn't require you to think of
the compiler scanning all the code of your function looking for
definitions of names. You can think of the interpreter pretty much
executing code as it sees it. You have to have a model for name
lookup that requires a chaining of namespaces based on where a
function is defined, but that's all still purely runtime (it involves
executing the def statement).

This requires some sophistication for a newbie to understand, but it's
been explained successfully for years, and the explanation would be
easier without UnboundLocalError.

Note that it explains your example above completely: the namespace
where f is defined contains a definition of x when f is called, and
thus the search stops there.

> BTW, I consider Python's treatment of global vs builtin namespaces
> dynamic scoping too, and it's nothing but trouble that globals can
> mask and unmask builtins dynamically. I'd rather make globals and
> builtins act more like locals now than make locals act more like
> they're dynamically scoped.

Um, that's not what I'd call dynamic scoping. It's dynamic lookup.
It's trouble for a compiler that wants to optimize builtins, but the
semantic model is nice and simple and easy to explain with the "only
runtime" rule.

> BTW2, I see plenty of UnboundLocalErrors in my own code, and some of
> those have occurred when the same name is also in use as a global.
> It's always been a logic error due to forgetting to initialize a
> local, and usually due to "moving code up" in an editor. It sure
> wouldn't be doing me any favor to let such code silently pick up
> whatever crap happened to be bound to the same-named global;
> UnboundLocalError is a fine bug-catcher.

You definitely have a point there -- like with most irritating errors,
the coin has two sides. I don't know which side would waste more
time. (When UnboundLocalError was still spelled as NameError, I'd bet
on the latter.)

--Guido van Rossum (home page: http://www.python.org/~guido/)
RE: LOAD_NAME & classes [ In reply to ]
[Guido]
> I seem to have trouble explaining what I meant.

I know, and I confess I'm giving you a hard time. There's a point to that
too: uniqueness also imposes costs on newbies and/or newcomers. Across the
world of programming languages now, dynamic scoping and lexical scoping are
"almost entirely *it*". For example, the Perl spelling of the running
example here does work the way you intend, but the explanation in Perl is
full-blown dynamic scoping:

sub g {
print "$x\n"; # prints 12 -- "full-blown dynamic scoping"
}

sub f {
print "$x\n"; # prints 10
local($x) = 12;
&g();
}

$x = 10;
&f();
print "$x\n"; # prints 10

Once you make f print 10, you're on that path as far as anyone coming from
any other language can tell at first glance (or even second and third). If
you go on to make g print 10 too, it's inexplicable via reference to how any
other language works. If there were a huge payback for "being different"
here, cool, but the only real payback I see is letting newbies avoid
learning how lexical scoping works, and only for a little while.

> Long ago, before I introduced LOAD_FAST and friends, Python had
> something that for want of a better term I'll call "lexical scoping
> with dynamic lookup".

I'm old enough to remember this <wink>.

> It did a dynamic lookup in a (max 3 deep: local / global / builtin)
> stack of namespaces, but the set of namespaces was determined by the
> compiler. This does not have the problems of dynamic scoping (the
> caller's stack frame can't cause trouble). But it also doesn't have
> the problem of the current strict static scoping.

Nor its advantages, including better error detection, and ease of
transferring hard-won knowledge among other lexically scoped languages.

> I like the older model better than the current model (apart from
> nested scopes) and I believe that the "only runtime" rule explains why
> the old model is more attractive: it doesn't require you to think of
> the compiler scanning all the code of your function looking for
> definitions of names. You can think of the interpreter pretty much
> executing code as it sees it. You have to have a model for name
> lookup that requires a chaining of namespaces based on where a
> function is defined, but that's all still purely runtime (it involves
> executing the def statement).
>
> This requires some sophistication for a newbie to understand, but it's
> been explained successfully for years, and the explanation would be
> easier without UnboundLocalError.
>
> Note that it explains your example above completely: the namespace
> where f is defined contains a definition of x when f is called, and
> thus the search stops there.

Does it scale?

x = 0

def f(i):
if i & 4:
x = 10
def g(i):
if i & 2:
x = 20
def h(i):
if i & 1:
x = 30
print x
h(i)
g(i)

f(3)

I can look at that today and predict with confidence that h() will either
print 30 (if and only if i is odd), or raise an exception. This is from
purely local analysis of h's body -- it doesn't matter that it's nested, and
it's irrelvant what the enclosing functions look like or do. That's a great
aid to writing correct code. If the value of x h sees *may* come from h, or
from g, or from f, or from the module scope instead, depending on i's
specific value at the time f is called, there's a lot more to think about.

I could keep local+global straight in pre-1.0 Python, although I never got
used to the inability to write nested functions that could refer to each
other (perhaps you've forgotten how many times you had to explain that one,
and how difficult it was to get across?). Now that Python has full-blown
nested scopes, the namespace interactions are potentially much more
convoluted, and the "purely local analysis" shortcut made possible by
everyone else's <wink> notion of lexical scoping becomes correspondingly
more valuable.

> ...
> Um, that's not what I'd call dynamic scoping. It's dynamic lookup.

I know -- the problem is that you're the only one in the world making this
distinction, and that makes it hard to maintain over time. If it had some
killer advantage ... but it doesn't seem to. When Python switched to
"strict local" names before 1.0, I don't recall anyone complaining -- if
there was a real advantage to dynamic lookup at the local scope, it appeared
to have escaped Python's users <wink>. I'll grant that it did make exec and
"import *" more predictable in corner cases.

> It's trouble for a compiler that wants to optimize builtins, but the
> semantic model is nice and simple and easy to explain with the "only
> runtime" rule.

Dynamic scoping is also easy to explain, but it doesn't scale. I'm afraid
dynamic lookup doesn't scale either. You should have stuck with Python's
original two-level namespace, you know <0.9 wink>.

the-builtins-didn't-count-ly y'rs - tim
RE: LOAD_NAME & classes [ In reply to ]
Tim Peters <tim.one@comcast.net>:

> Lexical scoping doesn't come naturally except to
> born Schemers, and indeed didn't come naturally to language designers either
> (e.g., early LISPs all had dynamic scoping).

I doubt whether the early designers of Lisp explicitly designed
it to have dynamic scoping. Coming from lambda calculus, they
were probably *thinking* of lexical scoping; it's just that
the implementation they chose had some unfortunate side effects.

So, I conjecture that the *idea* of lexical scoping comes
naturally enough, but a correct implementation of it doesn't. :-)

By the way, even in dynamically scoped Lisp, there's no
equivalent of an UnboundLocalError -- to get a local variable
at all, you have to bind some initial value to it. So, using
Guido's terminology, early Lisp had dynamic scoping, but not
dynamic lookup. Confused enough yet?-)

Greg Ewing, Computer Science Dept, +--------------------------------------+
University of Canterbury, | A citizen of NewZealandCorp, a |
Christchurch, New Zealand | wholly-owned subsidiary of USA Inc. |
greg@cosc.canterbury.ac.nz +--------------------------------------+
RE: LOAD_NAME & classes [ In reply to ]
[Greg Ewing]
> I doubt whether the early designers of Lisp explicitly designed
> it to have dynamic scoping. Coming from lambda calculus, they
> were probably *thinking* of lexical scoping; it's just that
> the implementation they chose had some unfortunate side effects.

I think McCarthy may be a good source <wink>:

http://www-formal.stanford.edu/jmc/history/lisp/lisp.html

The distinction between dynamic and lexical scoping is clear-cut only in
hindsight. As McCarthy relates, Algol 60 later bumped into much the same
scoping surprises they did.

The treatment of free variables in formal logic is also much clearer in
hindsight, and was the source of many difficulties before a proper
definition of substitution was crafted. Cute: Curry and Schoenfinkel
(independently) invented combinatory logic to get rid of free variables (and
the difficulties that come with them) entirely, and decades later
combinatory logic overthrew the lambda calculus as the basis for
high-performance implementations of pure functional languages.

Applying a contemporary understanding of these things retroactively just
doesn't fly; whatever clarity we have now was a result of their struggles.

> ...
> By the way, even in dynamically scoped Lisp, there's no
> equivalent of an UnboundLocalError -- to get a local variable
> at all, you have to bind some initial value to it.

There were certainly NameErrors. All you needed to do was evaluate any old
function that referenced a free variable, at a time when no binding for that
variable was on the stack. You can't do this in Scheme, but it was dead
easy in LISP; it's harder in Common Lisp, but dynamic scoping can still be
gotten at there.

> So, using Guido's terminology, early Lisp had dynamic scoping, but
> not dynamic lookup. Confused enough yet?-)

Not as confused as McCarthy turned out to have been <wink>.
RE: LOAD_NAME & classes [ In reply to ]
Tim Peters <tim.one@comcast.net>:

> I think McCarthy may be a good source <wink>:
>
> http://www-formal.stanford.edu/jmc/history/lisp/lisp.html

From there:

The difficulty was that when an inner recursion occurred, the value of
car[x] wanted was the outer value, but the inner value was actually
used. In modern terminology, lexical scoping was wanted, and dynamic
scoping was obtained.
...
I must confess that I regarded this difficulty as just a bug and
expressed confidence that Steve Russell would soon fix it.

So it seems I was more or less right -- the semantics they
wanted was lexical scoping, effectively, even if they didn't
explicitly think about it.

Greg Ewing, Computer Science Dept, +--------------------------------------+
University of Canterbury, | A citizen of NewZealandCorp, a |
Christchurch, New Zealand | wholly-owned subsidiary of USA Inc. |
greg@cosc.canterbury.ac.nz +--------------------------------------+
RE: LOAD_NAME & classes [ In reply to ]
[Greg Ewing]
>> http://www-formal.stanford.edu/jmc/history/lisp/lisp.html

> From there:
>
> The difficulty was that when an inner recursion occurred, the value
> of car[x] wanted was the outer value, but the inner value was
> actually used. In modern terminology, lexical scoping was wanted,
> and dynamic scoping was obtained.
> ...
> I must confess that I regarded this difficulty as just a bug and
> expressed confidence that Steve Russell would soon fix it.

> So it seems I was more or less right -- the semantics they
> wanted was lexical scoping, effectively, even if they didn't
> explicitly think about it.

They didn't even have names for it, Greg -- note the "in modern terminology"
there. This is McCarthy reflecting on things that happened more than 20
years before he wrote the paper. It was so muddy at the time that this
paper was reduced to

Unfortunately, time did not permit writing an appendix giving the
history of the problem, and the interested reader is referred to
(Moses 1970) as a place to start. (David Park tells me that Patrick
Fischer also had a hand in developing the FUNARG device).

Part of the problem is that the Lisp interpreter was written in Lisp, and
contrary to hopes it didn't fully specify the semantics: it turned out that
what the Lisp implementation just happened to do for a thing sometimes
reflected directly in how the Lisp interpreter emulated that thing, and so
studying the interpreter code sheds no light on intent in such cases -- it's
a kind of "self-fulfilling algorithm" then. "Metacircular interpreters"
fell out of favor for specifying language semantics as a result. Modern
attempts to rehabilitate the idea are still stumbling over how to nail the
intended scoping semantics; see, e.g. (skip to Conclusions):

http://linux.rice.edu/~rahul/hbaker/MetaCircular.html
Re: LOAD_NAME & classes [ In reply to ]
Tim Peters <tim.one@comcast.net> writes:

> BTW2, I see plenty of UnboundLocalErrors in my own code, and some of
> those have occurred when the same name is also in use as a global.
> It's always been a logic error due to forgetting to initialize a
> local, and usually due to "moving code up" in an editor. It sure
> wouldn't be doing me any favor to let such code silently pick up
> whatever crap happened to be bound to the same-named global;
> UnboundLocalError is a fine bug-catcher.

Wouldn't it be nice if these were flagged at compile time! More work
for Jeremy <wink>.

Cheers,
M.

--
Strangely enough I saw just such a beast at the grocery store
last night. Starbucks sells Javachip. (It's ice cream, but that
shouldn't be an obstacle for the Java marketing people.)
-- Jeremy Hylton, 29 Apr 1997
Re: LOAD_NAME & classes [ In reply to ]
[Tim]
> There's a point to that too: uniqueness also imposes costs on
> newbies and/or newcomers. Across the world of programming languages
> now, dynamic scoping and lexical scoping are "almost entirely *it*".
[...]
> Nor its advantages, including better error detection, and ease of
> transferring hard-won knowledge among other lexically scoped
> languages.

But Python *is* unique in that it doesn't require declarations. (I've
got to admit that the Perl example saddened me. But then in Perl,
local variables are a recent invention. :-)

We've found before that this can go against what's common knowledge
for other language (e.g. integer division).

[...]
> Does it scale?
>
> x = 0
>
> def f(i):
> if i & 4:
> x = 10
> def g(i):
> if i & 2:
> x = 20
> def h(i):
> if i & 1:
> x = 30
> print x
> h(i)
> g(i)
>
> f(3)
>
> I can look at that today and predict with confidence that h() will
> either print 30 (if and only if i is odd), or raise an exception.
> This is from purely local analysis of h's body -- it doesn't matter
> that it's nested, and it's irrelvant what the enclosing functions
> look like or do. That's a great aid to writing correct code. If
> the value of x h sees *may* come from h, or from g, or from f, or
> from the module scope instead, depending on i's specific value at
> the time f is called, there's a lot more to think about.

Yup. But it also requires intricate knowledge of Python's rules,
which are different than any other language's rules. You simply can't
have a variable declaration inside an if statement in other languages
that extends to the entire function body -- either the scope would be
limited to the block it's in, or the syntax wouldn't allow it.

Python's original semantic model on the other hand, and the model
that's still used for globals at the global level, gives a clear
explanation: a namespace is implemented as a dictionary, and name
lookup searches a pre-set sequence of namespaces until it finds a hit.
The lexical scoping rule determines how namespaces are combined.
Doing the lookup at runtime is easier to understand than doing it at
compile time -- even if the compile version might catch more bugs.
But I'm repeating myself; I already said that in my previous message.

> I could keep local+global straight in pre-1.0 Python, although I
> never got used to the inability to write nested functions that could
> refer to each other (perhaps you've forgotten how many times you had
> to explain that one, and how difficult it was to get across?).

No; apart from you, most people were happy with the rule "nested
functions don't work".

> Now that Python has full-blown nested scopes, the namespace
> interactions are potentially much more convoluted, and the "purely
> local analysis" shortcut made possible by everyone else's <wink>
> notion of lexical scoping becomes correspondingly more valuable.

I don't know. Full-blown nested scopes make namespace interactions
more convoluted no matter *what* set of rules we pick. An alternative
implementation model (with associated subtly different semantics
semantics) would have been to create an explicit list of the dicts
involved in the name resolution for a particular function invocation;
we rejected that model because we wanted this to be (nearly) as fast
as locals, so we moved more of the analysis to compile time.

But by doing so, we introduced more of a dependency on the
programmer's ability to understand what happens at compile time, and
that breaks the "only runtime exists" illusion.

In PEP 267 Jeremy is exploring how to optimize access to globals
*without* changing the rules. The change to LOAD_FAST that I
considered before would have optimized access to locals without
changing the rules, and I still regret that I didn't think of that
when I created LOAD_FAST (even though you disagree): the "only
runtime" rule is helpful for a large class of programmers, not only
newbies, and I'm not sure that adding more and more cruft from truly
compiled languages to Python's *semantics* is a good idea. (Adding
compiler technology that doesn't change the rules is fine, of course,
if it helps optimizations or better diagnostics.)

> > ...
> > Um, that's not what I'd call dynamic scoping. It's dynamic lookup.
>
> I know -- the problem is that you're the only one in the world
> making this distinction, and that makes it hard to maintain over
> time.

You can say that, but that doesn't make it so, and it doesn't convince
me. The three-scope was gospel in the Python world, and many people
actively disliked adding nested scopes (some still do).

> If it had some killer advantage ... but it doesn't seem to.
> When Python switched to "strict local" names before 1.0, I don't
> recall anyone complaining -- if there was a real advantage to
> dynamic lookup at the local scope, it appeared to have escaped
> Python's users <wink>. I'll grant that it did make exec and "import
> *" more predictable in corner cases.

Well, we gave them a big reason not to complain: this was the
singlemost biggest speedup in Python's history. But the rules were
definitely harder to explain, because for the first time we had to
explain a second compiler pass.

> > It's trouble for a compiler that wants to optimize builtins, but the
> > semantic model is nice and simple and easy to explain with the "only
> > runtime" rule.
>
> Dynamic scoping is also easy to explain, but it doesn't scale. I'm
> afraid dynamic lookup doesn't scale either. You should have stuck
> with Python's original two-level namespace, you know <0.9 wink>.

We need more than a single example to decide which rules bites worse
for large programs. Deep nesting is not common; long functions are.
And there the common annoyance is that a change in line 150 can break
the code in line 2 of the function.

--Guido van Rossum (home page: http://www.python.org/~guido/)
RE: LOAD_NAME & classes [ In reply to ]
[Guido van Rossum]
>
> We need more than a single example to decide which rules bites worse
> for large programs. Deep nesting is not common; long functions are.
> And there the common annoyance is that a change in line 150 can break
> the code in line 2 of the function.

I'm not exactly sure what you mean by this. Can you share an example? (Not
necessarily 150+ lines long, of course.) Thanks.

---
Patrick K. O'Brien
Orbtech
Re: LOAD_NAME & classes [ In reply to ]
> [Guido van Rossum]
> > We need more than a single example to decide which rules bites worse
> > for large programs. Deep nesting is not common; long functions are.
> > And there the common annoyance is that a change in line 150 can break
> > the code in line 2 of the function.
>
> I'm not exactly sure what you mean by this. Can you share an
> example? (Not necessarily 150+ lines long, of course.) Thanks.

It's a classic. Before we had UnboundLocalError (i.e. in 1.5.2 and
before) this was a common problem on c.l.py:

x = "a global"

def f():
print x # user thinks this should print the global
# 2000 lines of unrelated code
for x in "some sequence": # doesn't realize this overrides x
do_something_with(x)

Calling f() would raise NameError: x, which caused lots of confusion.

We added UnboundLocalError th make it clearer what's going on (so at
least the experienced c.l.py users would know right away where the
problem was :-), but still requires you to know about something
obscure that's going on at compile time (the compiler scanning your
entire function for variable definitions).

--Guido van Rossum (home page: http://www.python.org/~guido/)
RE: LOAD_NAME & classes [ In reply to ]
[Guido van Rossum]
> > > And there the common annoyance is that a change in line 150 can break
> > > the code in line 2 of the function.
> >
> > I'm not exactly sure what you mean by this. Can you share an
> > example? (Not necessarily 150+ lines long, of course.) Thanks.
>
> It's a classic. Before we had UnboundLocalError (i.e. in 1.5.2 and
> before) this was a common problem on c.l.py:
>
> x = "a global"
>
> def f():
> print x # user thinks this should print the global
> # 2000 lines of unrelated code
> for x in "some sequence": # doesn't realize this overrides x
> do_something_with(x)
>
> Calling f() would raise NameError: x, which caused lots of confusion.
>
> We added UnboundLocalError th make it clearer what's going on (so at
> least the experienced c.l.py users would know right away where the
> problem was :-), but still requires you to know about something
> obscure that's going on at compile time (the compiler scanning your
> entire function for variable definitions).

Okay. I think I'm following you, but I want to be certain about the
statement that "a change in line 150 can break the code in line 2 of the
function." Using your example, the function f() works, but only because of a
"fortunate" side effect of sorts. So if the code was later changed to "for y
in ..." then f() no longer works. But the example is fundamentally flawed to
begin with. Proper code shouldn't have to worry that "a change in line 150
can break the code in line 2 of the function." Right? Or am I still missing
something?

I've never felt that I needed to know about something obscure going on at
compile time in order to write decent Python code. Maybe I'm just being
paranoid, but this whole discussion just struck me as odd because I can't
recall ever having any problem like this. For the most part Python does
exactly what I think it should do. And when it doesn't, I'm usually wrong.

---
Patrick K. O'Brien
Orbtech
Re: LOAD_NAME & classes [ In reply to ]
> > [Guido van Rossum]
> > > We need more than a single example to decide which rules bites worse
> > > for large programs. Deep nesting is not common; long functions are.
> > > And there the common annoyance is that a change in line 150 can break
> > > the code in line 2 of the function.
> >
[Patrick]
> > I'm not exactly sure what you mean by this. Can you share an
> > example? (Not necessarily 150+ lines long, of course.) Thanks.
>
[Guido]
> It's a classic. Before we had UnboundLocalError (i.e. in 1.5.2 and
> before) this was a common problem on c.l.py:
>
> x = "a global"
>
> def f():
> print x # user thinks this should print the global
> # 2000 lines of unrelated code
> for x in "some sequence": # doesn't realize this overrides x
> do_something_with(x)
>
> Calling f() would raise NameError: x, which caused lots of confusion.
>
> We added UnboundLocalError th make it clearer what's going on (so at
> least the experienced c.l.py users would know right away where the
> problem was :-), but still requires you to know about something
> obscure that's going on at compile time (the compiler scanning your
> entire function for variable definitions).
>
So, the problem is the implicit local declaration that assumed when the
compiler detects a binding lower down the function's code than the first
reference to it, coupled with their function-wide scope. It is compounded by
the fact that although the analysis is performed at compile time the error
is only reported at run time.

Might it make more sense to issue a warning at compile time to the effect
that a variable is being used before it's assigned? How completely are
(re)bindings detected by the static analysis? Can you necessarily guarantee
that a LOAD_FAST is always looking at a local name? [.You'll understand I'm
not familiar with the details of code generation].

Seems to me the real problem here is explaining (by the interpreter's
behavior, rather than in the documentation ;-) the scope of locals and how a
name is determined to be local. It might help beginners if there was some
really easy way to get a fuller explanation of an error message. Not sure
how best that could be retrofitted, but it's better than code breakage.

Unfortunately the scoping rules are now "in the wild", so it's not possible
to change things too radically without mucho pain for those who have already
relied on them.

regards
Steve
--
home: http://www.holdenweb.com/
Python Web Programming:
http://pydish.holdenweb.com/pwp/
Re: LOAD_NAME & classes [ In reply to ]
> Okay. I think I'm following you, but I want to be certain about the
> statement that "a change in line 150 can break the code in line 2 of the
> function." Using your example, the function f() works, but only because of a
> "fortunate" side effect of sorts.


Why is the fact that this works:

x = 12
def f():
print x

a "fortunate" side effect? That's how the language works!

> So if the code was later changed to "for y
> in ..." then f() no longer works. But the example is fundamentally flawed to
> begin with. Proper code shouldn't have to worry that "a change in line 150
> can break the code in line 2 of the function." Right? Or am I still missing
> something?

I think you've got it backwards. My complaint is that if f() above
eventually grew 150 lines of unrelated code ending with an unrelated
assignment to a local variable x, the breakage would show up at an
unexpected point. Except for this one, it's hard to make a change at
the *tail* of a function that breaks something at the beginning!

> I've never felt that I needed to know about something obscure going
> on at compile time in order to write decent Python code. Maybe I'm
> just being paranoid, but this whole discussion just struck me as odd
> because I can't recall ever having any problem like this. For the
> most part Python does exactly what I think it should do. And when it
> doesn't, I'm usually wrong.

This particular form of breakage was a common error reported on c.l.py
and to help at python.org until we added UnboundLocalError to make the
diagnostic cleaner. Maybe that's all that's needed; getting a
NameError when you see this:

x = 12
def f():
print x # <--- NameError raised here!
...150 line of code you didn't think could cause the problem...

was very disturbing, causing people to look for places where x was
deleted from the global namespace later in the program.

--Guido van Rossum (home page: http://www.python.org/~guido/)
Re: LOAD_NAME & classes [ In reply to ]
On Tuesday 23 April 2002 05:30 pm, Steve Holden wrote:
...
> Might it make more sense to issue a warning at compile time to the effect
> that a variable is being used before it's assigned? How completely are

Hard to make sure, so, it would always be "MIGHT be used before
assignment". E.g.,

def f():
for i in range(6):
if i>0: print x,
x = str(i)

this is OK... no UnboundNameError. Now, change the if's guard to
if SolveTuringHaltingProblem(i): print x,
to see why it's hard to make sure about this at compile time.

A warning might still be OK of course, even though it may need to
be worded in wishy-washy "might" terms -- this IS peculiar usage.


Alex

1 2 3  View All