Mailing List Archive

Can't localize lexical variable $x at ...
Why can't we? I think I might have known the answer once, but if I did
I no longer recall it.

I think of local() as "temporarily replace the value at this lvalue
with a new value, restore it at the end of the current lexical scope".
This is hugely useful in many situations that would otherwise require
more and slower code with much more opportunity for error.

Is there a technical reason why it would not be possible to implement?
Or is it that we're worried it would be confusing for users? Or is it
only that way back when (around 5.0, I guess) nobody thought it would
be useful?

Is it something we would ever consider changing?

Hugo
Re: Can't localize lexical variable $x at ... [ In reply to ]
Ovid via perl5-porters <perl5-porters@perl.org> wrote:
:On Tuesday, 25 January 2022, 15:25:54 CET, <hv@crypt.org> wrote:
:
:> Why can't we? I think I might have known the answer once, but if I did
:> I no longer recall it.
:
:You remember how Randall always said "I have an article about that?"
:
:Nowadays, it's replaced by "LeoNerd has already written the code to do that." :)
:
:https://metacpan.org/pod/Syntax::Keyword::Dynamically

Super, I'll have to try that out - and it also answers the question of
whether it is (now) technically possible.

I remain interested in the answers to the rest of my questions.

Hugo
Re: Can't localize lexical variable $x at ... [ In reply to ]
On Tuesday, 25 January 2022, 15:25:54 CET, <hv@crypt.org> wrote:

> Why can't we? I think I might have known the answer once, but if I did
> I no longer recall it.

Hugo,

You remember how Randall always said "I have an article about that?"

Nowadays, it's replaced by "LeoNerd has already written the code to do that." :)

https://metacpan.org/pod/Syntax::Keyword::Dynamically

Best,
Ovid
-- 
IT consulting, training, specializing in Perl, databases, and agile development
http://www.allaroundtheworld.fr/. 

Buy my book! - http://bit.ly/beginning_perl
Re: Can't localize lexical variable $x at ... [ In reply to ]
yb@rslinux.fun wrote:
:On Tue, Jan 25, 2022 at 01:20:17PM +0000, hv@crypt.org wrote:
:> I think of local() as "temporarily replace the value at this lvalue
:> with a new value, restore it at the end of the current lexical scope".
:> This is hugely useful in many situations that would otherwise require
:> more and slower code with much more opportunity for error.
:
:Out of curiosity, is there any reason not to simply shadow it: my $x = ...?

In this case, it's an inline recursive function of which the relevant
parts look a bit like:

sub do_stuff {
my $prod = 1;
my $recurse;
$recurse = sub {
...
for my $new (@interesting_values) {
local $prod = $prod * $new; # aspirational
$recurse->();
}
};
$recurse->();
}

.. so shadowing doesn't help. Alternatives such as a) making it a
one-element array, b) passing it as an argument or c) manually
saving and restoring are all possible, but all likely to be slower,
more error-prone, and add complexity to already-complex code.

The real code's aim is to find new values for https://oeis.org/A292580.
It's very much a work-in-progress, but if you're interested look at
https://github.com/hvds/seq/blob/master/divrep/oul#L167 for current
implementation (as one-element array).

Hugo
Re: Can't localize lexical variable $x at ... [ In reply to ]
On Tue, Jan 25, 2022 at 01:20:17PM +0000, hv@crypt.org wrote:
> I think of local() as "temporarily replace the value at this lvalue
> with a new value, restore it at the end of the current lexical scope".
> This is hugely useful in many situations that would otherwise require
> more and slower code with much more opportunity for error.

Out of curiosity, is there any reason not to simply shadow it: my $x = ...?
Re: Can't localize lexical variable $x at ... [ In reply to ]
On Tue, Jan 25, 2022 at 9:25 AM <hv@crypt.org> wrote:

> Why can't we? I think I might have known the answer once, but if I did
> I no longer recall it.
>
> I think of local() as "temporarily replace the value at this lvalue
> with a new value, restore it at the end of the current lexical scope".
> This is hugely useful in many situations that would otherwise require
> more and slower code with much more opportunity for error.
>
> Is there a technical reason why it would not be possible to implement?
> Or is it that we're worried it would be confusing for users? Or is it
> only that way back when (around 5.0, I guess) nobody thought it would
> be useful?
>

I might be making this up but my understanding is the technical reason is
that local replaces the variable, not the value; which works fine with a
package variable that is always enumerable, or an element of an aggregate,
but not with a lexical variable.

-Dan
Re: Can't localize lexical variable $x at ... [ In reply to ]
demerphq <demerphq@gmail.com> wrote:
:On Tue, 25 Jan 2022, 23:43 , <hv@crypt.org> wrote:
:> In this case, it's an inline recursive function of which the relevant
:> parts look a bit like:
:>
:> sub do_stuff {
:> my $prod = 1;
:> my $recurse;
:> $recurse = sub {
:> ...
:> for my $new (@interesting_values) {
:> local $prod = $prod * $new; # aspirational
:> $recurse->();
:> }
:> };
:> $recurse->();
:> }
:>
:
:Just a minor nit, maybe not relevant, but that will leak.
[...]
:Solving the leak this causes is the reason we created __SUB__

Ah good point - not hugely relevant in this case (the outer sub is called
at most once), but I had forgotten about __SUB__.

:I think the reason it's not allowed was local $x predates my $x. And it was
:defined to operate on globals. But you could get the same result with a
:global. Just rearrange the clauses a bit. Eg, add an our $prod outside the
:outer sub. Then move and change your my $prod to right before the initial
:call to $recurse as a local $prod.
:
:You could fix the leakage and use globals for $prod by changing the recurse
:to
:
:our $prod;
:local *recurse = sub {};
:local $prod=1;
:recurse();

Just changing the inner call to '__SUB__->()' is enough to fix the leak,
right?

We avoid using globals for good reasons, I wouldn't want to use them
for this unless there really was no alternative.

Hugo
Re: Can't localize lexical variable $x at ... [ In reply to ]
On Tue, 25 Jan 2022 13:20:17 +0000
hv@crypt.org wrote:

> Why can't we? I think I might have known the answer once, but if I did
> I no longer recall it.
>
> I think of local() as "temporarily replace the value at this lvalue
> with a new value, restore it at the end of the current lexical scope".
> This is hugely useful in many situations that would otherwise require
> more and slower code with much more opportunity for error.

People often think of `local` as being equivalent to saving the old
value of a variable, assigning a new value, and arranging to assign the
new value back in. Perhaps equivalent to this using our new `defer`
blocks:

{
my $saved_VAR = $VAR;
defer { $VAR = $saved_VAR; }
$VAR = $new;

...
}

That's not really how it works at all though - don't forget that
`local` is a very old Perl feature that massively predates `my`
variables. In fact, `local` isn't about values but about variables.

What `local` really does is temporarily moves a *variable* out of the
way, assigning a new one into its place, to be restored back again.
Variables themselves can only be moved around where they exist by
reference - as members of the symbol table (i.e. `our` variables), or
elements of aggregate structures (arrays and hashes). Because regular
lexical scalars do not have this one-layer-distant referencing effect,
they cannot be moved out of the way by `local`.

This distinction isn't often visible, but it does come up for example
when you consider `tie`, or other magics applied to variables.

> Is there a technical reason why it would not be possible to implement?
> Or is it that we're worried it would be confusing for users? Or is it
> only that way back when (around 5.0, I guess) nobody thought it would
> be useful?
>
> Is it something we would ever consider changing?

Yes and no.

Since it's detected at compiletime, it would be quite easy for the
parser to implement a totally different kind of `local`isation - one
that acts on values - when dealing with lexical variables. That may
confuse users, because it would act differently for the purposes of
tie/magic.

My preference would be to add a new keyword (because "local" is a
terrible name anyway) to have the effect of temporarily assigning a new
*value* to any lvalue expression, and arranging for its old value to be
restored again at end of scope. In fact I already wrote a module to
provide a `dynamically` keyword doing just that:

https://metacpan.org/pod/Syntax::Keyword::Dynamically

As compared `local` it has three key differences:

1) Can `dynamically` modify a regular lexical variable

2) Can `dynamically` modify any lvalue-returning function or method.
Very handy for accessor-type methods on objects:

dynamically $logger->level = LOG_DEBUG;

3) Interacts with the `async/await` syntax provided by
Future::AsyncAwait, to properly swap the values around when context
switching:

async sub f {
dynamically $logger->level = LOG_DEBUG;

await some_operation();

$logger->print( "Still in debug level here" );
}

Bringing this keyword into perl core would be easy enough for the first
two points, but the third would be a tough sell because it first
depends on having async/await implemented in core. Arguing that into
perl core at the moment would be a Fun and Exciting challenge ;)

--
Paul "LeoNerd" Evans

leonerd@leonerd.org.uk | https://metacpan.org/author/PEVANS
http://www.leonerd.org.uk/ | https://www.tindie.com/stores/leonerd/
Re: Can't localize lexical variable $x at ... [ In reply to ]
On Tue, 25 Jan 2022, 23:43 , <hv@crypt.org> wrote:

> yb@rslinux.fun wrote:
> :On Tue, Jan 25, 2022 at 01:20:17PM +0000, hv@crypt.org wrote:
> :> I think of local() as "temporarily replace the value at this lvalue
> :> with a new value, restore it at the end of the current lexical scope".
> :> This is hugely useful in many situations that would otherwise require
> :> more and slower code with much more opportunity for error.
> :
> :Out of curiosity, is there any reason not to simply shadow it: my $x =
> ...?
>
> In this case, it's an inline recursive function of which the relevant
> parts look a bit like:
>
> sub do_stuff {
> my $prod = 1;
> my $recurse;
> $recurse = sub {
> ...
> for my $new (@interesting_values) {
> local $prod = $prod * $new; # aspirational
> $recurse->();
> }
> };
> $recurse->();
> }
>

Just a minor nit, maybe not relevant, but that will leak.

I think the reason it's not allowed was local $x predates my $x. And it was
defined to operate on globals. But you could get the same result with a
global. Just rearrange the clauses a bit. Eg, add an our $prod outside the
outer sub. Then move and change your my $prod to right before the initial
call to $recurse as a local $prod.

You could fix the leakage and use globals for $prod by changing the recurse
to

our $prod;
local *recurse = sub {};
local $prod=1;
recurse();

Solving the leak this causes is the reason we created __SUB__

Yves
Re: Can't localize lexical variable $x at ... [ In reply to ]
"Paul \"LeoNerd\" Evans" <leonerd@leonerd.org.uk> wrote:
:On Tue, 25 Jan 2022 13:20:17 +0000
:hv@crypt.org wrote:
:
:> Why can't we? I think I might have known the answer once, but if I did
:> I no longer recall it.
:>
:> I think of local() as "temporarily replace the value at this lvalue
:> with a new value, restore it at the end of the current lexical scope".
:> This is hugely useful in many situations that would otherwise require
:> more and slower code with much more opportunity for error.
:
:People often think of `local` as being equivalent to saving the old
:value of a variable, assigning a new value, and arranging to assign the
:new value back in. Perhaps equivalent to this using our new `defer`
:blocks:
:
: {
: my $saved_VAR = $VAR;
: defer { $VAR = $saved_VAR; }
: $VAR = $new;
:
: ...
: }
:
:That's not really how it works at all though - don't forget that
:`local` is a very old Perl feature that massively predates `my`
:variables. In fact, `local` isn't about values but about variables.

It would probably be useful to have examples where that distinction
makes a difference - I didn't see any (explicit) mention of that in
the lengthy perlsub section about local, nor in S::K::D. Are there
cases where changing an existing valid local() invocation to use
dynamically() instead would change the behaviour? My guess would be
that if anything it would revolve around magic.

[...]
:My preference would be to add a new keyword (because "local" is a
:terrible name anyway) to have the effect of temporarily assigning a new
:*value* to any lvalue expression, and arranging for its old value to be
:restored again at end of scope. In fact I already wrote a module to
:provide a `dynamically` keyword doing just that:
:
: https://metacpan.org/pod/Syntax::Keyword::Dynamically

Yes, thanks; Ovid already mentioned this.

I confess I don't understand the choice of "dynamically" for the name:
"temporarily" would seem the obvious choice to me (and I'd very much
want to abbreviate that to "temp"). "dynamic" is a concept with a few
specific meanings in perl jargon and lots more outside of that: I think
few average perl users would instinctively guess the correct meaning
in this context.

:As compared `local` it has three key differences:
:
: 1) Can `dynamically` modify a regular lexical variable
:
: 2) Can `dynamically` modify any lvalue-returning function or method.
: Very handy for accessor-type methods on objects:
:
: dynamically $logger->level = LOG_DEBUG;
:
: 3) Interacts with the `async/await` syntax provided by
: Future::AsyncAwait, to properly swap the values around when context
: switching:
:
: async sub f {
: dynamically $logger->level = LOG_DEBUG;
:
: await some_operation();
:
: $logger->print( "Still in debug level here" );
: }
:
:Bringing this keyword into perl core would be easy enough for the first
:two points, but the third would be a tough sell because it first
:depends on having async/await implemented in core. Arguing that into
:perl core at the moment would be a Fun and Exciting challenge ;)

As a module it does not appear to require Async even be installed, could
it not continue to act as it does now? Or is the problem that it could
not then easily be adapted (via a new release) to unknown future async
changes? Or just that it's not the done thing for core code to have
special handling for non-core modules?

I would have thought the apparent need to recompile after installing
DMD would be the bigger problem.

Hugo
Re: Can't localize lexical variable $x at ... [ In reply to ]
On Tuesday, 25 January 2022, 18:05:43 CET, <hv@crypt.org> wrote:

> I confess I don't understand the choice of "dynamically" for the name:
> "temporarily" would seem the obvious choice to me (and I'd very much
> want to abbreviate that to "temp"). "dynamic" is a concept with a few
> specific meanings in perl jargon and lots more outside of that: I think
> few average perl users would instinctively guess the correct meaning
> in this context.

I also thought that `temp` is a much better choice for this keyword. I think it's hard to misunderstand.

Best,
Ovid
-- 
IT consulting, training, specializing in Perl, databases, and agile development
http://www.allaroundtheworld.fr/. 


Buy my book! - http://bit.ly/beginning_perl
Re: Can't localize lexical variable $x at ... [ In reply to ]
On Tue, 25 Jan 2022 15:59:58 +0000
hv@crypt.org wrote:

> It would probably be useful to have examples where that distinction
> makes a difference - I didn't see any (explicit) mention of that in
> the lengthy perlsub section about local, nor in S::K::D. Are there
> cases where changing an existing valid local() invocation to use
> dynamically() instead would change the behaviour? My guess would be
> that if anything it would revolve around magic.

Anything involving subtle core magic would probably show the
differences. Outside of that it's not visible; even the magic that
implements tie goes to special steps to ensure it is handled in the
manner expected.

See the comments around `mg_localize` in mg.c:

https://github.com/Perl/perl5/blob/535068756f421813eff4aae3784af48f00ea94b8/mg.c#L500

> I confess I don't understand the choice of "dynamically" for the name:
> "temporarily" would seem the obvious choice to me (and I'd very much
> want to abbreviate that to "temp"). "dynamic" is a concept with a few
> specific meanings in perl jargon and lots more outside of that: I
> think few average perl users would instinctively guess the correct
> meaning in this context.

Temporary is all about time. Dynamic extent isn't quite the same thing.

Without async/await, coroutines, or other such-like ideas, the two
words mean the same thing. But crucially, once you have the idea of a
suspended coroutine (which is what an `async sub` is while it is
waiting), you have the possibility that a single *dynamic* extent can
live across two disjoint regions of time, with a gap in the middle.
Since the assignment is suspended and restored along with this dynamic
extent, the word "dynamically" is a better fit than "temporarily".

Example:

my $VAR = 1;

async sub f {
dynamically $VAR = 2;
say "VAR is $VAR";

await Future::IO->sleep(3);

say "VAR is $VAR";
}

At both of the `say` lines, the dynamic assignment to VAR has the value
2 visible. But outside of those, the value remains 1. This is true even
if the program pauses an invocation of f() in order to go off and do
other things in the meantime, and comes back to finish running f()
three seconds later. This was one dynamic extent, split across two
separate temporal extents with a 3-second gap.

> As a module it does not appear to require Async even be installed,
> could it not continue to act as it does now?

That's because it ships with a copy of Future-AsyncAwait's extension
API header file:

https://metacpan.org/release/PEVANS/Syntax-Keyword-Dynamically-0.09/source/AsyncAwait.h

It builds against that at compiletime, even if the Future::AsyncAwait
module isn't available.

> Or is the problem that
> it could not then easily be adapted (via a new release) to unknown
> future async changes? Or just that it's not the done thing for core
> code to have special handling for non-core modules?

I wouldn't want to have to include AsyncAwait.h with perl core :)

--
Paul "LeoNerd" Evans

leonerd@leonerd.org.uk | https://metacpan.org/author/PEVANS
http://www.leonerd.org.uk/ | https://www.tindie.com/stores/leonerd/
Re: Can't localize lexical variable $x at ... [ In reply to ]
On Tue, 25 Jan 2022 at 17:48, <hv@crypt.org> wrote:

> demerphq <demerphq@gmail.com> wrote:
> :On Tue, 25 Jan 2022, 23:43 , <hv@crypt.org> wrote:
> We avoid using globals for good reasons, I wouldn't want to use them
> for this unless there really was no alternative.
>

Which good reason? So long as you localize in your sub there isnt much
difference.

yves

--
perl -Mre=debug -e "/just|another|perl|hacker/"
Re: Can't localize lexical variable $x at ... [ In reply to ]
On Tue, Jan 25, 2022 at 02:37:19PM +0000, hv@crypt.org wrote:
> In this case, it's an inline recursive function of which the relevant
> parts look a bit like:
>
> sub do_stuff {
> my $prod = 1;
> my $recurse;
> $recurse = sub {
> ...
> for my $new (@interesting_values) {
> local $prod = $prod * $new; # aspirational
> $recurse->();
> }
> };
> $recurse->();
> }
>
> .. so shadowing doesn't help. Alternatives such as a) making it a
> one-element array, b) passing it as an argument or c) manually
> saving and restoring are all possible, but all likely to be slower,
> more error-prone, and add complexity to already-complex code.
>
> The real code's aim is to find new values for https://oeis.org/A292580.
> It's very much a work-in-progress, but if you're interested look at
> https://github.com/hvds/seq/blob/master/divrep/oul#L167 for current
> implementation (as one-element array).
>
> Hugo

Though personally I would agree with demerphq to just use a localized
global, I think one way around this to get what you want is to write:

my $h = {prod => 1};
...
$recurse = sub {
...
local $h->{prod} = $h->{prod} * $new;
...
};
Re: Can't localize lexical variable $x at ... [ In reply to ]
On Tue, 25 Jan 2022 at 20:15, <yb@rslinux.fun> wrote:

> On Tue, Jan 25, 2022 at 02:37:19PM +0000, hv@crypt.org wrote:
> > In this case, it's an inline recursive function of which the relevant
> > parts look a bit like:
> >
> > sub do_stuff {
> > my $prod = 1;
> > my $recurse;
> > $recurse = sub {
> > ...
> > for my $new (@interesting_values) {
> > local $prod = $prod * $new; # aspirational
> > $recurse->();
> > }
> > };
> > $recurse->();
> > }
> >
> > .. so shadowing doesn't help. Alternatives such as a) making it a
> > one-element array, b) passing it as an argument or c) manually
> > saving and restoring are all possible, but all likely to be slower,
> > more error-prone, and add complexity to already-complex code.
> >
> > The real code's aim is to find new values for https://oeis.org/A292580.
> > It's very much a work-in-progress, but if you're interested look at
> > https://github.com/hvds/seq/blob/master/divrep/oul#L167 for current
> > implementation (as one-element array).
> >
> > Hugo
>
> Though personally I would agree with demerphq to just use a localized
> global, I think one way around this to get what you want is to write:
>
> my $h = {prod => 1};
> ...
> $recurse = sub {
> ...
> local $h->{prod} = $h->{prod} * $new;
> ...
> };
>

Nod, but I might use an array instead of a hash, if speed is a concern.

Yves

--
perl -Mre=debug -e "/just|another|perl|hacker/"
Re: Can't localize lexical variable $x at ... [ In reply to ]
In this case, it's an inline recursive function of which the relevant
> parts look a bit like:
>
> sub do_stuff {
> my $prod = 1;
> my $recurse;
> $recurse = sub {
> ...
> for my $new (@interesting_values) {
> local $prod = $prod * $new; # aspirational
> $recurse->();
> }
> };
> $recurse->();
> }
>
> .. so shadowing doesn't help. Alternatives such as a) making it a
> one-element array, b) passing it as an argument or c) manually
> saving and restoring are all possible, but all likely to be slower,
> more error-prone, and add complexity to already-complex code.
>

okay here's my rewrite. I started with (c) then switched to (b).

sub do_stuff {
my $prod = shift;
my @interesting_values = wassup($prod);
for my $new (@interesting_values) {
do_stuff($prod * $new);
}
};
do_stuff(1);

I for one think passing an argument to a named recursive routine is less
error-prone and less complicated than trying to do arcane and fragile perl
four magic on a variable you're passing by shared outer scope to inner
iterations of an anonymous one.

It seems like it might be a good case for ditching recursion altogether in
favor of a generator/consumer pattern.

my @queue = (1);
while(@queue){
my $prod = shift @queue;
# push @queue, map { $prod * $_ } wassup($prod); # one way to
do it
push @queue, $prod * $_ for wassup($prod); # another way to
do it
}




--
"Lay off that whiskey, and let that cocaine be!" -- Johnny Cash
Re: Can't localize lexical variable $x at ... [ In reply to ]
On Tue, Jan 25, 2022 at 03:59:58PM +0000, hv@crypt.org wrote:
> It would probably be useful to have examples where that distinction
> makes a difference

Aren't we talking about the different between

1) container localisation:

my $x = 1;
my $rx = $x;
local $x = 2;
print $$rx, $x; # prints 2,2

2) value localisation:

my $x = 1;
my $rx = $x;
local $x = 2;
print $$rx, $x; # prints 1,2

local() normally does container localisation. I think for consistency it
should do that with lexical vars too.

i.e.

{
local $lex = $foo;
...
}

should be equivalent to


for $lex ($foo) {
...
}

--
Modern art:
"That's easy, I could have done that!"
"Ah, but you didn't!"