Mailing List Archive

Pre-RFC: A built-in ability for lexical import/export
One of the offshoots from the Pre-RFC rejections of the `module` and
`UNIVERSAL::import` ideas, was the thought that perl core really needs
to provide some nicer ways to implement lexical import into the
caller's lexical namespace.

Core didn't really have any code that does lexical import of normal
symbols, so when I created the `builtin::import` function I had to
create it all specially. It's fine there but feels a bit weird not
providing that as a general mechanism for others to use.

I was thinking that maybe it should be just exposed as a builtin::
function itself, perhaps as

builtin::lexically_export $name, \&func;

Perhaps it should take a whole kv-list of name/ref pairs, so you can
export lots in one go:

builtin::lexically_export
one => \&one,
two => \&two, ...

Though maybe it would be bad to encourage that because really the
caller ought to be specifying just a few symbols they want. Hmm.

Also, while syntactically this is written out "like a normal function"
and thus could go in `builtin::`, I wonder if it's a good fit here. It
can only possibly work during BEGIN time, because it affects the
compiletime lexical environment of the code being compiled. It isn't
useful at runtime. It syntactically looks like a regular function but
it doesn't really behave like one.

We probably do want this available as a core function, but what should
it be named? Is `builtin::` still a good place to put it?

--
Paul "LeoNerd" Evans

leonerd@leonerd.org.uk | https://metacpan.org/author/PEVANS
http://www.leonerd.org.uk/ | https://www.tindie.com/stores/leonerd/
Re: Pre-RFC: A built-in ability for lexical import/export [ In reply to ]
On Fri, Jun 24, 2022, at 11:44, Paul "LeoNerd" Evans wrote:
> Perhaps it should take a whole kv-list of name/ref pairs, so you can export lots in one go:
>
> builtin::lexically_export
> one => \&one,
> two => \&two, ...
>
> Though maybe it would be bad to encourage that because really the caller ought to be specifying just a few symbols they want. Hmm.

I think this is great. I think supplying multiple arguments is fine. The caller can specify what they want and the callee can behave accordingly.
sub import ($self, @names) {
die "unknown exports" if @names != grep {; defined } @EXPORTS{ @names };

builtin::lexically_export %EXPORTS{ @names };
}


> Also, while syntactically this is written out "like a normal function"
> and thus could go in `builtin::`, I wonder if it's a good fit here. It
> can only possibly work during BEGIN time, because it affects the
> compiletime lexical environment of the code being compiled. It isn't
> useful at runtime. It syntactically looks like a regular function but
> it doesn't really behave like one.

I think it's fine. I don't know that we actually have anything that "looks like it would happen at compile time". We could perhaps issue a warning if it was called when nothing was being compiled?

--
rjbs
Re: Pre-RFC: A built-in ability for lexical import/export [ In reply to ]
On Fri, Jun 24, 2022 at 11:44 AM Paul "LeoNerd" Evans <
leonerd@leonerd.org.uk> wrote:

> One of the offshoots from the Pre-RFC rejections of the `module` and
> `UNIVERSAL::import` ideas, was the thought that perl core really needs
> to provide some nicer ways to implement lexical import into the
> caller's lexical namespace.
>
> Core didn't really have any code that does lexical import of normal
> symbols, so when I created the `builtin::import` function I had to
> create it all specially. It's fine there but feels a bit weird not
> providing that as a general mechanism for others to use.
>
> I was thinking that maybe it should be just exposed as a builtin::
> function itself, perhaps as
>
> builtin::lexically_export $name, \&func;
>

I can't not suggest "lexport".


> Perhaps it should take a whole kv-list of name/ref pairs, so you can
> export lots in one go:
>
> builtin::lexically_export
> one => \&one,
> two => \&two, ...
>
> Though maybe it would be bad to encourage that because really the
> caller ought to be specifying just a few symbols they want. Hmm.
>

I don't think it's a big deal, being able to export multiple in one call
seems like a good feature.


> Also, while syntactically this is written out "like a normal function"
> and thus could go in `builtin::`, I wonder if it's a good fit here. It
> can only possibly work during BEGIN time, because it affects the
> compiletime lexical environment of the code being compiled. It isn't
> useful at runtime. It syntactically looks like a regular function but
> it doesn't really behave like one.
>
> We probably do want this available as a core function, but what should
> it be named? Is `builtin::` still a good place to put it?
>

I can see arguments for either way. If we had a namespace for "less normal
keywords" then I might argue for putting it there, but under a strict
interpretation of "parsed like a normal function", it certainly could fit
in builtin.

-Dan
Re: Pre-RFC: A built-in ability for lexical import/export [ In reply to ]
On Fri, 24 Jun 2022 16:44:27 +0100
"Paul \"LeoNerd\" Evans" <leonerd@leonerd.org.uk> wrote:

> I was thinking that maybe it should be just exposed as a builtin::
> function itself, perhaps as
>
> builtin::lexically_export $name, \&func;

I've had a bit of a go at implementing this:

https://github.com/leonerd/perl5/tree/lexically-export

> Perhaps it should take a whole kv-list of name/ref pairs, so you can
> export lots in one go:
>
> builtin::lexically_export
> one => \&one,
> two => \&two, ...

It currently doesn't do this yet, but shouldn't be hard to add.


In the process of writing it I've come up with some more questions, to
which I don't have an easy answer:

* Should it support things other than functions? Should we support

lexically_export VAR => \my $scalar;

* If so, should names be sigil-prefixed?

lexically_export '$VAR' => \my $scalar, '&func' => sub {...};

While I don't think it is common to want to export things other than
functions, it does occasionally happen (e.g. think the $LOG variable of
various logger modules, or the $METRICS of Metrics::Any). It would be
nice to support these in some way.

If we do support them, it raises the question on whether the "name"
argument to lexically_export() itself should include the sigil part of
it. In the vastly-common majority of cases, they'll all just be
functions, so it feels annoying to have to '&'-prefix them all -
especially if it would otherwise break the neat way of using
"name => \&name" notation.

On the other hand, it would lead to an awkward situation if you wanted
to export more than one item of the same basename:

lexically_export
FOO => \my $s, FOO => \my @arr, FOO => \my %hash, FOO => sub {};

This would technically be fine as it would create four separately named
items in the caller - $FOO, @FOO, %FOO and &FOO (callable as FOO()).
But it does feel strange, and breaks the otherwise neat hash-like
name/value pair structure of the arguments. I'm not sure I like it.

Instead, perhaps a compromise idea taken from the way Exporter.pm
works; which is to say that sigils would be required for
scalar/hash/array variables, but not required for regular functions.

lexically_export
name => \&function, # a regular function
'$VAR' => \my $scalar, # a scalar variable
'@VAR' => \my @arr, # an array variable
'%VAR' => \my %hash, # a hash variable

'&func' => \&func, # also permitted for symmetry, but not
# required

I.e. the model being "if no sigil is specified, presume &"; and
additionally, error if the sigil doesn't match the thing being
referenced.


I think this feels sane - does anyone disagree?

--
Paul "LeoNerd" Evans

leonerd@leonerd.org.uk | https://metacpan.org/author/PEVANS
http://www.leonerd.org.uk/ | https://www.tindie.com/stores/leonerd/
Re: Pre-RFC: A built-in ability for lexical import/export [ In reply to ]
"Paul \"LeoNerd\" Evans" <leonerd@leonerd.org.uk> writes:

> On Fri, 24 Jun 2022 16:44:27 +0100
> "Paul \"LeoNerd\" Evans" <leonerd@leonerd.org.uk> wrote:
>
>> I was thinking that maybe it should be just exposed as a builtin::
>> function itself, perhaps as
>>
>> builtin::lexically_export $name, \&func;
>
> I've had a bit of a go at implementing this:
>
> https://github.com/leonerd/perl5/tree/lexically-export
>
>> Perhaps it should take a whole kv-list of name/ref pairs, so you can
>> export lots in one go:
>>
>> builtin::lexically_export
>> one => \&one,
>> two => \&two, ...
>
> It currently doesn't do this yet, but shouldn't be hard to add.

I think it should support this, since it's common to export multiple
things, and build up data structures of things to be exported, e.g.:

my %exportables = ( ... );

sub import ($class, @items) {
# [ validation here ]

lexically_export %exportables{@items};
}

instead of having to do

lexically_export $_ => $exportables{$_} for @items;

> In the process of writing it I've come up with some more questions, to
> which I don't have an easy answer:
>
> * Should it support things other than functions? Should we support
>
> lexically_export VAR => \my $scalar;

Yes.

> * If so, should names be sigil-prefixed?
>
> lexically_export '$VAR' => \my $scalar, '&func' => sub {...};

Yes, but …

> Instead, perhaps a compromise idea taken from the way Exporter.pm
> works; which is to say that sigils would be required for
> scalar/hash/array variables, but not required for regular functions.
>
> lexically_export
> name => \&function, # a regular function
> '$VAR' => \my $scalar, # a scalar variable
> '@VAR' => \my @arr, # an array variable
> '%VAR' => \my %hash, # a hash variable
>
> '&func' => \&func, # also permitted for symmetry, but not
> # required
>
> I.e. the model being "if no sigil is specified, presume &"; and
> additionally, error if the sigil doesn't match the thing being
> referenced.

Yes.

> I think this feels sane - does anyone disagree?

I think this is a good approach, for reasons of convenience,
consistency, and safety.

- ilmari
Re: Pre-RFC: A built-in ability for lexical import/export [ In reply to ]
Hi there,

On Mon, 27 Jun 2022, Dagfinn Ilmari Mannsåker wrote:
> "Paul \"LeoNerd\" Evans" writes:
>> On Fri, 24 Jun 2022 16:44:27 +0100 "Paul \"LeoNerd\" Evans" wrote:
>>
>> ...
>> ...
>> * Should it support things other than functions? Should we support
>>
>> lexically_export VAR => \my $scalar;
>
> Yes.

Agreed.

>> * If so, should names be sigil-prefixed?
>>
>> lexically_export '$VAR' => \my $scalar, '&func' => sub {...};
>
> Yes ...

Agreed.

--

73,
Ged.
Re: Pre-RFC: A built-in ability for lexical import/export [ In reply to ]
On Mon, 27 Jun 2022 14:40:15 +0100
"Paul \"LeoNerd\" Evans" <leonerd@leonerd.org.uk> wrote:

> > Perhaps it should take a whole kv-list of name/ref pairs, so you can
> > export lots in one go:
> >
> > builtin::lexically_export
> > one => \&one,
> > two => \&two, ...
>
> It currently doesn't do this yet, but shouldn't be hard to add.

Update: it now does this.
>
> In the process of writing it I've come up with some more questions, to
> which I don't have an easy answer:
>
> * Should it support things other than functions? Should we support
>
> lexically_export VAR => \my $scalar;

And this.

> * If so, should names be sigil-prefixed?
>
> lexically_export '$VAR' => \my $scalar, '&func' => sub {...};

And this.

> I.e. the model being "if no sigil is specified, presume &"; and
> additionally, error if the sigil doesn't match the thing being
> referenced.

And this.

> I think this feels sane - does anyone disagree?

Thanks all for the comments (here and IRC).

Next (meta)-question: Should this be turned into a full RFC, or is it
small and concise enough that these questions and any subsequent
discussion on a PR would be sufficient?

In the hope it's the latter, I've now opened a PR for merging it:

https://github.com/Perl/perl5/pull/19895

but if people think this should go via the full RFC process first, I
can write something up.

--
Paul "LeoNerd" Evans

leonerd@leonerd.org.uk | https://metacpan.org/author/PEVANS
http://www.leonerd.org.uk/ | https://www.tindie.com/stores/leonerd/
Re: Pre-RFC: A built-in ability for lexical import/export [ In reply to ]
On Mon, Jun 27, 2022 at 5:30 PM Paul "LeoNerd" Evans <leonerd@leonerd.org.uk>
wrote:

> > * Should it support things other than functions? Should we support
> >
> > lexically_export VAR => \my $scalar;
>
> And this.
>

I'm concerned about this. Lexical import is designed to limit things to the
current scope and not expose them to the outside world. Because you're
exporting a reference, it's effectively a global variable. If more than one
bit of code imports that reference and alters it, other bits of code might
have already taken decisions based on that data, but the assumptions it
relied on are no longer true.

I've just spent several days chasing down a bug that's caused by this and
I've had to do so multiple times for my current client. My client is losing
money paying me to fix design flaws rather than build the new features they
want. I'd rather not add another feature to Perl which encourages this
practice.

Curtis "Ovid" Poe
CTO, All Around the World
World-class software development and consulting
https://allaroundtheworld.fr/
Re: Pre-RFC: A built-in ability for lexical import/export [ In reply to ]
On Mon, Jun 27, 2022 at 1:11 PM Ovid <curtis.poe@gmail.com> wrote:

> On Mon, Jun 27, 2022 at 5:30 PM Paul "LeoNerd" Evans <
> leonerd@leonerd.org.uk> wrote:
>
>> > * Should it support things other than functions? Should we support
>> >
>> > lexically_export VAR => \my $scalar;
>>
>> And this.
>>
>
> I'm concerned about this. Lexical import is designed to limit things to
> the current scope and not expose them to the outside world. Because you're
> exporting a reference, it's effectively a global variable. If more than one
> bit of code imports that reference and alters it, other bits of code might
> have already taken decisions based on that data, but the assumptions it
> relied on are no longer true.
>
> I've just spent several days chasing down a bug that's caused by this and
> I've had to do so multiple times for my current client. My client is losing
> money paying me to fix design flaws rather than build the new features they
> want. I'd rather not add another feature to Perl which encourages this
> practice.
>

I don't entirely agree. It's no different in semantics to passing a
variable to a function, which then has an alias to the variable to operate
on.

-Dan
Re: Pre-RFC: A built-in ability for lexical import/export [ In reply to ]
Op 27-06-2022 om 15:40 schreef Paul "LeoNerd" Evans:
> On Fri, 24 Jun 2022 16:44:27 +0100
> "Paul \"LeoNerd\" Evans" <leonerd@leonerd.org.uk> wrote:
>
>> I was thinking that maybe it should be just exposed as a builtin::
>> function itself, perhaps as
>>
>> builtin::lexically_export $name, \&func;
> I've had a bit of a go at implementing this:
>
> https://github.com/leonerd/perl5/tree/lexically-export
>
>> Perhaps it should take a whole kv-list of name/ref pairs, so you can
>> export lots in one go:
>>
>> builtin::lexically_export
>> one => \&one,
>> two => \&two, ...
> It currently doesn't do this yet, but shouldn't be hard to add.
>

What I do not understand is why one would want to use a different
external name than the internal name? Exporter doesn't work like this
and I for one, never felt the need to export under a different name.


TIA,

M4
Re: Pre-RFC: A built-in ability for lexical import/export [ In reply to ]
On Mon, 27 Jun 2022 19:46:54 +0200
Martijn Lievaart <m@rtij.nl> wrote:

> What I do not understand is why one would want to use a different
> external name than the internal name? Exporter doesn't work like this
> and I for one, never felt the need to export under a different name.

I'm not sure I follow the question...

--
Paul "LeoNerd" Evans

leonerd@leonerd.org.uk | https://metacpan.org/author/PEVANS
http://www.leonerd.org.uk/ | https://www.tindie.com/stores/leonerd/
Re: Pre-RFC: A built-in ability for lexical import/export [ In reply to ]
On Mon, Jun 27, 2022 at 7:48 PM Martijn Lievaart <m@rtij.nl> wrote:

>
> Op 27-06-2022 om 15:40 schreef Paul "LeoNerd" Evans:
> > On Fri, 24 Jun 2022 16:44:27 +0100
> > "Paul \"LeoNerd\" Evans" <leonerd@leonerd.org.uk> wrote:
> >
> >> I was thinking that maybe it should be just exposed as a builtin::
> >> function itself, perhaps as
> >>
> >> builtin::lexically_export $name, \&func;
>
[snip]

> What I do not understand is why one would want to use a different
> external name than the internal name? Exporter doesn't work like this
> and I for one, never felt the need to export under a different name.
>

There might not even be an "internal name" – the subroutine to be
exported might be a closure. I use this in Test::Trap to export trap{}
functions with different names, layers, and other options. Although that
might be a rather outlandish example …

Adapting instead the (oh-so-nostalgic) perlref example of closures, we
can imagine:

sub import ($class, @colors) {
# [ validation? caching? ]
builtin::lexically_export $_ => sub { "<FONT COLOR='$_'>@_</FONT>"
} for @colors;
}

No internal name.

And yes, this is very rarely useful. Exporter is usually enough, after
all. But if you are to offer a generic mechanism for lexical export, is
there any good reason to deny programmers this power?


Eirik
Re: Pre-RFC: A built-in ability for lexical import/export [ In reply to ]
Op 27-06-2022 om 21:45 schreef Paul "LeoNerd" Evans:
> On Mon, 27 Jun 2022 19:46:54 +0200
> Martijn Lievaart <m@rtij.nl> wrote:
>
>> What I do not understand is why one would want to use a different
>> external name than the internal name? Exporter doesn't work like this
>> and I for one, never felt the need to export under a different name.
> I'm not sure I follow the question...
>

If I understand the proposal correctly, you could do:

builtin::lexically_export two => \&one;

builtin::lexically_export three => \&two;


Why not implement:

builtin::lexically_export '\&one';

And have that export the sub one() under the name 'one'?


TIA,
M4
Re: Pre-RFC: A built-in ability for lexical import/export [ In reply to ]
On Mon, Jun 27, 2022 at 7:28 PM Dan Book <grinnz@gmail.com> wrote:

> I'm concerned about this. Lexical import is designed to limit things to
>> the current scope and not expose them to the outside world. Because you're
>> exporting a reference, it's effectively a global variable. If more than one
>> bit of code imports that reference and alters it, other bits of code might
>> have already taken decisions based on that data, but the assumptions it
>> relied on are no longer true.
>>
>> I don't entirely agree. It's no different in semantics to passing a
> variable to a function, which then has an alias to the variable to operate
> on
>

I agree with that. What I'm saying is that we shouldn't *add* to the issue.
At least if you export a sub which returns the data structure itself, you
have the option of cloning that structure or marking it read-only.

If we export variables, they would be much safer if they were read-only by
default, but with an option to mutate (e.g., you have to ask for the lack
of safety instead of making it the default):

lexically_export '$VAR' :mutable => \my $scalar

Otherwise, my vote would be to not support exporting them.

Best,
Curtis "Ovid" Poe
CTO, All Around the World
World-class software development and consulting
https://allaroundtheworld.fr/
Re: Pre-RFC: A built-in ability for lexical import/export [ In reply to ]
> Next (meta)-question: Should this be turned into a full RFC, or is it
> small and concise enough that these questions and any subsequent
> discussion on a PR would be sufficient?

I think this should go via an RFC. The idea is that changes to the language should go through the RFC process, and the process should be able to handle both heavy- and light-weight proposals.

In addition, I think at least a week should elapse before a Pre-RFC can move forward. Not everyone is able to keep on top of p5p on a daily basis, but there should be enough time allowed for people to read and comment, if they want to.

Neil
Re: Pre-RFC: A built-in ability for lexical import/export [ In reply to ]
On Tue, Jun 28, 2022, 11:53 Ovid <curtis.poe@gmail.com> wrote:

> On Mon, Jun 27, 2022 at 7:28 PM Dan Book <grinnz@gmail.com> wrote:
>
>> I'm concerned about this. Lexical import is designed to limit things to
>>> the current scope and not expose them to the outside world. Because you're
>>> exporting a reference, it's effectively a global variable. If more than one
>>> bit of code imports that reference and alters it, other bits of code might
>>> have already taken decisions based on that data, but the assumptions it
>>> relied on are no longer true.
>>>
>>> I don't entirely agree. It's no different in semantics to passing a
>> variable to a function, which then has an alias to the variable to operate
>> on
>>
>
> I agree with that. What I'm saying is that we shouldn't *add* to the
> issue. At least if you export a sub which returns the data structure
> itself, you have the option of cloning that structure or marking it
> read-only.
>
> If we export variables, they would be much safer if they were read-only by
> default, but with an option to mutate (e.g., you have to ask for the lack
> of safety instead of making it the default):
>
> lexically_export '$VAR' :mutable => \my $scalar
>
> Otherwise, my vote would be to not support exporting them.
>
> Best,
> Curtis "Ovid" Poe
> CTO, All Around the World
> World-class software development and consulting
> https://allaroundtheworld.fr/
>

In all of these examples, you're actually exporting lexical variables in
the scope of the import sub. None have the issue that you're worried about,
every call to the import routine gets its own variable.

If someone were to decide to export a variable with package scope, or a
state variable, then that's their choice. We can't stop them from exporting
an lvalue subroutine wrapper around such a variable, so there's no way to
fully guard against this problem.

I find this useful for exporting a lexical variable which has a scopeguard,
or for exporting a mock object with a test.

Please don't hold up a very useful feature with concerns that don't have to
do with it.

> <https://allaroundtheworld.fr/>
>
Re: Pre-RFC: A built-in ability for lexical import/export [ In reply to ]
On Tue, Jun 28, 2022 at 11:32 AM Veesh Goldman <rabbiveesh@gmail.com> wrote:

> Please don't hold up a very useful feature with concerns that don't have
> to do with it.
>
>> <https://allaroundtheworld.fr/>
>>
>
Fair enough. I think most people agree with you.

--
Curtis "Ovid" Poe
CTO, All Around the World
World-class software development and consulting
https://allaroundtheworld.fr/
Re: Pre-RFC: A built-in ability for lexical import/export [ In reply to ]
Martijn Lievaart <m@rtij.nl> writes:

> Op 27-06-2022 om 21:45 schreef Paul "LeoNerd" Evans:
>> On Mon, 27 Jun 2022 19:46:54 +0200
>> Martijn Lievaart <m@rtij.nl> wrote:
>>
>>> What I do not understand is why one would want to use a different
>>> external name than the internal name? Exporter doesn't work like this
>>> and I for one, never felt the need to export under a different name.
>> I'm not sure I follow the question...
>>
>
> If I understand the proposal correctly, you could do:
>
> builtin::lexically_export two => \&one;
>
> builtin::lexically_export three => \&two;
>
>
> Why not implement:
>
> builtin::lexically_export '\&one';
>
> And have that export the sub one() under the name 'one'?

That would only work if one() was a packge sub or variable in the
exporting module, since it would have to be looked up by name. The
whole point of this is to get away from package subs and variables, so
the item to be exported needs to be passed as a proper reference.

It could be possible to default the name to the (unqualified part of
the) name of the variable (using mechanisms similar to
Sub::Util::subname() and PadWalker::var_name()), but that strikes me as
Too Much Magic™ to avoid a tiny bit of explicitness.

> TIA,
> M4

- ilmari
Re: Pre-RFC: A built-in ability for lexical import/export [ In reply to ]
On Tue, 28 Jun 2022 10:29:06 +0100
Neil Bowers <neilb@neilb.org> wrote:

> > Next (meta)-question: Should this be turned into a full RFC, or is
> > it small and concise enough that these questions and any subsequent
> > discussion on a PR would be sufficient?
>
> I think this should go via an RFC. The idea is that changes to the
> language should go through the RFC process, and the process should be
> able to handle both heavy- and light-weight proposals.

Alrighty. Now written at

https://github.com/Perl/RFCs/pull/20

--
Paul "LeoNerd" Evans

leonerd@leonerd.org.uk | https://metacpan.org/author/PEVANS
http://www.leonerd.org.uk/ | https://www.tindie.com/stores/leonerd/