Mailing List Archive

On Sunday, 9 January 2022, 13:38:07 CET, Neil Bowers <neilb@neilb.org> wrote:

     
> 1. Remove the experimental sticker off signatures and release that way in 5.36 (so you'd still have to `use feature` or `use v5.36`), but no other changes.  
> 2. As for 1, but also add a runtime warning if you touch @_ inside a signatured-sub.
> 3. Like 2, but touching @_ is fatal.
> 4. Inside signatured-subs @_ becomes non-special.


I dislike #1 because that will drag out the @_ issue. if someone needs it, they can always use `sub foo (@args) {...}`

I dislike #3 and #4 (if I understand your intent) because that will break existing code, requiring more work for maintainers. They can't just enable/disable things. They have to rewrite code (again, assuming I am understanding correctly).

#2 seems to be to clearest path forward to removing this feature altogether.

Would there be a warnings category for disabling @_ warnings? Something like (no idea what to call it):

    no warnings "deprecated::args_array";

With that, we could silence that single warning, but it's easy to grep through a codebase for deprecated features. In fact, I'd love to see tons more deprecated warning categories to make it easier to track them in larger codebases. We have clients who could probably use something like this.

Best,
Ovid
-- 
IT consulting, training, specializing in Perl, databases, and agile development
http://www.allaroundtheworld.fr/. 

Buy my book! - http://bit.ly/beginning_perl

On Sun, Jan 9, 2022, at 7:37 AM, Neil Bowers wrote:
> Our overriding desire is to get signatures "out there", but what's the right next step? There are at least 4 options:
> 1. Remove the experimental sticker off signatures and release that way in 5.36 (so you'd still have to `use feature` or `use v5.36`), but no other changes.
> 2. As for 1, but also add a runtime warning if you touch @_ inside a signatured-sub.
> 3. Like 2, but touching @_ is fatal.
> 4. Inside signatured-subs @_ becomes non-special.

I am strongly in favor of #1.

It's possible I misunderstand the core goals of not setting up @_. I believe it's for performance. Signatured subroutines can be made faster than subs sans sigs. Instead of setting up @_ we can just not. So if I am missing a large component, somebody should say something.

But: the cost of this change is complexity at many levels. First, the ongoing complexity of writing subroutines with @_. "That's not complex, Rik, I've been doing that for twenty years!" Great, but for people who haven't been doing it for 20 years, it *is* complex and also weird.

For those of us who have been writing subroutines with @_ for 20 years, removing @_ is also weird. But *not* looking at @_ is pretty easy, so would we accept that tradeoff for some performance? Yes. But that tradeoff isn't actually on offer. We don't have a branch that provides this tradeoff. We have discussion over what one would be. Mostly, it's #4 above: accessing @_ is like accessing @~ or @(. It's some meaningless global inside a signatured sub, left with the contents of the last old-style sub. There's no warning for accessing it, because it's just some global. In fact, my understanding is that adding a warning would require that we add magic to @_ globally, slowing down more or less all subroutines. (Who remembers pseudohashes?)

So we can choose one the more complex versions of "make signatured subroutines faster" with no warnings, and do that *right away* and then remove the experimental flag after a year. But we don't have that branch. It's been the primary blocker on removing the experimental flag on signatures for years, I believe. Meanwhile, the only people who get the vastly improved experience of having signatures on subroutines are the ones who are willing to ignore the "this is experimental and might change at any time" label — which is exactly the behavior we want to avoid encouraging.

That "slow everything down" problem means that #2 is probably off the table.

Let's talk about #1 again. If we say "the signatures that you've had available for years, and that many of you have been using, are no longer experimental," then it's true: people can continue to see @_ in their subs, and that means that subroutine calls on signatured subs don't get the no-@_ optimization. *Things stay the same speed as now*. Meanwhile, the developer experience is significantly better.

If, later, we want to provide that optimization, we would need to provide a deprecation warning on accessing @_ in a subroutine with a signature. Given my belief about this slowing down all subroutines, I don't think we're likely to remove @_ in signatured subroutines if we make them non-experimental as is right now, *and I am okay with that.*

We introduced signatures in 5.20, over seven years ago, and they're still experimental. I think we're a lot better off making them non-experimental now and giving up on this optimization. Further, I think that signatures are *so much *better than not that I think we should immediately add the signatures feature to the 5.36 feature bundle.

--
rjbs

On 2022-01-09 4:37 a.m., Neil Bowers wrote:
> The topic we've discussed a lot recently is what to do with @_ in
> signatured-subs: if your sub has a signature then you shouldn’t be touching @_.
> If we can remove support for @_ in signatured subs, that could give a
> performance boost, and also remove a class of potential bugs.
>
> Our overriding desire is to get signatures "out there", but what's the right
> next step? There are at least 4 options:
>
> 1. Remove the experimental sticker off signatures and release that way in 5.36
> (so you'd still have to `use feature` or `use v5.36`), but no other changes.
> 2. As for 1, but also add a runtime warning if you touch @_ inside a
> signatured-sub.
> 3. Like 2, but touching @_ is fatal.
> 4. Inside signatured-subs @_ becomes non-special.

I vote for something like number 3.

When a file says "use v5.36" or a higher version, it causes signatures to be
available non-experimentally AND it causes @_ to not be declared within the
scope of any sub using a signature, wherein any references to @_ will fail AT
PARSE/COMPILE time.

Since "use v5.36" already enables strict, the failure with @_ can reported as a
reference to an undeclared variable like other violations of strict do.

Or at the very least, what I advocate for is that within the scope of any sub
having a signature, there does not exist an @_ with any kind of special
behavior, and the performance optimization exists.

That means any kind of explicit message to the user about @_, whether a warning
or a fatal error, it happens at parse/compile time, and not at runtime, of the
code, because runtime requires the special logic that slows things down.

I think making this fatal is best, so people don't gain bugs from @_ not having
the arguments because they just happened to miss new warnings.

This stricter behaviour should not break any existing code because it only
occurs when they explicitly "use v5.36" or they otherwise use-signatures without
experimental, meaning it is a change they explicitly opted into in the same file
as the affected sub.

-- Darren Duncan

On Sunday, 9 January 2022, 17:34:19 CET, Ricardo Signes <perl.p5p@rjbs.manxome.org> wrote:

> So we can choose one the more complex versions of "make signatured subroutines faster" with no 
> warnings, and do that right away and then remove the experimental flag after a year.  But we don't
> have that branch.  It's been the primary blocker on removing the experimental flag on signatures for
> years, I believe.  Meanwhile, the only people who get the vastly improved experience of having
> signatures on subroutines are the ones who are willing to ignore the "this is experimental and might
> change at any time" label — which is exactly the behavior we want to avoid encouraging.

I'm sold. Decoupling "make signatures non-experimental" and "deal with @_" sounds like the right way to go.

Best,
Ovid
-- 
IT consulting, training, specializing in Perl, databases, and agile development
http://www.allaroundtheworld.fr/. 

Buy my book! - http://bit.ly/beginning_perl

On Mon, Jan 10, 2022, at 6:44 AM, Darren Duncan wrote:.
> When a file says "use v5.36" or a higher version, it causes signatures to be
> available non-experimentally AND it causes @_ to not be declared within the
> scope of any sub using a signature, wherein any references to @_ will fail AT
> PARSE/COMPILE time.

What about this code:
use v5.36.0;

sub foo ($x) {
# Here we have a runtime reference to @_ -- what happens?
my $first = substr $x, 0, 1;

no strict 'refs';
say "Contents of \@$first: @{$first}"; # What if $x was "_foo"?
}

Does this become a runtime failure?

How do we enforce that runtime failure? Presumably we do it by making reading from @_ require magic. Right now, @_ is a standard array, and it's subroutine entry that does the magical work of filling it in. If we make reading from it magic, it will become magic everywhere, slowing down all subroutine calls, right?

I don't think we're going to get to have a warning on this. We can stop setting up @_ and let people deal with the fallout — that is, not knowing what the heck happened until they find the entry in perlsub detailing that @_ now has the leavings of other subroutines in it — or we can give up and ship what we already have.

Or maybe something else I haven't imagined. But given the choices so far, I want to ship what we have.

--
rjbs

On Mon, Jan 10, 2022 at 03:44:52AM -0800, Darren Duncan wrote:
> I vote for something like number 3.
>
> When a file says "use v5.36" or a higher version, it causes signatures to be
> available non-experimentally AND it causes @_ to not be declared within the
> scope of any sub using a signature, wherein any references to @_ will fail
> AT PARSE/COMPILE time.

@_ isn't "declared". It never has been, never would be.

After a dollar, at-sign or percent-sign, any single printing character
is permitted. If that printing char was a valid "start-of-identifier"
character, the parser then reads more chars to find a full identifier.
If not, then it stands for some superglobal punctuation var that is
always available - for instance, $!, $), etc...

Thus, by necessity, @_ always exists because _ is a printing character.

Making such code as

my $x = shift @_;

fail to parse at parse time requires adding a bunch of special-case code
to the parser/compiler, to recognise that @_ must not be permitted
inside signatured subs.

Also, breaking the surface-syntax part of @_ is insufficient here,
because of all the implicit uses of the array inside `@_`. For example

my $x = shift;

The parser never encounters the global *_ symbol here, nor the global @_
array. This gets compiled into an OP_SHIFT with a special flag which
tells the runtime to read GvAV(PL_defgv) to find the arguments array.

Forbidding this as well requires much more logic adding to the parser.

Now, all of theabove can be added - it's just a matter of writing more
code. Lots more code. In the parser. In lots of situations. It's a whole
bunch of work but a suitably-motivated individual could track down all
the cases and add them. Such a parser would be slower than it currently
is, because it has to check more things, but perhaps it wouldn't be
*much* slower that anyone could really measure. If it was, say, under 1%
slower on average, we're likely to be happy with the hit (because, on
average, computers get 1% faster every few weeks anyway so overall the
situation is already better than the last time we looked at it).

But now consider some purely-dynamic runtime things that the parser
could not know about:

no strict 'refs';
my $sym = "some code here";
my $val = shift @{$sym};

The parser can't possibly know that $sym might be given as "_" and thus
we'll have to defer this to runtime. Rik already points out in another
mail, why that is hard and requires adding magic to the GvAV(PL_defgv)
AV which would therefore slow down *all functions* in *all perl
programs*, regardless of whether that function happened to use
signatures, or even if any function at all in the program ever does.

...

> This stricter behaviour should not break any existing code because it only
> occurs when they explicitly "use v5.36" or they otherwise use-signatures
> without experimental, meaning it is a change they explicitly opted into in
> the same file as the affected sub.

Not directly related to your comment, but while we're here: There is
already much code on CPAN and elsewhere (and I know because I have
written some of it) that still makes use of @_ within a signatured sub.
There's two usecases:

1: Forwarding all the arguments to a wrapper function

sub wrap ($x, @) {
warn "called wrap with x=$x";
otherfunc(@_); # just makes this easier
}

2: Determining the argument-count, in order to distinguish passing an
`undef` from not having an argument at all:

sub field ($self, $new=undef) {
if(@_ > 1) {
$self->{field} = $new;
}
return $self->{field};
}

$obj->field(123);
print "field is ", $obj->field;

Yes I know this code is hideous and personally I hate that style
but it remains synatically- and semantically-valid perl that
nevertheless many folks actually do write, and want to continue to
write.

Quite aside from any "it might perform faster" or "it helps users avoid
bugs" arguments that might be put forward in favour of banning the use
of @_ in signatured subs, we need to keep these two usecases in mind
(among possibly others), and ensure we continue to provide for them.

It would be a shame if we had to tell users they can't perform behaviour
like either of these two cases, if they wish to use signatured subs.

--

Paul "LeoNerd" Evans

leonerd@leonerd.org.uk | https://metacpan.org/author/PEVANS
http://www.leonerd.org.uk/ | https://www.tindie.com/stores/leonerd/

On 2022-01-10 3:44 a.m., Darren Duncan wrote:
> On 2022-01-09 4:37 a.m., Neil Bowers wrote:
>> The topic we've discussed a lot recently is what to do with @_ in
>> signatured-subs: if your sub has a signature then you shouldn’t be touching
>> @_. If we can remove support for @_ in signatured subs, that could give a
>> performance boost, and also remove a class of potential bugs.
>>
>> Our overriding desire is to get signatures "out there", but what's the right
>> next step? There are at least 4 options:
>>
>>  1. Remove the experimental sticker off signatures and release that way in 5.36
>>     (so you'd still have to `use feature` or `use v5.36`), but no other changes.
>>  2. As for 1, but also add a runtime warning if you touch @_ inside a
>>     signatured-sub.
>>  3. Like 2, but touching @_ is fatal.
>>  4. Inside signatured-subs @_ becomes non-special.
>
> I vote for something like number 3.
> <snip>

I partially retract what I said before.

To be honest, as a design feature I actually LIKE the "@_" of Perl, or
specifically the concept of a single variable or thing that has all the routine
arguments as a collection. I even make this a fundamental feature of my own
Muldis Data Language, where routines always conceptually have exactly 1 argument
which is collection-typed, specifically it is a Tuple, I call it "args".

The main reason my previous answer picked number 3 was based on liking the
argument that making "@_" non-special would be a big performance boost, and that
was the main reason I supported it.

If it were possible to do in a performant way, my actual preference is for the
behavior of signatures to fundamentally be a pretty syntactic shorthand for
unpacking "@_" elements into lexicals, and @_ continues to also be available.

-- Darren Duncan

On Mon, Jan 10, 2022 at 03:44:52AM -0800, Darren Duncan wrote:
> When a file says "use v5.36" or a higher version, it causes signatures to be
> available non-experimentally AND it causes @_ to not be declared within the
> scope of any sub using a signature, wherein any references to @_ will fail
> AT PARSE/COMPILE time.

This isn't practical with the way features work.

When you:

use v5.36;

the hints word is updated with the number of the bundle of features
for that version.

Now just that could work, the code could check for that specific
feature bundle number, *but* if the code then enables or disables any
feature, then the bundle number is set to the "custom" bundle, and
features are stored in %^H (and mirrored in cop_features for
performance), and the feature bundle number check will no longer work.

So for this to work, we'd need a separate signatures
(eg. "signatures36") bundle with the new behaviour, if the user wanted
to disable signatures for a brief section of code, eg if they want to
define several prototyped functions, they'd neeed to C<no feature
"signatures36";>.

So it gets ugly.

Tony

I have no clue at all whether this is possible, so I'm guessing folks
who know have thought of and rejected this. I have not seen that
mentioned (I may have missed it) so I'll ask.

On Mon, Jan 10, 2022 at 06:36:24PM +0000, Paul LeoNerd Evans wrote:
> But now consider some purely-dynamic runtime things that the parser
> could not know about:
>
> no strict 'refs';
> my $sym = "some code here";
> my $val = shift @{$sym};


Would it be possible to detect when some common subset of things was
true that would allow disabling populating @_? For example, strict
enabled, no touching of @_, no "goto" and no "eval". Perhaps some other
disallowed things, but maybe few enough that it would be possible to
mark the sub as "doesn't need @_" and not initialize it for those?

Perhaps this optimization can be separated from signatures completely
and it could speed up:

{ my $n = 0; sub next { ++$n } sub current { $n } }

Sure your examples couldn't be, but maybe the things that could be
detected that don't need @_ is enough to be worth it?

If there was a way to get the argument count without checking @_, and
you could check somehow why your function was not able to get the
optimization, folks could make sure their speed sensitive functions got
it.

--
andrew

Instructions are just another man's opinion of how to do something.
-- Weldboy #DPWisdom

On Mon, Jan 10, 2022 at 02:46:36PM -0800, Darren Duncan wrote:
> If it were possible to do in a performant way, my actual preference is for
> the behavior of signatures to fundamentally be a pretty syntactic shorthand
> for unpacking "@_" elements into lexicals, and @_ continues to also be
> available.

Good news - that's exactly how it already works :)

--

Paul "LeoNerd" Evans

leonerd@leonerd.org.uk | https://metacpan.org/author/PEVANS
http://www.leonerd.org.uk/ | https://www.tindie.com/stores/leonerd/

On Wed, Jan 12, 2022, at 9:20 AM, Paul LeoNerd Evans wrote:
> On Mon, Jan 10, 2022 at 02:46:36PM -0800, Darren Duncan wrote:
> > If it were possible to do in a performant way, my actual preference is for
> > the behavior of signatures to fundamentally be a pretty syntactic shorthand
> > for unpacking "@_" elements into lexicals, and @_ continues to also be
> > available.
>
> Good news - that's exactly how it already works :)

…and let's talk about "performant." That just means "effective" or, often in programming, "fast enough to be acceptable." Are subroutine signatures unacceptably slow? No. The idea that they aren't seems bizarre to me, since they're often *faster than equivalent non-signatured code already.*

Consider* *this program:
use v5.34.0;
use warnings;

use experimental 'signatures';

use Time::HiRes qw(time);

sub sigs ($x) { return $x * $x }
sub bare { my ($x) = @_; return $x * $x }
sub full { die "arity" unless @_ == 1; my ($x) = @_; return $x * $x }

sub million {
my ($name, $code) = @_;
my $before = time;
for (1 .. 1e7) { $code->(1) };
my $after = time;
printf "%s: %0.4fs\n", $name, $after - $before
}

million(sigs => \&sigs);
million(bare => \&bare);
million(full => \&full);

This calls each subroutine 10,000,000 times (despite the name "million"). One has a signature and a tiny body. One copies the argument out of @_. One does that but also checks arity. The more features of signatures you use, the more code you need to compare it to for a fair comparison. But let's start with this statement:

*"sigs" is doing more work and providing more value than "bare", and isn't merely more compact.*

So what's the timing? On my recent Intel MacBook, I routinely get results roughly like this:
sigs: 0.8656s
bare: 1.1357s
full: 1.2632s

The signatures are already faster. Could they be *faster-er* by eliminating @_ setup (and adjusting to some resulting confusion over time)? Yes, I have reason to believe that. But in that scenario we're talking about accelerating our accelerator, not compensating for drag.

--
rjbs

On 2022-01-12 8:06 a.m., Ricardo Signes wrote:
> On Wed, Jan 12, 2022, at 9:20 AM, Paul LeoNerd Evans wrote:
>> On Mon, Jan 10, 2022 at 02:46:36PM -0800, Darren Duncan wrote:
>> > If it were possible to do in a performant way, my actual preference is for
>> > the behavior of signatures to fundamentally be a pretty syntactic shorthand
>> > for unpacking "@_" elements into lexicals, and @_ continues to also be
>> > available.
>>
>> Good news - that's exactly how it already works :)
>
> …and let's talk about "performant."  That just means "effective" or, often in
> programming, "fast enough to be acceptable."  Are subroutine signatures
> unacceptably slow?  No.  The idea that they aren't seems bizarre to me, since
> they're often /faster than equivalent non-signatured code already./

Let me be clear. I don't personally have any issues with the performance
leading me to mention this. Rather my reference to that is about OTHER people
talking about how avoiding doing stuff with @_ could improve performance, and I
was supporting them in principle. -- Darren Duncan

On Wed, Jan 12, 2022 at 5:06 PM Ricardo Signes <perl.p5p@rjbs.manxome.org>
wrote:

> On Wed, Jan 12, 2022, at 9:20 AM, Paul LeoNerd Evans wrote:
>
> On Mon, Jan 10, 2022 at 02:46:36PM -0800, Darren Duncan wrote:
> > If it were possible to do in a performant way, my actual preference is
> for
> > the behavior of signatures to fundamentally be a pretty syntactic
> shorthand
> > for unpacking "@_" elements into lexicals, and @_ continues to also be
> > available.
>
> Good news - that's exactly how it already works :)
>
>
> …and let's talk about "performant." That just means "effective" or, often
> in programming, "fast enough to be acceptable." Are subroutine signatures
> unacceptably slow? No. The idea that they aren't seems bizarre to me,
> since they're often *faster than equivalent non-signatured code already.*
>
> Consider this program:
>
> use v5.34.0;
> use warnings;
>
> use experimental 'signatures';
>
> use Time::HiRes qw(time);
>
> sub sigs ($x) { return $x * $x }
> sub bare { my ($x) = @_; return $x * $x }
> sub full { die "arity" unless @_ == 1; my ($x) = @_; return $x * $x }
>
> sub million {
> my ($name, $code) = @_;
> my $before = time;
> for (1 .. 1e7) { $code->(1) };
> my $after = time;
> printf "%s: %0.4fs\n", $name, $after - $before
> }
>
> million(sigs => \&sigs);
> million(bare => \&bare);
> million(full => \&full);
>
>
> This calls each subroutine 10,000,000 times (despite the name "million").
> One has a signature and a tiny body. One copies the argument out of @_.
> One does that but also checks arity. The more features of signatures you
> use, the more code you need to compare it to for a fair comparison. But
> let's start with this statement:
>
> *"sigs" is doing more work and providing more value than "bare", and isn't
> merely more compact.*
>
> So what's the timing? On my recent Intel MacBook, I routinely get results
> roughly like this:
>
> sigs: 0.8656s
> bare: 1.1357s
> full: 1.2632s
>
>
> The signatures are already faster. Could they be *faster-er* by
> eliminating @_ setup (and adjusting to some resulting confusion over
> time)? Yes, I have reason to believe that. But in that scenario we're
> talking about accelerating our accelerator, not compensating for drag.
>
> --
> rjbs
>

Wow, that's amazing! I wasn't aware that sub sigs are faster and also by
that much!

Here are the times from my PC (Ryzen 3800X):
sigs: 0.7614s
bare: 0.8764s
full: 1.0507s

The multiplication affects the difference even more, when replaced with
just 'return $x' these are the results:
sigs: 0.6784s
bare: 0.8124s
full: 0.9844s

I've tried to convert your script to use Benchmark cmpthese but failed.
The docs don't say what COUNT should be but for this function, for timethis
it seems to be either really a count, the number of iterations the code is
executed, or negative to signal how long it should run.
With 1e7 and even 1e9 it still (indented and in brackets?!?!) outputs:
(warning: too few iterations for a reliable count)
With -3 it just hangs at 100% cpu usage forever.

I took some time and put rjbs's base script in to a |benchmark|
framework. I added two functions to simplify extracting |$x| instead of
using |shift()|. The overhead of using an array for this seems to take a
toll. After simplifying it |base_simp| is the fastest, but |sigs| is
very close behind.

|#!/usr/bin/env perl use strict; use warnings; use Benchmark
qw(cmpthese); use experimental 'signatures';
###############################################################################
my $count = 10000000; cmpthese($count, { 'sigs' => sub { sigs(10) ; },
'bare' => sub { bare(10) ; }, 'bare_simp' => sub { bare_simp(10); },
'full' => sub { full(10) ; }, 'full_simp' => sub { full_simp(10); }, });
sub sigs ($x) { return $x * $x } sub bare { my ($x) = @_; return $x * $x
} sub bare_simp { my $x = $_[0]; return $x * $x } sub full { die "arity"
unless @_ == 1; my ($x) = @_; return $x * $x } sub full_simp { die
"arity" unless @_ == 1; my $x = $_[0]; return $x * $x } __END__ $ time
perl /tmp/x.pl Rate full_simp full bare sigs bare_simp full_simp
5263158/s -- -6% -21% -29% -33% full 5617978/s 7% -- -16% -24% -28% bare
6666667/s 27% 19% -- -10% -15% sigs 7407407/s 41% 32% 11% -- -5%
bare_simp 7812500/s 48% 39% 17% 5% -- real 0m19.556s user 0m19.550s sys
0m0.003s |

On 1/12/2022 1:06 PM, Alexander Hartmaier wrote:

> Wow, that's amazing! I wasn't aware that sub sigs are faster and also
> by that much!
>
> Here are the times from my PC (Ryzen 3800X):
> sigs: 0.7614s
> bare: 0.8764s
> full: 1.0507s
>
> The multiplication affects the difference even more, when replaced
> with just 'return $x' these are the results:
> sigs: 0.6784s
> bare: 0.8124s
> full: 0.9844s
>
> I've tried to convert your script to use Benchmark cmpthese but failed.

?

On Sun, Jan 9, 2022 at 4:38 AM Neil Bowers <neilb@neilb.org> wrote:

> Our overriding desire is to get signatures "out there", but what's the
> right next step? There are at least 4 options:
>
> 1. Remove the experimental sticker off signatures and release that way
> in 5.36 (so you'd still have to `use feature` or `use v5.36`), but no other
> changes.
> 2. As for 1, but also add a runtime warning if you touch @_ inside a
> signatured-sub.
> 3. Like 2, but touching @_ is fatal.
> 4. Inside signatured-subs @_ becomes non-special.
>
>
I am sympathetic to the concerns that #2/3/4 will have backcompat issues. I
don't recall what the current situation is regarding cpan smokes, but this
seems like a fairly easy change to make in a side branch and smoke against
cpan to see what might break? I use signatures a lot in new code, and in
the cases where I touch @_ directly, I don't use a signature for that sub.
Perhaps the majority of users are in the same situation and there won't be
much fallout at all.

On Thu, Jan 13, 2022, at 2:06 PM, Karen Etheridge wrote:
> On Sun, Jan 9, 2022 at 4:38 AM Neil Bowers <neilb@neilb.org> wrote:
>> Our overriding desire is to get signatures "out there", but what's the right next step? There are at least 4 options:
>> 1. Remove the experimental sticker off signatures and release that way in 5.36 (so you'd still have to `use feature` or `use v5.36`), but no other changes.
>> 2. As for 1, but also add a runtime warning if you touch @_ inside a signatured-sub.
>> 3. Like 2, but touching @_ is fatal.
>> 4. Inside signatured-subs @_ becomes non-special.
>
> I am sympathetic to the concerns that #2/3/4 will have backcompat issues. I don't recall what the current situation is regarding cpan smokes, but this seems like a fairly easy change to make in a side branch and smoke against cpan to see what might break? I use signatures a lot in new code, and in the cases where I touch @_ directly, I don't use a signature for that sub. Perhaps the majority of users are in the same situation and there won't be much fallout at all.

They would be easy to smoke if we had the patch, but we don't.

#2 and #3 are, so far as I understand, not really practical without slowing down quite a lot of code.

#4 is the one that has no runtime cost, but does introduce (I believe) a fair bit of lurking surprise for the unwary developer. Also, we don't have a patch.

(You can see some of the early discussion of this whole topic, and how we might stop setting up @_ at all, in this 2016 p5p thread <http://markmail.org/message/iehupzyfuuuth3pg>.)

--
rjbs

Part of the problem here is that we don’t know what the end-point is, and we’d like our next step to be a step in the right direction.

If we make signatures non-experimental, and part of `use v5.36`, then we’ll be encouraging "everyone" to start using signatures. This is a good step forward.

I initially argued for #2, as I was worried that if are going to remove @_ in the future, we’ll be setting regular Perl programmers up for pain, if we don’t help them avoid using @_ in signatured subs. Once we make something non-experimental, we have a responsibility to those programmers.

Going with #1 possibly restricts what we can do in the future, but I still think we should do it, otherwise another year will have gone by without signatures. And for the bulk of Perl programmers, I suspect the big win is giving them what we already have.

Neil

Is it possible to include an arity check discussion?

The arity check discussion seems to be ignored.



2022?1?14?(?) 19:42 Neil Bowers <neilb@neilb.org>:

> Part of the problem here is that we don’t know what the end-point is, and
> we’d like our next step to be a step in the right direction.
>
> If we make signatures non-experimental, and part of `use v5.36`, then
> we’ll be encouraging "everyone" to start using signatures. This is a good
> step forward.
>
> I initially argued for #2, as I was worried that if are going to remove @_
> in the future, we’ll be setting regular Perl programmers up for pain, if we
> don’t help them avoid using @_ in signatured subs. Once we make something
> non-experimental, we have a responsibility to those programmers.
>
> Going with #1 possibly restricts what we can do in the future, but I still
> think we should do it, otherwise another year will have gone by without
> signatures. And for the bulk of Perl programmers, I suspect the big win is
> giving them what we already have.
>
> Neil
>

On Sat, Jan 15, 2022, at 6:02 PM, Yuki Kimoto wrote:
> Is it possible to include an arity check discussion?
>
> The arity check discussion seems to be ignored.

http://markmail.org/message/e7lvyan2ctd24cun

*This topic has been discussed numerous times over the last seven years, and I'm not going through it again. This will be my last reply to any message on the topic of changing the strict arity checking of subroutine signatures.**
*

--
rjbs

On 2022-01-15 4:43 p.m., Ricardo Signes wrote:
> On Sat, Jan 15, 2022, at 6:02 PM, Yuki Kimoto wrote:
>> Is it possible to include an arity check discussion?
>>
>> The arity check discussion seems to be ignored.
>
> http://markmail.org/message/e7lvyan2ctd24cun
>
> *This topic has been discussed numerous times over the last seven years, and I'm
> not going through it again.  This will be my last reply to any message on the
> topic of changing the strict arity checking of subroutine signatures.**
> *

I for one also support keeping the existing strict arity checking in place for
signatures.

If one is using signatures then that usually implies that the arguments they
explicitly declare are the arguments they expect and that someone calling it in
any other way is making an error.

When a routine writer wants to allow a variable number of arguments, there is a
means provided to declare that explicitly in signatures.

And otherwise routine writers who want the old-style least strict behavior can
still get it for a sub by not using signatures and using @_ instead.

-- Darren Duncan

On Sun, 16 Jan 2022 at 03:46, Darren Duncan <darren@darrenduncan.net> wrote:

> On 2022-01-15 4:43 p.m., Ricardo Signes wrote:
> > On Sat, Jan 15, 2022, at 6:02 PM, Yuki Kimoto wrote:
> >> Is it possible to include an arity check discussion?
> >>
> >> The arity check discussion seems to be ignored.
> >
> > http://markmail.org/message/e7lvyan2ctd24cun
> >
> > *This topic has been discussed numerous times over the last seven years,
> and I'm
> > not going through it again. This will be my last reply to any message
> on the
> > topic of changing the strict arity checking of subroutine signatures.**
> > *
>
> I for one also support keeping the existing strict arity checking in place
> for
> signatures.
>
> If one is using signatures then that usually implies that the arguments
> they
> explicitly declare are the arguments they expect and that someone calling
> it in
> any other way is making an error.
>
> When a routine writer wants to allow a variable number of arguments, there
> is a
> means provided to declare that explicitly in signatures.
>
> And otherwise routine writers who want the old-style least strict behavior
> can
> still get it for a sub by not using signatures and using @_ instead.
>

+1

Yves

--
perl -Mre=debug -e "/just|another|perl|hacker/"

There is an important technical and usability point remaining, but I have
removed Ricardo's address from the CC list to avoid harrassment:

On Sun, 16 Jan 2022 at 08:43, Ricardo Signes <perl.p5p@rjbs.manxome.org>
wrote:

> On Sat, Jan 15, 2022, at 6:02 PM, Yuki Kimoto wrote:
>
> Is it possible to include an arity check discussion?
>
> The arity check discussion seems to be ignored.
>
>
> http://markmail.org/message/e7lvyan2ctd24cun
>
> *This topic has been discussed numerous times over the last seven years,
> and I'm not going through it again. This will be my last reply to any
> message on the topic of changing the strict arity checking of subroutine
> signatures.*
>

Could we get clarification on whether this is an official PSC decision
terminating *all* further conversation on the topic, or a personal
statement of withdrawal from a still-open discussion?

The linked message does not even attempt to address the point raised in the
original message in that thread - I'd go as far as to say it exhibits a
lack of understanding of the point in the first place. I'd further note
that the thread was started by a PSC member, even after the 7 years of
previous discussion!

For reference, original message:

https://markmail.org/message/4hiidblkjramen5e

and as for "discussion seems to be ignored", I agree with this sentiment -
there was no followup to my offer here:

https://markmail.org/message/hw4px3tlbua5nfcr

To recap the original problem: someone who learns Perl-with-signatures who
wants to pass a callback to a CPAN module will be writing fragile code.
This is something that library authors have no control over. The suggestion
is "library documentation should tell developers that they must always
include , @ in callbacks". There are no tools or mechanisms offered to
enforce this... so people will leave it out, and things will work. They
will work until a new version of that library is released which now passes
additional optional parameters: then they will break.

Providing a way for the library code to deal with that - an explicit
opt-out for arity checking, *not* a change to the defaults across the rest
of Perl - seems like a basic courtesy to developers. If we are outright
rejecting that, I find that disappointing. More so if the given reason is
"we've been discussing signatures for 7 years already", rather than any
specific technical impediment.

On Sun, 16 Jan 2022 at 09:12, Tom Molesworth via perl5-porters <
perl5-porters@perl.org> wrote:

> There is an important technical and usability point remaining, but I have
> removed Ricardo's address from the CC list to avoid harrassment:
>
> On Sun, 16 Jan 2022 at 08:43, Ricardo Signes <perl.p5p@rjbs.manxome.org>
> wrote:
>
>> On Sat, Jan 15, 2022, at 6:02 PM, Yuki Kimoto wrote:
>>
>> Is it possible to include an arity check discussion?
>>
>> The arity check discussion seems to be ignored.
>>
>>
>> http://markmail.org/message/e7lvyan2ctd24cun
>>
>> *This topic has been discussed numerous times over the last seven years,
>> and I'm not going through it again. This will be my last reply to any
>> message on the topic of changing the strict arity checking of subroutine
>> signatures.*
>>
>
> Could we get clarification on whether this is an official PSC decision
> terminating *all* further conversation on the topic, or a personal
> statement of withdrawal from a still-open discussion?
>
> The linked message does not even attempt to address the point raised in
> the original message in that thread - I'd go as far as to say it exhibits a
> lack of understanding of the point in the first place. I'd further note
> that the thread was started by a PSC member, even after the 7 years of
> previous discussion!
>
> For reference, original message:
>
> https://markmail.org/message/4hiidblkjramen5e
>
> and as for "discussion seems to be ignored", I agree with this sentiment -
> there was no followup to my offer here:
>
> https://markmail.org/message/hw4px3tlbua5nfcr
>
> To recap the original problem: someone who learns Perl-with-signatures who
> wants to pass a callback to a CPAN module will be writing fragile code.
> This is something that library authors have no control over. The suggestion
> is "library documentation should tell developers that they must always
> include , @ in callbacks".
>

If those callbacks do not share a common signature then yes, that is what
they must do. I dont understand why you assume that any case where
callbacks are used will have inconsistent signatures. It seems to me that
it is equally likely most callbacks would be the other way around. Eg,
favoring one case or another is an arbitrary decision that comes down to
some external factoid about the situation.

FWIW, I used to have *very* strongly *aligned* views with you about the
warnings coming from sprintf Avar introduced ages back. At first I loathed
that sprintf would warn if I passed in more arguments than it used. But
over time I found that it actually saved me from embarrassment more often
than it hindered my projects. Not exactly your point, but not exactly a
different one either.


> There are no tools or mechanisms offered to enforce this... so people will
> leave it out, and things will work. They will work until a new version of
> that library is released which now passes additional optional parameters:
> then they will break.
>

Isn't this a question for the implementer of the callback mechanism?


>
> Providing a way for the library code to deal with that - an explicit
> opt-out for arity checking, *not* a change to the defaults across the rest
> of Perl - seems like a basic courtesy to developers. If we are outright
> rejecting that, I find that disappointing. More so if the given reason is
> "we've been discussing signatures for 7 years already", rather than any
> specific technical impediment.
>

But doesn't the @ mechanism do that? Eg, if an author of an API that takes
callbacks as a parameter should document that the callback API might change
over time and thus all subs SHOULD have a @ in it?

Is there some introspective approach that would help here? Eg, if I was a
callback author and I could interrogate that the signature did not have a @
as its last clause then I could warn if someone supplied me a callback that
didnt satisfy.

Anyway, I see the problem you are worried about and agree it has merit. But
I'm not sure that the policy approach that rjbs is taking is unreasonable
either. Is there a middle ground that would satisfy?

cheers,
Yves
--
perl -Mre=debug -e "/just|another|perl|hacker/"

On Sun, 16 Jan 2022 at 16:44, demerphq <demerphq@gmail.com> wrote:

> On Sun, 16 Jan 2022 at 09:12, Tom Molesworth via perl5-porters <
> perl5-porters@perl.org> wrote:
>
>> To recap the original problem: someone who learns Perl-with-signatures
>> who wants to pass a callback to a CPAN module will be writing fragile code.
>> This is something that library authors have no control over. The suggestion
>> is "library documentation should tell developers that they must always
>> include , @ in callbacks".
>>
>
> If those callbacks do not share a common signature then yes, that is what
> they must do. I dont understand why you assume that any case where
> callbacks are used will have inconsistent signatures. It seems to me that
> it is equally likely most callbacks would be the other way around. Eg,
> favoring one case or another is an arbitrary decision that comes down to
> some external factoid about the situation.
>

hmm - I think the external factoid here would be "was written for the
earlier version, before the new option was added"? There's momentary
convenience in being able to leave out the parameters without the `,@`, but
that's not really the focus. At the time the original code was written,
signatures would presumably be consistent - there's only one case, the
documented API with however many callback parameters were passed at the
time, so there's no decision or favouring involved...

From your questions here, I get the impression that perhaps you're still
not clear on exactly what type of situation is problematic? Just in case,
here's a trivial example:

Let's say we have a module X that provides `->each(sub ($item) { ... })` as
a method. Perhaps it has an array-like structure internally and allows you
to iterate over items.

As code builds up over time, such as `$obj->each(sub ($item) { say "Item:
$item" })`, everyone's happy, there may be an obscure note in the
documentation about putting in `, @` in the signature somewhere but things
work fine without that, and half the developers using this don't understand
what that meant (or didn't read it in the first place). This module starts
to occupy a key position in the CPAN dependency stream, and everyone loves
it...

... except for one unsatisfied customer who points out that this isn't
useful enough in the current state: it should also pass the index. They
raise a patch supporting their very-important usecase: `$obj->each(sub
($item, $index) { say "Item $item had index $index" })`.

Maybe all the tests already have the `,@` as advised, maybe tests are
updated to fix everything that broke... either way, that new version is
released. Anyone who upgrades and *didn't* put in that `,@` now faces
breakage: they have to go and update all their code to cope with it. You
might say that they should have read the documentation, that upgrades need
careful testing, or that they are just bad programmers who should have
written perfect code in the first place. Either way, there's unhappiness
and perhaps a tendency to look again at other languages like Javascript
where this type of extensible callback API change is comparatively easier.

If the fallout is too bad, the change may be reverted, perhaps a new method
is added - `->each_with_index`. Huffman proponents declare that the library
is now unusable and proceed to fork it or rewrite from scratch, much
acrimony ensues, and in the middle of all that someone points out that we
clearly needed a *third* parameter, a reference to the underlying arrayref
so that you can look at items either side of the current one. This is seen
as a welcome distraction and immediately the userbase fragments into 3
different camps: the document said ,@ so get used to it,
`->each_with_index_and_arrayref`, and `->each(sub ($item, $index,
$arrayref) { ... }, WITH_INDEX | WITH_ARRAYREF)`. Perhaps someone else
proposes a `use Library ":v2"` to get the new behaviour, or puts in a
try-catch-until-we-stop-seeing-signature-failure workaround. The resulting
discussion becomes heated and is posted on various social media outlets,
with wise and informed commentary on all sides, but for some strange reason
the development on that original library is slowing down a bit, and perhaps
it's just a tiny bit less popular than it once was.

Anyway, you get the point. As a developer, I'm left thinking "if I can do
various boundary-breaking things like inspecting the caller's lexicals via
PadWalker, then _not_ being able to check what a coderef expects to be
called with seems like an odd omission".


> FWIW, I used to have *very* strongly *aligned* views with you about the
> warnings coming from sprintf Avar introduced ages back. At first I loathed
> that sprintf would warn if I passed in more arguments than it used. But
> over time I found that it actually saved me from embarrassment more often
> than it hindered my projects. Not exactly your point, but not exactly a
> different one either.
>

Indeed so - and that's just a warning. This is a step further - it's a
runtime error. And again, in the sprintf case you controlled the format
string *and* the parameters, so it's understandable that you'd need to take
responsibility for making those consistent.


>
>> There are no tools or mechanisms offered to enforce this... so people
>> will leave it out, and things will work. They will work until a new version
>> of that library is released which now passes additional optional
>> parameters: then they will break.
>>
>
> Isn't this a question for the implementer of the callback mechanism?
>
>
>> Providing a way for the library code to deal with that - an explicit
>> opt-out for arity checking, *not* a change to the defaults across the rest
>> of Perl - seems like a basic courtesy to developers. If we are outright
>> rejecting that, I find that disappointing. More so if the given reason is
>> "we've been discussing signatures for 7 years already", rather than any
>> specific technical impediment.
>>
>
> But doesn't the @ mechanism do that? Eg, if an author of an API that
> takes callbacks as a parameter should document that the callback API might
> change over time and thus all subs SHOULD have a @ in it?
>

The @ mechanism does not do that, though:

> There are no tools or mechanisms offered to *enforce* this... so *people
will leave it out, and things will work*. They will work until a new
version of that library is released which now passes additional optional
parameters: then they will break.

Making it easier to be backwards-compatible would benefit both the library
developers *and* the users.

Is there some introspective approach that would help here? Eg, if I was a
> callback author and I could interrogate that the signature did not have a @
> as its last clause then I could warn if someone supplied me a callback that
> didnt satisfy.
>

Yes. From the original email in that thread:

> If we had some sort of introspection we could do some kind of
> `bounded_invoke( $callback, @args )` which would slice @args to no more
> than the signature says it wants.

Sadly the conversation drifted away from that original suggestion - I
didn't see any technical discussion on it.


> Anyway, I see the problem you are worried about and agree it has merit.
> But I'm not sure that the policy approach that rjbs is taking is
> unreasonable either. Is there a middle ground that would satisfy?
>

There is, yes - options have been proposed, including:

- introspection, as above
- provide a way to bypass arity checks for a specific call

If the answer is "no, we are not providing any of those", and that's an
official PSC decision, then there's not much more to be said. I have not
seen that stated explicitly (yet!). Would be nice to have a summary in the
archives explaining _why_ we aren't considering them - performance impact,
for example.