Mailing List Archive

On Sunday, 9 January 2022, 13:38:07 CET, Neil Bowers <neilb@neilb.org> wrote:

     
> 1. Remove the experimental sticker off signatures and release that way in 5.36 (so you'd still have to `use feature` or `use v5.36`), but no other changes.  
> 2. As for 1, but also add a runtime warning if you touch @_ inside a signatured-sub.
> 3. Like 2, but touching @_ is fatal.
> 4. Inside signatured-subs @_ becomes non-special.


I dislike #1 because that will drag out the @_ issue. if someone needs it, they can always use `sub foo (@args) {...}`

I dislike #3 and #4 (if I understand your intent) because that will break existing code, requiring more work for maintainers. They can't just enable/disable things. They have to rewrite code (again, assuming I am understanding correctly).

#2 seems to be to clearest path forward to removing this feature altogether.

Would there be a warnings category for disabling @_ warnings? Something like (no idea what to call it):

    no warnings "deprecated::args_array";

With that, we could silence that single warning, but it's easy to grep through a codebase for deprecated features. In fact, I'd love to see tons more deprecated warning categories to make it easier to track them in larger codebases. We have clients who could probably use something like this.

Best,
Ovid
-- 
IT consulting, training, specializing in Perl, databases, and agile development
http://www.allaroundtheworld.fr/. 

Buy my book! - http://bit.ly/beginning_perl

On Sun, Jan 9, 2022, at 7:37 AM, Neil Bowers wrote:
> Our overriding desire is to get signatures "out there", but what's the right next step? There are at least 4 options:
> 1. Remove the experimental sticker off signatures and release that way in 5.36 (so you'd still have to `use feature` or `use v5.36`), but no other changes.
> 2. As for 1, but also add a runtime warning if you touch @_ inside a signatured-sub.
> 3. Like 2, but touching @_ is fatal.
> 4. Inside signatured-subs @_ becomes non-special.

I am strongly in favor of #1.

It's possible I misunderstand the core goals of not setting up @_. I believe it's for performance. Signatured subroutines can be made faster than subs sans sigs. Instead of setting up @_ we can just not. So if I am missing a large component, somebody should say something.

But: the cost of this change is complexity at many levels. First, the ongoing complexity of writing subroutines with @_. "That's not complex, Rik, I've been doing that for twenty years!" Great, but for people who haven't been doing it for 20 years, it *is* complex and also weird.

For those of us who have been writing subroutines with @_ for 20 years, removing @_ is also weird. But *not* looking at @_ is pretty easy, so would we accept that tradeoff for some performance? Yes. But that tradeoff isn't actually on offer. We don't have a branch that provides this tradeoff. We have discussion over what one would be. Mostly, it's #4 above: accessing @_ is like accessing @~ or @(. It's some meaningless global inside a signatured sub, left with the contents of the last old-style sub. There's no warning for accessing it, because it's just some global. In fact, my understanding is that adding a warning would require that we add magic to @_ globally, slowing down more or less all subroutines. (Who remembers pseudohashes?)

So we can choose one the more complex versions of "make signatured subroutines faster" with no warnings, and do that *right away* and then remove the experimental flag after a year. But we don't have that branch. It's been the primary blocker on removing the experimental flag on signatures for years, I believe. Meanwhile, the only people who get the vastly improved experience of having signatures on subroutines are the ones who are willing to ignore the "this is experimental and might change at any time" label — which is exactly the behavior we want to avoid encouraging.

That "slow everything down" problem means that #2 is probably off the table.

Let's talk about #1 again. If we say "the signatures that you've had available for years, and that many of you have been using, are no longer experimental," then it's true: people can continue to see @_ in their subs, and that means that subroutine calls on signatured subs don't get the no-@_ optimization. *Things stay the same speed as now*. Meanwhile, the developer experience is significantly better.

If, later, we want to provide that optimization, we would need to provide a deprecation warning on accessing @_ in a subroutine with a signature. Given my belief about this slowing down all subroutines, I don't think we're likely to remove @_ in signatured subroutines if we make them non-experimental as is right now, *and I am okay with that.*

We introduced signatures in 5.20, over seven years ago, and they're still experimental. I think we're a lot better off making them non-experimental now and giving up on this optimization. Further, I think that signatures are *so much *better than not that I think we should immediately add the signatures feature to the 5.36 feature bundle.

--
rjbs

On 2022-01-09 4:37 a.m., Neil Bowers wrote:
> The topic we've discussed a lot recently is what to do with @_ in
> signatured-subs: if your sub has a signature then you shouldn’t be touching @_.
> If we can remove support for @_ in signatured subs, that could give a
> performance boost, and also remove a class of potential bugs.
>
> Our overriding desire is to get signatures "out there", but what's the right
> next step? There are at least 4 options:
>
> 1. Remove the experimental sticker off signatures and release that way in 5.36
> (so you'd still have to `use feature` or `use v5.36`), but no other changes.
> 2. As for 1, but also add a runtime warning if you touch @_ inside a
> signatured-sub.
> 3. Like 2, but touching @_ is fatal.
> 4. Inside signatured-subs @_ becomes non-special.

I vote for something like number 3.

When a file says "use v5.36" or a higher version, it causes signatures to be
available non-experimentally AND it causes @_ to not be declared within the
scope of any sub using a signature, wherein any references to @_ will fail AT
PARSE/COMPILE time.

Since "use v5.36" already enables strict, the failure with @_ can reported as a
reference to an undeclared variable like other violations of strict do.

Or at the very least, what I advocate for is that within the scope of any sub
having a signature, there does not exist an @_ with any kind of special
behavior, and the performance optimization exists.

That means any kind of explicit message to the user about @_, whether a warning
or a fatal error, it happens at parse/compile time, and not at runtime, of the
code, because runtime requires the special logic that slows things down.

I think making this fatal is best, so people don't gain bugs from @_ not having
the arguments because they just happened to miss new warnings.

This stricter behaviour should not break any existing code because it only
occurs when they explicitly "use v5.36" or they otherwise use-signatures without
experimental, meaning it is a change they explicitly opted into in the same file
as the affected sub.

-- Darren Duncan

On Sunday, 9 January 2022, 17:34:19 CET, Ricardo Signes <perl.p5p@rjbs.manxome.org> wrote:

> So we can choose one the more complex versions of "make signatured subroutines faster" with no 
> warnings, and do that right away and then remove the experimental flag after a year.  But we don't
> have that branch.  It's been the primary blocker on removing the experimental flag on signatures for
> years, I believe.  Meanwhile, the only people who get the vastly improved experience of having
> signatures on subroutines are the ones who are willing to ignore the "this is experimental and might
> change at any time" label — which is exactly the behavior we want to avoid encouraging.

I'm sold. Decoupling "make signatures non-experimental" and "deal with @_" sounds like the right way to go.

Best,
Ovid
-- 
IT consulting, training, specializing in Perl, databases, and agile development
http://www.allaroundtheworld.fr/. 

Buy my book! - http://bit.ly/beginning_perl

On Mon, Jan 10, 2022, at 6:44 AM, Darren Duncan wrote:.
> When a file says "use v5.36" or a higher version, it causes signatures to be
> available non-experimentally AND it causes @_ to not be declared within the
> scope of any sub using a signature, wherein any references to @_ will fail AT
> PARSE/COMPILE time.

What about this code:
use v5.36.0;

sub foo ($x) {
# Here we have a runtime reference to @_ -- what happens?
my $first = substr $x, 0, 1;

no strict 'refs';
say "Contents of \@$first: @{$first}"; # What if $x was "_foo"?
}

Does this become a runtime failure?

How do we enforce that runtime failure? Presumably we do it by making reading from @_ require magic. Right now, @_ is a standard array, and it's subroutine entry that does the magical work of filling it in. If we make reading from it magic, it will become magic everywhere, slowing down all subroutine calls, right?

I don't think we're going to get to have a warning on this. We can stop setting up @_ and let people deal with the fallout — that is, not knowing what the heck happened until they find the entry in perlsub detailing that @_ now has the leavings of other subroutines in it — or we can give up and ship what we already have.

Or maybe something else I haven't imagined. But given the choices so far, I want to ship what we have.

--
rjbs

On Mon, Jan 10, 2022 at 03:44:52AM -0800, Darren Duncan wrote:
> I vote for something like number 3.
>
> When a file says "use v5.36" or a higher version, it causes signatures to be
> available non-experimentally AND it causes @_ to not be declared within the
> scope of any sub using a signature, wherein any references to @_ will fail
> AT PARSE/COMPILE time.

@_ isn't "declared". It never has been, never would be.

After a dollar, at-sign or percent-sign, any single printing character
is permitted. If that printing char was a valid "start-of-identifier"
character, the parser then reads more chars to find a full identifier.
If not, then it stands for some superglobal punctuation var that is
always available - for instance, $!, $), etc...

Thus, by necessity, @_ always exists because _ is a printing character.

Making such code as

my $x = shift @_;

fail to parse at parse time requires adding a bunch of special-case code
to the parser/compiler, to recognise that @_ must not be permitted
inside signatured subs.

Also, breaking the surface-syntax part of @_ is insufficient here,
because of all the implicit uses of the array inside `@_`. For example

my $x = shift;

The parser never encounters the global *_ symbol here, nor the global @_
array. This gets compiled into an OP_SHIFT with a special flag which
tells the runtime to read GvAV(PL_defgv) to find the arguments array.

Forbidding this as well requires much more logic adding to the parser.

Now, all of theabove can be added - it's just a matter of writing more
code. Lots more code. In the parser. In lots of situations. It's a whole
bunch of work but a suitably-motivated individual could track down all
the cases and add them. Such a parser would be slower than it currently
is, because it has to check more things, but perhaps it wouldn't be
*much* slower that anyone could really measure. If it was, say, under 1%
slower on average, we're likely to be happy with the hit (because, on
average, computers get 1% faster every few weeks anyway so overall the
situation is already better than the last time we looked at it).

But now consider some purely-dynamic runtime things that the parser
could not know about:

no strict 'refs';
my $sym = "some code here";
my $val = shift @{$sym};

The parser can't possibly know that $sym might be given as "_" and thus
we'll have to defer this to runtime. Rik already points out in another
mail, why that is hard and requires adding magic to the GvAV(PL_defgv)
AV which would therefore slow down *all functions* in *all perl
programs*, regardless of whether that function happened to use
signatures, or even if any function at all in the program ever does.

...

> This stricter behaviour should not break any existing code because it only
> occurs when they explicitly "use v5.36" or they otherwise use-signatures
> without experimental, meaning it is a change they explicitly opted into in
> the same file as the affected sub.

Not directly related to your comment, but while we're here: There is
already much code on CPAN and elsewhere (and I know because I have
written some of it) that still makes use of @_ within a signatured sub.
There's two usecases:

1: Forwarding all the arguments to a wrapper function

sub wrap ($x, @) {
warn "called wrap with x=$x";
otherfunc(@_); # just makes this easier
}

2: Determining the argument-count, in order to distinguish passing an
`undef` from not having an argument at all:

sub field ($self, $new=undef) {
if(@_ > 1) {
$self->{field} = $new;
}
return $self->{field};
}

$obj->field(123);
print "field is ", $obj->field;

Yes I know this code is hideous and personally I hate that style
but it remains synatically- and semantically-valid perl that
nevertheless many folks actually do write, and want to continue to
write.

Quite aside from any "it might perform faster" or "it helps users avoid
bugs" arguments that might be put forward in favour of banning the use
of @_ in signatured subs, we need to keep these two usecases in mind
(among possibly others), and ensure we continue to provide for them.

It would be a shame if we had to tell users they can't perform behaviour
like either of these two cases, if they wish to use signatured subs.

--

Paul "LeoNerd" Evans

leonerd@leonerd.org.uk | https://metacpan.org/author/PEVANS
http://www.leonerd.org.uk/ | https://www.tindie.com/stores/leonerd/

On 2022-01-10 3:44 a.m., Darren Duncan wrote:
> On 2022-01-09 4:37 a.m., Neil Bowers wrote:
>> The topic we've discussed a lot recently is what to do with @_ in
>> signatured-subs: if your sub has a signature then you shouldn’t be touching
>> @_. If we can remove support for @_ in signatured subs, that could give a
>> performance boost, and also remove a class of potential bugs.
>>
>> Our overriding desire is to get signatures "out there", but what's the right
>> next step? There are at least 4 options:
>>
>>  1. Remove the experimental sticker off signatures and release that way in 5.36
>>     (so you'd still have to `use feature` or `use v5.36`), but no other changes.
>>  2. As for 1, but also add a runtime warning if you touch @_ inside a
>>     signatured-sub.
>>  3. Like 2, but touching @_ is fatal.
>>  4. Inside signatured-subs @_ becomes non-special.
>
> I vote for something like number 3.
> <snip>

I partially retract what I said before.

To be honest, as a design feature I actually LIKE the "@_" of Perl, or
specifically the concept of a single variable or thing that has all the routine
arguments as a collection. I even make this a fundamental feature of my own
Muldis Data Language, where routines always conceptually have exactly 1 argument
which is collection-typed, specifically it is a Tuple, I call it "args".

The main reason my previous answer picked number 3 was based on liking the
argument that making "@_" non-special would be a big performance boost, and that
was the main reason I supported it.

If it were possible to do in a performant way, my actual preference is for the
behavior of signatures to fundamentally be a pretty syntactic shorthand for
unpacking "@_" elements into lexicals, and @_ continues to also be available.

-- Darren Duncan

On Mon, Jan 10, 2022 at 03:44:52AM -0800, Darren Duncan wrote:
> When a file says "use v5.36" or a higher version, it causes signatures to be
> available non-experimentally AND it causes @_ to not be declared within the
> scope of any sub using a signature, wherein any references to @_ will fail
> AT PARSE/COMPILE time.

This isn't practical with the way features work.

When you:

use v5.36;

the hints word is updated with the number of the bundle of features
for that version.

Now just that could work, the code could check for that specific
feature bundle number, *but* if the code then enables or disables any
feature, then the bundle number is set to the "custom" bundle, and
features are stored in %^H (and mirrored in cop_features for
performance), and the feature bundle number check will no longer work.

So for this to work, we'd need a separate signatures
(eg. "signatures36") bundle with the new behaviour, if the user wanted
to disable signatures for a brief section of code, eg if they want to
define several prototyped functions, they'd neeed to C<no feature
"signatures36";>.

So it gets ugly.

Tony

I have no clue at all whether this is possible, so I'm guessing folks
who know have thought of and rejected this. I have not seen that
mentioned (I may have missed it) so I'll ask.

On Mon, Jan 10, 2022 at 06:36:24PM +0000, Paul LeoNerd Evans wrote:
> But now consider some purely-dynamic runtime things that the parser
> could not know about:
>
> no strict 'refs';
> my $sym = "some code here";
> my $val = shift @{$sym};


Would it be possible to detect when some common subset of things was
true that would allow disabling populating @_? For example, strict
enabled, no touching of @_, no "goto" and no "eval". Perhaps some other
disallowed things, but maybe few enough that it would be possible to
mark the sub as "doesn't need @_" and not initialize it for those?

Perhaps this optimization can be separated from signatures completely
and it could speed up:

{ my $n = 0; sub next { ++$n } sub current { $n } }

Sure your examples couldn't be, but maybe the things that could be
detected that don't need @_ is enough to be worth it?

If there was a way to get the argument count without checking @_, and
you could check somehow why your function was not able to get the
optimization, folks could make sure their speed sensitive functions got
it.

--
andrew

Instructions are just another man's opinion of how to do something.
-- Weldboy #DPWisdom

On Mon, Jan 10, 2022 at 02:46:36PM -0800, Darren Duncan wrote:
> If it were possible to do in a performant way, my actual preference is for
> the behavior of signatures to fundamentally be a pretty syntactic shorthand
> for unpacking "@_" elements into lexicals, and @_ continues to also be
> available.

Good news - that's exactly how it already works :)

--

Paul "LeoNerd" Evans

leonerd@leonerd.org.uk | https://metacpan.org/author/PEVANS
http://www.leonerd.org.uk/ | https://www.tindie.com/stores/leonerd/

On Wed, Jan 12, 2022, at 9:20 AM, Paul LeoNerd Evans wrote:
> On Mon, Jan 10, 2022 at 02:46:36PM -0800, Darren Duncan wrote:
> > If it were possible to do in a performant way, my actual preference is for
> > the behavior of signatures to fundamentally be a pretty syntactic shorthand
> > for unpacking "@_" elements into lexicals, and @_ continues to also be
> > available.
>
> Good news - that's exactly how it already works :)

…and let's talk about "performant." That just means "effective" or, often in programming, "fast enough to be acceptable." Are subroutine signatures unacceptably slow? No. The idea that they aren't seems bizarre to me, since they're often *faster than equivalent non-signatured code already.*

Consider* *this program:
use v5.34.0;
use warnings;

use experimental 'signatures';

use Time::HiRes qw(time);

sub sigs ($x) { return $x * $x }
sub bare { my ($x) = @_; return $x * $x }
sub full { die "arity" unless @_ == 1; my ($x) = @_; return $x * $x }

sub million {
my ($name, $code) = @_;
my $before = time;
for (1 .. 1e7) { $code->(1) };
my $after = time;
printf "%s: %0.4fs\n", $name, $after - $before
}

million(sigs => \&sigs);
million(bare => \&bare);
million(full => \&full);

This calls each subroutine 10,000,000 times (despite the name "million"). One has a signature and a tiny body. One copies the argument out of @_. One does that but also checks arity. The more features of signatures you use, the more code you need to compare it to for a fair comparison. But let's start with this statement:

*"sigs" is doing more work and providing more value than "bare", and isn't merely more compact.*

So what's the timing? On my recent Intel MacBook, I routinely get results roughly like this:
sigs: 0.8656s
bare: 1.1357s
full: 1.2632s

The signatures are already faster. Could they be *faster-er* by eliminating @_ setup (and adjusting to some resulting confusion over time)? Yes, I have reason to believe that. But in that scenario we're talking about accelerating our accelerator, not compensating for drag.

--
rjbs

On 2022-01-12 8:06 a.m., Ricardo Signes wrote:
> On Wed, Jan 12, 2022, at 9:20 AM, Paul LeoNerd Evans wrote:
>> On Mon, Jan 10, 2022 at 02:46:36PM -0800, Darren Duncan wrote:
>> > If it were possible to do in a performant way, my actual preference is for
>> > the behavior of signatures to fundamentally be a pretty syntactic shorthand
>> > for unpacking "@_" elements into lexicals, and @_ continues to also be
>> > available.
>>
>> Good news - that's exactly how it already works :)
>
> …and let's talk about "performant."  That just means "effective" or, often in
> programming, "fast enough to be acceptable."  Are subroutine signatures
> unacceptably slow?  No.  The idea that they aren't seems bizarre to me, since
> they're often /faster than equivalent non-signatured code already./

Let me be clear. I don't personally have any issues with the performance
leading me to mention this. Rather my reference to that is about OTHER people
talking about how avoiding doing stuff with @_ could improve performance, and I
was supporting them in principle. -- Darren Duncan

On Wed, Jan 12, 2022 at 5:06 PM Ricardo Signes <perl.p5p@rjbs.manxome.org>
wrote:

> On Wed, Jan 12, 2022, at 9:20 AM, Paul LeoNerd Evans wrote:
>
> On Mon, Jan 10, 2022 at 02:46:36PM -0800, Darren Duncan wrote:
> > If it were possible to do in a performant way, my actual preference is
> for
> > the behavior of signatures to fundamentally be a pretty syntactic
> shorthand
> > for unpacking "@_" elements into lexicals, and @_ continues to also be
> > available.
>
> Good news - that's exactly how it already works :)
>
>
> …and let's talk about "performant." That just means "effective" or, often
> in programming, "fast enough to be acceptable." Are subroutine signatures
> unacceptably slow? No. The idea that they aren't seems bizarre to me,
> since they're often *faster than equivalent non-signatured code already.*
>
> Consider this program:
>
> use v5.34.0;
> use warnings;
>
> use experimental 'signatures';
>
> use Time::HiRes qw(time);
>
> sub sigs ($x) { return $x * $x }
> sub bare { my ($x) = @_; return $x * $x }
> sub full { die "arity" unless @_ == 1; my ($x) = @_; return $x * $x }
>
> sub million {
> my ($name, $code) = @_;
> my $before = time;
> for (1 .. 1e7) { $code->(1) };
> my $after = time;
> printf "%s: %0.4fs\n", $name, $after - $before
> }
>
> million(sigs => \&sigs);
> million(bare => \&bare);
> million(full => \&full);
>
>
> This calls each subroutine 10,000,000 times (despite the name "million").
> One has a signature and a tiny body. One copies the argument out of @_.
> One does that but also checks arity. The more features of signatures you
> use, the more code you need to compare it to for a fair comparison. But
> let's start with this statement:
>
> *"sigs" is doing more work and providing more value than "bare", and isn't
> merely more compact.*
>
> So what's the timing? On my recent Intel MacBook, I routinely get results
> roughly like this:
>
> sigs: 0.8656s
> bare: 1.1357s
> full: 1.2632s
>
>
> The signatures are already faster. Could they be *faster-er* by
> eliminating @_ setup (and adjusting to some resulting confusion over
> time)? Yes, I have reason to believe that. But in that scenario we're
> talking about accelerating our accelerator, not compensating for drag.
>
> --
> rjbs
>

Wow, that's amazing! I wasn't aware that sub sigs are faster and also by
that much!

Here are the times from my PC (Ryzen 3800X):
sigs: 0.7614s
bare: 0.8764s
full: 1.0507s

The multiplication affects the difference even more, when replaced with
just 'return $x' these are the results:
sigs: 0.6784s
bare: 0.8124s
full: 0.9844s

I've tried to convert your script to use Benchmark cmpthese but failed.
The docs don't say what COUNT should be but for this function, for timethis
it seems to be either really a count, the number of iterations the code is
executed, or negative to signal how long it should run.
With 1e7 and even 1e9 it still (indented and in brackets?!?!) outputs:
(warning: too few iterations for a reliable count)
With -3 it just hangs at 100% cpu usage forever.

I took some time and put rjbs's base script in to a |benchmark|
framework. I added two functions to simplify extracting |$x| instead of
using |shift()|. The overhead of using an array for this seems to take a
toll. After simplifying it |base_simp| is the fastest, but |sigs| is
very close behind.

|#!/usr/bin/env perl use strict; use warnings; use Benchmark
qw(cmpthese); use experimental 'signatures';
###############################################################################
my $count = 10000000; cmpthese($count, { 'sigs' => sub { sigs(10) ; },
'bare' => sub { bare(10) ; }, 'bare_simp' => sub { bare_simp(10); },
'full' => sub { full(10) ; }, 'full_simp' => sub { full_simp(10); }, });
sub sigs ($x) { return $x * $x } sub bare { my ($x) = @_; return $x * $x
} sub bare_simp { my $x = $_[0]; return $x * $x } sub full { die "arity"
unless @_ == 1; my ($x) = @_; return $x * $x } sub full_simp { die
"arity" unless @_ == 1; my $x = $_[0]; return $x * $x } __END__ $ time
perl /tmp/x.pl Rate full_simp full bare sigs bare_simp full_simp
5263158/s -- -6% -21% -29% -33% full 5617978/s 7% -- -16% -24% -28% bare
6666667/s 27% 19% -- -10% -15% sigs 7407407/s 41% 32% 11% -- -5%
bare_simp 7812500/s 48% 39% 17% 5% -- real 0m19.556s user 0m19.550s sys
0m0.003s |

On 1/12/2022 1:06 PM, Alexander Hartmaier wrote:

> Wow, that's amazing! I wasn't aware that sub sigs are faster and also
> by that much!
>
> Here are the times from my PC (Ryzen 3800X):
> sigs: 0.7614s
> bare: 0.8764s
> full: 1.0507s
>
> The multiplication affects the difference even more, when replaced
> with just 'return $x' these are the results:
> sigs: 0.6784s
> bare: 0.8124s
> full: 0.9844s
>
> I've tried to convert your script to use Benchmark cmpthese but failed.

?

On Sun, Jan 9, 2022 at 4:38 AM Neil Bowers <neilb@neilb.org> wrote:

> Our overriding desire is to get signatures "out there", but what's the
> right next step? There are at least 4 options:
>
> 1. Remove the experimental sticker off signatures and release that way
> in 5.36 (so you'd still have to `use feature` or `use v5.36`), but no other
> changes.
> 2. As for 1, but also add a runtime warning if you touch @_ inside a
> signatured-sub.
> 3. Like 2, but touching @_ is fatal.
> 4. Inside signatured-subs @_ becomes non-special.
>
>
I am sympathetic to the concerns that #2/3/4 will have backcompat issues. I
don't recall what the current situation is regarding cpan smokes, but this
seems like a fairly easy change to make in a side branch and smoke against
cpan to see what might break? I use signatures a lot in new code, and in
the cases where I touch @_ directly, I don't use a signature for that sub.
Perhaps the majority of users are in the same situation and there won't be
much fallout at all.

On Thu, Jan 13, 2022, at 2:06 PM, Karen Etheridge wrote:
> On Sun, Jan 9, 2022 at 4:38 AM Neil Bowers <neilb@neilb.org> wrote:
>> Our overriding desire is to get signatures "out there", but what's the right next step? There are at least 4 options:
>> 1. Remove the experimental sticker off signatures and release that way in 5.36 (so you'd still have to `use feature` or `use v5.36`), but no other changes.
>> 2. As for 1, but also add a runtime warning if you touch @_ inside a signatured-sub.
>> 3. Like 2, but touching @_ is fatal.
>> 4. Inside signatured-subs @_ becomes non-special.
>
> I am sympathetic to the concerns that #2/3/4 will have backcompat issues. I don't recall what the current situation is regarding cpan smokes, but this seems like a fairly easy change to make in a side branch and smoke against cpan to see what might break? I use signatures a lot in new code, and in the cases where I touch @_ directly, I don't use a signature for that sub. Perhaps the majority of users are in the same situation and there won't be much fallout at all.

They would be easy to smoke if we had the patch, but we don't.

#2 and #3 are, so far as I understand, not really practical without slowing down quite a lot of code.

#4 is the one that has no runtime cost, but does introduce (I believe) a fair bit of lurking surprise for the unwary developer. Also, we don't have a patch.

(You can see some of the early discussion of this whole topic, and how we might stop setting up @_ at all, in this 2016 p5p thread <http://markmail.org/message/iehupzyfuuuth3pg>.)

--
rjbs

Part of the problem here is that we don’t know what the end-point is, and we’d like our next step to be a step in the right direction.

If we make signatures non-experimental, and part of `use v5.36`, then we’ll be encouraging "everyone" to start using signatures. This is a good step forward.

I initially argued for #2, as I was worried that if are going to remove @_ in the future, we’ll be setting regular Perl programmers up for pain, if we don’t help them avoid using @_ in signatured subs. Once we make something non-experimental, we have a responsibility to those programmers.

Going with #1 possibly restricts what we can do in the future, but I still think we should do it, otherwise another year will have gone by without signatures. And for the bulk of Perl programmers, I suspect the big win is giving them what we already have.

Neil

Is it possible to include an arity check discussion?

The arity check discussion seems to be ignored.



2022?1?14?(?) 19:42 Neil Bowers <neilb@neilb.org>:

> Part of the problem here is that we don’t know what the end-point is, and
> we’d like our next step to be a step in the right direction.
>
> If we make signatures non-experimental, and part of `use v5.36`, then
> we’ll be encouraging "everyone" to start using signatures. This is a good
> step forward.
>
> I initially argued for #2, as I was worried that if are going to remove @_
> in the future, we’ll be setting regular Perl programmers up for pain, if we
> don’t help them avoid using @_ in signatured subs. Once we make something
> non-experimental, we have a responsibility to those programmers.
>
> Going with #1 possibly restricts what we can do in the future, but I still
> think we should do it, otherwise another year will have gone by without
> signatures. And for the bulk of Perl programmers, I suspect the big win is
> giving them what we already have.
>
> Neil
>

On Sat, Jan 15, 2022, at 6:02 PM, Yuki Kimoto wrote:
> Is it possible to include an arity check discussion?
>
> The arity check discussion seems to be ignored.

http://markmail.org/message/e7lvyan2ctd24cun

*This topic has been discussed numerous times over the last seven years, and I'm not going through it again. This will be my last reply to any message on the topic of changing the strict arity checking of subroutine signatures.**
*

--
rjbs

On 2022-01-15 4:43 p.m., Ricardo Signes wrote:
> On Sat, Jan 15, 2022, at 6:02 PM, Yuki Kimoto wrote:
>> Is it possible to include an arity check discussion?
>>
>> The arity check discussion seems to be ignored.
>
> http://markmail.org/message/e7lvyan2ctd24cun
>
> *This topic has been discussed numerous times over the last seven years, and I'm
> not going through it again.  This will be my last reply to any message on the
> topic of changing the strict arity checking of subroutine signatures.**
> *

I for one also support keeping the existing strict arity checking in place for
signatures.

If one is using signatures then that usually implies that the arguments they
explicitly declare are the arguments they expect and that someone calling it in
any other way is making an error.

When a routine writer wants to allow a variable number of arguments, there is a
means provided to declare that explicitly in signatures.

And otherwise routine writers who want the old-style least strict behavior can
still get it for a sub by not using signatures and using @_ instead.

-- Darren Duncan

On Sun, 16 Jan 2022 at 03:46, Darren Duncan <darren@darrenduncan.net> wrote:

> On 2022-01-15 4:43 p.m., Ricardo Signes wrote:
> > On Sat, Jan 15, 2022, at 6:02 PM, Yuki Kimoto wrote:
> >> Is it possible to include an arity check discussion?
> >>
> >> The arity check discussion seems to be ignored.
> >
> > http://markmail.org/message/e7lvyan2ctd24cun
> >
> > *This topic has been discussed numerous times over the last seven years,
> and I'm
> > not going through it again. This will be my last reply to any message
> on the
> > topic of changing the strict arity checking of subroutine signatures.**
> > *
>
> I for one also support keeping the existing strict arity checking in place
> for
> signatures.
>
> If one is using signatures then that usually implies that the arguments
> they
> explicitly declare are the arguments they expect and that someone calling
> it in
> any other way is making an error.
>
> When a routine writer wants to allow a variable number of arguments, there
> is a
> means provided to declare that explicitly in signatures.
>
> And otherwise routine writers who want the old-style least strict behavior
> can
> still get it for a sub by not using signatures and using @_ instead.
>

+1

Yves

--
perl -Mre=debug -e "/just|another|perl|hacker/"

There is an important technical and usability point remaining, but I have
removed Ricardo's address from the CC list to avoid harrassment:

On Sun, 16 Jan 2022 at 08:43, Ricardo Signes <perl.p5p@rjbs.manxome.org>
wrote:

> On Sat, Jan 15, 2022, at 6:02 PM, Yuki Kimoto wrote:
>
> Is it possible to include an arity check discussion?
>
> The arity check discussion seems to be ignored.
>
>
> http://markmail.org/message/e7lvyan2ctd24cun
>
> *This topic has been discussed numerous times over the last seven years,
> and I'm not going through it again. This will be my last reply to any
> message on the topic of changing the strict arity checking of subroutine
> signatures.*
>

Could we get clarification on whether this is an official PSC decision
terminating *all* further conversation on the topic, or a personal
statement of withdrawal from a still-open discussion?

The linked message does not even attempt to address the point raised in the
original message in that thread - I'd go as far as to say it exhibits a
lack of understanding of the point in the first place. I'd further note
that the thread was started by a PSC member, even after the 7 years of
previous discussion!

For reference, original message:

https://markmail.org/message/4hiidblkjramen5e

and as for "discussion seems to be ignored", I agree with this sentiment -
there was no followup to my offer here:

https://markmail.org/message/hw4px3tlbua5nfcr

To recap the original problem: someone who learns Perl-with-signatures who
wants to pass a callback to a CPAN module will be writing fragile code.
This is something that library authors have no control over. The suggestion
is "library documentation should tell developers that they must always
include , @ in callbacks". There are no tools or mechanisms offered to
enforce this... so people will leave it out, and things will work. They
will work until a new version of that library is released which now passes
additional optional parameters: then they will break.

Providing a way for the library code to deal with that - an explicit
opt-out for arity checking, *not* a change to the defaults across the rest
of Perl - seems like a basic courtesy to developers. If we are outright
rejecting that, I find that disappointing. More so if the given reason is
"we've been discussing signatures for 7 years already", rather than any
specific technical impediment.

On Sun, 16 Jan 2022 at 09:12, Tom Molesworth via perl5-porters <
perl5-porters@perl.org> wrote:

> There is an important technical and usability point remaining, but I have
> removed Ricardo's address from the CC list to avoid harrassment:
>
> On Sun, 16 Jan 2022 at 08:43, Ricardo Signes <perl.p5p@rjbs.manxome.org>
> wrote:
>
>> On Sat, Jan 15, 2022, at 6:02 PM, Yuki Kimoto wrote:
>>
>> Is it possible to include an arity check discussion?
>>
>> The arity check discussion seems to be ignored.
>>
>>
>> http://markmail.org/message/e7lvyan2ctd24cun
>>
>> *This topic has been discussed numerous times over the last seven years,
>> and I'm not going through it again. This will be my last reply to any
>> message on the topic of changing the strict arity checking of subroutine
>> signatures.*
>>
>
> Could we get clarification on whether this is an official PSC decision
> terminating *all* further conversation on the topic, or a personal
> statement of withdrawal from a still-open discussion?
>
> The linked message does not even attempt to address the point raised in
> the original message in that thread - I'd go as far as to say it exhibits a
> lack of understanding of the point in the first place. I'd further note
> that the thread was started by a PSC member, even after the 7 years of
> previous discussion!
>
> For reference, original message:
>
> https://markmail.org/message/4hiidblkjramen5e
>
> and as for "discussion seems to be ignored", I agree with this sentiment -
> there was no followup to my offer here:
>
> https://markmail.org/message/hw4px3tlbua5nfcr
>
> To recap the original problem: someone who learns Perl-with-signatures who
> wants to pass a callback to a CPAN module will be writing fragile code.
> This is something that library authors have no control over. The suggestion
> is "library documentation should tell developers that they must always
> include , @ in callbacks".
>

If those callbacks do not share a common signature then yes, that is what
they must do. I dont understand why you assume that any case where
callbacks are used will have inconsistent signatures. It seems to me that
it is equally likely most callbacks would be the other way around. Eg,
favoring one case or another is an arbitrary decision that comes down to
some external factoid about the situation.

FWIW, I used to have *very* strongly *aligned* views with you about the
warnings coming from sprintf Avar introduced ages back. At first I loathed
that sprintf would warn if I passed in more arguments than it used. But
over time I found that it actually saved me from embarrassment more often
than it hindered my projects. Not exactly your point, but not exactly a
different one either.


> There are no tools or mechanisms offered to enforce this... so people will
> leave it out, and things will work. They will work until a new version of
> that library is released which now passes additional optional parameters:
> then they will break.
>

Isn't this a question for the implementer of the callback mechanism?


>
> Providing a way for the library code to deal with that - an explicit
> opt-out for arity checking, *not* a change to the defaults across the rest
> of Perl - seems like a basic courtesy to developers. If we are outright
> rejecting that, I find that disappointing. More so if the given reason is
> "we've been discussing signatures for 7 years already", rather than any
> specific technical impediment.
>

But doesn't the @ mechanism do that? Eg, if an author of an API that takes
callbacks as a parameter should document that the callback API might change
over time and thus all subs SHOULD have a @ in it?

Is there some introspective approach that would help here? Eg, if I was a
callback author and I could interrogate that the signature did not have a @
as its last clause then I could warn if someone supplied me a callback that
didnt satisfy.

Anyway, I see the problem you are worried about and agree it has merit. But
I'm not sure that the policy approach that rjbs is taking is unreasonable
either. Is there a middle ground that would satisfy?

cheers,
Yves
--
perl -Mre=debug -e "/just|another|perl|hacker/"

On Sun, 16 Jan 2022 at 16:44, demerphq <demerphq@gmail.com> wrote:

> On Sun, 16 Jan 2022 at 09:12, Tom Molesworth via perl5-porters <
> perl5-porters@perl.org> wrote:
>
>> To recap the original problem: someone who learns Perl-with-signatures
>> who wants to pass a callback to a CPAN module will be writing fragile code.
>> This is something that library authors have no control over. The suggestion
>> is "library documentation should tell developers that they must always
>> include , @ in callbacks".
>>
>
> If those callbacks do not share a common signature then yes, that is what
> they must do. I dont understand why you assume that any case where
> callbacks are used will have inconsistent signatures. It seems to me that
> it is equally likely most callbacks would be the other way around. Eg,
> favoring one case or another is an arbitrary decision that comes down to
> some external factoid about the situation.
>

hmm - I think the external factoid here would be "was written for the
earlier version, before the new option was added"? There's momentary
convenience in being able to leave out the parameters without the `,@`, but
that's not really the focus. At the time the original code was written,
signatures would presumably be consistent - there's only one case, the
documented API with however many callback parameters were passed at the
time, so there's no decision or favouring involved...

From your questions here, I get the impression that perhaps you're still
not clear on exactly what type of situation is problematic? Just in case,
here's a trivial example:

Let's say we have a module X that provides `->each(sub ($item) { ... })` as
a method. Perhaps it has an array-like structure internally and allows you
to iterate over items.

As code builds up over time, such as `$obj->each(sub ($item) { say "Item:
$item" })`, everyone's happy, there may be an obscure note in the
documentation about putting in `, @` in the signature somewhere but things
work fine without that, and half the developers using this don't understand
what that meant (or didn't read it in the first place). This module starts
to occupy a key position in the CPAN dependency stream, and everyone loves
it...

... except for one unsatisfied customer who points out that this isn't
useful enough in the current state: it should also pass the index. They
raise a patch supporting their very-important usecase: `$obj->each(sub
($item, $index) { say "Item $item had index $index" })`.

Maybe all the tests already have the `,@` as advised, maybe tests are
updated to fix everything that broke... either way, that new version is
released. Anyone who upgrades and *didn't* put in that `,@` now faces
breakage: they have to go and update all their code to cope with it. You
might say that they should have read the documentation, that upgrades need
careful testing, or that they are just bad programmers who should have
written perfect code in the first place. Either way, there's unhappiness
and perhaps a tendency to look again at other languages like Javascript
where this type of extensible callback API change is comparatively easier.

If the fallout is too bad, the change may be reverted, perhaps a new method
is added - `->each_with_index`. Huffman proponents declare that the library
is now unusable and proceed to fork it or rewrite from scratch, much
acrimony ensues, and in the middle of all that someone points out that we
clearly needed a *third* parameter, a reference to the underlying arrayref
so that you can look at items either side of the current one. This is seen
as a welcome distraction and immediately the userbase fragments into 3
different camps: the document said ,@ so get used to it,
`->each_with_index_and_arrayref`, and `->each(sub ($item, $index,
$arrayref) { ... }, WITH_INDEX | WITH_ARRAYREF)`. Perhaps someone else
proposes a `use Library ":v2"` to get the new behaviour, or puts in a
try-catch-until-we-stop-seeing-signature-failure workaround. The resulting
discussion becomes heated and is posted on various social media outlets,
with wise and informed commentary on all sides, but for some strange reason
the development on that original library is slowing down a bit, and perhaps
it's just a tiny bit less popular than it once was.

Anyway, you get the point. As a developer, I'm left thinking "if I can do
various boundary-breaking things like inspecting the caller's lexicals via
PadWalker, then _not_ being able to check what a coderef expects to be
called with seems like an odd omission".


> FWIW, I used to have *very* strongly *aligned* views with you about the
> warnings coming from sprintf Avar introduced ages back. At first I loathed
> that sprintf would warn if I passed in more arguments than it used. But
> over time I found that it actually saved me from embarrassment more often
> than it hindered my projects. Not exactly your point, but not exactly a
> different one either.
>

Indeed so - and that's just a warning. This is a step further - it's a
runtime error. And again, in the sprintf case you controlled the format
string *and* the parameters, so it's understandable that you'd need to take
responsibility for making those consistent.


>
>> There are no tools or mechanisms offered to enforce this... so people
>> will leave it out, and things will work. They will work until a new version
>> of that library is released which now passes additional optional
>> parameters: then they will break.
>>
>
> Isn't this a question for the implementer of the callback mechanism?
>
>
>> Providing a way for the library code to deal with that - an explicit
>> opt-out for arity checking, *not* a change to the defaults across the rest
>> of Perl - seems like a basic courtesy to developers. If we are outright
>> rejecting that, I find that disappointing. More so if the given reason is
>> "we've been discussing signatures for 7 years already", rather than any
>> specific technical impediment.
>>
>
> But doesn't the @ mechanism do that? Eg, if an author of an API that
> takes callbacks as a parameter should document that the callback API might
> change over time and thus all subs SHOULD have a @ in it?
>

The @ mechanism does not do that, though:

> There are no tools or mechanisms offered to *enforce* this... so *people
will leave it out, and things will work*. They will work until a new
version of that library is released which now passes additional optional
parameters: then they will break.

Making it easier to be backwards-compatible would benefit both the library
developers *and* the users.

Is there some introspective approach that would help here? Eg, if I was a
> callback author and I could interrogate that the signature did not have a @
> as its last clause then I could warn if someone supplied me a callback that
> didnt satisfy.
>

Yes. From the original email in that thread:

> If we had some sort of introspection we could do some kind of
> `bounded_invoke( $callback, @args )` which would slice @args to no more
> than the signature says it wants.

Sadly the conversation drifted away from that original suggestion - I
didn't see any technical discussion on it.


> Anyway, I see the problem you are worried about and agree it has merit.
> But I'm not sure that the policy approach that rjbs is taking is
> unreasonable either. Is there a middle ground that would satisfy?
>

There is, yes - options have been proposed, including:

- introspection, as above
- provide a way to bypass arity checks for a specific call

If the answer is "no, we are not providing any of those", and that's an
official PSC decision, then there's not much more to be said. I have not
seen that stated explicitly (yet!). Would be nice to have a summary in the
archives explaining _why_ we aren't considering them - performance impact,
for example.

On Sun, 16 Jan 2022 at 11:30, Tom Molesworth <tom@deriv.com> wrote:

> On Sun, 16 Jan 2022 at 16:44, demerphq <demerphq@gmail.com> wrote:
>
>> On Sun, 16 Jan 2022 at 09:12, Tom Molesworth via perl5-porters <
>> perl5-porters@perl.org> wrote:
>>
>>> To recap the original problem: someone who learns Perl-with-signatures
>>> who wants to pass a callback to a CPAN module will be writing fragile code.
>>> This is something that library authors have no control over. The suggestion
>>> is "library documentation should tell developers that they must always
>>> include , @ in callbacks".
>>>
>>
>> If those callbacks do not share a common signature then yes, that is what
>> they must do. I dont understand why you assume that any case where
>> callbacks are used will have inconsistent signatures. It seems to me that
>> it is equally likely most callbacks would be the other way around. Eg,
>> favoring one case or another is an arbitrary decision that comes down to
>> some external factoid about the situation.
>>
>
> hmm - I think the external factoid here would be "was written for the
> earlier version, before the new option was added"? There's momentary
> convenience in being able to leave out the parameters without the `,@`, but
> that's not really the focus. At the time the original code was written,
> signatures would presumably be consistent - there's only one case, the
> documented API with however many callback parameters were passed at the
> time, so there's no decision or favouring involved...
>

Ok, I see where you are coming from, i got it in my head before, but your
clarication moved that to the heart, if you know what I mean. I do see that
this would be quite irritating a major blocker to rolling our a new API
version with enhancements over a previous version. Old code that used to be
"just fine" would break because it doesn't accept the new parameter.


>
> From your questions here, I get the impression that perhaps you're still
> not clear on exactly what type of situation is problematic?
>

FWIW, I think I did, but the clarification is still helpful, it drives home
the point well.


> Just in case, here's a trivial example:
>
> Let's say we have a module X that provides `->each(sub ($item) { ... })`
> as a method. Perhaps it has an array-like structure internally and allows
> you to iterate over items.
>
> As code builds up over time, such as `$obj->each(sub ($item) { say "Item:
> $item" })`, everyone's happy, there may be an obscure note in the
> documentation about putting in `, @` in the signature somewhere but things
> work fine without that, and half the developers using this don't understand
> what that meant (or didn't read it in the first place). This module starts
> to occupy a key position in the CPAN dependency stream, and everyone loves
> it...
>
> ... except for one unsatisfied customer who points out that this isn't
> useful enough in the current state: it should also pass the index. They
> raise a patch supporting their very-important usecase: `$obj->each(sub
> ($item, $index) { say "Item $item had index $index" })`.
>
> Maybe all the tests already have the `,@` as advised, maybe tests are
> updated to fix everything that broke... either way, that new version is
> released. Anyone who upgrades and *didn't* put in that `,@` now faces
> breakage: they have to go and update all their code to cope with it. You
> might say that they should have read the documentation, that upgrades need
> careful testing, or that they are just bad programmers who should have
> written perfect code in the first place. Either way, there's unhappiness
> and perhaps a tendency to look again at other languages like Javascript
> where this type of extensible callback API change is comparatively easier.
>
> If the fallout is too bad, the change may be reverted, perhaps a new
> method is added - `->each_with_index`. Huffman proponents declare that the
> library is now unusable and proceed to fork it or rewrite from scratch,
> much acrimony ensues, and in the middle of all that someone points out that
> we clearly needed a *third* parameter, a reference to the underlying
> arrayref so that you can look at items either side of the current one. This
> is seen as a welcome distraction and immediately the userbase fragments
> into 3 different camps: the document said ,@ so get used to it,
> `->each_with_index_and_arrayref`, and `->each(sub ($item, $index,
> $arrayref) { ... }, WITH_INDEX | WITH_ARRAYREF)`. Perhaps someone else
> proposes a `use Library ":v2"` to get the new behaviour, or puts in a
> try-catch-until-we-stop-seeing-signature-failure workaround. The resulting
> discussion becomes heated and is posted on various social media outlets,
> with wise and informed commentary on all sides, but for some strange reason
> the development on that original library is slowing down a bit, and perhaps
> it's just a tiny bit less popular than it once was.
>
> Anyway, you get the point. As a developer, I'm left thinking "if I can do
> various boundary-breaking things like inspecting the caller's lexicals via
> PadWalker, then _not_ being able to check what a coderef expects to be
> called with seems like an odd omission".
>

Yeah im with you, a seemingly reasonable policy decision permutes into a
bundle of secondary problems that it need not.


>
>
>> FWIW, I used to have *very* strongly *aligned* views with you about the
>> warnings coming from sprintf Avar introduced ages back. At first I loathed
>> that sprintf would warn if I passed in more arguments than it used. But
>> over time I found that it actually saved me from embarrassment more often
>> than it hindered my projects. Not exactly your point, but not exactly a
>> different one either.
>>
>
> Indeed so - and that's just a warning. This is a step further - it's a
> runtime error. And again, in the sprintf case you controlled the format
> string *and* the parameters, so it's understandable that you'd need to take
> responsibility for making those consistent.
>

Actually the case I had in mind was where an external caller provides a
format to a sprintf which is provided a set of arguments in a standard way.
So it's much closer to your use case than this. Probably less problematic
to work around however as you say its a warning not a fatal exception.



>
>
>>
>>> There are no tools or mechanisms offered to enforce this... so people
>>> will leave it out, and things will work. They will work until a new version
>>> of that library is released which now passes additional optional
>>> parameters: then they will break.
>>>
>>
>> Isn't this a question for the implementer of the callback mechanism?
>>
>>
>>> Providing a way for the library code to deal with that - an explicit
>>> opt-out for arity checking, *not* a change to the defaults across the rest
>>> of Perl - seems like a basic courtesy to developers. If we are outright
>>> rejecting that, I find that disappointing. More so if the given reason is
>>> "we've been discussing signatures for 7 years already", rather than any
>>> specific technical impediment.
>>>
>>
>> But doesn't the @ mechanism do that? Eg, if an author of an API that
>> takes callbacks as a parameter should document that the callback API might
>> change over time and thus all subs SHOULD have a @ in it?
>>
>
> The @ mechanism does not do that, though:
>
> > There are no tools or mechanisms offered to *enforce* this... so *people
> will leave it out, and things will work*. They will work until a new
> version of that library is released which now passes additional optional
> parameters: then they will break.
>
> Making it easier to be backwards-compatible would benefit both the library
> developers *and* the users.
>

Yeah I get you.


>
> Is there some introspective approach that would help here? Eg, if I was a
>> callback author and I could interrogate that the signature did not have a @
>> as its last clause then I could warn if someone supplied me a callback that
>> didnt satisfy.
>>
>
> Yes. From the original email in that thread:
>
> > If we had some sort of introspection we could do some kind of
> > `bounded_invoke( $callback, @args )` which would slice @args to no more
> > than the signature says it wants.
>
> Sadly the conversation drifted away from that original suggestion - I
> didn't see any technical discussion on it.
>

Hrm, it seems like that would be solution so rjbs could have what he thinks
is sensibel, and you could have what you think is sensible. If one could
introspect on the arity of a sub then to some level this could be avoided.
Eg, when the new framework accepts a sub designed for the old 1 argument
API, it could internally ensure that sub was called with 1 argument, and
call the subs intended for more than 1 argument as required. If you can't
introspect on the arity then you can not harden yourself for this. Eg, a
forward looking dev could say "my callback framework insists the callback
accepts a slurpy argument list" and warn or die when someone passes in one
that doesnt have artiy==-1 (-1 for infinity), and one less prescient could
do something else internally, something like:

my $arity=arity($sub_ref);
if ($arity == 1) {
$sub_ref->($key)
}
elsif ($arity < 0 or $arity == 2 ) {
$sub_ref->($key,$index);
}


>
>
>> Anyway, I see the problem you are worried about and agree it has merit.
>> But I'm not sure that the policy approach that rjbs is taking is
>> unreasonable either. Is there a middle ground that would satisfy?
>>
>
> There is, yes - options have been proposed, including:
>
> - introspection, as above
>

This seems reasonable.


> - provide a way to bypass arity checks for a specific call
>

Although I am not sure I can elucidate why I think this would be a bad
idea, I am intuitively sympathetic to those who think it is a bad idea to
be avoided.


>
> If the answer is "no, we are not providing any of those", and that's an
> official PSC decision, then there's not much more to be said. I have not
> seen that stated explicitly (yet!). Would be nice to have a summary in the
> archives explaining _why_ we aren't considering them - performance impact,
> for example.
>

Personally I cant understand why if perl can validate the arity of a sub we
cannot provide a mechanism to determine what that is at a code level.

While in this new PSC world I dont believe my opinion matters much, it at
all, I personally think you have case to go to rjbs and the PSC and say, "I
do not want to change the rules about arity, but I *do* think we should
provide a way to introspect on it", and I think rjbs might be a bit more
amenable to talk about it if he understand it wasnt the "bypass arity
checks" argument rehashed. I dont him that well, and we do disagree from
time to time, but IMO he is usually a pretty reasonable guy.

I am going to CC him on this, he can yell at me if he wants. :-)

Yves

--
perl -Mre=debug -e "/just|another|perl|hacker/"

On Sun, 16 Jan 2022 12:23:53 +0100
demerphq <demerphq@gmail.com> wrote:

> Hrm, it seems like that would be solution so rjbs could have what he
> thinks is sensibel, and you could have what you think is sensible.
> If one could introspect on the arity of a sub then to some level this
> could be avoided. Eg, when the new framework accepts a sub designed
> for the old 1 argument API, it could internally ensure that sub was
> called with 1 argument, and call the subs intended for more than 1
> argument as required. If you can't introspect on the arity then you
> can not harden yourself for this. Eg, a forward looking dev could say
> "my callback framework insists the callback accepts a slurpy argument
> list" and warn or die when someone passes in one that doesnt have
> artiy==-1 (-1 for infinity), and one less prescient could do
> something else internally, something like:
>
> my $arity=arity($sub_ref);
> if ($arity == 1) {
> $sub_ref->($key)
> }
> elsif ($arity < 0 or $arity == 2 ) {
> $sub_ref->($key,$index);
> }

...

> Personally I cant understand why if perl can validate the arity of a
> sub we cannot provide a mechanism to determine what that is at a code
> level.

> While in this new PSC world I dont believe my opinion matters much,
> it at all, I personally think you have case to go to rjbs and the PSC
> and say, "I do not want to change the rules about arity, but I *do*
> think we should provide a way to introspect on it", and I think rjbs
> might be a bit more amenable to talk about it if he understand it
> wasnt the "bypass arity checks" argument rehashed. I dont him that
> well, and we do disagree from time to time, but IMO he is usually a
> pretty reasonable guy.

((With my personal hat and not my PSC hat))

I've also been wanting (needing) an arity-inspection mechanism,
currently required to make Syntax::Keyword::MultiSub work.

Also in the presence of such a mechanism it would be possible to write
a little call-wrapper to call a sub with no more arguments than it was
expecting, even if I accept that it will *SILENTLY* discard extra ones:

sub call_no_arity ($code, @args) {
my $nmax = max_arity_of $code;
splice @args, $nmax, -1, () if defined $nmax;
return $code->(@args);
}

...

sub each ($code) {
...
call_no_arity $code, $item, $index, $context;
}

Could be used with any of the following, without error:

->each(sub ($item) { ... });
->each(sub ($item, $idx) { ... });
->each(sub ($item, $idx, $c) { ... });


That said, my ideal wish would be for arity-checking to appear as a
default-fatal exception from the caller's perspective; so the caller
could do something like:

sub each ($code) {
...
{
no fatal 'signatures::max_arity';
$code->($item, $index, $context);
}
}

To, within that small lexical scope, disarm the fatality of maximum
arity checking.

--
Paul "LeoNerd" Evans

leonerd@leonerd.org.uk | https://metacpan.org/author/PEVANS
http://www.leonerd.org.uk/ | https://www.tindie.com/stores/leonerd/

I've put some concrete examples of arity-checking API functionality here:

https://metacpan.org/pod/Acme::Signature::Arity

in case anyone else is in a similar library-author situation, needs a
workaround, and we end up deciding "no, not needed as a core/builtin
feature".

On Sun, Jan 16, 2022, at 6:23 AM, demerphq wrote:
> Personally I cant understand why if perl can validate the arity of a sub we cannot provide a mechanism to determine what that is at a code level.

I don't mind the existence of a "Sub::Util::arity_of" or something. (I'm not sure what return would be best, since it needs to handle:
* undeclared
* fixed number
* n or more
* n or more, but even/odd)
What I don't want is to change the behavior we have out of the box, or to further delay shipping this based on something that can be added in the future.

> I am going to CC him on this, he can yell at me if he wants. :-)

I mean, I read p5p, which got all the previous messages, too. ???? I will only yell if you spill my beer.

--
rjbs

2022-1-17 4:15 Ricardo Signes <rjbs@cpan.org> wrote:

> On Sun, Jan 16, 2022, at 6:23 AM, demerphq wrote:
>
> Personally I cant understand why if perl can validate the arity of a sub
> we cannot provide a mechanism to determine what that is at a code level.
>
>
> I don't mind the existence of a "Sub::Util::arity_of" or something. (I'm
> not sure what return would be best, since it needs to handle:
>
> - undeclared
> - fixed number
> - n or more
> - n or more, but even/odd)
>
> What I don't want is to change the behavior we have out of the box, or to
> further delay shipping this based on something that can be added in the
> future.
>
> I am going to CC him on this, he can yell at me if he wants. :-)
>
>
> I mean, I read p5p, which got all the previous messages, too. ???? I will
> only yell if you spill my beer.
>
> --
> rjbs
>

I have the same thinking as Tom about possible accidents.

My proposal makes arity checking optional.

use feature 'no_arity_check';

- Don't break anything, including experimental ones.
- Users can bypass arity checks. This is convenient because users can
rewrite the existing codes using signatures without changing any behavior.
- If we want to implement multi-sub in the future, use subroutines that the
arity checking is enabled.
- Subroutine arguments are used by everyone, so it should be selectable.
Coexistence of those who like arity checking and those who do not.
- There is no need to postpone the release of the current signatures.
- This way is very easy. Remove only ops of "... if @_> ..." "... if @ <
..." and "... if @_ % 2 != 0" when enabling "no_arity_check"
- A little faster than with arity checking.

On Sun, Jan 09, 2022 at 11:33:29AM -0500, Ricardo Signes wrote:
> On Sun, Jan 9, 2022, at 7:37 AM, Neil Bowers wrote:
> > Our overriding desire is to get signatures "out there", but what's the right next step? There are at least 4 options:
> > 1. Remove the experimental sticker off signatures and release that way in 5.36 (so you'd still have to `use feature` or `use v5.36`), but no other changes.
> > 2. As for 1, but also add a runtime warning if you touch @_ inside a signatured-sub.
> > 3. Like 2, but touching @_ is fatal.
> > 4. Inside signatured-subs @_ becomes non-special.
>
> I am strongly in favor of #1.
[snip] stuff

There seems to be a lot of confusion and misconceptions present in this
thread.

My preferred solution consists of doing all of the following:

1) leaving @_ untouched when calling a signatured-sub (i.e. it is still
the @_ of the caller).

This will have a significant performance boost, especially when calling
small stub functions like accessors. At the moment perl has to do the
equivalent of

my @args = (...);
local *_ = \@args;

for every sub call (even ones without args), setting up, aliasing,
unaliasing etc. It will also make assigning args to parameters faster.

Eventually over time as signatures become the norm, @_ will be remembered
just as some old weird thing perl used to do (like we occasionally
encounter perl4-isms).

2) To detect signature code which relies on @_ from breaking with this
change, we also make the presence of @_, $_[...] etc within the *lexical
scope* of a signatured sub a *compile* time warning. This is relatively
easy to do (at least to catch most common cases) and has exactly *zero*
run-time performance issues.

I think a warning (rather than being fatal) is best - it's compile time so
will catch most things during development, but still allows legitimate
uses of @_, e.g.:

sub foo($x) {
....;
no warnings 'snails';
local @_ = (1,2,3);
goto &bar; # tail call optimisation
}


3) to allow introspection and other processing of the args, introduce a
new mechanism - either my 'query parameters' proposal or something else.

(3) can be done later as long as we're happy with the argument that "for
now, people should stick with the old sub syntax if they want access to
@_".


I would be strongly opposed to any final decision which didn't include
(1) and (2).

I would be uncomfortable with signatures becoming non-experimental until
both (1) and (2) were done.

I could live with (2) being done before (1). So I might accept:

5.36: - @_ gives compile-time warning of "don't use @_, subject to change".

5.38: - @_ gives compile-time warning of "don't use @_, no longer does what
you think it does";
- @_ stops being populated (i.e. it becomes still just caller's @_).
- signatures stop being experimental

5.40: - query parameters (or similar) added.

5.40+ other signature features like named signatures added piecemeal.


Note that if (2) happens before (1), I think the warning shouldn't be a
deprecation warning - we don't want the category of the warning changing
after we implement (1). (Which, thinking about it, makes me more keen that
(1) and (2) happen together).





--
Decaffeinated coffee is like dehydrated water

> On Jan 17, 2022, at 07:31, Dave Mitchell <davem@iabyn.com> wrote:
>
> On Sun, Jan 09, 2022 at 11:33:29AM -0500, Ricardo Signes wrote:
>> On Sun, Jan 9, 2022, at 7:37 AM, Neil Bowers wrote:
>>> Our overriding desire is to get signatures "out there", but what's the right next step? There are at least 4 options:
>>> 1. Remove the experimental sticker off signatures and release that way in 5.36 (so you'd still have to `use feature` or `use v5.36`), but no other changes.
>>> 2. As for 1, but also add a runtime warning if you touch @_ inside a signatured-sub.
>>> 3. Like 2, but touching @_ is fatal.
>>> 4. Inside signatured-subs @_ becomes non-special.
>>
>> I am strongly in favor of #1.
> [snip] stuff
>
> There seems to be a lot of confusion and misconceptions present in this
> thread.
>
> My preferred solution consists of doing all of the following:
>
> 1) leaving @_ untouched when calling a signatured-sub (i.e. it is still
> the @_ of the caller).
>
> This will have a significant performance boost, especially when calling
> small stub functions like accessors. At the moment perl has to do the
> equivalent of
>
> my @args = (...);
> local *_ = \@args;
>
> for every sub call (even ones without args), setting up, aliasing,
> unaliasing etc. It will also make assigning args to parameters faster.
>
> Eventually over time as signatures become the norm, @_ will be remembered
> just as some old weird thing perl used to do (like we occasionally
> encounter perl4-isms).

The points heretofore raised in response to this seem to be:

1) There is no viable branch currently that implements leaving @_ untouched.

2) The performance gain has yet to be shown.

I’d love to help in either of these regards, but I lack the knowledge to assist with #1, and #2 can’t happen without the former.

-F

On Mon, 17 Jan 2022 09:55:34 -0500
Felipe Gasper <felipe@felipegasper.com> wrote:

> 1) There is no viable branch currently that implements leaving @_
> untouched.
>
> 2) The performance gain has yet to be shown.
>
> I’d love to help in either of these regards, but I lack the knowledge
> to assist with #1, and #2 can’t happen without the former.

Working on it... Hope to have results later today.

--
Paul "LeoNerd" Evans

leonerd@leonerd.org.uk | https://metacpan.org/author/PEVANS
http://www.leonerd.org.uk/ | https://www.tindie.com/stores/leonerd/

Hi Dave,

This is written as Neil, not a PSC position.

How about the following tweak to your proposed timeline?

1. 5.36: compile-time warning on use of @_ in signatured-subs, AND make signatures non-experimental
2. 5.38: @_ stops being populated; change wording of compile-time warning

This would give us non-experimental signatures a year earlier, and we’d have been warning about the impending change for a year as well.

I realise this slightly breaks the spirit of perlpolicy, but this isn’t proposing a change to signatures themselves. Perlpolicy also says "we'll always do our best to smooth the transition path for users of experimental features".

If we want people to continue to use Perl, and even new people to start, we need to move things forward, and signatures is probably one of the best things we can deliver in 5.36. And if we don’t get around to non-population-of-@_ in 5.38, well at least we made signatures non-experimental in 5.36.

You might argue, "Neil, you don’t know what you’re talking about", and to a close approximation you’d be right, but I feel like I need to lobby on the behalf of regular Perl developers.

Cheers,
Neil

??, 17 ???. 2022 ?. ? 15:32, Dave Mitchell <davem@iabyn.com>:
>
> 1) leaving @_ untouched when calling a signatured-sub (i.e. it is still
> the @_ of the caller).
>
> This will have a significant performance boost

While it does, as-is it breaks stack traces. Without stack traces,
debugging becomes a pain. Unless any proposal for (1) also includes
solution for stack traces, it makes programmer's life worse, not
better.

Best regards,
Sergey Aleynikov

On Mon, Jan 17, 2022, at 7:31 AM, Dave Mitchell wrote:
> On Sun, Jan 09, 2022 at 11:33:29AM -0500, Ricardo Signes wrote:
> > On Sun, Jan 9, 2022, at 7:37 AM, Neil Bowers wrote:
> > > Our overriding desire is to get signatures "out there", but what's the right next step? There are at least 4 options:
> > > 1. Remove the experimental sticker off signatures and release that way in 5.36 (so you'd still have to `use feature` or `use v5.36`), but no other changes.
> > > 2. As for 1, but also add a runtime warning if you touch @_ inside a signatured-sub.
> > > 3. Like 2, but touching @_ is fatal.
> > > 4. Inside signatured-subs @_ becomes non-special.
> >
> > I am strongly in favor of #1.
> [snip] stuff
>
> There seems to be a lot of confusion and misconceptions present in this thread.

I am always glad to see either or both of those things cleared up!

> 1) leaving @_ untouched when calling a signatured-sub (i.e. it is still the @_ of the caller).
> [...]
> 2) To detect signature code which relies on @_ from breaking with this change, we also make the presence of @_, $_[...] etc within the *lexical scope* of a signatured sub a *compile* time warning. This is relatively easy to do (at least to catch most common cases) and has exactly *zero* run-time performance issues.
> [...]
> 3) to allow introspection and other processing of the args, introduce a new mechanism - either my 'query parameters' proposal or something else.

So far, I think I'm with you. (I guess that the idea here is that behavior that can't be detected at compile time won't issue a warning at runtime, right?)

> I would be strongly opposed to any final decision which didn't include (1) and (2).
>
> I would be uncomfortable with signatures becoming non-experimental until both (1) and (2) were done.

I have mixed feelings about both 1 and 2. I'd like to see the actual performance change. That said, I acknowledge generally what we said we'd be doing.

But we've said that for years, now, and it hasn't happened and signatures have remained an experiment. Without a change in hand, showing that we have the changes prepared and know they'll work, it feels like "we kept this in experimental for years, planning to do this. We didn't do it, but this year is different." Is this year different?

There are at least three possibilities ahead of us:

1. we ship signatures as non-experimental this year
2. we ship signatures as non-experimental next year, and it's faster and a little different
3. we skip #1 in favor of #2 but do neither

What I want most of all is to avoid #3.

Our policy on experiments is:
> Experimental features will only have their experimental status revoked when they no longer have any design-changing bugs open against them and when they have remained unchanged in behavior for the entire length of a development cycle.

So if we want to change how @_ works and then make the feature stable, we're obliged to get that change into 5.even.0 so we can release it as non-experimental in the next stable .0 release. Could we bend the rules? Yes, but all the better if we can get this change into 5.36 if we want stable in 5.38.

In fact, this is what I proposed (with a difference in the phase in which warning occurs) in November: http://markmail.org/message/23qppsq4qjmegq6m

…but it's January now, which makes me start to worry that we're not 1.5yr from stable signatures, but 2.5yr, or more worryingly, 1.5 plus n. This is also why I've said it seemed practical to look at shipping stable now with a warning that one aspect could not be relied upon. If no actual changes to the @_ handling were produced, we'd be in a better place. If they were, we'd have given warning.

When 5.36 comes out, it will be eight years since signature were added to the language. I just want to feel like I can tell people to use them without feeling like a hypocrite, because I also tell people we need to act like experiments are really unreliable.

--
rjbs

On 1/17/22 12:39 PM, Neil Bowers wrote:
> Hi Dave,
>
> This is written as Neil, not a PSC position.
>
> How about the following tweak to your proposed timeline?
>
> 1. 5.36: compile-time warning on use of @_ in signatured-subs, AND make
> signatures non-experimental
> 2. 5.38: @_ stops being populated; change wording of compile-time warning

Neil,

Just to be clear: Does this mean "@_ stops being populated in
subroutines using signatures? (As distinct from *all* subroutines).
>

On Mon, Jan 17, 2022, at 12:48 PM, James E Keenan wrote:
> Just to be clear: Does this mean "@_ stops being populated in subroutines using signatures? (As distinct from *all* subroutines)

(I am not Neil but: yes. We can't stop populating @_ in other subroutines without breaking absolutely enormous amounts of code.)

--
rjbs

On Mon, Jan 17, 2022 at 08:41:24PM +0300, Sergey Aleynikov wrote:
> ??, 17 ???. 2022 ?. ? 15:32, Dave Mitchell <davem@iabyn.com>:
> >
> > 1) leaving @_ untouched when calling a signatured-sub (i.e. it is still
> > the @_ of the caller).
> >
> > This will have a significant performance boost
>
> While it does, as-is it breaks stack traces. Without stack traces,
> debugging becomes a pain. Unless any proposal for (1) also includes
> solution for stack traces, it makes programmer's life worse, not
> better.

It doesn't necessarily break stack backtraces; that is an entirely
separate choice. It boils down to (on non-@_-setting) sub calls, whether
we leave the original args on the stack or not. At the moment the args are
popped off the stack but stored in @_ instead.

The current behaviour means that for deeply nested sub calls with many
arguments, the stack doesn't tend to grow much, but lots of subs get their
private @_'s AvARRAY array of pointers left grown.


As an extreme example, this code

sub recurse {
return if $_[0]-- < 0;
recurse(@_);
}
recurse(10, 1..1_000_000);

would currently:
expand the stack by 1M pointer sizes (so 8Mb)
create 10 permanent (but private) @_'s (one for each depth of
recursion) each with a 1M pointer size buffer that is never freed.

When removing @_, we could either:

1) leave the arguments on the stack:

- this makes @DB::Args continue to work, and changes the memory usage to:
expand the stack by 10M pointer sizes (80Mb)
(there are no @_'s using memory)

2) pop the stack after processing args:

- this breaks @DB::Args (and thus stack backtraces), and changes the
memory usage to:

expand the stack by 1M pointer sizes (so 8Mb)
(there are no @_'s using memory)

So we have a complete choice. Leaving the args on the stack (and
thus not breaking stack backtraces) would result in a larger maximum stack
growth (which can then be reused for different sub calls) compared with
now where we have smaller stack growth, but leave lots of @_ buffers
lying around which can only be reused if *those* subs are called again.

--
This is a great day for France!
-- Nixon at Charles De Gaulle's funeral

On Sun, Jan 09, 2022 at 12:37:32PM +0000, Neil Bowers wrote:
> The topic we've discussed a lot recently is what to do with @_ in
> signatured-subs:

A few more facts I've just remembered to chuck on the fire.

1) Another advantage of not populating @_ is that it removes a whole bunch
of edge cases concerning what happens if @_ is modified during arg
processing. In particular things become more complicated if we ever
implement named args: this is where the caller does foo(y =>2, x =>1) to
call sub foo(:$x, :$y) {} - here the processing of default args etc
doesn't follow a simple L-R ordering and it's more likely that @_ could
get modified mid-processing.

As well as confusing or undefined behaviour, it requires more arg
processing overhead, because for example the size of @_ needs to be
rechecked after each arg, in case it has shrunk or grown in the meantime.

2) Some miscellaneous behaviours we may want to pin down before making
signatures non-experimental.

a) we should more clearly document the ordering of side effects (such as
calling FETCH() on tied values), or more to the point, document what
things are officially undefined, giving us more flexibility to change
things later when optimising.

b) banning of goto's into and out of default blocks:

we currently warn when entering a block containing a default expression:

Use of "goto" to jump into a construct is deprecated at ...

Perhaps that should be upgraded to a fatal error.
Jumping into the main block of a sub from a default expression doesn't
currently warn - maybe this should become a fatal error?

sub foo2 ($x = do { goto MAIN }, $y = ...)
{
MAIN: print "in main body\n";
}

c) decide where whitespace is is allowed or forbidden: this is currently
legal:

sub foo ($ x) {}

I think it would be nice to make it a compile-time error.

d) should duplicate parameters be a compile error? (at the moment it just
warns):

sub f ($a,$a) { ... }

"my" variable $a masks earlier declaration in same scope


Having now remembered a bunch of stuff in addition to @_ which might effect
experimentality, I'm leaning more strongly towards:

5.36 - make use of @_ a compile-time warning;
- fix up the other stuff mentioned above if we have time.

5.38 -stop populating @_
- make signatures non-experimental if we feel confident, else defer
for 1 more release.

5.40+ add new signature features.




--
Modern art:
"That's easy, I could have done that!"
"Ah, but you didn't!"