On Sun, 16 Jan 2022 at 11:30, Tom Molesworth <tom@deriv.com> wrote:
> On Sun, 16 Jan 2022 at 16:44, demerphq <demerphq@gmail.com> wrote:
>
>> On Sun, 16 Jan 2022 at 09:12, Tom Molesworth via perl5-porters <
>> perl5-porters@perl.org> wrote:
>>
>>> To recap the original problem: someone who learns Perl-with-signatures
>>> who wants to pass a callback to a CPAN module will be writing fragile code.
>>> This is something that library authors have no control over. The suggestion
>>> is "library documentation should tell developers that they must always
>>> include , @ in callbacks".
>>>
>>
>> If those callbacks do not share a common signature then yes, that is what
>> they must do. I dont understand why you assume that any case where
>> callbacks are used will have inconsistent signatures. It seems to me that
>> it is equally likely most callbacks would be the other way around. Eg,
>> favoring one case or another is an arbitrary decision that comes down to
>> some external factoid about the situation.
>>
>
> hmm - I think the external factoid here would be "was written for the
> earlier version, before the new option was added"? There's momentary
> convenience in being able to leave out the parameters without the `,@`, but
> that's not really the focus. At the time the original code was written,
> signatures would presumably be consistent - there's only one case, the
> documented API with however many callback parameters were passed at the
> time, so there's no decision or favouring involved...
>
Ok, I see where you are coming from, i got it in my head before, but your
clarication moved that to the heart, if you know what I mean. I do see that
this would be quite irritating a major blocker to rolling our a new API
version with enhancements over a previous version. Old code that used to be
"just fine" would break because it doesn't accept the new parameter.
>
> From your questions here, I get the impression that perhaps you're still
> not clear on exactly what type of situation is problematic?
>
FWIW, I think I did, but the clarification is still helpful, it drives home
the point well.
> Just in case, here's a trivial example:
>
> Let's say we have a module X that provides `->each(sub ($item) { ... })`
> as a method. Perhaps it has an array-like structure internally and allows
> you to iterate over items.
>
> As code builds up over time, such as `$obj->each(sub ($item) { say "Item:
> $item" })`, everyone's happy, there may be an obscure note in the
> documentation about putting in `, @` in the signature somewhere but things
> work fine without that, and half the developers using this don't understand
> what that meant (or didn't read it in the first place). This module starts
> to occupy a key position in the CPAN dependency stream, and everyone loves
> it...
>
> ... except for one unsatisfied customer who points out that this isn't
> useful enough in the current state: it should also pass the index. They
> raise a patch supporting their very-important usecase: `$obj->each(sub
> ($item, $index) { say "Item $item had index $index" })`.
>
> Maybe all the tests already have the `,@` as advised, maybe tests are
> updated to fix everything that broke... either way, that new version is
> released. Anyone who upgrades and *didn't* put in that `,@` now faces
> breakage: they have to go and update all their code to cope with it. You
> might say that they should have read the documentation, that upgrades need
> careful testing, or that they are just bad programmers who should have
> written perfect code in the first place. Either way, there's unhappiness
> and perhaps a tendency to look again at other languages like Javascript
> where this type of extensible callback API change is comparatively easier.
>
> If the fallout is too bad, the change may be reverted, perhaps a new
> method is added - `->each_with_index`. Huffman proponents declare that the
> library is now unusable and proceed to fork it or rewrite from scratch,
> much acrimony ensues, and in the middle of all that someone points out that
> we clearly needed a *third* parameter, a reference to the underlying
> arrayref so that you can look at items either side of the current one. This
> is seen as a welcome distraction and immediately the userbase fragments
> into 3 different camps: the document said ,@ so get used to it,
> `->each_with_index_and_arrayref`, and `->each(sub ($item, $index,
> $arrayref) { ... }, WITH_INDEX | WITH_ARRAYREF)`. Perhaps someone else
> proposes a `use Library ":v2"` to get the new behaviour, or puts in a
> try-catch-until-we-stop-seeing-signature-failure workaround. The resulting
> discussion becomes heated and is posted on various social media outlets,
> with wise and informed commentary on all sides, but for some strange reason
> the development on that original library is slowing down a bit, and perhaps
> it's just a tiny bit less popular than it once was.
>
> Anyway, you get the point. As a developer, I'm left thinking "if I can do
> various boundary-breaking things like inspecting the caller's lexicals via
> PadWalker, then _not_ being able to check what a coderef expects to be
> called with seems like an odd omission".
>
Yeah im with you, a seemingly reasonable policy decision permutes into a
bundle of secondary problems that it need not.
>
>
>> FWIW, I used to have *very* strongly *aligned* views with you about the
>> warnings coming from sprintf Avar introduced ages back. At first I loathed
>> that sprintf would warn if I passed in more arguments than it used. But
>> over time I found that it actually saved me from embarrassment more often
>> than it hindered my projects. Not exactly your point, but not exactly a
>> different one either.
>>
>
> Indeed so - and that's just a warning. This is a step further - it's a
> runtime error. And again, in the sprintf case you controlled the format
> string *and* the parameters, so it's understandable that you'd need to take
> responsibility for making those consistent.
>
Actually the case I had in mind was where an external caller provides a
format to a sprintf which is provided a set of arguments in a standard way.
So it's much closer to your use case than this. Probably less problematic
to work around however as you say its a warning not a fatal exception.
>
>
>>
>>> There are no tools or mechanisms offered to enforce this... so people
>>> will leave it out, and things will work. They will work until a new version
>>> of that library is released which now passes additional optional
>>> parameters: then they will break.
>>>
>>
>> Isn't this a question for the implementer of the callback mechanism?
>>
>>
>>> Providing a way for the library code to deal with that - an explicit
>>> opt-out for arity checking, *not* a change to the defaults across the rest
>>> of Perl - seems like a basic courtesy to developers. If we are outright
>>> rejecting that, I find that disappointing. More so if the given reason is
>>> "we've been discussing signatures for 7 years already", rather than any
>>> specific technical impediment.
>>>
>>
>> But doesn't the @ mechanism do that? Eg, if an author of an API that
>> takes callbacks as a parameter should document that the callback API might
>> change over time and thus all subs SHOULD have a @ in it?
>>
>
> The @ mechanism does not do that, though:
>
> > There are no tools or mechanisms offered to *enforce* this... so *people
> will leave it out, and things will work*. They will work until a new
> version of that library is released which now passes additional optional
> parameters: then they will break.
>
> Making it easier to be backwards-compatible would benefit both the library
> developers *and* the users.
>
Yeah I get you.
>
> Is there some introspective approach that would help here? Eg, if I was a
>> callback author and I could interrogate that the signature did not have a @
>> as its last clause then I could warn if someone supplied me a callback that
>> didnt satisfy.
>>
>
> Yes. From the original email in that thread:
>
> > If we had some sort of introspection we could do some kind of
> > `bounded_invoke( $callback, @args )` which would slice @args to no more
> > than the signature says it wants.
>
> Sadly the conversation drifted away from that original suggestion - I
> didn't see any technical discussion on it.
>
Hrm, it seems like that would be solution so rjbs could have what he thinks
is sensibel, and you could have what you think is sensible. If one could
introspect on the arity of a sub then to some level this could be avoided.
Eg, when the new framework accepts a sub designed for the old 1 argument
API, it could internally ensure that sub was called with 1 argument, and
call the subs intended for more than 1 argument as required. If you can't
introspect on the arity then you can not harden yourself for this. Eg, a
forward looking dev could say "my callback framework insists the callback
accepts a slurpy argument list" and warn or die when someone passes in one
that doesnt have artiy==-1 (-1 for infinity), and one less prescient could
do something else internally, something like:
my $arity=arity($sub_ref);
if ($arity == 1) {
$sub_ref->($key)
}
elsif ($arity < 0 or $arity == 2 ) {
$sub_ref->($key,$index);
}
>
>
>> Anyway, I see the problem you are worried about and agree it has merit.
>> But I'm not sure that the policy approach that rjbs is taking is
>> unreasonable either. Is there a middle ground that would satisfy?
>>
>
> There is, yes - options have been proposed, including:
>
> - introspection, as above
>
This seems reasonable.
> - provide a way to bypass arity checks for a specific call
>
Although I am not sure I can elucidate why I think this would be a bad
idea, I am intuitively sympathetic to those who think it is a bad idea to
be avoided.
>
> If the answer is "no, we are not providing any of those", and that's an
> official PSC decision, then there's not much more to be said. I have not
> seen that stated explicitly (yet!). Would be nice to have a summary in the
> archives explaining _why_ we aren't considering them - performance impact,
> for example.
>
Personally I cant understand why if perl can validate the arity of a sub we
cannot provide a mechanism to determine what that is at a code level.
While in this new PSC world I dont believe my opinion matters much, it at
all, I personally think you have case to go to rjbs and the PSC and say, "I
do not want to change the rules about arity, but I *do* think we should
provide a way to introspect on it", and I think rjbs might be a bit more
amenable to talk about it if he understand it wasnt the "bypass arity
checks" argument rehashed. I dont him that well, and we do disagree from
time to time, but IMO he is usually a pretty reasonable guy.
I am going to CC him on this, he can yell at me if he wants. :-)
Yves
--
perl -Mre=debug -e "/just|another|perl|hacker/"
> On Sun, 16 Jan 2022 at 16:44, demerphq <demerphq@gmail.com> wrote:
>
>> On Sun, 16 Jan 2022 at 09:12, Tom Molesworth via perl5-porters <
>> perl5-porters@perl.org> wrote:
>>
>>> To recap the original problem: someone who learns Perl-with-signatures
>>> who wants to pass a callback to a CPAN module will be writing fragile code.
>>> This is something that library authors have no control over. The suggestion
>>> is "library documentation should tell developers that they must always
>>> include , @ in callbacks".
>>>
>>
>> If those callbacks do not share a common signature then yes, that is what
>> they must do. I dont understand why you assume that any case where
>> callbacks are used will have inconsistent signatures. It seems to me that
>> it is equally likely most callbacks would be the other way around. Eg,
>> favoring one case or another is an arbitrary decision that comes down to
>> some external factoid about the situation.
>>
>
> hmm - I think the external factoid here would be "was written for the
> earlier version, before the new option was added"? There's momentary
> convenience in being able to leave out the parameters without the `,@`, but
> that's not really the focus. At the time the original code was written,
> signatures would presumably be consistent - there's only one case, the
> documented API with however many callback parameters were passed at the
> time, so there's no decision or favouring involved...
>
Ok, I see where you are coming from, i got it in my head before, but your
clarication moved that to the heart, if you know what I mean. I do see that
this would be quite irritating a major blocker to rolling our a new API
version with enhancements over a previous version. Old code that used to be
"just fine" would break because it doesn't accept the new parameter.
>
> From your questions here, I get the impression that perhaps you're still
> not clear on exactly what type of situation is problematic?
>
FWIW, I think I did, but the clarification is still helpful, it drives home
the point well.
> Just in case, here's a trivial example:
>
> Let's say we have a module X that provides `->each(sub ($item) { ... })`
> as a method. Perhaps it has an array-like structure internally and allows
> you to iterate over items.
>
> As code builds up over time, such as `$obj->each(sub ($item) { say "Item:
> $item" })`, everyone's happy, there may be an obscure note in the
> documentation about putting in `, @` in the signature somewhere but things
> work fine without that, and half the developers using this don't understand
> what that meant (or didn't read it in the first place). This module starts
> to occupy a key position in the CPAN dependency stream, and everyone loves
> it...
>
> ... except for one unsatisfied customer who points out that this isn't
> useful enough in the current state: it should also pass the index. They
> raise a patch supporting their very-important usecase: `$obj->each(sub
> ($item, $index) { say "Item $item had index $index" })`.
>
> Maybe all the tests already have the `,@` as advised, maybe tests are
> updated to fix everything that broke... either way, that new version is
> released. Anyone who upgrades and *didn't* put in that `,@` now faces
> breakage: they have to go and update all their code to cope with it. You
> might say that they should have read the documentation, that upgrades need
> careful testing, or that they are just bad programmers who should have
> written perfect code in the first place. Either way, there's unhappiness
> and perhaps a tendency to look again at other languages like Javascript
> where this type of extensible callback API change is comparatively easier.
>
> If the fallout is too bad, the change may be reverted, perhaps a new
> method is added - `->each_with_index`. Huffman proponents declare that the
> library is now unusable and proceed to fork it or rewrite from scratch,
> much acrimony ensues, and in the middle of all that someone points out that
> we clearly needed a *third* parameter, a reference to the underlying
> arrayref so that you can look at items either side of the current one. This
> is seen as a welcome distraction and immediately the userbase fragments
> into 3 different camps: the document said ,@ so get used to it,
> `->each_with_index_and_arrayref`, and `->each(sub ($item, $index,
> $arrayref) { ... }, WITH_INDEX | WITH_ARRAYREF)`. Perhaps someone else
> proposes a `use Library ":v2"` to get the new behaviour, or puts in a
> try-catch-until-we-stop-seeing-signature-failure workaround. The resulting
> discussion becomes heated and is posted on various social media outlets,
> with wise and informed commentary on all sides, but for some strange reason
> the development on that original library is slowing down a bit, and perhaps
> it's just a tiny bit less popular than it once was.
>
> Anyway, you get the point. As a developer, I'm left thinking "if I can do
> various boundary-breaking things like inspecting the caller's lexicals via
> PadWalker, then _not_ being able to check what a coderef expects to be
> called with seems like an odd omission".
>
Yeah im with you, a seemingly reasonable policy decision permutes into a
bundle of secondary problems that it need not.
>
>
>> FWIW, I used to have *very* strongly *aligned* views with you about the
>> warnings coming from sprintf Avar introduced ages back. At first I loathed
>> that sprintf would warn if I passed in more arguments than it used. But
>> over time I found that it actually saved me from embarrassment more often
>> than it hindered my projects. Not exactly your point, but not exactly a
>> different one either.
>>
>
> Indeed so - and that's just a warning. This is a step further - it's a
> runtime error. And again, in the sprintf case you controlled the format
> string *and* the parameters, so it's understandable that you'd need to take
> responsibility for making those consistent.
>
Actually the case I had in mind was where an external caller provides a
format to a sprintf which is provided a set of arguments in a standard way.
So it's much closer to your use case than this. Probably less problematic
to work around however as you say its a warning not a fatal exception.
>
>
>>
>>> There are no tools or mechanisms offered to enforce this... so people
>>> will leave it out, and things will work. They will work until a new version
>>> of that library is released which now passes additional optional
>>> parameters: then they will break.
>>>
>>
>> Isn't this a question for the implementer of the callback mechanism?
>>
>>
>>> Providing a way for the library code to deal with that - an explicit
>>> opt-out for arity checking, *not* a change to the defaults across the rest
>>> of Perl - seems like a basic courtesy to developers. If we are outright
>>> rejecting that, I find that disappointing. More so if the given reason is
>>> "we've been discussing signatures for 7 years already", rather than any
>>> specific technical impediment.
>>>
>>
>> But doesn't the @ mechanism do that? Eg, if an author of an API that
>> takes callbacks as a parameter should document that the callback API might
>> change over time and thus all subs SHOULD have a @ in it?
>>
>
> The @ mechanism does not do that, though:
>
> > There are no tools or mechanisms offered to *enforce* this... so *people
> will leave it out, and things will work*. They will work until a new
> version of that library is released which now passes additional optional
> parameters: then they will break.
>
> Making it easier to be backwards-compatible would benefit both the library
> developers *and* the users.
>
Yeah I get you.
>
> Is there some introspective approach that would help here? Eg, if I was a
>> callback author and I could interrogate that the signature did not have a @
>> as its last clause then I could warn if someone supplied me a callback that
>> didnt satisfy.
>>
>
> Yes. From the original email in that thread:
>
> > If we had some sort of introspection we could do some kind of
> > `bounded_invoke( $callback, @args )` which would slice @args to no more
> > than the signature says it wants.
>
> Sadly the conversation drifted away from that original suggestion - I
> didn't see any technical discussion on it.
>
Hrm, it seems like that would be solution so rjbs could have what he thinks
is sensibel, and you could have what you think is sensible. If one could
introspect on the arity of a sub then to some level this could be avoided.
Eg, when the new framework accepts a sub designed for the old 1 argument
API, it could internally ensure that sub was called with 1 argument, and
call the subs intended for more than 1 argument as required. If you can't
introspect on the arity then you can not harden yourself for this. Eg, a
forward looking dev could say "my callback framework insists the callback
accepts a slurpy argument list" and warn or die when someone passes in one
that doesnt have artiy==-1 (-1 for infinity), and one less prescient could
do something else internally, something like:
my $arity=arity($sub_ref);
if ($arity == 1) {
$sub_ref->($key)
}
elsif ($arity < 0 or $arity == 2 ) {
$sub_ref->($key,$index);
}
>
>
>> Anyway, I see the problem you are worried about and agree it has merit.
>> But I'm not sure that the policy approach that rjbs is taking is
>> unreasonable either. Is there a middle ground that would satisfy?
>>
>
> There is, yes - options have been proposed, including:
>
> - introspection, as above
>
This seems reasonable.
> - provide a way to bypass arity checks for a specific call
>
Although I am not sure I can elucidate why I think this would be a bad
idea, I am intuitively sympathetic to those who think it is a bad idea to
be avoided.
>
> If the answer is "no, we are not providing any of those", and that's an
> official PSC decision, then there's not much more to be said. I have not
> seen that stated explicitly (yet!). Would be nice to have a summary in the
> archives explaining _why_ we aren't considering them - performance impact,
> for example.
>
Personally I cant understand why if perl can validate the arity of a sub we
cannot provide a mechanism to determine what that is at a code level.
While in this new PSC world I dont believe my opinion matters much, it at
all, I personally think you have case to go to rjbs and the PSC and say, "I
do not want to change the rules about arity, but I *do* think we should
provide a way to introspect on it", and I think rjbs might be a bit more
amenable to talk about it if he understand it wasnt the "bypass arity
checks" argument rehashed. I dont him that well, and we do disagree from
time to time, but IMO he is usually a pretty reasonable guy.
I am going to CC him on this, he can yell at me if he wants. :-)
Yves
--
perl -Mre=debug -e "/just|another|perl|hacker/"