Mailing List Archive

Top-posting
For those who want to know exactly what's going on and don't want to wade through the diff or tests:

$ mkdir lib/{A,B,C}pack.pm

And in each of those, add this:

package Apack;use Bpack;1;
package Bpack;use Cpack;1;
package Cpack;my $i = 0;
while ( my ( $package, $file, $line ) = caller( $i++ ) ) {    push @Cpack::callers, "$file:$line";}1;

And then create run.pl:

#!/usr/bin/env perl
use lib 'lib';use Apack;use Data::Dumper;print Dumper( \@Cpack::callers );

If you have a patched Perl, you get this:

$VAR1 = [.          'lib/Bpack.pm:2',          'lib/Bpack.pm:2',          'lib/Bpack.pm:2',          'lib/Apack.pm:2',          'lib/Apack.pm:2',          'lib/Apack.pm:2',          'run.pl:4',          'run.pl:4',          'run.pl:4'        ];

That's perfectly sane and you can figure out what's going on (though I confess I don't know why some stack frames are repeated).
For an unpatched Perl, you get something like this:

$VAR1 = [.          'lib/Bpack.pm:2',          'lib/Cpack.pm:0',          'lib/Cpack.pm:0',          'lib/Apack.pm:2',          'lib/Cpack.pm:0',          'lib/Cpack.pm:0',          'run.pl:4',          'lib/Cpack.pm:0',          'lib/Cpack.pm:0'        ];

I cannot use that stack trace for debugging.
I have verified the buggy behavior all the way back to 5.8.9.
The patch which reverted the fix notes the following:

commit c0d05305c156c83e4f9f3a207451b3175fbb7f24 (HEAD)Merge: c6b2e294d8 79f75eaa71Author: David Mitchell <davem@iabyn.com>Date:   Mon Apr 27 22:04:23 2020 +0100
    [MERGE] Revert BEGIN { caller() } fixups
    These commits were intended to fix a problem with stack backtraces    reporting wrong file and line numbers in nested use's.
    A side-effect of the commits was to fix the package name returned by    caller() too; but quite a few distributions were relying on the old    behaviour.
    So for now, revert to the old behaviour and re-address after 5.32.0 is    released.
    The reverted commits are:
    v5.31.6-141-gf2f32cd638 avoid identical stack traces    v5.31.9-122-gee428a211d docs: clarify effect of $^H, %^H, ${^WARNING_BITS}    v5.31.9-162-gad89278aa2 fixup to "avoid identical stack traces" - try 2

I don't know which distributions were failing or why, but having a few distributions making it harder for all Perl developers to debug their programs seems backwards.
What can we do to get this fixed?
Best,Ovid-- IT consulting, training, specializing in Perl, databases, and agile development
http://www.allaroundtheworld.fr/. 

Buy my book! - http://bit.ly/beginning_perl






On Thursday, 13 January 2022, 06:05:07 CET, demerphq <demerphq@gmail.com> wrote:





This subject has come up multiple times and is a source of deep frustration for me, and deep confusion for the many non-perl developers who work on perl code at Booking.com.

This was fixed by Dave Mitchell in f2f32cd638746f538da6db804dab6dd54e654f30
https://github.com/Perl/perl5/commit/f2f32cd638746f538da6db804dab6dd54e654f30

This patch was then reverted in merge commit c0d05305c156c83e4f9f3a207451b3175fbb7f24.

Thus stack traces are *still* broken. I am looking at many examples right now and it is impossible to debug what is happening. I cannot post the work examples but the patch referenced includes examples from its tests.

Can we please fix this once and for all? We are running patched perl's at work so that we can fix this (without any sign of any related fallout!), and every time we release a new perl we have to reapply f2f32cd638746f538da6db804dab6dd54e654f30. When the people who build new perls forget then we end up with mass confusion amongst our devs.

Perl has a really bad name amongst the devs we hire, this one of the reasons. Can we please please please FINALLY DO SOMETHING about this? Perl has so many weird ways to establish a dependency and this bug makes it almost impossible to debug non-trivial cases even for an expert class perl dev like myself. 

I beg of you, do not let another perl get released with this bug in place!

Thanks,
Yves



--
perl -Mre=debug -e "/just|another|perl|hacker/"

On Thu, 13 Jan 2022, 18:45 Ovid, <curtis_ovid_poe@yahoo.com> wrote:

> Top-posting
>
> For those who want to know exactly what's going on and don't want to wade
> through the diff or tests:
>
> $ mkdir lib/{A,B,C}pack.pm
>
>
> And in each of those, add this:
>
> package Apack;
> use Bpack;
> 1;
>
> package Bpack;
> use Cpack;
> 1;
>
> package Cpack;
> my $i = 0;
>
> while ( my ( $package, $file, $line ) = caller( $i++ ) ) {
> push @Cpack::callers, "$file:$line";
> }
> 1;
>
>
> And then create run.pl:
>
> #!/usr/bin/env perl
>
> use lib 'lib';
> use Apack;
> use Data::Dumper;
> print Dumper( \@Cpack::callers );
>
>
> If you have a patched Perl, you get this:
>
> $VAR1 = [.
> 'lib/Bpack.pm:2',
> 'lib/Bpack.pm:2',
> 'lib/Bpack.pm:2',
> 'lib/Apack.pm:2',
> 'lib/Apack.pm:2',
> 'lib/Apack.pm:2',
> 'run.pl:4',
> 'run.pl:4',
> 'run.pl:4'
> ];
>
>
> That's perfectly sane and you can figure out what's going on (though I
> confess I don't know why some stack frames are repeated).
>

The repeats come from the expansion of use into BEGIN, require, and the use
itself I believe.


> For an unpatched Perl, you get something like this:
>
> $VAR1 = [.
> 'lib/Bpack.pm:2',
> 'lib/Cpack.pm:0',
> 'lib/Cpack.pm:0',
> 'lib/Apack.pm:2',
> 'lib/Cpack.pm:0',
> 'lib/Cpack.pm:0',
> 'run.pl:4',
> 'lib/Cpack.pm:0',
> 'lib/Cpack.pm:0'
> ];
>
>
> I cannot use that stack trace for debugging.
>
> I have verified the buggy behavior all the way back to 5.8.9.
>
> The patch which reverted the fix notes the following:
>
> commit c0d05305c156c83e4f9f3a207451b3175fbb7f24 (HEAD)
> Merge: c6b2e294d8 79f75eaa71
> Author: David Mitchell <davem@iabyn.com>
> Date: Mon Apr 27 22:04:23 2020 +0100
>
> [MERGE] Revert BEGIN { caller() } fixups
>
> These commits were intended to fix a problem with stack backtraces
> reporting wrong file and line numbers in nested use's.
>
> A side-effect of the commits was to fix the package name returned by
> caller() too; but quite a few distributions were relying on the old
> behaviour.
>
> So for now, revert to the old behaviour and re-address after 5.32.0 is
> released.
>
> The reverted commits are:
>
> v5.31.6-141-gf2f32cd638 avoid identical stack traces
> v5.31.9-122-gee428a211d docs: clarify effect of $^H, %^H,
> ${^WARNING_BITS}
> v5.31.9-162-gad89278aa2 fixup to "avoid identical stack traces" - try 2
>
>
> I don't know which distributions were failing or why, but having a few
> distributions making it harder for *all* Perl developers to debug their
> programs seems backwards.
>

Hearty agreement.


> What can we do to get this fixed?
>

Yves

>
>

For other's reference - it broke DBIX::Class
https://github.com/Perl/perl5/issues/17663, and you know it. It's
quite late in a release cycle to bring this up, since it'd require a
lot of efforts to release a fixed version of it to CPAN.

Best regards,
Sergey Aleynikov

On Thu, 13 Jan 2022 at 14:26, Sergey Aleynikov <sergey.aleynikov@gmail.com>
wrote:

> For other's reference - it broke DBIX::Class
> https://github.com/Perl/perl5/issues/17663, and you know it. It's
> quite late in a release cycle to bring this up, since it'd require a
> lot of efforts to release a fixed version of it to CPAN.
>

Yeah, I get your point, so maybe we have to suffer one more release, but
bottom line if DBIx::Class depends on this bug then it is depending on
clearly *broken* behaviour, and you have to wonder what insanity is being
hidden.

cheers,
Yves

On 2022-01-13 6:25 a.m., demerphq wrote:
> On Thu, 13 Jan 2022 at 14:26, Sergey Aleynikov wrote:
> For other's reference - it broke DBIX::Class
> https://github.com/Perl/perl5/issues/17663, and you know it. It's
> quite late in a release cycle to bring this up, since it'd require a
> lot of efforts to release a fixed version of it to CPAN.
>
> Yeah, I get your point, so maybe we have to suffer one more release, but bottom
> line if DBIx::Class depends on this bug then it is depending on clearly *broken*
> behaviour, and you have to wonder what insanity is being hidden.

I feel that Perl 5.36.0 SHOULD fix its own broken behavior regarding stack traces.

Just don't fix it in any 5.34.x or 5.32.x etc.

The 5.36.0 release is a major release and is far enough away that DBIx::Class
should be able to get fixed to not rely on the broken behavior in time for its
release.

The fix shouldn't be put off another year. If anyone depends on something that
depends on the broken behavior, they should not upgrade past 5.34.x until their
other dependencies are updated for compatibility.

-- Darren Duncan

On Fri, 14 Jan 2022 at 11:57, Darren Duncan <darren@darrenduncan.net> wrote:

> On 2022-01-13 6:25 a.m., demerphq wrote:
> > On Thu, 13 Jan 2022 at 14:26, Sergey Aleynikov wrote:
> > For other's reference - it broke DBIX::Class
> > https://github.com/Perl/perl5/issues/17663, and you know it. It's
> > quite late in a release cycle to bring this up, since it'd require a
> > lot of efforts to release a fixed version of it to CPAN.
> >
> > Yeah, I get your point, so maybe we have to suffer one more release, but
> bottom
> > line if DBIx::Class depends on this bug then it is depending on clearly
> *broken*
> > behaviour, and you have to wonder what insanity is being hidden.
>
> I feel that Perl 5.36.0 SHOULD fix its own broken behavior regarding stack
> traces.
>
> Just don't fix it in any 5.34.x or 5.32.x etc.
>
> The 5.36.0 release is a major release and is far enough away that
> DBIx::Class
> should be able to get fixed to not rely on the broken behavior in time for
> its
> release.
>
> The fix shouldn't be put off another year. If anyone depends on something
> that
> depends on the broken behavior, they should not upgrade past 5.34.x until
> their
> other dependencies are updated for compatibility.
>

I can live with that.

I am not really convinced the patch really does break anything however, we
have been running with the fix backported for years and we never noticed
any issues with our DBIx class code. *shrug*

Yves




--
perl -Mre=debug -e "/just|another|perl|hacker/"

On 2022-01-14 3:40 a.m., demerphq wrote:
> On Fri, 14 Jan 2022 at 11:57, Darren Duncan wrote:
> I feel that Perl 5.36.0 SHOULD fix its own broken behavior regarding stack
> traces.
>
> The 5.36.0 release is a major release and is far enough away that DBIx::Class
> should be able to get fixed to not rely on the broken behavior in time for its
> release.
>
> The fix shouldn't be put off another year.  If anyone depends on something that
> depends on the broken behavior, they should not upgrade past 5.34.x until their
> other dependencies are updated for compatibility.
>
> I can live with that.
>
> I am not really convinced the patch really does break anything however, we have
> been running with the fix backported for years and we never noticed any issues
> with our DBIx class code. *shrug*

That's all the more reason to re-apply the bug fix to blead right away.

Your experience would imply that any reliance in DBIx::Class on the broken
behavior isn't in its core functionality and more in some non-core functionality
or alternately just a test is broken and main code isn't.

-- Darren Duncan

On Fri, 14 Jan 2022, 20:00 Darren Duncan, <darren@darrenduncan.net> wrote:

> On 2022-01-14 3:40 a.m., demerphq wrote:
> > On Fri, 14 Jan 2022 at 11:57, Darren Duncan wrote:
> > I feel that Perl 5.36.0 SHOULD fix its own broken behavior regarding
> stack
> > traces.
> >
> > The 5.36.0 release is a major release and is far enough away that
> DBIx::Class
> > should be able to get fixed to not rely on the broken behavior in
> time for its
> > release.
> >
> > The fix shouldn't be put off another year. If anyone depends on
> something that
> > depends on the broken behavior, they should not upgrade past 5.34.x
> until their
> > other dependencies are updated for compatibility.
> >
> > I can live with that.
> >
> > I am not really convinced the patch really does break anything however,
> we have
> > been running with the fix backported for years and we never noticed any
> issues
> > with our DBIx class code. *shrug*
>
> That's all the more reason to re-apply the bug fix to blead right away.
>
> Your experience would imply that any reliance in DBIx::Class on the broken
> behavior isn't in its core functionality and more in some non-core
> functionality
> or alternately just a test is broken and main code isn't.
>

That would be my intuition, but I don't know that we use enough DBIx to be
certain. But it's extremely hard for me to understand how such broken data
could ever be truly useful.

Yves

>
>

As per https://github.com/Perl/perl5/issues/17663#issuecomment-602244332,
the "bug" in DBIC is an issue only in a test, not in actually code.

On Fri, Jan 14, 2022 at 2:14 PM demerphq <demerphq@gmail.com> wrote:

>
>
> On Fri, 14 Jan 2022, 20:00 Darren Duncan, <darren@darrenduncan.net> wrote:
>
>> On 2022-01-14 3:40 a.m., demerphq wrote:
>> > On Fri, 14 Jan 2022 at 11:57, Darren Duncan wrote:
>> > I feel that Perl 5.36.0 SHOULD fix its own broken behavior
>> regarding stack
>> > traces.
>> >
>> > The 5.36.0 release is a major release and is far enough away that
>> DBIx::Class
>> > should be able to get fixed to not rely on the broken behavior in
>> time for its
>> > release.
>> >
>> > The fix shouldn't be put off another year. If anyone depends on
>> something that
>> > depends on the broken behavior, they should not upgrade past 5.34.x
>> until their
>> > other dependencies are updated for compatibility.
>> >
>> > I can live with that.
>> >
>> > I am not really convinced the patch really does break anything however,
>> we have
>> > been running with the fix backported for years and we never noticed any
>> issues
>> > with our DBIx class code. *shrug*
>>
>> That's all the more reason to re-apply the bug fix to blead right away.
>>
>> Your experience would imply that any reliance in DBIx::Class on the
>> broken
>> behavior isn't in its core functionality and more in some non-core
>> functionality
>> or alternately just a test is broken and main code isn't.
>>
>
> That would be my intuition, but I don't know that we use enough DBIx to be
> certain. But it's extremely hard for me to understand how such broken data
> could ever be truly useful.
>
> Yves
>
>>
>>

On Fri, Jan 14, 2022 at 1:15 PM demerphq <demerphq@gmail.com> wrote:

> On Fri, 14 Jan 2022, 20:00 Darren Duncan, <darren@darrenduncan.net> wrote:
>
>> On 2022-01-14 3:40 a.m., demerphq wrote:
>> > On Fri, 14 Jan 2022 at 11:57, Darren Duncan wrote:
>> > I feel that Perl 5.36.0 SHOULD fix its own broken behavior
>> regarding stack
>> > traces.
>> >
>> > The 5.36.0 release is a major release and is far enough away that
>> DBIx::Class
>> > should be able to get fixed to not rely on the broken behavior in
>> time for its
>> > release.
>> >
>> > The fix shouldn't be put off another year. If anyone depends on
>> something that
>> > depends on the broken behavior, they should not upgrade past 5.34.x
>> until their
>> > other dependencies are updated for compatibility.
>> >
>> > I can live with that.
>> >
>> > I am not really convinced the patch really does break anything however,
>> we have
>> > been running with the fix backported for years and we never noticed any
>> issues
>> > with our DBIx class code. *shrug*
>>
>> That's all the more reason to re-apply the bug fix to blead right away.
>>
>> Your experience would imply that any reliance in DBIx::Class on the
>> broken
>> behavior isn't in its core functionality and more in some non-core
>> functionality
>> or alternately just a test is broken and main code isn't.
>>
>
> That would be my intuition, but I don't know that we use enough DBIx to be
> certain. But it's extremely hard for me to understand how such broken data
> could ever be truly useful.
>

What breaks is a test that checks what modules are loaded by DBIx::Class.
The change in behavior causes it to make different conclusions as to who is
loading what. I suspect this is representative for the sort of code that
would be affected by this change.

Leon

On Fri, 14 Jan 2022, 21:27 Leon Timmermans, <fawaka@gmail.com> wrote:

> On Fri, Jan 14, 2022 at 1:15 PM demerphq <demerphq@gmail.com> wrote:
>
>> On Fri, 14 Jan 2022, 20:00 Darren Duncan, <darren@darrenduncan.net>
>> wrote:
>>
>>> On 2022-01-14 3:40 a.m., demerphq wrote:
>>> > On Fri, 14 Jan 2022 at 11:57, Darren Duncan wrote:
>>> > I feel that Perl 5.36.0 SHOULD fix its own broken behavior
>>> regarding stack
>>> > traces.
>>> >
>>> > The 5.36.0 release is a major release and is far enough away that
>>> DBIx::Class
>>> > should be able to get fixed to not rely on the broken behavior in
>>> time for its
>>> > release.
>>> >
>>> > The fix shouldn't be put off another year. If anyone depends on
>>> something that
>>> > depends on the broken behavior, they should not upgrade past
>>> 5.34.x until their
>>> > other dependencies are updated for compatibility.
>>> >
>>> > I can live with that.
>>> >
>>> > I am not really convinced the patch really does break anything
>>> however, we have
>>> > been running with the fix backported for years and we never noticed
>>> any issues
>>> > with our DBIx class code. *shrug*
>>>
>>> That's all the more reason to re-apply the bug fix to blead right away.
>>>
>>> Your experience would imply that any reliance in DBIx::Class on the
>>> broken
>>> behavior isn't in its core functionality and more in some non-core
>>> functionality
>>> or alternately just a test is broken and main code isn't.
>>>
>>
>> That would be my intuition, but I don't know that we use enough DBIx to
>> be certain. But it's extremely hard for me to understand how such broken
>> data could ever be truly useful.
>>
>
> What breaks is a test that checks what modules are loaded by DBIx::Class.
> The change in behavior causes it to make different conclusions as to who is
> loading what. I suspect this is representative for the sort of code that
> would be affected by this change.
>


But those assumptions are basically broken, so I'm not sure your point.
Classic case of GIGO.

Yves

>

On Fri, Jan 14, 2022 at 3:33 PM demerphq <demerphq@gmail.com> wrote:

> What breaks is a test that checks what modules are loaded by DBIx::Class.
>> The change in behavior causes it to make different conclusions as to who is
>> loading what. I suspect this is representative for the sort of code that
>> would be affected by this change.
>>
>
> But those assumptions are basically broken, so I'm not sure your point.
> Classic case of GIGO.
>

It's not broken if it gets the job done, is it?

I agree with you that the new behavior is far more usable than the older
behavior, but that doesn't mean people haven't put the old behavior to use.

Leon

On Friday, 14 January 2022, 15:54:27 CET, Leon Timmermans <fawaka@gmail.com> wrote:

> > But those assumptions are basically broken, so I'm not sure your point. Classic case of GIGO.
>
> It's not broken if it gets the job done, is it?

> I agree with you that the new behavior is far more usable than the older behavior, but that doesn't
> mean people haven't put the old behavior to use.

I think the old behavior kind of worked because generally, stack traces work. It appears to be the use/require case which is problematic.

However, in the use/require case, if the apparently pseudo-random ordering is deterministic, on my machine the first stack frame grabbed happens to be correct. If that holds (pretty sure it must), then for the 'import' case where you're trying to figure out where you're exporting to, you're going to export those functions to the right spot. It's when you hit stack traces that it becomes an unusable mess.

As it stands:

* The stack trace is clearly incorrect
* That makes it useless for debugging
* When you're working on a *huge* system and get those traces, it's miserable

Given that, I suspect the most useful discussions are about *how* to fix it and/or *when* to fix it..

Best,
Ovid
-- 
IT consulting, training, specializing in Perl, databases, and agile development
http://www.allaroundtheworld.fr/. 


Buy my book! - http://bit.ly/beginning_perl

On Fri, Jan 14, 2022 at 4:59 PM Ovid <curtis_ovid_poe@yahoo.com> wrote:

> On Friday, 14 January 2022, 15:54:27 CET, Leon Timmermans <
> fawaka@gmail.com> wrote:
>
> > > But those assumptions are basically broken, so I'm not sure your
> point. Classic case of GIGO.
> >
> > It's not broken if it gets the job done, is it?
>
> > I agree with you that the new behavior is far more usable than the older
> behavior, but that doesn't
> > mean people haven't put the old behavior to use.
>
> I think the old behavior kind of worked because generally, stack traces
> work. It appears to be the use/require case which is problematic.
>
> However, in the use/require case, if the apparently pseudo-random ordering
> is deterministic, on my machine the first stack frame grabbed happens to be
> correct. If that holds (pretty sure it must), then for the 'import' case
> where you're trying to figure out where you're exporting to, you're going
> to export those functions to the right spot. It's when you hit stack traces
> that it becomes an unusable mess.
>
> As it stands:
>
> * The stack trace is clearly incorrect
> * That makes it useless for debugging
> * When you're working on a *huge* system and get those traces, it's
> miserable
>
> Given that, I suspect the most useful discussions are about *how* to fix
> it and/or *when* to fix it.
>

"How to fix it" probably includes people stepping up to deal with the
fallout. IME the most useful thing proponents of this change could do is
actually write patches for the 3 known affected modules. I don't really
understand how that hasn't happened yet in the past two years.

Leon

On Sat, 15 Jan 2022, 00:45 Leon Timmermans, <fawaka@gmail.com> wrote:

> On Fri, Jan 14, 2022 at 4:59 PM Ovid <curtis_ovid_poe@yahoo.com> wrote:
>
>> On Friday, 14 January 2022, 15:54:27 CET, Leon Timmermans <
>> fawaka@gmail.com> wrote:
>>
>> > > But those assumptions are basically broken, so I'm not sure your
>> point. Classic case of GIGO.
>> >
>> > It's not broken if it gets the job done, is it?
>>
>> > I agree with you that the new behavior is far more usable than the
>> older behavior, but that doesn't
>> > mean people haven't put the old behavior to use.
>>
>> I think the old behavior kind of worked because generally, stack traces
>> work. It appears to be the use/require case which is problematic.
>>
>> However, in the use/require case, if the apparently pseudo-random
>> ordering is deterministic, on my machine the first stack frame grabbed
>> happens to be correct. If that holds (pretty sure it must), then for the
>> 'import' case where you're trying to figure out where you're exporting to,
>> you're going to export those functions to the right spot. It's when you hit
>> stack traces that it becomes an unusable mess.
>>
>> As it stands:
>>
>> * The stack trace is clearly incorrect
>> * That makes it useless for debugging
>> * When you're working on a *huge* system and get those traces, it's
>> miserable
>>
>> Given that, I suspect the most useful discussions are about *how* to fix
>> it and/or *when* to fix it.
>>
>
> "How to fix it" probably includes people stepping up to deal with the
> fallout. IME the most useful thing proponents of this change could do is
> actually write patches for the 3 known affected modules. I don't really
> understand how that hasn't happened yet in the past two years.
>

I think there is a feeling that this is just like people baking a
dependency on hash order into their tests. All I saw on this subject was
people claiming the obviously broken behavior was "correct" and they
wouldn't accept that the change had a reasonable foundation. I disengaged
after people started arguing that the stack trace claiming things were used
on lines where they clearly weren't was correct and reasonable. There is to
much to do in life than to argue with people who vehemently insist that up
is down. Furthermore I didn't see anybody demonstrate a logical issue that
was clearly a problem, just broken and badly designed tests.

Anyway I'll find some time to dig into the DBIx part, but I don't know what
the other two are. If you do it would save me some time digging through the
email backlogs.

Yves

>

On Fri, Jan 14, 2022 at 5:44 PM Leon Timmermans <fawaka@gmail.com> wrote:

> On Fri, Jan 14, 2022 at 4:59 PM Ovid <curtis_ovid_poe@yahoo.com> wrote:
>
>> On Friday, 14 January 2022, 15:54:27 CET, Leon Timmermans <
>> fawaka@gmail.com> wrote:
>>
>> > > But those assumptions are basically broken, so I'm not sure your
>> point. Classic case of GIGO.
>> >
>> > It's not broken if it gets the job done, is it?
>>
>> > I agree with you that the new behavior is far more usable than the
>> older behavior, but that doesn't
>> > mean people haven't put the old behavior to use.
>>
>> I think the old behavior kind of worked because generally, stack traces
>> work. It appears to be the use/require case which is problematic.
>>
>> However, in the use/require case, if the apparently pseudo-random
>> ordering is deterministic, on my machine the first stack frame grabbed
>> happens to be correct. If that holds (pretty sure it must), then for the
>> 'import' case where you're trying to figure out where you're exporting to,
>> you're going to export those functions to the right spot. It's when you hit
>> stack traces that it becomes an unusable mess.
>>
>> As it stands:
>>
>> * The stack trace is clearly incorrect
>> * That makes it useless for debugging
>> * When you're working on a *huge* system and get those traces, it's
>> miserable
>>
>> Given that, I suspect the most useful discussions are about *how* to fix
>> it and/or *when* to fix it.
>>
>
> "How to fix it" probably includes people stepping up to deal with the
> fallout. IME the most useful thing proponents of this change could do is
> actually write patches for the 3 known affected modules. I don't really
> understand how that hasn't happened yet in the past two years.
>

I think I just did that for DBIx::Class.

Leon

On Sat, 15 Jan 2022 at 05:18, Leon Timmermans <fawaka@gmail.com> wrote:

> On Fri, Jan 14, 2022 at 5:44 PM Leon Timmermans <fawaka@gmail.com> wrote:
>
>> On Fri, Jan 14, 2022 at 4:59 PM Ovid <curtis_ovid_poe@yahoo.com> wrote:
>>
>>> On Friday, 14 January 2022, 15:54:27 CET, Leon Timmermans <
>>> fawaka@gmail.com> wrote:
>>>
>>> > > But those assumptions are basically broken, so I'm not sure your
>>> point. Classic case of GIGO.
>>> >
>>> > It's not broken if it gets the job done, is it?
>>>
>>> > I agree with you that the new behavior is far more usable than the
>>> older behavior, but that doesn't
>>> > mean people haven't put the old behavior to use.
>>>
>>> I think the old behavior kind of worked because generally, stack traces
>>> work. It appears to be the use/require case which is problematic.
>>>
>>> However, in the use/require case, if the apparently pseudo-random
>>> ordering is deterministic, on my machine the first stack frame grabbed
>>> happens to be correct. If that holds (pretty sure it must), then for the
>>> 'import' case where you're trying to figure out where you're exporting to,
>>> you're going to export those functions to the right spot. It's when you hit
>>> stack traces that it becomes an unusable mess.
>>>
>>> As it stands:
>>>
>>> * The stack trace is clearly incorrect
>>> * That makes it useless for debugging
>>> * When you're working on a *huge* system and get those traces, it's
>>> miserable
>>>
>>> Given that, I suspect the most useful discussions are about *how* to fix
>>> it and/or *when* to fix it.
>>>
>>
>> "How to fix it" probably includes people stepping up to deal with the
>> fallout. IME the most useful thing proponents of this change could do is
>> actually write patches for the 3 known affected modules. I don't really
>> understand how that hasn't happened yet in the past two years.
>>
>
> I think I just did that for DBIx::Class.
>
>
Thanks, although I feel it is worth noting that this was not a fix to
DBIx::Class and the patch never broke DBIx::Class, it was the tests for
DBIx::Class that were broken only.

cheers,
Yves

--
perl -Mre=debug -e "/just|another|perl|hacker/"

On Sat, 15 Jan 2022 at 07:06, demerphq <demerphq@gmail.com> wrote:

> On Sat, 15 Jan 2022 at 05:18, Leon Timmermans <fawaka@gmail.com> wrote:
>
>> On Fri, Jan 14, 2022 at 5:44 PM Leon Timmermans <fawaka@gmail.com> wrote:
>>
>>> On Fri, Jan 14, 2022 at 4:59 PM Ovid <curtis_ovid_poe@yahoo.com> wrote:
>>>
>>>> On Friday, 14 January 2022, 15:54:27 CET, Leon Timmermans <
>>>> fawaka@gmail.com> wrote:
>>>>
>>>> > > But those assumptions are basically broken, so I'm not sure your
>>>> point. Classic case of GIGO.
>>>> >
>>>> > It's not broken if it gets the job done, is it?
>>>>
>>>> > I agree with you that the new behavior is far more usable than the
>>>> older behavior, but that doesn't
>>>> > mean people haven't put the old behavior to use.
>>>>
>>>> I think the old behavior kind of worked because generally, stack traces
>>>> work. It appears to be the use/require case which is problematic.
>>>>
>>>> However, in the use/require case, if the apparently pseudo-random
>>>> ordering is deterministic, on my machine the first stack frame grabbed
>>>> happens to be correct. If that holds (pretty sure it must), then for the
>>>> 'import' case where you're trying to figure out where you're exporting to,
>>>> you're going to export those functions to the right spot. It's when you hit
>>>> stack traces that it becomes an unusable mess.
>>>>
>>>> As it stands:
>>>>
>>>> * The stack trace is clearly incorrect
>>>> * That makes it useless for debugging
>>>> * When you're working on a *huge* system and get those traces, it's
>>>> miserable
>>>>
>>>> Given that, I suspect the most useful discussions are about *how* to
>>>> fix it and/or *when* to fix it.
>>>>
>>>
>>> "How to fix it" probably includes people stepping up to deal with the
>>> fallout. IME the most useful thing proponents of this change could do is
>>> actually write patches for the 3 known affected modules. I don't really
>>> understand how that hasn't happened yet in the past two years.
>>>
>>
>> I think I just did that for DBIx::Class.
>>
>>
> Thanks, although I feel it is worth noting that this was not a fix to
> DBIx::Class and the patch never broke DBIx::Class, it was the tests for
> DBIx::Class that were broken only.
>
> I have looked at the other two cases and as far as I can tell Dave M did
actually dig into them and fix them. They are in the thread here:

https://github.com/Perl/perl5/issues/17663

BTW, in that thread you said: "This ticket raises the question «what is
correctness». The current behavior is truthful, but not intuitive. This
discussion really seems to be about which of those two values are more
relevant for correctness."

Consider the following:

~/tlib$ cat {A..D}.pm
package A;
BEGIN { Carp::cluck "\n--\nIn ", __PACKAGE__; }
use B;
1;
__END__


package B;
BEGIN { Carp::cluck "\n--\nIn ", __PACKAGE__; }
use C;
1;
__END__


package C;
BEGIN { Carp::cluck "\n--\nIn ", __PACKAGE__; }
use D;
1;
__END__


package D;
BEGIN { Carp::cluck "\n--\nIn ", __PACKAGE__; }
1;
__END__


~/tlib$ perl -I. -MCarp -e'use A;'

--
In A at A.pm line 2.
A::BEGIN() called at A.pm line 2
eval {...} called at A.pm line 2
require A.pm called at -e line 1
main::BEGIN() called at A.pm line 2
eval {...} called at A.pm line 2

--
In B at B.pm line 2.
B::BEGIN() called at B.pm line 2
eval {...} called at B.pm line 2
require B.pm called at A.pm line 3
A::BEGIN() called at B.pm line 2
eval {...} called at B.pm line 2
require A.pm called at -e line 1
main::BEGIN() called at B.pm line 2
eval {...} called at B.pm line 2

--
In C at C.pm line 2.
C::BEGIN() called at C.pm line 2
eval {...} called at C.pm line 2
require C.pm called at B.pm line 3
B::BEGIN() called at C.pm line 2
eval {...} called at C.pm line 2
require B.pm called at A.pm line 3
A::BEGIN() called at C.pm line 2
eval {...} called at C.pm line 2
require A.pm called at -e line 1
main::BEGIN() called at C.pm line 2
eval {...} called at C.pm line 2

--
In D at D.pm line 2.
D::BEGIN() called at D.pm line 2
eval {...} called at D.pm line 2
require D.pm called at C.pm line 3
C::BEGIN() called at D.pm line 2
eval {...} called at D.pm line 2
require C.pm called at B.pm line 3
B::BEGIN() called at D.pm line 2
eval {...} called at D.pm line 2
require B.pm called at A.pm line 3
A::BEGIN() called at D.pm line 2
eval {...} called at D.pm line 2
require A.pm called at -e line 1
main::BEGIN() called at D.pm line 2
eval {...} called at D.pm line 2

So respectfully I would like to know how you can consider such
contradictory results (eg the same stack frame being attributed to
different lines of code depending on when caller was called) can be
truthful, especially given some of the stack frames are BEGIN's which fire
once and are never fired again, it seems to me that we should be able to
agree that such a construct should be shown to come from exactly one place,
and one place alone.

Cheers,
Yves

On Sat, Jan 15, 2022 at 8:22 AM demerphq <demerphq@gmail.com> wrote:

> On Sat, 15 Jan 2022 at 07:06, demerphq <demerphq@gmail.com> wrote:
>
>> On Sat, 15 Jan 2022 at 05:18, Leon Timmermans <fawaka@gmail.com> wrote:
>>
>>> On Fri, Jan 14, 2022 at 5:44 PM Leon Timmermans <fawaka@gmail.com>
>>> wrote:
>>>
>>>> On Fri, Jan 14, 2022 at 4:59 PM Ovid <curtis_ovid_poe@yahoo.com> wrote:
>>>>
>>>>> On Friday, 14 January 2022, 15:54:27 CET, Leon Timmermans <
>>>>> fawaka@gmail.com> wrote:
>>>>>
>>>>> > > But those assumptions are basically broken, so I'm not sure your
>>>>> point. Classic case of GIGO.
>>>>> >
>>>>> > It's not broken if it gets the job done, is it?
>>>>>
>>>>> > I agree with you that the new behavior is far more usable than the
>>>>> older behavior, but that doesn't
>>>>> > mean people haven't put the old behavior to use.
>>>>>
>>>>> I think the old behavior kind of worked because generally, stack
>>>>> traces work. It appears to be the use/require case which is problematic.
>>>>>
>>>>> However, in the use/require case, if the apparently pseudo-random
>>>>> ordering is deterministic, on my machine the first stack frame grabbed
>>>>> happens to be correct. If that holds (pretty sure it must), then for the
>>>>> 'import' case where you're trying to figure out where you're exporting to,
>>>>> you're going to export those functions to the right spot. It's when you hit
>>>>> stack traces that it becomes an unusable mess.
>>>>>
>>>>> As it stands:
>>>>>
>>>>> * The stack trace is clearly incorrect
>>>>> * That makes it useless for debugging
>>>>> * When you're working on a *huge* system and get those traces, it's
>>>>> miserable
>>>>>
>>>>> Given that, I suspect the most useful discussions are about *how* to
>>>>> fix it and/or *when* to fix it.
>>>>>
>>>>
>>>> "How to fix it" probably includes people stepping up to deal with the
>>>> fallout. IME the most useful thing proponents of this change could do is
>>>> actually write patches for the 3 known affected modules. I don't really
>>>> understand how that hasn't happened yet in the past two years.
>>>>
>>>
>>> I think I just did that for DBIx::Class.
>>>
>>>
>> Thanks, although I feel it is worth noting that this was not a fix to
>> DBIx::Class and the patch never broke DBIx::Class, it was the tests for
>> DBIx::Class that were broken only.
>>
>> I have looked at the other two cases and as far as I can tell Dave M did
> actually dig into them and fix them. They are in the thread here:
>
> https://github.com/Perl/perl5/issues/17663
>
> BTW, in that thread you said: "This ticket raises the question «what is
> correctness». The current behavior is truthful, but not intuitive. This
> discussion really seems to be about which of those two values are more
> relevant for correctness."
>
> Consider the following:
>
> ...
>
> So respectfully I would like to know how you can consider such
> contradictory results (eg the same stack frame being attributed to
> different lines of code depending on when caller was called) can be
> truthful, especially given some of the stack frames are BEGIN's which fire
> once and are never fired again, it seems to me that we should be able to
> agree that such a construct should be shown to come from exactly one place,
> and one place alone.
>

As far as I can tell, it's an accurate representation of how the sausage is
made. BEGIN blocks aren't implemented the way one would expect them to, and
these stacktraces represent that. They're not false, just very unhelpful.

Leon

On Fri, Jan 14, 2022 at 10:06 PM demerphq <demerphq@gmail.com> wrote:

> Thanks, although I feel it is worth noting that this was not a fix to
> DBIx::Class and the patch never broke DBIx::Class, it was the tests for
> DBIx::Class that were broken only.
>

I don't think that's very fair to say. The tests are verifying behaviour
expected by the runtime code, and there is a reason why they are written
the way they are. Otherwise, any broken test could be "fixed" by simply
removing the test altogether. We've all worked with junior programmers who
want to do that, and we've all rightfully scolded them for it.

Part of fixing the test is understanding why it was written the way it was,
and what runtime behaviour is associated with that test. Changing a test's
expected result means that runtime behaviour is also changing, and other
things might need to be addressed there as a result.

On the other hand, it could be that this test isn't testing anything useful
at all, but I don't think that's been established here.

On 1/15/22 19:01, Karen Etheridge wrote:

>
>
> On Fri, Jan 14, 2022 at 10:06 PM demerphq <demerphq@gmail.com> wrote:
>
> Thanks, although I feel it is worth noting that this was not a fix
> to DBIx::Class and the patch never broke DBIx::Class, it was the
> tests for DBIx::Class that were broken only.
>
> I don't think that's very fair to say. The tests are verifying
> behaviour expected by the runtime code, and there is a reason why they
> are written the way they are. Otherwise, any broken test could be
> "fixed" by simply removing the test altogether. We've all worked with
> junior programmers who want to do that, and we've all rightfully
> scolded them for it.
>
> Part of fixing the test is understanding why it was written the way it
> was, and what runtime behaviour is associated with that test. Changing
> a test's expected result means that runtime behaviour is also
> changing, and other things might need to be addressed there as a result.
>
> On the other hand, it could be that this test isn't testing anything
> useful at all, but I don't think that's been established here.
>

Or the test tests something useful, works around bugs to do so, and when
those bugs get fixed, the test starts failing. It looks like that is the
case here.


HTH,

M4

On Sat, 15 Jan 2022 at 14:27, Leon Timmermans <fawaka@gmail.com> wrote:

> On Sat, Jan 15, 2022 at 8:22 AM demerphq <demerphq@gmail.com> wrote:
>
>>
>> As far as I can tell, it's an accurate representation of how the sausage
> is made. BEGIN blocks aren't implemented the way one would expect them to,
> and these stacktraces represent that. They're not false, just very
> unhelpful.
>

I really don't understand your position on this. We see output that says 4
contradictory things, none of which are actually accurate. So which one of
the four is true and which ones are false? I dont think the following
statements can all be true simultaneously.

main::BEGIN() called at A.pm line 2
main::BEGIN() called at B.pm line 2
main::BEGIN() called at C.pm line 2
main::BEGIN() called at D.pm line 2

But main::BEGIN is actually called at -e line 1. I mean, if the BEGIN
statement were created in a module it would would A::BEGIN or B::BEGIN. For
example:

$ perl -MCarp=cluck -le'package Foo; BEGIN { Carp::cluck("in begin") }'
in begin at -e line 1.
Foo::BEGIN() called at -e line 1
eval {...} called at -e line 1

I really think this is an important point that keeps getting lost. Sure you
can say "this is how the sausage is made", as this is the result of using a
single global to track the origin of ALL synthesized BEGIN blocks which are
part of a use statement. But that is circular logic. "it is correct and
true because that is how it works" could be used to justify leaving unfixed
ANY bug we encounter.

If the output did not say "called at A.pm line 2" but said "fake-frame
injected by compiler" or something like that, I could buy the "this is how
the sausage is made" argument. But it doesn't. It says something different
each time. None of which seem to have any relationship to reality.

It seems to me we should be able to agree on whether the output is true,
and whether it is correct. I am really struggling to understand why this is
controversial. It seems to me we should be able to agree that main::BEGIN
*cannot* be called from anywhere but main, and this means that the output
claiming that the block was called from a module file cannot be correct.

Cheers,
Yves

--
perl -Mre=debug -e "/just|another|perl|hacker/"

On Sat, 15 Jan 2022 at 19:01, Karen Etheridge <perl@froods.org> wrote:

>
>
> On Fri, Jan 14, 2022 at 10:06 PM demerphq <demerphq@gmail.com> wrote:
>
>> Thanks, although I feel it is worth noting that this was not a fix to
>> DBIx::Class and the patch never broke DBIx::Class, it was the tests for
>> DBIx::Class that were broken only.
>>
>
> I don't think that's very fair to say.
>

Fair is a question of subjective opinion. Correctness should be of
universal agreement. If something isnt correct it is broken.

Would you take this position if someone built a dependency on the exact
hash order returned by an old Perl?

Yves

On Sat, 15 Jan 2022 at 20:20, Martijn Lievaart <m@rtij.nl> wrote:

> On 1/15/22 19:01, Karen Etheridge wrote:
>
>
>
> On Fri, Jan 14, 2022 at 10:06 PM demerphq <demerphq@gmail.com> wrote:
>
>> Thanks, although I feel it is worth noting that this was not a fix to
>> DBIx::Class and the patch never broke DBIx::Class, it was the tests for
>> DBIx::Class that were broken only.
>>
>
> I don't think that's very fair to say. The tests are verifying behaviour
> expected by the runtime code, and there is a reason why they are written
> the way they are. Otherwise, any broken test could be "fixed" by simply
> removing the test altogether. We've all worked with junior programmers who
> want to do that, and we've all rightfully scolded them for it.
>
> Part of fixing the test is understanding why it was written the way it
> was, and what runtime behaviour is associated with that test. Changing a
> test's expected result means that runtime behaviour is also changing, and
> other things might need to be addressed there as a result.
>
> On the other hand, it could be that this test isn't testing anything
> useful at all, but I don't think that's been established here.
>
>
> Or the test tests something useful, works around bugs to do so, and when
> those bugs get fixed, the test starts failing. It looks like that is the
> case here.
>
That doesn't change the fact that the *test* was broken by this change, NOT
the code it was testing. Fixing the test to pass under the caller() fix did
not require modifying anything but the test code, and as such I feel
entirely entirely to say that the *test* was broken, not the code.

cheers,
Yves

--
perl -Mre=debug -e "/just|another|perl|hacker/"