Mailing List Archive

Is there a way to turn on extra debugging in perl
.... I just started getting
Out of memory! at random times I'm trying to figure out
where or what statement might be triggering

free shows plenty of memory ....
> free
total used free shared buff/cache
available
Mem: 165036564 7948136 152956592 1648 4131836
155818956
Swap: 8388604 227112 8161492

I have Carp::Always on but I guess that doesn't apply to out-of-memory
errors?
Re: Is there a way to turn on extra debugging in perl [ In reply to ]
On Mon, Aug 9, 2021 at 7:16 PM L. Walsh <perl-diddler@tlinx.org> wrote:

> .... I just started getting
> Out of memory! at random times I'm trying to figure out
> where or what statement might be triggering
>
> free shows plenty of memory ....
> > free
> total used free shared buff/cache
> available
> Mem: 165036564 7948136 152956592 1648 4131836
> 155818956
> Swap: 8388604 227112 8161492
>
> I have Carp::Always on but I guess that doesn't apply to out-of-memory
> errors?
>

Correct, the only way to affect out of memory errors is by recompiling Perl
with specific options (to reserve a buffer for out-of-memory use) and then
using https://perldoc.perl.org/variables/$%5EM. Another option is to use
something like https://metacpan.org/pod/Devel::Trace as a hammer to log
everything that happens up until you run out of memory.

-Dan
Re: Is there a way to turn on extra debugging in perl [ In reply to ]
On 2021/08/09 16:25, Dan Book wrote:
>
> Correct, the only way to affect out of memory errors is by recompiling
> Perl with specific options (to reserve a buffer for out-of-memory use)
> and then using https://perldoc.perl.org/variables/$%5EM.
---
%5EM?...new one on me.
> Another option is to use something
> like https://metacpan.org/pod/Devel::Trace as a hammer to log
> everything that happens up until you run out of memory.
>
> -Dan
----
Wouldn't that be likely to affect the outcome? Though rt now, I'm seeing
a wide variance in fail times (with system memory barely scratched).

Just now, I started a D/l of all the sources in my distro -- failed after
<100 packages. Restarted, and it completed 44000 packages. such randomness
makes for more likely Heisenbugs... Thanks for the ptrs,
Re: Is there a way to turn on extra debugging in perl [ In reply to ]
"L. Walsh" <perl-diddler@tlinx.org> wrote:
:On 2021/08/09 16:25, Dan Book wrote:
:>
:> Correct, the only way to affect out of memory errors is by recompiling
:> Perl with specific options (to reserve a buffer for out-of-memory use)
:> and then using https://perldoc.perl.org/variables/$%5EM.
:---
:%5EM?...new one on me.

That's the $^M variable, lightly documented in perlvar. There's a bit
more info on perlmonks at eg
https://perlmonks.org/index.pl?node_id=11130839
which also has a couple of additional links that look handy.

In this context, using $^M could get you a stacktrace at the point of
failure, which could well point you directly at the problem.

:> Another option is to use something
:> like https://metacpan.org/pod/Devel::Trace as a hammer to log
:> everything that happens up until you run out of memory.
:
:Wouldn't that be likely to affect the outcome? Though rt now, I'm seeing
:a wide variance in fail times (with system memory barely scratched).

If it is suddenly running out of memory rather than gradually growing
in size, it could well be a wild allocation of some sort. Commonest
cause of that in pure perl is using the numeric value of a reference
in a context that causes an allocation, such as an array index.

perl will give a warning if you directly use a reference as an array index:

% perl -wle 'my @a; my $index = {}; $a[$index] = 1'
Use of reference "HASH(0x5574970ad1e0)" as array index at -e line 1.
Out of memory!

It is not able to do so if you use it indirectly:

% perl -wle 'my @a; my $key = {}; my $index = $key + 0; $a[$index] = 1'
Out of memory!

And the same problem can arise in other ways:

% perl -wle 'my $index = {}; my $s = "x" x $index'
Out of memory!

[.p5p: any reason we couldn't warn on this, same as for array indices?]

There has been talk of a 'strict'-like mode that would trap all
stringification and numification of references, but as far as I know
no such module exists at the moment.

_If_ something like that is the cause (and if $^M seems like too much
effort), logging seems like the best way to narrow down which code
to audit for the error, and there's a good chance that the additional
logging will not affect the outcome.

Hugo
Re: Is there a way to turn on extra debugging in perl [ In reply to ]
On 2021/08/09 19:21, hv@crypt.org wrote:
> "L. Walsh" <perl-diddler@tlinx.org> wrote:
> :On 2021/08/09 16:25, Dan Book wrote:
>
---
Now the problem seems to have gone deep....
Just went through full downloads of latest distro...no prob...


Looked at the source code, and found about 6+ locations for
the same error message. hmmmmmm....