Mailing List Archive

`cpan` unusable on small-RAM machines
I have a Raspberry Pi Zero. It has 512MiB of RAM. Hardly small even by
modern standards. The `cpan` command is basically useless on it.


In more detail: I cannot install anything with the `cpan`. It gets
OOM-killed.

Or, more specifically, if I use

$ cpan Some::Module
or
$ cpan
cpan> install Some::Module

Then it manages to download, unpack, build and test but then gets killed
after the test phase but before the install phase.

...
All tests successful.
Files=1, Tests=32, 0 wallclock secs ( 0.13 usr 0.02 sys + 0.36 cusr
0.08 csys = 0.59 CPU) Result: PASS
Lockfile removed.
Killed

More detail from `dmesg`:

[Mon Nov 13 21:51:44 2023] oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=/,mems_allowed=0,global_oom,task_memcg=/,task=cpan,pid=31971,uid=999
[Mon Nov 13 21:51:44 2023] Out of memory: Killed process 31971 (cpan) total-vm:237996kB, anon-rss:151540kB, file-rss:0kB, shmem-rss:0kB, UID:999 pgtables:244kB oom_score_adj:0

However, I can use the `look` command and manually

$ perl Build.PL
$ perl Build && perl Build test && perl Build install

and that works just fine, no OOM-kill - even from within the `cpan`
shell while CPAN.pm is still running.

First off - I'm not looking to be told how to install stuff. I know how
I can install stuff on my machine - that isn't the point of this email.
See below.

The point of this email is two-fold:

1) We seem to have ended up in a situation where "Perl is unsuable"
on small-memory systems like Raspberry Pis ((Pies?)). I don't have
this problem with Node or Python or... really anything else. Just
Perl. This doesn't paint Perl in a good light on these platforms as
compared to other languages. It's not a good look for us.

2) What on earth is CPAN.pm doing after the test phase but before the
install phase that suddenly needs all this memory, anyway? I could
just write myself a small script to automate around using CPAN.pm
to only grab the tarball, then unpack it and run the build scripts
locally myself. And a bit of extra magic around that would recurse
dependencies and ... wait why on earth would I do that when that
is LITERALLY what CPAN.pm is supposed to be doing?

Perhaps if I hadn't said "don't tell me" above, some folks would
have told me to use cpanm instead? To you folks, I have one
question in reply - Why? More specifically, why is cpan (the
core-shipped and therefore always available, officially-blessed
tool) unable to do this task, but cpanm (a third-party needs extra
steps to install it itself presuming that an end-user even knows
it exists) seems to be the "better" version?


Can we fix this please?

Either find out and fix why CPAN.pm eats all that memory, or write and
supply *IN CORE* a better, more minimised tool, that works better in
this situation.

--
Paul "LeoNerd" Evans

leonerd@leonerd.org.uk | https://metacpan.org/author/PEVANS
http://www.leonerd.org.uk/ | https://www.tindie.com/stores/leonerd/
Re: `cpan` unusable on small-RAM machines [ In reply to ]
Hi there,

Well, somebody cares. :)

On Mon, 13 Nov 2023, Paul "LeoNerd" Evans wrote:

> I have a Raspberry Pi Zero. It has 512MiB of RAM. Hardly small even by
> modern standards. The `cpan` command is basically useless on it.

I agree that half a gig of RAM should be enough to do almost anything
but it seems the world is no longer really interested in efficiency.
I used to serve all the computing needs of a small business with just
a 5MHz 8086 box, with 768kiB of RAM and a 10MB hard disc. Nowadays,
the initrd wouldn't fit on that disc.

But I'm afraid I can't agree that 512MiB of RAM is 'hardly small' by
modern standards. If you'd said that it ought to be enough to install
a Perl module I'd agree. If you'd said that it ought to be enough to
run a browser (after all when I was Compatibility King at Glendon Ave.
back in 2000, we had 32Mbytes of RAM if we were lucky but managed to
run any existing browser) I'd say, "Not any more! I can't even run a
browser reliably on a 1GiB Pi". 4GiB is *almost* enough. To avoid
problems, I find 8GiB is better. At the moment. We use 512MiB Zeros
here just to run the odd camera. I think they're a fiver apiece.

Having said that, I can't remember the last time I used cpan.pm to
install anything. I usually look for a distro package and use that.
I think once you start a cpan shell you're off the beaten track, and
you're supposed to manage all by yourself.

> In more detail: I cannot install anything with the `cpan`. It gets
> OOM-killed.

The OOM killer doesn't kill processes because there isn't enough RAM.
It kills them because there isn't enough *remaining* RAM. It might or
it might not be cpan's fault that your RAM is full, but wouldn't it be
easier just to tune the OOM killer for the process? On one of my Pis
I had to do that for - wait for it - the NFS daemon. I suppose swap
would be too painful.

Maybe this should be mentioned in the docs.

--

73,
Ged.
Re: `cpan` unusable on small-RAM machines [ In reply to ]
On Mon, 13 Nov 2023 23:14:08 +0100, Paul "LeoNerd" Evans <leonerd@leonerd.org.uk> wrote:

> In more detail: I cannot install anything with the `cpan`. It gets
> OOM-killed.

I don't know if you're aware of this, but Andreas K?nig (andk) is available on IRC as klapperl and in general one of the nicest, and more importantly responsible and thoughtful people i know, whom you should find it very easy to talk to.

I'm sure if you report details to him he'd not disappoint.

--
With regards,
Christian Walde
Re: `cpan` unusable on small-RAM machines [ In reply to ]
On Mon, 13 Nov 2023 22:14:08 +0000
"Paul \"LeoNerd\" Evans" <leonerd@leonerd.org.uk> wrote:

> In more detail: I cannot install anything with the `cpan`. It gets
> OOM-killed.
>
> Or, more specifically, if I use
>
> $ cpan Some::Module
> or
> $ cpan
> cpan> install Some::Module
>
> Then it manages to download, unpack, build and test but then gets
> killed after the test phase but before the install phase.
>
> ...
> All tests successful.
> Files=1, Tests=32, 0 wallclock secs ( 0.13 usr 0.02 sys + 0.36
> cusr 0.08 csys = 0.59 CPU) Result: PASS
> Lockfile removed.
> Killed

On further investigation, there are other things that don't get
OOM-killed.

I can perfectly well

cpan> test Some::Module
cpan> notest install Some::Module

in two steps, and it will download/unpack/build/test the module in the
first stage, then install it from that pre-built location in the second
stage. No OOMdeath there.

But regardless of what module, a simple `install Some::Module` always
dies.

And on closer inspection, when it fails, it sits quite a while spinning
CPU and (presumably) burning memory. It get as far as printing
"Lockfile removed." nicely quickly, then waits aaaaages (over 10
seconds) before the OOM-killer kills it.

All tests successful.
Files=8, Tests=43, 3 wallclock secs ( 0.21 usr 0.02 sys + 2.94 cusr 0.34 csys = 3.51 CPU)
Result: PASS
Lockfile removed.
<== it waits for over 10 seconds here...
Killed


--
Paul "LeoNerd" Evans

leonerd@leonerd.org.uk | https://metacpan.org/author/PEVANS
http://www.leonerd.org.uk/ | https://www.tindie.com/stores/leonerd/