Hi,
TL;DR: mod_perl is using perl_parse() incorrectly and not NULL-terminating the argv array passed to it.
While setting up a perl web application with mod_perl & apache, apache kept segfaulting.
Broke out gdb, and found that it was segfaulting within perl itself:
Program received signal SIGSEGV, Segmentation fault.
0x00007ffff7358ff5 in perl_parse () from /lib/x86_64-linux-gnu/libperl.so.5.30
(gdb) bt
#0 0x00007ffff7358ff5 in perl_parse () from /lib/x86_64-linux-gnu/libperl.so.5.30
#1 0x00007ffff764cd0c in modperl_startup () from /usr/lib/apache2/modules/mod_perl.so
#2 0x00007ffff764cc97 in modperl_startup () from /usr/lib/apache2/modules/mod_perl.so
#3 0x00007ffff764d0fa in modperl_init () from /usr/lib/apache2/modules/mod_perl.so
#4 0x00007ffff764d27b in modperl_hook_init () from /usr/lib/apache2/modules/mod_perl.so
#5 0x00005555555b23d4 in ap_run_open_logs ()
#6 0x000055555558c440 in main ()
# valgrind apache2 -k start -X
==22529== Memcheck, a memory error detector
==22529== Copyright (C) 2002-2017, and GNU GPL'd, by Julian Seward et al.
==22529== Using Valgrind-3.15.0 and LibVEX; rerun with -h for copyright info
==22529== Command: apache2 -k start -X
==22529==
==22529== Invalid read of size 8
==22529== at 0x564AFF5: perl_parse (in /usr/lib/x86_64-linux-gnu/libperl.so.5.30.0)
==22529== by 0x55A8D0B: modperl_startup (in /usr/lib/apache2/modules/mod_perl.so)
==22529== by 0x55A8C96: modperl_startup (in /usr/lib/apache2/modules/mod_perl.so)
==22529== by 0x55A90F9: modperl_init (in /usr/lib/apache2/modules/mod_perl.so)
==22529== by 0x55A927A: modperl_hook_init (in /usr/lib/apache2/modules/mod_perl.so)
==22529== by 0x1663D3: ap_run_open_logs (in /usr/sbin/apache2)
==22529== by 0x14043F: main (in /usr/sbin/apache2)
==22529== Address 0x5a44000 is not stack'd, malloc'd or (recently) free'd
==22529==
==22529==
==22529== Process terminating with default action of signal 11 (SIGSEGV)
==22529== Access not within mapped region at address 0x5A44000
==22529== at 0x564AFF5: perl_parse (in /usr/lib/x86_64-linux-gnu/libperl.so.5.30.0)
==22529== by 0x55A8D0B: modperl_startup (in /usr/lib/apache2/modules/mod_perl.so)
==22529== by 0x55A8C96: modperl_startup (in /usr/lib/apache2/modules/mod_perl.so)
==22529== by 0x55A90F9: modperl_init (in /usr/lib/apache2/modules/mod_perl.so)
==22529== by 0x55A927A: modperl_hook_init (in /usr/lib/apache2/modules/mod_perl.so)
==22529== by 0x1663D3: ap_run_open_logs (in /usr/sbin/apache2)
==22529== by 0x14043F: main (in /usr/sbin/apache2)
When using debug symbols, gdb indicated that it was erroring in very early in perl's runtime before it had got to any perl code - the exact line it was failing on was perl.c:2365 scriptname = argv[0];. It wasn't possible to reason beyond that as stepping through optimised code even with debug symbols is next to impossible to make any sense of.
I did find that building perl without optimisations made the error go away.
I found the following closed issue: https://github.com/Perl/perl5/issues/15806 which describes the same issue I was having.
Looking at the source for mod_perl, I found that the argv array passed to perl_parse() is not NULL terminated as is required by perl - ( documentation: https://perldoc.perl.org/perlembed#Adding-a-Perl-interpreter-to-your-C-program )
As for why this works in 99.99% of cases, I have no idea. With the set up I have (Ubuntu 20.04 LXD container) it is very reproducible initially, but after I fiddle a bit (uninstall & reinstall packages, install debugging packages, etc etc) it stops being reproducible. Such is undefined behaviour and invalid memory accesses, I suppose, but quite irritating.
I've attached my suggested patch.
See also Ubuntu bug report - https://bugs.launchpad.net/ubuntu/+source/libapache2-mod-perl2/+bug/1915959
I don't believe it's strictly relevant, but package versions are:
apache2: 2.4.41-4ubuntu3.1
libapache2-mod-perl2: 2.0.11-2
perl: 5.30.0-9ubuntu0.2
Thanks,
Charles
--
Register for our online DO-178C & Multicore training with ConsuNova: 8-12th March 2021<https://www.rapitasystems.com/training/DO178C/?utm_source=email_footer>.
TL;DR: mod_perl is using perl_parse() incorrectly and not NULL-terminating the argv array passed to it.
While setting up a perl web application with mod_perl & apache, apache kept segfaulting.
Broke out gdb, and found that it was segfaulting within perl itself:
Program received signal SIGSEGV, Segmentation fault.
0x00007ffff7358ff5 in perl_parse () from /lib/x86_64-linux-gnu/libperl.so.5.30
(gdb) bt
#0 0x00007ffff7358ff5 in perl_parse () from /lib/x86_64-linux-gnu/libperl.so.5.30
#1 0x00007ffff764cd0c in modperl_startup () from /usr/lib/apache2/modules/mod_perl.so
#2 0x00007ffff764cc97 in modperl_startup () from /usr/lib/apache2/modules/mod_perl.so
#3 0x00007ffff764d0fa in modperl_init () from /usr/lib/apache2/modules/mod_perl.so
#4 0x00007ffff764d27b in modperl_hook_init () from /usr/lib/apache2/modules/mod_perl.so
#5 0x00005555555b23d4 in ap_run_open_logs ()
#6 0x000055555558c440 in main ()
# valgrind apache2 -k start -X
==22529== Memcheck, a memory error detector
==22529== Copyright (C) 2002-2017, and GNU GPL'd, by Julian Seward et al.
==22529== Using Valgrind-3.15.0 and LibVEX; rerun with -h for copyright info
==22529== Command: apache2 -k start -X
==22529==
==22529== Invalid read of size 8
==22529== at 0x564AFF5: perl_parse (in /usr/lib/x86_64-linux-gnu/libperl.so.5.30.0)
==22529== by 0x55A8D0B: modperl_startup (in /usr/lib/apache2/modules/mod_perl.so)
==22529== by 0x55A8C96: modperl_startup (in /usr/lib/apache2/modules/mod_perl.so)
==22529== by 0x55A90F9: modperl_init (in /usr/lib/apache2/modules/mod_perl.so)
==22529== by 0x55A927A: modperl_hook_init (in /usr/lib/apache2/modules/mod_perl.so)
==22529== by 0x1663D3: ap_run_open_logs (in /usr/sbin/apache2)
==22529== by 0x14043F: main (in /usr/sbin/apache2)
==22529== Address 0x5a44000 is not stack'd, malloc'd or (recently) free'd
==22529==
==22529==
==22529== Process terminating with default action of signal 11 (SIGSEGV)
==22529== Access not within mapped region at address 0x5A44000
==22529== at 0x564AFF5: perl_parse (in /usr/lib/x86_64-linux-gnu/libperl.so.5.30.0)
==22529== by 0x55A8D0B: modperl_startup (in /usr/lib/apache2/modules/mod_perl.so)
==22529== by 0x55A8C96: modperl_startup (in /usr/lib/apache2/modules/mod_perl.so)
==22529== by 0x55A90F9: modperl_init (in /usr/lib/apache2/modules/mod_perl.so)
==22529== by 0x55A927A: modperl_hook_init (in /usr/lib/apache2/modules/mod_perl.so)
==22529== by 0x1663D3: ap_run_open_logs (in /usr/sbin/apache2)
==22529== by 0x14043F: main (in /usr/sbin/apache2)
When using debug symbols, gdb indicated that it was erroring in very early in perl's runtime before it had got to any perl code - the exact line it was failing on was perl.c:2365 scriptname = argv[0];. It wasn't possible to reason beyond that as stepping through optimised code even with debug symbols is next to impossible to make any sense of.
I did find that building perl without optimisations made the error go away.
I found the following closed issue: https://github.com/Perl/perl5/issues/15806 which describes the same issue I was having.
Looking at the source for mod_perl, I found that the argv array passed to perl_parse() is not NULL terminated as is required by perl - ( documentation: https://perldoc.perl.org/perlembed#Adding-a-Perl-interpreter-to-your-C-program )
As for why this works in 99.99% of cases, I have no idea. With the set up I have (Ubuntu 20.04 LXD container) it is very reproducible initially, but after I fiddle a bit (uninstall & reinstall packages, install debugging packages, etc etc) it stops being reproducible. Such is undefined behaviour and invalid memory accesses, I suppose, but quite irritating.
I've attached my suggested patch.
See also Ubuntu bug report - https://bugs.launchpad.net/ubuntu/+source/libapache2-mod-perl2/+bug/1915959
I don't believe it's strictly relevant, but package versions are:
apache2: 2.4.41-4ubuntu3.1
libapache2-mod-perl2: 2.0.11-2
perl: 5.30.0-9ubuntu0.2
Thanks,
Charles
--
Register for our online DO-178C & Multicore training with ConsuNova: 8-12th March 2021<https://www.rapitasystems.com/training/DO178C/?utm_source=email_footer>.