Hi all,
I would like to pay some attention on the following problem, which
keeps hurting me once a while. Thanks!
https://superuser.com/questions/1216484/how-to-debug-mod-perl-compilation-errors-during-start-and-reload-of-apache-httpd
I'm running Ubuntu 16.04 LTS with httpd 2.4 and mod_perl2 to host a
[PerlAuthenHandler][1] for some web app. It has been 14.04 LTS before
and was upgraded some weeks ago.
This works in general but sometimes, and especially during reloads and
restarts of httpd, mod_perl runs into problems during compilation of
my handler which leads to errors so that while httpd keeps running,
mod_perl is in a state not able to process anything anymore. As
mod_perl uses multiple interpreters during runtime and some of those
might be used for the first time on any request, compilation might
happen even without a reload/restart of the httpd service itself. But
that doesn't seem to be the case very often compared to the fact that
at least a reload of httpd happens once a day in the morning because
of log rotation.
Those compilation errors don't occur always, really only sometimes.
The handler might run well for some days and during reload of log
rotation it simply stops working. Or the same with restarting httpd:
In most cases this works fine as well, but sometimes compilation
errors in mod_perl already occur during such a restart. In most cases
those errors go away with an additional restart afterwards.
The errors don't happen in my own code directly, but always only in
libs I'm `use`ing, therefore during compilation time. Those libs are
not installed using CPAN or such, but by the package manger of my
distro instead, which should be the most "correct" and compatible
thing one could do. Of course those libs and my own code don't change
between working/not working cycles, there's really absolutely no
difference in anything installed, not even automatic security updates
or such. It just seems to stop working some day for some reason.
The errors itself are never runtime problems, but really always
regarding compilation of source code, like syntax errors in Perl code
and such. And of course the errors are not always the same, though it
seems that some of those occur more often than others.
Example 1:
> [Mon Jun 05 06:25:23.818862 2017] [core:notice] [pid 21652:tid 139994496042880] AH00094: Command line: '/usr/sbin/apache2'
> String found where operator expected at /usr/share/perl/5.22/Tie/Hash.pm line 25, near "croak "$pkg must define either a TIEHASH() or a new() method""
> (Do you need to predeclare croak?)
> String found where operator expected at /usr/share/perl/5.22/Tie/Hash.pm line 33, near "croak "$pkg doesn't define a TIEHASH method""
> (Do you need to predeclare croak?)
> String found where operator expected at /usr/share/perl/5.22/Tie/Hash.pm line 39, near "croak "$pkg doesn't define an EXISTS method""
> (Do you need to predeclare croak?)
> [Mon Jun 05 16:14:21.233643 2017] [mpm_event:notice] [pid 21652:tid 139994496042880] AH00491: caught SIGTERM, shutting down
Example 2:
> [Tue Jun 06 06:25:23.200424 2017] [core:notice] [pid 7051:tid 139688717936512] AH00094: Command line: '/usr/sbin/apache2'
> String found where operator expected at /usr/share/perl/5.22/File/Temp.pm line 199, near "croak 'Usage: ($fh, $name) = _gettemp($template, OPTIONS);'"
> (Do you need to predeclare croak?)
> String found where operator expected at /usr/share/perl/5.22/File/Temp.pm line 221, near "carp "File::Temp::_gettemp: template must not be a reference""
> (Do you need to predeclare carp?)
> String found where operator expected at /usr/share/perl/5.22/File/Temp.pm line 228, near "carp "File::Temp::_gettemp: Must have even number of options""
> (Do you need to predeclare carp?)
> String found where operator expected at /usr/share/perl/5.22/File/Temp.pm line 482, near "croak 'Usage: _replace_XX($template, $ignore)'"
> (Do you need to predeclare croak?)
> String found where operator expected at /usr/share/perl/5.22/File/Temp.pm line 808, near "croak 'Usage: _deferred_unlink($fh, $fname, $isdir)'"
> (Do you need to predeclare croak?)
> String found where operator expected at /usr/share/perl/5.22/File/Temp.pm line 835, near "carp "Request to remove directory $fname could not be completed since it does not exist!\n""
> (Do you need to predeclare carp?)
> String found where operator expected at /usr/share/perl/5.22/File/Temp.pm line 848, near "carp "Request to remove file $fname could not be completed since it is not there!\n""
> (Do you need to predeclare carp?)
> String found where operator expected at /usr/share/perl/5.22/File/Temp.pm line 993, near "croak "'tempfile' can't be called as a method""
> (Do you need to predeclare croak?)
> String found where operator expected at /usr/share/perl/5.22/File/Temp.pm line 1080, near "croak "Error in tempfile() using template $template: $errstr""
> (Do you need to predeclare croak?)
> String found where operator expected at /usr/share/perl/5.22/File/Temp.pm line 1111, near "croak "Error unlinking file $path using unlink0""
> [Tue Jun 06 07:54:18.220486 2017] [mpm_event:notice] [pid 7051:tid 139688717936512] AH00491: caught SIGTERM, shutting down
Example 3:
> [Tue Jun 06 11:17:40.789534 2017] [core:notice] [pid 4498:tid 139925979752320] AH00094: Command line: '/usr/sbin/apache2'
> String found where operator expected at /usr/share/perl/5.22/SelectSaver.pm line 10, near "croak 'usage: SelectSaver->new( [FILEHANDLE] )'"
> (Do you need to predeclare croak?)
> [Tue Jun 06 11:25:25.344061 2017] [mpm_event:notice] [pid 4498:tid 139925979752320] AH00491: caught SIGTERM, shutting down
Example 4:
> [Tue Jun 06 12:09:54.763670 2017] [core:notice] [pid 16855:tid 139853164218240] AH00094: Command line: '/usr/sbin/apache2'
> String found where operator expected at (eval 10) line 2, near "croak 'usage: $io->getline()'"
> (Do you need to predeclare croak?)
> String found where operator expected at (eval 10) line 8, near "croak 'usage: $io->getlines()'"
> (Do you need to predeclare croak?)
> String found where operator expected at (eval 10) line 10, near "croak 'Can\'t call $io->getlines in a scalar context, use $io->getline'"
> (Do you need to predeclare croak?)
> String found where operator expected at /usr/share/perl5/Config/IniFiles.pm line 400, near "carp "Invalid -import value \"$parms{-import}\" was ignored.""
> (Do you need to predeclare carp?)
> String found where operator expected at /usr/share/perl5/Config/IniFiles.pm line 446, near "carp "Comment character must be unique.""
> (Do you need to predeclare carp?)
> String found where operator expected at /usr/share/perl5/Config/IniFiles.pm line 451, near "carp "Illegal comment character.""
> (Do you need to predeclare carp?)
> String found where operator expected at /usr/share/perl5/Config/IniFiles.pm line 461, near "carp "Illegal value for -allowedcommentchars.""
> (Do you need to predeclare carp?)
> String found where operator expected at /usr/share/perl5/Config/IniFiles.pm line 482, near "carp "Unknown named parameter $k=>$v""
> (Do you need to predeclare carp?)
> String found where operator expected at /usr/share/perl5/Config/IniFiles.pm line 768, near "carp "Unhandled data structure in $ref, cannot _deepcopy()""
> (Do you need to predeclare carp?)
> String found where operator expected at /usr/share/perl5/Config/IniFiles.pm line 1273, near "carp "Failed to open $self->{cf}: $!""
> (Do you need to predeclare carp?)
> String found where operator expected at /usr/share/perl5/Config/IniFiles.pm line 1769, near "carp "Unable to rename temp config file ($new_file) to ${filename}: $!""
> (Do you need to predeclare carp?)
> [Tue Jun 06 12:10:23.827500 2017] [mpm_event:notice] [pid 16855:tid 139853164218240] AH00491: caught SIGTERM, shutting down
Sometimes the error even claims the line 4 in the following code of
`parent.pm` is wrong, because `$VERSION` is not defined:
> package parent;
> use strict;
> use vars qw($VERSION);
> $VERSION = '0.232';
>
> sub import {
So in the end, it seems to be some random stuff because of internal
errors in the mod_perl compiler or some wrong internal status or such.
Any ideas on how I can debug and find the root cause of the problem?
Or ideas on what the problem could actually be already?
[1]: https://perl.apache.org/docs/2.0/user/handlers/http.html#PerlAuthenHandler
Mit freundlichen Grüßen,
Thorsten Schöning
--
Thorsten Schöning E-Mail: Thorsten.Schoening@AM-SoFT.de
AM-SoFT IT-Systeme http://www.AM-SoFT.de/
Telefon...........05151- 9468- 55
Fax...............05151- 9468- 88
Mobil..............0178-8 9468- 04
AM-SoFT GmbH IT-Systeme, Brandenburger Str. 7c, 31789 Hameln
AG Hannover HRB 207 694 - Geschäftsführer: Andreas Muchow
I would like to pay some attention on the following problem, which
keeps hurting me once a while. Thanks!
https://superuser.com/questions/1216484/how-to-debug-mod-perl-compilation-errors-during-start-and-reload-of-apache-httpd
I'm running Ubuntu 16.04 LTS with httpd 2.4 and mod_perl2 to host a
[PerlAuthenHandler][1] for some web app. It has been 14.04 LTS before
and was upgraded some weeks ago.
This works in general but sometimes, and especially during reloads and
restarts of httpd, mod_perl runs into problems during compilation of
my handler which leads to errors so that while httpd keeps running,
mod_perl is in a state not able to process anything anymore. As
mod_perl uses multiple interpreters during runtime and some of those
might be used for the first time on any request, compilation might
happen even without a reload/restart of the httpd service itself. But
that doesn't seem to be the case very often compared to the fact that
at least a reload of httpd happens once a day in the morning because
of log rotation.
Those compilation errors don't occur always, really only sometimes.
The handler might run well for some days and during reload of log
rotation it simply stops working. Or the same with restarting httpd:
In most cases this works fine as well, but sometimes compilation
errors in mod_perl already occur during such a restart. In most cases
those errors go away with an additional restart afterwards.
The errors don't happen in my own code directly, but always only in
libs I'm `use`ing, therefore during compilation time. Those libs are
not installed using CPAN or such, but by the package manger of my
distro instead, which should be the most "correct" and compatible
thing one could do. Of course those libs and my own code don't change
between working/not working cycles, there's really absolutely no
difference in anything installed, not even automatic security updates
or such. It just seems to stop working some day for some reason.
The errors itself are never runtime problems, but really always
regarding compilation of source code, like syntax errors in Perl code
and such. And of course the errors are not always the same, though it
seems that some of those occur more often than others.
Example 1:
> [Mon Jun 05 06:25:23.818862 2017] [core:notice] [pid 21652:tid 139994496042880] AH00094: Command line: '/usr/sbin/apache2'
> String found where operator expected at /usr/share/perl/5.22/Tie/Hash.pm line 25, near "croak "$pkg must define either a TIEHASH() or a new() method""
> (Do you need to predeclare croak?)
> String found where operator expected at /usr/share/perl/5.22/Tie/Hash.pm line 33, near "croak "$pkg doesn't define a TIEHASH method""
> (Do you need to predeclare croak?)
> String found where operator expected at /usr/share/perl/5.22/Tie/Hash.pm line 39, near "croak "$pkg doesn't define an EXISTS method""
> (Do you need to predeclare croak?)
> [Mon Jun 05 16:14:21.233643 2017] [mpm_event:notice] [pid 21652:tid 139994496042880] AH00491: caught SIGTERM, shutting down
Example 2:
> [Tue Jun 06 06:25:23.200424 2017] [core:notice] [pid 7051:tid 139688717936512] AH00094: Command line: '/usr/sbin/apache2'
> String found where operator expected at /usr/share/perl/5.22/File/Temp.pm line 199, near "croak 'Usage: ($fh, $name) = _gettemp($template, OPTIONS);'"
> (Do you need to predeclare croak?)
> String found where operator expected at /usr/share/perl/5.22/File/Temp.pm line 221, near "carp "File::Temp::_gettemp: template must not be a reference""
> (Do you need to predeclare carp?)
> String found where operator expected at /usr/share/perl/5.22/File/Temp.pm line 228, near "carp "File::Temp::_gettemp: Must have even number of options""
> (Do you need to predeclare carp?)
> String found where operator expected at /usr/share/perl/5.22/File/Temp.pm line 482, near "croak 'Usage: _replace_XX($template, $ignore)'"
> (Do you need to predeclare croak?)
> String found where operator expected at /usr/share/perl/5.22/File/Temp.pm line 808, near "croak 'Usage: _deferred_unlink($fh, $fname, $isdir)'"
> (Do you need to predeclare croak?)
> String found where operator expected at /usr/share/perl/5.22/File/Temp.pm line 835, near "carp "Request to remove directory $fname could not be completed since it does not exist!\n""
> (Do you need to predeclare carp?)
> String found where operator expected at /usr/share/perl/5.22/File/Temp.pm line 848, near "carp "Request to remove file $fname could not be completed since it is not there!\n""
> (Do you need to predeclare carp?)
> String found where operator expected at /usr/share/perl/5.22/File/Temp.pm line 993, near "croak "'tempfile' can't be called as a method""
> (Do you need to predeclare croak?)
> String found where operator expected at /usr/share/perl/5.22/File/Temp.pm line 1080, near "croak "Error in tempfile() using template $template: $errstr""
> (Do you need to predeclare croak?)
> String found where operator expected at /usr/share/perl/5.22/File/Temp.pm line 1111, near "croak "Error unlinking file $path using unlink0""
> [Tue Jun 06 07:54:18.220486 2017] [mpm_event:notice] [pid 7051:tid 139688717936512] AH00491: caught SIGTERM, shutting down
Example 3:
> [Tue Jun 06 11:17:40.789534 2017] [core:notice] [pid 4498:tid 139925979752320] AH00094: Command line: '/usr/sbin/apache2'
> String found where operator expected at /usr/share/perl/5.22/SelectSaver.pm line 10, near "croak 'usage: SelectSaver->new( [FILEHANDLE] )'"
> (Do you need to predeclare croak?)
> [Tue Jun 06 11:25:25.344061 2017] [mpm_event:notice] [pid 4498:tid 139925979752320] AH00491: caught SIGTERM, shutting down
Example 4:
> [Tue Jun 06 12:09:54.763670 2017] [core:notice] [pid 16855:tid 139853164218240] AH00094: Command line: '/usr/sbin/apache2'
> String found where operator expected at (eval 10) line 2, near "croak 'usage: $io->getline()'"
> (Do you need to predeclare croak?)
> String found where operator expected at (eval 10) line 8, near "croak 'usage: $io->getlines()'"
> (Do you need to predeclare croak?)
> String found where operator expected at (eval 10) line 10, near "croak 'Can\'t call $io->getlines in a scalar context, use $io->getline'"
> (Do you need to predeclare croak?)
> String found where operator expected at /usr/share/perl5/Config/IniFiles.pm line 400, near "carp "Invalid -import value \"$parms{-import}\" was ignored.""
> (Do you need to predeclare carp?)
> String found where operator expected at /usr/share/perl5/Config/IniFiles.pm line 446, near "carp "Comment character must be unique.""
> (Do you need to predeclare carp?)
> String found where operator expected at /usr/share/perl5/Config/IniFiles.pm line 451, near "carp "Illegal comment character.""
> (Do you need to predeclare carp?)
> String found where operator expected at /usr/share/perl5/Config/IniFiles.pm line 461, near "carp "Illegal value for -allowedcommentchars.""
> (Do you need to predeclare carp?)
> String found where operator expected at /usr/share/perl5/Config/IniFiles.pm line 482, near "carp "Unknown named parameter $k=>$v""
> (Do you need to predeclare carp?)
> String found where operator expected at /usr/share/perl5/Config/IniFiles.pm line 768, near "carp "Unhandled data structure in $ref, cannot _deepcopy()""
> (Do you need to predeclare carp?)
> String found where operator expected at /usr/share/perl5/Config/IniFiles.pm line 1273, near "carp "Failed to open $self->{cf}: $!""
> (Do you need to predeclare carp?)
> String found where operator expected at /usr/share/perl5/Config/IniFiles.pm line 1769, near "carp "Unable to rename temp config file ($new_file) to ${filename}: $!""
> (Do you need to predeclare carp?)
> [Tue Jun 06 12:10:23.827500 2017] [mpm_event:notice] [pid 16855:tid 139853164218240] AH00491: caught SIGTERM, shutting down
Sometimes the error even claims the line 4 in the following code of
`parent.pm` is wrong, because `$VERSION` is not defined:
> package parent;
> use strict;
> use vars qw($VERSION);
> $VERSION = '0.232';
>
> sub import {
So in the end, it seems to be some random stuff because of internal
errors in the mod_perl compiler or some wrong internal status or such.
Any ideas on how I can debug and find the root cause of the problem?
Or ideas on what the problem could actually be already?
[1]: https://perl.apache.org/docs/2.0/user/handlers/http.html#PerlAuthenHandler
Mit freundlichen Grüßen,
Thorsten Schöning
--
Thorsten Schöning E-Mail: Thorsten.Schoening@AM-SoFT.de
AM-SoFT IT-Systeme http://www.AM-SoFT.de/
Telefon...........05151- 9468- 55
Fax...............05151- 9468- 88
Mobil..............0178-8 9468- 04
AM-SoFT GmbH IT-Systeme, Brandenburger Str. 7c, 31789 Hameln
AG Hannover HRB 207 694 - Geschäftsführer: Andreas Muchow