https://bz.apache.org/bugzilla/show_bug.cgi?id=67912
Bug ID: 67912
Summary: httpd segmentation violation (or other strange
symptoms) after OS library updates
Product: Apache httpd-2
Version: 2.4.57
Hardware: PC
OS: Linux
Status: NEW
Severity: critical
Priority: P2
Component: All
Assignee: bugs@httpd.apache.org
Reporter: alex.nishri@utoronto.ca
Target Milestone: ---
After system updates on hundreds of Linux servers we operate, followed by
graceful httpd restart, many servers experienced httpd segment violation fault,
while other servers experienced strange httpd appliction symptoms, all
requiring an Apache httpd reload to correct. (We saw similar problems a year
ago when something else was updated, but this ticket focuses on the current
instance.)
On Oct 21, 2023 early morning, on hundreds of CentOS 7 servers we manage,
anacron running /etc/cron.daily invoked /usr/sbin/yum-cron, which installed 22
updates, followed by /usr/sbin/logrotate, which rotated the httpd logs and
requested Apache httpd graceful restart.
On many servers, the result was the parent httpd process was killed with a SEGV
signal. On all such servers, httpd "start" was successful to resume service.
On other servers we saw other strange symptoms. e.g. on one service httpd
stayed up, accessing the service URL lead to correct redirection to Shibboleth
to authenticate people, but after correct redirection from Shibboleth back to
service application, the authenticated credentials were not recognized--this
problem onset correlates to /etc/cron.daily running, was not corrected by httpd
graceful reload, and was successfully corrected by httpd stop/start.
On samples of httpd instances that did not crash, but showed other symptoms, we
examined open file handles. Following the yum updates, the only open files that
the morning update had changed were to file /usr/lib64/libsoftokn3.so and to
file /usr/lib64/libfreeblpriv3.so, which are included in that morning's yum
update of nss-softokn-3.90.0-6.el7_9.x86_64.
We observed that the parent httpd process had an open file handle to the
deleted (and hence not updated version of) file /usr/lib64/libfreebl3.so, but
did not have an open handle to file /usr/lib64/libsoftokn3.so.
And that all the httpd child processes had an open file handle to the deleted
(and hence not updated version of) file /usr/lib64/libfreebl3.so, and an open
file to current (and hence updated version of) file /usr/lib64/libsoftokn3.so.
A hypothesis is that Apache is invoking routines inside an old version of
libfreebl3.so, and a new version of libsoftokn3.so, and that this leads to the
observed problems. We have not examined the Apache 2.4.57 or Network Security
Services (NSS) 3.92 sources to verify this hypothesis. However, we see no other
httpd parent or child process open files that were changed by the Oct 21
updates leading to these symptoms.
Other Linux daemons survive such updates, and it would be appreciated if Apache
also did so.
We note that in the recent Apache "Bug 67787 - httpd fails to gracefully
restart after update in CentOS 7", which seems to be about a different update
leading to different httpd problem symptoms, it says, "In Fedora/RHEL/CentOS
the httpd package will do a (full) restart of httpd.service after updating the
package, unless you've manually disabled that." In light of this statement, can
you clarify what component was already configured to perform an httpd reload
when nss-softokn-3.90.0-6.el7_9.x86_64 was updated ?
As a bypass to this problem (which happens "randomly" about once a year) we
plan to put in code, probably using the "needs-restarting" Linux program, to
detect this Apache issue, and perform a non-graceful restart. But we would much
prefer a solution where Apache behaves better following OS library updates.
--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org
Bug ID: 67912
Summary: httpd segmentation violation (or other strange
symptoms) after OS library updates
Product: Apache httpd-2
Version: 2.4.57
Hardware: PC
OS: Linux
Status: NEW
Severity: critical
Priority: P2
Component: All
Assignee: bugs@httpd.apache.org
Reporter: alex.nishri@utoronto.ca
Target Milestone: ---
After system updates on hundreds of Linux servers we operate, followed by
graceful httpd restart, many servers experienced httpd segment violation fault,
while other servers experienced strange httpd appliction symptoms, all
requiring an Apache httpd reload to correct. (We saw similar problems a year
ago when something else was updated, but this ticket focuses on the current
instance.)
On Oct 21, 2023 early morning, on hundreds of CentOS 7 servers we manage,
anacron running /etc/cron.daily invoked /usr/sbin/yum-cron, which installed 22
updates, followed by /usr/sbin/logrotate, which rotated the httpd logs and
requested Apache httpd graceful restart.
On many servers, the result was the parent httpd process was killed with a SEGV
signal. On all such servers, httpd "start" was successful to resume service.
On other servers we saw other strange symptoms. e.g. on one service httpd
stayed up, accessing the service URL lead to correct redirection to Shibboleth
to authenticate people, but after correct redirection from Shibboleth back to
service application, the authenticated credentials were not recognized--this
problem onset correlates to /etc/cron.daily running, was not corrected by httpd
graceful reload, and was successfully corrected by httpd stop/start.
On samples of httpd instances that did not crash, but showed other symptoms, we
examined open file handles. Following the yum updates, the only open files that
the morning update had changed were to file /usr/lib64/libsoftokn3.so and to
file /usr/lib64/libfreeblpriv3.so, which are included in that morning's yum
update of nss-softokn-3.90.0-6.el7_9.x86_64.
We observed that the parent httpd process had an open file handle to the
deleted (and hence not updated version of) file /usr/lib64/libfreebl3.so, but
did not have an open handle to file /usr/lib64/libsoftokn3.so.
And that all the httpd child processes had an open file handle to the deleted
(and hence not updated version of) file /usr/lib64/libfreebl3.so, and an open
file to current (and hence updated version of) file /usr/lib64/libsoftokn3.so.
A hypothesis is that Apache is invoking routines inside an old version of
libfreebl3.so, and a new version of libsoftokn3.so, and that this leads to the
observed problems. We have not examined the Apache 2.4.57 or Network Security
Services (NSS) 3.92 sources to verify this hypothesis. However, we see no other
httpd parent or child process open files that were changed by the Oct 21
updates leading to these symptoms.
Other Linux daemons survive such updates, and it would be appreciated if Apache
also did so.
We note that in the recent Apache "Bug 67787 - httpd fails to gracefully
restart after update in CentOS 7", which seems to be about a different update
leading to different httpd problem symptoms, it says, "In Fedora/RHEL/CentOS
the httpd package will do a (full) restart of httpd.service after updating the
package, unless you've manually disabled that." In light of this statement, can
you clarify what component was already configured to perform an httpd reload
when nss-softokn-3.90.0-6.el7_9.x86_64 was updated ?
As a bypass to this problem (which happens "randomly" about once a year) we
plan to put in code, probably using the "needs-restarting" Linux program, to
detect this Apache issue, and perform a non-graceful restart. But we would much
prefer a solution where Apache behaves better following OS library updates.
--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org