Mailing List Archive

Httpd is hanging intermittently
Hi All,
We are using httpd version 2.4.46 and its working fine for a long time.
But recently, we started seeing an issue where apache hangs indefinitely
even when the system is in idle state.
And when apache hangs, I see below entries in error_log:
[Tue Sep 21 22:05:53.243013 2021] [ssl:warn] [pid 5769:tid 2644435888]
AH01990: Server: PRNG still contains insufficient entropy!
[Tue Sep 21 22:05:54.501476 2021] [ssl:warn] [pid 5769:tid 2787111856]
AH01990: Server: PRNG still contains insufficient entropy!
[Tue Sep 21 22:05:54.502449 2021] [ssl:warn] [pid 5769:tid 2787111856]
AH01990: Server: PRNG still contains insufficient entropy!
...
....
....

I am pretty sure, we not changed anything related to httpd config for quite
a time time and have no idea, why this issue started getting manifested now.
Please help me how to RC this and what logs can be looked to debug further?

PS: Occurence of issue is more in systems where FIPS is enabled. In FIPS
disabled systems, occurrence is less.

With Regards
Venkat
Re: Httpd is hanging intermittently [ In reply to ]
On Wed, Sep 22, 2021 at 8:12 PM alchemist vk <alchemist.vk@gmail.com> wrote:
>
> I am pretty sure, we not changed anything related to httpd config for quite a time time and have no idea, why this issue started getting manifested now.

Which operating system and openssl version are you using? Did you
upgrade openssl recently?
What are your SSLRandomSeed settings?


Regards;
Yann.

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org
Re: Httpd is hanging intermittently [ In reply to ]
I don't think "insufficient entropy" has anything to do with Apache, but
you could try installing "haveged" rpm.
That may solve your problem.

On Wed, Sep 22, 2021 at 2:11 PM alchemist vk <alchemist.vk@gmail.com> wrote:

> Hi All,
> We are using httpd version 2.4.46 and its working fine for a long time.
> But recently, we started seeing an issue where apache hangs indefinitely
> even when the system is in idle state.
> And when apache hangs, I see below entries in error_log:
> [Tue Sep 21 22:05:53.243013 2021] [ssl:warn] [pid 5769:tid 2644435888]
> AH01990: Server: PRNG still contains insufficient entropy!
> [Tue Sep 21 22:05:54.501476 2021] [ssl:warn] [pid 5769:tid 2787111856]
> AH01990: Server: PRNG still contains insufficient entropy!
> [Tue Sep 21 22:05:54.502449 2021] [ssl:warn] [pid 5769:tid 2787111856]
> AH01990: Server: PRNG still contains insufficient entropy!
> ...
> ....
> ....
>
> I am pretty sure, we not changed anything related to httpd config for
> quite a time time and have no idea, why this issue started getting
> manifested now.
> Please help me how to RC this and what logs can be looked to debug further?
>
> PS: Occurence of issue is more in systems where FIPS is enabled. In FIPS
> disabled systems, occurrence is less.
>
> With Regards
> Venkat
>
>
>
>
>
Re: Httpd is hanging intermittently [ In reply to ]
Thanks *Jon *for openssl command confirmation.
*@ylavik*,
Its linux OS and openssl version is 1.1.1k-fips. I not yet explored
with SSLRandomSeed changes.
Yes, we upgraded openssl few months back to 1.1.1k, but we are seeing
this httpd hangs issue from last month.

*@otis Dewitt*, Since its production code in systems, I cant install
haveged and try it out.


On Thu, Sep 23, 2021 at 4:57 AM Otis Dewitt - NOAA Affiliate
<otis.dewitt@noaa.gov.invalid> wrote:

>
> I don't think "insufficient entropy" has anything to do with Apache, but
> you could try installing "haveged" rpm.
> That may solve your problem.
>
> On Wed, Sep 22, 2021 at 2:11 PM alchemist vk <alchemist.vk@gmail.com>
> wrote:
>
>> Hi All,
>> We are using httpd version 2.4.46 and its working fine for a long time.
>> But recently, we started seeing an issue where apache hangs indefinitely
>> even when the system is in idle state.
>> And when apache hangs, I see below entries in error_log:
>> [Tue Sep 21 22:05:53.243013 2021] [ssl:warn] [pid 5769:tid 2644435888]
>> AH01990: Server: PRNG still contains insufficient entropy!
>> [Tue Sep 21 22:05:54.501476 2021] [ssl:warn] [pid 5769:tid 2787111856]
>> AH01990: Server: PRNG still contains insufficient entropy!
>> [Tue Sep 21 22:05:54.502449 2021] [ssl:warn] [pid 5769:tid 2787111856]
>> AH01990: Server: PRNG still contains insufficient entropy!
>> ...
>> ....
>> ....
>>
>> I am pretty sure, we not changed anything related to httpd config for
>> quite a time time and have no idea, why this issue started getting
>> manifested now.
>> Please help me how to RC this and what logs can be looked to debug
>> further?
>>
>> PS: Occurence of issue is more in systems where FIPS is enabled. In FIPS
>> disabled systems, occurrence is less.
>>
>> With Regards
>> Venkat
>>
>>
>>
>>
>>
Re: Httpd is hanging intermittently [ In reply to ]
Hmm I see, I not sure why you did not get this right away when switching
from openssl to openssl-fips because FIPS require a lot of entropy
and if this is on VMWARE, that has very poor entropy unless you use entropy
generator like "*haveged*" or load *virtio_rng *kernel module.
As I said before I am not sure how you will fix this without generating
more entropy, it seems the system is unable to create enough and
there is no way around this.


On Thu, Sep 23, 2021 at 1:15 AM alchemist vk <alchemist.vk@gmail.com> wrote:

> Thanks *Jon *for openssl command confirmation.
> *@ylavik*,
> Its linux OS and openssl version is 1.1.1k-fips. I not yet explored
> with SSLRandomSeed changes.
> Yes, we upgraded openssl few months back to 1.1.1k, but we are seeing
> this httpd hangs issue from last month.
>
> *@otis Dewitt*, Since its production code in systems, I cant install
> haveged and try it out.
>
>
> On Thu, Sep 23, 2021 at 4:57 AM Otis Dewitt - NOAA Affiliate
> <otis.dewitt@noaa.gov.invalid> wrote:
>
>>
>> I don't think "insufficient entropy" has anything to do with Apache, but
>> you could try installing "haveged" rpm.
>> That may solve your problem.
>>
>> On Wed, Sep 22, 2021 at 2:11 PM alchemist vk <alchemist.vk@gmail.com>
>> wrote:
>>
>>> Hi All,
>>> We are using httpd version 2.4.46 and its working fine for a long time.
>>> But recently, we started seeing an issue where apache hangs indefinitely
>>> even when the system is in idle state.
>>> And when apache hangs, I see below entries in error_log:
>>> [Tue Sep 21 22:05:53.243013 2021] [ssl:warn] [pid 5769:tid 2644435888]
>>> AH01990: Server: PRNG still contains insufficient entropy!
>>> [Tue Sep 21 22:05:54.501476 2021] [ssl:warn] [pid 5769:tid 2787111856]
>>> AH01990: Server: PRNG still contains insufficient entropy!
>>> [Tue Sep 21 22:05:54.502449 2021] [ssl:warn] [pid 5769:tid 2787111856]
>>> AH01990: Server: PRNG still contains insufficient entropy!
>>> ...
>>> ....
>>> ....
>>>
>>> I am pretty sure, we not changed anything related to httpd config for
>>> quite a time time and have no idea, why this issue started getting
>>> manifested now.
>>> Please help me how to RC this and what logs can be looked to debug
>>> further?
>>>
>>> PS: Occurence of issue is more in systems where FIPS is enabled. In FIPS
>>> disabled systems, occurrence is less.
>>>
>>> With Regards
>>> Venkat
>>>
>>>
>>>
>>>
>>>
Re: Httpd is hanging intermittently [ In reply to ]
Thanks Dewitt for your inputs.
Will check from system perspective how to generate more entropy and resolve
this issue.

Do you know, how to generate more entropy in system or via apache so that
it can never be deprived of entropy?

With Regards,
Venkatesh

On Thu, Sep 23, 2021 at 8:46 PM Otis Dewitt - NOAA Affiliate
<otis.dewitt@noaa.gov.invalid> wrote:

> Hmm I see, I not sure why you did not get this right away when switching
> from openssl to openssl-fips because FIPS require a lot of entropy
> and if this is on VMWARE, that has very poor entropy unless you use
> entropy generator like "*haveged*" or load *virtio_rng *kernel module.
> As I said before I am not sure how you will fix this without generating
> more entropy, it seems the system is unable to create enough and
> there is no way around this.
>
>
> On Thu, Sep 23, 2021 at 1:15 AM alchemist vk <alchemist.vk@gmail.com>
> wrote:
>
>> Thanks *Jon *for openssl command confirmation.
>> *@ylavik*,
>> Its linux OS and openssl version is 1.1.1k-fips. I not yet explored
>> with SSLRandomSeed changes.
>> Yes, we upgraded openssl few months back to 1.1.1k, but we are
>> seeing this httpd hangs issue from last month.
>>
>> *@otis Dewitt*, Since its production code in systems, I cant install
>> haveged and try it out.
>>
>>
>> On Thu, Sep 23, 2021 at 4:57 AM Otis Dewitt - NOAA Affiliate
>> <otis.dewitt@noaa.gov.invalid> wrote:
>>
>>>
>>> I don't think "insufficient entropy" has anything to do with Apache, but
>>> you could try installing "haveged" rpm.
>>> That may solve your problem.
>>>
>>> On Wed, Sep 22, 2021 at 2:11 PM alchemist vk <alchemist.vk@gmail.com>
>>> wrote:
>>>
>>>> Hi All,
>>>> We are using httpd version 2.4.46 and its working fine for a long
>>>> time. But recently, we started seeing an issue where apache hangs
>>>> indefinitely even when the system is in idle state.
>>>> And when apache hangs, I see below entries in error_log:
>>>> [Tue Sep 21 22:05:53.243013 2021] [ssl:warn] [pid 5769:tid 2644435888]
>>>> AH01990: Server: PRNG still contains insufficient entropy!
>>>> [Tue Sep 21 22:05:54.501476 2021] [ssl:warn] [pid 5769:tid 2787111856]
>>>> AH01990: Server: PRNG still contains insufficient entropy!
>>>> [Tue Sep 21 22:05:54.502449 2021] [ssl:warn] [pid 5769:tid 2787111856]
>>>> AH01990: Server: PRNG still contains insufficient entropy!
>>>> ...
>>>> ....
>>>> ....
>>>>
>>>> I am pretty sure, we not changed anything related to httpd config for
>>>> quite a time time and have no idea, why this issue started getting
>>>> manifested now.
>>>> Please help me how to RC this and what logs can be looked to debug
>>>> further?
>>>>
>>>> PS: Occurence of issue is more in systems where FIPS is enabled. In
>>>> FIPS disabled systems, occurrence is less.
>>>>
>>>> With Regards
>>>> Venkat
>>>>
>>>>
>>>>
>>>>
>>>>
Re: Httpd is hanging intermittently [ In reply to ]
No problem Venkatesh.

No, I don't know how to generate entropy in Apache because I think Apache
uses the system entropy.
You can check how many are available via: "cat
/proc/sys/kernel/random/entropy_avail".

Under the system I know of two different packages, one *rngd *and the other
*haveged.*

The *rngd* daemon, which is a part of the rng-tools package, is capable of
using both environmental noise and hardware random number generators for
extracting entropy. The daemon checks whether the data supplied by the
source of randomness is sufficiently random and then stores it in the
kernel's random-number entropy pool. The random numbers it generates are
made available through the /dev/random and /dev/urandom character devices.

The *haveged *project is an attempt to provide an easy-to-use,
unpredictable random number generator based upon an adaptation of the HAVEGE
<http://www.irisa.fr/caps/projects/hipsor/> algorithm. Haveged was created
to remedy low-entropy conditions in the Linux random device that can occur
under some workloads, especially on headless servers. Current development
of haveged is directed towards improving overall reliability and
adaptability while minimizing the barriers to using haveged for other tasks.

What OS are you using? Redhat CentOS etc . . .


On Thu, Sep 23, 2021 at 2:06 PM alchemist vk <alchemist.vk@gmail.com> wrote:

> Thanks Dewitt for your inputs.
> Will check from system perspective how to generate more entropy and
> resolve this issue.
>
> Do you know, how to generate more entropy in system or via apache so that
> it can never be deprived of entropy?
>
> With Regards,
> Venkatesh
>
> On Thu, Sep 23, 2021 at 8:46 PM Otis Dewitt - NOAA Affiliate
> <otis.dewitt@noaa.gov.invalid> wrote:
>
>> Hmm I see, I not sure why you did not get this right away when switching
>> from openssl to openssl-fips because FIPS require a lot of entropy
>> and if this is on VMWARE, that has very poor entropy unless you use
>> entropy generator like "*haveged*" or load *virtio_rng *kernel module.
>> As I said before I am not sure how you will fix this without generating
>> more entropy, it seems the system is unable to create enough and
>> there is no way around this.
>>
>>
>> On Thu, Sep 23, 2021 at 1:15 AM alchemist vk <alchemist.vk@gmail.com>
>> wrote:
>>
>>> Thanks *Jon *for openssl command confirmation.
>>> *@ylavik*,
>>> Its linux OS and openssl version is 1.1.1k-fips. I not yet explored
>>> with SSLRandomSeed changes.
>>> Yes, we upgraded openssl few months back to 1.1.1k, but we are
>>> seeing this httpd hangs issue from last month.
>>>
>>> *@otis Dewitt*, Since its production code in systems, I cant install
>>> haveged and try it out.
>>>
>>>
>>> On Thu, Sep 23, 2021 at 4:57 AM Otis Dewitt - NOAA Affiliate
>>> <otis.dewitt@noaa.gov.invalid> wrote:
>>>
>>>>
>>>> I don't think "insufficient entropy" has anything to do with Apache,
>>>> but you could try installing "haveged" rpm.
>>>> That may solve your problem.
>>>>
>>>> On Wed, Sep 22, 2021 at 2:11 PM alchemist vk <alchemist.vk@gmail.com>
>>>> wrote:
>>>>
>>>>> Hi All,
>>>>> We are using httpd version 2.4.46 and its working fine for a long
>>>>> time. But recently, we started seeing an issue where apache hangs
>>>>> indefinitely even when the system is in idle state.
>>>>> And when apache hangs, I see below entries in error_log:
>>>>> [Tue Sep 21 22:05:53.243013 2021] [ssl:warn] [pid 5769:tid 2644435888]
>>>>> AH01990: Server: PRNG still contains insufficient entropy!
>>>>> [Tue Sep 21 22:05:54.501476 2021] [ssl:warn] [pid 5769:tid 2787111856]
>>>>> AH01990: Server: PRNG still contains insufficient entropy!
>>>>> [Tue Sep 21 22:05:54.502449 2021] [ssl:warn] [pid 5769:tid 2787111856]
>>>>> AH01990: Server: PRNG still contains insufficient entropy!
>>>>> ...
>>>>> ....
>>>>> ....
>>>>>
>>>>> I am pretty sure, we not changed anything related to httpd config for
>>>>> quite a time time and have no idea, why this issue started getting
>>>>> manifested now.
>>>>> Please help me how to RC this and what logs can be looked to debug
>>>>> further?
>>>>>
>>>>> PS: Occurence of issue is more in systems where FIPS is enabled. In
>>>>> FIPS disabled systems, occurrence is less.
>>>>>
>>>>> With Regards
>>>>> Venkat
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
Re: Httpd is hanging intermittently [ In reply to ]
Thanks Dewitt for very thorough and insightful explanation. We are using
Yocto packaged linux version with openssl version being OpenSSL 1.1.1k-fips
25 Mar 2021.

With Regards,
Venkatesh

On Fri, Sep 24, 2021 at 12:11 AM Otis Dewitt - NOAA Affiliate
<otis.dewitt@noaa.gov.invalid> wrote:

> No problem Venkatesh.
>
> No, I don't know how to generate entropy in Apache because I think Apache
> uses the system entropy.
> You can check how many are available via: "cat
> /proc/sys/kernel/random/entropy_avail".
>
> Under the system I know of two different packages, one *rngd *and the
> other *haveged.*
>
> The *rngd* daemon, which is a part of the rng-tools package, is capable
> of using both environmental noise and hardware random number generators for
> extracting entropy. The daemon checks whether the data supplied by the
> source of randomness is sufficiently random and then stores it in the
> kernel's random-number entropy pool. The random numbers it generates are
> made available through the /dev/random and /dev/urandom character devices.
>
> The *haveged *project is an attempt to provide an easy-to-use,
> unpredictable random number generator based upon an adaptation of the
> HAVEGE <http://www.irisa.fr/caps/projects/hipsor/> algorithm. Haveged was
> created to remedy low-entropy conditions in the Linux random device that
> can occur under some workloads, especially on headless servers. Current
> development of haveged is directed towards improving overall reliability
> and adaptability while minimizing the barriers to using haveged for other
> tasks.
>
> What OS are you using? Redhat CentOS etc . . .
>
>
> On Thu, Sep 23, 2021 at 2:06 PM alchemist vk <alchemist.vk@gmail.com>
> wrote:
>
>> Thanks Dewitt for your inputs.
>> Will check from system perspective how to generate more entropy and
>> resolve this issue.
>>
>> Do you know, how to generate more entropy in system or via apache so that
>> it can never be deprived of entropy?
>>
>> With Regards,
>> Venkatesh
>>
>> On Thu, Sep 23, 2021 at 8:46 PM Otis Dewitt - NOAA Affiliate
>> <otis.dewitt@noaa.gov.invalid> wrote:
>>
>>> Hmm I see, I not sure why you did not get this right away when switching
>>> from openssl to openssl-fips because FIPS require a lot of entropy
>>> and if this is on VMWARE, that has very poor entropy unless you use
>>> entropy generator like "*haveged*" or load *virtio_rng *kernel module.
>>> As I said before I am not sure how you will fix this without generating
>>> more entropy, it seems the system is unable to create enough and
>>> there is no way around this.
>>>
>>>
>>> On Thu, Sep 23, 2021 at 1:15 AM alchemist vk <alchemist.vk@gmail.com>
>>> wrote:
>>>
>>>> Thanks *Jon *for openssl command confirmation.
>>>> *@ylavik*,
>>>> Its linux OS and openssl version is 1.1.1k-fips. I not yet
>>>> explored with SSLRandomSeed changes.
>>>> Yes, we upgraded openssl few months back to 1.1.1k, but we are
>>>> seeing this httpd hangs issue from last month.
>>>>
>>>> *@otis Dewitt*, Since its production code in systems, I cant install
>>>> haveged and try it out.
>>>>
>>>>
>>>> On Thu, Sep 23, 2021 at 4:57 AM Otis Dewitt - NOAA Affiliate
>>>> <otis.dewitt@noaa.gov.invalid> wrote:
>>>>
>>>>>
>>>>> I don't think "insufficient entropy" has anything to do with Apache,
>>>>> but you could try installing "haveged" rpm.
>>>>> That may solve your problem.
>>>>>
>>>>> On Wed, Sep 22, 2021 at 2:11 PM alchemist vk <alchemist.vk@gmail.com>
>>>>> wrote:
>>>>>
>>>>>> Hi All,
>>>>>> We are using httpd version 2.4.46 and its working fine for a long
>>>>>> time. But recently, we started seeing an issue where apache hangs
>>>>>> indefinitely even when the system is in idle state.
>>>>>> And when apache hangs, I see below entries in error_log:
>>>>>> [Tue Sep 21 22:05:53.243013 2021] [ssl:warn] [pid 5769:tid
>>>>>> 2644435888] AH01990: Server: PRNG still contains insufficient entropy!
>>>>>> [Tue Sep 21 22:05:54.501476 2021] [ssl:warn] [pid 5769:tid
>>>>>> 2787111856] AH01990: Server: PRNG still contains insufficient entropy!
>>>>>> [Tue Sep 21 22:05:54.502449 2021] [ssl:warn] [pid 5769:tid
>>>>>> 2787111856] AH01990: Server: PRNG still contains insufficient entropy!
>>>>>> ...
>>>>>> ....
>>>>>> ....
>>>>>>
>>>>>> I am pretty sure, we not changed anything related to httpd config for
>>>>>> quite a time time and have no idea, why this issue started getting
>>>>>> manifested now.
>>>>>> Please help me how to RC this and what logs can be looked to debug
>>>>>> further?
>>>>>>
>>>>>> PS: Occurence of issue is more in systems where FIPS is enabled. In
>>>>>> FIPS disabled systems, occurrence is less.
>>>>>>
>>>>>> With Regards
>>>>>> Venkat
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
Re: Httpd is hanging intermittently [ In reply to ]
I did not find many but here are some notes for Yocto.

1.)
http://ch.ege.io/blog/2015/05/04/using-h-slash-w-randaom-generator-on-odrod-c1-with-yocto/
2.) https://wiki.yoctoproject.org/wiki/Entropy_on_Autobuilders

Thanks,
Otis

On Fri, Sep 24, 2021 at 9:14 AM alchemist vk <alchemist.vk@gmail.com> wrote:

> Thanks Dewitt for very thorough and insightful explanation. We are using
> Yocto packaged linux version with openssl version being OpenSSL 1.1.1k-fips
> 25 Mar 2021.
>
> With Regards,
> Venkatesh
>
> On Fri, Sep 24, 2021 at 12:11 AM Otis Dewitt - NOAA Affiliate
> <otis.dewitt@noaa.gov.invalid> wrote:
>
>> No problem Venkatesh.
>>
>> No, I don't know how to generate entropy in Apache because I think Apache
>> uses the system entropy.
>> You can check how many are available via: "cat
>> /proc/sys/kernel/random/entropy_avail".
>>
>> Under the system I know of two different packages, one *rngd *and the
>> other *haveged.*
>>
>> The *rngd* daemon, which is a part of the rng-tools package, is capable
>> of using both environmental noise and hardware random number generators for
>> extracting entropy. The daemon checks whether the data supplied by the
>> source of randomness is sufficiently random and then stores it in the
>> kernel's random-number entropy pool. The random numbers it generates are
>> made available through the /dev/random and /dev/urandom character
>> devices.
>>
>> The *haveged *project is an attempt to provide an easy-to-use,
>> unpredictable random number generator based upon an adaptation of the
>> HAVEGE <http://www.irisa.fr/caps/projects/hipsor/> algorithm. Haveged
>> was created to remedy low-entropy conditions in the Linux random device
>> that can occur under some workloads, especially on headless servers.
>> Current development of haveged is directed towards improving overall
>> reliability and adaptability while minimizing the barriers to using haveged
>> for other tasks.
>>
>> What OS are you using? Redhat CentOS etc . . .
>>
>>
>> On Thu, Sep 23, 2021 at 2:06 PM alchemist vk <alchemist.vk@gmail.com>
>> wrote:
>>
>>> Thanks Dewitt for your inputs.
>>> Will check from system perspective how to generate more entropy and
>>> resolve this issue.
>>>
>>> Do you know, how to generate more entropy in system or via apache so
>>> that it can never be deprived of entropy?
>>>
>>> With Regards,
>>> Venkatesh
>>>
>>> On Thu, Sep 23, 2021 at 8:46 PM Otis Dewitt - NOAA Affiliate
>>> <otis.dewitt@noaa.gov.invalid> wrote:
>>>
>>>> Hmm I see, I not sure why you did not get this right away when
>>>> switching from openssl to openssl-fips because FIPS require a lot of entropy
>>>> and if this is on VMWARE, that has very poor entropy unless you use
>>>> entropy generator like "*haveged*" or load *virtio_rng *kernel module.
>>>> As I said before I am not sure how you will fix this without generating
>>>> more entropy, it seems the system is unable to create enough and
>>>> there is no way around this.
>>>>
>>>>
>>>> On Thu, Sep 23, 2021 at 1:15 AM alchemist vk <alchemist.vk@gmail.com>
>>>> wrote:
>>>>
>>>>> Thanks *Jon *for openssl command confirmation.
>>>>> *@ylavik*,
>>>>> Its linux OS and openssl version is 1.1.1k-fips. I not yet
>>>>> explored with SSLRandomSeed changes.
>>>>> Yes, we upgraded openssl few months back to 1.1.1k, but we are
>>>>> seeing this httpd hangs issue from last month.
>>>>>
>>>>> *@otis Dewitt*, Since its production code in systems, I cant install
>>>>> haveged and try it out.
>>>>>
>>>>>
>>>>> On Thu, Sep 23, 2021 at 4:57 AM Otis Dewitt - NOAA Affiliate
>>>>> <otis.dewitt@noaa.gov.invalid> wrote:
>>>>>
>>>>>>
>>>>>> I don't think "insufficient entropy" has anything to do with Apache,
>>>>>> but you could try installing "haveged" rpm.
>>>>>> That may solve your problem.
>>>>>>
>>>>>> On Wed, Sep 22, 2021 at 2:11 PM alchemist vk <alchemist.vk@gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>>> Hi All,
>>>>>>> We are using httpd version 2.4.46 and its working fine for a long
>>>>>>> time. But recently, we started seeing an issue where apache hangs
>>>>>>> indefinitely even when the system is in idle state.
>>>>>>> And when apache hangs, I see below entries in error_log:
>>>>>>> [Tue Sep 21 22:05:53.243013 2021] [ssl:warn] [pid 5769:tid
>>>>>>> 2644435888] AH01990: Server: PRNG still contains insufficient entropy!
>>>>>>> [Tue Sep 21 22:05:54.501476 2021] [ssl:warn] [pid 5769:tid
>>>>>>> 2787111856] AH01990: Server: PRNG still contains insufficient entropy!
>>>>>>> [Tue Sep 21 22:05:54.502449 2021] [ssl:warn] [pid 5769:tid
>>>>>>> 2787111856] AH01990: Server: PRNG still contains insufficient entropy!
>>>>>>> ...
>>>>>>> ....
>>>>>>> ....
>>>>>>>
>>>>>>> I am pretty sure, we not changed anything related to httpd config
>>>>>>> for quite a time time and have no idea, why this issue started getting
>>>>>>> manifested now.
>>>>>>> Please help me how to RC this and what logs can be looked to debug
>>>>>>> further?
>>>>>>>
>>>>>>> PS: Occurence of issue is more in systems where FIPS is enabled. In
>>>>>>> FIPS disabled systems, occurrence is less.
>>>>>>>
>>>>>>> With Regards
>>>>>>> Venkat
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
Re: Httpd is hanging intermittently [ In reply to ]
Thanks Dewitt for the links. These are very useful.
Will check with our kernel team on the instructions/points made in the
given links and will try to find a workaround to resove httpd hanging.
Thanks a lot for your help.

With Regards,
Venkatesh

On Fri, Sep 24, 2021 at 9:57 PM Otis Dewitt - NOAA Affiliate
<otis.dewitt@noaa.gov.invalid> wrote:

> I did not find many but here are some notes for Yocto.
>
> 1.)
> http://ch.ege.io/blog/2015/05/04/using-h-slash-w-randaom-generator-on-odrod-c1-with-yocto/
> 2.) https://wiki.yoctoproject.org/wiki/Entropy_on_Autobuilders
>
> Thanks,
> Otis
>
> On Fri, Sep 24, 2021 at 9:14 AM alchemist vk <alchemist.vk@gmail.com>
> wrote:
>
>> Thanks Dewitt for very thorough and insightful explanation. We are using
>> Yocto packaged linux version with openssl version being OpenSSL 1.1.1k-fips
>> 25 Mar 2021.
>>
>> With Regards,
>> Venkatesh
>>
>> On Fri, Sep 24, 2021 at 12:11 AM Otis Dewitt - NOAA Affiliate
>> <otis.dewitt@noaa.gov.invalid> wrote:
>>
>>> No problem Venkatesh.
>>>
>>> No, I don't know how to generate entropy in Apache because I think
>>> Apache uses the system entropy.
>>> You can check how many are available via: "cat
>>> /proc/sys/kernel/random/entropy_avail".
>>>
>>> Under the system I know of two different packages, one *rngd *and the
>>> other *haveged.*
>>>
>>> The *rngd* daemon, which is a part of the rng-tools package, is capable
>>> of using both environmental noise and hardware random number generators for
>>> extracting entropy. The daemon checks whether the data supplied by the
>>> source of randomness is sufficiently random and then stores it in the
>>> kernel's random-number entropy pool. The random numbers it generates are
>>> made available through the /dev/random and /dev/urandom character
>>> devices.
>>>
>>> The *haveged *project is an attempt to provide an easy-to-use,
>>> unpredictable random number generator based upon an adaptation of the
>>> HAVEGE <http://www.irisa.fr/caps/projects/hipsor/> algorithm. Haveged
>>> was created to remedy low-entropy conditions in the Linux random device
>>> that can occur under some workloads, especially on headless servers.
>>> Current development of haveged is directed towards improving overall
>>> reliability and adaptability while minimizing the barriers to using haveged
>>> for other tasks.
>>>
>>> What OS are you using? Redhat CentOS etc . . .
>>>
>>>
>>> On Thu, Sep 23, 2021 at 2:06 PM alchemist vk <alchemist.vk@gmail.com>
>>> wrote:
>>>
>>>> Thanks Dewitt for your inputs.
>>>> Will check from system perspective how to generate more entropy and
>>>> resolve this issue.
>>>>
>>>> Do you know, how to generate more entropy in system or via apache so
>>>> that it can never be deprived of entropy?
>>>>
>>>> With Regards,
>>>> Venkatesh
>>>>
>>>> On Thu, Sep 23, 2021 at 8:46 PM Otis Dewitt - NOAA Affiliate
>>>> <otis.dewitt@noaa.gov.invalid> wrote:
>>>>
>>>>> Hmm I see, I not sure why you did not get this right away when
>>>>> switching from openssl to openssl-fips because FIPS require a lot of entropy
>>>>> and if this is on VMWARE, that has very poor entropy unless you use
>>>>> entropy generator like "*haveged*" or load *virtio_rng *kernel module.
>>>>> As I said before I am not sure how you will fix this without
>>>>> generating more entropy, it seems the system is unable to create enough and
>>>>> there is no way around this.
>>>>>
>>>>>
>>>>> On Thu, Sep 23, 2021 at 1:15 AM alchemist vk <alchemist.vk@gmail.com>
>>>>> wrote:
>>>>>
>>>>>> Thanks *Jon *for openssl command confirmation.
>>>>>> *@ylavik*,
>>>>>> Its linux OS and openssl version is 1.1.1k-fips. I not yet
>>>>>> explored with SSLRandomSeed changes.
>>>>>> Yes, we upgraded openssl few months back to 1.1.1k, but we are
>>>>>> seeing this httpd hangs issue from last month.
>>>>>>
>>>>>> *@otis Dewitt*, Since its production code in systems, I cant install
>>>>>> haveged and try it out.
>>>>>>
>>>>>>
>>>>>> On Thu, Sep 23, 2021 at 4:57 AM Otis Dewitt - NOAA Affiliate
>>>>>> <otis.dewitt@noaa.gov.invalid> wrote:
>>>>>>
>>>>>>>
>>>>>>> I don't think "insufficient entropy" has anything to do with Apache,
>>>>>>> but you could try installing "haveged" rpm.
>>>>>>> That may solve your problem.
>>>>>>>
>>>>>>> On Wed, Sep 22, 2021 at 2:11 PM alchemist vk <alchemist.vk@gmail.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> Hi All,
>>>>>>>> We are using httpd version 2.4.46 and its working fine for a long
>>>>>>>> time. But recently, we started seeing an issue where apache hangs
>>>>>>>> indefinitely even when the system is in idle state.
>>>>>>>> And when apache hangs, I see below entries in error_log:
>>>>>>>> [Tue Sep 21 22:05:53.243013 2021] [ssl:warn] [pid 5769:tid
>>>>>>>> 2644435888] AH01990: Server: PRNG still contains insufficient entropy!
>>>>>>>> [Tue Sep 21 22:05:54.501476 2021] [ssl:warn] [pid 5769:tid
>>>>>>>> 2787111856] AH01990: Server: PRNG still contains insufficient entropy!
>>>>>>>> [Tue Sep 21 22:05:54.502449 2021] [ssl:warn] [pid 5769:tid
>>>>>>>> 2787111856] AH01990: Server: PRNG still contains insufficient entropy!
>>>>>>>> ...
>>>>>>>> ....
>>>>>>>> ....
>>>>>>>>
>>>>>>>> I am pretty sure, we not changed anything related to httpd config
>>>>>>>> for quite a time time and have no idea, why this issue started getting
>>>>>>>> manifested now.
>>>>>>>> Please help me how to RC this and what logs can be looked to debug
>>>>>>>> further?
>>>>>>>>
>>>>>>>> PS: Occurence of issue is more in systems where FIPS is enabled. In
>>>>>>>> FIPS disabled systems, occurrence is less.
>>>>>>>>
>>>>>>>> With Regards
>>>>>>>> Venkat
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>