Mailing List Archive

On Friday 04 June 2021 at 11:09:33, Olli Pasanen wrote:

> Hello,
>
> I’ve encountered an interesting local DOS issue on Apache 2.4.48, and I am
> not sure whether this is intended behavior.
>
> Symptom:
> - Apache2 hangs indefinitely (stops processing incoming requests)
>
> Setup:
> - Apache 2.4.48 built from source, running on Debian 10
> - A dummy CustomLog program that does not consume stdin

Do you believe the first step to be a requirement, or can this situation be
reproduced using standard packaged versions of Apache (it sounds like this
would be true)?

> An example of a directive causing the issue:
> CustomLog “|/tmp/logger.py” combined OR
> CustomLog “|$/tmp/logger.py” combined
>
> To reproduce the issue, simply configure CustomLog as above where the
> target logger is a dummy executable that does not consume stdin. After
> this, generate enough requests to hang the server (in my case around
> 700-800, but this depends on what the log format is and probably other
> directives too). Apache then stops processing any incoming requests.

I don't think I would call this a DoS problem - it's not as though there is
anything an external agent can do to a correctly configured Apache to cause
this behaviour. I'd say it is an error in configuration (admin needs to ensure
that anything told to accept log entries from Apache does actually do so),
although I agree it might be good if Apache's logging could be decoupled from
request processing, so that a blockage in one does not affect the other.

> I am not entirely sure about the root cause of the issue, but I believe it
> apache wants to write to the stdin buffer of the target logger, and since
> it is full, it will try to wait until there is space. Since the logger is
> not consuming the stdin, this causes apache to hang indefinitely.
>
> Is this intended behaviour? If so, are there any ways to mitigate the issue
> by changing the Apache / OS configuration? The logger that is actually
> used is not a dummy program, but unfortunately has similar behaviour (full
> buffers) under heavy load.

I would guess that it's not exactly "intended", but it's a consequence of
request processing being dependent on log processing. Maybe a feature request
to the developers to have these decoupled would be worthwhile?


Antony.

--
Just when you think you're done, a cat floats by with buttered toast strapped
to its back.

- Steve Krug, "Don't make me think"

Please reply to the list;
please *don't* CC me.

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org

Thanks for the reply.

> Do you believe the first step to be a requirement, or can this situation be reproduced using standard packaged versions of Apache (it sounds like this would be true)?

It is not specific to any modifications made to the source code. The standard packaged versions behave exactly the same.

> I don't think I would call this a DoS problem - it's not as though there is anything an external agent can do to a correctly configured Apache to cause this behaviour. I'd say it is an error in configuration (admin needs to ensure that anything told to accept log entries from Apache does actually do so), although I agree it might be good if Apache's logging could be decoupled from request processing, so that a blockage in one does not affect the other.

I agree that it is not a typical DoS caused by an external user, but depending on the custom logger, this behaviour might go unnoticed for a while even in production and cause outages.
If the buffer is not consumed at all, I believe the issue persists until someone restarts/reloads the main process.

A misbehaving logger is definitely a configuration error, but I think it would make more sense to drop the log if the buffers are full, rather than wait for them to empty.

> I would guess that it's not exactly "intended", but it's a consequence of request processing being dependent on log processing. Maybe a feature request to the developers to have these decoupled would be worthwhile?

It would be nice to have this behavior at least mentioned on the CustomLog documentation page.
I am not sure how to go about creating a new feature request.

- Olli

-----Original Message-----
From: Antony Stone <Antony.Stone@apache.open.source.it>
Sent: Friday, 4 June 2021 12.55
To: users@httpd.apache.org
Subject: Re: [users@httpd] Misbehaving CustomLog logger causing local denial-of-service, intended behaviour?

On Friday 04 June 2021 at 11:09:33, Olli Pasanen wrote:

> Hello,
>
> I’ve encountered an interesting local DOS issue on Apache 2.4.48, and
> I am not sure whether this is intended behavior.
>
> Symptom:
> - Apache2 hangs indefinitely (stops processing incoming requests)
>
> Setup:
> - Apache 2.4.48 built from source, running on Debian 10
> - A dummy CustomLog program that does not consume stdin

Do you believe the first step to be a requirement, or can this situation be reproduced using standard packaged versions of Apache (it sounds like this would be true)?

> An example of a directive causing the issue:
> CustomLog “|/tmp/logger.py” combined OR CustomLog “|$/tmp/logger.py”
> combined
>
> To reproduce the issue, simply configure CustomLog as above where the
> target logger is a dummy executable that does not consume stdin. After
> this, generate enough requests to hang the server (in my case around
> 700-800, but this depends on what the log format is and probably other
> directives too). Apache then stops processing any incoming requests.

I don't think I would call this a DoS problem - it's not as though there is anything an external agent can do to a correctly configured Apache to cause this behaviour. I'd say it is an error in configuration (admin needs to ensure that anything told to accept log entries from Apache does actually do so), although I agree it might be good if Apache's logging could be decoupled from request processing, so that a blockage in one does not affect the other.

> I am not entirely sure about the root cause of the issue, but I
> believe it apache wants to write to the stdin buffer of the target
> logger, and since it is full, it will try to wait until there is
> space. Since the logger is not consuming the stdin, this causes apache to hang indefinitely.
>
> Is this intended behaviour? If so, are there any ways to mitigate the
> issue by changing the Apache / OS configuration? The logger that is
> actually used is not a dummy program, but unfortunately has similar
> behaviour (full
> buffers) under heavy load.

I would guess that it's not exactly "intended", but it's a consequence of request processing being dependent on log processing. Maybe a feature request to the developers to have these decoupled would be worthwhile?


Antony.

--
Just when you think you're done, a cat floats by with buttered toast strapped to its back.

- Steve Krug, "Don't make me think"

Please reply to the list;
please *don't* CC me.

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org