Mailing List Archive

Conditional Logging ( using varnishncsa )
Hi,

We have logging turned on using varnishncsa.

/usr/bin/varnishncsa -a -w /var/log/varnish/varnishncsa.log -D -f
/etc/sysconfig/varnishncsa

Here is what's defined in varnishncsa

%{X-Forwarded-For}i %l %u %t %D \"%r\" %s %b \"%{Referer}i\"
\"%{User-agent}i\" \"%{Host}i\" %{Varnish:hitmiss}x

However, this would log EVERY request that goes through varnish.

We have a monitoring server that hits it aggressively ( and also static
files ).

x.x.x.x - - [07/Nov/2019:00:22:53 -0600] 2080 "GET
http://localhost/index.php HTTP/1.0" 200 8 "-" "HTTP-Monitor/1.1" "-" miss
x.x.x.x - - [07/Nov/2019:00:22:58 -0600] 2472 "GET
http://localhost/index.php HTTP/1.0" 200 8 "-" "HTTP-Monitor/1.1" "-" miss
x.x.x.x - - [07/Nov/2019:00:22:59 -0600] 1919 "GET
http://localhost/index.php HTTP/1.0" 200 8 "-" "HTTP-Monitor/1.1" "-" miss

Is there a way in which I can exclude these from varnish logs ?

In apache I would just do

SetEnvIf Request_URI
"\.(jpeg|jpg|xml|png|gif|ico|js|css|swf|woff|ttf|eot\?|js?.|css?.)$" DontLog
SetEnvIfNoCase User-Agent "(HTTP-Monitor)" DontLog
CustomLog /var/www/logs/access_80_log combined env=!DontLog

This would otherwise just keep filling up the logs.

Let me know.

Thanks.
Re: Conditional Logging ( using varnishncsa ) [ In reply to ]
On Thu, Nov 7, 2019 at 6:34 AM Maninder Singh <mandys@gmail.com> wrote:
>
> Hi,
>
> We have logging turned on using varnishncsa.
>
> /usr/bin/varnishncsa -a -w /var/log/varnish/varnishncsa.log -D -f /etc/sysconfig/varnishncsa
>
> Here is what's defined in varnishncsa
>
> %{X-Forwarded-For}i %l %u %t %D \"%r\" %s %b \"%{Referer}i\" \"%{User-agent}i\" \"%{Host}i\" %{Varnish:hitmiss}x
>
> However, this would log EVERY request that goes through varnish.
>
> We have a monitoring server that hits it aggressively ( and also static files ).
>
> x.x.x.x - - [07/Nov/2019:00:22:53 -0600] 2080 "GET http://localhost/index.php HTTP/1.0" 200 8 "-" "HTTP-Monitor/1.1" "-" miss
> x.x.x.x - - [07/Nov/2019:00:22:58 -0600] 2472 "GET http://localhost/index.php HTTP/1.0" 200 8 "-" "HTTP-Monitor/1.1" "-" miss
> x.x.x.x - - [07/Nov/2019:00:22:59 -0600] 1919 "GET http://localhost/index.php HTTP/1.0" 200 8 "-" "HTTP-Monitor/1.1" "-" miss
>
> Is there a way in which I can exclude these from varnish logs ?
>
> In apache I would just do
>
> SetEnvIf Request_URI "\.(jpeg|jpg|xml|png|gif|ico|js|css|swf|woff|ttf|eot\?|js?.|css?.)$" DontLog
> SetEnvIfNoCase User-Agent "(HTTP-Monitor)" DontLog
> CustomLog /var/www/logs/access_80_log combined env=!DontLog
>
> This would otherwise just keep filling up the logs.

Do something like this with your command line:

> varnishncsa [...] -q 'not (ReqHeader:User-Agent ~ "HTTP-Monitor" or ReqURL ~ "\.(jpeg|jpg|xml|png|gif|ico|js|css|swf|woff|ttf|eot\?|js?.|css?.)$")'

See man varnishncsa, man vsl and man vsl-query.

Dridi
_______________________________________________
varnish-misc mailing list
varnish-misc@varnish-cache.org
https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
Re: Conditional Logging ( using varnishncsa ) [ In reply to ]
Thank you Dridi.

This worked :-)

On Thu, 7 Nov 2019 at 12:31, Dridi Boukelmoune <dridi@varni.sh> wrote:

> On Thu, Nov 7, 2019 at 6:34 AM Maninder Singh <mandys@gmail.com> wrote:
> >
> > Hi,
> >
> > We have logging turned on using varnishncsa.
> >
> > /usr/bin/varnishncsa -a -w /var/log/varnish/varnishncsa.log -D -f
> /etc/sysconfig/varnishncsa
> >
> > Here is what's defined in varnishncsa
> >
> > %{X-Forwarded-For}i %l %u %t %D \"%r\" %s %b \"%{Referer}i\"
> \"%{User-agent}i\" \"%{Host}i\" %{Varnish:hitmiss}x
> >
> > However, this would log EVERY request that goes through varnish.
> >
> > We have a monitoring server that hits it aggressively ( and also static
> files ).
> >
> > x.x.x.x - - [07/Nov/2019:00:22:53 -0600] 2080 "GET
> http://localhost/index.php HTTP/1.0" 200 8 "-" "HTTP-Monitor/1.1" "-" miss
> > x.x.x.x - - [07/Nov/2019:00:22:58 -0600] 2472 "GET
> http://localhost/index.php HTTP/1.0" 200 8 "-" "HTTP-Monitor/1.1" "-" miss
> > x.x.x.x - - [07/Nov/2019:00:22:59 -0600] 1919 "GET
> http://localhost/index.php HTTP/1.0" 200 8 "-" "HTTP-Monitor/1.1" "-" miss
> >
> > Is there a way in which I can exclude these from varnish logs ?
> >
> > In apache I would just do
> >
> > SetEnvIf Request_URI
> "\.(jpeg|jpg|xml|png|gif|ico|js|css|swf|woff|ttf|eot\?|js?.|css?.)$" DontLog
> > SetEnvIfNoCase User-Agent "(HTTP-Monitor)" DontLog
> > CustomLog /var/www/logs/access_80_log combined env=!DontLog
> >
> > This would otherwise just keep filling up the logs.
>
> Do something like this with your command line:
>
> > varnishncsa [...] -q 'not (ReqHeader:User-Agent ~ "HTTP-Monitor" or
> ReqURL ~
> "\.(jpeg|jpg|xml|png|gif|ico|js|css|swf|woff|ttf|eot\?|js?.|css?.)$")'
>
> See man varnishncsa, man vsl and man vsl-query.
>
> Dridi
>