Mailing List Archive: varnishncsa -F flag custom log format

varnishncsa -F flag custom log format

May 26, 2012, 2:24 PM

Post #1 of 4 (6906 views)

Running Varnish 3.02 rev. cbf1284 on Ubuntu Server 12.04 LTS x64:

I can get varnishncsa writing to logs successfully with:

VARNISHNCSA_ENABLED=1

DAEMON_OPTS="-a \

-c \

-D \

-P ${PIDFILE} \

-w /var/log/varnish/hollywood.varnishncsa.log"

in /etc/default/varnishncsa

but I'm really struggling with the -F flag for custom log formats. I'm
trying to achieve an Apache combined log format prefixed by the virtual
host, which'd look something like this:

%{Host}i %h %l %u %t \"%r\" %s %b \"%{Referer}i\" \"%{User-agent}i\"

but all I'm managing to do is capture the %{Host}i bit in the logfile.

There's an old thread on here about issues with spaces and escaping quotes,
but the responses are all about setting the format on the command line,
which I'm really trying to avoid. I've got to:

VARNISHNCSA_ENABLED=1

FORMAT="%{Host}i %h %l %u %t \"%r\" %s %b \"%{Referer}i\"
\"%{User-agent}i\""

DAEMON_OPTS="-a \

-c \

-D \

-F ${FORMAT} \

-P ${PIDFILE} \

-w /var/log/varnish/hollywood.varnishncsa.log"

I've tried various permutations of single and double quotes, and the format
both in and out of a variable. No luck so far. Can anyone help me out?

Re: varnishncsa -F flag custom log format [ In reply to ]

daniel.carrillo at gmail

May 27, 2012, 3:29 AM

Post #2 of 4 (6813 views)

Permalink

2012/5/26 Rowan @ Jetboy <rowan@jetboy.co.uk>:
> Running Varnish 3.02 rev. cbf1284 on Ubuntu Server 12.04 LTS x64:
>
>
>
> I can get varnishncsa writing to logs successfully with:
>
>
>
> VARNISHNCSA_ENABLED=1
>
> DAEMON_OPTS="-a \
>
>              -c \
>
>              -D \
>
>              -P ${PIDFILE} \
>
>              -w /var/log/varnish/hollywood.varnishncsa.log"
>
>
>
> in /etc/default/varnishncsa
>
>
>
> but I'm really struggling with the -F flag for custom log formats. I'm
> trying to achieve an Apache combined log format prefixed by the virtual
> host, which'd look something like this:
>
>
>
> %{Host}i %h %l %u %t \"%r\" %s %b \"%{Referer}i\" \"%{User-agent}i\"
>
>
>
> but all I'm managing to do is capture the %{Host}i bit in the logfile.
>
>
>
> There's an old thread on here about issues with spaces and escaping quotes,
> but the responses are all about setting the format on the command line,
> which I'm really trying to avoid. I've got to:
>
>
>
> VARNISHNCSA_ENABLED=1
>
> FORMAT="%{Host}i %h %l %u %t \"%r\" %s %b \"%{Referer}i\"
> \"%{User-agent}i\""
>
> DAEMON_OPTS="-a \
>
>              -c \
>
>              -D \
>
>              -F ${FORMAT} \
>
>              -P ${PIDFILE} \
>
>              -w /var/log/varnish/hollywood.varnishncsa.log"
>
>
>
> I've tried various permutations of single and double quotes, and the format
> both in and out of a variable. No luck so far. Can anyone help me out?

This one works for me:

LOG_FORMAT="%h %l %u %t \"%r\" %s %b \"%{Referer}i\"
\"%{User-agent}i\" %{Host}i %{Varnish:time_firstbyte}x"
DAEMON_OPTS="-c -a -w $logfile -D -P $pidfile -F '${LOG_FORMAT}'"

Please, note the single quotes in -F

Hope this help.

_______________________________________________
varnish-misc mailing list
varnish-misc@varnish-cache.org
https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc

RE: varnishncsa -F flag custom log format [ In reply to ]

rowan at jetboy

May 27, 2012, 7:09 AM

Post #3 of 4 (6820 views)

Permalink

Hi Daniel,

Thanks for the reply. I've tried entirely replacing my code with yours
(other than the capitalisation of $PIDFILE, and putting $logfile inline),
but I get similar results. It's only recording the first log parameter in
the log (in this case, the host IP). The single quotes result in the first
single quote being recorded in the log. E.g.:

'66.249.72.57
'157.55.17.194

Are you running Varnish on Linux with bash, or some other OS flavour?

<snip>

This one works for me:

LOG_FORMAT="%h %l %u %t \"%r\" %s %b \"%{Referer}i\"
\"%{User-agent}i\" %{Host}i %{Varnish:time_firstbyte}x"
DAEMON_OPTS="-c -a -w $logfile -D -P $pidfile -F '${LOG_FORMAT}'"

Please, note the single quotes in -F

Hope this help.

</snip>

_______________________________________________
varnish-misc mailing list
varnish-misc@varnish-cache.org
https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc

Re: varnishncsa -F flag custom log format [ In reply to ]

rekcah78 at gmail

May 27, 2012, 1:20 PM

Post #4 of 4 (6814 views)

Permalink

Hi,

in the init file (Ubuntu):

I set:
+ LOG_FORMAT="%{Host}i %h %l %u %t \"%r\" %s %b \"%{Referer}i\"
\"%{User-agent}i\""
- DAEMON_OPTS="-a -w ${LOGFILE} -D -P ${PIDFILE}"
+ DAEMON_OPTS="-a -w ${LOGFILE} -D -P ${PIDFILE} -F"

- --chuid $USER --exec ${DAEMON} -- ${DAEMON_OPTS} \
+ --chuid $USER --exec ${DAEMON} -- ${DAEMON_OPTS} "${LOG_FORMAT}" \

Now I have a correct log line:
www.XXXXXX.com 2a01:e35:2f16:....:21b:..ff:....:1c0f - -
[27/May/2012:22:07:05 +0200] "GET http://www.XXXXXX.com/favicon.ico
HTTP/1.1" 404 188 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_6_8)
AppleWebKit/536.5 (KHTML, like Gecko) Chrome/19.0.1084.52
Safari/536.5"

But now I want to change the value %r not contain http://www.XXXXXX.com :)

Christophe

2012/5/27 Rowan @ Jetboy <rowan@jetboy.co.uk>:
> Hi Daniel,
>
> Thanks for the reply. I've tried entirely replacing my code with yours
> (other than the capitalisation of $PIDFILE, and putting $logfile inline),
> but I get similar results. It's only recording the first log parameter in
> the log (in this case, the host IP). The single quotes result in the first
> single quote being recorded in the log. E.g.:
>
> '66.249.72.57
> '157.55.17.194
>
> Are you running Varnish on Linux with bash, or some other OS flavour?
>
>
> <snip>
>
> This one works for me:
>
> LOG_FORMAT="%h %l %u %t \"%r\" %s %b \"%{Referer}i\"
> \"%{User-agent}i\" %{Host}i %{Varnish:time_firstbyte}x"
> DAEMON_OPTS="-c -a -w $logfile -D -P $pidfile -F '${LOG_FORMAT}'"
>
> Please, note the single quotes in -F
>
> Hope this help.
>
> </snip>
>
>
> _______________________________________________
> varnish-misc mailing list
> varnish-misc@varnish-cache.org
> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc

_______________________________________________
varnish-misc mailing list
varnish-misc@varnish-cache.org
https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc