Mailing List Archive

Hello Andrei,

thanks a lot for your reply and thoughts ...

perhaps I used the wrong words in the subject?

I'm actually not looking to split a varnish logfile into different
domains and log files ...
instead I was looking to create single log files for each domain with
varnishncsa

similar to this:

varnishncsa -q "ReqHost ~ '^mydomain.com$' " -a -w
/var/log/varnish/mydomain.com -D
varnishncsa -q "ReqHost ~ '^myotherdomain.com$' " -a -w
/var/log/varnish/myotherdomain.com -D

is something like this possible?

thanks & greetings
becki



Am 12.11.2016 um 13:30 schrieb Andrei:
> Hello,
>
> I suggest looking into splitlogs with piped logging:
>
> http://httpd.apache.org/docs/current/logs.html#piped
> https://httpd.apache.org/docs/2.4/programs/split-logfile.html
>
> This is similarly used with cPanel/WHM
> (https://documentation.cpanel.net/display/ALD/The+splitlogs+Binary)
> which you can likely pipe to directly from varnishncsa granted the
> expected format is used. I haven't tried this personally, but now that
> you've mentioned it this would seem pretty useful. Hope this helps!
>
> On 11/12/2016 12:52 PM, Admin Beckspaced wrote:
>> hello there ;)
>>
>> since putting varnish version 5 in front of my apache backend the
>> apache access logs don't fill up and webalizer has no data to process.
>>
>> I know that I can use varnishncsa to create logs in apache format.
>>
>> but since I'm using multiple domains I would need logs per domain.
>> I see that there's the -q <query> option
>>
>> I found stuff by google like:
>>
>> varnishncsa -m "RxHeader:^Host: www.domain1.com$" -a -w
>> /var/log/varnish/www.domain1.com -D
>>
>> but I see that the -m param is no longer available in version 5?
>>
>> how could i use that -q <query> param so varnishncsa would produce
>> different logs for different domains?
>>
>> would a regex query on the host like below be possible?
>>
>> varnishncsa -q "ReqHost ~ '^mydomain.com$' " -a -w
>> /var/log/varnish/mydomain.com -D
>> varnishncsa -q "ReqHost ~ '^myotherdomain.com$' " -a -w
>> /var/log/varnish/myotherdomain.com -D
>>
>> where can I find a list of variables I can use in the vsl-query language?
>> a RTFM with the proper link would be perfect ;)
>>
>> thanks & greetings
>> becki
>>
>> _______________________________________________
>> varnish-misc mailing list
>> varnish-misc@varnish-cache.org
>> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
>
>
>
>


_______________________________________________
varnish-misc mailing list
varnish-misc@varnish-cache.org
https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc

>> I suggest looking into splitlogs with piped logging:
>>
>> http://httpd.apache.org/docs/current/logs.html#piped
>> https://httpd.apache.org/docs/2.4/programs/split-logfile.html
>>
>> This is similarly used with cPanel/WHM
>> (https://documentation.cpanel.net/display/ALD/The+splitlogs+Binary)
>> which you can likely pipe to directly from varnishncsa granted the
>> expected format is used. I haven't tried this personally, but now that
>> you've mentioned it this would seem pretty useful. Hope this helps!


Sorry to sort of hijack the thread but this is interesting to me.

I have Varnish 4.1 in front of WHM. Is it possible to pipe all
varnishncsa logs into one log in Apache (WHM in my case) so after Apache
processes those logs in webalizer or awstats?

I would like to avoid to have to add manually everytime I add a new
virtualhost in WHM.

Miguel

_______________________________________________
varnish-misc mailing list
varnish-misc@varnish-cache.org
https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc

Same Hello here ;)

did have a more in depth look in the manual and figured out that
varnishncsa does support VSL query.
so someone could filter on the Request Header and Host

varnishncsa -q "ReqHeader ~ '^Host: .*example.com'"

which would produce a log for a specific domain only

it then would need multiple varnishncsa instances for logging per
domain, which I found here:

https://kevops.com/2015/11/varnish-logging-per-host-with-init-script/

I use varnish version 5 and then there would be no need for splitlog and
the logs would be created directly.

please correct me if I'm wrong?

thanks for your time & help
Becki


Am 12.11.2016 um 17:05 schrieb Andrei:
> Hello again,
>
> My apologies for not explaining my thoughts better earlier then. Afaik,
> varnishncsa does not have a native method to split output based on
> different parameters. The method I was thinking of was based on piping
> varnishncsa output through splitlogs (or similar) for the log processing
> and writeouts. Since replying earlier, I've got this working on a cPanel
> server with piped logging enabled for Apache using the following two for
> example (X-Port is a custom header set in vcl_recv related to SSL
> offloading, but you can use a static value or similar custom header):
>
> varnishncsa -F "%{HOST}i:%{X-Port}i %h %l %u %t \"%m %U%q %H\" %s %b
> \"%{Referer}i\" \"%{User-agent}i\""|sed -e
> 's#^www\.##g'|/usr/local/cpanel/bin/splitlogs --main=`hostname`
> --mainout=/usr/local/apache/logs/access_log
> varnishncsa -F "%{HOST}i %{%s}t %b ."|sed -e
> 's#^www\.##g'|/usr/local/cpanel/bin/splitlogs --main=`hostname`
> --suffix=-bytes_log
>
> The above pipes the requests to the splitlogs binary which queues then
> writes to separate logs per domain, that are later processed by the
> cPanel log stats apps. Either way, I believe you need an intermediary
> script to queue and write the log entries per domain. While looking into
> this process, I ran across this little tidbit which you may find of use
> https://gist.github.com/garlandkr/4954272 for logstash style output.
>
>


_______________________________________________
varnish-misc mailing list
varnish-misc@varnish-cache.org
https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc

Hello,

By not using splitlogs or an intermediary script, you will be forced to run
multiple instances of varnishncsa, which isn't optimal if you host multiple
domains. The more traffic/domains you have, the more resources you will
consume on parsing the same data across multiple channels. Yes, varnishncsa
supports VSL, however I think your approach is a bit off (no offense). As
in, you need to shift the log writing process away from varnishncsa. By
doing so, you only have one instance of varnishncsa using resources to
gather the data, which is then fed to a parser that handles the per domain
log splits and writes. That's where 'splitlogs' came into play.

As your question does raise some interest in the cPanel community (myself
and Miguel González on this list for example), I threw together a quick
Perl script that will in short, pipe and parse data between varnishncsa and
the splitlogs binary for cache hits. This lets splitlogs handle the queued
log writes which are later parsed for cPanel bandwidth usage and graphs,
webalizer, awstats, logaholic, etc - https://github.com/AndreiG6/vscp



On Sun, Nov 13, 2016 at 12:18 PM, Admin Beckspaced <admin@beckspaced.com>
wrote:

> Same Hello here ;)
>
> did have a more in depth look in the manual and figured out that
> varnishncsa does support VSL query.
> so someone could filter on the Request Header and Host
>
> varnishncsa -q "ReqHeader ~ '^Host: .*example.com'"
>
> which would produce a log for a specific domain only
>
> it then would need multiple varnishncsa instances for logging per domain,
> which I found here:
>
> https://kevops.com/2015/11/varnish-logging-per-host-with-init-script/
>
> I use varnish version 5 and then there would be no need for splitlog and
> the logs would be created directly.
>
> please correct me if I'm wrong?
>
> thanks for your time & help
> Becki
>
>
> Am 12.11.2016 um 17:05 schrieb Andrei:
>
>> Hello again,
>>
>> My apologies for not explaining my thoughts better earlier then. Afaik,
>> varnishncsa does not have a native method to split output based on
>> different parameters. The method I was thinking of was based on piping
>> varnishncsa output through splitlogs (or similar) for the log processing
>> and writeouts. Since replying earlier, I've got this working on a cPanel
>> server with piped logging enabled for Apache using the following two for
>> example (X-Port is a custom header set in vcl_recv related to SSL
>> offloading, but you can use a static value or similar custom header):
>>
>> varnishncsa -F "%{HOST}i:%{X-Port}i %h %l %u %t \"%m %U%q %H\" %s %b
>> \"%{Referer}i\" \"%{User-agent}i\""|sed -e
>> 's#^www\.##g'|/usr/local/cpanel/bin/splitlogs --main=`hostname`
>> --mainout=/usr/local/apache/logs/access_log
>> varnishncsa -F "%{HOST}i %{%s}t %b ."|sed -e
>> 's#^www\.##g'|/usr/local/cpanel/bin/splitlogs --main=`hostname`
>> --suffix=-bytes_log
>>
>> The above pipes the requests to the splitlogs binary which queues then
>> writes to separate logs per domain, that are later processed by the
>> cPanel log stats apps. Either way, I believe you need an intermediary
>> script to queue and write the log entries per domain. While looking into
>> this process, I ran across this little tidbit which you may find of use
>> https://gist.github.com/garlandkr/4954272 for logstash style output.
>>
>>
>>
>
> _______________________________________________
> varnish-misc mailing list
> varnish-misc@varnish-cache.org
> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
>

Is the resources argument that compelling? I find it cleaner to have one
ncsa per domain, plus the ncsas will read from memory, which is super fast.

Sure you can then split logs afterwards using awk or whatever, but that's
adding an extra layer and serializing a process that doesn't need to be.

But I'm not an admin, so I may be off.

On Nov 13, 2016 16:27, "Andrei" <lagged@gmail.com> wrote:

> Hello,
>
> By not using splitlogs or an intermediary script, you will be forced to
> run multiple instances of varnishncsa, which isn't optimal if you host
> multiple domains. The more traffic/domains you have, the more resources you
> will consume on parsing the same data across multiple channels.
> Yes, varnishncsa supports VSL, however I think your approach is a bit off
> (no offense). As in, you need to shift the log writing process away
> from varnishncsa. By doing so, you only have one instance of varnishncsa
> using resources to gather the data, which is then fed to a parser that
> handles the per domain log splits and writes. That's where 'splitlogs' came
> into play.
>
> As your question does raise some interest in the cPanel community (myself
> and Miguel González on this list for example), I threw together a quick
> Perl script that will in short, pipe and parse data between varnishncsa and
> the splitlogs binary for cache hits. This lets splitlogs handle the queued
> log writes which are later parsed for cPanel bandwidth usage and graphs,
> webalizer, awstats, logaholic, etc - https://github.com/AndreiG6/vscp
>
>
>
> On Sun, Nov 13, 2016 at 12:18 PM, Admin Beckspaced <admin@beckspaced.com>
> wrote:
>
>> Same Hello here ;)
>>
>> did have a more in depth look in the manual and figured out that
>> varnishncsa does support VSL query.
>> so someone could filter on the Request Header and Host
>>
>> varnishncsa -q "ReqHeader ~ '^Host: .*example.com'"
>>
>> which would produce a log for a specific domain only
>>
>> it then would need multiple varnishncsa instances for logging per domain,
>> which I found here:
>>
>> https://kevops.com/2015/11/varnish-logging-per-host-with-init-script/
>>
>> I use varnish version 5 and then there would be no need for splitlog and
>> the logs would be created directly.
>>
>> please correct me if I'm wrong?
>>
>> thanks for your time & help
>> Becki
>>
>>
>> Am 12.11.2016 um 17:05 schrieb Andrei:
>>
>>> Hello again,
>>>
>>> My apologies for not explaining my thoughts better earlier then. Afaik,
>>> varnishncsa does not have a native method to split output based on
>>> different parameters. The method I was thinking of was based on piping
>>> varnishncsa output through splitlogs (or similar) for the log processing
>>> and writeouts. Since replying earlier, I've got this working on a cPanel
>>> server with piped logging enabled for Apache using the following two for
>>> example (X-Port is a custom header set in vcl_recv related to SSL
>>> offloading, but you can use a static value or similar custom header):
>>>
>>> varnishncsa -F "%{HOST}i:%{X-Port}i %h %l %u %t \"%m %U%q %H\" %s %b
>>> \"%{Referer}i\" \"%{User-agent}i\""|sed -e
>>> 's#^www\.##g'|/usr/local/cpanel/bin/splitlogs --main=`hostname`
>>> --mainout=/usr/local/apache/logs/access_log
>>> varnishncsa -F "%{HOST}i %{%s}t %b ."|sed -e
>>> 's#^www\.##g'|/usr/local/cpanel/bin/splitlogs --main=`hostname`
>>> --suffix=-bytes_log
>>>
>>> The above pipes the requests to the splitlogs binary which queues then
>>> writes to separate logs per domain, that are later processed by the
>>> cPanel log stats apps. Either way, I believe you need an intermediary
>>> script to queue and write the log entries per domain. While looking into
>>> this process, I ran across this little tidbit which you may find of use
>>> https://gist.github.com/garlandkr/4954272 for logstash style output.
>>>
>>>
>>>
>>
>> _______________________________________________
>> varnish-misc mailing list
>> varnish-misc@varnish-cache.org
>> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
>>
>
>
> _______________________________________________
> varnish-misc mailing list
> varnish-misc@varnish-cache.org
> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
>

I haven't done any actual benchmarks for the resource usage, however as an
admin I would rather not have numerous processes for a single data channel
that needs to be parsed and split.

My apologies for sort of hijacking this thread with a cPanel integration.
It's just so widely used, that I'm sure others here have had the need or
thought at some point and figured I would share my findings. As cPanel
servers are known to house hundreds of domains, per domain varnishncsa
simply isn't an option, which is why I chose to piggyback on splitlogs as
it's already a supported option of theirs.

On Nov 13, 2016 17:44, "Guillaume Quintard" <guillaume@varnish-software.com>
wrote:

> Is the resources argument that compelling? I find it cleaner to have one
> ncsa per domain, plus the ncsas will read from memory, which is super fast.
>
> Sure you can then split logs afterwards using awk or whatever, but that's
> adding an extra layer and serializing a process that doesn't need to be.
>
> But I'm not an admin, so I may be off.
>
> On Nov 13, 2016 16:27, "Andrei" <lagged@gmail.com> wrote:
>
>> Hello,
>>
>> By not using splitlogs or an intermediary script, you will be forced to
>> run multiple instances of varnishncsa, which isn't optimal if you host
>> multiple domains. The more traffic/domains you have, the more resources you
>> will consume on parsing the same data across multiple channels.
>> Yes, varnishncsa supports VSL, however I think your approach is a bit off
>> (no offense). As in, you need to shift the log writing process away
>> from varnishncsa. By doing so, you only have one instance of varnishncsa
>> using resources to gather the data, which is then fed to a parser that
>> handles the per domain log splits and writes. That's where 'splitlogs' came
>> into play.
>>
>> As your question does raise some interest in the cPanel community (myself
>> and Miguel González on this list for example), I threw together a quick
>> Perl script that will in short, pipe and parse data between varnishncsa and
>> the splitlogs binary for cache hits. This lets splitlogs handle the queued
>> log writes which are later parsed for cPanel bandwidth usage and graphs,
>> webalizer, awstats, logaholic, etc - https://github.com/AndreiG6/vscp
>>
>>
>>
>> On Sun, Nov 13, 2016 at 12:18 PM, Admin Beckspaced <admin@beckspaced.com>
>> wrote:
>>
>>> Same Hello here ;)
>>>
>>> did have a more in depth look in the manual and figured out that
>>> varnishncsa does support VSL query.
>>> so someone could filter on the Request Header and Host
>>>
>>> varnishncsa -q "ReqHeader ~ '^Host: .*example.com'"
>>>
>>> which would produce a log for a specific domain only
>>>
>>> it then would need multiple varnishncsa instances for logging per
>>> domain, which I found here:
>>>
>>> https://kevops.com/2015/11/varnish-logging-per-host-with-init-script/
>>>
>>> I use varnish version 5 and then there would be no need for splitlog and
>>> the logs would be created directly.
>>>
>>> please correct me if I'm wrong?
>>>
>>> thanks for your time & help
>>> Becki
>>>
>>>
>>> Am 12.11.2016 um 17:05 schrieb Andrei:
>>>
>>>> Hello again,
>>>>
>>>> My apologies for not explaining my thoughts better earlier then. Afaik,
>>>> varnishncsa does not have a native method to split output based on
>>>> different parameters. The method I was thinking of was based on piping
>>>> varnishncsa output through splitlogs (or similar) for the log processing
>>>> and writeouts. Since replying earlier, I've got this working on a cPanel
>>>> server with piped logging enabled for Apache using the following two for
>>>> example (X-Port is a custom header set in vcl_recv related to SSL
>>>> offloading, but you can use a static value or similar custom header):
>>>>
>>>> varnishncsa -F "%{HOST}i:%{X-Port}i %h %l %u %t \"%m %U%q %H\" %s %b
>>>> \"%{Referer}i\" \"%{User-agent}i\""|sed -e
>>>> 's#^www\.##g'|/usr/local/cpanel/bin/splitlogs --main=`hostname`
>>>> --mainout=/usr/local/apache/logs/access_log
>>>> varnishncsa -F "%{HOST}i %{%s}t %b ."|sed -e
>>>> 's#^www\.##g'|/usr/local/cpanel/bin/splitlogs --main=`hostname`
>>>> --suffix=-bytes_log
>>>>
>>>> The above pipes the requests to the splitlogs binary which queues then
>>>> writes to separate logs per domain, that are later processed by the
>>>> cPanel log stats apps. Either way, I believe you need an intermediary
>>>> script to queue and write the log entries per domain. While looking into
>>>> this process, I ran across this little tidbit which you may find of use
>>>> https://gist.github.com/garlandkr/4954272 for logstash style output.
>>>>
>>>>
>>>>
>>>
>>> _______________________________________________
>>> varnish-misc mailing list
>>> varnish-misc@varnish-cache.org
>>> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
>>>
>>
>>
>> _______________________________________________
>> varnish-misc mailing list
>> varnish-misc@varnish-cache.org
>> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
>>
>

Hello Andrei,

No offense taken ;) that's the reason we talk to find the best solution!

and I think you are right. It's better to have one instance of
varnishncsa and shift the log writing away.

I tried piping varnishncsa to the apache split-logfile perl script:

/usr/sbin/varnishncsa -f /etc/varnish/varnishncsa-log-format-string -P
/var/run/varnishlog.pid | sed -e 's#^www\.##g' | /usr/bin/split-logfile &

and split-logfile started creating different log files for the vhosts.

but still having a bit of an issue with this approach as most of the
data just gets written to the logs once I kill the background process.
the different log files for the vhosts get created but are mostly empty
and start filling up on killing the varnishncsa main process

perhaps the apache split-logfile is not the same as the splitlogs from
cpanel?
where could I download a version to test?

another thing is the naming of the vhosts. you actually never know what
clients are requesting?
so Baidu spider requests URL like 333.domain.com ... or ww3.domain.com
... wwq.domain.com

in your pipe command I see sed removing the www. subdomain but you can
actually never know what comes in?

in apache I can easily filter all those subdomains and redirect to the
proper domain.

even in varnish VCL I can normalize the host

not sure how to do this in varnishncsa? is there a way to do that?

anyway, I think a single varnishncsa process and shifting the log
writing away is the way to go!
but still needs a bit of tweaking if you ask me ;)

thanks & greetings
becki


Am 13.11.2016 um 16:01 schrieb Andrei:
> Hello,
>
> By not using splitlogs or an intermediary script, you will be forced
> to run multiple instances of varnishncsa, which isn't optimal if you
> host multiple domains. The more traffic/domains you have, the more
> resources you will consume on parsing the same data across multiple
> channels. Yes, varnishncsa supports VSL, however I think your approach
> is a bit off (no offense). As in, you need to shift the log writing
> process away from varnishncsa. By doing so, you only have one instance
> of varnishncsa using resources to gather the data, which is then fed
> to a parser that handles the per domain log splits and writes. That's
> where 'splitlogs' came into play.
>
> As your question does raise some interest in the cPanel community
> (myself and Miguel González on this list for example), I threw
> together a quick Perl script that will in short, pipe and parse data
> between varnishncsa and the splitlogs binary for cache hits. This lets
> splitlogs handle the queued log writes which are later parsed for
> cPanel bandwidth usage and graphs, webalizer, awstats, logaholic, etc
> - https://github.com/AndreiG6/vscp
>
>
>
> On Sun, Nov 13, 2016 at 12:18 PM, Admin Beckspaced
> <admin@beckspaced.com <mailto:admin@beckspaced.com>> wrote:
>
> Same Hello here ;)
>
> did have a more in depth look in the manual and figured out that
> varnishncsa does support VSL query.
> so someone could filter on the Request Header and Host
>
> varnishncsa -q "ReqHeader ~ '^Host: .*example.com
> <http://example.com>'"
>
> which would produce a log for a specific domain only
>
> it then would need multiple varnishncsa instances for logging per
> domain, which I found here:
>
> https://kevops.com/2015/11/varnish-logging-per-host-with-init-script/
> <https://kevops.com/2015/11/varnish-logging-per-host-with-init-script/>
>
> I use varnish version 5 and then there would be no need for
> splitlog and the logs would be created directly.
>
> please correct me if I'm wrong?
>
> thanks for your time & help
> Becki
>
>
> Am 12.11.2016 um 17:05 schrieb Andrei:
>
> Hello again,
>
> My apologies for not explaining my thoughts better earlier
> then. Afaik,
> varnishncsa does not have a native method to split output based on
> different parameters. The method I was thinking of was based
> on piping
> varnishncsa output through splitlogs (or similar) for the log
> processing
> and writeouts. Since replying earlier, I've got this working
> on a cPanel
> server with piped logging enabled for Apache using the
> following two for
> example (X-Port is a custom header set in vcl_recv related to SSL
> offloading, but you can use a static value or similar custom
> header):
>
> varnishncsa -F "%{HOST}i:%{X-Port}i %h %l %u %t \"%m %U%q %H\"
> %s %b
> \"%{Referer}i\" \"%{User-agent}i\""|sed -e
> 's#^www\.##g'|/usr/local/cpanel/bin/splitlogs --main=`hostname`
> --mainout=/usr/local/apache/logs/access_log
> varnishncsa -F "%{HOST}i %{%s}t %b ."|sed -e
> 's#^www\.##g'|/usr/local/cpanel/bin/splitlogs --main=`hostname`
> --suffix=-bytes_log
>
> The above pipes the requests to the splitlogs binary which
> queues then
> writes to separate logs per domain, that are later processed
> by the
> cPanel log stats apps. Either way, I believe you need an
> intermediary
> script to queue and write the log entries per domain. While
> looking into
> this process, I ran across this little tidbit which you may
> find of use
> https://gist.github.com/garlandkr/4954272
> <https://gist.github.com/garlandkr/4954272> for logstash style
> output.
>
>
>
>
> _______________________________________________
> varnish-misc mailing list
> varnish-misc@varnish-cache.org <mailto:varnish-misc@varnish-cache.org>
> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
> <https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc>
>
>


_______________________________________________
varnish-misc mailing list
varnish-misc@varnish-cache.org
https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc

> even in varnish VCL I can normalize the host
>
> not sure how to do this in varnishncsa? is there a way to do that?

In VCL you can std.log the normalized host, and use a custom pattern
in varnishncsa to pick it up.

Dridi

_______________________________________________
varnish-misc mailing list
varnish-misc@varnish-cache.org
https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc

Am 15.11.2016 um 10:54 schrieb Dridi Boukelmoune:
>> even in varnish VCL I can normalize the host
>>
>> not sure how to do this in varnishncsa? is there a way to do that?
> In VCL you can std.log the normalized host, and use a custom pattern
> in varnishncsa to pick it up.
>
> Dridi
>
>
hello Dridi,

thanks a lot for your reply. and how would i do that?

I already normalize the host in vcl_recv

//normalize the req.http.host
set req.http.host = regsub(req.http.Host, "^www\.", "");

but not seeing that normalized host in varnishncsa ...

a bit more hints and info would be nice ;)

thanks & greetings
becki



_______________________________________________
varnish-misc mailing list
varnish-misc@varnish-cache.org
https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc

Probably because varnishncsa sees the ReqHeader before it is normalized.
Try putting the normalized host in req.http.x-host, and filtering on that.

--
Guillaume Quintard

On Tue, Nov 15, 2016 at 4:31 PM, Admin Beckspaced <admin@beckspaced.com>
wrote:

>
> Am 15.11.2016 um 10:54 schrieb Dridi Boukelmoune:
>
>> even in varnish VCL I can normalize the host
>>>
>>> not sure how to do this in varnishncsa? is there a way to do that?
>>>
>> In VCL you can std.log the normalized host, and use a custom pattern
>> in varnishncsa to pick it up.
>>
>> Dridi
>>
>>
>> hello Dridi,
>
> thanks a lot for your reply. and how would i do that?
>
> I already normalize the host in vcl_recv
>
> //normalize the req.http.host
> set req.http.host = regsub(req.http.Host, "^www\.", "");
>
> but not seeing that normalized host in varnishncsa ...
>
> a bit more hints and info would be nice ;)
>
> thanks & greetings
> becki
>
>
>
> _______________________________________________
> varnish-misc mailing list
> varnish-misc@varnish-cache.org
> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
>

> a bit more hints and info would be nice ;)

man vmod_std
man varnishncsa

That's how much "nice" I'm willing to do :p

Dridi

_______________________________________________
varnish-misc mailing list
varnish-misc@varnish-cache.org
https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc

Am 15.11.2016 um 16:51 schrieb Dridi Boukelmoune:
>> a bit more hints and info would be nice ;)
> man vmod_std
> man varnishncsa
>
> That's how much "nice" I'm willing to do :p
>
> Dridi
>
>
ok. first I want to say thanks for being nice and pointing me to the man
pages.

after a bit of reading I finally found the parts I was looking for:

import std;

sub vcl_recv {

std.log(“myhost:” + regsub(req.http.Host, "^www\.", "") );

}

in varnishncsa:

# varnishncsa -F ‘%h %l %u %t “%r” %s %b “%{Referer}i”
“%{User-agent}i %{VCL_Log:myhost}x’


not yet tested but I think this is what Dridi was pointing to?

Ok ... I also do understand that people need to read manuals a.k.a. RTFM ;)

But is a mailing list ONLY here to get the finger pointed to MAN and RTFM???

I'm part of quite a few mailing lists and if I see others having
problems I already encountered and solved earlier I'm always happy to
give them the bits and pieces they need.
why just point them to MAN or RTFM if I know that specific info?

Isn't that what a mailing list is there for? To help others?

Your thoughts please ;)

thanks & greetings
becki





_______________________________________________
varnish-misc mailing list
varnish-misc@varnish-cache.org
https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc

> ok. first I want to say thanks for being nice and pointing me to the man
> pages.

Sometimes it's hard to even know that man pages even exist (I'm
looking at vsl and vsl-query in particular) so I tend to do that a lot ;)

> not yet tested but I think this is what Dridi was pointing to?

Yes, that was the suggestion.

> Ok ... I also do understand that people need to read manuals a.k.a. RTFM ;)
>
> But is a mailing list ONLY here to get the finger pointed to MAN and RTFM???

No but it saves some time and typing for lazy me.

> Isn't that what a mailing list is there for? To help others?
>
> Your thoughts please ;)

Now that the finger is pointed at me, while I agree that I often point
to the manual, I'm rarely saying to RTFM. I can be weeks between to
peeks at the misc list for me, so sometimes I get extra lazy. I'm also
sure you learned more than just how to pick the normalized host by
looking at varnishncsa's manual.

I could have been very nasty and not mention vmod_std(3) because
after all, I had already hinted "std.log" in a previous email ;)

I gave you a minimalist solution [1], you didn't know how to do it, I
pointed to the right manuals since you already had the recipe. The
little push that should get the ball rolling.

Then if it doesn't work out as expected I can still try helping further.
I'm usually reluctant to give a turn-key solution (except in the docs)
and that's probably a habit I got from being a trainer. DYI and I will
back you up, sort of thing.

Cheers

[1] https://www.varnish-cache.org/lists/pipermail/varnish-misc/2016-November/025402.html

_______________________________________________
varnish-misc mailing list
varnish-misc@varnish-cache.org
https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc

>> Isn't that what a mailing list is there for? To help others?
>>
>> Your thoughts please ;)
> Now that the finger is pointed at me, while I agree that I often point
> to the manual, I'm rarely saying to RTFM. I can be weeks between to
> peeks at the misc list for me, so sometimes I get extra lazy. I'm also
> sure you learned more than just how to pick the normalized host by
> looking at varnishncsa's manual.
>
> I could have been very nasty and not mention vmod_std(3) because
> after all, I had already hinted "std.log" in a previous email ;)
>
> I gave you a minimalist solution [1], you didn't know how to do it, I
> pointed to the right manuals since you already had the recipe. The
> little push that should get the ball rolling.
>
> Then if it doesn't work out as expected I can still try helping further.
> I'm usually reluctant to give a turn-key solution (except in the docs)
> and that's probably a habit I got from being a trainer. DYI and I will
> back you up, sort of thing.
>
> Cheers
Thanks lazy you,

glad you gave me the right hints and pointers. good to know that it's
only your extra laziness ;)

anyway, no offense taken! in general I'm in the same boat with you and
DYI instead of turn-key solutions
sometimes I just wish for a short explanation to get the ball a better
rolling instead of a MAN and then asking for further help.

anyway, thanks again for mentioning std.log which took me on the right
path ...

wish you lots of health & happiness so you can perhaps kick that extra
bit of laziness and just be plain lazy and perhaps add a tiny
explanation / example the next time ;)

greetings & all the best
Becki


_______________________________________________
varnish-misc mailing list
varnish-misc@varnish-cache.org
https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc

On 15.11.2016 17:57, Admin Beckspaced wrote:
>
> Am 15.11.2016 um 16:51 schrieb Dridi Boukelmoune:
>>> a bit more hints and info would be nice ;)
>> man vmod_std
>> man varnishncsa
>>
>> That's how much "nice" I'm willing to do :p
>>
>> Dridi
>>
>>
> ok. first I want to say thanks for being nice and pointing me to the
> man pages.
>
> after a bit of reading I finally found the parts I was looking for:
>
> import std;
>
> sub vcl_recv {
>
> std.log(“myhost:” + regsub(req.http.Host, "^www\.", "") );
>
> }
>
> in varnishncsa:
>
> # varnishncsa -F ‘%h %l %u %t “%r” %s %b “%{Referer}i”
> “%{User-agent}i %{VCL_Log:myhost}x’
>
>
> not yet tested but I think this is what Dridi was pointing to?
>
Hello again,

sorry, it has been a while but I just thought to finish the thread I
started and point to the solution I decided to go with at last.

The question in the beginning was: How can I split the varnishncsa logs
per domain.

My first thinking was to use the query -q option, e.g. varnishncsa -q
"ReqHeader ~ '^Host: .*example.com'"

But this approach would end up in a lot varnishncsa instances, as
pointed out by Andrei, and also the problem with not being able to
normalize the Request Host header.
Then Dridi pointed me to man vmod_std and using std.log in VCL, which
was the final bit needed ;)

So here's my current solution:

I run a single instance of varnishncsa with the following params:

VARNISHLOG_PARAMS="-f /etc/varnish/varnishncsa-log-format-string -a -w
/var/log/varnish/varnish.log"

the varnishncsa-log-format-string is as follows:

%{VCL_Log:myhost}x %h %l %u %t "%r" %s %b "%{Referer}i" "%{User-agent}i"

at the beginning VCL_Log:key The value set by std.log("key:value")
in VCL, more on that later

My varnish sits in front of Apache with 30 something different domains.
Currently I don't use varnish for all domains, but have setup varnish
VCL in such a way that I can filter on the domains and decide to cache
with varnish or just skip caching and pass to the apache backend.

My varnish VCL is based on the varnish boilerplate which I found on the net:

http://verticalprogramming.com/2013/09/15/varnish-virtual-host-boilerplate

So if I decide to cache a particular domain with varnish I can normalize
the Request Host header in VCL

sub vcl_recv {

if (req.http.host ~ "somedomain\.com") {

# tilde ~ uses regex
if (req.http.host ~ "^(www.)?somedomain\.com$") {

//normalize the req.http.host
set req.http.host = regsub(req.http.Host, "^www\.", "");

std.log("myhost:" + req.http.Host );
...

so this std.log(myhost:somedomian.com) gets picked up by varnishncsa and
the custom format string, see above.

which then produces a nice & steady varnish.log file with a normalized
host at the beginning for domains I want to cache and an empty space if
I don't want to.

then we got the split-logfile from apache:

https://httpd.apache.org/docs/2.4/programs/split-logfile.html

which was exactly made for a setup with the host names at the very
beginning of the log file. Only thing someone needs to take care of is
that the logfiles will get created in the directory where the script is
run. so therefore I created a small bash script wrapper split-logfile.sh
which first changes to the right working directory:

#!/bin/bash
cd /var/log/varnish
/usr/bin/split-logfile < varnish.log

and on the daily logrotate on the /var/log/varnish/varnish.log I added
the following:

/var/log/varnish/varnish.log {
...
prerotate
/var/log/varnish/split-logfile.sh
endscript
...
}

so before the varnish.log gets rotated split-logfile.sh gets called and
creates the different log files per normalized host and an access.log
for all the requests without a hostname at the beginning.
After a view logrotate runs the /var/log/varnish/ could look like that:

mydomain.com.log
myotherdomain.com.log
access.log
varnish.log
varnish.log-20161130
varnish.log-20161201
varnish.log-20161202

which finally give me exactly what I wanted! A single instance of
varnishncsa producing a varnish.log for all domains.
a per domain split via split-logfile.sh on each logrotate run resulting
in log files per domain ready to use with webalizer, which gets also
called on prerotate the logfile:

/var/log/varnish/mydomain.com.log {
...
prerotate
/usr/bin/webalizer -qc /etc/webalizer/mydomain.conf
endscript
...
}

perhaps this might help someone here looking for something similar?

thanks & greetings
becki



_______________________________________________
varnish-misc mailing list
varnish-misc@varnish-cache.org
https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc

Thanks for reporting back, great job!

On Dec 2, 2016 19:57, "Admin Beckspaced" <admin@beckspaced.com> wrote:

>
> On 15.11.2016 17:57, Admin Beckspaced wrote:
>
>>
>> Am 15.11.2016 um 16:51 schrieb Dridi Boukelmoune:
>>
>>> a bit more hints and info would be nice ;)
>>>>
>>> man vmod_std
>>> man varnishncsa
>>>
>>> That's how much "nice" I'm willing to do :p
>>>
>>> Dridi
>>>
>>>
>>> ok. first I want to say thanks for being nice and pointing me to the man
>> pages.
>>
>> after a bit of reading I finally found the parts I was looking for:
>>
>> import std;
>>
>> sub vcl_recv {
>>
>> std.log(“myhost:” + regsub(req.http.Host, "^www\.", "") );
>>
>> }
>>
>> in varnishncsa:
>>
>> # varnishncsa -F ‘%h %l %u %t “%r” %s %b “%{Referer}i”
>> “%{User-agent}i %{VCL_Log:myhost}x’
>>
>>
>> not yet tested but I think this is what Dridi was pointing to?
>>
>> Hello again,
>
> sorry, it has been a while but I just thought to finish the thread I
> started and point to the solution I decided to go with at last.
>
> The question in the beginning was: How can I split the varnishncsa logs
> per domain.
>
> My first thinking was to use the query -q option, e.g. varnishncsa -q
> "ReqHeader ~ '^Host: .*example.com'"
>
> But this approach would end up in a lot varnishncsa instances, as pointed
> out by Andrei, and also the problem with not being able to normalize the
> Request Host header.
> Then Dridi pointed me to man vmod_std and using std.log in VCL, which was
> the final bit needed ;)
>
> So here's my current solution:
>
> I run a single instance of varnishncsa with the following params:
>
> VARNISHLOG_PARAMS="-f /etc/varnish/varnishncsa-log-format-string -a -w
> /var/log/varnish/varnish.log"
>
> the varnishncsa-log-format-string is as follows:
>
> %{VCL_Log:myhost}x %h %l %u %t "%r" %s %b "%{Referer}i" "%{User-agent}i"
>
> at the beginning VCL_Log:key The value set by std.log("key:value") in
> VCL, more on that later
>
> My varnish sits in front of Apache with 30 something different domains.
> Currently I don't use varnish for all domains, but have setup varnish VCL
> in such a way that I can filter on the domains and decide to cache with
> varnish or just skip caching and pass to the apache backend.
>
> My varnish VCL is based on the varnish boilerplate which I found on the
> net:
>
> http://verticalprogramming.com/2013/09/15/varnish-virtual-host-boilerplate
>
> So if I decide to cache a particular domain with varnish I can normalize
> the Request Host header in VCL
>
> sub vcl_recv {
>
> if (req.http.host ~ "somedomain\.com") {
>
> # tilde ~ uses regex
> if (req.http.host ~ "^(www.)?somedomain\.com$") {
>
> //normalize the req.http.host
> set req.http.host = regsub(req.http.Host, "^www\.", "");
>
> std.log("myhost:" + req.http.Host );
> ...
>
> so this std.log(myhost:somedomian.com) gets picked up by varnishncsa and
> the custom format string, see above.
>
> which then produces a nice & steady varnish.log file with a normalized
> host at the beginning for domains I want to cache and an empty space if I
> don't want to.
>
> then we got the split-logfile from apache:
>
> https://httpd.apache.org/docs/2.4/programs/split-logfile.html
>
> which was exactly made for a setup with the host names at the very
> beginning of the log file. Only thing someone needs to take care of is that
> the logfiles will get created in the directory where the script is run. so
> therefore I created a small bash script wrapper split-logfile.sh which
> first changes to the right working directory:
>
> #!/bin/bash
> cd /var/log/varnish
> /usr/bin/split-logfile < varnish.log
>
> and on the daily logrotate on the /var/log/varnish/varnish.log I added the
> following:
>
> /var/log/varnish/varnish.log {
> ...
> prerotate
> /var/log/varnish/split-logfile.sh
> endscript
> ...
> }
>
> so before the varnish.log gets rotated split-logfile.sh gets called and
> creates the different log files per normalized host and an access.log for
> all the requests without a hostname at the beginning.
> After a view logrotate runs the /var/log/varnish/ could look like that:
>
> mydomain.com.log
> myotherdomain.com.log
> access.log
> varnish.log
> varnish.log-20161130
> varnish.log-20161201
> varnish.log-20161202
>
> which finally give me exactly what I wanted! A single instance of
> varnishncsa producing a varnish.log for all domains.
> a per domain split via split-logfile.sh on each logrotate run resulting in
> log files per domain ready to use with webalizer, which gets also called on
> prerotate the logfile:
>
> /var/log/varnish/mydomain.com.log {
> ...
> prerotate
> /usr/bin/webalizer -qc /etc/webalizer/mydomain.conf
> endscript
> ...
> }
>
> perhaps this might help someone here looking for something similar?
>
> thanks & greetings
> becki
>
>
>
> _______________________________________________
> varnish-misc mailing list
> varnish-misc@varnish-cache.org
> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc