Mailing List Archive

Hello Hardik,

> Is it any change require regarding this? can we change any tunable parameter ? any VCL flow change can help ?

Can you please tell us more about which version of Varnish you are using?

$ varnishd -V

Please also specify which OS you are running Varnish on, and how you
installed Varnish.

Thanks,
Dridi
_______________________________________________
varnish-misc mailing list
varnish-misc@varnish-cache.org
https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc

Hi Dridi,

Varnish version - 6.0.1
OS - centos 7

I checked out vanish 6.0.1 source code, built rpm and installed.

Please let me know if more information is required.

Thank you
Hardik

On Tue, 12 Mar 2019 at 16:20, Dridi Boukelmoune <dridi@varni.sh> wrote:

> Hello Hardik,
>
> > Is it any change require regarding this? can we change any tunable
> parameter ? any VCL flow change can help ?
>
> Can you please tell us more about which version of Varnish you are using?
>
> $ varnishd -V
>
> Please also specify which OS you are running Varnish on, and how you
> installed Varnish.
>
> Thanks,
> Dridi
>

On Tue, Mar 12, 2019 at 11:54 AM Hardik <hetardik.p@gmail.com> wrote:
>
> Hi Dridi,
>
> Varnish version - 6.0.1
> OS - centos 7
>
> I checked out vanish 6.0.1 source code, built rpm and installed.
>
> Please let me know if more information is required.

Do you have a reliable way to reproduce this?

If so, can you describe it and does it still happen with 6.0.3?

Thanks,
Dridi
_______________________________________________
varnish-misc mailing list
varnish-misc@varnish-cache.org
https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc

Hi,

On 12/03/2019 11:07, Hardik wrote:
> *Error cases when ReqAcct is not present in varnishlog*
>
> 1.
> -   VSL            timeout
> -   End            synt
>
> 2.
> -   VSL            store overflow
> -   End            synth

The VSL clients (varnishlog and friends) store transactions in the client
program's memory when varnishd comes close to overwriting the respective region
of the VSL shared memory ring.

The -L and -T options allow you to specify these limits, see for example the
varnishlog man page.

I _suspect_ you are seeing this with the -g request or -g session transaction
grouping (please specify how you invoke the respective command). If yes, an
alternative would be to reduce the time the VCL client needs to store open
transactions by switching to the vxid or raw grouping (-g argument).

hth, Nils
_______________________________________________
varnish-misc mailing list
varnish-misc@varnish-cache.org
https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc

Hi Dridi,

We should able to recreate with load and mobile requests. I have not tried
with 6.0.3.

@Nils,
We are seeing issue with both varnishlog and varnishlog with -g option. But
here problem is, shared memory it self does not have ReqAcct tag I think (
please correct me if I am wrong). Because all the clients which are reading
shm all are getting same thing..means no ReqAcct. But yes I am agree that
impact with "varnishlog -g session" is more.

So If shared memory it self has no ReqAcct tag then all clients will also
not get right ? How to fix this problem ? Please help with some details
which I can understand because we are loosing bills for which we are
serving traffic...!

*Normal varnish command we use to grep running logs*
varnishlog -g request -q "ReqURL ~ '/abc/xyz'"

*command uses to read shared memory directly for billing*
varnishlog -g session
--> we are already planing to use "varnishlog -g xvid" for billing api.
Because I understood is, -g session option is taking more time to arrange
in particular order and delivery final output. Please help with some more
detail.. It will really helpful.

Thank you
Hardik


On Tue, 12 Mar 2019 at 17:32, Nils Goroll <slink@schokola.de> wrote:

> Hi,
>
> On 12/03/2019 11:07, Hardik wrote:
> > *Error cases when ReqAcct is not present in varnishlog*
> >
> > 1.
> > - VSL timeout
> > - End synt
> >
> > 2.
> > - VSL store overflow
> > - End synth
>
> The VSL clients (varnishlog and friends) store transactions in the client
> program's memory when varnishd comes close to overwriting the respective
> region
> of the VSL shared memory ring.
>
> The -L and -T options allow you to specify these limits, see for example
> the
> varnishlog man page.
>
> I _suspect_ you are seeing this with the -g request or -g session
> transaction
> grouping (please specify how you invoke the respective command). If yes, an
> alternative would be to reduce the time the VCL client needs to store open
> transactions by switching to the vxid or raw grouping (-g argument).
>
> hth, Nils
>

On Wed, Mar 13, 2019 at 3:56 PM Hardik <hetardik.p@gmail.com> wrote:
>
> Hi Dridi,
>
> We should able to recreate with load and mobile requests. I have not tried with 6.0.3.

I guess there's no need for that. Your varnishlog setup is stretched
too thin to cope with your load.

I was completely oblivious to the problems slink spotted right away...

> @Nils,
> We are seeing issue with both varnishlog and varnishlog with -g option. But here problem is, shared memory it self does not have ReqAcct tag I think ( please correct me if I am wrong). Because all the clients which are reading shm all are getting same thing..means no ReqAcct. But yes I am agree that impact with "varnishlog -g session" is more.
>
> So If shared memory it self has no ReqAcct tag then all clients will also not get right ? How to fix this problem ? Please help with some details which I can understand because we are loosing bills for which we are serving traffic...!

You could try avoiding grouping, use varnishncsa with a custom format,
overall store and process less data in memory.

> Normal varnish command we use to grep running logs
> varnishlog -g request -q "ReqURL ~ '/abc/xyz'"
>
> command uses to read shared memory directly for billing
> varnishlog -g session
> --> we are already planing to use "varnishlog -g xvid" for billing api. Because I understood is, -g session option is taking more time to arrange in particular order and delivery final output. Please help with some more detail.. It will really helpful.

There are few practical uses of -g session for live traffic. This
works better on offline logs for example when examining traffic
post-mortem.

If only ReqAcct is important, and the rest of the information you need
is available on the request side, you should definitely stop using the
-g option, and use the -c option to further limit internal processing.

If you need information from the backend transactions too, try to
figure how you could make this information available on the client
side.

Dridi
_______________________________________________
varnish-misc mailing list
varnish-misc@varnish-cache.org
https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc

Thanks a lot Dridi & Team for details..

You could try avoiding grouping, use varnishncsa with a custom format,
overall store and process less data in memory.
*-I was analyzing varnishncsa logs as well and could see problem with it
also. Means missing ReqAcct sometime.*
*Does that means shared memory its self does not have ReqAcct tag ? *
*-Is grouping happen to clients memory ? means client reads from shared
memory and then does grouping in client's memory ? Please give some details
so based on that I can arrange the things. Also I am curious to understand.
Any document link or point out any coding part, I will go through that as
well.*

If only ReqAcct is important, and the rest of the information you need
is available on the request side, you should definitely stop using the
-g option, and use the -c option to further limit internal processing.
*-Yes, we are reading now request side only ( till now was reading session
also but found out other alternative so stopped that after your comment ). *
*-I have few doubts here.. is there a difference between NOT assigning -g
option to g_arg(means keep default to xvid ) and assigning -c option ? *
*-Currently I have stopped using -g option in service which is reading shm
so kept to default option. To by pass berequest tags I have started using
"VSL_t_bereq" structure member. If -c option is still better than this then
I will test out that also. Please suggest..*

Thank you
Hardik

On Wed, 13 Mar 2019 at 22:49, Dridi Boukelmoune <dridi@varni.sh> wrote:

> On Wed, Mar 13, 2019 at 3:56 PM Hardik <hetardik.p@gmail.com> wrote:
> >
> > Hi Dridi,
> >
> > We should able to recreate with load and mobile requests. I have not
> tried with 6.0.3.
>
> I guess there's no need for that. Your varnishlog setup is stretched
> too thin to cope with your load.
>
> I was completely oblivious to the problems slink spotted right away...
>
> > @Nils,
> > We are seeing issue with both varnishlog and varnishlog with -g option.
> But here problem is, shared memory it self does not have ReqAcct tag I
> think ( please correct me if I am wrong). Because all the clients which are
> reading shm all are getting same thing..means no ReqAcct. But yes I am
> agree that impact with "varnishlog -g session" is more.
> >
> > So If shared memory it self has no ReqAcct tag then all clients will
> also not get right ? How to fix this problem ? Please help with some
> details which I can understand because we are loosing bills for which we
> are serving traffic...!
>
> You could try avoiding grouping, use varnishncsa with a custom format,
> overall store and process less data in memory.
>
> > Normal varnish command we use to grep running logs
> > varnishlog -g request -q "ReqURL ~ '/abc/xyz'"
> >
> > command uses to read shared memory directly for billing
> > varnishlog -g session
> > --> we are already planing to use "varnishlog -g xvid" for billing api.
> Because I understood is, -g session option is taking more time to arrange
> in particular order and delivery final output. Please help with some more
> detail.. It will really helpful.
>
> There are few practical uses of -g session for live traffic. This
> works better on offline logs for example when examining traffic
> post-mortem.
>
> If only ReqAcct is important, and the rest of the information you need
> is available on the request side, you should definitely stop using the
> -g option, and use the -c option to further limit internal processing.
>
> If you need information from the backend transactions too, try to
> figure how you could make this information available on the client
> side.
>
> Dridi
>

On Sun, Mar 17, 2019 at 1:12 PM Hardik <hetardik.p@gmail.com> wrote:
>
> Thanks a lot Dridi & Team for details..

Before I can answer your questions, can you explain exactly what you
are trying to do?

We could probably give you better advice if we knew what you need to collect.

Dridi
_______________________________________________
varnish-misc mailing list
varnish-misc@varnish-cache.org
https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc

Hi Dridi,

I am reading few tags for Billing purpose. I have added VMOD for this.In
vmod currently I was passing "-g session" option to varnish callback
function. But *I found out that not getting ReqAcct tag. Also I am missing
lots of logs itself.*

After with discussion with you guys it seems problem is due to "-g session"
option. So I removed that and tested again and looks better then before.
Reduced frequency of missing ReqAcct and Log loss decreased a lot.

Now, my doubt, Is it better not to use any option and keep default setting,
or to use -c option ? So I can minimize log loss as many as I can.

Here is small from vmod which is reading shared memory (*it was lot easier
in varnish 3 because I was able to call dispatch function directly*),
vut = VUT_Init(argv[0], 1, argv, &s);
vut->dispatch_f = VarnishLog::handler;
vut->dispatch_priv = this;
vut->g_arg = 3; <---- I commented out this now ( -g session )
vut->sighup = 1;
vut->sighup_f = VarnishLog::sighup;
VUT_Setup(vut);
VUT_Main(vut);
VUT_Fini(&vut);


I can not change above whole setup but can modify few things in that.

Now, If you can answer previous questions will be really helpful.
Particularly how -g session option creating problem ?

Thank you
Hardik



On Mon, 18 Mar 2019 at 20:43, Dridi Boukelmoune <dridi@varni.sh> wrote:

> On Sun, Mar 17, 2019 at 1:12 PM Hardik <hetardik.p@gmail.com> wrote:
> >
> > Thanks a lot Dridi & Team for details..
>
> Before I can answer your questions, can you explain exactly what you
> are trying to do?
>
> We could probably give you better advice if we knew what you need to
> collect.
>
> Dridi
>

On Mon, Mar 18, 2019 at 5:53 PM Hardik <hetardik.p@gmail.com> wrote:
>
> Hi Dridi,
>
> I am reading few tags for Billing purpose. I have added VMOD for this.In vmod currently I was passing "-g session" option to varnish callback function. But I found out that not getting ReqAcct tag. Also I am missing lots of logs itself.
>
> After with discussion with you guys it seems problem is due to "-g session" option. So I removed that and tested again and looks better then before. Reduced frequency of missing ReqAcct and Log loss decreased a lot.
>
> Now, my doubt, Is it better not to use any option and keep default setting, or to use -c option ? So I can minimize log loss as many as I can.
>
> Here is small from vmod which is reading shared memory (it was lot easier in varnish 3 because I was able to call dispatch function directly),
> vut = VUT_Init(argv[0], 1, argv, &s);
> vut->dispatch_f = VarnishLog::handler;
> vut->dispatch_priv = this;
> vut->g_arg = 3; <---- I commented out this now ( -g session )
> vut->sighup = 1;
> vut->sighup_f = VarnishLog::sighup;
> VUT_Setup(vut);
> VUT_Main(vut);
> VUT_Fini(&vut);

So you wrote your own log utility in C++? Wow!

> I can not change above whole setup but can modify few things in that.
>
> Now, If you can answer previous questions will be really helpful. Particularly how -g session option creating problem ?

Sorry but this is not what I was looking for. [1]

Can you give me a list of log records you need to collect? And
possibly how you are trying to group them if they come from different
transactions? There's a lot we can do without building a new utility.

Dridi

[1] http://xyproblem.info/
_______________________________________________
varnish-misc mailing list
varnish-misc@varnish-cache.org
https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc

Hi Dridi,

Can you give me a list of log records you need to collect?

SLT_Timestamp :
SLT_ReqStart :
SLT_ReqMethod :
SLT_ReqURL:
SLT_ReqProtocol :
SLT_RespStatus :
SLT_ReqHeader :
SLT_RespHeader :
SLT_ReqAcct :
SLT_BereqAcct :
SLT_VCL_Log :

And
possibly how you are trying to group them if they come from different
transactions?
*I am reading based xid ( by FD ). Means reading full records per fd.*

Please let me know if any other information I can provide..


If this is not related to my problem still I am curious to know how
grouping is happening. You can point out some code or links with some
details, I will go through.

Thank you
Hardik

On Tue, 19 Mar 2019 at 00:43, Dridi Boukelmoune <dridi@varni.sh> wrote:

> On Mon, Mar 18, 2019 at 5:53 PM Hardik <hetardik.p@gmail.com> wrote:
> >
> > Hi Dridi,
> >
> > I am reading few tags for Billing purpose. I have added VMOD for this.In
> vmod currently I was passing "-g session" option to varnish callback
> function. But I found out that not getting ReqAcct tag. Also I am missing
> lots of logs itself.
> >
> > After with discussion with you guys it seems problem is due to "-g
> session" option. So I removed that and tested again and looks better then
> before. Reduced frequency of missing ReqAcct and Log loss decreased a lot.
> >
> > Now, my doubt, Is it better not to use any option and keep default
> setting, or to use -c option ? So I can minimize log loss as many as I can.
> >
> > Here is small from vmod which is reading shared memory (it was lot
> easier in varnish 3 because I was able to call dispatch function directly),
> > vut = VUT_Init(argv[0], 1, argv, &s);
> > vut->dispatch_f = VarnishLog::handler;
> > vut->dispatch_priv = this;
> > vut->g_arg = 3; <---- I commented out this now ( -g session )
> > vut->sighup = 1;
> > vut->sighup_f = VarnishLog::sighup;
> > VUT_Setup(vut);
> > VUT_Main(vut);
> > VUT_Fini(&vut);
>
> So you wrote your own log utility in C++? Wow!
>
> > I can not change above whole setup but can modify few things in that.
> >
> > Now, If you can answer previous questions will be really helpful.
> Particularly how -g session option creating problem ?
>
> Sorry but this is not what I was looking for. [1]
>
> Can you give me a list of log records you need to collect? And
> possibly how you are trying to group them if they come from different
> transactions? There's a lot we can do without building a new utility.
>
> Dridi
>
> [1] http://xyproblem.info/
>

Hi,

Thank you for the details, more questions to come.

On Tue, Mar 19, 2019 at 1:42 AM Hardik <hetardik.p@gmail.com> wrote:
>
> Hi Dridi,
>
> Can you give me a list of log records you need to collect?
>
> SLT_Timestamp :

Do you need all timestamps or a specific metric?

> SLT_ReqStart :
> SLT_ReqMethod :
> SLT_ReqURL:
> SLT_ReqProtocol :
> SLT_RespStatus :
> SLT_ReqHeader :
> SLT_RespHeader :
> SLT_ReqAcct :
> SLT_BereqAcct :

Do you need the BereqAcct records for all transactions? Including cache hits?

This one is tricky in terms of billing.

> SLT_VCL_Log :
>
> And
> possibly how you are trying to group them if they come from different
> transactions?

You can do the grouping with the -g option, but that didn't go well
for you so that's what I'm trying to figure out.

> I am reading based xid ( by FD ). Means reading full records per fd.

What does FD mean here? File descriptor? From ReqStart?

> Please let me know if any other information I can provide..
>
>
> If this is not related to my problem still I am curious to know how grouping is happening. You can point out some code or links with some details, I will go through.

Well, utilities like varnishlog or varnishncsa accumulate transactions
via libvarnishapi in memory (which may take a long time) and then
libvarnishapi presents them in order. So utilities don't assume this
logic and simply get the data presented to them in a callback
function.

That's where timeouts, overruns or transaction limits may result in
data loss since slow log consumers don't slow down Varnish, and
Varnish isn't slowed down by logs more than writing them to memory
requires.



Dridi
_______________________________________________
varnish-misc mailing list
varnish-misc@varnish-cache.org
https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc

Hi Dridi,

Do you need all timestamps or a specific metric?
Regarding timestamp, want to read two tags,
Timestamp Start: *1516269224.184112* 0.000000 0.000000
Timestamp Resp: 1516269224.184920 *0.000808 0.000087*

Do you need the BereqAcct records for all transactions? Including cache
hits?
Sorry it is my mistake. I am not reading any of the beck-end records. So
can ignore BereqAcct.
I need fields from Req records only.

What does FD mean here? File descriptor? From ReqStart?
Yes, Its file descriptor. And yes reading from ReqStart till ReqAcct. Using
switch case to read needed records.
ex:
switch (tag)
{
case SLT_ReqURL:


Using below condition as I do not want to read Sess and Bereq records.(
Please correct me if doing something wrong)
if (!VSL_Match(vsl, t->c) || (t->type == VSL_t_bereq)
|| (t->type == VSL_t_sess))
continue;

Well, utilities like varnishlog or varnishncsa accumulate transactions
via libvarnishapi in memory (which may take a long time) and then
libvarnishapi presents them in order. So utilities don't assume this
logic and simply get the data presented to them in a callback
function.
Thanks for this..
It means varnishlog (any utility) asks for logs which will come via
libvarnishapi. Just trying to confirm that what i understood from above is
correct. Here, memory means varnishd process memory or utility's memory ?
As per my understadning its varnishd's memory because libvarnishapi will be
holding logs to arrange.
In case of varnish 3, we were able to call direct dispatch function so it
was faster, if my understanding is correct.

Are log loss and missing of ReqAcct tag both problems are related ?
I am thinking missing of ReqAcct is due to timeout problem also. For
example mobile request are taking more time to serve.. In this case we
might hit timeout of reading logs from VSL right ? I do not know in detail.
But if buy modifying any timeout value can help please point out, I will
try that sure.

Thank you
hardik


On Tue, 19 Mar 2019 at 22:22, Dridi Boukelmoune <dridi@varni.sh> wrote:

> Hi,
>
> Thank you for the details, more questions to come.
>
> On Tue, Mar 19, 2019 at 1:42 AM Hardik <hetardik.p@gmail.com> wrote:
> >
> > Hi Dridi,
> >
> > Can you give me a list of log records you need to collect?
> >
> > SLT_Timestamp :
>
> Do you need all timestamps or a specific metric?
>
> > SLT_ReqStart :
> > SLT_ReqMethod :
> > SLT_ReqURL:
> > SLT_ReqProtocol :
> > SLT_RespStatus :
> > SLT_ReqHeader :
> > SLT_RespHeader :
> > SLT_ReqAcct :
> > SLT_BereqAcct :
>
> Do you need the BereqAcct records for all transactions? Including cache
> hits?
>
> This one is tricky in terms of billing.
>
> > SLT_VCL_Log :
> >
> > And
> > possibly how you are trying to group them if they come from different
> > transactions?
>
> You can do the grouping with the -g option, but that didn't go well
> for you so that's what I'm trying to figure out.
>
> > I am reading based xid ( by FD ). Means reading full records per fd.
>
> What does FD mean here? File descriptor? From ReqStart?
>
> > Please let me know if any other information I can provide..
> >
> >
> > If this is not related to my problem still I am curious to know how
> grouping is happening. You can point out some code or links with some
> details, I will go through.
>
> Well, utilities like varnishlog or varnishncsa accumulate transactions
> via libvarnishapi in memory (which may take a long time) and then
> libvarnishapi presents them in order. So utilities don't assume this
> logic and simply get the data presented to them in a callback
> function.
>
> That's where timeouts, overruns or transaction limits may result in
> data loss since slow log consumers don't slow down Varnish, and
> Varnish isn't slowed down by logs more than writing them to memory
> requires.
>
>
>
> Dridi
>

On Wed, Mar 20, 2019 at 10:40 AM Hardik <hetardik.p@gmail.com> wrote:
>
> Hi Dridi,
>
> Do you need all timestamps or a specific metric?
> Regarding timestamp, want to read two tags,
> Timestamp Start: 1516269224.184112 0.000000 0.000000
> Timestamp Resp: 1516269224.184920 0.000808 0.000087
>
> Do you need the BereqAcct records for all transactions? Including cache hits?
> Sorry it is my mistake. I am not reading any of the beck-end records. So can ignore BereqAcct.
> I need fields from Req records only.

Ok, in this case you can probably get away with just varnishncsa to
collect all you need.

No grouping (the default -g vxid), client mode (-c) only, with a
custom -F format to grab only what you need.

This should help reduce the churn to the point where you lose data.

If you struggle with this, I can help you later with that, but start
by reading the following manuals:

- varnishncsa
- vsl
- vsl-query

For example, the format for the timestamps you wish to collect would
look like this:

> %{VSL:Timestamp:Start[1]}x %{VSL:Timestamp:Resp[2]}x %{VSL:Timestamp:Resp[3]}x

Rinse and repeat for all the things you need to capture for the logs,
put them in the order you prefer and off you go. No need to write your
own utility.

> What does FD mean here? File descriptor? From ReqStart?
> Yes, Its file descriptor. And yes reading from ReqStart till ReqAcct. Using switch case to read needed records.

If you already work with VXIDs, the FD becomes redundant.



Dridi
_______________________________________________
varnish-misc mailing list
varnish-misc@varnish-cache.org
https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc

Thanks Dridi.

I was ill and off work for few days so did not reply.

Hey Dridi, I am already using varnishncsa. And use case perspective I am
much familiar. I am using below configuration for varnishncsa. This logs I
am calling as raw-logs/Apache and providing to customer directly without
any modification.Means no processing on this.

#the format of the log lines
FORMAT='-F "%h %v %u %t \"%m %U%q %H\" %s %O \"%{Referer}i\"
\"%{User-agent}i\" %S [%{VCL_Log:x-product}x lg=\"%{Accept-Language}i\"
fwd=\"%{X-Forwarded-For}i\" %{VCL_Log:namespace}x
%{VCL_Log:x-cache-hit}x/%{VCL_Log:x-revalidate-cache}x
%{VCL_Log:sc-substatus}x]

Note- few parameter I have added in varnishncsa to get that information
ex:%S.

I have written other utility because want to parse all the fields based on
billing requirements. I have tried removing "-g session" option and now log
loss reduced a lot(I can say 95%).

If you can help on my previous reply/doubts then will be really helpful (
because billing really difficult to change billing setup).I am only facing
two issues here, 1. log loss 2. missing of ReqAcct ( which is also case in
varnishncsa).

If I am doing something wrong other then "-g session" option in the code I
have pasted before from utility please point out.

Thank you
Hardik

On Wed, 20 Mar 2019 at 21:02, Dridi Boukelmoune <dridi@varni.sh> wrote:

> On Wed, Mar 20, 2019 at 10:40 AM Hardik <hetardik.p@gmail.com> wrote:
> >
> > Hi Dridi,
> >
> > Do you need all timestamps or a specific metric?
> > Regarding timestamp, want to read two tags,
> > Timestamp Start: 1516269224.184112 0.000000 0.000000
> > Timestamp Resp: 1516269224.184920 0.000808 0.000087
> >
> > Do you need the BereqAcct records for all transactions? Including cache
> hits?
> > Sorry it is my mistake. I am not reading any of the beck-end records. So
> can ignore BereqAcct.
> > I need fields from Req records only.
>
> Ok, in this case you can probably get away with just varnishncsa to
> collect all you need.
>
> No grouping (the default -g vxid), client mode (-c) only, with a
> custom -F format to grab only what you need.
>
> This should help reduce the churn to the point where you lose data.
>
> If you struggle with this, I can help you later with that, but start
> by reading the following manuals:
>
> - varnishncsa
> - vsl
> - vsl-query
>
> For example, the format for the timestamps you wish to collect would
> look like this:
>
> %{VSL:Timestamp:Resp[3]}x
>
> Rinse and repeat for all the things you need to capture for the logs,
> put them in the order you prefer and off you go. No need to write your
> own utility.
>
> > What does FD mean here? File descriptor? From ReqStart?
> > Yes, Its file descriptor. And yes reading from ReqStart till ReqAcct.
> Using switch case to read needed records.
>
> If you already work with VXIDs, the FD becomes redundant.
>
>
>
> Dridi
>

Hi Dridi / Team,

Gentle reminder..

Thank you
Hardik

On Thu, 4 Apr 2019 at 11:33, Hardik <hetardik.p@gmail.com> wrote:

> Thanks Dridi.
>
> I was ill and off work for few days so did not reply.
>
> Hey Dridi, I am already using varnishncsa. And use case perspective I am
> much familiar. I am using below configuration for varnishncsa. This logs I
> am calling as raw-logs/Apache and providing to customer directly without
> any modification.Means no processing on this.
>
> #the format of the log lines
> FORMAT='-F "%h %v %u %t \"%m %U%q %H\" %s %O \"%{Referer}i\"
> \"%{User-agent}i\" %S [%{VCL_Log:x-product}x lg=\"%{Accept-Language}i\"
> fwd=\"%{X-Forwarded-For}i\" %{VCL_Log:namespace}x
> %{VCL_Log:x-cache-hit}x/%{VCL_Log:x-revalidate-cache}x
> %{VCL_Log:sc-substatus}x]
>
> Note- few parameter I have added in varnishncsa to get that information
> ex:%S.
>
> I have written other utility because want to parse all the fields based on
> billing requirements. I have tried removing "-g session" option and now log
> loss reduced a lot(I can say 95%).
>
> If you can help on my previous reply/doubts then will be really helpful (
> because billing really difficult to change billing setup).I am only facing
> two issues here, 1. log loss 2. missing of ReqAcct ( which is also case in
> varnishncsa).
>
> If I am doing something wrong other then "-g session" option in the code I
> have pasted before from utility please point out.
>
> Thank you
> Hardik
>
> On Wed, 20 Mar 2019 at 21:02, Dridi Boukelmoune <dridi@varni.sh> wrote:
>
>> On Wed, Mar 20, 2019 at 10:40 AM Hardik <hetardik.p@gmail.com> wrote:
>> >
>> > Hi Dridi,
>> >
>> > Do you need all timestamps or a specific metric?
>> > Regarding timestamp, want to read two tags,
>> > Timestamp Start: 1516269224.184112 0.000000 0.000000
>> > Timestamp Resp: 1516269224.184920 0.000808 0.000087
>> >
>> > Do you need the BereqAcct records for all transactions? Including cache
>> hits?
>> > Sorry it is my mistake. I am not reading any of the beck-end records.
>> So can ignore BereqAcct.
>> > I need fields from Req records only.
>>
>> Ok, in this case you can probably get away with just varnishncsa to
>> collect all you need.
>>
>> No grouping (the default -g vxid), client mode (-c) only, with a
>> custom -F format to grab only what you need.
>>
>> This should help reduce the churn to the point where you lose data.
>>
>> If you struggle with this, I can help you later with that, but start
>> by reading the following manuals:
>>
>> - varnishncsa
>> - vsl
>> - vsl-query
>>
>> For example, the format for the timestamps you wish to collect would
>> look like this:
>>
>> > %{VSL:Timestamp:Start[1]}x %{VSL:Timestamp:Resp[2]}x
>> %{VSL:Timestamp:Resp[3]}x
>>
>> Rinse and repeat for all the things you need to capture for the logs,
>> put them in the order you prefer and off you go. No need to write your
>> own utility.
>>
>> > What does FD mean here? File descriptor? From ReqStart?
>> > Yes, Its file descriptor. And yes reading from ReqStart till ReqAcct.
>> Using switch case to read needed records.
>>
>> If you already work with VXIDs, the FD becomes redundant.
>>
>>
>>
>> Dridi
>>
>