Mailing List Archive

How to cache req.body and then use it in response for coming requests in varnish
Hi??I am trying to configure varnish as a simple cache service, which means the content posted by clients could be cached as an object. The content chould be returned in responses for the coming GET requests(with same url...). I am not sure wether this can be realized by VCL configuration.

Any recommendation for this situation? Thanks for any help!

I was noticed vmod-bodyaccess can cache req.body and then it could be used in bereq(https://info.varnish-software.com/blog/caching-post-requests-with-varnish), but I do not know how to put it into responses for clients.

thanks!
Zafir
________________________________
eyz12315@live.cn
Re: How to cache req.body and then use it in response for coming requests in varnish [ In reply to ]
Hi,

There are a few things to do here:
- make sure that if req.method == "POST", you return(hash) from vcl_recv
(otherwise, the builtin.vcl (
https://github.com/varnish/Varnish-Cache/blob/4.1/bin/varnishd/builtin.vcl#L63)
will pass)
- but before that, add 'set req.http.x-method = req.method' at the top of
vcl_recv, and 'if (bereq.http.x-method == "POST") {set bereq.method =
"POST"; }' at the top of vcl_backend_fetch, because for cacheable request,
varnish will want to set the method to "GET", so we need to reset it back

On top of that:
- you can use req.hash_always_miss (man vcl) to push new content (and avoid
just getting the cached response)
- make sure you only cache when the response is OK, otherwise a wrong POST
request will mask the previous valid ones
- if you are acting upon the URL of the resource itself, I'd say it should
be a PUT/PATCH, rather than a POST, but that's just pedantry at this point
- realize that the content cached is the response returned by the backend,
not the body of the request itself. They may be the same, but maybe not
(there are ways to do it though)

Hope that helps
--
Guillaume Quintard


On Sun, Aug 11, 2019 at 10:42 PM Yuan Zafir <eyz12315@live.cn> wrote:

> Hi?I am trying to configure varnish as a simple cache service, which
> means the content posted by clients could be cached as an object. The
> content chould be returned in responses for the coming GET requests(with
> same url...). I am not sure wether this can be realized by VCL
> configuration.
>
> Any recommendation for this situation? Thanks for any help!
>
> I was noticed vmod-bodyaccess can cache req.body and then it could be used
> in bereq(
> https://info.varnish-software.com/blog/caching-post-requests-with-varnish),
> but I do not know how to put it into responses for clients.
>
> thanks!
> Zafir
> ------------------------------
> eyz12315@live.cn
> _______________________________________________
> varnish-misc mailing list
> varnish-misc@varnish-cache.org
> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
>
Re: Re: How to cache req.body and then use it in response for coming requests in varnish [ In reply to ]
hi Guillaume?

Thanks for your reply.

I think there should be ambiguity description about my question in my first mail.
Actually what I want is to use varnish as a cache service *without backend servers*, clients can push/pull the resource to/from varnish cache storage using specific http method, such as POST/GET.
For example, one client can send " POST /string1 "abcd" " to make a string-"abcd" cached into varnish, other clients could then use "GET /string1" request to get this string. If Get request miss the cache, a 404 error could be sent (through synth command in vcl_miss).

I am not sure whether it's feasible to make the string-"abcd" cached into varnish. Do you have any recommendation for this requrement?
Thanks again for your attention!


Zafir Yuan

________________________________
eyz12315@live.cn

From: Guillaume Quintard<mailto:guillaume@varnish-software.com>
Date: 2019-08-12 14:03
To: Yuan Zafir<mailto:eyz12315@live.cn>
CC: varnish-misc<mailto:varnish-misc@varnish-cache.org>
Subject: Re: How to cache req.body and then use it in response for coming requests in varnish
Hi,

There are a few things to do here:
- make sure that if req.method == "POST", you return(hash) from vcl_recv (otherwise, the builtin.vcl (https://github.com/varnish/Varnish-Cache/blob/4.1/bin/varnishd/builtin.vcl#L63) will pass)
- but before that, add 'set req.http.x-method = req.method' at the top of vcl_recv, and 'if (bereq.http.x-method == "POST") {set bereq.method = "POST"; }' at the top of vcl_backend_fetch, because for cacheable request, varnish will want to set the method to "GET", so we need to reset it back

On top of that:
- you can use req.hash_always_miss (man vcl) to push new content (and avoid just getting the cached response)
- make sure you only cache when the response is OK, otherwise a wrong POST request will mask the previous valid ones
- if you are acting upon the URL of the resource itself, I'd say it should be a PUT/PATCH, rather than a POST, but that's just pedantry at this point
- realize that the content cached is the response returned by the backend, not the body of the request itself. They may be the same, but maybe not (there are ways to do it though)

Hope that helps
--
Guillaume Quintard


On Sun, Aug 11, 2019 at 10:42 PM Yuan Zafir <eyz12315@live.cn<mailto:eyz12315@live.cn>> wrote:
Hi?I am trying to configure varnish as a simple cache service, which means the content posted by clients could be cached as an object. The content chould be returned in responses for the coming GET requests(with same url...). I am not sure wether this can be realized by VCL configuration.

Any recommendation for this situation? Thanks for any help!

I was noticed vmod-bodyaccess can cache req.body and then it could be used in bereq(https://info.varnish-software.com/blog/caching-post-requests-with-varnish), but I do not know how to put it into responses for clients.

thanks!
Zafir
________________________________
eyz12315@live.cn<mailto:eyz12315@live.cn>
_______________________________________________
varnish-misc mailing list
varnish-misc@varnish-cache.org<mailto:varnish-misc@varnish-cache.org>
https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
Re: Re: How to cache req.body and then use it in response for coming requests in varnish [ In reply to ]
Hi,

then you need something like
https://docs.varnish-software.com/varnish-cache-plus/vmods/synthbackend/ (with
the .mirror() function) which was created for this purpose, but it's a
Varnish Plus feature.

It may be possible usable with the new backend error mechanism in the next
version, coupled with some vmod_bodyaccess feature (but you would need a
blob instead of a string)

--
Guillaume Quintard


On Mon, Aug 12, 2019 at 12:57 AM Yuan Zafir <eyz12315@live.cn> wrote:

> hi Guillaume?
>
> Thanks for your reply.
>
> I think there should be ambiguity description about my question in my
> first mail.
> Actually what I want is to use varnish as a cache service *without backend
> servers*, clients can push/pull the resource to/from varnish cache storage
> using specific http method, such as POST/GET.
> For example, one client can send " POST /string1 "abcd" " to make a
> string-"abcd" cached into varnish, other clients could then use "GET
> /string1" request to get this string. If Get request miss the cache, a 404
> error could be sent (through synth command in vcl_miss).
>
> I am not sure whether it's feasible to make the string-"abcd" cached into
> varnish. Do you have any recommendation for this requrement?
> Thanks again for your attention!
>
>
> Zafir Yuan
>
> ------------------------------
> eyz12315@live.cn
>
>
> *From:* Guillaume Quintard <guillaume@varnish-software.com>
> *Date:* 2019-08-12 14:03
> *To:* Yuan Zafir <eyz12315@live.cn>
> *CC:* varnish-misc <varnish-misc@varnish-cache.org>
> *Subject:* Re: How to cache req.body and then use it in response for
> coming requests in varnish
> Hi,
>
> There are a few things to do here:
> - make sure that if req.method == "POST", you return(hash) from vcl_recv
> (otherwise, the builtin.vcl (
> https://github.com/varnish/Varnish-Cache/blob/4.1/bin/varnishd/builtin.vcl#L63)
> will pass)
> - but before that, add 'set req.http.x-method = req.method' at the top of
> vcl_recv, and 'if (bereq.http.x-method == "POST") {set bereq.method =
> "POST"; }' at the top of vcl_backend_fetch, because for cacheable request,
> varnish will want to set the method to "GET", so we need to reset it back
>
> On top of that:
> - you can use req.hash_always_miss (man vcl) to push new content (and
> avoid just getting the cached response)
> - make sure you only cache when the response is OK, otherwise a wrong POST
> request will mask the previous valid ones
> - if you are acting upon the URL of the resource itself, I'd say it should
> be a PUT/PATCH, rather than a POST, but that's just pedantry at this point
> - realize that the content cached is the response returned by the backend,
> not the body of the request itself. They may be the same, but maybe not
> (there are ways to do it though)
>
> Hope that helps
> --
> Guillaume Quintard
>
>
> On Sun, Aug 11, 2019 at 10:42 PM Yuan Zafir <eyz12315@live.cn> wrote:
>
>> Hi?I am trying to configure varnish as a simple cache service, which
>> means the content posted by clients could be cached as an object. The
>> content chould be returned in responses for the coming GET requests(with
>> same url...). I am not sure wether this can be realized by VCL
>> configuration.
>>
>> Any recommendation for this situation? Thanks for any help!
>>
>> I was noticed vmod-bodyaccess can cache req.body and then it could be
>> used in bereq(
>> https://info.varnish-software.com/blog/caching-post-requests-with-varnish),
>> but I do not know how to put it into responses for clients.
>>
>> thanks!
>> Zafir
>> ------------------------------
>> eyz12315@live.cn
>> _______________________________________________
>> varnish-misc mailing list
>> varnish-misc@varnish-cache.org
>> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
>>
>
Re: Re: How to cache req.body and then use it in response for coming requests in varnish [ In reply to ]
Thanks, Guillaume!

Zafir Yuan
________________________________
eyz12315@live.cn

From: Guillaume Quintard<mailto:guillaume@varnish-software.com>
Date: 2019-08-12 22:05
To: Yuan Zafir<mailto:eyz12315@live.cn>
CC: varnish-misc<mailto:varnish-misc@varnish-cache.org>
Subject: Re: Re: How to cache req.body and then use it in response for coming requests in varnish
Hi,

then you need something like https://docs.varnish-software.com/varnish-cache-plus/vmods/synthbackend/ (with the .mirror() function) which was created for this purpose, but it's a Varnish Plus feature.

It may be possible usable with the new backend error mechanism in the next version, coupled with some vmod_bodyaccess feature (but you would need a blob instead of a string)

--
Guillaume Quintard


On Mon, Aug 12, 2019 at 12:57 AM Yuan Zafir <eyz12315@live.cn<mailto:eyz12315@live.cn>> wrote:
hi Guillaume?

Thanks for your reply.

I think there should be ambiguity description about my question in my first mail.
Actually what I want is to use varnish as a cache service *without backend servers*, clients can push/pull the resource to/from varnish cache storage using specific http method, such as POST/GET.
For example, one client can send " POST /string1 "abcd" " to make a string-"abcd" cached into varnish, other clients could then use "GET /string1" request to get this string. If Get request miss the cache, a 404 error could be sent (through synth command in vcl_miss).

I am not sure whether it's feasible to make the string-"abcd" cached into varnish. Do you have any recommendation for this requrement?
Thanks again for your attention!


Zafir Yuan

________________________________
eyz12315@live.cn<mailto:eyz12315@live.cn>

From: Guillaume Quintard<mailto:guillaume@varnish-software.com>
Date: 2019-08-12 14:03
To: Yuan Zafir<mailto:eyz12315@live.cn>
CC: varnish-misc<mailto:varnish-misc@varnish-cache.org>
Subject: Re: How to cache req.body and then use it in response for coming requests in varnish
Hi,

There are a few things to do here:
- make sure that if req.method == "POST", you return(hash) from vcl_recv (otherwise, the builtin.vcl (https://github.com/varnish/Varnish-Cache/blob/4.1/bin/varnishd/builtin.vcl#L63) will pass)
- but before that, add 'set req.http.x-method = req.method' at the top of vcl_recv, and 'if (bereq.http.x-method == "POST") {set bereq.method = "POST"; }' at the top of vcl_backend_fetch, because for cacheable request, varnish will want to set the method to "GET", so we need to reset it back

On top of that:
- you can use req.hash_always_miss (man vcl) to push new content (and avoid just getting the cached response)
- make sure you only cache when the response is OK, otherwise a wrong POST request will mask the previous valid ones
- if you are acting upon the URL of the resource itself, I'd say it should be a PUT/PATCH, rather than a POST, but that's just pedantry at this point
- realize that the content cached is the response returned by the backend, not the body of the request itself. They may be the same, but maybe not (there are ways to do it though)

Hope that helps
--
Guillaume Quintard


On Sun, Aug 11, 2019 at 10:42 PM Yuan Zafir <eyz12315@live.cn<mailto:eyz12315@live.cn>> wrote:
Hi?I am trying to configure varnish as a simple cache service, which means the content posted by clients could be cached as an object. The content chould be returned in responses for the coming GET requests(with same url...). I am not sure wether this can be realized by VCL configuration.

Any recommendation for this situation? Thanks for any help!

I was noticed vmod-bodyaccess can cache req.body and then it could be used in bereq(https://info.varnish-software.com/blog/caching-post-requests-with-varnish), but I do not know how to put it into responses for clients.

thanks!
Zafir
________________________________
eyz12315@live.cn<mailto:eyz12315@live.cn>
_______________________________________________
varnish-misc mailing list
varnish-misc@varnish-cache.org<mailto:varnish-misc@varnish-cache.org>
https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc