Mailing List Archive

Hi Uday,

Ultimately, you'll probably want to learn and use this vmod:
https://github.com/Dridi/libvmod-querystring , but in the meantime, we can
use a quick hack.

Essentially, we don't need to modify the URL, but we can just alter the
cache key computation.

By default, the key logic looks like this:

sub vcl_hash {
hash_data(req.url);
if (req.http.host) {
hash_data(req.http.host);
} else {
hash_data(server.ip);
}
return (lookup);
}

(It's done by default if you don't have a vcl_hash section in your VCL)
We can tweak it slightly so that we ignore the whole querystring:

sub vcl_hash {
hash_data(regsub(req.url, "\?.*",""));
if (req.http.host) {
hash_data(req.http.host);
} else {
hash_data(server.ip);
}
return (lookup);
}

It's crude, but should do the job. To use it, just copy the code above in
your VCL, for example just after the vcl_recv definition (not inside it
though). Of course, if you already have a vlc_hash definition in your code,
you'll need to modify that one, instead of adding a new one.

Relevant documentation:
- https://varnish-cache.org/docs/trunk/users-guide/vcl-hashing.html
- https://www.varnish-software.com/developers/tutorials/varnish-builtin-vcl/

Last note: it would probably be better if the tomcat server didn't need
that unique parameter, or at the very least, if Varnish could just add it
itself rather than relying on client information as you're caching
something public using something that was user-specific, so there's
potential for snafus here.

Hope that helps,


On Tue, May 30, 2023, 03:45 Uday Kumar <uday.polu@indiamart.com> wrote:

> Hello everyone,
>
> In our system, we're currently using Varnish Cache in front of our Tomcat
> Server for caching content.
>
> As part of our new requirement, we've started passing a unique parameter
> with every URL. The addition of this unique parameter in each request is
> causing a cache miss, as Varnish treats each request as distinct due to the
> difference in the parameter. Our intent is to have Varnish ignore this
> specific parameter for caching purposes, so that it can treat similar
> requests with different unique parameters as identical for caching purposes.
>
> Expected Functionality of Varnish:
>
> 1. We need Varnish to ignore the unique parameter when determining if a
> request is in the cache or while caching a request.
>
> 2. We also need Varnish to retain this unique parameter in the request
> URL when it's passed along to the Tomcat Server.
>
> We're looking for a way to modify our Varnish configuration to address the
> above issues, your assistance would be greatly appreciated.
>
>
> Thanks & Regards
> Uday Kumar
> _______________________________________________
> varnish-misc mailing list
> varnish-misc@varnish-cache.org
> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
>

Hello Guillaume,

Thank you so much for your help, will try modifying vcl_hash as suggested!


> Last note: it would probably be better if the tomcat server didn't need
> that unique parameter, or at the very least, if Varnish could just add it
> itself rather than relying on client information as you're caching
> something public using something that was user-specific, so there's
> potential for snafus here.
>

Could you please also suggest how to configure Varnish so that Varnish can
add Unique Parameter by itself??

> Could you please also suggest how to configure Varnish so that Varnish
can add Unique Parameter by itself??

We'd need more context, is there any kind of check that tomcat does on this
parameter, does it need to have a specific length, or match a regex?
If we know that, we can have Varnish check the user request to make sure
it's valid, and potentially generate its own parameter.

But it all depends on what Tomcat expects from that parameter.

--
Guillaume Quintard


On Tue, May 30, 2023 at 11:18?PM Uday Kumar <uday.polu@indiamart.com> wrote:

> Hello Guillaume,
>
> Thank you so much for your help, will try modifying vcl_hash as suggested!
>
>
>> Last note: it would probably be better if the tomcat server didn't need
>> that unique parameter, or at the very least, if Varnish could just add
>> it itself rather than relying on client information as you're caching
>> something public using something that was user-specific, so there's
>> potential for snafus here.
>>
>
> Could you please also suggest how to configure Varnish so that Varnish
> can add Unique Parameter by itself??
>

Hello,

We would like to configure varnish to create unique parameter such that its
value should be of 20 characters (alphanumeric characters that are URL
safe).

On Wed, May 31, 2023, 13:34 Guillaume Quintard <guillaume.quintard@gmail.com>
wrote:

> > Could you please also suggest how to configure Varnish so that Varnish
> can add Unique Parameter by itself??
>
> We'd need more context, is there any kind of check that tomcat does on
> this parameter, does it need to have a specific length, or match a regex?
> If we know that, we can have Varnish check the user request to make sure
> it's valid, and potentially generate its own parameter.
>
> But it all depends on what Tomcat expects from that parameter.
>
> --
> Guillaume Quintard
>
>
> On Tue, May 30, 2023 at 11:18?PM Uday Kumar <uday.polu@indiamart.com>
> wrote:
>
>> Hello Guillaume,
>>
>> Thank you so much for your help, will try modifying vcl_hash as suggested!
>>
>>
>>> Last note: it would probably be better if the tomcat server didn't need
>>> that unique parameter, or at the very least, if Varnish could just add
>>> it itself rather than relying on client information as you're caching
>>> something public using something that was user-specific, so there's
>>> potential for snafus here.
>>>
>>
>> Could you please also suggest how to configure Varnish so that Varnish
>> can add Unique Parameter by itself??
>>
>

Does it need to be unique? can't we just get away with
"aaaaaaaaaaaaaaaaaaaa"?

the crude VCL code would look like:
set req.url = regsub(req.url, "\?.*","?yourparam=aaaaaaaaaaaaaaaaaaaa");

i.e. getting rid of the whole query string and just putting yours in place.

--
Guillaume Quintard


On Wed, May 31, 2023 at 11:07?AM Uday Kumar <uday.polu@indiamart.com> wrote:

> Hello,
>
> We would like to configure varnish to create unique parameter such that
> its value should be of 20 characters (alphanumeric characters that are URL
> safe).
>
> On Wed, May 31, 2023, 13:34 Guillaume Quintard <
> guillaume.quintard@gmail.com> wrote:
>
>> > Could you please also suggest how to configure Varnish so that Varnish
>> can add Unique Parameter by itself??
>>
>> We'd need more context, is there any kind of check that tomcat does on
>> this parameter, does it need to have a specific length, or match a regex?
>> If we know that, we can have Varnish check the user request to make sure
>> it's valid, and potentially generate its own parameter.
>>
>> But it all depends on what Tomcat expects from that parameter.
>>
>> --
>> Guillaume Quintard
>>
>>
>> On Tue, May 30, 2023 at 11:18?PM Uday Kumar <uday.polu@indiamart.com>
>> wrote:
>>
>>> Hello Guillaume,
>>>
>>> Thank you so much for your help, will try modifying vcl_hash as
>>> suggested!
>>>
>>>
>>>> Last note: it would probably be better if the tomcat server didn't need
>>>> that unique parameter, or at the very least, if Varnish could just add
>>>> it itself rather than relying on client information as you're caching
>>>> something public using something that was user-specific, so there's
>>>> potential for snafus here.
>>>>
>>>
>>> Could you please also suggest how to configure Varnish so that Varnish
>>> can add Unique Parameter by itself??
>>>
>>

On 5/31/23 20:06, Uday Kumar wrote:
>
> We would like to configure varnish to create unique parameter such that
> its value should be of 20 characters (alphanumeric characters that are
> URL safe).

I have used VMOD uuid to set unique values in each request:

https://github.com/otto-de/libvmod-uuid

The standard UUID string is 36 characters long. If you really have to
have exactly 20 characters, you could take the last 3 blocks of hex
characters, removing the hyphens. It would take a bit of regex slinging.

For example from: 6ba7b810-9dad-11d1-80b4-00c04fd430c8
extract: 11d180b400c04fd430c8

Then you'd have 80 bits represented in 20 hex characters, instead of the
128 bits represented by the standard string. The 80 bits are all
obtained from the random number generator (the urandom device on Linux),
so I suppose that would be enough to ensure uniqueness.


Best,
Geoff
--
** * * UPLEX - Nils Goroll Systemoptimierung

Scheffelstraße 32
22301 Hamburg

Tel +49 40 2880 5731
Mob +49 176 636 90917
Fax +49 40 42949753

http://uplex.de

> Does it need to be unique? can't we just get away with
> "aaaaaaaaaaaaaaaaaaaa"?
>

Our Requirements:
1. New Parameter should be *appended *to already existing parameters in
Query String. (should not replace entire query string)
2. Parameter Value *Must be Unique for each request* (ideally unique
randomness is preferred)
3. Allowed Characters are Alphanumeric which are *URL safe* [can be
lowercase, uppercase in case of alphabets]
4. Characters can be repeated in parameter value EX: Gn4lT*Y*gBgpPaRi6hw6*Y*S
(here, Y is repeated) But as mentioned above the value must be unique as a
whole.

Ex: Parameter value for 1st request can be "Gn4lT*Y*gBgpPaRi6hw6*Y*S",
2nd request can be
"G34lTYgBgpPaRi6hyaaF" and so on


Thanks & Regards
Uday Kumar

Thanks, so, to make things clean you are going to need to use a couple of
vmods, which means being able to compile them first:
- https://github.com/otto-de/libvmod-uuid as Geoff offered
- https://github.com/Dridi/libvmod-querystring that will allow easy
manipulation of the querystring

unfortunately, the install-vmod tool that is bundled into the Varnish
docker image isn't able to cleanly compile/install them. I'll have a look
this week-end if I can, or at least I'll open a ticket on
https://github.com/varnish/docker-varnish

But, if you are able to install those two, then your life is easy:
- once you receive a request, you can start by creating a unique ID,
which'll be the the vcl equivalent of `uuidgen | sed -E
's/(\w+)-(\w+)-(\w+)-(\w+).*/\1\2\3\4/'` (without having testing it,
probably `regsub(uuid.uuid_v4(), "s/(\w+)-(\w+)-(\w+)-(\w+).*",
"\1\2\3\4/"`)
- then just add/replace the parameter in the querystring with
vmod_querystring

and...that's about it?

Problem is getting the vmods to compile/install which I can help with this
week-end. There's black magic that you can do using regex to manipulate
querystring, but it's a terrible idea.

--
Guillaume Quintard


On Wed, May 31, 2023 at 6:48?PM Uday Kumar <uday.polu@indiamart.com> wrote:

>
> Does it need to be unique? can't we just get away with
>> "aaaaaaaaaaaaaaaaaaaa"?
>>
>
> Our Requirements:
> 1. New Parameter should be *appended *to already existing parameters in
> Query String. (should not replace entire query string)
> 2. Parameter Value *Must be Unique for each request* (ideally unique
> randomness is preferred)
> 3. Allowed Characters are Alphanumeric which are *URL safe* [can be
> lowercase, uppercase in case of alphabets]
> 4. Characters can be repeated in parameter value EX: Gn4lT*Y*gBgpPaRi6hw6
> *Y*S (here, Y is repeated) But as mentioned above the value must be
> unique as a whole.
>
> Ex: Parameter value for 1st request can be "Gn4lT*Y*gBgpPaRi6hw6*Y*S",
> 2nd request can be
> "G34lTYgBgpPaRi6hyaaF" and so on
>
>
> Thanks & Regards
> Uday Kumar
>

On 6/1/23 03:47, Uday Kumar wrote:
>
> Our Requirements:
[...]
> 3. Allowed Characters are Alphanumeric which are *URL safe* [can be
> lowercase, uppercase in case of alphabets]
[...]
This time you didn't mention a requirement of exactly 20 characters. If
that's not strictly required, it's worth pointing out that the hyphen
character '-' is also URL safe.

So in that case you could just use the standard 36-character
representation of a UUID, with its four hyphens. It would save you the
trouble of regex matching and substitution, as Guillaume spelled out.


Best,
Geoff
--
** * * UPLEX - Nils Goroll Systemoptimierung

Scheffelstraße 32
22301 Hamburg

Tel +49 40 2880 5731
Mob +49 176 636 90917
Fax +49 40 42949753

http://uplex.de

Thanks for the prompt response!

Thanks & Regards
Uday Kumar


On Thu, Jun 1, 2023 at 11:12?AM Guillaume Quintard <
guillaume.quintard@gmail.com> wrote:

> Thanks, so, to make things clean you are going to need to use a couple of
> vmods, which means being able to compile them first:
> - https://github.com/otto-de/libvmod-uuid as Geoff offered
> - https://github.com/Dridi/libvmod-querystring that will allow easy
> manipulation of the querystring
>
> unfortunately, the install-vmod tool that is bundled into the Varnish
> docker image isn't able to cleanly compile/install them. I'll have a look
> this week-end if I can, or at least I'll open a ticket on
> https://github.com/varnish/docker-varnish
>
> But, if you are able to install those two, then your life is easy:
> - once you receive a request, you can start by creating a unique ID,
> which'll be the the vcl equivalent of `uuidgen | sed -E
> 's/(\w+)-(\w+)-(\w+)-(\w+).*/\1\2\3\4/'` (without having testing it,
> probably `regsub(uuid.uuid_v4(), "s/(\w+)-(\w+)-(\w+)-(\w+).*",
> "\1\2\3\4/"`)
> - then just add/replace the parameter in the querystring with
> vmod_querystring
>
> and...that's about it?
>
> Problem is getting the vmods to compile/install which I can help with this
> week-end. There's black magic that you can do using regex to manipulate
> querystring, but it's a terrible idea.
>
> --
> Guillaume Quintard
>
>
> On Wed, May 31, 2023 at 6:48?PM Uday Kumar <uday.polu@indiamart.com>
> wrote:
>
>>
>> Does it need to be unique? can't we just get away with
>>> "aaaaaaaaaaaaaaaaaaaa"?
>>>
>>
>> Our Requirements:
>> 1. New Parameter should be *appended *to already existing parameters in
>> Query String. (should not replace entire query string)
>> 2. Parameter Value *Must be Unique for each request* (ideally unique
>> randomness is preferred)
>> 3. Allowed Characters are Alphanumeric which are *URL safe* [can be
>> lowercase, uppercase in case of alphabets]
>> 4. Characters can be repeated in parameter value EX: Gn4lT*Y*gBgpPaRi6hw6
>> *Y*S (here, Y is repeated) But as mentioned above the value must be
>> unique as a whole.
>>
>> Ex: Parameter value for 1st request can be "Gn4lT*Y*gBgpPaRi6hw6*Y*S",
>> 2nd request can be
>> "G34lTYgBgpPaRi6hyaaF" and so on
>>
>>
>> Thanks & Regards
>> Uday Kumar
>>
>

Hi all,

Turns out install-vmod works, just needed to grab the right tarballs and
have the right dependencies installed. Here's the Dockerfile I used:
FROM varnish:7.3

USER root
RUN set -e; \
EXTRA_DEPS="autoconf-archive libossp-uuid-dev"; \
apt-get update; \
apt-get -y install $VMOD_DEPS $EXTRA_DEPS libossp-uuid16 libuuid1
/pkgs/*.deb; \
# vmod_querystring
install-vmod
https://github.com/Dridi/libvmod-querystring/releases/download/v2.0.3/vmod-querystring-2.0.3.tar.gz;
\
# vmod_uuid
install-vmod
https://github.com/otto-de/libvmod-uuid/archive/refs/heads/master.tar.gz; \
apt-get -y purge --auto-remove $VMOD_DEPS $EXTRA_DEPS varnish-dev; \
rm -rf /var/lib/apt/lists/*
USER varnish

and here's the VCL:

vcl 4.1;

import querystring;
import uuid;

backend default {
.host = "localhost";
.port = "4444";
}

sub vcl_init {
new qf = querystring.filter(sort = true);
qf.add_string("myparam");
}

# clear the url from param as it goes in
sub vcl_recv {
# clear myparam from the incoming url
set req.url = qf.apply(mode = drop);
}

# add the querystring parameter back if we go to the backend
sub vcl_backend_fetch {
# create the unique string
set bereq.http.mynewparam = regsub(uuid.uuid_v4(), "^(.{20}).*", "\1");

# add our own myparam
if (bereq.url ~ "\?") {
set bereq.url = bereq.url + "&myparam=" + bereq.http.mynewparam;
} else {
set bereq.url = bereq.url + "?myparam=" + bereq.http.mynewparam;
}
}

It's a bit crude, but it fulfills your requirements. Make sure you test it
though.

--
Guillaume Quintard


On Thu, Jun 1, 2023 at 6:10?AM Uday Kumar <uday.polu@indiamart.com> wrote:

> Thanks for the prompt response!
>
> Thanks & Regards
> Uday Kumar
>
>
> On Thu, Jun 1, 2023 at 11:12?AM Guillaume Quintard <
> guillaume.quintard@gmail.com> wrote:
>
>> Thanks, so, to make things clean you are going to need to use a couple of
>> vmods, which means being able to compile them first:
>> - https://github.com/otto-de/libvmod-uuid as Geoff offered
>> - https://github.com/Dridi/libvmod-querystring that will allow easy
>> manipulation of the querystring
>>
>> unfortunately, the install-vmod tool that is bundled into the Varnish
>> docker image isn't able to cleanly compile/install them. I'll have a look
>> this week-end if I can, or at least I'll open a ticket on
>> https://github.com/varnish/docker-varnish
>>
>> But, if you are able to install those two, then your life is easy:
>> - once you receive a request, you can start by creating a unique ID,
>> which'll be the the vcl equivalent of `uuidgen | sed -E
>> 's/(\w+)-(\w+)-(\w+)-(\w+).*/\1\2\3\4/'` (without having testing it,
>> probably `regsub(uuid.uuid_v4(), "s/(\w+)-(\w+)-(\w+)-(\w+).*",
>> "\1\2\3\4/"`)
>> - then just add/replace the parameter in the querystring with
>> vmod_querystring
>>
>> and...that's about it?
>>
>> Problem is getting the vmods to compile/install which I can help with
>> this week-end. There's black magic that you can do using regex to
>> manipulate querystring, but it's a terrible idea.
>>
>> --
>> Guillaume Quintard
>>
>>
>> On Wed, May 31, 2023 at 6:48?PM Uday Kumar <uday.polu@indiamart.com>
>> wrote:
>>
>>>
>>> Does it need to be unique? can't we just get away with
>>>> "aaaaaaaaaaaaaaaaaaaa"?
>>>>
>>>
>>> Our Requirements:
>>> 1. New Parameter should be *appended *to already existing parameters in
>>> Query String. (should not replace entire query string)
>>> 2. Parameter Value *Must be Unique for each request* (ideally unique
>>> randomness is preferred)
>>> 3. Allowed Characters are Alphanumeric which are *URL safe* [can be
>>> lowercase, uppercase in case of alphabets]
>>> 4. Characters can be repeated in parameter value EX: Gn4lT*Y*
>>> gBgpPaRi6hw6*Y*S (here, Y is repeated) But as mentioned above the value
>>> must be unique as a whole.
>>>
>>> Ex: Parameter value for 1st request can be "Gn4lT*Y*gBgpPaRi6hw6*Y*S",
>>> 2nd request can be
>>> "G34lTYgBgpPaRi6hyaaF" and so on
>>>
>>>
>>> Thanks & Regards
>>> Uday Kumar
>>>
>>

Hello Guillaume,

Thanks for the update!


(It's done by default if you don't have a vcl_hash section in your VCL)
>>>> We can tweak it slightly so that we ignore the whole querystring:
>>>> sub vcl_hash {
>>>> hash_data(regsub(req.url, "\?.*",""));
>>>> if (req.http.host) {
>>>> hash_data(req.http.host);
>>>> } else {
>>>> hash_data(server.ip);
>>>> }
>>>> return (lookup);
>>>> }
>>>>
>>>
Would like to discuss about above suggestion.

*FYI:*
*In our current vcl_hash subroutine, we didnt had any return lookup
statement in production , and the code is as below*
#Working
sub vcl_hash{
hash_data(req.url);
hash_data(req.http.Accept-Encoding);
}
The above code is *working without any issues on production even without
return (lookup)* statement.

For our new requirement * to ignore the parameter in URL while caching, * as
per your suggestion we have made changes to the vcl_hash subroutine, new
code is as below.

#Not Working
sub vcl_hash{
set req.http.hash-url = regsuball(req.url, "traceId=.*?(\&|$)", "");
hash_data(req.http.hash-url);
unset req.http.hash-url;
hash_data(req.http.Accept-Encoding);
}

The above code is *not hashing the URL by ignoring traceId (not as
expected)** but if I add return lookup at the end of subroutine its working
as expected.*

#Working Code
sub vcl_hash{
set req.http.hash-url = regsuball(req.url, "traceId=.*?(\&|$)", "");
hash_data(req.http.hash-url);
unset req.http.hash-url;
hash_data(req.http.Accept-Encoding);
*return (lookup);*
}


*I have few doubts to be clarified:*
1. May I know what difference return (lookup) statement makes?
2. Will there be any side effects with modified code, if I use return
(lookup)? (Because original code was not causing any issue even without
return lookup in production)

Hi,

Relevant documentation:
- https://varnish-cache.org/docs/trunk/users-guide/vcl-hashing.html
- https://www.varnish-software.com/developers/tutorials/varnish-builtin-vcl/
- https://varnish-cache.org/docs/trunk/users-guide/vcl-built-in-code.html

Essentially: if you don't use a return statement, then the built-in vcl
code is executed, and so the logic will be different with and without that
statement.

You wrote that the code isn't working, but don't explain further, which
makes it hard to debug, my best guess is that you're hashing too much
because of the built-in code.
One thing you can do is this:
```
sub vcl_deliver {
set resp.http.req-hash = req.hash;
...
}
```
That will allow you to see objects get the same hash, or a different one.
On that topic, I'm pretty certain that hashing the Accept-Encoding header
is useless and will fragment your cache needlessly, as Varnish already
takes that header into account implicitly.

Note that the vcl I shared in my last email doesn't have a vcl_hash
function because it relies entirely on modifying the url before it is
hashed by the built-in vcl.

Hope that helps.

--
Guillaume Quintard


On Mon, Jun 5, 2023 at 4:31?AM Uday Kumar <uday.polu@indiamart.com> wrote:

> Hello Guillaume,
>
> Thanks for the update!
>
>
> (It's done by default if you don't have a vcl_hash section in your VCL)
>>>>> We can tweak it slightly so that we ignore the whole querystring:
>>>>> sub vcl_hash {
>>>>> hash_data(regsub(req.url, "\?.*",""));
>>>>> if (req.http.host) {
>>>>> hash_data(req.http.host);
>>>>> } else {
>>>>> hash_data(server.ip);
>>>>> }
>>>>> return (lookup);
>>>>> }
>>>>>
>>>>
> Would like to discuss about above suggestion.
>
> *FYI:*
> *In our current vcl_hash subroutine, we didnt had any return lookup
> statement in production , and the code is as below*
> #Working
> sub vcl_hash{
> hash_data(req.url);
> hash_data(req.http.Accept-Encoding);
> }
> The above code is *working without any issues on production even without
> return (lookup)* statement.
>
> For our new requirement * to ignore the parameter in URL while caching, * as
> per your suggestion we have made changes to the vcl_hash subroutine, new
> code is as below.
>
> #Not Working
> sub vcl_hash{
> set req.http.hash-url = regsuball(req.url, "traceId=.*?(\&|$)", "");
> hash_data(req.http.hash-url);
> unset req.http.hash-url;
> hash_data(req.http.Accept-Encoding);
> }
>
> The above code is *not hashing the URL by ignoring traceId (not as
> expected)** but if I add return lookup at the end of subroutine its
> working as expected.*
>
> #Working Code
> sub vcl_hash{
> set req.http.hash-url = regsuball(req.url, "traceId=.*?(\&|$)", "");
> hash_data(req.http.hash-url);
> unset req.http.hash-url;
> hash_data(req.http.Accept-Encoding);
> *return (lookup);*
> }
>
>
> *I have few doubts to be clarified:*
> 1. May I know what difference return (lookup) statement makes?
> 2. Will there be any side effects with modified code, if I use return
> (lookup)? (Because original code was not causing any issue even without
> return lookup in production)
>
>