Mailing List Archive

Varnish removing tags incorrectly fromURL
Hello,



I would highly appreciate if I get some help on the following issu:



The query string from the end (== erstellen4) is being incorrectly
appended to the token because the Varnish is not removing the tags
correctly.



https://xxxxxx.my/aa/?ResetPasswordToken=4P/weCg49hetX25dVAJxGW0i2GcwuN3bB3z
xbMiYLo+3Kfpk199F9ZjwvSP3g8mrPq/opmCosoDmkTHYx3CYK+ABEFrF92y+R0V9icpnLep+f+z
fPJjVOZ+M6wa1egt+GNktWIdBIruXXREYAboEQyBtHmgGJQe25KoCUvfUe1ySZlcFre5Dk913ktB
D/wvwrtt/O6T2e9aUn2aiKkKdtA==&utm_source=acc_activation&utm_medium=email&utm
_campaign=FW_new_customer_activation_2-2019032713&utm_content=Zugangsdaten+e
rstellen4



This is what I get by looking at the logs:



After reset password token:



4P/weCg49hetX25dVAJxGW0i2GcwuN3bB3zxbMiYLo3Kfpk199F9ZjwvSP3g8mrPq/opmCosoDmk
THYx3CYK
ABEFrF92yR0V9icpnLepfzfPJjVOZM6wa1egtGNktWIdBIruXXREYAboEQyBtHmgGJQe25KoCUvf
Ue1ySZlcFre5Dk913ktBD/wvwrtt/O6T2e9aUn2aiKkKdtA== erstellen4







This is my varnish config :





#



# Marker to tell the VCL compiler that this VCL has been adapted to the

# new 4.0 format.

vcl 4.0;



import directors;

import std;



acl monitoring {

"localhost";

"192.xxx.xxx.xxx"/32; /* Collector */

"83.xxx.xxx.xxx"/32; /* LB */

}



acl purge {

"xxx.xxx.xxx.xxx"/32; /* */

"xxx.xxx.xxx.xxx"/32; /* */

}





include "/etc/varnish/backend.vcl";



sub vcl_init {



include "/etc/varnish/director.vcl";



}



sub vcl_recv {

# Happens before we check if we have this in cache already.

#

# Typically you clean up the request here, removing cookies you don't
need,

#A

#rewriting the request, etc.

#set req.backend_hint = vweb.backend(req.http.X-Forwarded-For);

#set req.backend_hint = fbdirector.backend();

#

# Set hash directory with hashing option X-Forwarded for becuase we use
nginx between the client and vanrish

#set req.backend_hint = hashdirector.backend(req.http.X-Forwarded-For);

#



# Monitoring for FortiADC if faild, the proxy is taken out, if all fails
then hit maintance page.

if (req.method == "GET" && req.url == "/varnish-status") {

if (client.ip ~ monitoring) {

#if (std.healthy(hashdirector.backend(req.http.X-Forwarded-For))) {

return(synth(200, "OK"));

#} else {

# return(synth(503, "No backends available"));

#}

} else {

return(synth(403, "Access denied."));

}

}



include "/etc/varnish/vhost.vcl";



# Remove the proxy header (see https://httpoxy.org/#mitigate-varnish)

unset req.http.proxy;



# Allow purging

if (req.method == "PURGE") {

if (!client.ip ~ purge) { # purge is the ACL defined at the begining

# Not from an allowed IP? Then die with an error.

return (synth(405, "IP: " + client.ip + " is not allowed to send
PURGE requests."));

}

# If you got this stage (and didn't error out above), purge the cached
result

return (purge);

}



# Only allow BAN requests from IP addresses in the 'purge' ACL.

if (req.method == "BAN") {

# Same ACL check as above:

if (!client.ip ~ purge) {

return (synth(405, "IP: " + client.ip + " is not allowed
to send BAN requests."));

}



# manual sudo varnishadm "ban req.http.host ~ www.mydomain.com"

ban("req.http.host ~ " + req.http.host);

# Throw a synthetic page so the request won't go to the backend.

return (synth(200, "BAN for " + req.http.host + " done"));

}



# Only cache GET or HEAD requests. This makes sure the POST requests are
always passed.

if (req.method != "GET" && req.method != "HEAD") {

return (pass);

}



# Some generic URL manipulation, useful for all templates that follow

# First remove the Google Analytics added parameters, useless for our
backend

if (req.url ~
"(\?|&)(utm_source|utm_medium|utm_campaign|utm_content|gclid|cx|ie|cof|siteu
rl)=") {

set req.url = regsuball(req.url,
"&(utm_source|utm_medium|utm_campaign|utm_content|gclid|cx|ie|cof|siteurl)=(
[A-z0-9_\-\.%25]+)", "");

set req.url = regsuball(req.url,
"\?(utm_source|utm_medium|utm_campaign|utm_content|gclid|cx|ie|cof|siteurl)=
([A-z0-9_\-\.%25]+)", "?");

set req.url = regsub(req.url, "\?&", "?");

set req.url = regsub(req.url, "\?$", "");

}



# Strip hash, server doesn't need it.

if (req.url ~ "\#") {

set req.url = regsub(req.url, "\#.*$", "");

}



# Strip a trailing ? if it exists

if (req.url ~ "\?$") {

set req.url = regsub(req.url, "\?$", "");

}



# Some generic cookie manipulation, useful for all templates that follow

# Remove the "has_js" cookie

set req.http.Cookie = regsuball(req.http.Cookie, "has_js=[^;]+(; )?", "");



# Remove any Google Analytics based cookies

set req.http.Cookie = regsuball(req.http.Cookie, "__utm.=[^;]+(; )?", "");

set req.http.Cookie = regsuball(req.http.Cookie, "_ga=[^;]+(; )?", "");

set req.http.Cookie = regsuball(req.http.Cookie, "_gat=[^;]+(; )?", "");

set req.http.Cookie = regsuball(req.http.Cookie, "utmctr=[^;]+(; )?", "");

set req.http.Cookie = regsuball(req.http.Cookie, "utmcmd.=[^;]+(; )?",
"");

set req.http.Cookie = regsuball(req.http.Cookie, "utmccn.=[^;]+(; )?",
"");



# Remove DoubleClick offensive cookies

set req.http.Cookie = regsuball(req.http.Cookie, "__gads=[^;]+(; )?", "");



# Remove the Quant Capital cookies (added by some plugin, all __qca)

set req.http.Cookie = regsuball(req.http.Cookie, "__qc.=[^;]+(; )?", "");



# Remove the AddThis cookies

set req.http.Cookie = regsuball(req.http.Cookie, "__atuv.=[^;]+(; )?",
"");



# Remove a ";" prefix in the cookie if present

set req.http.Cookie = regsuball(req.http.Cookie, "^;\s*", "");



# Are there cookies left with only spaces or that are empty?

if (req.http.cookie ~ "^\s*$") {

unset req.http.cookie;

}



# Large static files are delivered directly to the end-user without

# waiting for Varnish to fully read the file first.

# Varnish 4 fully supports Streaming, so set do_stream in
vcl_backend_response()

if (req.url ~
"^[^?]*\.(7z|avi|bz2|flac|flv|gz|mka|mkv|mov|mp3|mp4|mpeg|mpg|ogg|ogm|opus|r
ar|tar|tgz|tbz|txz|wav|webm|xz|zip)(\?.*)?$") {

unset req.http.Cookie;

return (hash);

}



#Remove all cookies for static files

if (req.url ~
"^[^?]*\.(7z|avi|bmp|bz2|css|csv|doc|docx|eot|flac|flv|gif|gz|ico|jpeg|jpg|j
s|less|mka|mkv|mov|mp3|mp4|mpeg|mpg|odt|otf|ogg|ogm|opus|pdf|png|ppt|pptx|ra
r|rtf|svg|svgz|swf|tar|tbz|tgz|ttf|txt|txz|wav|webm|webp|woff|woff2|xls|xlsx
|xml|xz|zip)(\?.*)?$") {

unset req.http.Cookie;

return (hash);

}



# Send Surrogate-Capability headers to announce ESI support to backend

set req.http.Surrogate-Capability = "key=ESI/1.0";



if (req.http.Authorization) {

# Not cacheable by default

return (pass);

}



if (req.url == "/checksite.aspx") {

# Dont cache monitoring url

return (pass);

}

return (hash);



}



sub vcl_backend_response {

# Happens after we have read the response headers from the backend.

#

# Here you clean the response headers, removing silly Set-Cookie headers

# and other mistakes your backend does.

#

#

set beresp.http.X-Backend = beresp.backend.name;

# Remove some headers: ASP version

unset beresp.http.X-Powered-By;



# Remove cookie with empty basketid useless... VL should fix on backend

if (beresp.http.set-cookie == "BasketID=; path=/") {

unset beresp.http.set-cookie;

}



# Pause ESI request and remove Surrogate-Control header

if (beresp.http.Surrogate-Control ~ "ESI/1.0") {

unset beresp.http.Surrogate-Control;

set beresp.do_esi = true;

}



# Enable cache for all static files

# The same argument as the static caches from above: monitor your cache
size, if you get data nuked out of it, consider giving up the static file
cache.

# Before you blindly enable this, have a read here:
https://ma.ttias.be/stop-caching-static-files/

if (bereq.url ~
"^[^?]*\.(7z|avi|bmp|bz2|css|csv|doc|docx|eot|flac|flv|gif|gz|ico|jpeg|jpg|j
s|less|mka|mkv|mov|mp3|mp4|mpeg|mpg|odt|otf|ogg|ogm|opus|pdf|png|ppt|pptx|ra
r|rtf|svg|svgz|swf|tar|tbz|tgz|ttf|txt|txz|wav|webm|webp|woff|woff2|xls|xlsx
|xml|xz|zip)(\?.*)?$") {

unset beresp.http.set-cookie;

}



# Large static files are delivered directly to the end-user without

# waiting for Varnish to fully read the file first.

# Varnish 4 fully supports Streaming, so use streaming here to avoid
locking.

if (bereq.url ~
"^[^?]*\.(7z|avi|bz2|flac|flv|gz|mka|mkv|mov|mp3|mp4|mpeg|mpg|ogg|ogm|opus|r
ar|tar|tgz|tbz|txz|wav|webm|xz|zip)(\?.*)?$") {

unset beresp.http.set-cookie;

set beresp.do_stream = true; # Check memory usage it'll grow in
fetch_chunksize blocks (128k by default) if the backend doesn't send a
Content-Length header, so only enable it for big objects

}



# Don't cache 50x responses

if (beresp.status == 500 || beresp.status == 502 || beresp.status == 503
|| beresp.status == 504) {

return (abandon);

}





if (bereq.http.Cookie ~ "(UserID|_session)") {

#set beresp.http.X-Cacheable = "NO:Got Session";

set beresp.uncacheable = true;

return (deliver);



} elsif (beresp.ttl <= 0s) {

# Varnish determined the object was not cacheable

#set beresp.http.X-Cacheable = "NO:Not Cacheable";



} elsif (beresp.http.set-cookie) {

# You don't wish to cache content for logged in users

#set beresp.http.X-Cacheable = "NO:Set-Cookie";

set beresp.uncacheable = true;

return (deliver);



} elsif (beresp.http.Cache-Control ~ "private") {

# You are respecting the Cache-Control=private header from the
backend

#set beresp.http.X-Cacheable = "NO:Cache-Control=private";

set beresp.uncacheable = true;

return (deliver);



} else {

# Varnish determined the object was cacheable

#set beresp.http.X-Cacheable = "YES";

}





return(deliver);



}



sub vcl_deliver {

# Happens when we have all the pieces we need, and are about to send the

# response to the client.

#

# You can do accounting or modifying the final object here.

if (obj.hits > 0) { # Add debug header to see if it's a HIT/MISS and the
number of hits, disable when not needed

set resp.http.X-Cache = "HIT";

} else {

set resp.http.X-Cache = "MISS";

}



# Unset some headers

unset resp.http.Via;

unset resp.http.X-Varnish;



# Please note that obj.hits behaviour changed in 4.0, now it counts per
objecthead, not per object

# and obj.hits may not be reset in some cases where bans are in use. See
bug 1492 for details.

# So take hits with a grain of salt

set resp.http.X-Cache-Hits = obj.hits;



}





Regards,

Cris
Re: Varnish removing tags incorrectly fromURL [ In reply to ]
On Wed, Mar 27, 2019 at 2:33 PM <cristian.c@istream.today> wrote:
>
> Hello,
>
>
>
> I would highly appreciate if I get some help on the following issu:

If you need to filter out or extract parameters from a query-string I
recommend this:

https://github.com/Dridi/libvmod-querystring/#vmod-querystring

If you are running Varnish below 6.0 I encourage you to upgrade but
meanwhile you also have this:

https://github.com/Dridi/libvmod-querystring/tree/v1.0.6#vmod-querystring

It should be much easier than using regular expressions, at the
expense of having to manage a vmod.


Dridi
_______________________________________________
varnish-misc mailing list
varnish-misc@varnish-cache.org
https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
RE: Varnish removing tags incorrectly fromURL [ In reply to ]
Hello,

Thank you for answering, I'm running on version 5.2.1, the problem is that I can't install this version https://github.com/Dridi/libvmod-querystring/tree/v1.0.6#vmod-querystring because there is no "configure" file inside (./configure not working) if I download the last version of vmod I get the following error configure error: varnish version 6.0.0. or higher is required ...

Thank you!


Regards,
Cristian

-----Original Message-----
From: Dridi Boukelmoune <dridi@varni.sh>
Sent: Wednesday, March 27, 2019 5:40 PM
To: cristian.c@istream.today
Cc: varnish-misc <varnish-misc@varnish-cache.org>
Subject: Re: Varnish removing tags incorrectly fromURL

On Wed, Mar 27, 2019 at 2:33 PM <cristian.c@istream.today> wrote:
>
> Hello,
>
>
>
> I would highly appreciate if I get some help on the following issu:

If you need to filter out or extract parameters from a query-string I recommend this:

https://github.com/Dridi/libvmod-querystring/#vmod-querystring

If you are running Varnish below 6.0 I encourage you to upgrade but meanwhile you also have this:

https://github.com/Dridi/libvmod-querystring/tree/v1.0.6#vmod-querystring

It should be much easier than using regular expressions, at the expense of having to manage a vmod.


Dridi

_______________________________________________
varnish-misc mailing list
varnish-misc@varnish-cache.org
https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
RE: Varnish removing tags incorrectly fromURL [ In reply to ]
Update: Ignore this... I have found. thx

-----Original Message-----
From: cristian.c@istream.today <cristian.c@istream.today>
Sent: Thursday, March 28, 2019 3:00 PM
To: 'Dridi Boukelmoune' <dridi@varni.sh>
Cc: 'varnish-misc' <varnish-misc@varnish-cache.org>
Subject: RE: Varnish removing tags incorrectly fromURL

Hello,

Thank you for answering, I'm running on version 5.2.1, the problem is that I can't install this version https://github.com/Dridi/libvmod-querystring/tree/v1.0.6#vmod-querystring because there is no "configure" file inside (./configure not working) if I download the last version of vmod I get the following error configure error: varnish version 6.0.0. or higher is required ...

Thank you!


Regards,
Cristian

-----Original Message-----
From: Dridi Boukelmoune <dridi@varni.sh>
Sent: Wednesday, March 27, 2019 5:40 PM
To: cristian.c@istream.today
Cc: varnish-misc <varnish-misc@varnish-cache.org>
Subject: Re: Varnish removing tags incorrectly fromURL

On Wed, Mar 27, 2019 at 2:33 PM <cristian.c@istream.today> wrote:
>
> Hello,
>
>
>
> I would highly appreciate if I get some help on the following issu:

If you need to filter out or extract parameters from a query-string I recommend this:

https://github.com/Dridi/libvmod-querystring/#vmod-querystring

If you are running Varnish below 6.0 I encourage you to upgrade but meanwhile you also have this:

https://github.com/Dridi/libvmod-querystring/tree/v1.0.6#vmod-querystring

It should be much easier than using regular expressions, at the expense of having to manage a vmod.


Dridi


_______________________________________________
varnish-misc mailing list
varnish-misc@varnish-cache.org
https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
Re: Varnish removing tags incorrectly fromURL [ In reply to ]
Hello Cristian,

Would you care sharing your findings in order to help people having the
same issue in the future?

Cheers,

On Thu, Mar 28, 2019, 07:42 <cristian.c@istream.today> wrote:

> Update: Ignore this... I have found. thx
>
> -----Original Message-----
> From: cristian.c@istream.today <cristian.c@istream.today>
> Sent: Thursday, March 28, 2019 3:00 PM
> To: 'Dridi Boukelmoune' <dridi@varni.sh>
> Cc: 'varnish-misc' <varnish-misc@varnish-cache.org>
> Subject: RE: Varnish removing tags incorrectly fromURL
>
> Hello,
>
> Thank you for answering, I'm running on version 5.2.1, the problem is
> that I can't install this version
> https://github.com/Dridi/libvmod-querystring/tree/v1.0.6#vmod-querystring
> because there is no "configure" file inside (./configure not working) if
> I download the last version of vmod I get the following error configure
> error: varnish version 6.0.0. or higher is required ...
>
> Thank you!
>
>
> Regards,
> Cristian
>
> -----Original Message-----
> From: Dridi Boukelmoune <dridi@varni.sh>
> Sent: Wednesday, March 27, 2019 5:40 PM
> To: cristian.c@istream.today
> Cc: varnish-misc <varnish-misc@varnish-cache.org>
> Subject: Re: Varnish removing tags incorrectly fromURL
>
> On Wed, Mar 27, 2019 at 2:33 PM <cristian.c@istream.today> wrote:
> >
> > Hello,
> >
> >
> >
> > I would highly appreciate if I get some help on the following issu:
>
> If you need to filter out or extract parameters from a query-string I
> recommend this:
>
> https://github.com/Dridi/libvmod-querystring/#vmod-querystring
>
> If you are running Varnish below 6.0 I encourage you to upgrade but
> meanwhile you also have this:
>
> https://github.com/Dridi/libvmod-querystring/tree/v1.0.6#vmod-querystring
>
> It should be much easier than using regular expressions, at the expense of
> having to manage a vmod.
>
>
> Dridi
>
>
> _______________________________________________
> varnish-misc mailing list
> varnish-misc@varnish-cache.org
> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
>
RE: Varnish removing tags incorrectly fromURL [ In reply to ]
Hi,



It was just my mistake, you must download the tar.gz version from releases tab : https://github.com/Dridi/libvmod-querystring/releases







Regards,

Cristian



From: Guillaume Quintard <guillaume@varnish-software.com>
Sent: Thursday, March 28, 2019 5:43 PM
To: cristian.c@istream.today
Cc: varnish-misc <varnish-misc@varnish-cache.org>
Subject: Re: Varnish removing tags incorrectly fromURL



Hello Cristian,



Would you care sharing your findings in order to help people having the same issue in the future?



Cheers,



On Thu, Mar 28, 2019, 07:42 <cristian.c@istream.today <mailto:cristian.c@istream.today> > wrote:

Update: Ignore this... I have found. thx

-----Original Message-----
From: cristian.c@istream.today <mailto:cristian.c@istream.today> <cristian.c@istream.today <mailto:cristian.c@istream.today> >
Sent: Thursday, March 28, 2019 3:00 PM
To: 'Dridi Boukelmoune' <dridi@varni.sh <mailto:dridi@varni.sh> >
Cc: 'varnish-misc' <varnish-misc@varnish-cache.org <mailto:varnish-misc@varnish-cache.org> >
Subject: RE: Varnish removing tags incorrectly fromURL

Hello,

Thank you for answering, I'm running on version 5.2.1, the problem is that I can't install this version https://github.com/Dridi/libvmod-querystring/tree/v1.0.6#vmod-querystring because there is no "configure" file inside (./configure not working) if I download the last version of vmod I get the following error configure error: varnish version 6.0.0. or higher is required ...

Thank you!


Regards,
Cristian

-----Original Message-----
From: Dridi Boukelmoune <dridi@varni.sh <mailto:dridi@varni.sh> >
Sent: Wednesday, March 27, 2019 5:40 PM
To: cristian.c@istream.today <mailto:cristian.c@istream.today>
Cc: varnish-misc <varnish-misc@varnish-cache.org <mailto:varnish-misc@varnish-cache.org> >
Subject: Re: Varnish removing tags incorrectly fromURL

On Wed, Mar 27, 2019 at 2:33 PM <cristian.c@istream.today <mailto:cristian.c@istream.today> > wrote:
>
> Hello,
>
>
>
> I would highly appreciate if I get some help on the following issu:

If you need to filter out or extract parameters from a query-string I recommend this:

https://github.com/Dridi/libvmod-querystring/#vmod-querystring

If you are running Varnish below 6.0 I encourage you to upgrade but meanwhile you also have this:

https://github.com/Dridi/libvmod-querystring/tree/v1.0.6#vmod-querystring

It should be much easier than using regular expressions, at the expense of having to manage a vmod.


Dridi


_______________________________________________
varnish-misc mailing list
varnish-misc@varnish-cache.org <mailto:varnish-misc@varnish-cache.org>
https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
Re: Varnish removing tags incorrectly fromURL [ In reply to ]
On Fri, Mar 29, 2019 at 8:18 AM <cristian.c@istream.today> wrote:
>
> Hi,
>
> It was just my mistake, you must download the tar.gz version from releases tab : https://github.com/Dridi/libvmod-querystring/releases

I would have been surprised if a 1.x release didn't work for you :)

Thanks for confirming and don't forget to move to 6.0 LTS to get bug
fixes regularly.

Dridi
_______________________________________________
varnish-misc mailing list
varnish-misc@varnish-cache.org
https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc