Mailing List Archive

On Tue, Jan 23, 2018 at 5:12 PM, Alexandros Kechagias
<alexandros.kechagias@gmail.com> wrote:
> Hi Experts,
> I am looking for a way to invalidate a cache based on a set of different
> "keys" that can be combined with each other with operators like "AND", "OR"
> or "NOT".
>
> So I am looking for something that basically behaves like the xkey mod from
> varnish-modules[1] with the only difference that it can invalidate cache
> objects in an SQLish way.
>
> For example:
> xkey.NewPurge("userfoo OR projectbar AND NOT instancebaz")
>
> Do you know a way i could do that in varnish? I am capable of programming
> and writing VCL.

Hi,

Not really an expert, but I'll share my two cents anyway.

I would recommend bans if it had all the boolean operations you are
looking for, but I think as of today only AND aka && is supported (and
NOT semantics by using negative operators like != or !~). You could
emulate an OR operator to some extent with regular|expressions but
that's limited to one header at a time.

That's the closest I'm aware of.

Dridi
_______________________________________________
varnish-misc mailing list
varnish-misc@varnish-cache.org
https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc

Regarding bans:

On 23/01/18 18:04, Dridi Boukelmoune wrote:
> "keys" that can be combined with each other with operators like "AND", "OR"
> or "NOT".

OR is just the same as several bans.

As any boolean expression can be converted to a disjunction (= OR semantics),
the existing '&&' operator with the existing comparison operators

* ``==``: *<field>* and *<arg>* are equal strings (case sensitive)
* ``!=``: *<field>* and *<arg>* are unequal strings (case sensitive)
* ``~``: *<field>* matches the regular expression *<arg>*
* ``!~``:*<field>* does not match the regular expression *<arg>*

should suffice to implement arbitrary logic based on

* ``req.url``: The request url
* ``req.http.*``: Any request header
* ``obj.status``: The cache object status
* ``obj.http.*``: Any cache object header

Nils

P.S. On a related issue, I got an open PR to add obj.ttl, obj.age, obj.grace and
obj.keep at https://github.com/varnishcache/varnish-cache/pull/2462
_______________________________________________
varnish-misc mailing list
varnish-misc@varnish-cache.org
https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc

Hi there,
thanks for the replies. I see, I didn't give you enough details for you to
be able to help me. Sorry for that, I had a little bit of tunnel vision and
also the title is not optimal.

The reason I am using xkey, is that I am tagging content that comes from
the Backend, with tags/names that I use afterwards to invalidate everything
that has to do with the tags/names. The problem is that in my scenario the
URLs can't always reliably reflect their content. So I decided to use tags,
so the backend can tell varnish how to group the content.

For Example:
1. User wants the pages called :
- mysite.mars/foo
- mysite.mars/bar
- mysite.mars/baz

2. Varnish asks the backend for the sites and tags them with xkey
accordingly to data that comes from the backend through beresp.http.xkey
Let's say beresp.http.xkey gives me back the following keys/tags for each
site (I will visualize them with brackets)
- mysite.mars/foo [project1] [inst6]
- mysite.mars/bar [project2] [inst6]
- mysite.mars/baz [project2] [inst6]

3. Now I know that the purpose of xkey is that i can say:
xkey.purge("inst6")
That would delete all the caches.
xkey.purge("project2")
Would delete the last two.

But I have a different problem.
I want to be able to delete "inst6" only from "project2".
So something like:

xkey.purge("project2" && "inst6")

would be nice.

Has someone an idea how I can solve this problem with already existing
varnish modules or with a VCL algorithm of a reasonable complexity?
If I can't find anything I would have to write a module myself or maybe
look for some different caching technology. If you could also drop me a
hint there, that would also be nice.

Alexandros

On Fri, Jan 26, 2018 at 4:12 PM, Alexandros Kechagias
<alexandros.kechagias@gmail.com> wrote:
> Hi there,
> thanks for the replies. I see, I didn't give you enough details for you to
> be able to help me. Sorry for that, I had a little bit of tunnel vision and
> also the title is not optimal.
>
> The reason I am using xkey, is that I am tagging content that comes from the
> Backend, with tags/names that I use afterwards to invalidate everything that
> has to do with the tags/names. The problem is that in my scenario the URLs
> can't always reliably reflect their content. So I decided to use tags, so
> the backend can tell varnish how to group the content.
>
> For Example:
> 1. User wants the pages called :
> - mysite.mars/foo
> - mysite.mars/bar
> - mysite.mars/baz
>
> 2. Varnish asks the backend for the sites and tags them with xkey
> accordingly to data that comes from the backend through beresp.http.xkey
> Let's say beresp.http.xkey gives me back the following keys/tags for each
> site (I will visualize them with brackets)
> - mysite.mars/foo [project1] [inst6]
> - mysite.mars/bar [project2] [inst6]
> - mysite.mars/baz [project2] [inst6]
>
> 3. Now I know that the purpose of xkey is that i can say:
> xkey.purge("inst6")
> That would delete all the caches.
> xkey.purge("project2")
> Would delete the last two.
>
> But I have a different problem.
> I want to be able to delete "inst6" only from "project2".
> So something like:
>
> xkey.purge("project2" && "inst6")
>
> would be nice.
>
> Has someone an idea how I can solve this problem with already existing
> varnish modules or with a VCL algorithm of a reasonable complexity?
> If I can't find anything I would have to write a module myself or maybe look
> for some different caching technology. If you could also drop me a hint
> there, that would also be nice.
>
> Alexandros

Hi,

I actually understood you well, and basically vmod-xkey sits
in-between native purge and ban. vmod-xkey invalidates based on
criteria (like bans) with purge-like behavior/performance.

So if you need complex expressions and are ready to give up the
real-time nature of xkey purges, you can reuse whatever headers (xkey
supports a couple) that contain your invalidation keys and issue bans
instead:

ban obj.http.xkey ~ project2 && obj.http.xkey ~ inst6

Like I said, xkey gives up some of the ban flexibility and moves the
cursor to real-time processing while bans are deferred by design. What
I'm trying to say here is if you need complex invalidation schemes
combinating multiple criteria, then you have access to whatever was
present when beresp was cached. Today xkey accepts multiple keys at
once, but invalidates their union, the opposite of what you want.

Dridi
_______________________________________________
varnish-misc mailing list
varnish-misc@varnish-cache.org
https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc

Hi,

On 26/01/18 17:07, Dridi Boukelmoune wrote:
> So if you need complex expressions and are ready to give up the
> real-time nature of xkey purges, you can reuse whatever headers (xkey
> supports a couple) that contain your invalidation keys and issue bans
> instead:
>
> ban obj.http.xkey ~ project2 && obj.http.xkey ~ inst6
>
> Like I said, xkey gives up some of the ban flexibility and moves the
> cursor to real-time processing while bans are deferred by design.

Dridi, I completely agree with your response, except for one thing: IMHO, bans
are in no way less real time than purges: While the ban *lurker* processing is
deferred, actual ban checks at lookup time happen as immediately as whatever purges.

Other than that, purges act on objectheads ("cache keys") affect all variants
under that cache key. bans are checked per object (variant).

Nils
_______________________________________________
varnish-misc mailing list
varnish-misc@varnish-cache.org
https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc

On Fri, Jan 26, 2018 at 6:15 PM, Nils Goroll <slink@schokola.de> wrote:
<snip>
> Dridi, I completely agree with your response, except for one thing: IMHO, bans
> are in no way less real time than purges: While the ban *lurker* processing is
> deferred, actual ban checks at lookup time happen as immediately as whatever purges.

What I meant by deferred is that objects don't become candidates for
eviction right away, so when storage runs low you may LRU-nuke objects
still alive instead of banned objects that haven't been processed yet
(lurk-able or not).

I think, but like I said I'm no expert in this area.

> Other than that, purges act on objectheads ("cache keys") affect all variants
> under that cache key. bans are checked per object (variant).

Good point, and to clarify xkey operates on objects like bans, so not
all variants may be xkey-purged at once.

Dridi
_______________________________________________
varnish-misc mailing list
varnish-misc@varnish-cache.org
https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc

> So if you need complex expressions and are ready to give up the
> real-time nature of xkey purges, you can reuse whatever headers (xkey
> supports a couple) that contain your invalidation keys and issue bans
> instead:
>
> ban obj.http.xkey ~ project2 && obj.http.xkey ~ inst6

That's what i am going to do.

> * ``==``: *<field>* and *<arg>* are equal strings (case sensitive)
> * ``!=``: *<field>* and *<arg>* are unequal strings (case sensitive)
> * ``~``: *<field>* matches the regular expression *<arg>*
> * ``!~``:*<field>* does not match the regular expression *<arg>*
>
>should suffice to implement arbitrary logic based on
>
> * ``req.url``: The request url
> * ``req.http.*``: Any request header
> * ``obj.status``: The cache object status
> * ``obj.http.*``: Any cache object header

Ok, now i get that too.

I think I have to improve my understanding of varnish.
I didn't knew that I can access the all the data that xkey uses to tag the
caches.

Thanks a bunch for your time guys, this really helped.
Alexandros

Hi there,
I have a question about your recommendation of banning caches with
obj.http.xkey.

I am also getting the following error from the compiler:
'obj.http.xkey': Not available in method 'vcl_recv'

Also according to the documentation [1] I can only access "obj.*"
inside vcl_hit[2] or vcl_deliver[3]
and I feel like this is the wrong place to use ban to ban objects.

I wanted to add this functionality into vcl_recv like:

if (req.method == "BAN") {
# ACL purgers go here
ban(obj.http.xkey ~ req.http.banrgx);
return (synth(200, "Banned !"));
}

Any suggestions?

I use varnish-cache 5.2.1

[1] variables in vcl subroutines :
https://book.varnish-software.com/4.0/chapters/VCL_Basics.html#variables-in-vcl-subroutines
[2] vcl_hit : https://book.varnish-software.com/4.0/chapters/VCL_Subroutines.html#vcl-vcl-hit
[3] vcl_deliver :
https://book.varnish-software.com/4.0/chapters/VCL_Subroutines.html#vcl-vcl-deliver

Thanks
Alexandros
_______________________________________________
varnish-misc mailing list
varnish-misc@varnish-cache.org
https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc

ban() takes a string :-)

ban("obj.http.xkey ~ " + req.http.banrgx);


--
Guillaume Quintard

On Tue, Jan 30, 2018 at 5:06 PM, Alexandros Kechagias <
alexandros.kechagias@gmail.com> wrote:

> Hi there,
> I have a question about your recommendation of banning caches with
> obj.http.xkey.
>
> I am also getting the following error from the compiler:
> 'obj.http.xkey': Not available in method 'vcl_recv'
>
> Also according to the documentation [1] I can only access "obj.*"
> inside vcl_hit[2] or vcl_deliver[3]
> and I feel like this is the wrong place to use ban to ban objects.
>
> I wanted to add this functionality into vcl_recv like:
>
> if (req.method == "BAN") {
> # ACL purgers go here
> ban(obj.http.xkey ~ req.http.banrgx);
> return (synth(200, "Banned !"));
> }
>
> Any suggestions?
>
> I use varnish-cache 5.2.1
>
> [1] variables in vcl subroutines :
> https://book.varnish-software.com/4.0/chapters/VCL_Basics.
> html#variables-in-vcl-subroutines
> [2] vcl_hit : https://book.varnish-software.com/4.0/chapters/VCL_
> Subroutines.html#vcl-vcl-hit
> [3] vcl_deliver :
> https://book.varnish-software.com/4.0/chapters/VCL_
> Subroutines.html#vcl-vcl-deliver
>
> Thanks
> Alexandros
> _______________________________________________
> varnish-misc mailing list
> varnish-misc@varnish-cache.org
> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
>

> ban() takes a string :-)

D'oh! Thanks Guillaume, it works now. :-)

The documentation says that:
"Bans are checked when we hit an object in the cache, but before we
deliver it." [1]
I guess that's why obj.* is only available in vcl_hit and vcl_deliver, right?

[1] http://varnish-cache.org/docs/4.0/users-guide/purging.html#bans
_______________________________________________
varnish-misc mailing list
varnish-misc@varnish-cache.org
https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc

Technically, bans are evaluated outside of vcl, after vcl_hash. We grab a
tentative obj, then test it, if it matches, we destroy the objcontinue
searching, otherwise we go to vcl_hit

--
Guillaume Quintard

On Jan 30, 2018 18:49, "Alexandros Kechagias" <
alexandros.kechagias@gmail.com> wrote:

> > ban() takes a string :-)
>
> D'oh! Thanks Guillaume, it works now. :-)
>
> The documentation says that:
> "Bans are checked when we hit an object in the cache, but before we
> deliver it." [1]
> I guess that's why obj.* is only available in vcl_hit and vcl_deliver,
> right?
>
> [1] http://varnish-cache.org/docs/4.0/users-guide/purging.html#bans
>

Ok, I see, there's still a lot to learn for me. :-)

Thanks for your time Guillaume!

2018-01-30 19:08 GMT+01:00 Guillaume Quintard <guillaume@varnish-software.com>:
> Technically, bans are evaluated outside of vcl, after vcl_hash. We grab a
> tentative obj, then test it, if it matches, we destroy the objcontinue
> searching, otherwise we go to vcl_hit
>
> --
> Guillaume Quintard
>
> On Jan 30, 2018 18:49, "Alexandros Kechagias"
> <alexandros.kechagias@gmail.com> wrote:
>>
>> > ban() takes a string :-)
>>
>> D'oh! Thanks Guillaume, it works now. :-)
>>
>> The documentation says that:
>> "Bans are checked when we hit an object in the cache, but before we
>> deliver it." [1]
>> I guess that's why obj.* is only available in vcl_hit and vcl_deliver,
>> right?
>>
>> [1] http://varnish-cache.org/docs/4.0/users-guide/purging.html#bans
_______________________________________________
varnish-misc mailing list
varnish-misc@varnish-cache.org
https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc