Mailing List Archive

My Varnish project
Hello,

Thank you very much for an excellent product. I wish to use Varnish
cache as a front end to my system, but I have some problems in using it,
and the way forward probably involves hacking the code. I will run this
past you all first to get your opinion.



Background:

Our site (http://www.adofms.com.au) is a traditional LAMP application,
using PHP / Apache / MySQL. Users login to the application, and upon
login, a session is created in the database, and a cookie is set for the
user.

The ADOFMS system contains a large number of Units - Fuel Card -
Vehicles for the users to maintain. When a user logs in and gets a valid
session, they can only see the vehicles and cards associated with their
unit. So far so good.

Problem that I have is that the session identification is done entirely
by the cookie - I do not repeat the session ID in the URL.

So for example -

User number 1 visits :
http://www.adofms.com.au/vehicles.php - and gets a list of vehicles
that they own.

User number 2 visits
http://www.adofms.com.au/vehicles.php - and they get a different list of
vehicles to what user 1 sees.

So, of course, with Varnish, the vehicles.php output from user 1 is
cached, and presented to user 2. This is very quick and efficient, but
not what we want. I need things to be cached on a session by session basis.


I believe I have 3 options here :

1) Re-write the whole PHP application to repeat the session ID as part
of every URL in the system. That is do-able, but boring.

2) Cook up some VCL code to cache pages on a per-session basis, by
appending the req.http.Cookie value to the URL before it is stored in
the cache, and then doing the same thing when looking up the cache. VCL
does not easily allow this to happen though ???? Correct me if I am wrong.

3) Hack the source code of Varnish to use the session ID (from the
cookie) to segregate cached results by session.


I am going to have a go at method 3) anyway, but would like your opinion
before starting out on this adventure :) I would think its not too
hard, since I know exactly what I am trying to achieve, which is often
half the battle when coding. I have some similar enhancements that would
suit my application well. I will explain them here:

Example for extra enchancement :

I have a URL such as

http://www.adofms.com.au/vehicles.php?op=view&id=12345
- Displays the full details of vehicle ID 12345, which can generate a
lot of SQL calls to create. I would love to cache the results of this on
a per user basis.

If the user updates the vehicle - the naming conventions in this
application guarantee that the backend receives a HTTP POST request with
a URL of :
http://www.adofms.com.au/vehicles.php?op=update .. followed by a call to
redisplay the record, which in this case is
http://www.adofms.com.au/vehicles.php?op=view&id=12345&refresh=1

Now - If I am going to hack the Varnish code anyway, I can get around
this by intercepting POST requests that have an op=update GET variable
set, and an id=<some value> POST variable set .. and if so, remove the
existing vehicles.php?op=view&id=<some value> entry from the cache.

Again, running this idea past you for feedback. I understand that such a
change would be very specific to our application and the way it works,
but thats OK with me.

Another alternative would be to make VCL a little more powerful - a
couple of simple ways of changing the key value for the "obj" variable
would go a long way I think .. and the ability to grab GET and POST
variables would be very handy too.


Would appreciate your thoughts on these issues.

Thank you
Steve OConnor
ADOFMS Chief Developer
My Varnish project [ In reply to ]
In message <465BF076.9030804 at adofms.com.au>, admin writes:

>2) Cook up some VCL code to cache pages on a per-session basis, by
>appending the req.http.Cookie value to the URL before it is stored in
>the cache, and then doing the same thing when looking up the cache. VCL
>does not easily allow this to happen though ???? Correct me if I am wrong.

Actually, we just added a facility to do this:

vcl_hash {
req.hash += req.http.cookie;
}

It will (not yet) do the right thing if there are multiple cookie
header lines, but that is in the works.

--
Poul-Henning Kamp | UNIX since Zilog Zeus 3.20
phk at FreeBSD.ORG | TCP/IP since RFC 956
FreeBSD committer | BSD since 4.3-tahoe
Never attribute to malice what can adequately be explained by incompetence.
My Varnish project [ In reply to ]
Poul-Henning Kamp wrote:
> In message <465BF076.9030804 at adofms.com.au>, admin writes:
>
>
>> 2) Cook up some VCL code to cache pages on a per-session basis, by
>> appending the req.http.Cookie value to the URL before it is stored in
>> the cache, and then doing the same thing when looking up the cache. VCL
>> does not easily allow this to happen though ???? Correct me if I am wrong.
>>
>
> Actually, we just added a facility to do this:
>
> vcl_hash {
> req.hash += req.http.cookie;
> }
>
> It will (not yet) do the right thing if there are multiple cookie
> header lines, but that is in the works.
>
>
Excellent - I am on gentoo, using varnish-1.0.4

The man pages for VCL do mention a vcl_hash interface, but it says it's
not implemented yet. Wasnt sure what it was, but I assume that is called
whenever the hash value for the key of the object is calculated ?

That would be before every insert command, and before every lookup
command ? In that case, its exactly what I need. I notice that req.hash
is not mentioned in the man pages either.

Will make sure that I have the latest version and have a play again.
Actually - it wont take a minute, and its all quiet here now (8pm in
Australia), so Ill try it on the production servers now .. :)

Steve
My Varnish project [ In reply to ]
In message <465BFE12.5070001 at adofms.com.au>, admin writes:
>Poul-Henning Kamp wrote:

>The man pages for VCL do mention a vcl_hash interface, but it says it's
>not implemented yet. Wasnt sure what it was, but I assume that is called
>whenever the hash value for the key of the object is calculated ?

Yes.

--
Poul-Henning Kamp | UNIX since Zilog Zeus 3.20
phk at FreeBSD.ORG | TCP/IP since RFC 956
FreeBSD committer | BSD since 4.3-tahoe
Never attribute to malice what can adequately be explained by incompetence.
My Varnish project [ In reply to ]
admin <admin at adofms.com.au> writes:
> Poul-Henning Kamp <phk at phk.freebsd.dk> writes:
> > Actually, we just added a facility to do this:
> >
> > vcl_hash {
> > req.hash += req.http.cookie;
> > }
> >
> > It will (not yet) do the right thing if there are multiple cookie
> > header lines, but that is in the works.
> Excellent - I am on gentoo, using varnish-1.0.4

1.0.4 does not have this functionality, you'll have to check out the
latest code from Subversion and build / install manually.

DES
--
Dag-Erling Sm?rgrav
Senior Software Developer
Linpro AS - www.linpro.no
My Varnish project [ In reply to ]
Poul-Henning Kamp wrote:
> In message <465BFE12.5070001 at adofms.com.au>, admin writes:
>
>> Poul-Henning Kamp wrote:
>>
>
>
>> The man pages for VCL do mention a vcl_hash interface, but it says it's
>> not implemented yet. Wasnt sure what it was, but I assume that is called
>> whenever the hash value for the key of the object is calculated ?
>>
>
> Yes.
>
>
Thanks

I have this now :
sub vcl_hash {
req.hash += req.http.cookie;
}

and get this when I run varnishd :

(/etc/varnish/adofms.vcl Line 22 Pos 14)
req.hash += req.http.cookie;
-------------########--------------------

I think I need a more up to date source file ?

When I emerge varnish , it fetches and builds :
mirror://sourceforge/varnish/varnish-1.0.4.tar.gz

Is there a newer version in SVN or something ?
My Varnish project [ In reply to ]
Dag-Erling Sm?rgrav wrote:
> admin <admin at adofms.com.au> writes:
>
>> Poul-Henning Kamp <phk at phk.freebsd.dk> writes:
>>
>>> Actually, we just added a facility to do this:
>>>
>>> vcl_hash {
>>> req.hash += req.http.cookie;
>>> }
>>>
>>> It will (not yet) do the right thing if there are multiple cookie
>>> header lines, but that is in the works.
>>>
>> Excellent - I am on gentoo, using varnish-1.0.4
>>
>
> 1.0.4 does not have this functionality, you'll have to check out the
> latest code from Subversion and build / install manually.
>
> DES
>
You guys are great :)

Doing this now ... shouldnt be long

SteveOC
My Varnish project [ In reply to ]
"ADOFMS Admin, SteveOC" <admin at adofms.com.au> writes:
> I have this now :
> sub vcl_hash {
> req.hash += req.http.cookie;
> }
>
> and get this when I run varnishd :
>
> (/etc/varnish/adofms.vcl Line 22 Pos 14)
> req.hash += req.http.cookie;
> -------------########--------------------
>
> I think I need a more up to date source file ?

You need the latest sources from Subversion, and you left out the "set"
keyword.

DES
--
Dag-Erling Sm?rgrav
Senior Software Developer
Linpro AS - www.linpro.no
My Varnish project [ In reply to ]
Just to finish this thread then :

I am currently running the latest SVN release of Varnish on our
production server, and implemented the hash upgrade using just VCL.

My VCL setup has :

sub vcl_hash {
set req.hash += req.url;
set req.hash += req.http.host;
set req.hash += req.http.cookie;
hash;
}

Very easy, but it also needs to insert into the cache on vcl_fetch when
resp.http.Set-Cookie is true. (ie - DONT pass when set-cookie is set)

Its now working fine, and passed a few obvious test cases.

I will run this on our production machine for the next 24 hours. We will
have about 400 login sessions tomorrow with live users each doing an
hours worth of work, so it will be good to keep an eye on things during
this period.

Also - I am building this one with x86_64, and everything looks fine to
me. Its worth flagging it in Gentoo with at least an ~amd64 keyword.

The codebase looks very clean - I will spend some time experimenting
soon, making up some new VCL extensions to solve my other problems and
tailor Varnish to exactly fit my application. My aim here is to make
sure that all the hackery can be done using simple VCL variables, so
that others can get Varnish fitting in with other weird setups too.

I really like this project, and its good break from PHP coding. I miss
my C compiler !.

Thanks again guys.
SteveOC
My Varnish project [ In reply to ]
Hi,

If I would like to do a check based on the response code from the backend how can I do this in VCL

I would like to be able to not cache 404's or to set the timeout to a very low number for 404's.

I have checked the vcl man page, but I'm unable to find which statement to use for the if clause.

--

Thomas Westlund
Aftenposten AS, UNIX/nettverksavd.
Postboks 1, 0051 Oslo
Tlf: +47 98 20 30 33
Fax: +47 22 86 40 74