Mailing List Archive

Moving a redirect from backend to varnish
Hi there. I am currently responsible for a reasonably large cms
(http://www.hio.no). Now, the problem is, my predecessor has *not*
implemented a cache for this site, and the server is getting loaded down
quite heavily due to many mysql-requests.

Enter varnish (1.0.2-7 on Red Hat AS4) .... but, there is a problem:
poorly documented vcl ;)

One of the most important aspects about the site is that we have an
apache-rewrite on "/" for all http-requests from our OWN networks when
there is no Referer header set. The reason is we (or, our information
department.... ;) want all internal requests to / to end up on the news
page on first open, and subsequent clicks on links to / to actually take
you to /.

Now, a proxy will certainly mess up this apache rewrite. The apache
rewrite itself looks like this (truncated for brevity):

RewriteCond %{REMOTE_ADDR} ^128\.39\.75\.* [OR]
RewriteCond %{REMOTE_ADDR} ^128\.39\.89\.* [OR]
RewriteCond %{REMOTE_ADDR} ^2001:700:700:\.*
RewriteCond %{HTTP_REFERER} !^http://www\.hio\.no
RewriteRule ^/$ /content/view/full/27 [R=permanent,L]

A rather simple redirect.

The problem is, with varnish in place, this will of course not have the
desired effect. the vcl man page tells us a few things, but there have
been some problems implementing this redirect.

At the moment, I am playing with using an acl to check if the client is
on an internal network, and then see if Referer is set and the page
being requested is /, and then act accordingly.

There are two problems in this. The first is making the test act
accordingly, the second is the acl syntax.. first, the test. I am
currently using something like:

if (client ~ internal && req.url == "/" && req.http.referer) {...}

The acl uses a syntax which I've noticed in the list archives for this
list is possibly bugged for checking IP's (?):

acl internal {
"158.36.161.1"/24;
"158.36.78.1"/23;
...
...
}

Now, the man page uses .1 for the IPs, but .0 should be more intuitive?
Also the /24 should probably be placed inside the quotes?

Now, provided I get the right test in place - the next question is how
to handle the redirect itself. You could either send the request on to
the backend (but this is not good caching... ;), using pipe (?), or we
could tell the client directly that the document is moved, and what the
correct url is. But how could I get varnish to simply say "beeeep, no,
go to THIS page instead" using something like a http 301?

Or is there another solution that the current documentation does not expose?

Oh, and is there any need for credits to the varnishproject if I use it
on http://www.hio.no?

--
Kenneth R?rvik, IT HiO
Tlf 22 45 20 83
Kenneth.Rorvik at hio.no
Moving a redirect from backend to varnish [ In reply to ]
On Jan 17, 2007, at 09:08, Kenneth R?rvik wrote:

> Hi there. I am currently responsible for a reasonably large cms
> (http://www.hio.no). Now, the problem is, my predecessor has *not*
> implemented a cache for this site, and the server is getting loaded
> down quite heavily due to many mysql-requests.
>
> Enter varnish (1.0.2-7 on Red Hat AS4) .... but, there is a
> problem: poorly documented vcl ;)
>
> One of the most important aspects about the site is that we have an
> apache-rewrite on "/" for all http-requests from our OWN networks
> when there is no Referer header set. The reason is we (or, our
> information department.... ;) want all internal requests to / to
> end up on the news page on first open, and subsequent clicks on
> links to / to actually take you to /.
>
> Now, a proxy will certainly mess up this apache rewrite. The apache
> rewrite itself looks like this (truncated for brevity):
>
> RewriteCond %{REMOTE_ADDR} ^128\.39\.75\.* [OR]
> RewriteCond %{REMOTE_ADDR} ^128\.39\.89\.* [OR]
> RewriteCond %{REMOTE_ADDR} ^2001:700:700:\.*
> RewriteCond %{HTTP_REFERER} !^http://www\.hio\.no
> RewriteRule ^/$ /content/view/full/27 [R=permanent,L]
>
> A rather simple redirect.

I'm not going to solve your problem here, but I have a question: Why
are you using a permanent redirect here?? It is, from my
understanding, not at all a permanent redirect, it's a conditional
one -- and you have no intention to blocking access to the / page
from internal users; you just want them to get there following a link
somewhere in the www.hio.no hive.
From my limited knowledge of HTTP, the way you are doing it here
should be causing trouble with browsers who do what they are allowed
to - i.e. cache the redirect (as the browser will never see any of
the conditions anyway). Not to mention all the caches you could be
poisoning, if you have any such internally (and with hio having a
presumably rather large network, I wouldn't automatically assume that
your cache would be the only one, authorized or not).

Just curious -- and looking forward to some helpful answers to your
questions ;)

/Eirik

>
> The problem is, with varnish in place, this will of course not have
> the desired effect. the vcl man page tells us a few things, but
> there have been some problems implementing this redirect.
>
> At the moment, I am playing with using an acl to check if the
> client is on an internal network, and then see if Referer is set
> and the page being requested is /, and then act accordingly.
>
> There are two problems in this. The first is making the test act
> accordingly, the second is the acl syntax.. first, the test. I am
> currently using something like:
>
> if (client ~ internal && req.url == "/" && req.http.referer) {...}
>
> The acl uses a syntax which I've noticed in the list archives for
> this list is possibly bugged for checking IP's (?):
>
> acl internal {
> "158.36.161.1"/24;
> "158.36.78.1"/23;
> ...
> ...
> }
>
> Now, the man page uses .1 for the IPs, but .0 should be more
> intuitive? Also the /24 should probably be placed inside the quotes?
>
> Now, provided I get the right test in place - the next question is
> how to handle the redirect itself. You could either send the
> request on to the backend (but this is not good caching... ;),
> using pipe (?), or we could tell the client directly that the
> document is moved, and what the correct url is. But how could I get
> varnish to simply say "beeeep, no, go to THIS page instead" using
> something like a http 301?
>
> Or is there another solution that the current documentation does
> not expose?
>
> Oh, and is there any need for credits to the varnishproject if I
> use it on http://www.hio.no?
>
> --
> Kenneth R?rvik, IT HiO
> Tlf 22 45 20 83
> Kenneth.Rorvik at hio.no
> _______________________________________________
> varnish-misc mailing list
> varnish-misc at projects.linpro.no
> http://projects.linpro.no/mailman/listinfo/varnish-misc
>
>
Moving a redirect from backend to varnish [ In reply to ]
Eirik ?verby wrote:

> I'm not going to solve your problem here, but I have a question: Why are
> you using a permanent redirect here?? It is, from my understanding, not
> at all a permanent redirect, it's a conditional one -- and you have no
> intention to blocking access to the / page from internal users; you just
> want them to get there following a link somewhere in the www.hio.no hive.
> From my limited knowledge of HTTP, the way you are doing it here should
> be causing trouble with browsers who do what they are allowed to - i.e.
> cache the redirect (as the browser will never see any of the conditions
> anyway). Not to mention all the caches you could be poisoning, if you
> have any such internally (and with hio having a presumably rather large
> network, I wouldn't automatically assume that your cache would be the
> only one, authorized or not).

Hmm, thinking about this, the client IS getting a http 301 and a correct
URL for subsequent use, so I don't think this would be poisoning caches?


--
Kenneth R?rvik, IT HiO
Tlf 22 45 20 83
Kenneth.Rorvik at hio.no
Moving a redirect from backend to varnish [ In reply to ]
Kenneth R?rvik wrote:

> Now, provided I get the right test in place - the next question is how
> to handle the redirect itself. You could either send the request on to

Looks like the test is OK now. Next problem: piping to the server now
makes it impossible for the server to distinguish clients (since they
all come from varnish), so the next Q is:

How can I have varnish itself send a http 301 or 307 to the client with
the correct URL? Cannot find this documented, or on google for that matter.

--
Kenneth R?rvik, IT HiO
Tlf 22 45 20 83
Kenneth.Rorvik at hio.no
Moving a redirect from backend to varnish [ In reply to ]
> How can I have varnish itself send a http 301 or 307 to the client with
> the correct URL? Cannot find this documented, or on google for that matter.

Hmm, a bit quick:

http://projects.linpro.no/pipermail/varnish-misc/2006-October/000104.html

Any thoughts on when this can be expected?

--
Kenneth R?rvik, IT HiO
Tlf 22 45 20 83
Kenneth.Rorvik at hio.no
Moving a redirect from backend to varnish [ In reply to ]
Kenneth R?rvik <Kenneth.Rorvik at hio.no> writes:

> Kenneth R?rvik wrote:
>
>> Now, provided I get the right test in place - the next question is
>> how to handle the redirect itself. You could either send the request
>> on to
>
> Looks like the test is OK now. Next problem: piping to the server now
> makes it impossible for the server to distinguish clients (since they
> all come from varnish), so the next Q is:
>
> How can I have varnish itself send a http 301 or 307 to the client
> with the correct URL? Cannot find this documented, or on google for
> that matter.

Varnish adds (or modifies an existing?) X-Forwarded-For header,
containing a path of IP addresses of the client and additional
proxies.

You should be able to use the server-variable %{HTTP_FORWARDED}, or
perhaps %{HTTP_X_FORWARDED_FROM}, for matching the client address with
RewriteCond.

Note that it is possible for the user to add information to this
header when sending a request.

--
Stig Sandbeck Mathisen, Linpro