Mailing List Archive

varnish with apache mod_auth
Hi,

We are having an issue with VARNISH and apache mod_auth. Varnish is on port
80 serving users and Apache is the backend.

We have servers restricting access only to authenticated users or certain
IP addresses. Since we installed Varnish the issue is that we need to
enable 127.0.0.1 as a permitted IP (required ip rule) so the Varnish can
fetch content. The problem, is that the real IP is not used and all the
other rules does not apply.

Bottom line, how can we still control who is requesting using MOD_AUTH and
having Varnish?

Regards
Hernán.
Re: varnish with apache mod_auth [ In reply to ]
I don't believe there's a trivial way to do this.

Varnish will return the cached response to any IP address that comes
calling. Even if the first request comes from a valid IP, which gets
passed through via X-Forward or similar, and mod_auth is tweaked to respond
to that, any subsequent request will not be seen by either apache or
mod_auth at all.

You have a few options:
1) IP Whitelists are a rather poor means of authentication. Moving to
something else might be prudent. But that's not easy.
2) There are probably VMODs that do something similar. If not and if the
list of IPs isn't too long, you could limit the IPs in VCL rather than
mod_auth.
3) Push the list of IP addresses that can connect to the external port down
to IPTables or similar.
4) Push the list of IP addresses to external Firewall, or Security Group or
whatever.



On Thu, Mar 16, 2017 at 5:46 PM, Hernán Marsili <hernan@cmsmedios.com>
wrote:

> Hi,
>
> We are having an issue with VARNISH and apache mod_auth. Varnish is on
> port 80 serving users and Apache is the backend.
>
> We have servers restricting access only to authenticated users or certain
> IP addresses. Since we installed Varnish the issue is that we need to
> enable 127.0.0.1 as a permitted IP (required ip rule) so the Varnish can
> fetch content. The problem, is that the real IP is not used and all the
> other rules does not apply.
>
> Bottom line, how can we still control who is requesting using MOD_AUTH and
> having Varnish?
>
> Regards
> Hernán.
>
> _______________________________________________
> varnish-misc mailing list
> varnish-misc@varnish-cache.org
> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
>
Re: varnish with apache mod_auth [ In reply to ]
Authenticated requests should typically bypass cache, unless you want to
hash the related session id(s), however that can get "interesting". I
suggest using an Apache module such as rpaf or remoteip in order for Apache
to set the client IP from the X-Forwarded-For header set by Varnish. This
way, you will not need to worry about whitelisting localhost, or other
cucumbersome iptables rules, and your IP restrictions will work as intended.

On Fri, Mar 17, 2017 at 1:32 AM, Jason Price <japrice@gmail.com> wrote:

> I don't believe there's a trivial way to do this.
>
> Varnish will return the cached response to any IP address that comes
> calling. Even if the first request comes from a valid IP, which gets
> passed through via X-Forward or similar, and mod_auth is tweaked to respond
> to that, any subsequent request will not be seen by either apache or
> mod_auth at all.
>
> You have a few options:
> 1) IP Whitelists are a rather poor means of authentication. Moving to
> something else might be prudent. But that's not easy.
> 2) There are probably VMODs that do something similar. If not and if the
> list of IPs isn't too long, you could limit the IPs in VCL rather than
> mod_auth.
> 3) Push the list of IP addresses that can connect to the external port
> down to IPTables or similar.
> 4) Push the list of IP addresses to external Firewall, or Security Group
> or whatever.
>
>
>
> On Thu, Mar 16, 2017 at 5:46 PM, Hernán Marsili <hernan@cmsmedios.com>
> wrote:
>
>> Hi,
>>
>> We are having an issue with VARNISH and apache mod_auth. Varnish is on
>> port 80 serving users and Apache is the backend.
>>
>> We have servers restricting access only to authenticated users or certain
>> IP addresses. Since we installed Varnish the issue is that we need to
>> enable 127.0.0.1 as a permitted IP (required ip rule) so the Varnish can
>> fetch content. The problem, is that the real IP is not used and all the
>> other rules does not apply.
>>
>> Bottom line, how can we still control who is requesting using MOD_AUTH
>> and having Varnish?
>>
>> Regards
>> Hernán.
>>
>> _______________________________________________
>> varnish-misc mailing list
>> varnish-misc@varnish-cache.org
>> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
>>
>
>
> _______________________________________________
> varnish-misc mailing list
> varnish-misc@varnish-cache.org
> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
>
Re: varnish with apache mod_auth [ In reply to ]
Thank you! so, I figure I can parse the x-forwarded-for in which I have 3
ips. The first one is the customer, the second one is the one 1 need (the
CDN) and the third I think is the load balancer.

I can assign it to a new header x-cdn-ip and use apache_remoteip to use
that ip as the connecting ip.

What do you think?

Only problem here is to parse the second iP. I have something like this:

set req.http.x-cdn-ip = regsub(req.http.X-Forwarded-For, "^([^,]+),?.*$",
"\1");

I was able to get the first IP but not the second only which is the one I
need. Any one can point me in the right direction with the regsub?

Thank you!

On Fri, Mar 17, 2017 at 4:43 AM Andrei <lagged@gmail.com> wrote:

> Authenticated requests should typically bypass cache, unless you want to
> hash the related session id(s), however that can get "interesting". I
> suggest using an Apache module such as rpaf or remoteip in order for Apache
> to set the client IP from the X-Forwarded-For header set by Varnish. This
> way, you will not need to worry about whitelisting localhost, or other
> cucumbersome iptables rules, and your IP restrictions will work as intended.
>
> On Fri, Mar 17, 2017 at 1:32 AM, Jason Price <japrice@gmail.com> wrote:
>
> I don't believe there's a trivial way to do this.
>
> Varnish will return the cached response to any IP address that comes
> calling. Even if the first request comes from a valid IP, which gets
> passed through via X-Forward or similar, and mod_auth is tweaked to respond
> to that, any subsequent request will not be seen by either apache or
> mod_auth at all.
>
> You have a few options:
> 1) IP Whitelists are a rather poor means of authentication. Moving to
> something else might be prudent. But that's not easy.
> 2) There are probably VMODs that do something similar. If not and if the
> list of IPs isn't too long, you could limit the IPs in VCL rather than
> mod_auth.
> 3) Push the list of IP addresses that can connect to the external port
> down to IPTables or similar.
> 4) Push the list of IP addresses to external Firewall, or Security Group
> or whatever.
>
>
>
> On Thu, Mar 16, 2017 at 5:46 PM, Hernán Marsili <hernan@cmsmedios.com>
> wrote:
>
> Hi,
>
> We are having an issue with VARNISH and apache mod_auth. Varnish is on
> port 80 serving users and Apache is the backend.
>
> We have servers restricting access only to authenticated users or certain
> IP addresses. Since we installed Varnish the issue is that we need to
> enable 127.0.0.1 as a permitted IP (required ip rule) so the Varnish can
> fetch content. The problem, is that the real IP is not used and all the
> other rules does not apply.
>
> Bottom line, how can we still control who is requesting using MOD_AUTH and
> having Varnish?
>
> Regards
> Hernán.
>
> _______________________________________________
> varnish-misc mailing list
> varnish-misc@varnish-cache.org
> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
>
>
>
> _______________________________________________
> varnish-misc mailing list
> varnish-misc@varnish-cache.org
> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
>
>
>
Re: varnish with apache mod_auth [ In reply to ]
If you have the ability to compile a vmod, you can use split() from
vmod-str (disclaimer: I wrote that)
https://github.com/gquintard/libvmod-str/blob/master/src/vmod_str.vcc

otherwise, to get the second ip, something like :

regsub(req.http.xff, "([^,]+), *([^ ,]+)[ ,]?.*", "\2")

should work. Fell free to test, using regex101.com for example. or better,
a Varnish Test case Case:
https://gist.github.com/gquintard/ee47432bb8b5c97b615d973b57b6338e
test it using: varnishtest foo.vtc

--
Guillaume Quintard

On Fri, Mar 17, 2017 at 1:33 PM, Hernán Marsili <hernan@cmsmedios.com>
wrote:

> Thank you! so, I figure I can parse the x-forwarded-for in which I have 3
> ips. The first one is the customer, the second one is the one 1 need (the
> CDN) and the third I think is the load balancer.
>
> I can assign it to a new header x-cdn-ip and use apache_remoteip to use
> that ip as the connecting ip.
>
> What do you think?
>
> Only problem here is to parse the second iP. I have something like this:
>
> set req.http.x-cdn-ip = regsub(req.http.X-Forwarded-For, "^([^,]+),?.*$",
> "\1");
>
> I was able to get the first IP but not the second only which is the one I
> need. Any one can point me in the right direction with the regsub?
>
> Thank you!
>
> On Fri, Mar 17, 2017 at 4:43 AM Andrei <lagged@gmail.com> wrote:
>
>> Authenticated requests should typically bypass cache, unless you want to
>> hash the related session id(s), however that can get "interesting". I
>> suggest using an Apache module such as rpaf or remoteip in order for Apache
>> to set the client IP from the X-Forwarded-For header set by Varnish. This
>> way, you will not need to worry about whitelisting localhost, or other
>> cucumbersome iptables rules, and your IP restrictions will work as intended.
>>
>> On Fri, Mar 17, 2017 at 1:32 AM, Jason Price <japrice@gmail.com> wrote:
>>
>> I don't believe there's a trivial way to do this.
>>
>> Varnish will return the cached response to any IP address that comes
>> calling. Even if the first request comes from a valid IP, which gets
>> passed through via X-Forward or similar, and mod_auth is tweaked to respond
>> to that, any subsequent request will not be seen by either apache or
>> mod_auth at all.
>>
>> You have a few options:
>> 1) IP Whitelists are a rather poor means of authentication. Moving to
>> something else might be prudent. But that's not easy.
>> 2) There are probably VMODs that do something similar. If not and if the
>> list of IPs isn't too long, you could limit the IPs in VCL rather than
>> mod_auth.
>> 3) Push the list of IP addresses that can connect to the external port
>> down to IPTables or similar.
>> 4) Push the list of IP addresses to external Firewall, or Security Group
>> or whatever.
>>
>>
>>
>> On Thu, Mar 16, 2017 at 5:46 PM, Hernán Marsili <hernan@cmsmedios.com>
>> wrote:
>>
>> Hi,
>>
>> We are having an issue with VARNISH and apache mod_auth. Varnish is on
>> port 80 serving users and Apache is the backend.
>>
>> We have servers restricting access only to authenticated users or certain
>> IP addresses. Since we installed Varnish the issue is that we need to
>> enable 127.0.0.1 as a permitted IP (required ip rule) so the Varnish can
>> fetch content. The problem, is that the real IP is not used and all the
>> other rules does not apply.
>>
>> Bottom line, how can we still control who is requesting using MOD_AUTH
>> and having Varnish?
>>
>> Regards
>> Hernán.
>>
>> _______________________________________________
>> varnish-misc mailing list
>> varnish-misc@varnish-cache.org
>> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
>>
>>
>>
>> _______________________________________________
>> varnish-misc mailing list
>> varnish-misc@varnish-cache.org
>> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
>>
>>
>>
> _______________________________________________
> varnish-misc mailing list
> varnish-misc@varnish-cache.org
> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
>
Re: varnish with apache mod_auth [ In reply to ]
Does the CDN not provide the IP you want in a separate header? Typically
CDN's have custom headers for just that which you can use as well

On Fri, Mar 17, 2017 at 3:31 PM, Guillaume Quintard <
guillaume@varnish-software.com> wrote:

> If you have the ability to compile a vmod, you can use split() from
> vmod-str (disclaimer: I wrote that) https://github.com/
> gquintard/libvmod-str/blob/master/src/vmod_str.vcc
>
> otherwise, to get the second ip, something like :
>
> regsub(req.http.xff, "([^,]+), *([^ ,]+)[ ,]?.*", "\2")
>
> should work. Fell free to test, using regex101.com for example. or
> better, a Varnish Test case Case: https://gist.github.com/gquintard/
> ee47432bb8b5c97b615d973b57b6338e
> test it using: varnishtest foo.vtc
>
> --
> Guillaume Quintard
>
> On Fri, Mar 17, 2017 at 1:33 PM, Hernán Marsili <hernan@cmsmedios.com>
> wrote:
>
>> Thank you! so, I figure I can parse the x-forwarded-for in which I have 3
>> ips. The first one is the customer, the second one is the one 1 need (the
>> CDN) and the third I think is the load balancer.
>>
>> I can assign it to a new header x-cdn-ip and use apache_remoteip to use
>> that ip as the connecting ip.
>>
>> What do you think?
>>
>> Only problem here is to parse the second iP. I have something like this:
>>
>> set req.http.x-cdn-ip = regsub(req.http.X-Forwarded-For,
>> "^([^,]+),?.*$", "\1");
>>
>> I was able to get the first IP but not the second only which is the one I
>> need. Any one can point me in the right direction with the regsub?
>>
>> Thank you!
>>
>> On Fri, Mar 17, 2017 at 4:43 AM Andrei <lagged@gmail.com> wrote:
>>
>>> Authenticated requests should typically bypass cache, unless you want to
>>> hash the related session id(s), however that can get "interesting". I
>>> suggest using an Apache module such as rpaf or remoteip in order for Apache
>>> to set the client IP from the X-Forwarded-For header set by Varnish. This
>>> way, you will not need to worry about whitelisting localhost, or other
>>> cucumbersome iptables rules, and your IP restrictions will work as intended.
>>>
>>> On Fri, Mar 17, 2017 at 1:32 AM, Jason Price <japrice@gmail.com> wrote:
>>>
>>> I don't believe there's a trivial way to do this.
>>>
>>> Varnish will return the cached response to any IP address that comes
>>> calling. Even if the first request comes from a valid IP, which gets
>>> passed through via X-Forward or similar, and mod_auth is tweaked to respond
>>> to that, any subsequent request will not be seen by either apache or
>>> mod_auth at all.
>>>
>>> You have a few options:
>>> 1) IP Whitelists are a rather poor means of authentication. Moving to
>>> something else might be prudent. But that's not easy.
>>> 2) There are probably VMODs that do something similar. If not and if
>>> the list of IPs isn't too long, you could limit the IPs in VCL rather than
>>> mod_auth.
>>> 3) Push the list of IP addresses that can connect to the external port
>>> down to IPTables or similar.
>>> 4) Push the list of IP addresses to external Firewall, or Security Group
>>> or whatever.
>>>
>>>
>>>
>>> On Thu, Mar 16, 2017 at 5:46 PM, Hernán Marsili <hernan@cmsmedios.com>
>>> wrote:
>>>
>>> Hi,
>>>
>>> We are having an issue with VARNISH and apache mod_auth. Varnish is on
>>> port 80 serving users and Apache is the backend.
>>>
>>> We have servers restricting access only to authenticated users or
>>> certain IP addresses. Since we installed Varnish the issue is that we need
>>> to enable 127.0.0.1 as a permitted IP (required ip rule) so the Varnish can
>>> fetch content. The problem, is that the real IP is not used and all the
>>> other rules does not apply.
>>>
>>> Bottom line, how can we still control who is requesting using MOD_AUTH
>>> and having Varnish?
>>>
>>> Regards
>>> Hernán.
>>>
>>> _______________________________________________
>>> varnish-misc mailing list
>>> varnish-misc@varnish-cache.org
>>> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
>>>
>>>
>>>
>>> _______________________________________________
>>> varnish-misc mailing list
>>> varnish-misc@varnish-cache.org
>>> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
>>>
>>>
>>>
>> _______________________________________________
>> varnish-misc mailing list
>> varnish-misc@varnish-cache.org
>> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
>>
>
>
Re: varnish with apache mod_auth [ In reply to ]
Ok, so I finally make it work with the suggested rule.

On the vcl_recv I have:

if (req.http.x-forwarded-for) {

set req.http.X-Forwarded-For = req.http.X-Forwarded-For + ", " +
client.ip;

set req.http.x-cdn-ip = regsub(req.http.X-Forwarded-For, "([^,]+), *([^
,]+)[ ,]?.*", "\2");

} else {

set req.http.X-Forwarded-For = client.ip;

set req.http.x-cdn-ip = regsub(req.http.X-Forwarded-For, "([^,]+), *([^
,]+)[ ,]?.*", "\2");

}

I then use Apache remote_ip to listen to x-cd-ip with this:

RemoteIPHeader x-cdn-ip

RemoteIPTrustedProxy 127.0.0.1 172.31.29.204

I don't probable need the IF but since this was in place for some reason, I
just leave it.

It seems to be working just fine. What do you think?

On Fri, Mar 17, 2017 at 10:32 AM Andrei <lagged@gmail.com> wrote:

> Does the CDN not provide the IP you want in a separate header? Typically
> CDN's have custom headers for just that which you can use as well
>
> On Fri, Mar 17, 2017 at 3:31 PM, Guillaume Quintard <
> guillaume@varnish-software.com> wrote:
>
> If you have the ability to compile a vmod, you can use split() from
> vmod-str (disclaimer: I wrote that)
> https://github.com/gquintard/libvmod-str/blob/master/src/vmod_str.vcc
>
> otherwise, to get the second ip, something like :
>
> regsub(req.http.xff, "([^,]+), *([^ ,]+)[ ,]?.*", "\2")
>
> should work. Fell free to test, using regex101.com for example. or
> better, a Varnish Test case Case:
> https://gist.github.com/gquintard/ee47432bb8b5c97b615d973b57b6338e
> test it using: varnishtest foo.vtc
>
> --
> Guillaume Quintard
>
> On Fri, Mar 17, 2017 at 1:33 PM, Hernán Marsili <hernan@cmsmedios.com>
> wrote:
>
> Thank you! so, I figure I can parse the x-forwarded-for in which I have 3
> ips. The first one is the customer, the second one is the one 1 need (the
> CDN) and the third I think is the load balancer.
>
> I can assign it to a new header x-cdn-ip and use apache_remoteip to use
> that ip as the connecting ip.
>
> What do you think?
>
> Only problem here is to parse the second iP. I have something like this:
>
> set req.http.x-cdn-ip = regsub(req.http.X-Forwarded-For, "^([^,]+),?.*$",
> "\1");
>
> I was able to get the first IP but not the second only which is the one I
> need. Any one can point me in the right direction with the regsub?
>
> Thank you!
>
> On Fri, Mar 17, 2017 at 4:43 AM Andrei <lagged@gmail.com> wrote:
>
> Authenticated requests should typically bypass cache, unless you want to
> hash the related session id(s), however that can get "interesting". I
> suggest using an Apache module such as rpaf or remoteip in order for Apache
> to set the client IP from the X-Forwarded-For header set by Varnish. This
> way, you will not need to worry about whitelisting localhost, or other
> cucumbersome iptables rules, and your IP restrictions will work as intended.
>
> On Fri, Mar 17, 2017 at 1:32 AM, Jason Price <japrice@gmail.com> wrote:
>
> I don't believe there's a trivial way to do this.
>
> Varnish will return the cached response to any IP address that comes
> calling. Even if the first request comes from a valid IP, which gets
> passed through via X-Forward or similar, and mod_auth is tweaked to respond
> to that, any subsequent request will not be seen by either apache or
> mod_auth at all.
>
> You have a few options:
> 1) IP Whitelists are a rather poor means of authentication. Moving to
> something else might be prudent. But that's not easy.
> 2) There are probably VMODs that do something similar. If not and if the
> list of IPs isn't too long, you could limit the IPs in VCL rather than
> mod_auth.
> 3) Push the list of IP addresses that can connect to the external port
> down to IPTables or similar.
> 4) Push the list of IP addresses to external Firewall, or Security Group
> or whatever.
>
>
>
> On Thu, Mar 16, 2017 at 5:46 PM, Hernán Marsili <hernan@cmsmedios.com>
> wrote:
>
> Hi,
>
> We are having an issue with VARNISH and apache mod_auth. Varnish is on
> port 80 serving users and Apache is the backend.
>
> We have servers restricting access only to authenticated users or certain
> IP addresses. Since we installed Varnish the issue is that we need to
> enable 127.0.0.1 as a permitted IP (required ip rule) so the Varnish can
> fetch content. The problem, is that the real IP is not used and all the
> other rules does not apply.
>
> Bottom line, how can we still control who is requesting using MOD_AUTH and
> having Varnish?
>
> Regards
> Hernán.
>
> _______________________________________________
> varnish-misc mailing list
> varnish-misc@varnish-cache.org
> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
>
>
>
> _______________________________________________
> varnish-misc mailing list
> varnish-misc@varnish-cache.org
> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
>
>
>
> _______________________________________________
> varnish-misc mailing list
> varnish-misc@varnish-cache.org
> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
>
>
>
>
Re: varnish with apache mod_auth [ In reply to ]
Actually, Varnish should set the XFF header even before you enter vcl_recv.

--
Guillaume Quintard

On Mar 17, 2017 19:23, "Hernán Marsili" <hernan@cmsmedios.com> wrote:

> Ok, so I finally make it work with the suggested rule.
>
> On the vcl_recv I have:
>
> if (req.http.x-forwarded-for) {
>
> set req.http.X-Forwarded-For = req.http.X-Forwarded-For + ", " +
> client.ip;
>
> set req.http.x-cdn-ip = regsub(req.http.X-Forwarded-For, "([^,]+), *([^
> ,]+)[ ,]?.*", "\2");
>
> } else {
>
> set req.http.X-Forwarded-For = client.ip;
>
> set req.http.x-cdn-ip = regsub(req.http.X-Forwarded-For, "([^,]+), *([^
> ,]+)[ ,]?.*", "\2");
>
> }
>
> I then use Apache remote_ip to listen to x-cd-ip with this:
>
> RemoteIPHeader x-cdn-ip
>
> RemoteIPTrustedProxy 127.0.0.1 172.31.29.204
>
> I don't probable need the IF but since this was in place for some reason,
> I just leave it.
>
> It seems to be working just fine. What do you think?
>
> On Fri, Mar 17, 2017 at 10:32 AM Andrei <lagged@gmail.com> wrote:
>
>> Does the CDN not provide the IP you want in a separate header? Typically
>> CDN's have custom headers for just that which you can use as well
>>
>> On Fri, Mar 17, 2017 at 3:31 PM, Guillaume Quintard <
>> guillaume@varnish-software.com> wrote:
>>
>> If you have the ability to compile a vmod, you can use split() from
>> vmod-str (disclaimer: I wrote that) https://github.com/
>> gquintard/libvmod-str/blob/master/src/vmod_str.vcc
>>
>> otherwise, to get the second ip, something like :
>>
>> regsub(req.http.xff, "([^,]+), *([^ ,]+)[ ,]?.*", "\2")
>>
>> should work. Fell free to test, using regex101.com for example. or
>> better, a Varnish Test case Case: https://gist.github.com/gquintard/
>> ee47432bb8b5c97b615d973b57b6338e
>> test it using: varnishtest foo.vtc
>>
>> --
>> Guillaume Quintard
>>
>> On Fri, Mar 17, 2017 at 1:33 PM, Hernán Marsili <hernan@cmsmedios.com>
>> wrote:
>>
>> Thank you! so, I figure I can parse the x-forwarded-for in which I have 3
>> ips. The first one is the customer, the second one is the one 1 need (the
>> CDN) and the third I think is the load balancer.
>>
>> I can assign it to a new header x-cdn-ip and use apache_remoteip to use
>> that ip as the connecting ip.
>>
>> What do you think?
>>
>> Only problem here is to parse the second iP. I have something like this:
>>
>> set req.http.x-cdn-ip = regsub(req.http.X-Forwarded-For,
>> "^([^,]+),?.*$", "\1");
>>
>> I was able to get the first IP but not the second only which is the one I
>> need. Any one can point me in the right direction with the regsub?
>>
>> Thank you!
>>
>> On Fri, Mar 17, 2017 at 4:43 AM Andrei <lagged@gmail.com> wrote:
>>
>> Authenticated requests should typically bypass cache, unless you want to
>> hash the related session id(s), however that can get "interesting". I
>> suggest using an Apache module such as rpaf or remoteip in order for Apache
>> to set the client IP from the X-Forwarded-For header set by Varnish. This
>> way, you will not need to worry about whitelisting localhost, or other
>> cucumbersome iptables rules, and your IP restrictions will work as intended.
>>
>> On Fri, Mar 17, 2017 at 1:32 AM, Jason Price <japrice@gmail.com> wrote:
>>
>> I don't believe there's a trivial way to do this.
>>
>> Varnish will return the cached response to any IP address that comes
>> calling. Even if the first request comes from a valid IP, which gets
>> passed through via X-Forward or similar, and mod_auth is tweaked to respond
>> to that, any subsequent request will not be seen by either apache or
>> mod_auth at all.
>>
>> You have a few options:
>> 1) IP Whitelists are a rather poor means of authentication. Moving to
>> something else might be prudent. But that's not easy.
>> 2) There are probably VMODs that do something similar. If not and if the
>> list of IPs isn't too long, you could limit the IPs in VCL rather than
>> mod_auth.
>> 3) Push the list of IP addresses that can connect to the external port
>> down to IPTables or similar.
>> 4) Push the list of IP addresses to external Firewall, or Security Group
>> or whatever.
>>
>>
>>
>> On Thu, Mar 16, 2017 at 5:46 PM, Hernán Marsili <hernan@cmsmedios.com>
>> wrote:
>>
>> Hi,
>>
>> We are having an issue with VARNISH and apache mod_auth. Varnish is on
>> port 80 serving users and Apache is the backend.
>>
>> We have servers restricting access only to authenticated users or certain
>> IP addresses. Since we installed Varnish the issue is that we need to
>> enable 127.0.0.1 as a permitted IP (required ip rule) so the Varnish can
>> fetch content. The problem, is that the real IP is not used and all the
>> other rules does not apply.
>>
>> Bottom line, how can we still control who is requesting using MOD_AUTH
>> and having Varnish?
>>
>> Regards
>> Hernán.
>>
>> _______________________________________________
>> varnish-misc mailing list
>> varnish-misc@varnish-cache.org
>> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
>>
>>
>>
>> _______________________________________________
>> varnish-misc mailing list
>> varnish-misc@varnish-cache.org
>> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
>>
>>
>>
>> _______________________________________________
>> varnish-misc mailing list
>> varnish-misc@varnish-cache.org
>> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
>>
>>
>>
>>
Re: varnish with apache mod_auth [ In reply to ]
Out of curiosity, has anyone done a CDN of Varnish servers? I have 4
Varnish servers in different datacenters around the world, and use anycast
IPs to direct traffic based on the region. I managed to do cache
replication using a "fanout" method for new cache hits to be replicated
through an intermediary server to the related group of Varnish servers, but
was wondering if anyone had a better method.

On Fri, Mar 17, 2017 at 2:48 PM, Guillaume Quintard <
guillaume@varnish-software.com> wrote:

> Actually, Varnish should set the XFF header even before you enter
> vcl_recv.
>
> --
> Guillaume Quintard
>
> On Mar 17, 2017 19:23, "Hernán Marsili" <hernan@cmsmedios.com> wrote:
>
>> Ok, so I finally make it work with the suggested rule.
>>
>> On the vcl_recv I have:
>>
>> if (req.http.x-forwarded-for) {
>>
>> set req.http.X-Forwarded-For = req.http.X-Forwarded-For + ", " +
>> client.ip;
>>
>> set req.http.x-cdn-ip = regsub(req.http.X-Forwarded-For, "([^,]+), *([^
>> ,]+)[ ,]?.*", "\2");
>>
>> } else {
>>
>> set req.http.X-Forwarded-For = client.ip;
>>
>> set req.http.x-cdn-ip = regsub(req.http.X-Forwarded-For, "([^,]+), *([^
>> ,]+)[ ,]?.*", "\2");
>>
>> }
>>
>> I then use Apache remote_ip to listen to x-cd-ip with this:
>>
>> RemoteIPHeader x-cdn-ip
>>
>> RemoteIPTrustedProxy 127.0.0.1 172.31.29.204
>>
>> I don't probable need the IF but since this was in place for some reason,
>> I just leave it.
>>
>> It seems to be working just fine. What do you think?
>>
>> On Fri, Mar 17, 2017 at 10:32 AM Andrei <lagged@gmail.com> wrote:
>>
>>> Does the CDN not provide the IP you want in a separate header? Typically
>>> CDN's have custom headers for just that which you can use as well
>>>
>>> On Fri, Mar 17, 2017 at 3:31 PM, Guillaume Quintard <
>>> guillaume@varnish-software.com> wrote:
>>>
>>> If you have the ability to compile a vmod, you can use split() from
>>> vmod-str (disclaimer: I wrote that) https://github.com/gquin
>>> tard/libvmod-str/blob/master/src/vmod_str.vcc
>>>
>>> otherwise, to get the second ip, something like :
>>>
>>> regsub(req.http.xff, "([^,]+), *([^ ,]+)[ ,]?.*", "\2")
>>>
>>> should work. Fell free to test, using regex101.com for example. or
>>> better, a Varnish Test case Case: https://gist.github.com/
>>> gquintard/ee47432bb8b5c97b615d973b57b6338e
>>> test it using: varnishtest foo.vtc
>>>
>>> --
>>> Guillaume Quintard
>>>
>>> On Fri, Mar 17, 2017 at 1:33 PM, Hernán Marsili <hernan@cmsmedios.com>
>>> wrote:
>>>
>>> Thank you! so, I figure I can parse the x-forwarded-for in which I have
>>> 3 ips. The first one is the customer, the second one is the one 1 need (the
>>> CDN) and the third I think is the load balancer.
>>>
>>> I can assign it to a new header x-cdn-ip and use apache_remoteip to use
>>> that ip as the connecting ip.
>>>
>>> What do you think?
>>>
>>> Only problem here is to parse the second iP. I have something like this:
>>>
>>> set req.http.x-cdn-ip = regsub(req.http.X-Forwarded-For,
>>> "^([^,]+),?.*$", "\1");
>>>
>>> I was able to get the first IP but not the second only which is the one
>>> I need. Any one can point me in the right direction with the regsub?
>>>
>>> Thank you!
>>>
>>> On Fri, Mar 17, 2017 at 4:43 AM Andrei <lagged@gmail.com> wrote:
>>>
>>> Authenticated requests should typically bypass cache, unless you want to
>>> hash the related session id(s), however that can get "interesting". I
>>> suggest using an Apache module such as rpaf or remoteip in order for Apache
>>> to set the client IP from the X-Forwarded-For header set by Varnish. This
>>> way, you will not need to worry about whitelisting localhost, or other
>>> cucumbersome iptables rules, and your IP restrictions will work as intended.
>>>
>>> On Fri, Mar 17, 2017 at 1:32 AM, Jason Price <japrice@gmail.com> wrote:
>>>
>>> I don't believe there's a trivial way to do this.
>>>
>>> Varnish will return the cached response to any IP address that comes
>>> calling. Even if the first request comes from a valid IP, which gets
>>> passed through via X-Forward or similar, and mod_auth is tweaked to respond
>>> to that, any subsequent request will not be seen by either apache or
>>> mod_auth at all.
>>>
>>> You have a few options:
>>> 1) IP Whitelists are a rather poor means of authentication. Moving to
>>> something else might be prudent. But that's not easy.
>>> 2) There are probably VMODs that do something similar. If not and if
>>> the list of IPs isn't too long, you could limit the IPs in VCL rather than
>>> mod_auth.
>>> 3) Push the list of IP addresses that can connect to the external port
>>> down to IPTables or similar.
>>> 4) Push the list of IP addresses to external Firewall, or Security Group
>>> or whatever.
>>>
>>>
>>>
>>> On Thu, Mar 16, 2017 at 5:46 PM, Hernán Marsili <hernan@cmsmedios.com>
>>> wrote:
>>>
>>> Hi,
>>>
>>> We are having an issue with VARNISH and apache mod_auth. Varnish is on
>>> port 80 serving users and Apache is the backend.
>>>
>>> We have servers restricting access only to authenticated users or
>>> certain IP addresses. Since we installed Varnish the issue is that we need
>>> to enable 127.0.0.1 as a permitted IP (required ip rule) so the Varnish can
>>> fetch content. The problem, is that the real IP is not used and all the
>>> other rules does not apply.
>>>
>>> Bottom line, how can we still control who is requesting using MOD_AUTH
>>> and having Varnish?
>>>
>>> Regards
>>> Hernán.
>>>
>>> _______________________________________________
>>> varnish-misc mailing list
>>> varnish-misc@varnish-cache.org
>>> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
>>>
>>>
>>>
>>> _______________________________________________
>>> varnish-misc mailing list
>>> varnish-misc@varnish-cache.org
>>> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
>>>
>>>
>>>
>>> _______________________________________________
>>> varnish-misc mailing list
>>> varnish-misc@varnish-cache.org
>>> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
>>>
>>>
>>>
>>>
Re: varnish with apache mod_auth [ In reply to ]
whoops, I totally meant to reply to the load balancing thread :|

On Sat, Mar 18, 2017 at 1:11 AM, Andrei <lagged@gmail.com> wrote:

> Out of curiosity, has anyone done a CDN of Varnish servers? I have 4
> Varnish servers in different datacenters around the world, and use anycast
> IPs to direct traffic based on the region. I managed to do cache
> replication using a "fanout" method for new cache hits to be replicated
> through an intermediary server to the related group of Varnish servers, but
> was wondering if anyone had a better method.
>
> On Fri, Mar 17, 2017 at 2:48 PM, Guillaume Quintard <
> guillaume@varnish-software.com> wrote:
>
>> Actually, Varnish should set the XFF header even before you enter
>> vcl_recv.
>>
>> --
>> Guillaume Quintard
>>
>> On Mar 17, 2017 19:23, "Hernán Marsili" <hernan@cmsmedios.com> wrote:
>>
>>> Ok, so I finally make it work with the suggested rule.
>>>
>>> On the vcl_recv I have:
>>>
>>> if (req.http.x-forwarded-for) {
>>>
>>> set req.http.X-Forwarded-For = req.http.X-Forwarded-For + ", "
>>> + client.ip;
>>>
>>> set req.http.x-cdn-ip = regsub(req.http.X-Forwarded-For, "([^,]+), *([^
>>> ,]+)[ ,]?.*", "\2");
>>>
>>> } else {
>>>
>>> set req.http.X-Forwarded-For = client.ip;
>>>
>>> set req.http.x-cdn-ip = regsub(req.http.X-Forwarded-For, "([^,]+), *([^
>>> ,]+)[ ,]?.*", "\2");
>>>
>>> }
>>>
>>> I then use Apache remote_ip to listen to x-cd-ip with this:
>>>
>>> RemoteIPHeader x-cdn-ip
>>>
>>> RemoteIPTrustedProxy 127.0.0.1 172.31.29.204
>>>
>>> I don't probable need the IF but since this was in place for some
>>> reason, I just leave it.
>>>
>>> It seems to be working just fine. What do you think?
>>>
>>> On Fri, Mar 17, 2017 at 10:32 AM Andrei <lagged@gmail.com> wrote:
>>>
>>>> Does the CDN not provide the IP you want in a separate header?
>>>> Typically CDN's have custom headers for just that which you can use as well
>>>>
>>>> On Fri, Mar 17, 2017 at 3:31 PM, Guillaume Quintard <
>>>> guillaume@varnish-software.com> wrote:
>>>>
>>>> If you have the ability to compile a vmod, you can use split() from
>>>> vmod-str (disclaimer: I wrote that) https://github.com/gquin
>>>> tard/libvmod-str/blob/master/src/vmod_str.vcc
>>>>
>>>> otherwise, to get the second ip, something like :
>>>>
>>>> regsub(req.http.xff, "([^,]+), *([^ ,]+)[ ,]?.*", "\2")
>>>>
>>>> should work. Fell free to test, using regex101.com for example. or
>>>> better, a Varnish Test case Case: https://gist.github.com/
>>>> gquintard/ee47432bb8b5c97b615d973b57b6338e
>>>> test it using: varnishtest foo.vtc
>>>>
>>>> --
>>>> Guillaume Quintard
>>>>
>>>> On Fri, Mar 17, 2017 at 1:33 PM, Hernán Marsili <hernan@cmsmedios.com>
>>>> wrote:
>>>>
>>>> Thank you! so, I figure I can parse the x-forwarded-for in which I have
>>>> 3 ips. The first one is the customer, the second one is the one 1 need (the
>>>> CDN) and the third I think is the load balancer.
>>>>
>>>> I can assign it to a new header x-cdn-ip and use apache_remoteip to use
>>>> that ip as the connecting ip.
>>>>
>>>> What do you think?
>>>>
>>>> Only problem here is to parse the second iP. I have something like this:
>>>>
>>>> set req.http.x-cdn-ip = regsub(req.http.X-Forwarded-For,
>>>> "^([^,]+),?.*$", "\1");
>>>>
>>>> I was able to get the first IP but not the second only which is the one
>>>> I need. Any one can point me in the right direction with the regsub?
>>>>
>>>> Thank you!
>>>>
>>>> On Fri, Mar 17, 2017 at 4:43 AM Andrei <lagged@gmail.com> wrote:
>>>>
>>>> Authenticated requests should typically bypass cache, unless you want
>>>> to hash the related session id(s), however that can get "interesting". I
>>>> suggest using an Apache module such as rpaf or remoteip in order for Apache
>>>> to set the client IP from the X-Forwarded-For header set by Varnish. This
>>>> way, you will not need to worry about whitelisting localhost, or other
>>>> cucumbersome iptables rules, and your IP restrictions will work as intended.
>>>>
>>>> On Fri, Mar 17, 2017 at 1:32 AM, Jason Price <japrice@gmail.com> wrote:
>>>>
>>>> I don't believe there's a trivial way to do this.
>>>>
>>>> Varnish will return the cached response to any IP address that comes
>>>> calling. Even if the first request comes from a valid IP, which gets
>>>> passed through via X-Forward or similar, and mod_auth is tweaked to respond
>>>> to that, any subsequent request will not be seen by either apache or
>>>> mod_auth at all.
>>>>
>>>> You have a few options:
>>>> 1) IP Whitelists are a rather poor means of authentication. Moving to
>>>> something else might be prudent. But that's not easy.
>>>> 2) There are probably VMODs that do something similar. If not and if
>>>> the list of IPs isn't too long, you could limit the IPs in VCL rather than
>>>> mod_auth.
>>>> 3) Push the list of IP addresses that can connect to the external port
>>>> down to IPTables or similar.
>>>> 4) Push the list of IP addresses to external Firewall, or Security
>>>> Group or whatever.
>>>>
>>>>
>>>>
>>>> On Thu, Mar 16, 2017 at 5:46 PM, Hernán Marsili <hernan@cmsmedios.com>
>>>> wrote:
>>>>
>>>> Hi,
>>>>
>>>> We are having an issue with VARNISH and apache mod_auth. Varnish is on
>>>> port 80 serving users and Apache is the backend.
>>>>
>>>> We have servers restricting access only to authenticated users or
>>>> certain IP addresses. Since we installed Varnish the issue is that we need
>>>> to enable 127.0.0.1 as a permitted IP (required ip rule) so the Varnish can
>>>> fetch content. The problem, is that the real IP is not used and all the
>>>> other rules does not apply.
>>>>
>>>> Bottom line, how can we still control who is requesting using MOD_AUTH
>>>> and having Varnish?
>>>>
>>>> Regards
>>>> Hernán.
>>>>
>>>> _______________________________________________
>>>> varnish-misc mailing list
>>>> varnish-misc@varnish-cache.org
>>>> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
>>>>
>>>>
>>>>
>>>> _______________________________________________
>>>> varnish-misc mailing list
>>>> varnish-misc@varnish-cache.org
>>>> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
>>>>
>>>>
>>>>
>>>> _______________________________________________
>>>> varnish-misc mailing list
>>>> varnish-misc@varnish-cache.org
>>>> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
>>>>
>>>>
>>>>
>>>>
>
Re: varnish with apache mod_auth [ In reply to ]
Well, this finally did not work since 0,7% of the users end up with a 401
ACCESS DENIED. The problem seems to be users behind a proxy in which the
structure of the IPs on the xff is changed. ;(

On Fri, Mar 17, 2017 at 4:48 PM Guillaume Quintard <
guillaume@varnish-software.com> wrote:

> Actually, Varnish should set the XFF header even before you enter
> vcl_recv.
>
>
> --
> Guillaume Quintard
>
> On Mar 17, 2017 19:23, "Hernán Marsili" <hernan@cmsmedios.com> wrote:
>
> Ok, so I finally make it work with the suggested rule.
>
> On the vcl_recv I have:
>
> if (req.http.x-forwarded-for) {
>
> set req.http.X-Forwarded-For = req.http.X-Forwarded-For + ", " +
> client.ip;
>
> set req.http.x-cdn-ip = regsub(req.http.X-Forwarded-For, "([^,]+), *([^
> ,]+)[ ,]?.*", "\2");
>
> } else {
>
> set req.http.X-Forwarded-For = client.ip;
>
> set req.http.x-cdn-ip = regsub(req.http.X-Forwarded-For, "([^,]+), *([^
> ,]+)[ ,]?.*", "\2");
>
> }
>
> I then use Apache remote_ip to listen to x-cd-ip with this:
>
> RemoteIPHeader x-cdn-ip
>
> RemoteIPTrustedProxy 127.0.0.1 172.31.29.204
>
> I don't probable need the IF but since this was in place for some reason,
> I just leave it.
>
> It seems to be working just fine. What do you think?
>
> On Fri, Mar 17, 2017 at 10:32 AM Andrei <lagged@gmail.com> wrote:
>
> Does the CDN not provide the IP you want in a separate header? Typically
> CDN's have custom headers for just that which you can use as well
>
> On Fri, Mar 17, 2017 at 3:31 PM, Guillaume Quintard <
> guillaume@varnish-software.com> wrote:
>
> If you have the ability to compile a vmod, you can use split() from
> vmod-str (disclaimer: I wrote that)
> https://github.com/gquintard/libvmod-str/blob/master/src/vmod_str.vcc
>
> otherwise, to get the second ip, something like :
>
> regsub(req.http.xff, "([^,]+), *([^ ,]+)[ ,]?.*", "\2")
>
> should work. Fell free to test, using regex101.com for example. or
> better, a Varnish Test case Case:
> https://gist.github.com/gquintard/ee47432bb8b5c97b615d973b57b6338e
> test it using: varnishtest foo.vtc
>
> --
> Guillaume Quintard
>
> On Fri, Mar 17, 2017 at 1:33 PM, Hernán Marsili <hernan@cmsmedios.com>
> wrote:
>
> Thank you! so, I figure I can parse the x-forwarded-for in which I have 3
> ips. The first one is the customer, the second one is the one 1 need (the
> CDN) and the third I think is the load balancer.
>
> I can assign it to a new header x-cdn-ip and use apache_remoteip to use
> that ip as the connecting ip.
>
> What do you think?
>
> Only problem here is to parse the second iP. I have something like this:
>
> set req.http.x-cdn-ip = regsub(req.http.X-Forwarded-For, "^([^,]+),?.*$",
> "\1");
>
> I was able to get the first IP but not the second only which is the one I
> need. Any one can point me in the right direction with the regsub?
>
> Thank you!
>
> On Fri, Mar 17, 2017 at 4:43 AM Andrei <lagged@gmail.com> wrote:
>
> Authenticated requests should typically bypass cache, unless you want to
> hash the related session id(s), however that can get "interesting". I
> suggest using an Apache module such as rpaf or remoteip in order for Apache
> to set the client IP from the X-Forwarded-For header set by Varnish. This
> way, you will not need to worry about whitelisting localhost, or other
> cucumbersome iptables rules, and your IP restrictions will work as intended.
>
> On Fri, Mar 17, 2017 at 1:32 AM, Jason Price <japrice@gmail.com> wrote:
>
> I don't believe there's a trivial way to do this.
>
> Varnish will return the cached response to any IP address that comes
> calling. Even if the first request comes from a valid IP, which gets
> passed through via X-Forward or similar, and mod_auth is tweaked to respond
> to that, any subsequent request will not be seen by either apache or
> mod_auth at all.
>
> You have a few options:
> 1) IP Whitelists are a rather poor means of authentication. Moving to
> something else might be prudent. But that's not easy.
> 2) There are probably VMODs that do something similar. If not and if the
> list of IPs isn't too long, you could limit the IPs in VCL rather than
> mod_auth.
> 3) Push the list of IP addresses that can connect to the external port
> down to IPTables or similar.
> 4) Push the list of IP addresses to external Firewall, or Security Group
> or whatever.
>
>
>
> On Thu, Mar 16, 2017 at 5:46 PM, Hernán Marsili <hernan@cmsmedios.com>
> wrote:
>
> Hi,
>
> We are having an issue with VARNISH and apache mod_auth. Varnish is on
> port 80 serving users and Apache is the backend.
>
> We have servers restricting access only to authenticated users or certain
> IP addresses. Since we installed Varnish the issue is that we need to
> enable 127.0.0.1 as a permitted IP (required ip rule) so the Varnish can
> fetch content. The problem, is that the real IP is not used and all the
> other rules does not apply.
>
> Bottom line, how can we still control who is requesting using MOD_AUTH and
> having Varnish?
>
> Regards
> Hernán.
>
> _______________________________________________
> varnish-misc mailing list
> varnish-misc@varnish-cache.org
> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
>
>
>
> _______________________________________________
> varnish-misc mailing list
> varnish-misc@varnish-cache.org
> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
>
>
>
> _______________________________________________
> varnish-misc mailing list
> varnish-misc@varnish-cache.org
> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
>
>
>
>