Mailing List Archive

Varnish and AWS ALBs
Hi all,

Is anyone else running Varnish behind AWS ALBs? I just encountered an issue today with how I have been using X-Forwarded-For to check against a Varnish ACL in that is more restrictive than the ALB's security group, but I realized the hard way that since X-Forwarded-For can be arbitrarily set, a malicious actor can set it to an address that is permitted by the Varnish ACL, whether through guessing or other knowledge. Since Varnish gets XFF from the ALB, which in turn trusts existing XFF headers, you can't then really trust client.ip since it's just taken from XFF. Unless I'm missing something...

I've opened a support case with AWS to see if there's a way to configure an ALB to not trust XFF and use the IP from the original TCP connection, but I'm not hopeful. I'll likely have to go back to using two ALBs rather than one relatively open one and one with a Varnish ACL for tigher controls to a certain subset of the web sites behind the single ALB.

Justin
Re: Varnish and AWS ALBs [ In reply to ]
Hi Justin,

You cannot rely on the first IP in XFF (I guess you're doing that at the moment), but you can rely on the next-to-last. The last one is added to XFF by Varnish before entering 'vcl_recv', and the next-to-last is added by the ALB. That's the client IP as seen by the ALB and cannot be forged by clients.

Best,

--
Carlos Abalde

> On 19 Aug 2021, at 22:11, Justin Lloyd <justinl@arena.net> wrote:
>
> Hi all,
>
> Is anyone else running Varnish behind AWS ALBs? I just encountered an issue today with how I have been using X-Forwarded-For to check against a Varnish ACL in that is more restrictive than the ALB’s security group, but I realized the hard way that since X-Forwarded-For can be arbitrarily set, a malicious actor can set it to an address that is permitted by the Varnish ACL, whether through guessing or other knowledge. Since Varnish gets XFF from the ALB, which in turn trusts existing XFF headers, you can’t then really trust client.ip since it’s just taken from XFF. Unless I’m missing something...
>
> I’ve opened a support case with AWS to see if there’s a way to configure an ALB to not trust XFF and use the IP from the original TCP connection, but I’m not hopeful. I’ll likely have to go back to using two ALBs rather than one relatively open one and one with a Varnish ACL for tigher controls to a certain subset of the web sites behind the single ALB.
>
> Justin
>
> _______________________________________________
> varnish-misc mailing list
> varnish-misc@varnish-cache.org <mailto:varnish-misc@varnish-cache.org>
> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc <https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc>
RE: Varnish and AWS ALBs [ In reply to ]
Hi Carlos,

Correct, but that’s my problem. There are only two IPs in XFF: the original sender (and thus the one that can be spoofed) and the ALB’s IP, which is basically useless. I realize I can’t trust the first IP, but the client IP being the ALB doesn’t help because I need to restrict by the actual source IP, which apparently (unless AWS support tells me differently) I can’t actually know for sure once the request is behind the ALB. If that is the case, then I’ll definitely have to go back to the two ALB solution since then the special Varnish ACL can be handled once again by the ALB’s SG. I don’t like having two ALBs to manage, but it seems it’s probably the only way to handle this situation.

Justin

From: Carlos Abalde <carlos.abalde@gmail.com>
Sent: Thursday, August 19, 2021 1:35 PM
To: Justin Lloyd <justinl@arena.net>
Cc: varnish-misc@varnish-cache.org
Subject: Re: Varnish and AWS ALBs

Hi Justin,

You cannot rely on the first IP in XFF (I guess you're doing that at the moment), but you can rely on the next-to-last. The last one is added to XFF by Varnish before entering 'vcl_recv', and the next-to-last is added by the ALB. That's the client IP as seen by the ALB and cannot be forged by clients.

Best,

--
Carlos Abalde


On 19 Aug 2021, at 22:11, Justin Lloyd <justinl@arena.net<mailto:justinl@arena.net>> wrote:

Hi all,

Is anyone else running Varnish behind AWS ALBs? I just encountered an issue today with how I have been using X-Forwarded-For to check against a Varnish ACL in that is more restrictive than the ALB’s security group, but I realized the hard way that since X-Forwarded-For can be arbitrarily set, a malicious actor can set it to an address that is permitted by the Varnish ACL, whether through guessing or other knowledge. Since Varnish gets XFF from the ALB, which in turn trusts existing XFF headers, you can’t then really trust client.ip since it’s just taken from XFF. Unless I’m missing something...

I’ve opened a support case with AWS to see if there’s a way to configure an ALB to not trust XFF and use the IP from the original TCP connection, but I’m not hopeful. I’ll likely have to go back to using two ALBs rather than one relatively open one and one with a Varnish ACL for tigher controls to a certain subset of the web sites behind the single ALB.

Justin

_______________________________________________
varnish-misc mailing list
varnish-misc@varnish-cache.org<mailto:varnish-misc@varnish-cache.org>
https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
Re: Varnish and AWS ALBs [ In reply to ]
Hi,

No so sure about that. Let's assume the client address is 1.1.1.1. Two possible scenarios:

- The client request reaches the ALB without XFF. The ALB will inject XFF with value 1.1.1.1. Then Varnish will modify XFF adding the ALB's address (i.e., 1.1.1.1,<ALB IP>). Using the next-to-last IP you're using the right client address.

- The client request reaches the ALB with a forged XFF (e.g. 127.0.0.1). The ALB will will modify XFF (i.e. 127.0.0.1,1.1.1.1). The Varnish will do the same (i.e. 127.0.0.1,1.1.1.1,<ALB IP>). Using the next-to-last IP you're still using the right client address.

I've not checked using a ALB, but that should be the expected behaviour for me.

Best,

--
Carlos Abalde
Re: Varnish and AWS ALBs [ In reply to ]
Hi,

If I read this correctly:
https://docs.aws.amazon.com/elasticloadbalancing/latest/application/x-forwarded-headers.html
, you can trust the before-last IP, because it was added by the ALB,
always. (and using vmod_str makes it easy to retrieve
https://github.com/varnish/varnish-modules/blob/master/src/vmod_str.vcc#L42)

Side question: would an NLB work? They support proxy-protocol, that would
also solve your problem.

Cheers,

--
Guillaume Quintard


On Thu, Aug 19, 2021 at 1:52 PM Carlos Abalde <carlos.abalde@gmail.com>
wrote:

> Hi,
>
> No so sure about that. Let's assume the client address is 1.1.1.1. Two
> possible scenarios:
>
> - The client request reaches the ALB without XFF. The ALB will inject XFF
> with value 1.1.1.1. Then Varnish will modify XFF adding the ALB's address
> (i.e., 1.1.1.1,<ALB IP>). Using the next-to-last IP you're using the right
> client address.
>
> - The client request reaches the ALB with a forged XFF (e.g. 127.0.0.1).
> The ALB will will modify XFF (i.e. 127.0.0.1,1.1.1.1). The Varnish will do
> the same (i.e. 127.0.0.1,1.1.1.1,<ALB IP>). Using the next-to-last IP
> you're still using the right client address.
>
>
> I've not checked using a ALB, but that should be the expected behaviour
> for me.
>
> Best,
>
> --
> Carlos Abalde
>
> _______________________________________________
> varnish-misc mailing list
> varnish-misc@varnish-cache.org
> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
>
RE: Varnish and AWS ALBs [ In reply to ]
Hi Guillaume!

It looks like you and Carlos are both correct. For some reason, before I was not seeing the Varnish XFF values from faked XFFs, not sure why, but now I’m seeing the fakes I’m using against one of my dev sites and I’m seeing the three values where it’s FAKED_IP, REAL_IP, ALB_IP. So with a little bit more VCL code (or probably easier once I move to Varnish Enterprise next year), I should be able to handle this. I’ll give it a whirl and see how it goes.

Thanks!

Justin

From: Guillaume Quintard <guillaume.quintard@gmail.com>
Sent: Thursday, August 19, 2021 2:00 PM
To: Carlos Abalde <carlos.abalde@gmail.com>
Cc: Justin Lloyd <justinl@arena.net>; varnish-misc@varnish-cache.org
Subject: Re: Varnish and AWS ALBs

Hi,

If I read this correctly: https://docs.aws.amazon.com/elasticloadbalancing/latest/application/x-forwarded-headers.html , you can trust the before-last IP, because it was added by the ALB, always. (and using vmod_str makes it easy to retrieve https://github.com/varnish/varnish-modules/blob/master/src/vmod_str.vcc#L42)

Side question: would an NLB work? They support proxy-protocol, that would also solve your problem.

Cheers,

--
Guillaume Quintard


On Thu, Aug 19, 2021 at 1:52 PM Carlos Abalde <carlos.abalde@gmail.com<mailto:carlos.abalde@gmail.com>> wrote:
Hi,

No so sure about that. Let's assume the client address is 1.1.1.1. Two possible scenarios:

- The client request reaches the ALB without XFF. The ALB will inject XFF with value 1.1.1.1. Then Varnish will modify XFF adding the ALB's address (i.e., 1.1.1.1,<ALB IP>). Using the next-to-last IP you're using the right client address.

- The client request reaches the ALB with a forged XFF (e.g. 127.0.0.1). The ALB will will modify XFF (i.e. 127.0.0.1,1.1.1.1). The Varnish will do the same (i.e. 127.0.0.1,1.1.1.1,<ALB IP>). Using the next-to-last IP you're still using the right client address.

I've not checked using a ALB, but that should be the expected behaviour for me.

Best,

--
Carlos Abalde

_______________________________________________
varnish-misc mailing list
varnish-misc@varnish-cache.org<mailto:varnish-misc@varnish-cache.org>
https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
RE: Varnish and AWS ALBs [ In reply to ]
I was just trying to get varnish-modules to build (having to install varnish and build tools on my dev manager server) and hit the limitation that vmod_str isn’t available until Varnish 6.6. I’m on Varnish 6.5 so I’d need to test the 6.6 upgrade in dev and then roll that out to live, which will take some time (higher priority and urgency issues and projects on my plate). I’ll play with regsub() some more to see if I can figure out a temporary approach.

Thanks,
Justin


From: varnish-misc <varnish-misc-bounces+justinl=arena.net@varnish-cache.org> On Behalf Of Justin Lloyd
Sent: Thursday, August 19, 2021 2:39 PM
To: Guillaume Quintard <guillaume.quintard@gmail.com>; Carlos Abalde <carlos.abalde@gmail.com>
Cc: varnish-misc@varnish-cache.org
Subject: RE: Varnish and AWS ALBs

Hi Guillaume!

It looks like you and Carlos are both correct. For some reason, before I was not seeing the Varnish XFF values from faked XFFs, not sure why, but now I’m seeing the fakes I’m using against one of my dev sites and I’m seeing the three values where it’s FAKED_IP, REAL_IP, ALB_IP. So with a little bit more VCL code (or probably easier once I move to Varnish Enterprise next year), I should be able to handle this. I’ll give it a whirl and see how it goes.

Thanks!

Justin


From: Guillaume Quintard <guillaume.quintard@gmail.com<mailto:guillaume.quintard@gmail.com>>
Sent: Thursday, August 19, 2021 2:00 PM
To: Carlos Abalde <carlos.abalde@gmail.com<mailto:carlos.abalde@gmail.com>>
Cc: Justin Lloyd <justinl@arena.net<mailto:justinl@arena.net>>; varnish-misc@varnish-cache.org<mailto:varnish-misc@varnish-cache.org>
Subject: Re: Varnish and AWS ALBs

Hi,

If I read this correctly: https://docs.aws.amazon.com/elasticloadbalancing/latest/application/x-forwarded-headers.html , you can trust the before-last IP, because it was added by the ALB, always. (and using vmod_str makes it easy to retrieve https://github.com/varnish/varnish-modules/blob/master/src/vmod_str.vcc#L42)

Side question: would an NLB work? They support proxy-protocol, that would also solve your problem.

Cheers,

--
Guillaume Quintard


On Thu, Aug 19, 2021 at 1:52 PM Carlos Abalde <carlos.abalde@gmail.com<mailto:carlos.abalde@gmail.com>> wrote:
Hi,

No so sure about that. Let's assume the client address is 1.1.1.1. Two possible scenarios:

- The client request reaches the ALB without XFF. The ALB will inject XFF with value 1.1.1.1. Then Varnish will modify XFF adding the ALB's address (i.e., 1.1.1.1,<ALB IP>). Using the next-to-last IP you're using the right client address.

- The client request reaches the ALB with a forged XFF (e.g. 127.0.0.1). The ALB will will modify XFF (i.e. 127.0.0.1,1.1.1.1). The Varnish will do the same (i.e. 127.0.0.1,1.1.1.1,<ALB IP>). Using the next-to-last IP you're still using the right client address.

I've not checked using a ALB, but that should be the expected behaviour for me.

Best,

--
Carlos Abalde

_______________________________________________
varnish-misc mailing list
varnish-misc@varnish-cache.org<mailto:varnish-misc@varnish-cache.org>
https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
Re: Varnish and AWS ALBs [ In reply to ]
This is a possible regsub() to extract the next-to-last IP address (it assumes at lest two are available):

set req.http.X-Client-Ip = regsub(
req.http.X-Forwarded-For,
"^.*(?:^|,)\s*([^,\s]+)\s*,[^,]+$",
"\1");

Best,

--
Carlos Abalde

> On 20 Aug 2021, at 13:08, Justin Lloyd <justinl@arena.net> wrote:
>
> I was just trying to get varnish-modules to build (having to install varnish and build tools on my dev manager server) and hit the limitation that vmod_str isn’t available until Varnish 6.6. I’m on Varnish 6.5 so I’d need to test the 6.6 upgrade in dev and then roll that out to live, which will take some time (higher priority and urgency issues and projects on my plate). I’ll play with regsub() some more to see if I can figure out a temporary approach.
>
> Thanks,
> Justin
>
>
> From: varnish-misc <varnish-misc-bounces+justinl=arena.net@varnish-cache.org> On Behalf Of Justin Lloyd
> Sent: Thursday, August 19, 2021 2:39 PM
> To: Guillaume Quintard <guillaume.quintard@gmail.com>; Carlos Abalde <carlos.abalde@gmail.com>
> Cc: varnish-misc@varnish-cache.org
> Subject: RE: Varnish and AWS ALBs
>
> Hi Guillaume!
>
> It looks like you and Carlos are both correct. For some reason, before I was not seeing the Varnish XFF values from faked XFFs, not sure why, but now I’m seeing the fakes I’m using against one of my dev sites and I’m seeing the three values where it’s FAKED_IP, REAL_IP, ALB_IP. So with a little bit more VCL code (or probably easier once I move to Varnish Enterprise next year), I should be able to handle this. I’ll give it a whirl and see how it goes.
>
> Thanks!
>
> Justin
>
>
> From: Guillaume Quintard <guillaume.quintard@gmail.com <mailto:guillaume.quintard@gmail.com>>
> Sent: Thursday, August 19, 2021 2:00 PM
> To: Carlos Abalde <carlos.abalde@gmail.com <mailto:carlos.abalde@gmail.com>>
> Cc: Justin Lloyd <justinl@arena.net <mailto:justinl@arena.net>>; varnish-misc@varnish-cache.org <mailto:varnish-misc@varnish-cache.org>
> Subject: Re: Varnish and AWS ALBs
>
> Hi,
>
> If I read this correctly: https://docs.aws.amazon.com/elasticloadbalancing/latest/application/x-forwarded-headers.html <https://docs.aws.amazon.com/elasticloadbalancing/latest/application/x-forwarded-headers.html> , you can trust the before-last IP, because it was added by the ALB, always. (and using vmod_str makes it easy to retrieve https://github.com/varnish/varnish-modules/blob/master/src/vmod_str.vcc#L42 <https://github.com/varnish/varnish-modules/blob/master/src/vmod_str.vcc#L42>)
>
> Side question: would an NLB work? They support proxy-protocol, that would also solve your problem.
>
> Cheers,
>
> --
> Guillaume Quintard
>
>
> On Thu, Aug 19, 2021 at 1:52 PM Carlos Abalde <carlos.abalde@gmail.com <mailto:carlos.abalde@gmail.com>> wrote:
> Hi,
>
> No so sure about that. Let's assume the client address is 1.1.1.1. Two possible scenarios:
>
> - The client request reaches the ALB without XFF. The ALB will inject XFF with value 1.1.1.1. Then Varnish will modify XFF adding the ALB's address (i.e., 1.1.1.1,<ALB IP>). Using the next-to-last IP you're using the right client address.
>
> - The client request reaches the ALB with a forged XFF (e.g. 127.0.0.1). The ALB will will modify XFF (i.e. 127.0.0.1,1.1.1.1). The Varnish will do the same (i.e. 127.0.0.1,1.1.1.1,<ALB IP>). Using the next-to-last IP you're still using the right client address.
>
> I've not checked using a ALB, but that should be the expected behaviour for me.
>
> Best,
>
> --
> Carlos Abalde
>
> _______________________________________________
> varnish-misc mailing list
> varnish-misc@varnish-cache.org <mailto:varnish-misc@varnish-cache.org>
> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc <https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc>
RE: Varnish and AWS ALBs [ In reply to ]
Hey Carlos,

That seems to do the trick, thanks! I’ve always thought I was pretty good with PCRE (used Perl heavily for like 15 years) but for some reason this one was eluding me. I appreciate the help!

Justin

From: Carlos Abalde <carlos.abalde@gmail.com>
Sent: Friday, August 20, 2021 4:12 AM
To: Justin Lloyd <justinl@arena.net>
Cc: Guillaume Quintard <guillaume.quintard@gmail.com>; varnish-misc@varnish-cache.org
Subject: Re: Varnish and AWS ALBs

This is a possible regsub() to extract the next-to-last IP address (it assumes at lest two are available):

set req.http.X-Client-Ip = regsub(
req.http.X-Forwarded-For,
"^.*(?:^|,)\s*([^,\s]+)\s*,[^,]+$",
"\1");

Best,

--
Carlos Abalde


On 20 Aug 2021, at 13:08, Justin Lloyd <justinl@arena.net<mailto:justinl@arena.net>> wrote:

I was just trying to get varnish-modules to build (having to install varnish and build tools on my dev manager server) and hit the limitation that vmod_str isn’t available until Varnish 6.6. I’m on Varnish 6.5 so I’d need to test the 6.6 upgrade in dev and then roll that out to live, which will take some time (higher priority and urgency issues and projects on my plate). I’ll play with regsub() some more to see if I can figure out a temporary approach.

Thanks,
Justin


From: varnish-misc <varnish-misc-bounces+justinl=arena.net@varnish-cache.org<mailto:varnish-misc-bounces+justinl=arena.net@varnish-cache.org>> On Behalf Of Justin Lloyd
Sent: Thursday, August 19, 2021 2:39 PM
To: Guillaume Quintard <guillaume.quintard@gmail.com<mailto:guillaume.quintard@gmail.com>>; Carlos Abalde <carlos.abalde@gmail.com<mailto:carlos.abalde@gmail.com>>
Cc: varnish-misc@varnish-cache.org<mailto:varnish-misc@varnish-cache.org>
Subject: RE: Varnish and AWS ALBs

Hi Guillaume!

It looks like you and Carlos are both correct. For some reason, before I was not seeing the Varnish XFF values from faked XFFs, not sure why, but now I’m seeing the fakes I’m using against one of my dev sites and I’m seeing the three values where it’s FAKED_IP, REAL_IP, ALB_IP. So with a little bit more VCL code (or probably easier once I move to Varnish Enterprise next year), I should be able to handle this. I’ll give it a whirl and see how it goes.

Thanks!

Justin


From: Guillaume Quintard <guillaume.quintard@gmail.com<mailto:guillaume.quintard@gmail.com>>
Sent: Thursday, August 19, 2021 2:00 PM
To: Carlos Abalde <carlos.abalde@gmail.com<mailto:carlos.abalde@gmail.com>>
Cc: Justin Lloyd <justinl@arena.net<mailto:justinl@arena.net>>; varnish-misc@varnish-cache.org<mailto:varnish-misc@varnish-cache.org>
Subject: Re: Varnish and AWS ALBs

Hi,

If I read this correctly: https://docs.aws.amazon.com/elasticloadbalancing/latest/application/x-forwarded-headers.html , you can trust the before-last IP, because it was added by the ALB, always. (and using vmod_str makes it easy to retrieve https://github.com/varnish/varnish-modules/blob/master/src/vmod_str.vcc#L42)

Side question: would an NLB work? They support proxy-protocol, that would also solve your problem.

Cheers,

--
Guillaume Quintard


On Thu, Aug 19, 2021 at 1:52 PM Carlos Abalde <carlos.abalde@gmail.com<mailto:carlos.abalde@gmail.com>> wrote:
Hi,

No so sure about that. Let's assume the client address is 1.1.1.1. Two possible scenarios:

- The client request reaches the ALB without XFF. The ALB will inject XFF with value 1.1.1.1. Then Varnish will modify XFF adding the ALB's address (i.e., 1.1.1.1,<ALB IP>). Using the next-to-last IP you're using the right client address.

- The client request reaches the ALB with a forged XFF (e.g. 127.0.0.1). The ALB will will modify XFF (i.e. 127.0.0.1,1.1.1.1). The Varnish will do the same (i.e. 127.0.0.1,1.1.1.1,<ALB IP>). Using the next-to-last IP you're still using the right client address.

I've not checked using a ALB, but that should be the expected behaviour for me.

Best,

--
Carlos Abalde

_______________________________________________
varnish-misc mailing list
varnish-misc@varnish-cache.org<mailto:varnish-misc@varnish-cache.org>
https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc