Mailing List Archive: Block Unauthorized Requests at Varnish [Code Optimization]

Block Unauthorized Requests at Varnish [Code Optimization]

Oct 12, 2023, 10:14 AM

Post #1 of 7 (278 views)

Hello everyone,

We use varnish in our production environment for caching content.

Our Requirement:

We are trying to block unauthorized requests at varnish based on the source
parameter in the URL and the client IP in the request header.

For example:

Sample URL:

www.hostname:port/path?source=mobile&keyword= bags

Let's assume there are 3 IPs [which are allowed to access varnish]
associated with the above request of mobile source.

i.e *IP1, IP2, IP3*

So if any request comes with the source as *mobile *and client-ip as *IP4*,
it's treated as an unauthorized request and should be blocked at varnish.

What we have done for blocking?

*Sample URL:*
www.hostname:port/path?source=mobile&keyword= bags

Created a map using ACL as below:

acl mobile_source{

"IP1";

"IP2";

"IP3";

}

If(req.url ~ "source=mobile" && client.ip !~ mobile_source) {

return(Synth(403, "varnish access denied!"))

}

The problem we are facing:

The source parameter can have different values like mobile, desktop,
laptop, tablet, etc. and each value can have different IPs associated with
it.

ACL Rules will be as below:

acl mobile_source{

"IP1";

"IP2";

"IP3";

}

acl desktop_source{

"IP4";

"IP5";

"IP6";

}

and so on,

If we wanted to block unauthorized access from different source vs IP
combinations, we would have to add that many conditions as below.

If(

(req.url ~ "source=mobile" && client.ip != mobile_source) ||

(req.url ~ "source=desktop" && client.ip != desktop_source) ||

(req.url ~ "source=laptop" && client.ip != laptop_source) ||

(req.url ~ "source=tablet" && client.ip != tablet_source)

){

return(Synth(403, "access denied!"))

}

This becomes worse, if we have 10's or 20's of source values.

Our question:

We would like to know if there is any way to optimize the code by removing
redundant checks so that we can scale it even if we have many sources vs IP
combinations.

Thanks & Regards
Uday Kumar

Re: Block Unauthorized Requests at Varnish [Code Optimization] [ In reply to ]

guillaume.quintard at gmail

Oct 12, 2023, 10:41 AM

Post #2 of 7 (278 views)

Permalink

Hi Uday,

I don't think those are redundant checks, from what you are showing, they
are all justified. Sure, there may be a bunch of them, but you have to go
through to them.

One thing I would do though is to generate the VCL from a source file, like
a YAML one:

mobile:
- IP1
- IP2
- IP3
desktop:
- IP4
- IP5
- IP6

From that, you can build the VCL without having to manually write
"client.ip" or "(req.url ~ "source=" every time.

--
Guillaume Quintard

On Thu, Oct 12, 2023 at 10:17?AM Uday Kumar <uday.polu@indiamart.com> wrote:

> Hello everyone,
>
> We use varnish in our production environment for caching content.
>
> Our Requirement:
>
> We are trying to block unauthorized requests at varnish based on the
> source parameter in the URL and the client IP in the request header.
>
> For example:
>
> Sample URL:
>
> www.hostname:port/path?source=mobile&keyword= bags
>
> Let's assume there are 3 IPs [which are allowed to access varnish]
> associated with the above request of mobile source.
>
> i.e *IP1, IP2, IP3*
>
> So if any request comes with the source as *mobile *and client-ip as *IP4*,
> it's treated as an unauthorized request and should be blocked at varnish.
>
>
> What we have done for blocking?
>
> *Sample URL:*
> www.hostname:port/path?source=mobile&keyword= bags
>
> Created a map using ACL as below:
>
> acl mobile_source{
>
> "IP1";
>
> "IP2";
>
> "IP3";
>
> }
>
> If(req.url ~ "source=mobile" && client.ip !~ mobile_source) {
>
> return(Synth(403, "varnish access denied!"))
>
> }
>
>
> The problem we are facing:
>
> The source parameter can have different values like mobile, desktop,
> laptop, tablet, etc. and each value can have different IPs associated with
> it.
>
> ACL Rules will be as below:
>
> acl mobile_source{
>
> "IP1";
>
> "IP2";
>
> "IP3";
>
> }
>
> acl desktop_source{
>
> "IP4";
>
> "IP5";
>
> "IP6";
>
> }
>
> and so on,
>
>
> If we wanted to block unauthorized access from different source vs IP
> combinations, we would have to add that many conditions as below.
>
> If(
>
> (req.url ~ "source=mobile" && client.ip != mobile_source) ||
>
> (req.url ~ "source=desktop" && client.ip != desktop_source) ||
>
> (req.url ~ "source=laptop" && client.ip != laptop_source) ||
>
> (req.url ~ "source=tablet" && client.ip != tablet_source)
>
> ){
>
> return(Synth(403, "access denied!"))
>
> }
>
> This becomes worse, if we have 10's or 20's of source values.
>
> Our question:
>
> We would like to know if there is any way to optimize the code by
> removing redundant checks so that we can scale it even if we have many
> sources vs IP combinations.
>
>
> Thanks & Regards
> Uday Kumar
> _______________________________________________
> varnish-misc mailing list
> varnish-misc@varnish-cache.org
> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
>

Re: Block Unauthorized Requests at Varnish [Code Optimization] [ In reply to ]

uday.polu at indiamart

Oct 12, 2023, 11:45 AM

Post #3 of 7 (278 views)

Permalink

Hi Guillaume,

I don't think those are redundant checks, from what you are showing, they
are all justified. Sure, there may be a bunch of them, but you have to go
through to them.

By redundant I meant, I have to write multiple checks for each source and
list of IPs associated with it. [.which would be *worse *if the number of
sources are huge]

*Example:*

If(

(req.url ~ "source=mobile" && client.ip != mobile_source) ||

(req.url ~ "source=desktop" && client.ip != desktop_source) ||

(req.url ~ "source=laptop" && client.ip != laptop_source) ||

(req.url ~ "source=tablet" && client.ip != tablet_source)

){

return(Synth(403, "access denied!"))

}

In the above example, if the request URL is source=tablet *[for which
condition is present at the end]*, still I have to check all the above
conditions.

One thing I would do though is to generate the VCL from a source file, like
a YAML one:

Didn't understand, can you please elaborate?

Thanks & Regards
Uday Kumar

On Thu, Oct 12, 2023 at 11:11?PM Guillaume Quintard <
guillaume.quintard@gmail.com> wrote:

> Hi Uday,
>
> I don't think those are redundant checks, from what you are showing, they
> are all justified. Sure, there may be a bunch of them, but you have to go
> through to them.
>
> One thing I would do though is to generate the VCL from a source file,
> like a YAML one:
>
> mobile:
> - IP1
> - IP2
> - IP3
> desktop:
> - IP4
> - IP5
> - IP6
>
>
> From that, you can build the VCL without having to manually write
> "client.ip" or "(req.url ~ "source=" every time.
>
> --
> Guillaume Quintard
>
>
> On Thu, Oct 12, 2023 at 10:17?AM Uday Kumar <uday.polu@indiamart.com>
> wrote:
>
>> Hello everyone,
>>
>> We use varnish in our production environment for caching content.
>>
>> Our Requirement:
>>
>> We are trying to block unauthorized requests at varnish based on the
>> source parameter in the URL and the client IP in the request header.
>>
>> For example:
>>
>> Sample URL:
>>
>> www.hostname:port/path?source=mobile&keyword= bags
>>
>> Let's assume there are 3 IPs [which are allowed to access varnish]
>> associated with the above request of mobile source.
>>
>> i.e *IP1, IP2, IP3*
>>
>> So if any request comes with the source as *mobile *and client-ip as
>> *IP4*, it's treated as an unauthorized request and should be blocked at
>> varnish.
>>
>>
>> What we have done for blocking?
>>
>> *Sample URL:*
>> www.hostname:port/path?source=mobile&keyword= bags
>>
>> Created a map using ACL as below:
>>
>> acl mobile_source{
>>
>> "IP1";
>>
>> "IP2";
>>
>> "IP3";
>>
>> }
>>
>> If(req.url ~ "source=mobile" && client.ip !~ mobile_source) {
>>
>> return(Synth(403, "varnish access denied!"))
>>
>> }
>>
>>
>> The problem we are facing:
>>
>> The source parameter can have different values like mobile, desktop,
>> laptop, tablet, etc. and each value can have different IPs associated with
>> it.
>>
>> ACL Rules will be as below:
>>
>> acl mobile_source{
>>
>> "IP1";
>>
>> "IP2";
>>
>> "IP3";
>>
>> }
>>
>> acl desktop_source{
>>
>> "IP4";
>>
>> "IP5";
>>
>> "IP6";
>>
>> }
>>
>> and so on,
>>
>>
>> If we wanted to block unauthorized access from different source vs IP
>> combinations, we would have to add that many conditions as below.
>>
>> If(
>>
>> (req.url ~ "source=mobile" && client.ip != mobile_source) ||
>>
>> (req.url ~ "source=desktop" && client.ip != desktop_source) ||
>>
>> (req.url ~ "source=laptop" && client.ip != laptop_source) ||
>>
>> (req.url ~ "source=tablet" && client.ip != tablet_source)
>>
>> ){
>>
>> return(Synth(403, "access denied!"))
>>
>> }
>>
>> This becomes worse, if we have 10's or 20's of source values.
>>
>> Our question:
>>
>> We would like to know if there is any way to optimize the code by
>> removing redundant checks so that we can scale it even if we have many
>> sources vs IP combinations.
>>
>>
>> Thanks & Regards
>> Uday Kumar
>> _______________________________________________
>> varnish-misc mailing list
>> varnish-misc@varnish-cache.org
>> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
>>
>

Re: Block Unauthorized Requests at Varnish [Code Optimization] [ In reply to ]

guillaume.quintard at gmail

Oct 12, 2023, 12:05 PM

Post #4 of 7 (278 views)

Permalink

> In the above example, if the request URL is source=tablet [for which
condition is present at the end], still I have to check all the above
conditions.

That's mainly how computers work, processing will be linear. You *could*
create a vmod that packs ACLs into a hashmap to simplify the apparent
logic, but you will pay that price developing the vmod, and for a very
modest performance gain. If you have less than 50 sources, or even less
than a 100, I don't think it's worth agonizing over that kind of
optimization (unless you've actually measured and you did see a
performance drop).

> One thing I would do though is to generate the VCL from a source file,
like a YAML one:

All I'm saying is that you should focus on increasing the maintainability
of the project before worrying about performance. I assume that the VCL is
currently committed in a repo somewhere and gets edited every time you need
to add a new IP or source. If so, it's not great because editing such
repetitive code is error-prone, and therefore you should use templating to
create the VCL from a simpler, more maintainable source.

Tools like go templates or jinja can provide that feature and save you from
repeating yourself when writing configuration.

--
Guillaume Quintard

On Thu, Oct 12, 2023 at 11:46?AM Uday Kumar <uday.polu@indiamart.com> wrote:

> Hi Guillaume,
>
> I don't think those are redundant checks, from what you are showing, they
> are all justified. Sure, there may be a bunch of them, but you have to go
> through to them.
>
> By redundant I meant, I have to write multiple checks for each source and
> list of IPs associated with it. [.which would be *worse *if the number of
> sources are huge]
>
> *Example:*
>
> If(
>
> (req.url ~ "source=mobile" && client.ip != mobile_source) ||
>
> (req.url ~ "source=desktop" && client.ip != desktop_source) ||
>
> (req.url ~ "source=laptop" && client.ip != laptop_source) ||
>
> (req.url ~ "source=tablet" && client.ip != tablet_source)
>
> ){
>
> return(Synth(403, "access denied!"))
>
> }
>
>
> In the above example, if the request URL is source=tablet *[for which
> condition is present at the end]*, still I have to check all the above
> conditions.
>
>
>
>
>
> One thing I would do though is to generate the VCL from a source file,
> like a YAML one:
>
> Didn't understand, can you please elaborate?
>
> Thanks & Regards
> Uday Kumar
>
>
> On Thu, Oct 12, 2023 at 11:11?PM Guillaume Quintard <
> guillaume.quintard@gmail.com> wrote:
>
>> Hi Uday,
>>
>> I don't think those are redundant checks, from what you are showing, they
>> are all justified. Sure, there may be a bunch of them, but you have to go
>> through to them.
>>
>> One thing I would do though is to generate the VCL from a source file,
>> like a YAML one:
>>
>> mobile:
>> - IP1
>> - IP2
>> - IP3
>> desktop:
>> - IP4
>> - IP5
>> - IP6
>>
>>
>> From that, you can build the VCL without having to manually write
>> "client.ip" or "(req.url ~ "source=" every time.
>>
>> --
>> Guillaume Quintard
>>
>>
>> On Thu, Oct 12, 2023 at 10:17?AM Uday Kumar <uday.polu@indiamart.com>
>> wrote:
>>
>>> Hello everyone,
>>>
>>> We use varnish in our production environment for caching content.
>>>
>>> Our Requirement:
>>>
>>> We are trying to block unauthorized requests at varnish based on the
>>> source parameter in the URL and the client IP in the request header.
>>>
>>> For example:
>>>
>>> Sample URL:
>>>
>>> www.hostname:port/path?source=mobile&keyword= bags
>>>
>>> Let's assume there are 3 IPs [which are allowed to access varnish]
>>> associated with the above request of mobile source.
>>>
>>> i.e *IP1, IP2, IP3*
>>>
>>> So if any request comes with the source as *mobile *and client-ip as
>>> *IP4*, it's treated as an unauthorized request and should be blocked at
>>> varnish.
>>>
>>>
>>> What we have done for blocking?
>>>
>>> *Sample URL:*
>>> www.hostname:port/path?source=mobile&keyword= bags
>>>
>>> Created a map using ACL as below:
>>>
>>> acl mobile_source{
>>>
>>> "IP1";
>>>
>>> "IP2";
>>>
>>> "IP3";
>>>
>>> }
>>>
>>> If(req.url ~ "source=mobile" && client.ip !~ mobile_source) {
>>>
>>> return(Synth(403, "varnish access denied!"))
>>>
>>> }
>>>
>>>
>>> The problem we are facing:
>>>
>>> The source parameter can have different values like mobile, desktop,
>>> laptop, tablet, etc. and each value can have different IPs associated with
>>> it.
>>>
>>> ACL Rules will be as below:
>>>
>>> acl mobile_source{
>>>
>>> "IP1";
>>>
>>> "IP2";
>>>
>>> "IP3";
>>>
>>> }
>>>
>>> acl desktop_source{
>>>
>>> "IP4";
>>>
>>> "IP5";
>>>
>>> "IP6";
>>>
>>> }
>>>
>>> and so on,
>>>
>>>
>>> If we wanted to block unauthorized access from different source vs IP
>>> combinations, we would have to add that many conditions as below.
>>>
>>> If(
>>>
>>> (req.url ~ "source=mobile" && client.ip != mobile_source) ||
>>>
>>> (req.url ~ "source=desktop" && client.ip != desktop_source) ||
>>>
>>> (req.url ~ "source=laptop" && client.ip != laptop_source) ||
>>>
>>> (req.url ~ "source=tablet" && client.ip != tablet_source)
>>>
>>> ){
>>>
>>> return(Synth(403, "access denied!"))
>>>
>>> }
>>>
>>> This becomes worse, if we have 10's or 20's of source values.
>>>
>>> Our question:
>>>
>>> We would like to know if there is any way to optimize the code by
>>> removing redundant checks so that we can scale it even if we have many
>>> sources vs IP combinations.
>>>
>>>
>>> Thanks & Regards
>>> Uday Kumar
>>> _______________________________________________
>>> varnish-misc mailing list
>>> varnish-misc@varnish-cache.org
>>> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
>>>
>>

Re: Block Unauthorized Requests at Varnish [Code Optimization] [ In reply to ]

uday.polu at indiamart

Oct 12, 2023, 12:35 PM

Post #5 of 7 (278 views)

Permalink

> That's mainly how computers work, processing will be linear. You *could*
create a vmod that packs ACLs into a hashmap to simplify the apparent
logic, but you will pay that price developing the vmod, and for a very
modest performance gain. If you have less than 50 sources, or even less
than 100, I don't think it's worth agonizing over that kind of optimization
(unless you've actually measured and you did see a performance drop).

Okay, Thanks for your suggestion!

> I assume that the VCL is currently committed in a repo somewhere and
gets edited every time you need to add a new IP or source. If so, it's not
great because editing such repetitive code is error-prone, and therefore
you should use templating to create the VCL from a simpler, more
maintainable source.

Sure, will definitely explore!

Thanks & Regards
Uday Kumar

On Fri, Oct 13, 2023 at 12:35?AM Guillaume Quintard <
guillaume.quintard@gmail.com> wrote:

> > In the above example, if the request URL is source=tablet [for which
> condition is present at the end], still I have to check all the above
> conditions.
>
> That's mainly how computers work, processing will be linear. You *could*
> create a vmod that packs ACLs into a hashmap to simplify the apparent
> logic, but you will pay that price developing the vmod, and for a very
> modest performance gain. If you have less than 50 sources, or even less
> than a 100, I don't think it's worth agonizing over that kind of
> optimization (unless you've actually measured and you did see a
> performance drop).
>
> > One thing I would do though is to generate the VCL from a source file,
> like a YAML one:
>
> All I'm saying is that you should focus on increasing the maintainability
> of the project before worrying about performance. I assume that the VCL is
> currently committed in a repo somewhere and gets edited every time you need
> to add a new IP or source. If so, it's not great because editing such
> repetitive code is error-prone, and therefore you should use templating to
> create the VCL from a simpler, more maintainable source.
>
> Tools like go templates or jinja can provide that feature and save you
> from repeating yourself when writing configuration.
>
> --
> Guillaume Quintard
>
>
> On Thu, Oct 12, 2023 at 11:46?AM Uday Kumar <uday.polu@indiamart.com>
> wrote:
>
>> Hi Guillaume,
>>
>> I don't think those are redundant checks, from what you are showing, they
>> are all justified. Sure, there may be a bunch of them, but you have to go
>> through to them.
>>
>> By redundant I meant, I have to write multiple checks for each source and
>> list of IPs associated with it. [.which would be *worse *if the number of
>> sources are huge]
>>
>> *Example:*
>>
>> If(
>>
>> (req.url ~ "source=mobile" && client.ip != mobile_source) ||
>>
>> (req.url ~ "source=desktop" && client.ip != desktop_source) ||
>>
>> (req.url ~ "source=laptop" && client.ip != laptop_source) ||
>>
>> (req.url ~ "source=tablet" && client.ip != tablet_source)
>>
>> ){
>>
>> return(Synth(403, "access denied!"))
>>
>> }
>>
>>
>> In the above example, if the request URL is source=tablet *[for which
>> condition is present at the end]*, still I have to check all the above
>> conditions.
>>
>>
>>
>>
>>
>> One thing I would do though is to generate the VCL from a source file,
>> like a YAML one:
>>
>> Didn't understand, can you please elaborate?
>>
>> Thanks & Regards
>> Uday Kumar
>>
>>
>> On Thu, Oct 12, 2023 at 11:11?PM Guillaume Quintard <
>> guillaume.quintard@gmail.com> wrote:
>>
>>> Hi Uday,
>>>
>>> I don't think those are redundant checks, from what you are showing,
>>> they are all justified. Sure, there may be a bunch of them, but you have to
>>> go through to them.
>>>
>>> One thing I would do though is to generate the VCL from a source file,
>>> like a YAML one:
>>>
>>> mobile:
>>> - IP1
>>> - IP2
>>> - IP3
>>> desktop:
>>> - IP4
>>> - IP5
>>> - IP6
>>>
>>>
>>> From that, you can build the VCL without having to manually write
>>> "client.ip" or "(req.url ~ "source=" every time.
>>>
>>> --
>>> Guillaume Quintard
>>>
>>>
>>> On Thu, Oct 12, 2023 at 10:17?AM Uday Kumar <uday.polu@indiamart.com>
>>> wrote:
>>>
>>>> Hello everyone,
>>>>
>>>> We use varnish in our production environment for caching content.
>>>>
>>>> Our Requirement:
>>>>
>>>> We are trying to block unauthorized requests at varnish based on the
>>>> source parameter in the URL and the client IP in the request header.
>>>>
>>>> For example:
>>>>
>>>> Sample URL:
>>>>
>>>> www.hostname:port/path?source=mobile&keyword= bags
>>>>
>>>> Let's assume there are 3 IPs [which are allowed to access varnish]
>>>> associated with the above request of mobile source.
>>>>
>>>> i.e *IP1, IP2, IP3*
>>>>
>>>> So if any request comes with the source as *mobile *and client-ip as
>>>> *IP4*, it's treated as an unauthorized request and should be blocked
>>>> at varnish.
>>>>
>>>>
>>>> What we have done for blocking?
>>>>
>>>> *Sample URL:*
>>>> www.hostname:port/path?source=mobile&keyword= bags
>>>>
>>>> Created a map using ACL as below:
>>>>
>>>> acl mobile_source{
>>>>
>>>> "IP1";
>>>>
>>>> "IP2";
>>>>
>>>> "IP3";
>>>>
>>>> }
>>>>
>>>> If(req.url ~ "source=mobile" && client.ip !~ mobile_source) {
>>>>
>>>> return(Synth(403, "varnish access denied!"))
>>>>
>>>> }
>>>>
>>>>
>>>> The problem we are facing:
>>>>
>>>> The source parameter can have different values like mobile, desktop,
>>>> laptop, tablet, etc. and each value can have different IPs associated with
>>>> it.
>>>>
>>>> ACL Rules will be as below:
>>>>
>>>> acl mobile_source{
>>>>
>>>> "IP1";
>>>>
>>>> "IP2";
>>>>
>>>> "IP3";
>>>>
>>>> }
>>>>
>>>> acl desktop_source{
>>>>
>>>> "IP4";
>>>>
>>>> "IP5";
>>>>
>>>> "IP6";
>>>>
>>>> }
>>>>
>>>> and so on,
>>>>
>>>>
>>>> If we wanted to block unauthorized access from different source vs IP
>>>> combinations, we would have to add that many conditions as below.
>>>>
>>>> If(
>>>>
>>>> (req.url ~ "source=mobile" && client.ip != mobile_source) ||
>>>>
>>>> (req.url ~ "source=desktop" && client.ip != desktop_source) ||
>>>>
>>>> (req.url ~ "source=laptop" && client.ip != laptop_source) ||
>>>>
>>>> (req.url ~ "source=tablet" && client.ip != tablet_source)
>>>>
>>>> ){
>>>>
>>>> return(Synth(403, "access denied!"))
>>>>
>>>> }
>>>>
>>>> This becomes worse, if we have 10's or 20's of source values.
>>>>
>>>> Our question:
>>>>
>>>> We would like to know if there is any way to optimize the code by
>>>> removing redundant checks so that we can scale it even if we have many
>>>> sources vs IP combinations.
>>>>
>>>>
>>>> Thanks & Regards
>>>> Uday Kumar
>>>> _______________________________________________
>>>> varnish-misc mailing list
>>>> varnish-misc@varnish-cache.org
>>>> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
>>>>
>>>

Re: Block Unauthorized Requests at Varnish [Code Optimization] [ In reply to ]

guillaume.quintard at gmail

Oct 14, 2023, 7:02 PM

Post #6 of 7 (274 views)

Permalink

Hello Uday,

Quick follow-up as I realize that templating can be a bit scary when
confronted for the first time, and you are far from the first one to be
curious about, so I've committed this:
https://github.com/varnish/toolbox/tree/master/gotemplate-example
It probably won't get you very far, but it should at least get you started,
and help understand how templating can make things a tiny be simpler but
splitting data from business logic, for example to add more IPs/ACLs or
source without edit the VCL manually.

Hope that helps.

--
Guillaume Quintard

On Thu, Oct 12, 2023 at 12:36?PM Uday Kumar <uday.polu@indiamart.com> wrote:

> > That's mainly how computers work, processing will be linear. You *could*
> create a vmod that packs ACLs into a hashmap to simplify the apparent
> logic, but you will pay that price developing the vmod, and for a very
> modest performance gain. If you have less than 50 sources, or even less
> than 100, I don't think it's worth agonizing over that kind of optimization
> (unless you've actually measured and you did see a performance drop).
>
> Okay, Thanks for your suggestion!
>
> > I assume that the VCL is currently committed in a repo somewhere and
> gets edited every time you need to add a new IP or source. If so, it's not
> great because editing such repetitive code is error-prone, and therefore
> you should use templating to create the VCL from a simpler, more
> maintainable source.
>
> Sure, will definitely explore!
>
> Thanks & Regards
> Uday Kumar
>
>
> On Fri, Oct 13, 2023 at 12:35?AM Guillaume Quintard <
> guillaume.quintard@gmail.com> wrote:
>
>> > In the above example, if the request URL is source=tablet [for which
>> condition is present at the end], still I have to check all the above
>> conditions.
>>
>> That's mainly how computers work, processing will be linear. You *could*
>> create a vmod that packs ACLs into a hashmap to simplify the apparent
>> logic, but you will pay that price developing the vmod, and for a very
>> modest performance gain. If you have less than 50 sources, or even less
>> than a 100, I don't think it's worth agonizing over that kind of
>> optimization (unless you've actually measured and you did see a
>> performance drop).
>>
>> > One thing I would do though is to generate the VCL from a source file,
>> like a YAML one:
>>
>> All I'm saying is that you should focus on increasing the maintainability
>> of the project before worrying about performance. I assume that the VCL is
>> currently committed in a repo somewhere and gets edited every time you need
>> to add a new IP or source. If so, it's not great because editing such
>> repetitive code is error-prone, and therefore you should use templating to
>> create the VCL from a simpler, more maintainable source.
>>
>> Tools like go templates or jinja can provide that feature and save you
>> from repeating yourself when writing configuration.
>>
>> --
>> Guillaume Quintard
>>
>>
>> On Thu, Oct 12, 2023 at 11:46?AM Uday Kumar <uday.polu@indiamart.com>
>> wrote:
>>
>>> Hi Guillaume,
>>>
>>> I don't think those are redundant checks, from what you are showing,
>>> they are all justified. Sure, there may be a bunch of them, but you have to
>>> go through to them.
>>>
>>> By redundant I meant, I have to write multiple checks for each source
>>> and list of IPs associated with it. [.which would be *worse *if the
>>> number of sources are huge]
>>>
>>> *Example:*
>>>
>>> If(
>>>
>>> (req.url ~ "source=mobile" && client.ip != mobile_source) ||
>>>
>>> (req.url ~ "source=desktop" && client.ip != desktop_source) ||
>>>
>>> (req.url ~ "source=laptop" && client.ip != laptop_source) ||
>>>
>>> (req.url ~ "source=tablet" && client.ip != tablet_source)
>>>
>>> ){
>>>
>>> return(Synth(403, "access denied!"))
>>>
>>> }
>>>
>>>
>>> In the above example, if the request URL is source=tablet *[for which
>>> condition is present at the end]*, still I have to check all the above
>>> conditions.
>>>
>>>
>>>
>>>
>>>
>>> One thing I would do though is to generate the VCL from a source file,
>>> like a YAML one:
>>>
>>> Didn't understand, can you please elaborate?
>>>
>>> Thanks & Regards
>>> Uday Kumar
>>>
>>>
>>> On Thu, Oct 12, 2023 at 11:11?PM Guillaume Quintard <
>>> guillaume.quintard@gmail.com> wrote:
>>>
>>>> Hi Uday,
>>>>
>>>> I don't think those are redundant checks, from what you are showing,
>>>> they are all justified. Sure, there may be a bunch of them, but you have to
>>>> go through to them.
>>>>
>>>> One thing I would do though is to generate the VCL from a source file,
>>>> like a YAML one:
>>>>
>>>> mobile:
>>>> - IP1
>>>> - IP2
>>>> - IP3
>>>> desktop:
>>>> - IP4
>>>> - IP5
>>>> - IP6
>>>>
>>>>
>>>> From that, you can build the VCL without having to manually write
>>>> "client.ip" or "(req.url ~ "source=" every time.
>>>>
>>>> --
>>>> Guillaume Quintard
>>>>
>>>>
>>>> On Thu, Oct 12, 2023 at 10:17?AM Uday Kumar <uday.polu@indiamart.com>
>>>> wrote:
>>>>
>>>>> Hello everyone,
>>>>>
>>>>> We use varnish in our production environment for caching content.
>>>>>
>>>>> Our Requirement:
>>>>>
>>>>> We are trying to block unauthorized requests at varnish based on the
>>>>> source parameter in the URL and the client IP in the request header.
>>>>>
>>>>> For example:
>>>>>
>>>>> Sample URL:
>>>>>
>>>>> www.hostname:port/path?source=mobile&keyword= bags
>>>>>
>>>>> Let's assume there are 3 IPs [which are allowed to access varnish]
>>>>> associated with the above request of mobile source.
>>>>>
>>>>> i.e *IP1, IP2, IP3*
>>>>>
>>>>> So if any request comes with the source as *mobile *and client-ip as
>>>>> *IP4*, it's treated as an unauthorized request and should be blocked
>>>>> at varnish.
>>>>>
>>>>>
>>>>> What we have done for blocking?
>>>>>
>>>>> *Sample URL:*
>>>>> www.hostname:port/path?source=mobile&keyword= bags
>>>>>
>>>>> Created a map using ACL as below:
>>>>>
>>>>> acl mobile_source{
>>>>>
>>>>> "IP1";
>>>>>
>>>>> "IP2";
>>>>>
>>>>> "IP3";
>>>>>
>>>>> }
>>>>>
>>>>> If(req.url ~ "source=mobile" && client.ip !~ mobile_source) {
>>>>>
>>>>> return(Synth(403, "varnish access denied!"))
>>>>>
>>>>> }
>>>>>
>>>>>
>>>>> The problem we are facing:
>>>>>
>>>>> The source parameter can have different values like mobile, desktop,
>>>>> laptop, tablet, etc. and each value can have different IPs associated with
>>>>> it.
>>>>>
>>>>> ACL Rules will be as below:
>>>>>
>>>>> acl mobile_source{
>>>>>
>>>>> "IP1";
>>>>>
>>>>> "IP2";
>>>>>
>>>>> "IP3";
>>>>>
>>>>> }
>>>>>
>>>>> acl desktop_source{
>>>>>
>>>>> "IP4";
>>>>>
>>>>> "IP5";
>>>>>
>>>>> "IP6";
>>>>>
>>>>> }
>>>>>
>>>>> and so on,
>>>>>
>>>>>
>>>>> If we wanted to block unauthorized access from different source vs IP
>>>>> combinations, we would have to add that many conditions as below.
>>>>>
>>>>> If(
>>>>>
>>>>> (req.url ~ "source=mobile" && client.ip != mobile_source) ||
>>>>>
>>>>> (req.url ~ "source=desktop" && client.ip != desktop_source) ||
>>>>>
>>>>> (req.url ~ "source=laptop" && client.ip != laptop_source) ||
>>>>>
>>>>> (req.url ~ "source=tablet" && client.ip != tablet_source)
>>>>>
>>>>> ){
>>>>>
>>>>> return(Synth(403, "access denied!"))
>>>>>
>>>>> }
>>>>>
>>>>> This becomes worse, if we have 10's or 20's of source values.
>>>>>
>>>>> Our question:
>>>>>
>>>>> We would like to know if there is any way to optimize the code by
>>>>> removing redundant checks so that we can scale it even if we have many
>>>>> sources vs IP combinations.
>>>>>
>>>>>
>>>>> Thanks & Regards
>>>>> Uday Kumar
>>>>> _______________________________________________
>>>>> varnish-misc mailing list
>>>>> varnish-misc@varnish-cache.org
>>>>> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
>>>>>
>>>>

Re: Block Unauthorized Requests at Varnish [Code Optimization] [ In reply to ]

uday.polu at indiamart

Oct 15, 2023, 11:43 PM

Post #7 of 7 (268 views)

Permalink

Hello Guillaume,

Thank you so much!

I'll check it out!

Thanks & Regards
Uday Kumar

On Sun, Oct 15, 2023 at 7:32?AM Guillaume Quintard <
guillaume.quintard@gmail.com> wrote:

> Hello Uday,
>
> Quick follow-up as I realize that templating can be a bit scary when
> confronted for the first time, and you are far from the first one to be
> curious about, so I've committed this:
> https://github.com/varnish/toolbox/tree/master/gotemplate-example
> It probably won't get you very far, but it should at least get you
> started, and help understand how templating can make things a tiny be
> simpler but splitting data from business logic, for example to add more
> IPs/ACLs or source without edit the VCL manually.
>
> Hope that helps.
>
> --
> Guillaume Quintard
>
>
> On Thu, Oct 12, 2023 at 12:36?PM Uday Kumar <uday.polu@indiamart.com>
> wrote:
>
>> > That's mainly how computers work, processing will be linear. You
>> *could* create a vmod that packs ACLs into a hashmap to simplify the
>> apparent logic, but you will pay that price developing the vmod, and for a
>> very modest performance gain. If you have less than 50 sources, or even
>> less than 100, I don't think it's worth agonizing over that kind of
>> optimization (unless you've actually measured and you did see a
>> performance drop).
>>
>> Okay, Thanks for your suggestion!
>>
>> > I assume that the VCL is currently committed in a repo somewhere and
>> gets edited every time you need to add a new IP or source. If so, it's not
>> great because editing such repetitive code is error-prone, and therefore
>> you should use templating to create the VCL from a simpler, more
>> maintainable source.
>>
>> Sure, will definitely explore!
>>
>> Thanks & Regards
>> Uday Kumar
>>
>>
>> On Fri, Oct 13, 2023 at 12:35?AM Guillaume Quintard <
>> guillaume.quintard@gmail.com> wrote:
>>
>>> > In the above example, if the request URL is source=tablet [for which
>>> condition is present at the end], still I have to check all the above
>>> conditions.
>>>
>>> That's mainly how computers work, processing will be linear. You *could*
>>> create a vmod that packs ACLs into a hashmap to simplify the apparent
>>> logic, but you will pay that price developing the vmod, and for a very
>>> modest performance gain. If you have less than 50 sources, or even less
>>> than a 100, I don't think it's worth agonizing over that kind of
>>> optimization (unless you've actually measured and you did see a
>>> performance drop).
>>>
>>> > One thing I would do though is to generate the VCL from a source file,
>>> like a YAML one:
>>>
>>> All I'm saying is that you should focus on increasing the
>>> maintainability of the project before worrying about performance. I assume
>>> that the VCL is currently committed in a repo somewhere and gets edited
>>> every time you need to add a new IP or source. If so, it's not great
>>> because editing such repetitive code is error-prone, and therefore you
>>> should use templating to create the VCL from a simpler, more maintainable
>>> source.
>>>
>>> Tools like go templates or jinja can provide that feature and save you
>>> from repeating yourself when writing configuration.
>>>
>>> --
>>> Guillaume Quintard
>>>
>>>
>>> On Thu, Oct 12, 2023 at 11:46?AM Uday Kumar <uday.polu@indiamart.com>
>>> wrote:
>>>
>>>> Hi Guillaume,
>>>>
>>>> I don't think those are redundant checks, from what you are showing,
>>>> they are all justified. Sure, there may be a bunch of them, but you have to
>>>> go through to them.
>>>>
>>>> By redundant I meant, I have to write multiple checks for each source
>>>> and list of IPs associated with it. [.which would be *worse *if the
>>>> number of sources are huge]
>>>>
>>>> *Example:*
>>>>
>>>> If(
>>>>
>>>> (req.url ~ "source=mobile" && client.ip != mobile_source) ||
>>>>
>>>> (req.url ~ "source=desktop" && client.ip != desktop_source) ||
>>>>
>>>> (req.url ~ "source=laptop" && client.ip != laptop_source) ||
>>>>
>>>> (req.url ~ "source=tablet" && client.ip != tablet_source)
>>>>
>>>> ){
>>>>
>>>> return(Synth(403, "access denied!"))
>>>>
>>>> }
>>>>
>>>>
>>>> In the above example, if the request URL is source=tablet *[for which
>>>> condition is present at the end]*, still I have to check all the above
>>>> conditions.
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> One thing I would do though is to generate the VCL from a source file,
>>>> like a YAML one:
>>>>
>>>> Didn't understand, can you please elaborate?
>>>>
>>>> Thanks & Regards
>>>> Uday Kumar
>>>>
>>>>
>>>> On Thu, Oct 12, 2023 at 11:11?PM Guillaume Quintard <
>>>> guillaume.quintard@gmail.com> wrote:
>>>>
>>>>> Hi Uday,
>>>>>
>>>>> I don't think those are redundant checks, from what you are showing,
>>>>> they are all justified. Sure, there may be a bunch of them, but you have to
>>>>> go through to them.
>>>>>
>>>>> One thing I would do though is to generate the VCL from a source file,
>>>>> like a YAML one:
>>>>>
>>>>> mobile:
>>>>> - IP1
>>>>> - IP2
>>>>> - IP3
>>>>> desktop:
>>>>> - IP4
>>>>> - IP5
>>>>> - IP6
>>>>>
>>>>>
>>>>> From that, you can build the VCL without having to manually write
>>>>> "client.ip" or "(req.url ~ "source=" every time.
>>>>>
>>>>> --
>>>>> Guillaume Quintard
>>>>>
>>>>>
>>>>> On Thu, Oct 12, 2023 at 10:17?AM Uday Kumar <uday.polu@indiamart.com>
>>>>> wrote:
>>>>>
>>>>>> Hello everyone,
>>>>>>
>>>>>> We use varnish in our production environment for caching content.
>>>>>>
>>>>>> Our Requirement:
>>>>>>
>>>>>> We are trying to block unauthorized requests at varnish based on the
>>>>>> source parameter in the URL and the client IP in the request header.
>>>>>>
>>>>>> For example:
>>>>>>
>>>>>> Sample URL:
>>>>>>
>>>>>> www.hostname:port/path?source=mobile&keyword= bags
>>>>>>
>>>>>> Let's assume there are 3 IPs [which are allowed to access varnish]
>>>>>> associated with the above request of mobile source.
>>>>>>
>>>>>> i.e *IP1, IP2, IP3*
>>>>>>
>>>>>> So if any request comes with the source as *mobile *and client-ip as
>>>>>> *IP4*, it's treated as an unauthorized request and should be blocked
>>>>>> at varnish.
>>>>>>
>>>>>>
>>>>>> What we have done for blocking?
>>>>>>
>>>>>> *Sample URL:*
>>>>>> www.hostname:port/path?source=mobile&keyword= bags
>>>>>>
>>>>>> Created a map using ACL as below:
>>>>>>
>>>>>> acl mobile_source{
>>>>>>
>>>>>> "IP1";
>>>>>>
>>>>>> "IP2";
>>>>>>
>>>>>> "IP3";
>>>>>>
>>>>>> }
>>>>>>
>>>>>> If(req.url ~ "source=mobile" && client.ip !~ mobile_source) {
>>>>>>
>>>>>> return(Synth(403, "varnish access denied!"))
>>>>>>
>>>>>> }
>>>>>>
>>>>>>
>>>>>> The problem we are facing:
>>>>>>
>>>>>> The source parameter can have different values like mobile, desktop,
>>>>>> laptop, tablet, etc. and each value can have different IPs associated with
>>>>>> it.
>>>>>>
>>>>>> ACL Rules will be as below:
>>>>>>
>>>>>> acl mobile_source{
>>>>>>
>>>>>> "IP1";
>>>>>>
>>>>>> "IP2";
>>>>>>
>>>>>> "IP3";
>>>>>>
>>>>>> }
>>>>>>
>>>>>> acl desktop_source{
>>>>>>
>>>>>> "IP4";
>>>>>>
>>>>>> "IP5";
>>>>>>
>>>>>> "IP6";
>>>>>>
>>>>>> }
>>>>>>
>>>>>> and so on,
>>>>>>
>>>>>>
>>>>>> If we wanted to block unauthorized access from different source vs IP
>>>>>> combinations, we would have to add that many conditions as below.
>>>>>>
>>>>>> If(
>>>>>>
>>>>>> (req.url ~ "source=mobile" && client.ip != mobile_source) ||
>>>>>>
>>>>>> (req.url ~ "source=desktop" && client.ip != desktop_source) ||
>>>>>>
>>>>>> (req.url ~ "source=laptop" && client.ip != laptop_source) ||
>>>>>>
>>>>>> (req.url ~ "source=tablet" && client.ip != tablet_source)
>>>>>>
>>>>>> ){
>>>>>>
>>>>>> return(Synth(403, "access denied!"))
>>>>>>
>>>>>> }
>>>>>>
>>>>>> This becomes worse, if we have 10's or 20's of source values.
>>>>>>
>>>>>> Our question:
>>>>>>
>>>>>> We would like to know if there is any way to optimize the code by
>>>>>> removing redundant checks so that we can scale it even if we have many
>>>>>> sources vs IP combinations.
>>>>>>
>>>>>>
>>>>>> Thanks & Regards
>>>>>> Uday Kumar
>>>>>> _______________________________________________
>>>>>> varnish-misc mailing list
>>>>>> varnish-misc@varnish-cache.org
>>>>>> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
>>>>>>
>>>>>