Mailing List Archive: Unexpected Cache-Control Header Transmission in Dual-Server API Setup

Unexpected Cache-Control Header Transmission in Dual-Server API Setup

Jun 28, 2023, 6:25 AM

Post #1 of 5 (329 views)

Hello All,

Our application operates on a dual-server setup, where each server is
dedicated to running a distinct API.

*Technical specifications:*
Framework: Spring-boot v2.4 (Java 1.8)
Runtime Environment: Tomcat
Version: Apache Tomcat/7.0.42
Server1 runs API-1 and Server2 runs API-2. Both servers are equipped with
an installed Varnish application. When either API is accessed, the request
is processed through the Varnish instance associated with the respective
server.

*Issue Description:*
In a typical scenario, a client (browser) sends a request to API-1, which
is handled by the Varnish instance on Server1. After initial processing,
API-1 makes a subsequent request to API-2 on Server2.

The Request Flow is as follows:
*Browser --> Varnish on Server1 --> Tomcat on Server1 --> Varnish on
Server2 --> Tomcat on Server2*

*Assuming, the request from Browser will be a miss at Server1 Varnish so
that the request reaches Tomcat(Backend) on server1.*

In cases where the browser *does not include any cache-control headers in
the request* (e.g., no-cache, max-age=0), the Server1 Varnish instance
correctly *does not receive any cache-control headers*.

*However, when API-1 calls API-2, we observe that a cache-control: no-cache
and p**ragma: no-cache headers are being transmitted to the Varnish
instance on Server2*, despite the following conditions:

1. We are not explicitly sending any cache-control header in our
application code during the call from API-1 to API-2.
2. Our application does not use the Spring-security dependency, which by
default might add such a header.
3. The cache-control header is not being set by the Varnish instance on
Server2.

This unexpected behavior of receiving a cache-control header at Server2's
Varnish instance when invoking API-2 from API-1 is the crux of our issue.

We kindly request your assistance in understanding the cause of this
unexpected behavior. Additionally, we would greatly appreciate any guidance
on how to effectively prevent this issue from occurring in the future.

Thanks & Regards
Uday Kumar

Re: Unexpected Cache-Control Header Transmission in Dual-Server API Setup [ In reply to ]

guillaume.quintard at gmail

Jun 28, 2023, 8:02 AM

Post #2 of 5 (329 views)

Permalink

Hi Uday,

That one should be quick: Varnish doesn't add cache-control headers on its
own.

So, from what I understand it can come from two places:
- either the VCL in varnish1
- something in tomcat1

It should be very easy to check with varnishlog's. Essentially, run
"varnishlog -H request -q 'ReqHeader:uday'" on both varnish nodes and send
a curl request like "curl http://varnish1/some/request/not/in/cache.html -H
"uday: true"

You should see the request going through both varnish and should be able to
pinpoint what created the header. Or at least identify whether it's a
varnish thing or not.

Kind regards

For a reminder on varnishlog:
https://docs.varnish-software.com/tutorials/vsl-query/

On Wed, Jun 28, 2023, 06:28 Uday Kumar <uday.polu@indiamart.com> wrote:

> Hello All,
>
> Our application operates on a dual-server setup, where each server is
> dedicated to running a distinct API.
>
> *Technical specifications:*
> Framework: Spring-boot v2.4 (Java 1.8)
> Runtime Environment: Tomcat
> Version: Apache Tomcat/7.0.42
> Server1 runs API-1 and Server2 runs API-2. Both servers are equipped with
> an installed Varnish application. When either API is accessed, the request
> is processed through the Varnish instance associated with the respective
> server.
>
> *Issue Description:*
> In a typical scenario, a client (browser) sends a request to API-1, which
> is handled by the Varnish instance on Server1. After initial processing,
> API-1 makes a subsequent request to API-2 on Server2.
>
> The Request Flow is as follows:
> *Browser --> Varnish on Server1 --> Tomcat on Server1 --> Varnish on
> Server2 --> Tomcat on Server2*
>
> *Assuming, the request from Browser will be a miss at Server1 Varnish so
> that the request reaches Tomcat(Backend) on server1.*
>
> In cases where the browser *does not include any cache-control headers in
> the request* (e.g., no-cache, max-age=0), the Server1 Varnish instance
> correctly *does not receive any cache-control headers*.
>
> *However, when API-1 calls API-2, we observe that a cache-control:
> no-cache and p**ragma: no-cache headers are being transmitted to the
> Varnish instance on Server2*, despite the following conditions:
>
> 1. We are not explicitly sending any cache-control header in our
> application code during the call from API-1 to API-2.
> 2. Our application does not use the Spring-security dependency, which by
> default might add such a header.
> 3. The cache-control header is not being set by the Varnish instance on
> Server2.
>
> This unexpected behavior of receiving a cache-control header at Server2's
> Varnish instance when invoking API-2 from API-1 is the crux of our issue.
>
> We kindly request your assistance in understanding the cause of this
> unexpected behavior. Additionally, we would greatly appreciate any guidance
> on how to effectively prevent this issue from occurring in the future.
>
> Thanks & Regards
> Uday Kumar
> _______________________________________________
> varnish-misc mailing list
> varnish-misc@varnish-cache.org
> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
>

Re: Unexpected Cache-Control Header Transmission in Dual-Server API Setup [ In reply to ]

uday.polu at indiamart

Jun 28, 2023, 10:02 AM

Post #3 of 5 (329 views)

Permalink

Hi Guillaume,

You are right!
varnish is not adding any cache-control headers.

*Observations when trying to replicate the issue locally:*
I was trying to replicate the issue using Local Machine by creating a
Spring Boot Application that acts as API-1 and tried hitting API-2 that's
on Server2.

*Request Flow:* Local Machine ----> Server2 varnish --> Server2 Tomcat

Point-1: When using* integrated tomcat (Tomcat 9) the spring-boot* issue
was *not *replicable [*Just ran Application in intellij*] (meaning, the
cache-control header is *not *being transmitted to Varnish of Server2)

*Point-2:* When *Tomcat 9 was explicitly installed in my local machine* and
built the* corresponding war of API-1 and used this to hit API-2* that's on
Server2, *Now issue got replicated* (meaning, *cache-control: no-cache,
pragma: no-cache is being transmitted to Varnish of Server2*)

Any insights?

Thanks & Regards
Uday Kumar

On Wed, Jun 28, 2023 at 8:32?PM Guillaume Quintard <
guillaume.quintard@gmail.com> wrote:

> Hi Uday,
>
> That one should be quick: Varnish doesn't add cache-control headers on its
> own.
>
> So, from what I understand it can come from two places:
> - either the VCL in varnish1
> - something in tomcat1
>
> It should be very easy to check with varnishlog's. Essentially, run
> "varnishlog -H request -q 'ReqHeader:uday'" on both varnish nodes and send
> a curl request like "curl http://varnish1/some/request/not/in/cache.html
> -H "uday: true"
>
> You should see the request going through both varnish and should be able
> to pinpoint what created the header. Or at least identify whether it's a
> varnish thing or not.
>
> Kind regards
>
> For a reminder on varnishlog:
> https://docs.varnish-software.com/tutorials/vsl-query/
>
>
> On Wed, Jun 28, 2023, 06:28 Uday Kumar <uday.polu@indiamart.com> wrote:
>
>> Hello All,
>>
>> Our application operates on a dual-server setup, where each server is
>> dedicated to running a distinct API.
>>
>> *Technical specifications:*
>> Framework: Spring-boot v2.4 (Java 1.8)
>> Runtime Environment: Tomcat
>> Version: Apache Tomcat/7.0.42
>> Server1 runs API-1 and Server2 runs API-2. Both servers are equipped with
>> an installed Varnish application. When either API is accessed, the request
>> is processed through the Varnish instance associated with the respective
>> server.
>>
>> *Issue Description:*
>> In a typical scenario, a client (browser) sends a request to API-1, which
>> is handled by the Varnish instance on Server1. After initial processing,
>> API-1 makes a subsequent request to API-2 on Server2.
>>
>> The Request Flow is as follows:
>> *Browser --> Varnish on Server1 --> Tomcat on Server1 --> Varnish on
>> Server2 --> Tomcat on Server2*
>>
>> *Assuming, the request from Browser will be a miss at Server1 Varnish so
>> that the request reaches Tomcat(Backend) on server1.*
>>
>> In cases where the browser *does not include any cache-control
>> headers in the request* (e.g., no-cache, max-age=0), the Server1 Varnish
>> instance correctly *does not receive any cache-control headers*.
>>
>> *However, when API-1 calls API-2, we observe that a cache-control:
>> no-cache and p**ragma: no-cache headers are being transmitted to the
>> Varnish instance on Server2*, despite the following conditions:
>>
>> 1. We are not explicitly sending any cache-control header in our
>> application code during the call from API-1 to API-2.
>> 2. Our application does not use the Spring-security dependency, which by
>> default might add such a header.
>> 3. The cache-control header is not being set by the Varnish instance on
>> Server2.
>>
>> This unexpected behavior of receiving a cache-control header at Server2's
>> Varnish instance when invoking API-2 from API-1 is the crux of our issue.
>>
>> We kindly request your assistance in understanding the cause of this
>> unexpected behavior. Additionally, we would greatly appreciate any guidance
>> on how to effectively prevent this issue from occurring in the future.
>>
>> Thanks & Regards
>> Uday Kumar
>> _______________________________________________
>> varnish-misc mailing list
>> varnish-misc@varnish-cache.org
>> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
>>
>

Re: Unexpected Cache-Control Header Transmission in Dual-Server API Setup [ In reply to ]

guillaume.quintard at gmail

Jun 28, 2023, 10:06 AM

Post #4 of 5 (329 views)

Permalink

Not really, I have no tomcat expertise, which is where the issue should be
fixed. That being said, if you can't prevent tomcat from adding the header,
then you can use the VCL on varnish2 to scrub the headers ("unset
req.http.cache-control;").

--
Guillaume Quintard

On Wed, Jun 28, 2023 at 10:03?AM Uday Kumar <uday.polu@indiamart.com> wrote:

> Hi Guillaume,
>
> You are right!
> varnish is not adding any cache-control headers.
>
>
> *Observations when trying to replicate the issue locally:*
> I was trying to replicate the issue using Local Machine by creating a
> Spring Boot Application that acts as API-1 and tried hitting API-2 that's
> on Server2.
>
> *Request Flow:* Local Machine ----> Server2 varnish --> Server2 Tomcat
>
> Point-1: When using* integrated tomcat (Tomcat 9) the spring-boot* issue
> was *not *replicable [*Just ran Application in intellij*] (meaning, the
> cache-control header is *not *being transmitted to Varnish of Server2)
>
> *Point-2:* When *Tomcat 9 was explicitly installed in my local machine*
> and built the* corresponding war of API-1 and used this to hit API-2*
> that's on Server2, *Now issue got replicated* (meaning, *cache-control:
> no-cache, pragma: no-cache is being transmitted to Varnish of Server2*)
>
>
> Any insights?
>
> Thanks & Regards
> Uday Kumar
>
>
> On Wed, Jun 28, 2023 at 8:32?PM Guillaume Quintard <
> guillaume.quintard@gmail.com> wrote:
>
>> Hi Uday,
>>
>> That one should be quick: Varnish doesn't add cache-control headers on
>> its own.
>>
>> So, from what I understand it can come from two places:
>> - either the VCL in varnish1
>> - something in tomcat1
>>
>> It should be very easy to check with varnishlog's. Essentially, run
>> "varnishlog -H request -q 'ReqHeader:uday'" on both varnish nodes and send
>> a curl request like "curl http://varnish1/some/request/not/in/cache.html
>> -H "uday: true"
>>
>> You should see the request going through both varnish and should be able
>> to pinpoint what created the header. Or at least identify whether it's a
>> varnish thing or not.
>>
>> Kind regards
>>
>> For a reminder on varnishlog:
>> https://docs.varnish-software.com/tutorials/vsl-query/
>>
>>
>> On Wed, Jun 28, 2023, 06:28 Uday Kumar <uday.polu@indiamart.com> wrote:
>>
>>> Hello All,
>>>
>>> Our application operates on a dual-server setup, where each server is
>>> dedicated to running a distinct API.
>>>
>>> *Technical specifications:*
>>> Framework: Spring-boot v2.4 (Java 1.8)
>>> Runtime Environment: Tomcat
>>> Version: Apache Tomcat/7.0.42
>>> Server1 runs API-1 and Server2 runs API-2. Both servers are equipped
>>> with an installed Varnish application. When either API is accessed, the
>>> request is processed through the Varnish instance associated with the
>>> respective server.
>>>
>>> *Issue Description:*
>>> In a typical scenario, a client (browser) sends a request to API-1,
>>> which is handled by the Varnish instance on Server1. After initial
>>> processing, API-1 makes a subsequent request to API-2 on Server2.
>>>
>>> The Request Flow is as follows:
>>> *Browser --> Varnish on Server1 --> Tomcat on Server1 --> Varnish on
>>> Server2 --> Tomcat on Server2*
>>>
>>> *Assuming, the request from Browser will be a miss at Server1 Varnish so
>>> that the request reaches Tomcat(Backend) on server1.*
>>>
>>> In cases where the browser *does not include any cache-control
>>> headers in the request* (e.g., no-cache, max-age=0), the Server1
>>> Varnish instance correctly *does not receive any cache-control headers*.
>>>
>>> *However, when API-1 calls API-2, we observe that a cache-control:
>>> no-cache and p**ragma: no-cache headers are being transmitted to the
>>> Varnish instance on Server2*, despite the following conditions:
>>>
>>> 1. We are not explicitly sending any cache-control header in our
>>> application code during the call from API-1 to API-2.
>>> 2. Our application does not use the Spring-security dependency, which by
>>> default might add such a header.
>>> 3. The cache-control header is not being set by the Varnish instance on
>>> Server2.
>>>
>>> This unexpected behavior of receiving a cache-control header at
>>> Server2's Varnish instance when invoking API-2 from API-1 is the crux of
>>> our issue.
>>>
>>> We kindly request your assistance in understanding the cause of this
>>> unexpected behavior. Additionally, we would greatly appreciate any guidance
>>> on how to effectively prevent this issue from occurring in the future.
>>>
>>> Thanks & Regards
>>> Uday Kumar
>>> _______________________________________________
>>> varnish-misc mailing list
>>> varnish-misc@varnish-cache.org
>>> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
>>>
>>

Re: Unexpected Cache-Control Header Transmission in Dual-Server API Setup [ In reply to ]

uday.polu at indiamart

Jun 28, 2023, 10:26 AM

Post #5 of 5 (329 views)

Permalink

Okay thank you!

On Wed, Jun 28, 2023, 22:36 Guillaume Quintard <guillaume.quintard@gmail.com>
wrote:

> Not really, I have no tomcat expertise, which is where the issue should be
> fixed. That being said, if you can't prevent tomcat from adding the header,
> then you can use the VCL on varnish2 to scrub the headers ("unset
> req.http.cache-control;").
>
> --
> Guillaume Quintard
>
>
> On Wed, Jun 28, 2023 at 10:03?AM Uday Kumar <uday.polu@indiamart.com>
> wrote:
>
>> Hi Guillaume,
>>
>> You are right!
>> varnish is not adding any cache-control headers.
>>
>>
>> *Observations when trying to replicate the issue locally:*
>> I was trying to replicate the issue using Local Machine by creating a
>> Spring Boot Application that acts as API-1 and tried hitting API-2 that's
>> on Server2.
>>
>> *Request Flow:* Local Machine ----> Server2 varnish --> Server2 Tomcat
>>
>> Point-1: When using* integrated tomcat (Tomcat 9) the spring-boot* issue
>> was *not *replicable [*Just ran Application in intellij*] (meaning, the
>> cache-control header is *not *being transmitted to Varnish of Server2)
>>
>> *Point-2:* When *Tomcat 9 was explicitly installed in my local machine*
>> and built the* corresponding war of API-1 and used this to hit API-2*
>> that's on Server2, *Now issue got replicated* (meaning, *cache-control:
>> no-cache, pragma: no-cache is being transmitted to Varnish of Server2*)
>>
>>
>> Any insights?
>>
>> Thanks & Regards
>> Uday Kumar
>>
>>
>> On Wed, Jun 28, 2023 at 8:32?PM Guillaume Quintard <
>> guillaume.quintard@gmail.com> wrote:
>>
>>> Hi Uday,
>>>
>>> That one should be quick: Varnish doesn't add cache-control headers on
>>> its own.
>>>
>>> So, from what I understand it can come from two places:
>>> - either the VCL in varnish1
>>> - something in tomcat1
>>>
>>> It should be very easy to check with varnishlog's. Essentially, run
>>> "varnishlog -H request -q 'ReqHeader:uday'" on both varnish nodes and send
>>> a curl request like "curl http://varnish1/some/request/not/in/cache.html
>>> -H "uday: true"
>>>
>>> You should see the request going through both varnish and should be able
>>> to pinpoint what created the header. Or at least identify whether it's a
>>> varnish thing or not.
>>>
>>> Kind regards
>>>
>>> For a reminder on varnishlog:
>>> https://docs.varnish-software.com/tutorials/vsl-query/
>>>
>>>
>>> On Wed, Jun 28, 2023, 06:28 Uday Kumar <uday.polu@indiamart.com> wrote:
>>>
>>>> Hello All,
>>>>
>>>> Our application operates on a dual-server setup, where each server is
>>>> dedicated to running a distinct API.
>>>>
>>>> *Technical specifications:*
>>>> Framework: Spring-boot v2.4 (Java 1.8)
>>>> Runtime Environment: Tomcat
>>>> Version: Apache Tomcat/7.0.42
>>>> Server1 runs API-1 and Server2 runs API-2. Both servers are equipped
>>>> with an installed Varnish application. When either API is accessed, the
>>>> request is processed through the Varnish instance associated with the
>>>> respective server.
>>>>
>>>> *Issue Description:*
>>>> In a typical scenario, a client (browser) sends a request to API-1,
>>>> which is handled by the Varnish instance on Server1. After initial
>>>> processing, API-1 makes a subsequent request to API-2 on Server2.
>>>>
>>>> The Request Flow is as follows:
>>>> *Browser --> Varnish on Server1 --> Tomcat on Server1 --> Varnish on
>>>> Server2 --> Tomcat on Server2*
>>>>
>>>> *Assuming, the request from Browser will be a miss at Server1 Varnish
>>>> so that the request reaches Tomcat(Backend) on server1.*
>>>>
>>>> In cases where the browser *does not include any cache-control
>>>> headers in the request* (e.g., no-cache, max-age=0), the Server1
>>>> Varnish instance correctly *does not receive any cache-control headers*
>>>> .
>>>>
>>>> *However, when API-1 calls API-2, we observe that a cache-control:
>>>> no-cache and p**ragma: no-cache headers are being transmitted to the
>>>> Varnish instance on Server2*, despite the following conditions:
>>>>
>>>> 1. We are not explicitly sending any cache-control header in our
>>>> application code during the call from API-1 to API-2.
>>>> 2. Our application does not use the Spring-security dependency, which
>>>> by default might add such a header.
>>>> 3. The cache-control header is not being set by the Varnish instance on
>>>> Server2.
>>>>
>>>> This unexpected behavior of receiving a cache-control header at
>>>> Server2's Varnish instance when invoking API-2 from API-1 is the crux of
>>>> our issue.
>>>>
>>>> We kindly request your assistance in understanding the cause of this
>>>> unexpected behavior. Additionally, we would greatly appreciate any guidance
>>>> on how to effectively prevent this issue from occurring in the future.
>>>>
>>>> Thanks & Regards
>>>> Uday Kumar
>>>> _______________________________________________
>>>> varnish-misc mailing list
>>>> varnish-misc@varnish-cache.org
>>>> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
>>>>
>>>