Mailing List Archive

On Sun, Nov 3, 2019, at 17:27, Sven Oehme wrote:
>
> i installed varnish and tried to set this up, but all examples i found
> are to configure it to speed up access to a certain backend host, i
> don't want that, i try to proxy/cache content to everything
> destination that goes trough varnish. any suggestions on how to set
> this up ?

Maybe try with something like Squid instead of Varnish?

http://www.squid-cache.org/

--
Cosimo
_______________________________________________
varnish-misc mailing list
varnish-misc@varnish-cache.org
https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc

you are saying you can't do this with varnish or you are just suggesting to
see if I can make it work with squid ?

Sven


On Sun, Nov 3, 2019, 9:33 AM Cosimo Streppone <cosimo@streppone.it> wrote:

> On Sun, Nov 3, 2019, at 17:27, Sven Oehme wrote:
> >
> > i installed varnish and tried to set this up, but all examples i found
> > are to configure it to speed up access to a certain backend host, i
> > don't want that, i try to proxy/cache content to everything
> > destination that goes trough varnish. any suggestions on how to set
> > this up ?
>
> Maybe try with something like Squid instead of Varnish?
>
> http://www.squid-cache.org/
>
> --
> Cosimo
> _______________________________________________
> varnish-misc mailing list
> varnish-misc@varnish-cache.org
> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
>

> Am 03.11.2019 um 17:35 schrieb Sven Oehme <oehmes@gmail.com>:
>
> you are saying you can't do this with varnish or you are just suggesting to see if I can make it work with squid ?
>



Varnish is a cache for incoming request to a website (or a couple of websites)

It was never intended as a forward-cache, like Squid. And I doubt it can actually be made to work that way in any even remotely reasonable fashion.

That said, I would really like to know if Squid (which would be the primary tool to try this) does bring any kind of significant improvement these days - at all.

A lot of content is personalized (everything that carries a cookie) and Squid does (hopefully) not store and cache it.
On top of that, Squid, per definition, cannot store content delivered over HTTPS (which is at least 90 and probably closer to 97%) of content these days.
You’d need to setup SSL interception etc.pp.

Browsers are a lot better at caching locally, too, these days, as are websites at instructing browsers to do so.

So, in summary, it’s not the 90s anymore, better get a faster WiFi/internet connection or apply some traffic shaping to nobody can abuse all the bandwidth.




_______________________________________________
varnish-misc mailing list
varnish-misc@varnish-cache.org
https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc

my case is very special. all the nodes download several GB size files
and they are all static, think more about a CDN case. i will take a
look at squid.

thx for the reply. Sven

On Sun, Nov 3, 2019 at 10:05 AM Rainer Duffner <rainer@ultra-secure.de> wrote:
>
>
>
> > Am 03.11.2019 um 17:35 schrieb Sven Oehme <oehmes@gmail.com>:
> >
> > you are saying you can't do this with varnish or you are just suggesting to see if I can make it work with squid ?
> >
>
>
>
> Varnish is a cache for incoming request to a website (or a couple of websites)
>
> It was never intended as a forward-cache, like Squid. And I doubt it can actually be made to work that way in any even remotely reasonable fashion.
>
> That said, I would really like to know if Squid (which would be the primary tool to try this) does bring any kind of significant improvement these days - at all.
>
> A lot of content is personalized (everything that carries a cookie) and Squid does (hopefully) not store and cache it.
> On top of that, Squid, per definition, cannot store content delivered over HTTPS (which is at least 90 and probably closer to 97%) of content these days.
> You’d need to setup SSL interception etc.pp.
>
> Browsers are a lot better at caching locally, too, these days, as are websites at instructing browsers to do so.
>
> So, in summary, it’s not the 90s anymore, better get a faster WiFi/internet connection or apply some traffic shaping to nobody can abuse all the bandwidth.
>
>
>
>
_______________________________________________
varnish-misc mailing list
varnish-misc@varnish-cache.org
https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc

> Am 03.11.2019 um 18:10 schrieb Sven Oehme <oehmes@gmail.com>:
>
> my case is very special. all the nodes download several GB size files
> and they are all static, think more about a CDN case. i will take a
> look at squid.


Then create a local cache and point the nodes there?

Varnish exists because Squid has a number of fundamental problems.
Those problems didn’t go away in the last two decades ;-)

i can't create a cache as the content is non predictable and comes
from various sources. i also can't change the tool thats accessing the
files unfortunate. as i said, a very special case :-)
the only way i see i can solve this is simply cache all accessed data
via http, lets see if squid can do it.

sven

On Sun, Nov 3, 2019 at 10:12 AM Rainer Duffner <rainer@ultra-secure.de> wrote:
>
>
>
> Am 03.11.2019 um 18:10 schrieb Sven Oehme <oehmes@gmail.com>:
>
> my case is very special. all the nodes download several GB size files
> and they are all static, think more about a CDN case. i will take a
> look at squid.
>
>
>
> Then create a local cache and point the nodes there?
>
> Varnish exists because Squid has a number of fundamental problems.
> Those problems didn’t go away in the last two decades ;-)
>
>
>
>
_______________________________________________
varnish-misc mailing list
varnish-misc@varnish-cache.org
https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc

Are we talking about HTTP or HTTPS here?

In the first case, you can use dynamic backends to point at arbitrary hosts.

HTTPS is doable but it's going to be another can of worms because varnish
basically need to be a man-in-the-middle and you'll need special
certificates on all the clients.

On Sun, Nov 3, 2019, 09:18 Sven Oehme <oehmes@gmail.com> wrote:

> i can't create a cache as the content is non predictable and comes
> from various sources. i also can't change the tool thats accessing the
> files unfortunate. as i said, a very special case :-)
> the only way i see i can solve this is simply cache all accessed data
> via http, lets see if squid can do it.
>
> sven
>
> On Sun, Nov 3, 2019 at 10:12 AM Rainer Duffner <rainer@ultra-secure.de>
> wrote:
> >
> >
> >
> > Am 03.11.2019 um 18:10 schrieb Sven Oehme <oehmes@gmail.com>:
> >
> > my case is very special. all the nodes download several GB size files
> > and they are all static, think more about a CDN case. i will take a
> > look at squid.
> >
> >
> >
> > Then create a local cache and point the nodes there?
> >
> > Varnish exists because Squid has a number of fundamental problems.
> > Those problems didn’t go away in the last two decades ;-)
> >
> >
> >
> >
> _______________________________________________
> varnish-misc mailing list
> varnish-misc@varnish-cache.org
> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
>

> So, in summary, it’s not the 90s anymore

I would agree with that, yes :-)

--
Cosimo

--------
In message <CALssuR0=uyz38-Q7OOGfVcHpXkTS4gFjYhFE=TCRK32dpzF8fA@mail.gmail.com>, Sven Oehme writes:

>i can't create a cache as the content is non predictable and comes
>from various sources. i also can't change the tool thats accessing the
>files unfortunate. as i said, a very special case :-)
>the only way i see i can solve this is simply cache all accessed data
>via http, lets see if squid can do it.

Just to chime in: Squid's you tool for that, and it is not quite as horrible
as it once were, but still ... ugh!

That said, I have a squid running here myself, so that all my FreeBSD
machines do not need to drag updates into the house individually.

--
Poul-Henning Kamp | UNIX since Zilog Zeus 3.20
phk@FreeBSD.ORG | TCP/IP since RFC 956
FreeBSD committer | BSD since 4.3-tahoe
Never attribute to malice what can adequately be explained by incompetence.
_______________________________________________
varnish-misc mailing list
varnish-misc@varnish-cache.org
https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc

How do you deal with the client side certs ?

On Sun, Nov 3, 2019, 2:51 PM Poul-Henning Kamp <phk@phk.freebsd.dk> wrote:

> --------
> In message <CALssuR0=uyz38-Q7OOGfVcHpXkTS4gFjYhFE=
> TCRK32dpzF8fA@mail.gmail.com>, Sven Oehme writes:
>
> >i can't create a cache as the content is non predictable and comes
> >from various sources. i also can't change the tool thats accessing the
> >files unfortunate. as i said, a very special case :-)
> >the only way i see i can solve this is simply cache all accessed data
> >via http, lets see if squid can do it.
>
> Just to chime in: Squid's you tool for that, and it is not quite as
> horrible
> as it once were, but still ... ugh!
>
> That said, I have a squid running here myself, so that all my FreeBSD
> machines do not need to drag updates into the house individually.
>
> --
> Poul-Henning Kamp | UNIX since Zilog Zeus 3.20
> phk@FreeBSD.ORG | TCP/IP since RFC 956
> FreeBSD committer | BSD since 4.3-tahoe
> Never attribute to malice what can adequately be explained by incompetence.
>

--------
In message <CALssuR3b=xzOU_ip9DmbFf1PVBq3p9yLVo9SHoFoic=0hK+K7w@mail.gmail.com>, Sven Oehme writes:

>How do you deal with the client side certs ?

I dont.

It's just regular http(s) traffic


--
Poul-Henning Kamp | UNIX since Zilog Zeus 3.20
phk@FreeBSD.ORG | TCP/IP since RFC 956
FreeBSD committer | BSD since 4.3-tahoe
Never attribute to malice what can adequately be explained by incompetence.
_______________________________________________
varnish-misc mailing list
varnish-misc@varnish-cache.org
https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc

could you please share your squid config, so i could try that as a
starting point.

thx. sven

On Sun, Nov 3, 2019 at 3:20 PM Poul-Henning Kamp <phk@phk.freebsd.dk> wrote:
>
> --------
> In message <CALssuR3b=xzOU_ip9DmbFf1PVBq3p9yLVo9SHoFoic=0hK+K7w@mail.gmail.com>, Sven Oehme writes:
>
> >How do you deal with the client side certs ?
>
> I dont.
>
> It's just regular http(s) traffic
>
>
> --
> Poul-Henning Kamp | UNIX since Zilog Zeus 3.20
> phk@FreeBSD.ORG | TCP/IP since RFC 956
> FreeBSD committer | BSD since 4.3-tahoe
> Never attribute to malice what can adequately be explained by incompetence.
_______________________________________________
varnish-misc mailing list
varnish-misc@varnish-cache.org
https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc

--------
In message <CALssuR3sshKftWHC+scCDvx+ukbCQj6JuDgk3mTs-jcNdy1TUQ@mail.gmail.com>, Sven Oehme writes:
>could you please share your squid config, so i could try that as a
>starting point.

Whatever is the default in FreeBSD's package...


--
Poul-Henning Kamp | UNIX since Zilog Zeus 3.20
phk@FreeBSD.ORG | TCP/IP since RFC 956
FreeBSD committer | BSD since 4.3-tahoe
Never attribute to malice what can adequately be explained by incompetence.
_______________________________________________
varnish-misc mailing list
varnish-misc@varnish-cache.org
https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc