Mailing List Archive

Varnish version 2 and beyond
Varnish version 1 has been out for some time now, and it is time
to talk about varnish version 2 and the future in general.

I told Anders that I wanted a bit of vacation before I even
contemplated the future. That is why I have been relatively silent
and inactive since the release.

But I have also talked to a number of early Varnish users and given
a couple of talks about Varnish in various settings and the reception
have been very warm and postive.

And then everybody asks me "What happens next ? What about version 2 ?"

So lets me open the discussion about version 2 with a loooong email:


What ?
======

The features I have heard people ask for so far are:

Things we already know how to do:
Bugfixes
Docs
More VCL
Simple Server side includes.
Vary: header processing (requirement for compression etc.)
Compression.
tar storage method for static/emergency contents
The cluster controller

Tasks which can be delegated:
Tcpdump-like specification to varnishlog(1) etc.
RRD statistics
GUI / Dashboard

Things we need to think more about how to do:
Prefetching
backend loadbalancing

Things we need to think a lot about how to do:
Cookie/Auth handling
URL-scrubber functionality (exploits etc)

As a programmer I want to add, high on the list:

Regression test driver
Regression test cases


How ?
=====

The plan Anders talked about for version 2, was to go through a
process setting up a user-group with a governing board, writing
specification documents for Varnish version 2, produce time estimates,
write a budget and pass the hat around and see what happened.

Even though I am not very interested in the administrative side of
the project, I would strongly advice setting up some kind of
"core-team" or governing board, well ahead of the time where it
becomes necessary to have one.


Who ?
=====

Some of the tasks above are probably best left in my hands while
others would be better off delegated. As good first approximation
is that if it is in the Varnishd binary it is probably "my turf"
if it is outside it can delegated.

If you guys get a better offer from somebody else or decide you
can't afford my services, I won't disappear from Varnish in toto.

Chances are I might still land a contract or two with people who
need help with Varnish, but my involvement would be a sparetime
activity and obviously a lot less intense than otherwise.


How Much ?
==========

As you know VG.no sponsored version 1.0 and I don't know if the
amount they spent is really secret or not. I trust Anders will
disclose it if he feels this is necessary for the discussion.

(Currently linpro.no runs the project servers, I don't know under
what arrangement that happens. If they cannot find enough Varnish
work in Norway, we may have to pay them to do so, or migrate it to
sourceforge or whatever.)

For my own part, I feed my kids by working as an independent
contractor and as such I have to send invoices to somebody every
so often. I have plenty of uncommitted hours in 2007 still, and
if people want to pay me to continue on Varnish, I'm game. (I
do have a little bit of available time left in 2006 also).

For reasons of personal development I prefer to not get nailed down
in just one area of computing. I also have a number of faithful
customers which keep coming back with interesting stuff for me to
work on.

So I try to only put half of my time up for any one customer when
it comes to long term contracts. I work better that way too, it
gives me more time to think about the issues in the background.

For the magnitude of work we are talking about here, my listprice
is DKR610/h (USD105/h) which after my 10% discount on BSD licensed
Open Source, becomes DKR550/h (USD94/h) (For danish customers sales
tax would be on top of that).

If you guys prefer to go through the formal excercise and define
tasks, milestones etc. etc, then it's fine for me. The only two
conditions I set is that you guys set up a governing board so I
have a competent and authoritative management to work with on all
the paper and that you tell me where to send my invoices. This
last bit could easily be the most tricky.

But Seeing the domainnames we get email from, I get the impression
that finding funding for future Varnish development may not be as
troublesome as we anticipated when we began, so a much simpler
alternative is also an option:

If you guys are willing to pay me DKR50K/month (USD8600/month) and
agree to a 3 month notice of termination, I will commit to using
half of my time on Varnish in whatever direction we together decide
to take it. In that scenario I'm willing to send invoices directly
to whoever the donor of the month is. That way we don't have to
wait for a formal user organization to get the financial side
hashed out.

The main downside to this way of doing things is that it does
exactly take away alot of the incentive to get a solid user
organization set up.

My advice is that you first decide the organizational aspect(s)
before you start the technical bits and prioritzation of the wishlist.

I think that was what I had to say right now and I think you guys
should talk/email/whisper in smokefilled rooms/fight it out with
EMACS macros or however else you want to settle between yourselves
how you want to progress from here.


Poul-Henning

--
Poul-Henning Kamp | UNIX since Zilog Zeus 3.20
phk at FreeBSD.ORG | TCP/IP since RFC 956
FreeBSD committer | BSD since 4.3-tahoe
Never attribute to malice what can adequately be explained by incompetence.
Varnish version 2 and beyond [ In reply to ]
...
> What ?
> ======
>
> The features I have heard people ask for so far are:
>
> Things we already know how to do:
> Bugfixes
> Docs
> More VCL
> Simple Server side includes.
> Vary: header processing (requirement for compression etc.)
> Compression.

I wrote the Vary support for Apache's mod_disk_cache at one time, and
would be happy to provide feedback on what worked and what didn't. To
me it seems the lack of Vary (and hence enable compression in most
cases) is a major blocker for larger adoption.

...
> As a programmer I want to add, high on the list:
>
> Regression test driver
> Regression test cases

Have you looked at the httpd-test project, used by mod_perl and httpd?

http://svn.apache.org/viewvc/httpd/test/trunk/perl-framework/

Its some crazy Perl stuff, but it makes it quite easy to build new test
cases quickly.

-Paul Querna
Varnish version 2 and beyond [ In reply to ]
On Tue, 31 Oct 2006, Poul-Henning Kamp wrote:

>
>
> Varnish version 1 has been out for some time now, and it is time
> to talk about varnish version 2 and the future in general.
>
> I told Anders that I wanted a bit of vacation before I even
> contemplated the future. That is why I have been relatively silent
> and inactive since the release.
>
> But I have also talked to a number of early Varnish users and given
> a couple of talks about Varnish in various settings and the reception
> have been very warm and postive.
>
> And then everybody asks me "What happens next ? What about version 2 ?"
>
> So lets me open the discussion about version 2 with a loooong email:
>
>
> What ?
> ======
>
> The features I have heard people ask for so far are:
>
> Things we already know how to do:
> Bugfixes
> Docs
> More VCL
> Simple Server side includes.
> Vary: header processing (requirement for compression etc.)
> Compression.
> tar storage method for static/emergency contents
> The cluster controller

I just now started looking at varnish, after watching your eurobsdcon
presentation. I'm looking into varnish as a frontend for remote and local
freebsd/linux mirrors. A small 1gb package cache reachable via l2 speeds
things up quite a bit.

The main thing that seems to be missing (for my purposes) is more control
over the request sent to the backends, Host: and url rewriting in vcl
would be nice. If some more flexibility is added then people who currently
use apache+mod_rewrite should feel right at home.

A trivial switch to use the backend hostname instead of the client
supplied hostname would cover most cases though.


Thanks for the great software.

--
Sten Spans

"There is a crack in everything, that's how the light gets in."
Leonard Cohen - Anthem
Varnish version 2 and beyond [ In reply to ]
Hello,
Regarding varnish 2 or later, have you had requests to support SSL ?
Damien,

Sten Spans writes:
> On Tue, 31 Oct 2006, Poul-Henning Kamp wrote:
>
> >
> >
> > Varnish version 1 has been out for some time now, and it is time
> > to talk about varnish version 2 and the future in general.
> >
> > I told Anders that I wanted a bit of vacation before I even
> > contemplated the future. That is why I have been relatively silent
> > and inactive since the release.
> >
> > But I have also talked to a number of early Varnish users and given
> > a couple of talks about Varnish in various settings and the reception
> > have been very warm and postive.
> >
> > And then everybody asks me "What happens next ? What about version 2 ?"
> >
> > So lets me open the discussion about version 2 with a loooong email:
> >
> >
> > What ?
> > ======
> >
> > The features I have heard people ask for so far are:
> >
> > Things we already know how to do:
> > Bugfixes
> > Docs
> > More VCL
> > Simple Server side includes.
> > Vary: header processing (requirement for compression etc.)
> > Compression.
> > tar storage method for static/emergency contents
> > The cluster controller
>
> I just now started looking at varnish, after watching your eurobsdcon
> presentation. I'm looking into varnish as a frontend for remote and local
> freebsd/linux mirrors. A small 1gb package cache reachable via l2 speeds
> things up quite a bit.
>
> The main thing that seems to be missing (for my purposes) is more control
> over the request sent to the backends, Host: and url rewriting in vcl
> would be nice. If some more flexibility is added then people who currently
> use apache+mod_rewrite should feel right at home.
>
> A trivial switch to use the backend hostname instead of the client
> supplied hostname would cover most cases though.
>
>
> Thanks for the great software.
>
> --
> Sten Spans
>
> "There is a crack in everything, that's how the light gets in."
> Leonard Cohen - Anthem
> _______________________________________________
> varnish-misc mailing list
> varnish-misc at projects.linpro.no
> http://projects.linpro.no/mailman/listinfo/varnish-misc
Varnish version 2 and beyond [ In reply to ]
For me SSL support is high on the list.

Other than that, it gets to be a question of what path you want to go:
More features, like URL rewrites, better virtual hosting, load
balancing, etc, etc, or keeping it simple.

I'd actually prefer the later. I'm used to having a very "layered"
approach to the hosting, with Apache first, then Squid (but now
Varnish) then Pound for load balancing and finally the actual
web-server.

This works fine, but I would like to put Varnish in the front so I can
have just one for multiple sites, which is why I's like SSL support.
That probably also would get rid of Apache completely in some cases,
but not all. And to get rid of the rest you would need to implement
Apaches URL rewrite and load balancing and probably a couple of more
things.

I'd probably prefer Varnish to just do caching, but do it really well. :)

//Lennart
Varnish version 2 and beyond [ In reply to ]
In message <319e029f0611220815r68a85393u7e7d2b5b6121588 at mail.gmail.com>, "Lenna
rt Regebro" writes:
>For me SSL support is high on the list.
>
>Other than that, it gets to be a question of what path you want to go:
>More features, like URL rewrites, better virtual hosting, load
>balancing, etc, etc, or keeping it simple.

The idea behind the VCL language is to be able to have features and
still keep it simple.

Features implemented via VCL will be design be clearly isolated
(and testable!) because they are only activated if the VCL code says so.

>This works fine, but I would like to put Varnish in the front so I can
>have just one for multiple sites, which is why I's like SSL support.

SSL support is noted, but not currently on the short list for
version two.

--
Poul-Henning Kamp | UNIX since Zilog Zeus 3.20
phk at FreeBSD.ORG | TCP/IP since RFC 956
FreeBSD committer | BSD since 4.3-tahoe
Never attribute to malice what can adequately be explained by incompetence.