Mailing List Archive

Hi,

The good news - my initial testing on a single 3Ghz Xeon (HT)
processor Varnish box indicates that it tops out at about 18,000 1k
responses a second, and it seems reasonable under overload. This
compares very favourably to many other implementations, and is much
improved from the last release.

The bad news - testing for HTTP conformance with Co-Advisor <http://
coad.measurement-factory.com/> finds a *lot* of violations. However,
I'm just testing stock Varnish with no configuration (as that's not
documented! ;). Can anyone comment on the state of Varnish's HTTP
conformance?

Cheers,

P.S. I can't provide full results, but I believe Measurement Factory
gives favourable terms to Open Source projects; you might want to
talk to them.

--
Mark Nottingham
mnot at yahoo-inc.com

In message <11A48748-CDA5-4185-9ECB-A6248A7351C3 at yahoo-inc.com>, Mark Nottingha
m writes:

>The bad news - testing for HTTP conformance with Co-Advisor <http://
>coad.measurement-factory.com/> finds a *lot* of violations. However,
>I'm just testing stock Varnish with no configuration (as that's not
>documented! ;). Can anyone comment on the state of Varnish's HTTP
>conformance?

Have you tested Varnish as a proxy or as an origin server ?

Varnish is not a proxy in the RFC2616 sense. Varnish
is an origin server. An origin server which gets its contents
using HTTP, but an origin server nontheless.

RFC2616 only defines proxies and caches in a client side context
and Varnish is (firmly!) a server-side facility. If in doubt,
we side with the origin server, not the client.

To the extent that we fail origin server requirements that affect
practical deployment of Varnish, we will fix, but I cannot guarantee
that we will strive for "to-the-letter-and-damn-the-cost" compliance.

Poul-Henning

--
Poul-Henning Kamp | UNIX since Zilog Zeus 3.20
phk at FreeBSD.ORG | TCP/IP since RFC 956
FreeBSD committer | BSD since 4.3-tahoe
Never attribute to malice what can adequately be explained by incompetence.

In message <11A48748-CDA5-4185-9ECB-A6248A7351C3 at yahoo-inc.com>, Mark Nottingha
m writes:

>The bad news - testing for HTTP conformance with Co-Advisor <http://
>coad.measurement-factory.com/> finds a *lot* of violations. However,
>I'm just testing stock Varnish with no configuration (as that's not
>documented! ;). Can anyone comment on the state of Varnish's HTTP
>conformance?

Have you tested Varnish as a proxy or as an origin server ?

Varnish is not a proxy in the RFC2616 sense. Varnish
is an origin server. An origin server which gets its contents
using HTTP, but an origin server nontheless.

RFC2616 only defines proxies and caches in a client side context
and Varnish is (firmly!) a server-side facility. If in doubt,
we side with the origin server, not the client.

To the extent that we fail origin server requirements that affect
practical deployment of Varnish, we will fix, but I cannot guarantee
that we will strive for "to-the-letter-and-damn-the-cost" compliance.

Poul-Henning

--
Poul-Henning Kamp | UNIX since Zilog Zeus 3.20
phk at FreeBSD.ORG | TCP/IP since RFC 956
FreeBSD committer | BSD since 4.3-tahoe
Never attribute to malice what can adequately be explained by incompetence.

On 2006/09/20, at 1:35 PM, Poul-Henning Kamp wrote:

> In message <11A48748-CDA5-4185-9ECB-A6248A7351C3 at yahoo-inc.com>,
> Mark Nottingham writes:
>
>> The bad news - testing for HTTP conformance with Co-Advisor <http://
>> coad.measurement-factory.com/> finds a *lot* of violations. However,
>> I'm just testing stock Varnish with no configuration (as that's not
>> documented! ;). Can anyone comment on the state of Varnish's HTTP
>> conformance?
>
> Have you tested Varnish as a proxy or as an origin server ?

As a gateway.

> Varnish is not a proxy in the RFC2616 sense. Varnish
> is an origin server. An origin server which gets its contents
> using HTTP, but an origin server nontheless.

That's a gateway, a.k.a. surrogate, a.k.a. reverse proxy. While it's
true that many of the requirements for HTTP proxies won't apply to
it, there are many HTTP requirements that do.

For example, Varnish is acting as a cache, and appears to violate
many of the requirements for caches.

> RFC2616 only defines proxies and caches in a client side context
> and Varnish is (firmly!) a server-side facility.

No. From RFC2616:

> cache
> A program's local store of response messages and the subsystem
> that controls its message storage, retrieval, and deletion. A
> cache stores cacheable responses in order to reduce the response
> time and network bandwidth consumption on future, equivalent
> requests. Any client or server may include a cache, though a
> cache
> cannot be used by a server that is acting as a tunnel.


> To the extent that we fail origin server requirements that affect
> practical deployment of Varnish, we will fix, but I cannot guarantee
> that we will strive for "to-the-letter-and-damn-the-cost" compliance.

That's fine by me; perfect conformance is very difficult to obtain,
and in many cases, not too useful when faced with real loads. My
concerns are mostly around interoperability with other devices, and
controlling how Varnish caches things.

Cheers,

--
Mark Nottingham
mnot at yahoo-inc.com

On 2006/09/20, at 1:35 PM, Poul-Henning Kamp wrote:

> In message <11A48748-CDA5-4185-9ECB-A6248A7351C3 at yahoo-inc.com>,
> Mark Nottingham writes:
>
>> The bad news - testing for HTTP conformance with Co-Advisor <http://
>> coad.measurement-factory.com/> finds a *lot* of violations. However,
>> I'm just testing stock Varnish with no configuration (as that's not
>> documented! ;). Can anyone comment on the state of Varnish's HTTP
>> conformance?
>
> Have you tested Varnish as a proxy or as an origin server ?

As a gateway.

> Varnish is not a proxy in the RFC2616 sense. Varnish
> is an origin server. An origin server which gets its contents
> using HTTP, but an origin server nontheless.

That's a gateway, a.k.a. surrogate, a.k.a. reverse proxy. While it's
true that many of the requirements for HTTP proxies won't apply to
it, there are many HTTP requirements that do.

For example, Varnish is acting as a cache, and appears to violate
many of the requirements for caches.

> RFC2616 only defines proxies and caches in a client side context
> and Varnish is (firmly!) a server-side facility.

No. From RFC2616:

> cache
> A program's local store of response messages and the subsystem
> that controls its message storage, retrieval, and deletion. A
> cache stores cacheable responses in order to reduce the response
> time and network bandwidth consumption on future, equivalent
> requests. Any client or server may include a cache, though a
> cache
> cannot be used by a server that is acting as a tunnel.


> To the extent that we fail origin server requirements that affect
> practical deployment of Varnish, we will fix, but I cannot guarantee
> that we will strive for "to-the-letter-and-damn-the-cost" compliance.

That's fine by me; perfect conformance is very difficult to obtain,
and in many cases, not too useful when faced with real loads. My
concerns are mostly around interoperability with other devices, and
controlling how Varnish caches things.

Cheers,

--
Mark Nottingham
mnot at yahoo-inc.com

In message <E077482A-817A-4F54-A91A-C74AA34FD7FC at yahoo-inc.com>, Mark Nottingha
m writes:

>> Varnish is not a proxy in the RFC2616 sense. Varnish
>> is an origin server. An origin server which gets its contents
>> using HTTP, but an origin server nontheless.
>
>That's a gateway, a.k.a. surrogate, a.k.a. reverse proxy.

In RFC2616 context, Varnish is an origin server and a client, but
it is not a proxy or a gateway. Varnish is designed and built to
be able to violate the semantic transparancy requirement for both
of those.

The name Varnish derives from the dictionary definition that says
"To give a deceptively good appearance" (...to a CMS system).

The entire point of Varnish is to make up for shortcomings in CMS
systems and these shortcomings are not limited merely to speed.

Varnish may be tasked with fixing up object lifetimes or anything
else the site administrator is unhappy with.

While varnish speaks "state of the art HTTP" on the client side,
it may have to use "pidgin HTTP" with the backend. RFC2616 does
not even begin to address a situation like that.

For instance section 14.15 says:

The Content-MD5 header field [...] Only origin servers or clients MAY
generate the Content-MD5 header field; proxies and gateways MUST NOT
generate it, [...]

This is a perfect example of what Varnish might be asked to do: if
the CMS system does not or can not produce Content-MD5 and the site
ovner wants these, Varnish should be able to add them.

Another example is that one of the hottest wishlist items for Varnish
version 2 is ESI(-like) facilities for content composition, there
is no way RFC2616 can allow us to do that, while sailing under a
"gateway" flag.

I want to stress, that this non-adherence to standard is not based
on disrespect for RFC2616 on our part.

The RFC2616 authors tacitly assume that a web-server can be fixed
or improved at will. Unfortunately experience has shown that other
parts of CMS systems are far more critical than the webserver part
and therefore many sites which would love to, do not have that
option.

Therefore we are trying fill a need the authors of RFC2616 did not
foresee: The need to put a glossy finish over a rubbish origin
server to make it look good to the clients.

>My concerns are mostly around interoperability with other devices, and
>controlling how Varnish caches things.

All of "policy" or "behaviour" aspects of Varnish are designed to
happen in the VCL programming language.

The default VCL code is intended to Do The Right Thing, so that a
site administrator hit my abnormal traffic at 2:20AM can kick in a
Varnish box, point it at the backend and have it cope with the spike
and revisit the situation when he wakes up next time.

Therefore Varnish will look a lot like a proxy or gateway when you
look at it first time. But hopefully it will be much more than
that to you, once you get to know it better.

All that however, does not change the fact that in the separate
roles of client and origin server, Varnish should comply with
RFC2616 to the extent possible, reasonable and productive.

Poul-Henning

PS: At this point, VCL is nowhere as expressive as we dream about,
but that is merely a matter of time, ideas and code.

--
Poul-Henning Kamp | UNIX since Zilog Zeus 3.20
phk at FreeBSD.ORG | TCP/IP since RFC 956
FreeBSD committer | BSD since 4.3-tahoe
Never attribute to malice what can adequately be explained by incompetence.

In message <E077482A-817A-4F54-A91A-C74AA34FD7FC at yahoo-inc.com>, Mark Nottingha
m writes:

>> Varnish is not a proxy in the RFC2616 sense. Varnish
>> is an origin server. An origin server which gets its contents
>> using HTTP, but an origin server nontheless.
>
>That's a gateway, a.k.a. surrogate, a.k.a. reverse proxy.

In RFC2616 context, Varnish is an origin server and a client, but
it is not a proxy or a gateway. Varnish is designed and built to
be able to violate the semantic transparancy requirement for both
of those.

The name Varnish derives from the dictionary definition that says
"To give a deceptively good appearance" (...to a CMS system).

The entire point of Varnish is to make up for shortcomings in CMS
systems and these shortcomings are not limited merely to speed.

Varnish may be tasked with fixing up object lifetimes or anything
else the site administrator is unhappy with.

While varnish speaks "state of the art HTTP" on the client side,
it may have to use "pidgin HTTP" with the backend. RFC2616 does
not even begin to address a situation like that.

For instance section 14.15 says:

The Content-MD5 header field [...] Only origin servers or clients MAY
generate the Content-MD5 header field; proxies and gateways MUST NOT
generate it, [...]

This is a perfect example of what Varnish might be asked to do: if
the CMS system does not or can not produce Content-MD5 and the site
ovner wants these, Varnish should be able to add them.

Another example is that one of the hottest wishlist items for Varnish
version 2 is ESI(-like) facilities for content composition, there
is no way RFC2616 can allow us to do that, while sailing under a
"gateway" flag.

I want to stress, that this non-adherence to standard is not based
on disrespect for RFC2616 on our part.

The RFC2616 authors tacitly assume that a web-server can be fixed
or improved at will. Unfortunately experience has shown that other
parts of CMS systems are far more critical than the webserver part
and therefore many sites which would love to, do not have that
option.

Therefore we are trying fill a need the authors of RFC2616 did not
foresee: The need to put a glossy finish over a rubbish origin
server to make it look good to the clients.

>My concerns are mostly around interoperability with other devices, and
>controlling how Varnish caches things.

All of "policy" or "behaviour" aspects of Varnish are designed to
happen in the VCL programming language.

The default VCL code is intended to Do The Right Thing, so that a
site administrator hit my abnormal traffic at 2:20AM can kick in a
Varnish box, point it at the backend and have it cope with the spike
and revisit the situation when he wakes up next time.

Therefore Varnish will look a lot like a proxy or gateway when you
look at it first time. But hopefully it will be much more than
that to you, once you get to know it better.

All that however, does not change the fact that in the separate
roles of client and origin server, Varnish should comply with
RFC2616 to the extent possible, reasonable and productive.

Poul-Henning

PS: At this point, VCL is nowhere as expressive as we dream about,
but that is merely a matter of time, ideas and code.

--
Poul-Henning Kamp | UNIX since Zilog Zeus 3.20
phk at FreeBSD.ORG | TCP/IP since RFC 956
FreeBSD committer | BSD since 4.3-tahoe
Never attribute to malice what can adequately be explained by incompetence.

On 2006/09/21, at 3:07 AM, Poul-Henning Kamp wrote:

> In message <E077482A-817A-4F54-A91A-C74AA34FD7FC at yahoo-inc.com>,
> Mark Nottingha
> m writes:
>
>>> Varnish is not a proxy in the RFC2616 sense. Varnish
>>> is an origin server. An origin server which gets its contents
>>> using HTTP, but an origin server nontheless.
>>
>> That's a gateway, a.k.a. surrogate, a.k.a. reverse proxy.
>
> In RFC2616 context, Varnish is an origin server and a client, but
> it is not a proxy or a gateway. Varnish is designed and built to
> be able to violate the semantic transparancy requirement for both
> of those.

Semantic transparency requirements are placed on caches, not gateways
(or proxies, for that matter). "Reverse proxies," "accelerators,"
"surrogates," most "CDNs" and similar systems are all gateways; they
act as the origin server, from the client's point of view. Most of
them have caches that aren't semantically transparent, because
they've been configured (either directly, or with protocol
extensions) to do things beyond "normal" HTTP.

> Therefore Varnish will look a lot like a proxy or gateway when you
> look at it first time. But hopefully it will be much more than
> that to you, once you get to know it better.

I think this is a good expression of my concern. Most products like
Varnish start from a base of being a conformant HTTP cache, and build
extensions and modifications on top of that. If that isn't the
intent, it needs to be said very clearly.

> All that however, does not change the fact that in the separate
> roles of client and origin server, Varnish should comply with
> RFC2616 to the extent possible, reasonable and productive.

That's good to hear. There seem to be a number of HTTP violations
(that aren't specific to proxies) in Varnish; again, I'd encourage
you to talk to the Measurement Factory folks and make their tests (or
something like them) part of your QA suite.

Cheers,

--
Mark Nottingham
mnot at yahoo-inc.com

On 2006/09/21, at 3:07 AM, Poul-Henning Kamp wrote:

> In message <E077482A-817A-4F54-A91A-C74AA34FD7FC at yahoo-inc.com>,
> Mark Nottingha
> m writes:
>
>>> Varnish is not a proxy in the RFC2616 sense. Varnish
>>> is an origin server. An origin server which gets its contents
>>> using HTTP, but an origin server nontheless.
>>
>> That's a gateway, a.k.a. surrogate, a.k.a. reverse proxy.
>
> In RFC2616 context, Varnish is an origin server and a client, but
> it is not a proxy or a gateway. Varnish is designed and built to
> be able to violate the semantic transparancy requirement for both
> of those.

Semantic transparency requirements are placed on caches, not gateways
(or proxies, for that matter). "Reverse proxies," "accelerators,"
"surrogates," most "CDNs" and similar systems are all gateways; they
act as the origin server, from the client's point of view. Most of
them have caches that aren't semantically transparent, because
they've been configured (either directly, or with protocol
extensions) to do things beyond "normal" HTTP.

> Therefore Varnish will look a lot like a proxy or gateway when you
> look at it first time. But hopefully it will be much more than
> that to you, once you get to know it better.

I think this is a good expression of my concern. Most products like
Varnish start from a base of being a conformant HTTP cache, and build
extensions and modifications on top of that. If that isn't the
intent, it needs to be said very clearly.

> All that however, does not change the fact that in the separate
> roles of client and origin server, Varnish should comply with
> RFC2616 to the extent possible, reasonable and productive.

That's good to hear. There seem to be a number of HTTP violations
(that aren't specific to proxies) in Varnish; again, I'd encourage
you to talk to the Measurement Factory folks and make their tests (or
something like them) part of your QA suite.

Cheers,

--
Mark Nottingham
mnot at yahoo-inc.com

In message <0F08FB20-F4EE-4345-88C3-A271B340E5B3 at yahoo-inc.com>, Mark Nottingha
m writes:

>Semantic transparency requirements are placed on caches, not gateways
>(or proxies, for that matter).

This is turning into a semantic fight, but RFC2616 clearly does
presuppose that gateways are transparent, and that is too narrow
a constraint for us.

>> All that however, does not change the fact that in the separate
>> roles of client and origin server, Varnish should comply with
>> RFC2616 to the extent possible, reasonable and productive.
>
>That's good to hear. There seem to be a number of HTTP violations
>(that aren't specific to proxies) in Varnish; again, I'd encourage
>you to talk to the Measurement Factory folks and make their tests (or
>something like them) part of your QA suite.

Yes, we have a lot of work to do in this area, and we need to build
some kind of test-rig so we can test the VCL primitives comprehensively
as well.

--
Poul-Henning Kamp | UNIX since Zilog Zeus 3.20
phk at FreeBSD.ORG | TCP/IP since RFC 956
FreeBSD committer | BSD since 4.3-tahoe
Never attribute to malice what can adequately be explained by incompetence.

In message <0F08FB20-F4EE-4345-88C3-A271B340E5B3 at yahoo-inc.com>, Mark Nottingha
m writes:

>Semantic transparency requirements are placed on caches, not gateways
>(or proxies, for that matter).

This is turning into a semantic fight, but RFC2616 clearly does
presuppose that gateways are transparent, and that is too narrow
a constraint for us.

>> All that however, does not change the fact that in the separate
>> roles of client and origin server, Varnish should comply with
>> RFC2616 to the extent possible, reasonable and productive.
>
>That's good to hear. There seem to be a number of HTTP violations
>(that aren't specific to proxies) in Varnish; again, I'd encourage
>you to talk to the Measurement Factory folks and make their tests (or
>something like them) part of your QA suite.

Yes, we have a lot of work to do in this area, and we need to build
some kind of test-rig so we can test the VCL primitives comprehensively
as well.

--
Poul-Henning Kamp | UNIX since Zilog Zeus 3.20
phk at FreeBSD.ORG | TCP/IP since RFC 956
FreeBSD committer | BSD since 4.3-tahoe
Never attribute to malice what can adequately be explained by incompetence.

> In message <0F08FB20-F4EE-4345-88C3-A271B340E5B3 at yahoo-inc.com>, Mark
> Nottingham writes:

>>> All that however, does not change the fact that in the separate
>>> roles of client and origin server, Varnish should comply with
>>> RFC2616 to the extent possible, reasonable and productive.
>>
>>That's good to hear. There seem to be a number of HTTP violations
>>(that aren't specific to proxies) in Varnish; again, I'd encourage
>>you to talk to the Measurement Factory folks and make their tests (or
>>something like them) part of your QA suite.
>
> Yes, we have a lot of work to do in this area, and we need to build
> some kind of test-rig so we can test the VCL primitives comprehensively
> as well.

I would like to add that when it came to RFC 2616 compliance, the team
discussed it a fair bit, without going into all the gory details possible.
"What??!!" You may ask, "You have to do just exactly that!". Well, here
comes one of the cool parts about Varnish: We discussed implementing RFC
2616 in VCL, as a default VCL config file. While, as you notice, we ended
up with a not so strict interpetation as default, I (with the fear of
saying something wrong, since I did not write the code) am pretty sure
that VCL is so flexible as to be able to do exactly that.

In other words, one can say that while Varnish is a "cache" that be
anything you want it to be :) Think of it as a cache that is _quick_ at
doing a lookup/serv and scale good on hardware, and at the same time has
the flexibility of a "programming language" (VCL). We are fully aware that
VCL itself puts a threshold on our users. Thats regretful, but we worked
hard at getting it as easy as possible (Insane to say right now since the
doc on VCL is hidden somewhere in /dev/null land), but at the same time
very powerful.
When the day is over, we/I feel that the primary users that Varnish tries
to serv, are users that may need to do complex things with HTTP 1.0-1. And
we give them the tool for this. Every website that will benefit alot from
Varnish has the resources to learn and appreciate VCL quickly. At the same
time we give the hobby website hackers a okay default run-and-forget
config, we think...

Thanks for pointing us towards Measurement Factory. Their tool(s) may be
great for finding/making/testing vcl files for their compliance or such.
I'll dig into it.

I will have to stand corrected if what I say here is completly off the
hook, but this is at least part of the dialog and conclusion the team has
ended up with in regards to RFC 2616 and VCL.

Anders Berg

> In message <0F08FB20-F4EE-4345-88C3-A271B340E5B3 at yahoo-inc.com>, Mark
> Nottingham writes:

>>> All that however, does not change the fact that in the separate
>>> roles of client and origin server, Varnish should comply with
>>> RFC2616 to the extent possible, reasonable and productive.
>>
>>That's good to hear. There seem to be a number of HTTP violations
>>(that aren't specific to proxies) in Varnish; again, I'd encourage
>>you to talk to the Measurement Factory folks and make their tests (or
>>something like them) part of your QA suite.
>
> Yes, we have a lot of work to do in this area, and we need to build
> some kind of test-rig so we can test the VCL primitives comprehensively
> as well.

I would like to add that when it came to RFC 2616 compliance, the team
discussed it a fair bit, without going into all the gory details possible.
"What??!!" You may ask, "You have to do just exactly that!". Well, here
comes one of the cool parts about Varnish: We discussed implementing RFC
2616 in VCL, as a default VCL config file. While, as you notice, we ended
up with a not so strict interpetation as default, I (with the fear of
saying something wrong, since I did not write the code) am pretty sure
that VCL is so flexible as to be able to do exactly that.

In other words, one can say that while Varnish is a "cache" that be
anything you want it to be :) Think of it as a cache that is _quick_ at
doing a lookup/serv and scale good on hardware, and at the same time has
the flexibility of a "programming language" (VCL). We are fully aware that
VCL itself puts a threshold on our users. Thats regretful, but we worked
hard at getting it as easy as possible (Insane to say right now since the
doc on VCL is hidden somewhere in /dev/null land), but at the same time
very powerful.
When the day is over, we/I feel that the primary users that Varnish tries
to serv, are users that may need to do complex things with HTTP 1.0-1. And
we give them the tool for this. Every website that will benefit alot from
Varnish has the resources to learn and appreciate VCL quickly. At the same
time we give the hobby website hackers a okay default run-and-forget
config, we think...

Thanks for pointing us towards Measurement Factory. Their tool(s) may be
great for finding/making/testing vcl files for their compliance or such.
I'll dig into it.

I will have to stand corrected if what I say here is completly off the
hook, but this is at least part of the dialog and conclusion the team has
ended up with in regards to RFC 2616 and VCL.

Anders Berg