Mailing List Archive: disable sendfile in Varnish, please

disable sendfile in Varnish, please

May 11, 2007, 8:12 AM

Post #1 of 9 (338 views)

I've nailed three different operating system kernels as having
sendfile(2) issues today, so I would advice all of you to
disable sendfile to avoid the various problems we've seen.

The easiest way is to specify

-p sendfile_threshold=-1

to varnishd, or by using the CLI:

param.set sendfile_threshold -1

Only those of you with high paging rates would be in risk of seing
any performance changes as result of this.

While it is quite easy to set up a test environment to look for
sendfile trouble, we cannot do it as part of varnish runtime so I
think we will ship with sendfile_threshold at -1 for the forseeable
future.

Once we start to see kernels where sendfile works correctly in all
instances, we can consider how to seletively enable those.

Alternatively, if nobody sees a performance loss from disabling it
now, we might as well just forget about it in toto.

--
Poul-Henning Kamp | UNIX since Zilog Zeus 3.20
phk at FreeBSD.ORG | TCP/IP since RFC 956
FreeBSD committer | BSD since 4.3-tahoe
Never attribute to malice what can adequately be explained by incompetence.

disable sendfile in Varnish, please [ In reply to ]

ltning at anduin

Dec 10, 2007, 7:23 AM

Post #2 of 9 (346 views)

Permalink

Hi,

is this still a problem?

I'm running varnish in front of a midsize site (~2m hits/day), and
have seen a couple of panics like the ones attached. It refers to
sendfile, and fails to do a core dump (I know core dumps in general
are functioning, as I had a couple of NIC-related ones not too long
ago). Thus the attached "screenshot" is all I've got.

FreeBSD anduin.net 6.2-STABLE FreeBSD 6.2-STABLE #1: Sat Apr 28
12:59:34 CEST 2007
Varnish version: varnish-1.1.1_2 (from ports)

/Eirik

On May 11, 2007, at 17:12, Poul-Henning Kamp wrote:

>
> I've nailed three different operating system kernels as having
> sendfile(2) issues today, so I would advice all of you to
> disable sendfile to avoid the various problems we've seen.
>
> The easiest way is to specify
>
> -p sendfile_threshold=-1
>
> to varnishd, or by using the CLI:
>
> param.set sendfile_threshold -1
>
> Only those of you with high paging rates would be in risk of seing
> any performance changes as result of this.
>
> While it is quite easy to set up a test environment to look for
> sendfile trouble, we cannot do it as part of varnish runtime so I
> think we will ship with sendfile_threshold at -1 for the forseeable
> future.
>
> Once we start to see kernels where sendfile works correctly in all
> instances, we can consider how to seletively enable those.
>
> Alternatively, if nobody sees a performance loss from disabling it
> now, we might as well just forget about it in toto.
>
> --
> Poul-Henning Kamp | UNIX since Zilog Zeus 3.20
> phk at FreeBSD.ORG | TCP/IP since RFC 956
> FreeBSD committer | BSD since 4.3-tahoe
> Never attribute to malice what can adequately be explained by
> incompetence.
> _______________________________________________
> varnish-misc mailing list
> varnish-misc at projects.linpro.no
> http://projects.linpro.no/mailman/listinfo/varnish-misc
>

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://projects.linpro.no/pipermail/varnish-misc/attachments/20071210/9adc9eef/attachment.htm
-------------- next part --------------
A non-text attachment was scrubbed...
Name: fbsd_sendfile_panic.png
Type: image/png
Size: 57917 bytes
Desc: not available
Url : http://projects.linpro.no/pipermail/varnish-misc/attachments/20071210/9adc9eef/attachment.png

disable sendfile in Varnish, please [ In reply to ]

phk at phk

Dec 10, 2007, 9:17 AM

Post #3 of 9 (339 views)

Permalink

In message <EEF3518D-EA59-4B21-9CB0-D3F71075E8DA at anduin.net>, =?ISO-8859-1?Q?Ei
rik_=D8verby?= writes:

>Hi,
>
>is this still a problem?

Yes, sendfile is not currently usable because it does not tell
us when it is _really_ done with the data we send, so it can
run afoul of our reuse of the memory for short lived objects.

I have not spent any time on it, as the performance hit from
using writev(2) seems to be trivial if one has plenty of RAM
and because the real fix for the sendfile issue is likely
to mean a redefinition of the sendfile system call.

Poul-Henning

>> I've nailed three different operating system kernels as having
>> sendfile(2) issues today, so I would advice all of you to
>> disable sendfile to avoid the various problems we've seen.
>>
>> The easiest way is to specify
>>
>> -p sendfile_threshold=-1
>>
>> to varnishd, or by using the CLI:
>>
>> param.set sendfile_threshold -1
--
Poul-Henning Kamp | UNIX since Zilog Zeus 3.20
phk at FreeBSD.ORG | TCP/IP since RFC 956
FreeBSD committer | BSD since 4.3-tahoe
Never attribute to malice what can adequately be explained by incompetence.

disable sendfile in Varnish, please [ In reply to ]

ltning at anduin

Dec 10, 2007, 12:00 PM

Post #4 of 9 (340 views)

Permalink

On Dec 10, 2007, at 17:17, Poul-Henning Kamp wrote:

> In message <EEF3518D-EA59-4B21-9CB0-D3F71075E8DA at anduin.net>, =?
> ISO-8859-1?Q?Ei
> rik_=D8verby?= writes:
>
>> Hi,
>>
>> is this still a problem?
>
> Yes, sendfile is not currently usable because it does not tell
> us when it is _really_ done with the data we send, so it can
> run afoul of our reuse of the memory for short lived objects.

Thanks. What's the likelihood that the error I saw today is in fact
caused by this weakness in the sendfile implementation, and that
Varnish has triggered it? In other words; now that I disabled it as
suggested, how surprised should I be to see a similiar error again?

/Eirik

> I have not spent any time on it, as the performance hit from
> using writev(2) seems to be trivial if one has plenty of RAM
> and because the real fix for the sendfile issue is likely
> to mean a redefinition of the sendfile system call.
>
> Poul-Henning
>
>>> I've nailed three different operating system kernels as having
>>> sendfile(2) issues today, so I would advice all of you to
>>> disable sendfile to avoid the various problems we've seen.
>>>
>>> The easiest way is to specify
>>>
>>> -p sendfile_threshold=-1
>>>
>>> to varnishd, or by using the CLI:
>>>
>>> param.set sendfile_threshold -1
> --
> Poul-Henning Kamp | UNIX since Zilog Zeus 3.20
> phk at FreeBSD.ORG | TCP/IP since RFC 956
> FreeBSD committer | BSD since 4.3-tahoe
> Never attribute to malice what can adequately be explained by
> incompetence.
>

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://projects.linpro.no/pipermail/varnish-misc/attachments/20071210/3a472b7b/attachment.htm

disable sendfile in Varnish, please [ In reply to ]

phk at phk

Dec 10, 2007, 12:28 PM

Post #5 of 9 (337 views)

Permalink

In message <F8756F73-58F0-4C7A-9E8A-8BA90EC61661 at anduin.net>, =?ISO-8859-1?Q?Ei
rik_=D8verby?= writes:

>>> is this still a problem?
>>
>> Yes, sendfile is not currently usable because it does not tell
>> us when it is _really_ done with the data we send, so it can
>> run afoul of our reuse of the memory for short lived objects.
>
>Thanks. What's the likelihood that the error I saw today is in fact
>caused by this weakness in the sendfile implementation, and that
>Varnish has triggered it? In other words; now that I disabled it as
>suggested, how surprised should I be to see a similiar error again?

Well, the fact that you have sendfile enabled at all means that
you run a pretty old[1] version of varnish, so there are a number
of other bugs you may hit, but at least the one where sendfile
sends wrong data to the user should be gone.

Poul-Henning

[1] For such a young project obviously :-)

--
Poul-Henning Kamp | UNIX since Zilog Zeus 3.20
phk at FreeBSD.ORG | TCP/IP since RFC 956
FreeBSD committer | BSD since 4.3-tahoe
Never attribute to malice what can adequately be explained by incompetence.

disable sendfile in Varnish, please [ In reply to ]

steinove at vg

Dec 11, 2007, 2:02 AM

Post #6 of 9 (342 views)

Permalink

Poul-Henning Kamp wrote:
> In message <EEF3518D-EA59-4B21-9CB0-D3F71075E8DA at anduin.net>, =?ISO-8859-1?Q?Ei
> rik_=D8verby?= writes:
>
>> Hi,
>>
>> is this still a problem?
>
> Yes, sendfile is not currently usable because it does not tell
> us when it is _really_ done with the data we send, so it can
> run afoul of our reuse of the memory for short lived objects.
>

As i understand it, the problem is in the bsd implementation of
sendfile, and does not occur in the linux implementation.

Or do You have another opinion on the matter?

Stein Ove

*****************************************************************
Denne fotnoten bekrefter at denne e-postmeldingen ble
skannet av MailSweeper og funnet fri for virus.
*****************************************************************
This footnote confirms that this email message has been
swept by MailSweeper for the presence of computer viruses.
*****************************************************************

disable sendfile in Varnish, please [ In reply to ]

phk at phk

Dec 11, 2007, 2:29 AM

Post #7 of 9 (341 views)

Permalink

In message <475E5236.5090205 at vg.no>, Stein Ove Rosseland writes:

>As i understand it, the problem is in the bsd implementation of
>sendfile, and does not occur in the linux implementation.
>
>Or do You have another opinion on the matter?

As far as I'm aware, the linux implementation has the same shortcoming
(in addition to lacking the ability to send a header)

--
Poul-Henning Kamp | UNIX since Zilog Zeus 3.20
phk at FreeBSD.ORG | TCP/IP since RFC 956
FreeBSD committer | BSD since 4.3-tahoe
Never attribute to malice what can adequately be explained by incompetence.

disable sendfile in Varnish, please [ In reply to ]

andre at idium

Dec 11, 2007, 5:27 AM

Post #8 of 9 (339 views)

Permalink

Yes, I was experiencing this problem w/ varnish on linux (2.6.17).

--
Andr? ?ien Langvand <andre at idium.no> - PGP: 0x7B1E3468
Systemadministrator - Idium AS - http://www.idium.no

Poul-Henning Kamp wrote:
> In message <475E5236.5090205 at vg.no>, Stein Ove Rosseland writes:
>
>> As i understand it, the problem is in the bsd implementation of
>> sendfile, and does not occur in the linux implementation.
>>
>> Or do You have another opinion on the matter?
>
> As far as I'm aware, the linux implementation has the same shortcoming
> (in addition to lacking the ability to send a header)
>

disable sendfile in Varnish, please [ In reply to ]

des at linpro

Jan 8, 2008, 8:35 AM

Post #9 of 9 (338 views)

Permalink

Eirik ?verby <ltning at anduin.net> writes:
> is this still a problem?

[a little late, but] yes, even more so now that we discovered that
sendfile() semantics are different from what we expected (the devil is
in the undocumented details) and that even with all known bugs fixed,
sendfile() may result in corrupted data being sent to the client.

This may be possible to fix by changing the way Varnish uses
sendfile(), but that's not currently a priority.

DES
--
Dag-Erling Sm?rgrav
Senior Software Developer
Linpro AS - www.linpro.no