Mailing List Archive

PaX kills Tor due to overflow
My Tor node gets killed once every day or two with the following message in
dmesg:

PAX: size overflow detected in function tcp_recvmsg net/ipv4/tcp.c:1696
Pid: 28327, comm: tor Not tainted 3.4.4-hardened-r1 #1
Call Trace:
[<ffffffff8112813b>] report_size_overflow+0x34/0x3e
[<ffffffff8154b573>] tcp_recvmsg+0x5fa/0x8ca
[<ffffffff81567bde>] inet_recvmsg+0x69/0x83
[<ffffffff814f6ca3>] sock_aio_read.part.21+0x104/0x11c
[<ffffffff814f6d01>] sock_aio_read+0x46/0x51
[<ffffffff8112072a>] do_sync_read+0xda/0x122
[<ffffffff81036b68>] ? __switch_to+0x1af/0x382
[<ffffffff81092835>] ? finish_task_switch+0xb6/0xc5
[<ffffffff811210ec>] vfs_read+0x140/0x1a5
[<ffffffff8112143b>] sys_read+0x50/0x78
[<ffffffff81610a5c>] system_call_fastpath+0x1c/0x21

Is there a way to figure out where Tor goes wrong and provokes this error? No
core dump is produced, and not even running Tor under GDB helps - execution
ends with:

Program terminated with signal SIGKILL, Killed.
The program no longer exists.

--
Karl-Johan Karlsson
Re: PaX kills Tor due to overflow [ In reply to ]
On Mon, Aug 20, 2012 at 11:16 AM, Karl-Johan Karlsson
<creideiki+gentoo-hardened@ferretporn.se> wrote:
> My Tor node gets killed once every day or two with the following message in
> dmesg:
>
> PAX: size overflow detected in function tcp_recvmsg net/ipv4/tcp.c:1696
> Pid: 28327, comm: tor Not tainted 3.4.4-hardened-r1 #1
> Call Trace:
> [<ffffffff8112813b>] report_size_overflow+0x34/0x3e
> [<ffffffff8154b573>] tcp_recvmsg+0x5fa/0x8ca
> [<ffffffff81567bde>] inet_recvmsg+0x69/0x83
> [<ffffffff814f6ca3>] sock_aio_read.part.21+0x104/0x11c
> [<ffffffff814f6d01>] sock_aio_read+0x46/0x51
> [<ffffffff8112072a>] do_sync_read+0xda/0x122
> [<ffffffff81036b68>] ? __switch_to+0x1af/0x382
> [<ffffffff81092835>] ? finish_task_switch+0xb6/0xc5
> [<ffffffff811210ec>] vfs_read+0x140/0x1a5
> [<ffffffff8112143b>] sys_read+0x50/0x78
> [<ffffffff81610a5c>] system_call_fastpath+0x1c/0x21
>
> Is there a way to figure out where Tor goes wrong and provokes this error? No
> core dump is produced, and not even running Tor under GDB helps - execution
> ends with:
>
> Program terminated with signal SIGKILL, Killed.
> The program no longer exists.
>
> --
> Karl-Johan Karlsson

Hi Karl-Johan,

Are you running Tor as a relay or is is only used as a client? Also,
what version of Tor and what version of the kernel are you running?

- Matt
Re: PaX kills Tor due to overflow [ In reply to ]
On Monday 20 August 2012 11.26.15 Matthew Finkel wrote:
> Are you running Tor as a relay or is is only used as a client?

It's a relay. It's usually a client as well, but nobody is using that
functionality at the moment. It is not an exit node.

> Also,
> what version of Tor and what version of the kernel are you running?

sys-kernel/hardened-sources-3.4.4-r1 configured without SELinux and with:

[*] Grsecurity
Configuration Method (Automatic) --->
Usage Type (Server) --->
Virtualization Type (Host) --->
Virtualization Hardware (EPT/RVI Processor Support) --->
Virtualization Software (KVM) --->
Required Priorities (Security) --->

I'm not sure when these crashes started, but I've seen them at least since
net-misc/tor-0.2.3.19_rc. I'm now on net-misc/tor-0.2.3.20_rc.

--
Karl-Johan Karlsson
Re: PaX kills Tor due to overflow [ In reply to ]
It's because of the size_overflow plugin.
Take a look at this bug:
https://bugs.gentoo.org/show_bug.cgi?id=430906

It is promised to get fixed soon.

Regards:
Dw.
--
dr Tóth Attila, Radiológus, 06-20-825-8057
Attila Toth MD, Radiologist, +36-20-825-8057

2012.Augusztus 20.(H) 17:43 időpontban Karl-Johan Karlsson ezt írta:
> On Monday 20 August 2012 11.26.15 Matthew Finkel wrote:
>> Are you running Tor as a relay or is is only used as a client?
>
> It's a relay. It's usually a client as well, but nobody is using that
> functionality at the moment. It is not an exit node.
>
>> Also,
>> what version of Tor and what version of the kernel are you running?
>
> sys-kernel/hardened-sources-3.4.4-r1 configured without SELinux and with:
>
> [*] Grsecurity
> Configuration Method (Automatic) --->
> Usage Type (Server) --->
> Virtualization Type (Host) --->
> Virtualization Hardware (EPT/RVI Processor Support) --->
> Virtualization Software (KVM) --->
> Required Priorities (Security) --->
>
> I'm not sure when these crashes started, but I've seen them at least since
> net-misc/tor-0.2.3.19_rc. I'm now on net-misc/tor-0.2.3.20_rc.
>
> --
> Karl-Johan Karlsson
Re: PaX kills Tor due to overflow [ In reply to ]
On Mon, 20 Aug 2012 16:16:27 +0100
Karl-Johan Karlsson <creideiki+gentoo-hardened@ferretporn.se> wrote:

> My Tor node gets killed once every day or two with the following message in
> dmesg:
>
> PAX: size overflow detected in function tcp_recvmsg net/ipv4/tcp.c:1696

That's a size_overflow false positive. Try some of the recent grsec
patches, it might be fixed there. Or disable PAX_SIZE_OVERFLOW and
rebuild your kernel.
Re: PaX kills Tor due to overflow [ In reply to ]
On 08/20/2012 01:59 PM, Pavel Labushev wrote:
> On Mon, 20 Aug 2012 16:16:27 +0100
> Karl-Johan Karlsson<creideiki+gentoo-hardened@ferretporn.se> wrote:
>
>> My Tor node gets killed once every day or two with the following message in
>> dmesg:
>>
>> PAX: size overflow detected in function tcp_recvmsg net/ipv4/tcp.c:1696
>
> That's a size_overflow false positive. Try some of the recent grsec
> patches, it might be fixed there. Or disable PAX_SIZE_OVERFLOW and
> rebuild your kernel.

That sounds about right. I'm not hitting this with tor-ramdisk, a tiny
ramdisk image for running tor relays, built with latest tor + busybox +
hardened kernel. I have PAX_SIZE_OVERFLOW off. I didn't even try
turning it on since its still very experimental.


--
Anthony G. Basile, Ph. D.
Chair of Information Technology
D'Youville College
Buffalo, NY 14201
(716) 829-8197
Re: PaX kills Tor due to overflow [ In reply to ]
On Tue, Aug 21, 2012 at 11:44 PM, Anthony G. Basile
<basile@opensource.dyc.edu> wrote:
> That sounds about right. I'm not hitting this with tor-ramdisk, a tiny
> ramdisk image for running tor relays, built with latest tor + busybox +
> hardened kernel. I have PAX_SIZE_OVERFLOW off. I didn't even try turning
> it on since its still very experimental.

Perhaps the Kconfig description should be more explicit about the
status of this feature (looking at one in 3.4.7 sources now)? When
upgrading to 3.4 hardened-source series, I suspected that this feature
was rather experimental, but couldn't confirm it even after some
thorough searching, so I figured out (maybe wrongly) that it was a
mature work that grsecurity / PaX finally decided to include in their
patches.

--
Maxim Kammerer
Liberté Linux: http://dee.su/liberte
Re: PaX kills Tor due to overflow [ In reply to ]
On 22 Aug 2012 at 1:37, Maxim Kammerer wrote:

> On Tue, Aug 21, 2012 at 11:44 PM, Anthony G. Basile
> <basile@opensource.dyc.edu> wrote:
> > That sounds about right. I'm not hitting this with tor-ramdisk, a tiny
> > ramdisk image for running tor relays, built with latest tor + busybox +
> > hardened kernel. I have PAX_SIZE_OVERFLOW off. I didn't even try turning
> > it on since its still very experimental.

it's actually getting better, since last week's version the number of
false positives should be much reduced.

> Perhaps the Kconfig description should be more explicit about the
> status of this feature (looking at one in 3.4.7 sources now)? When
> upgrading to 3.4 hardened-source series, I suspected that this feature
> was rather experimental, but couldn't confirm it even after some
> thorough searching, so I figured out (maybe wrongly) that it was a
> mature work that grsecurity / PaX finally decided to include in their
> patches.

did your thorough search include the grsec mailing list archives? if
you google '"size_overflow" pax' then it's like the first hit there ;).
Re: PaX kills Tor due to overflow [ In reply to ]
On Wed, Aug 22, 2012 at 2:19 AM, PaX Team <pageexec@freemail.hu> wrote:
> did your thorough search include the grsec mailing list archives? if
> you google '"size_overflow" pax' then it's like the first hit there ;).

Yes! :) I did read that post, and what I (probably wrongly) gathered
from it was that the plugin was developed for 3 years, and finalized
by Emese Revfy during the last year. I think I noted the
“experimental” part, but figured that it's just you being on the safe
side, and that booting and watching the system for a few minutes
should be enough. The referenced blog post could dispel the confusion,
but I couldn't find it (and I did search for '"Emese Revfy" blog').

--
Maxim Kammerer
Liberté Linux: http://dee.su/liberte
Re: PaX kills Tor due to overflow [ In reply to ]
On 22 Aug 2012 at 4:59, Maxim Kammerer wrote:

> Yes! :) I did read that post, and what I (probably wrongly) gathered
> from it was that the plugin was developed for 3 years, and finalized
> by Emese Revfy during the last year.

what we did 3 years ago was the macro hack, not much to do with the new
approach with the gcc plugin that she's worked on for a year now ;).

> I think I noted the "experimental" part, but figured that it's just you
> being on the safe side, and that booting and watching the system for a
> few minutes should be enough.

we normally take less time to shake out the initial bugs but this is
a tricky feature (we originally wanted to release it last autumn ;)
and there were always new surprises as more code was exposed to the
instrumentation.

> The referenced blog post could dispel the confusion, but I couldn't
> find it (and I did search for '"Emese Revfy" blog').

she finished it just the other day, now i'll have to translate it and
then we can publish it finally (on the grsec blog).