Mailing List Archive

Crashes after 3.7.0-hardened upgrade
I recently updated all of our servers to 3.7.0-hardened (from
3.4.2-hardened-r1) and re-did our iptables rules to avoid future pain[1]
from the state -> conntrack switch.

The first thing I noticed was that vsftpd apparently crashed on my own
box, michael.orlitzky.com. The server stayed up, though, until I did
something stupid and tried to kill the crashed process. Then it
panicked. I drove to work, rebooted, and disabled vsftpd. Naturally that
hasn't happened again.

Last night, our VPN firewall went down; panicked, around 11:30pm. Drove
to work today and rebooted it, but I'm not sure what the underlying
cause was -- I didn't get a shot of the panic message. The only thing it
does is OpenVPN on two e1000s.

I've been looking through the dmesg of our other servers, just to see if
anything looks out of the ordinary. There's one other machine still
running vsftpd that has a non-fatal (i.e. stuff is still running) crash.
There are more errors above this if needed, although I'm going to have
to reboot it now.

On the VPN box, I'll probably bump to 3.7.1-r2 and just pray unless
someone has a better suggestion.


grsec: From 61.160.222.83: Invalid alignment/Bus error occurred at
000000608f728691 in
/var/log/apache2/abogadosdeaccidentedeautoenmarylandblog.com/www/error/error-2013-01-06.log[vsftpd:7764]
uid/euid:0/0 gid/egid:0/0, parent
/var/log/apache2/abogadosdeaccidentedeautoenmarylandblog.com/www/error/error-2013-01-06.log[vsftpd:2583]
uid/euid:0/0 gid/egid:0/0
grsec: From 61.160.222.83: bruteforce prevention initiated for the next
30 minutes or until service restarted, stalling each fork 30 seconds.
Please investigate the crash report for
/var/log/apache2/abogadosdeaccidentedeautoenmarylandblog.com/www/error/error-2013-01-06.log[vsftpd:7764]
uid/euid:0/0 gid/egid:0/0, parent
/var/log/apache2/abogadosdeaccidentedeautoenmarylandblog.com/www/error/error-2013-01-06.log[vsftpd:2583]
uid/euid:0/0 gid/egid:0/0
grsec: From 61.160.222.83: denied resource overstep by requesting 4096
for RLIMIT_CORE against limit 0 for
/var/log/apache2/abogadosdeaccidentedeautoenmarylandblog.com/www/error/error-2013-01-06.log[vsftpd:7764]
uid/euid:0/0 gid/egid:0/0, parent
/var/log/apache2/abogadosdeaccidentedeautoenmarylandblog.com/www/error/error-2013-01-06.log[vsftpd:2583]
uid/euid:0/0 gid/egid:0/0
PAX: please report this to pageexec@freemail.hu
BUG: unable to handle kernel NULL pointer dereference at 0000000000000030
IP: [<ffffffff81029972>] dup_mm+0x261/0x4c0
PGD 18c661000
Thread overran stack, or stack corrupted
Oops: 0000 [#1] SMP
Modules linked in: xt_tcpudp xt_multiport nf_conntrack_ipv4
nf_defrag_ipv4 xt_conntrack nf_conntrack iptable_filter ip_tables
x_tables cpufreq_ondemand uhci_hcd ehci_hcd thermal usbcore acpi_cpufreq
tg3 microcode freq_table mperf usb_common processor libphy thermal_sys
hwmon unix
CPU 0
Pid: 2583, comm: vsftpd Not tainted 3.7.0-hardened #1 HP ProLiant DL380 G4
RIP: 0010:[<ffffffff81029972>] [<ffffffff81029972>] dup_mm+0x261/0x4c0
RSP: 0018:ffff880187a4ddc0 EFLAGS: 00010286
RAX: 0000000000000000 RBX: ffff880193c4c508 RCX: 0000000000000000
RDX: ffff88018c4df500 RSI: ffff880193c4c508 RDI: ffff880154c32cf0
RBP: ffff8801748fa3c0 R08: ffff88019bc112b0 R09: ffffffff810298cd
R10: 8000000000000000 R11: ffff88018c4c9e00 R12: ffff88018bfc30c0
R13: ffff880154c32cf0 R14: ffff8801748fa420 R15: ffff88018bfc3120
FS: 000002ef1e350700(0000) GS:ffff88019bc00000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 0000000000000030 CR3: 0000000001329000 CR4: 00000000000007b0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Process vsftpd (pid: 2583, threadinfo ffff8801907e3ca8, task
ffff8801907e38d0)
Stack:
0000000000000000 0000000000000000 0000000000000000 ffff8801748fa3c0
0000000000000000 ffff8801748fa3c8 ffff880194c52540 0000000001200011
ffff880174920000 0000000000000000 000002ef1e3509d0 0000000000000000
Call Trace:
[<ffffffff8102a42e>] ? copy_process+0x829/0x119e
[<ffffffff8102ae24>] ? do_fork+0x5c/0x2c2
[<ffffffff8131f873>] ? stub_clone+0x13/0x20
[<ffffffff8131f608>] ? system_call_fastpath+0x18/0x1d
Code: 00 00 00 00 49 c7 45 18 00 00 00 00 49 c7 85 b0 00 00 00 00 00 00
00 49 8b 95 98 00 00 00 48 85 d2 0f 84 85 00 00 00 48 8b 42 18 <48> 8b
48 30 48 8b 82 c8 00 00 00 f0 48 ff 42 30 71 07 f0 48 ff
RIP [<ffffffff81029972>] dup_mm+0x261/0x4c0
RSP <ffff880187a4ddc0>
CR2: 0000000000000030
---[ end trace 969655b532a2156e ]---




[1] https://bugs.gentoo.org/show_bug.cgi?id=448906
Re: Crashes after 3.7.0-hardened upgrade [ In reply to ]
Its e1000. This was an unknown issue until just recently. Is supposed
to be fixed in the latest 3.7.1-r2. Let me know if it is and I'll drop
3.7.0 in favor of 3.7.1-r2.

My appologies. I do test, but its impossible to test on every possible
hardware config.

--Tony

On 01/12/2013 05:22 PM, Michael Orlitzky wrote:
> I recently updated all of our servers to 3.7.0-hardened (from
> 3.4.2-hardened-r1) and re-did our iptables rules to avoid future pain[1]
> from the state -> conntrack switch.
>
> The first thing I noticed was that vsftpd apparently crashed on my own
> box, michael.orlitzky.com. The server stayed up, though, until I did
> something stupid and tried to kill the crashed process. Then it
> panicked. I drove to work, rebooted, and disabled vsftpd. Naturally that
> hasn't happened again.
>
> Last night, our VPN firewall went down; panicked, around 11:30pm. Drove
> to work today and rebooted it, but I'm not sure what the underlying
> cause was -- I didn't get a shot of the panic message. The only thing it
> does is OpenVPN on two e1000s.
>
> I've been looking through the dmesg of our other servers, just to see if
> anything looks out of the ordinary. There's one other machine still
> running vsftpd that has a non-fatal (i.e. stuff is still running) crash.
> There are more errors above this if needed, although I'm going to have
> to reboot it now.
>
> On the VPN box, I'll probably bump to 3.7.1-r2 and just pray unless
> someone has a better suggestion.
>
>
> grsec: From 61.160.222.83: Invalid alignment/Bus error occurred at
> 000000608f728691 in
> /var/log/apache2/abogadosdeaccidentedeautoenmarylandblog.com/www/error/error-2013-01-06.log[vsftpd:7764]
> uid/euid:0/0 gid/egid:0/0, parent
> /var/log/apache2/abogadosdeaccidentedeautoenmarylandblog.com/www/error/error-2013-01-06.log[vsftpd:2583]
> uid/euid:0/0 gid/egid:0/0
> grsec: From 61.160.222.83: bruteforce prevention initiated for the next
> 30 minutes or until service restarted, stalling each fork 30 seconds.
> Please investigate the crash report for
> /var/log/apache2/abogadosdeaccidentedeautoenmarylandblog.com/www/error/error-2013-01-06.log[vsftpd:7764]
> uid/euid:0/0 gid/egid:0/0, parent
> /var/log/apache2/abogadosdeaccidentedeautoenmarylandblog.com/www/error/error-2013-01-06.log[vsftpd:2583]
> uid/euid:0/0 gid/egid:0/0
> grsec: From 61.160.222.83: denied resource overstep by requesting 4096
> for RLIMIT_CORE against limit 0 for
> /var/log/apache2/abogadosdeaccidentedeautoenmarylandblog.com/www/error/error-2013-01-06.log[vsftpd:7764]
> uid/euid:0/0 gid/egid:0/0, parent
> /var/log/apache2/abogadosdeaccidentedeautoenmarylandblog.com/www/error/error-2013-01-06.log[vsftpd:2583]
> uid/euid:0/0 gid/egid:0/0
> PAX: please report this to pageexec@freemail.hu
> BUG: unable to handle kernel NULL pointer dereference at 0000000000000030
> IP: [<ffffffff81029972>] dup_mm+0x261/0x4c0
> PGD 18c661000
> Thread overran stack, or stack corrupted
> Oops: 0000 [#1] SMP
> Modules linked in: xt_tcpudp xt_multiport nf_conntrack_ipv4
> nf_defrag_ipv4 xt_conntrack nf_conntrack iptable_filter ip_tables
> x_tables cpufreq_ondemand uhci_hcd ehci_hcd thermal usbcore acpi_cpufreq
> tg3 microcode freq_table mperf usb_common processor libphy thermal_sys
> hwmon unix
> CPU 0
> Pid: 2583, comm: vsftpd Not tainted 3.7.0-hardened #1 HP ProLiant DL380 G4
> RIP: 0010:[<ffffffff81029972>] [<ffffffff81029972>] dup_mm+0x261/0x4c0
> RSP: 0018:ffff880187a4ddc0 EFLAGS: 00010286
> RAX: 0000000000000000 RBX: ffff880193c4c508 RCX: 0000000000000000
> RDX: ffff88018c4df500 RSI: ffff880193c4c508 RDI: ffff880154c32cf0
> RBP: ffff8801748fa3c0 R08: ffff88019bc112b0 R09: ffffffff810298cd
> R10: 8000000000000000 R11: ffff88018c4c9e00 R12: ffff88018bfc30c0
> R13: ffff880154c32cf0 R14: ffff8801748fa420 R15: ffff88018bfc3120
> FS: 000002ef1e350700(0000) GS:ffff88019bc00000(0000) knlGS:0000000000000000
> CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
> CR2: 0000000000000030 CR3: 0000000001329000 CR4: 00000000000007b0
> DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
> DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
> Process vsftpd (pid: 2583, threadinfo ffff8801907e3ca8, task
> ffff8801907e38d0)
> Stack:
> 0000000000000000 0000000000000000 0000000000000000 ffff8801748fa3c0
> 0000000000000000 ffff8801748fa3c8 ffff880194c52540 0000000001200011
> ffff880174920000 0000000000000000 000002ef1e3509d0 0000000000000000
> Call Trace:
> [<ffffffff8102a42e>] ? copy_process+0x829/0x119e
> [<ffffffff8102ae24>] ? do_fork+0x5c/0x2c2
> [<ffffffff8131f873>] ? stub_clone+0x13/0x20
> [<ffffffff8131f608>] ? system_call_fastpath+0x18/0x1d
> Code: 00 00 00 00 49 c7 45 18 00 00 00 00 49 c7 85 b0 00 00 00 00 00 00
> 00 49 8b 95 98 00 00 00 48 85 d2 0f 84 85 00 00 00 48 8b 42 18<48> 8b
> 48 30 48 8b 82 c8 00 00 00 f0 48 ff 42 30 71 07 f0 48 ff
> RIP [<ffffffff81029972>] dup_mm+0x261/0x4c0
> RSP<ffff880187a4ddc0>
> CR2: 0000000000000030
> ---[ end trace 969655b532a2156e ]---
>
>
>
>
> [1] https://bugs.gentoo.org/show_bug.cgi?id=448906


--
Anthony G. Basile, Ph. D.
Chair of Information Technology
D'Youville College
Buffalo, NY 14201
(716) 829-8197
Re: Crashes after 3.7.0-hardened upgrade [ In reply to ]
Regarding the panic also see:
CONFIG_GRKERNSEC_BRUTE kernel config option.
It tries to counteract brute-forcing probes.
In case of process running as a user it kills, if it's running as root it
makes the system panic.
--
dr Tóth Attila, Radiológus, 06-20-825-8057
Attila Toth MD, Radiologist, +36-20-825-8057

2013.Január 12.(Szo) 23:22 időpontban Michael Orlitzky ezt írta:
> I recently updated all of our servers to 3.7.0-hardened (from
> 3.4.2-hardened-r1) and re-did our iptables rules to avoid future pain[1]
> from the state -> conntrack switch.
>
> The first thing I noticed was that vsftpd apparently crashed on my own
> box, michael.orlitzky.com. The server stayed up, though, until I did
> something stupid and tried to kill the crashed process. Then it
> panicked. I drove to work, rebooted, and disabled vsftpd. Naturally that
> hasn't happened again.
>
> Last night, our VPN firewall went down; panicked, around 11:30pm. Drove
> to work today and rebooted it, but I'm not sure what the underlying
> cause was -- I didn't get a shot of the panic message. The only thing it
> does is OpenVPN on two e1000s.
>
> I've been looking through the dmesg of our other servers, just to see if
> anything looks out of the ordinary. There's one other machine still
> running vsftpd that has a non-fatal (i.e. stuff is still running) crash.
> There are more errors above this if needed, although I'm going to have
> to reboot it now.
>
> On the VPN box, I'll probably bump to 3.7.1-r2 and just pray unless
> someone has a better suggestion.
>
>
> grsec: From 61.160.222.83: Invalid alignment/Bus error occurred at
> 000000608f728691 in
> /var/log/apache2/abogadosdeaccidentedeautoenmarylandblog.com/www/error/error-2013-01-06.log[vsftpd:7764]
> uid/euid:0/0 gid/egid:0/0, parent
> /var/log/apache2/abogadosdeaccidentedeautoenmarylandblog.com/www/error/error-2013-01-06.log[vsftpd:2583]
> uid/euid:0/0 gid/egid:0/0
> grsec: From 61.160.222.83: bruteforce prevention initiated for the next
> 30 minutes or until service restarted, stalling each fork 30 seconds.
> Please investigate the crash report for
> /var/log/apache2/abogadosdeaccidentedeautoenmarylandblog.com/www/error/error-2013-01-06.log[vsftpd:7764]
> uid/euid:0/0 gid/egid:0/0, parent
> /var/log/apache2/abogadosdeaccidentedeautoenmarylandblog.com/www/error/error-2013-01-06.log[vsftpd:2583]
> uid/euid:0/0 gid/egid:0/0
> grsec: From 61.160.222.83: denied resource overstep by requesting 4096
> for RLIMIT_CORE against limit 0 for
> /var/log/apache2/abogadosdeaccidentedeautoenmarylandblog.com/www/error/error-2013-01-06.log[vsftpd:7764]
> uid/euid:0/0 gid/egid:0/0, parent
> /var/log/apache2/abogadosdeaccidentedeautoenmarylandblog.com/www/error/error-2013-01-06.log[vsftpd:2583]
> uid/euid:0/0 gid/egid:0/0
> PAX: please report this to pageexec@freemail.hu
> BUG: unable to handle kernel NULL pointer dereference at 0000000000000030
> IP: [<ffffffff81029972>] dup_mm+0x261/0x4c0
> PGD 18c661000
> Thread overran stack, or stack corrupted
> Oops: 0000 [#1] SMP
> Modules linked in: xt_tcpudp xt_multiport nf_conntrack_ipv4
> nf_defrag_ipv4 xt_conntrack nf_conntrack iptable_filter ip_tables
> x_tables cpufreq_ondemand uhci_hcd ehci_hcd thermal usbcore acpi_cpufreq
> tg3 microcode freq_table mperf usb_common processor libphy thermal_sys
> hwmon unix
> CPU 0
> Pid: 2583, comm: vsftpd Not tainted 3.7.0-hardened #1 HP ProLiant DL380 G4
> RIP: 0010:[<ffffffff81029972>] [<ffffffff81029972>] dup_mm+0x261/0x4c0
> RSP: 0018:ffff880187a4ddc0 EFLAGS: 00010286
> RAX: 0000000000000000 RBX: ffff880193c4c508 RCX: 0000000000000000
> RDX: ffff88018c4df500 RSI: ffff880193c4c508 RDI: ffff880154c32cf0
> RBP: ffff8801748fa3c0 R08: ffff88019bc112b0 R09: ffffffff810298cd
> R10: 8000000000000000 R11: ffff88018c4c9e00 R12: ffff88018bfc30c0
> R13: ffff880154c32cf0 R14: ffff8801748fa420 R15: ffff88018bfc3120
> FS: 000002ef1e350700(0000) GS:ffff88019bc00000(0000)
> knlGS:0000000000000000
> CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
> CR2: 0000000000000030 CR3: 0000000001329000 CR4: 00000000000007b0
> DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
> DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
> Process vsftpd (pid: 2583, threadinfo ffff8801907e3ca8, task
> ffff8801907e38d0)
> Stack:
> 0000000000000000 0000000000000000 0000000000000000 ffff8801748fa3c0
> 0000000000000000 ffff8801748fa3c8 ffff880194c52540 0000000001200011
> ffff880174920000 0000000000000000 000002ef1e3509d0 0000000000000000
> Call Trace:
> [<ffffffff8102a42e>] ? copy_process+0x829/0x119e
> [<ffffffff8102ae24>] ? do_fork+0x5c/0x2c2
> [<ffffffff8131f873>] ? stub_clone+0x13/0x20
> [<ffffffff8131f608>] ? system_call_fastpath+0x18/0x1d
> Code: 00 00 00 00 49 c7 45 18 00 00 00 00 49 c7 85 b0 00 00 00 00 00 00
> 00 49 8b 95 98 00 00 00 48 85 d2 0f 84 85 00 00 00 48 8b 42 18 <48> 8b
> 48 30 48 8b 82 c8 00 00 00 f0 48 ff 42 30 71 07 f0 48 ff
> RIP [<ffffffff81029972>] dup_mm+0x261/0x4c0
> RSP <ffff880187a4ddc0>
> CR2: 0000000000000030
> ---[ end trace 969655b532a2156e ]---
>
>
>
>
> [1] https://bugs.gentoo.org/show_bug.cgi?id=448906
>
Re: Crashes after 3.7.0-hardened upgrade [ In reply to ]
On 01/12/2013 06:16 PM, Anthony G. Basile wrote:
> Its e1000. This was an unknown issue until just recently. Is supposed
> to be fixed in the latest 3.7.1-r2. Let me know if it is and I'll drop
> 3.7.0 in favor of 3.7.1-r2.
>
> My appologies. I do test, but its impossible to test on every possible
> hardware config.
>

No problem, our VPN box is an odd Dell in a rack of HPs so it tests its
own kernel upgrades. I'll have it upgraded in a minute. Thanks for the
quick response.
Re: Crashes after 3.7.0-hardened upgrade [ In reply to ]
On 01/12/2013 06:22 PM, "Tóth Attila" wrote:
> Regarding the panic also see:
> CONFIG_GRKERNSEC_BRUTE kernel config option.
> It tries to counteract brute-forcing probes.
> In case of process running as a user it kills, if it's running as root it
> makes the system panic.

Oh, so it's just a normal vsftpd segfault and the protection is causing
the crash?

On my personal server I'll just ditch vsftpd. I'm giving free webhosting
to a few friends but the price just got raised to "learn SFTP."

For the other (work) web server, I think our customer is using
FileZilla, so it may be as easy as having them flip the dropdown box to
"SFTP" on Monday.

Thanks!
Re: Crashes after 3.7.0-hardened upgrade [ In reply to ]
On 01/12/13 18:16, Anthony G. Basile wrote:
> Its e1000. This was an unknown issue until just recently. Is supposed
> to be fixed in the latest 3.7.1-r2. Let me know if it is and I'll drop
> 3.7.0 in favor of 3.7.1-r2.

Bad news:

http://michael.orlitzky.com/tmp/e1000.jpg
Re: Crashes after 3.7.0-hardened upgrade [ In reply to ]
On 13 Jan 2013 at 11:28, Michael Orlitzky wrote:

> On 01/12/13 18:16, Anthony G. Basile wrote:
> > Its e1000. This was an unknown issue until just recently. Is supposed
> > to be fixed in the latest 3.7.1-r2. Let me know if it is and I'll drop
> > 3.7.0 in favor of 3.7.1-r2.
>
> Bad news:
>
> http://michael.orlitzky.com/tmp/e1000.jpg

that's a known false positive of the size overflow plugin,
see http://forums.grsecurity.net/viewtopic.php?f=3&t=3208&start=15

once you fix that and you can still reproduce the null deref,
can you email me the corresponding vmlinux along with the oops
please?
Re: Crashes after 3.7.0-hardened upgrade [ In reply to ]
On 01/13/2013 04:16 PM, PaX Team wrote:
>
> that's a known false positive of the size overflow plugin,
> see http://forums.grsecurity.net/viewtopic.php?f=3&t=3208&start=15
>
> once you fix that and you can still reproduce the null deref,
> can you email me the corresponding vmlinux along with the oops
> please?
>

I'm afraid I won't be able to crash it for about a week. I'm going to be
out of the office (modulo emergencies), so I've reverted to
3.4.5-hardened for the time being.

Once I'm back in and able to physically reboot it, I can try the fix and
not get crucified if it goes down.
Re: Crashes after 3.7.0-hardened upgrade [ In reply to ]
On 01/13/13 16:16, PaX Team wrote:
>
> that's a known false positive of the size overflow plugin,
> see http://forums.grsecurity.net/viewtopic.php?f=3&t=3208&start=15
>
> once you fix that and you can still reproduce the null deref,
> can you email me the corresponding vmlinux along with the oops
> please?
>
>

I'm back, and I see the new plugin is in the latest 3.7.3-hardened. So
I've upgraded to that and sprinkled some chicken blood on the server.
Here's hoping it stays up.
Re: Crashes after 3.7.0-hardened upgrade [ In reply to ]
On 01/23/13 10:17, Michael Orlitzky wrote:
> On 01/13/13 16:16, PaX Team wrote:
>>
>> that's a known false positive of the size overflow plugin,
>> see http://forums.grsecurity.net/viewtopic.php?f=3&t=3208&start=15
>>
>> once you fix that and you can still reproduce the null deref,
>> can you email me the corresponding vmlinux along with the oops
>> please?
>>
>>
>
> I'm back, and I see the new plugin is in the latest 3.7.3-hardened. So
> I've upgraded to that and sprinkled some chicken blood on the server.
> Here's hoping it stays up.
>

3.7.3-hardened (with the new overflow plugin) has been running for a
while, e1000 seems OK with it.

Thanks for your help.