I recently updated all of our servers to 3.7.0-hardened (from
3.4.2-hardened-r1) and re-did our iptables rules to avoid future pain[1]
from the state -> conntrack switch.
The first thing I noticed was that vsftpd apparently crashed on my own
box, michael.orlitzky.com. The server stayed up, though, until I did
something stupid and tried to kill the crashed process. Then it
panicked. I drove to work, rebooted, and disabled vsftpd. Naturally that
hasn't happened again.
Last night, our VPN firewall went down; panicked, around 11:30pm. Drove
to work today and rebooted it, but I'm not sure what the underlying
cause was -- I didn't get a shot of the panic message. The only thing it
does is OpenVPN on two e1000s.
I've been looking through the dmesg of our other servers, just to see if
anything looks out of the ordinary. There's one other machine still
running vsftpd that has a non-fatal (i.e. stuff is still running) crash.
There are more errors above this if needed, although I'm going to have
to reboot it now.
On the VPN box, I'll probably bump to 3.7.1-r2 and just pray unless
someone has a better suggestion.
grsec: From 61.160.222.83: Invalid alignment/Bus error occurred at
000000608f728691 in
/var/log/apache2/abogadosdeaccidentedeautoenmarylandblog.com/www/error/error-2013-01-06.log[vsftpd:7764]
uid/euid:0/0 gid/egid:0/0, parent
/var/log/apache2/abogadosdeaccidentedeautoenmarylandblog.com/www/error/error-2013-01-06.log[vsftpd:2583]
uid/euid:0/0 gid/egid:0/0
grsec: From 61.160.222.83: bruteforce prevention initiated for the next
30 minutes or until service restarted, stalling each fork 30 seconds.
Please investigate the crash report for
/var/log/apache2/abogadosdeaccidentedeautoenmarylandblog.com/www/error/error-2013-01-06.log[vsftpd:7764]
uid/euid:0/0 gid/egid:0/0, parent
/var/log/apache2/abogadosdeaccidentedeautoenmarylandblog.com/www/error/error-2013-01-06.log[vsftpd:2583]
uid/euid:0/0 gid/egid:0/0
grsec: From 61.160.222.83: denied resource overstep by requesting 4096
for RLIMIT_CORE against limit 0 for
/var/log/apache2/abogadosdeaccidentedeautoenmarylandblog.com/www/error/error-2013-01-06.log[vsftpd:7764]
uid/euid:0/0 gid/egid:0/0, parent
/var/log/apache2/abogadosdeaccidentedeautoenmarylandblog.com/www/error/error-2013-01-06.log[vsftpd:2583]
uid/euid:0/0 gid/egid:0/0
PAX: please report this to pageexec@freemail.hu
BUG: unable to handle kernel NULL pointer dereference at 0000000000000030
IP: [<ffffffff81029972>] dup_mm+0x261/0x4c0
PGD 18c661000
Thread overran stack, or stack corrupted
Oops: 0000 [#1] SMP
Modules linked in: xt_tcpudp xt_multiport nf_conntrack_ipv4
nf_defrag_ipv4 xt_conntrack nf_conntrack iptable_filter ip_tables
x_tables cpufreq_ondemand uhci_hcd ehci_hcd thermal usbcore acpi_cpufreq
tg3 microcode freq_table mperf usb_common processor libphy thermal_sys
hwmon unix
CPU 0
Pid: 2583, comm: vsftpd Not tainted 3.7.0-hardened #1 HP ProLiant DL380 G4
RIP: 0010:[<ffffffff81029972>] [<ffffffff81029972>] dup_mm+0x261/0x4c0
RSP: 0018:ffff880187a4ddc0 EFLAGS: 00010286
RAX: 0000000000000000 RBX: ffff880193c4c508 RCX: 0000000000000000
RDX: ffff88018c4df500 RSI: ffff880193c4c508 RDI: ffff880154c32cf0
RBP: ffff8801748fa3c0 R08: ffff88019bc112b0 R09: ffffffff810298cd
R10: 8000000000000000 R11: ffff88018c4c9e00 R12: ffff88018bfc30c0
R13: ffff880154c32cf0 R14: ffff8801748fa420 R15: ffff88018bfc3120
FS: 000002ef1e350700(0000) GS:ffff88019bc00000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 0000000000000030 CR3: 0000000001329000 CR4: 00000000000007b0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Process vsftpd (pid: 2583, threadinfo ffff8801907e3ca8, task
ffff8801907e38d0)
Stack:
0000000000000000 0000000000000000 0000000000000000 ffff8801748fa3c0
0000000000000000 ffff8801748fa3c8 ffff880194c52540 0000000001200011
ffff880174920000 0000000000000000 000002ef1e3509d0 0000000000000000
Call Trace:
[<ffffffff8102a42e>] ? copy_process+0x829/0x119e
[<ffffffff8102ae24>] ? do_fork+0x5c/0x2c2
[<ffffffff8131f873>] ? stub_clone+0x13/0x20
[<ffffffff8131f608>] ? system_call_fastpath+0x18/0x1d
Code: 00 00 00 00 49 c7 45 18 00 00 00 00 49 c7 85 b0 00 00 00 00 00 00
00 49 8b 95 98 00 00 00 48 85 d2 0f 84 85 00 00 00 48 8b 42 18 <48> 8b
48 30 48 8b 82 c8 00 00 00 f0 48 ff 42 30 71 07 f0 48 ff
RIP [<ffffffff81029972>] dup_mm+0x261/0x4c0
RSP <ffff880187a4ddc0>
CR2: 0000000000000030
---[ end trace 969655b532a2156e ]---
[1] https://bugs.gentoo.org/show_bug.cgi?id=448906
3.4.2-hardened-r1) and re-did our iptables rules to avoid future pain[1]
from the state -> conntrack switch.
The first thing I noticed was that vsftpd apparently crashed on my own
box, michael.orlitzky.com. The server stayed up, though, until I did
something stupid and tried to kill the crashed process. Then it
panicked. I drove to work, rebooted, and disabled vsftpd. Naturally that
hasn't happened again.
Last night, our VPN firewall went down; panicked, around 11:30pm. Drove
to work today and rebooted it, but I'm not sure what the underlying
cause was -- I didn't get a shot of the panic message. The only thing it
does is OpenVPN on two e1000s.
I've been looking through the dmesg of our other servers, just to see if
anything looks out of the ordinary. There's one other machine still
running vsftpd that has a non-fatal (i.e. stuff is still running) crash.
There are more errors above this if needed, although I'm going to have
to reboot it now.
On the VPN box, I'll probably bump to 3.7.1-r2 and just pray unless
someone has a better suggestion.
grsec: From 61.160.222.83: Invalid alignment/Bus error occurred at
000000608f728691 in
/var/log/apache2/abogadosdeaccidentedeautoenmarylandblog.com/www/error/error-2013-01-06.log[vsftpd:7764]
uid/euid:0/0 gid/egid:0/0, parent
/var/log/apache2/abogadosdeaccidentedeautoenmarylandblog.com/www/error/error-2013-01-06.log[vsftpd:2583]
uid/euid:0/0 gid/egid:0/0
grsec: From 61.160.222.83: bruteforce prevention initiated for the next
30 minutes or until service restarted, stalling each fork 30 seconds.
Please investigate the crash report for
/var/log/apache2/abogadosdeaccidentedeautoenmarylandblog.com/www/error/error-2013-01-06.log[vsftpd:7764]
uid/euid:0/0 gid/egid:0/0, parent
/var/log/apache2/abogadosdeaccidentedeautoenmarylandblog.com/www/error/error-2013-01-06.log[vsftpd:2583]
uid/euid:0/0 gid/egid:0/0
grsec: From 61.160.222.83: denied resource overstep by requesting 4096
for RLIMIT_CORE against limit 0 for
/var/log/apache2/abogadosdeaccidentedeautoenmarylandblog.com/www/error/error-2013-01-06.log[vsftpd:7764]
uid/euid:0/0 gid/egid:0/0, parent
/var/log/apache2/abogadosdeaccidentedeautoenmarylandblog.com/www/error/error-2013-01-06.log[vsftpd:2583]
uid/euid:0/0 gid/egid:0/0
PAX: please report this to pageexec@freemail.hu
BUG: unable to handle kernel NULL pointer dereference at 0000000000000030
IP: [<ffffffff81029972>] dup_mm+0x261/0x4c0
PGD 18c661000
Thread overran stack, or stack corrupted
Oops: 0000 [#1] SMP
Modules linked in: xt_tcpudp xt_multiport nf_conntrack_ipv4
nf_defrag_ipv4 xt_conntrack nf_conntrack iptable_filter ip_tables
x_tables cpufreq_ondemand uhci_hcd ehci_hcd thermal usbcore acpi_cpufreq
tg3 microcode freq_table mperf usb_common processor libphy thermal_sys
hwmon unix
CPU 0
Pid: 2583, comm: vsftpd Not tainted 3.7.0-hardened #1 HP ProLiant DL380 G4
RIP: 0010:[<ffffffff81029972>] [<ffffffff81029972>] dup_mm+0x261/0x4c0
RSP: 0018:ffff880187a4ddc0 EFLAGS: 00010286
RAX: 0000000000000000 RBX: ffff880193c4c508 RCX: 0000000000000000
RDX: ffff88018c4df500 RSI: ffff880193c4c508 RDI: ffff880154c32cf0
RBP: ffff8801748fa3c0 R08: ffff88019bc112b0 R09: ffffffff810298cd
R10: 8000000000000000 R11: ffff88018c4c9e00 R12: ffff88018bfc30c0
R13: ffff880154c32cf0 R14: ffff8801748fa420 R15: ffff88018bfc3120
FS: 000002ef1e350700(0000) GS:ffff88019bc00000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 0000000000000030 CR3: 0000000001329000 CR4: 00000000000007b0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Process vsftpd (pid: 2583, threadinfo ffff8801907e3ca8, task
ffff8801907e38d0)
Stack:
0000000000000000 0000000000000000 0000000000000000 ffff8801748fa3c0
0000000000000000 ffff8801748fa3c8 ffff880194c52540 0000000001200011
ffff880174920000 0000000000000000 000002ef1e3509d0 0000000000000000
Call Trace:
[<ffffffff8102a42e>] ? copy_process+0x829/0x119e
[<ffffffff8102ae24>] ? do_fork+0x5c/0x2c2
[<ffffffff8131f873>] ? stub_clone+0x13/0x20
[<ffffffff8131f608>] ? system_call_fastpath+0x18/0x1d
Code: 00 00 00 00 49 c7 45 18 00 00 00 00 49 c7 85 b0 00 00 00 00 00 00
00 49 8b 95 98 00 00 00 48 85 d2 0f 84 85 00 00 00 48 8b 42 18 <48> 8b
48 30 48 8b 82 c8 00 00 00 f0 48 ff 42 30 71 07 f0 48 ff
RIP [<ffffffff81029972>] dup_mm+0x261/0x4c0
RSP <ffff880187a4ddc0>
CR2: 0000000000000030
---[ end trace 969655b532a2156e ]---
[1] https://bugs.gentoo.org/show_bug.cgi?id=448906