Mailing List Archive

Remotely replaced sshd_config, CentOS 5.3/SSH 4.3p2-36el5_4.2
Hello Everyone,

Fighting a bit of a nasty morning... anyone seen this before?

We have a number of servers that have password authentication disabled
as well as shell access disabled for all users except those whom have
keys. These servers run cPanel and have been updated to the following specs:

2.6.18-164.el5PAE #1 SMP Thu Sep 3 04:10:44 EDT 2009 i686 i686 i386
GNU/Linux
openssh-4.3p2-36.el5_4.2

Early (around midnight-1am CST) this morning we had a widespread attack
via an unknown vector. In the attack, the only thing that I can find is
the following (IP blacked out, although it is the attackers' address):

Nov 12 04:31:22 sharedserver/sharedserver sshd[16083]: Received
disconnect from 100.100.100.100: 11: No supported authentication methods
available
Nov 12 04:32:14 sharedserver/sharedserver sshd[11265]: Received signal
15; terminating.
Nov 12 04:32:14 sharedserver/sharedserver sshd[16570]: Server listening
on :: port 2.
Nov 12 04:32:14 sharedserver/sharedserver sshd[16570]: error: Bind to
port 2 on 0.0.0.0 failed: Address already in use.
Nov 12 04:32:27 sharedserver/sharedserver sshd[16611]: Accepted password
for root from 100.100.100.100 port 3630 ssh2
Nov 12 04:32:27 sharedserver/sharedserver sshd[16611]:
pam_unix(sshd:session): session opened for user root by (uid=0)


The concerning part is that it obviously appears that there is someone
reloading SSHD, but there is no successful login (at all) via shell
prior to this.

This time corresponds with a modified sshd_config that then allows
password authentication, whereby the user then logs in as root and has a
good time, so to speak.

I know that the following vulnerability is out in the wild:

Linux Kernel 'sock_sendpage()' NULL Pointer Dereference Vulnerability

However, since the user never actually logged into the server from what
I can see, I'm still searching for the real way that this occurred.

I have logs from these servers, if you need other information to
possibly help track this down that is possible. I'm having a hard time
finding the vector for this attack though...

Any assistance would be greatly appreciated.
Re: Remotely replaced sshd_config, CentOS 5.3/SSH 4.3p2-36el5_4.2 [ In reply to ]
2009/11/12 Adam Hubscher <offbeatadam@gmail.com>:
> Early (around midnight-1am CST) this morning we had a widespread attack via
> an unknown vector. In the attack, the only thing that I can find is the
> following (IP blacked out, although it is the attackers' address):

A couple of colleagues at UK universities have reported seeing things
similar to the following (they run RHEL5/CentOS/Scientific Linux) :-

A user account was used to log in from two sites:

195.22.101.220 (server14.Xuna.nl)
195.22.100.126 (server12.xuna.nl)

On the compromised systems (RHEL5) the ssh and sshd binaries were
replaced with ones that logged username and plain text password
information to a file called /etc/X11/fonts/misc/s1

The new ssh and sshd had the dates set to the originals, but they didn't
have a and i attributes set. Their new sizes were

334768 /usr/bin/ssh
445512 /usr/sbin/sshd

The output of 'strings /usr/sbin/sshd' included the following:

/etc/X11/fonts/misc/S1
/etc/X11/fonts/misc/s1
/etc/X11/fonts/misc/s1.tmp
rm -rf /etc/X11/fonts/misc/s1; cp /etc/X11/fonts/misc/s1.tmp
/etc/X11/fonts/misc/s1; chmod o+w /etc/X11/fonts/misc/s1; rm -rf
/etc/X11/fonts/misc/s1.tmp
/usr/X11R6/bin/xauth
no-X11-forwarding

and 'strings /usr/sbin/ssh' included:

/etc/X11/fonts/misc/S1
/etc/X11/fonts/misc/s1

Where a compromised system had had the openssh-server and openssh-clients
rpms updated after the compromise, 'rpm -V' on openssh-server and
openssh-clients looked ok (but the /etc/X11/fonts/misc/s1 file still
existed).

Regards,

Mark
Re: Remotely replaced sshd_config, CentOS 5.3/SSH 4.3p2-36el5_4.2 [ In reply to ]
68.50.70.187 is the attackers' IP.

Leif Nixon wrote:
> Adam Hubscher <offbeatadam@gmail.com> writes:
>
>> These servers run cPanel and have been updated to the following
>> specs:
>>
>> 2.6.18-164.el5PAE #1 SMP Thu Sep 3 04:10:44 EDT 2009 i686 i686 i386
>> GNU/Linux
>
> This seems vulnerable to CVE-2009-3547 and CVE-2009-2695. If SELinux is
> enabled, you can trivially get root on these machines if you can run
> commands as a logged in user.
>
> I would start by looking very hard at all successful ssh logins the
> hours before the known intrusion. It is very possible that some of them
> are performed using stolen ssh keys.
>
>> I have logs from these servers, if you need other information to
>> possibly help track this down that is possible. I'm having a hard time
>> finding the vector for this attack though...
>
> If you could share the IP number of the attacking host, that could be
> useful. Does /root/.bash_history contain anything interesting? Is there
> anything suspicious in /dev/shm? (There won't be, if the machine has
> been rebooted after the intrusion.)
>