ITSS Information Security Services

On this page:
Summary
Technical Details
Countermeasures
References

Summary

Stanford, along with a large number of research institutions and high performance computing centers, has become a target for some sophisticated Linux and Solaris attacks. An unknown attacker (or group) has compromised numerous multi-user Solaris and Linux computers on Stanford's campus using a variety of mechanisms. In most cases, the attacker gets access to a machine by cracking or sniffing passwords. Local user accounts are escalated to root privileges by triggering a variety of local exploits, including the do_brk() and mremap() exploits on Linux and the sadmind, arbitrary kernel loading modules and passwd vulnerabilities on Solaris.

If you believe your Unix computer has been affected by these intrusions, please contact the Information Security Services office (650-723-2911 or security@stanford.edu). Please include the name or IP address of the affected machine, as well as any compromised userIDs.

The attacker appears to be deliberately targetting machines in academic and high performance computing environments, rather than attacking systems indiscriminately.

It's not always obvious that a machine has been compromised, but they've been discovered at Stanford because:

• a user notices that their last login (as reported by many computers when the user logs in) is not consistent with the last time she actually logged in
• a user complains that performance on a particular computer has severely degraded
• a system administrator notices that users are logging in from unusual locations -- in particular, a sys admin notices multiple failed logins, sometimes for more than one user ID, coming from another university or research location
• unexpected errors are generated when a computer reboots, because a rootkit has been installed on a compromised machine, but doesn't work with that particular machine's operating system or kernel version

If you are responsible for Solaris or Linux computers at Stanford, please be sure that your systems are running the most recent versions of their kernels, and all security relevant patches. These are described in the security alerts referenced above.

If for some reason you cannot upgrade to the latest Linux kernel, please consider taking other precautions to defend these computers from malicious use. Possible precautions are discussed in the Countermeasures section below.

Stanford system administrators and the Information Security Services office are in contact with a number of academic and research organizations involving ongoing Linux and Solaris compromises. The perpetrators regularly gain access to an unprivileged local user account, presumably by sniffing passwords, cracking passwords from other compromised systems, or by triggering vulnerabilities in remotely accessible services.

In cases where the victim machine is running up to date versions of the kernel and privileged applications, the compromised user account is typically used to run a password decoding application called John the Ripper. This program is what usually causes system performance to degrade. The attacker is knowledgeable about Kerberos as well as other authentication systems, and has been observed running dictionary attacks against Kerberos passwords, as well as local password databases.

In cases where the target machine is running known vulnerable versions of an operating system or an application, or where there are insecure trust relationships between multiple machines (one of which is compromised already), the attacker is able to "get root." With root privileges, he can replace core utilities and applications on the victim machine, usually with the intention of capturing more usernames and passwords, and making it easier for himself to access the machine at a later time.

Some compromised systems have rootkits installed. A rootkit modifies the behavior of a compromised machine by hiding malicious activity (like keylogging or network sniffing); replacing system binaries with malicious versions that enable the attacker to collect information from local users; etc. Unfortunately not all rootkits leave evidence on the disk; truff's Phrack article on loadable kernel modules describes mechanisms for changing the behavior of a running machine by modifying the image in memory. Even without support for loadable kernel modules enabled, which might be expected to provide some protection against kernel-based malware, some rootkits are able to modify system behavior -- the SucKIT rootkit is perhaps the best-known of these tools. SucKIT has been found on several of the Linux boxes involved in the current set of compromises.

An introduction to Linux Kernel Rootkits

Given the sophistication of these attacks, and the difficulty involved in removing rootkits and illicit access mechanisms, we strongly recommend that compromised hosts be taken off line and completely rebuilt, including a fresh install of the operating system and application of all relevant patches.

How are they compromised?

The attacks start with the compromise of an unprivileged local user account. Usually this is because the attacker's captured the password from somewhere else: it's been sniffed off the network (through the use of insecure protocols like telnet), it's been collected when the user signs on to or from another compromised machine, it's been harvested from the password file on a compromised system.

If the target machine is behind on its patches, the attacker then uses one of a number of public exploits to elevate the unprivileged account to root status. Exploits target the Linux mremap() vulnerabilities, the Solaris kernel module loading vulnerability (for which an attack was made public on 8 Apr), and a Solaris priocntl() issue.

Configuration mistakes with the Network File Service (NFS): NFS is a mechanism for sharing file systems and directories over a network. In its most conservative configuration, the amount of privileged access provided to a shared directory is strictly limited, and filesystems are marked nosuid*. Unfortunately, there are many circumstances in which more permissive access is granted, usually to facilitate the distribution of system management and data processing tasks -- in particular, programmers and system administrators often make programs setuid root in order to run them without requiring that the user running them have root privileges.

In several cases of compromises at other institutions, the attacker has taken advantage of an NFS server exporting a filesystem to the world (ie. with no access restrictions), and allowing root=0 NFS access. With unrestricted NFS access to the target system from another compromised machine, the attacker is able to install a setuid root version of the UNIX shell program /bin/sh. This enables him to operate on the system with superuser privileges.

* nosuid is an filesystem mounting option that prevents any program or script on that filesystem from running with superuser privileges.

Evidence of compromise

This following list summarizes the ways in which compromised machines have been detected, both at Stanford and at other organizations involved in these attacks. Not all compromised machines have all these symptoms. This list may not be complete.

Any Linux or Solaris system that is behind on security updates is subject to compromise by the attacker. Any system running NFS or maintaining trust relationships with known-compromised machines are also at risk.

- chkrootkit detects the SucKIT rootkit or other malware -- but do not assume that a clean bill of health from chkrootkit means that your system is genuinely clean. Rootkits are notoriously hard to find (that's kind of the point). The SK rootkit replaces /sbin/init with itself, and moves the real system init file to /sbin/initXXX, where XXX is a special suffix that causes the file to be hidden. Some victim machines have had the .d rootkit installed. This one replaces /usr/lib/syslogd and /bin/netstat -- it is sometimes detected by looking for the file /etc/rc3.d/S20xfst. [Note: in order to identify modified and/or hidden files on a compromised system, you'll need to use versions of your normal system tools (like ls and netstat) that are guaranteed to be unaffected -- use known-good copies on a CD or floppy. For more information on preparing a forensics toolkit, check out

Incident Response Tools for Unix, Part One: System Tools
Forensic Analysis of a Live Linux System Part I

and the references therein.]

- OpenSSH has been replaced by a malicious version. This affects the SSH server and client as well as scp. The SSH client is modified to capture usernames, passwords, private keys and private key passphrases and send them to a destination specified by the attacker, enabling him to masquerade as a legitimate system user.

- in.ftpd and the ftp client have been replaced by malicious versions. These programs are modified to capture usernames and passwords and send them to the attacker's specified destination.

- ettercap is installed and used to sniff usernames and passwords from the network, as well as SSH v1 credentials which can be extracted

- the user news, list, or smmsp is enabled for interactive login (and is not a legitimate system user) -- be on the lookout for any new user accounts, especially those with root privileges or sudo access

- successful login from an "unexpected" remote host without any preceding failures (indicating that the user account being compromised had not yet been associated with these attacks)

- multiple failed logins in sequence from a remote system, followed by a successful login for a different user (showing the attacker trying different combinations of userIDs and passwords):

Apr 3 04:41:00 victim-host sshd[14598]: Bad protocol version identification '' from remote-unexpected-host-1

Apr 3 04:41:27 victim-host sshd(pam_unix)[14599]: authentication failure; logname= uid=0 euid=0 tty=NODEVssh ruser= rhost=remote-unexpected-host-2 user=loginID-no-go-1

Apr 3 04:41:29 victim-host sshd[14599]: Failed password for loginID-no-go-1 from remote-unexpected-host-2 port 36554 ssh2

Apr 3 04:41:37 victim-host sshd(pam_unix)[14601]: authentication failure; logname= uid=0 euid=0 tty=NODEVssh ruser= rhost=remote-unexpected-host-2 user=loginID-no-go-2

Apr 3 04:41:39 victim-host sshd[14601]: Failed password for loginID-no-go-2 from remote-unexpected-host-2 port 36555 ssh2

Apr 3 04:41:46 victim-host sshd[14603]: Accepted password for loginID-bingo from remote-unexpected-host-2 port 36556 ssh2

Apr 3 04:41:46 victim-host sshd(pam_unix)[14605]: session opened for loginID-bingo from remote-unexpected-host-2 port 36556 ssh2

(the failed logins are attempts to use loginIDs that have already been identified as compromised, and are therefore disabled).

- the root user account executes the su command to switch into an unprivileged user account:

2004-04-06 16:44:25 PDT auth.info local@victim-host su: [ID 366847 auth.info] su loginID-bingo succeeded for root on /dev/pts/5

(there are some circumstances in which this is a legitimate command, but in most "legitimate" situations the root user can accomplish their tasks without moving into a less-privileged user context).

- The system programs /usr/bin/pppd, /usr/bin/ypwatch and/or /etc/inet/inetd.conf are replaced

- The attacker's SSH key is added to the authorized keys file, allowing key-based remote logins

- If rsh is enabled, the .rhosts file is modified to allow universal access.

Detection tools:

chkrootkit - checks local machine for rootkit infestation
Rootkit Hunter - another detection tool
swatch or grep or other log monitoring programs to look for unusual login patterns

CIAC relates these tests for rootkits on Linux:

- The SucKIT rootkit allows an attacker to hide malicious files by giving them a particular ending. The current attacker is hiding code that ends in xrk or mem. To test for the presence of the rootkit, create a file whose name ends in xrk or mem, then execute an "ls -l". If the files you just created are not shown in the output of ls, it means that the rootkit is hiding them, ie. your system is compromised and needs to be rebuilt.

- Change directories to /sbin and execute an "ls -l init" -- the link count should be 1. Create a hard link to init using ln, and then execute the "ls -l init" again. If the link count is still 1, the SK rootkit is installed.

- Rooted systems send usernames and passwords to other compromised machines using TCP port 55, so if you keep records of network connections, traffic to destination port TCP/55 merits further investigation.

Note: all of the countermeasures discussed below involve some amount of work and inconvenience, both for you and your users. As you're considering how to protect your machines, your research and your co-workers from these aggressive and sophisticated attackers, please remember that although it's disruptive, the time you spend today in changing passwords, upgrading kernels, applying patches, and instituting remote access restrictions is far less than the time you'll spend rebuilding your system if it's compromised!

No single protection mechanism will prevent computer compromises -- but implementing a few of these suggestions will minimize your chances of being successfully attacked.

Prevention:

Install all security-relevant patches for your multi-user UNIX systems:

Sun releases patches for two privilege escalation vulnerabilities
Solaris vulnerability involving the passwd command
Solaris security vulnerability involving the priocntl system call

Two vulnerabilities in Linux distributions
Linux kernel patch fixes memory management vulnerabilities

Be sure your users are picking hard-to-crack passwords:

Pluggable password strength checking for your servers

Use different passwords on all your UNIX accounts.

Use SSH with RSA keys or Kerberos for user authentication.. [Note: these mechanisms will not prevent an account from being compromised, especially on a machine with a rootkit installed, but they may make it harder for an attacker to access unprivileged user accounts.]

Limit remote access to unpatched multi-user systems, using host-based firewalls, TCP-wrappers or other mechanisms. This is a potentially management-intensive process, especially for machines with a lot of remote users -- but even relatively permissive rules (allow connections from aol.com; deny all else) provide some protection against remote attacks.

[1] Computer Emergency Response Team: Steps for Responding to a UNIX or Windows Compromise

[2] Securing a UNIX desktop