Dealing with Unix/Linux break-ins
Note: This information is provided to help people whose machines are not supported by SCS Facilities, and who are dealing with a host that has been broken into. If your machine is suppported by SCS Facilities and has been broken into, send mail to email@example.com or call the SCS Help Desk (x8-4231; M-F, 9-5) and somebody from Facilities will fix your machine.
If an intruder is sufficiently clever, you won't notice that your host has been hacked until somebody notifies you that it has been misbehaving. If an intruder is not sufficiently clever, there are a few signs of intrusion that you can look for:
- Unexpected password changes or strange entries in /etc/passwd
- Unexpected increases in network traffic, system load, and/or disk space use
- Files with strange names, such as "..."
- Modified files (be careful, since many system files on Facilitized hosts have been modified by us).
- ps or ls failing to list expected processes and files
- Strange open ports on your machine (note that some backdoors may use ICMP packets to communicate)
- Changes in file permissions
- Users logged in from unexpected places
- Strange system problems
There are also a few useful tools that you can use to help detect break-ins.
What an intruder may have done to your host
You should assume that if an intruder can login to your host, they can get root access on it. Almost invariably, an intruder will do two things:
- Install some method to ensure continued remote access of some kind (either via a shell or some form of remote command execution).
- Have some way for the remote access to survive reboots.
In addition, it's very common for an intruder to do one or more of the following:
- Replace system binaries such as ls,ps, and/or system libraries with with trojaned versions that hide files and processes (a collection of such files along with scripts to make various other system changes is often called a rootkit).
- Install some form of "sniffer" that either eavesdrops on network connections or snoops tty input (note that encrypted connections will not protect one against a tty sniffer).
- Install loadable kernel module(s) that hide files and processes. This is very hard to detect in many cases.
- On Linux: modify file attributes with chattr so trojaned files cannot be easily deleted.
- Use hacked hosts to launch denial of service attacks on other sites, which may cause our connection to the internet to become flooded.
Because of the dangers of keyboard/tty sniffing, it is important that you change any passwords that you may have typed at a hacked host.
The big question
If you have been hacked, the big question you should be asking yourself is, "How?" The most common reasons for Unix/Linux break-ins are:
- Unpatched software
- Sniffed passwords
- Poor NFS export access restrictions
- Weak (crackable) passwords
- Various other things such as indiscriminate xhosting, etc
Cleaning up after a break-in
This information is provided as a service to people in the SCS community who are dealing with a non-supported Unix/Linux host that has been hacked. SCS Facilities takes NO responsibility for data loss or other problems that may result by using this information, and can provide little or no assistance in fixing non-supported hosts.
If you do find out that your machine has been broken into, you should contact SCS Facilities if the machine is a SCS host or on the SCS network and let us know about the break-in. We would be interested in seeing any log files (in particular, sniffer logs) or other information about the intrusion, especially if they point to other SCS hosts that may be compromised If the host is maintained by us, we will fix it. If the machine is not maintained by SCS Facilities, then you have two choices:
- Re-install it: This is the safest thing to do, since it can be very difficult to make sure that all changes made by the intruder have been found.
- Try to fix your machine: Not recommended.
If you do decide to clean up a host yourself, the following tools are useful (off-site links will open in a new browser window):
- Clean copies of system binaries such as ls, find, & ps
- These won't be of immediate help in case of a loadable kernel module or changes to shared libraries, but can be handy in other cases. You might want to use statically linked copies.
- Used to list open files and network ports on a host. Suffers from the same lkm/shared library problems as the system binaries.
- Can be used to verify binaries, though an intruder may have removed or modified the RPM database (or the rpm program).
- Useful for finding common rootkits, and can detect many types of LKM-based kits. Note that chkrootkit will falsely label the Facilities passwd program as being infected, and may also try to do a find on /afs.
- Port scanner. Very useful in finding out which ports are really open on your machine.
- The Coroner's Toolkit
- A collection of programs for doing forensics analysis on Unix hosts. Some of these programs are Linux-specific, and some are not for the faint-of-heart.
The cleanup process
The cleanup process basically consists of:
- Figuring out roughly when the host was hacked.
- Trying to determine what the intruder may have changed since then. The first part of this step often involves determining which tools accurately reflect the state of the machine. Once you have such tools, you should look closely at all files with suspicious mtimes and ctimes.
- Undoing the changes the intruder made, and replacing modified system binaries with clean copies.
- Patching or otherwise removing the means by which the hacker managed to break-in.
Because there are so many variables in cleaning up these intrusions, each clean-up is a custom job.