Friends - Quick summary: After a recent breakin on a system I manage, I located a log called "2049hosts" which, I am reasonably confident, is a list of Internet connected hosts with port 2049 (NFS) enabled. I am sending this to the root and postmaster accounts at each of these systems to alert you that you may have the NFS service enabled on your machine. The mountd daemon that is part of NFS can be used to gain root access to your machine - even if you're not exporting any NFS volumes! If you have already taken care of this, know that your version of NFS is not vulnerable, or don't care, my apologies. Otherwise, read on. First, I'm sorry for sending out a form letter; I cannot write up a custom letter to over 3600 machines. I felt that the risk to your machines was greater than the annoyance you might feel toward unsolicited mail. Please understand I have nothing to gain by sending this. In fact, I've written this at some risk to myself and the systems I run; the person who attacked me may not be happy that I'm alerting others. The attacker used the mountd vulnerability documented at: http://www.cert.org/advisories/CA-98.12.mountd.html to get root access to my system and one other on that network. I also know this attacker got root access on at least two other systems mentioned in other logs we found. Here is a quick summary of what changes were made to our system; you might want to check the following to see if your system has been attacked as well. ---- What to look for: ---- - The following binaries were replaced with binaries having the same date and time, but different size and MD5 checksum. The replacements provide back doors, hide the existance of files and running tasks and other nasty stuff. On a RedHat, Caldera, or SUSE Linux system, running "rpm -Va | less" will show the Size and md5 checksums for these files changed from the original install: S.5..... /bin/ls S.5..... /usr/bin/du S.5..... /bin/netstat S.5..... /sbin/ifconfig S.5..... /bin/ps S.5..... /usr/bin/top S.5..... /usr/sbin/in.rlogind S.5..... /usr/sbin/syslogd S.5..... /bin/login S.5..... /usr/sbin/named S.5..... /usr/sbin/tcpd These system binaries should _not_ be trusted until you are sure they have not been attacked. It might be prudent to re-install them from the original CD-ROM for your operating system. - A new user called "sysop" was added to /etc/passwd. The UID for this user was 131072. Because of a quirk in how UID's are interpreted, this user has root privileges. This user's home directory was /tmp; you _may_ find a .bash_history file there. - The "in.telnetd" daemon was re-enabled on our machine (it's normally disabled as we use ssh). - The in.rlogind executable, above, had been enabled, allowing the attacker to get in via an executable that had a back door for him/her. - The "rexec" service in /etc/services had its port reassigned to 6969. - After replacing the "ps" command with a known good copy, "ps axf | less" showed some background tasks ("lpid"'s) running, sniffing for passwords on the network cable. You might want to investigate _any_ system binaries or running tasks that you don't recognize. On our system, the binaries were stored in /usr/info/.term/ . - The system logs had a number of entries erased from them. - /home/ftp/pub had its rights changed to 777 (anyone can read from, write to, or enter this directory). The xferlog had been erased, so the attacker probably transferred some files to or from this machine. ---- Suggestions: ---- I'm putting this section is as a guide to some _possible_ approaches you might want to take if you're not sure what to do, starting with the easiest and least intrusive to the hardest and most difficult. Please understand that I have absolutely no knowledge of your system, your needs of that system, or your experience level. I may be going over the heads of some and insulting others - my apologies to both groups in advance. - Disable NFS. If you're not using it at all, remove it from your system and stop the NFS and portmapper services. Note that the portmapper is used by non-nfs services; check carefully before removing it. - Upgrade NFS. If you need it, upgrade it to a level that no longer has the vulnerability. The Cert advisory page lists reports from a number of operating system vendors about which versions of their products might be vulnerable. - Upgrade IMAP. This and a number of other services have documented vulnerabilities - see http://www.cert.org for more details. - Inspect your system for evidence of a breakin. I include details on our breakin that you can use as a starting point, but _do_ _not_ assume that because you can't find any of our clues that your system is safe. - Spend some time learining about system vulnerabilities and exploits. http://www.cert.org and the website for your particular operating system might be two good places to start. - Construct a firewall to block access to the NFS service. The first two options (disabling or upgrading) are much simpler. - Re-install the operating system from scratch. This _might_ be a good choice if you know your system has been compromised and do not know how to clean it up. - Hire/ask/feed pizza and beverages to someone you trust to do the inspection and cleanup for you. Final notes: You should not trust my words completely. For all you know, I'm the attacker, trying to get you to do something that makes my job easier (I'm not, but can't prove it :-( ). If anything I've said does not make sense, _please_ consult someone you trust for guidance on how to proceed. Consulting a book on Unix system security might be a good idea. No repsonse to this message is necessary. I have used my real mail address in the headers and in the signature below if you wish to respond. I offer my sincere hope that you have not been attacked at all. Happy Holidays! Cheers, and thanks for taking the time to read this, - Bill --------------------------------------------------------------------------- Unix _is_ user friendly. It's just very selective about who its friends are. And sometimes even best friends have fights. William Stearns (wstearns@pobox.com) Mason, Buildkernel, and named2hosts are at: http://www.pobox.com/~wstearns ---------------------------------------------------------------------------