Skip to content

Recovering a hacked server

July 11, 2007

The computer I keep in my office for use as a web / subversion / backup server was compromised recently. I don’t know when for sure or how. Here is the procedure I followed recovering the machine. It is mainly for future reference.

Machine:

  • calamity.domain.suppressed.edu
  • Ubuntu 7.04 (Feisty Fawn) w/ updates current to 7/2/07

Details of Intrusion:

  • Complaints of (guestbook / comment) spam pointing to calamity:8082/d_special/xanax.html
  • None of the files, directories, filenames that appear in any of them spam appear in the filesystem so these must be kept in memory by whatever process is serving them.
  • Several other computers on the internet can be found using google which appear to have been compromised in the same way (i.e. they are serving the same exact pages from the same ports, etc).

Resources:

Steps:

  1. Removed from network and powered off. I know this isn’t what you are supposed to do since there may be processes running which would identify the method of break-in and type of attack. However, I was pressured to re-build the machine by the IT people who happened to drop by my office and I didn’t have time right then. Plus I’ve got important data on the machine that I didn’t want tampered with or destroyed. Powering the machine off seemed like the best idea at the time.
  2. (few days later) Powered on the machine (still detached from the network).
  3. Checked /var/log for anything I could find. Came up blank, except for alot of (apparently) unsuccessful break-in attempts through ssh.
  4. Ran rootkit detectors rkhunter and chkrootkit. Neither one found anything useful. I think setting up something like tripwire will be a good idea after the machine is rebuilt.
  5. Booted a Live CD version of Ubuntu 6.10, made backups of all important data to an external hard drive:
    • user directories
    • webserver files
    • subversion server files
    • mysql databases
    • Misc. config files and logs: /etc, /boot/grub, /var/log,
    • Mail in /var/mail
  6. Installed Ubuntu 7.04 from disk after formatting the main disk
  7. Ran the updates
  8. Installed extra software:
    • emacs
    • openssh-server
    • inkscape
    • apache2
    • subversion
    • tripwire (to verify system integrity)
    • snort (basic intrusion detection)
    • firestarter (firewall configurator)
  9. Thought about installing the ‘grsecurity‘ package, but it requires patching and compiling a kernel. I’ll save this for later.
  10. TODO:
Advertisements
Leave a Comment

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: