Recovering a hacked server

The computer I keep in my office for use as a web / subversion / backup server was compromised recently. I don’t know when for sure or how. Here is the procedure I followed recovering the machine. It is mainly for future reference.

Machine:

• calamity.domain.suppressed.edu
• Ubuntu 7.04 (Feisty Fawn) w/ updates current to 7/2/07

Details of Intrusion:

• Complaints of (guestbook / comment) spam pointing to calamity:8082/d_special/xanax.html
• None of the files, directories, filenames that appear in any of them spam appear in the filesystem so these must be kept in memory by whatever process is serving them.
• Several other computers on the internet can be found using google which appear to have been compromised in the same way (i.e. they are serving the same exact pages from the same ports, etc).

Resources:

• CERT
• BLOG 1
• BLOG 2

Steps:

1. Removed from network and powered off. I know this isn’t what you are supposed to do since there may be processes running which would identify the method of break-in and type of attack. However, I was pressured to re-build the machine by the IT people who happened to drop by my office and I didn’t have time right then. Plus I’ve got important data on the machine that I didn’t want tampered with or destroyed. Powering the machine off seemed like the best idea at the time.
2. (few days later) Powered on the machine (still detached from the network).
3. Checked /var/log for anything I could find. Came up blank, except for alot of (apparently) unsuccessful break-in attempts through ssh.
4. Ran rootkit detectors rkhunter and chkrootkit. Neither one found anything useful. I think setting up something like tripwire will be a good idea after the machine is rebuilt.
5. Booted a Live CD version of Ubuntu 6.10, made backups of all important data to an external hard drive:
   • user directories
   • webserver files
   • subversion server files
   • mysql databases
   • Misc. config files and logs: /etc, /boot/grub, /var/log,
   • Mail in /var/mail
6. Installed Ubuntu 7.04 from disk after formatting the main disk
7. Ran the updates
8. Installed extra software:
   • emacs
   • openssh-server
   • inkscape
   • apache2
   • subversion
   • tripwire (to verify system integrity)
   • snort (basic intrusion detection)
   • firestarter (firewall configurator)
9. Thought about installing the ‘grsecurity‘ package, but it requires patching and compiling a kernel. I’ll save this for later.
10. TODO:
    • Setup log greping for tripwire, and snort
    • Setup grsecurity
    • Investigate PHP, Apache, Subversion security issues

Trying Xgl and compiz

I’m testing out Xgl, the OpenGL accelerated X server for linux, along with the compiz compositing window manager. Most things seem to work.

One annoyance is that the opening window animation that is standard in gnome is incredibly slow for some reason when using Xgl. To solve this I got some internet advice and messed around with gconf which I haven’t done before:


ben@calamity:~$ gconftool-2 --get /apps/panel/global/enable_animations
true
ben@calamity:~$ gconftool-2 --type bool --set /apps/panel/global/enable_animations false


Howto burn Dual Layer DVD in Linux

A simple terminal command to burn an iso, just cut and paste:


growisofs -use-the-force-luke=dao \
-use-the-force-luke=break:1913760 -dvd-compat -speed=2 \
-Z /dev/hda=IMAGE.000


replace IMAGE.OOO with whatever your backup file is called, and of course /dev/hda with the mount-point of your burner.

To figure out which of my optical drives is which, I run:
dmesg | grep -e 'hd.:'