LokiSec.com

I thought it would be nice to write about what you should do when your *nix server is compromised.  The idea came from a conversation that was sparked on the Full Disclosure List.

I am going to list out steps I recommend from the point of using a VPS for your hosting solution.  The steps are similar if you have physical access to your box.

1. Take a snapshot of active connections to your server by running the below command and piping it into a file.  You may catch their hand in the cookie jar.
   1. lsof -i > *some file*
2. Limit all public ports to only allow traffic to and from the IP address you are going to use to troubleshoot the issue.  This gives you the opportunity to still interact with your server while blocking access to the attacker.
   1. iptables-save > ipsave
   2. iptables -A INPUT -s *your IP* -j ACCEPT
   3. iptables -A INPUT -m state –state ESTABLISHED,RELATED -j ACCEPT
   4. iptables -P INPUT DROP
   5. iptables -P FORWARD DROP
3. Backup the entire system to preserve evidence. This will be invaluable if your server was involved with malicious activities that garner a visit  from the authorities.  It also might be beneficial to report it to your ISP/Host.  This will help them explain to the authorities that want your information what has happened.
4. Check your logs for suspicious activity.
   1. cat /var/log/auth.log
      1. Look for instances of users logging in from unknown IP addresses. (sshd)
   2. cat /var/log/syslog
      1. Investigate suspicious/new cron jobs.
   3. cat /var/log/apache2/access.log and/or /var/log/apache2/other_vhosts_access.log
      1. Look for odd files that are being accessed by external IP addresses.
5. Check running processes for items that do not match your baseline you took of the server when you deployed it. You did do that right? If there is something that you feel is suspicious look it up.  Chances are it has been documented by someone.
6. Check for listening ports.  Be sure that the ports that are in the LISTEN state are the ports you want to make available.  Ports that are listening to [::]:* are listening for traffic from all IP addresses.
   1. netstat -lptu
7. Check for new users that were added to your server.
   1. cat /etc/passwd or lastlog
8. Remove the users that you identified as being from the attacker. Using this command will remove the user, home directory, and mail spool.
   1. userdel *username* -r
9. Remove the files that you identified as being from the attacker.
10. When you identify the method that the attacker used to infiltrate your server fix it.  A search for the avenue of attack may lead you to information on how to fix the issue.  If a patch exists for the application that allowed the breach check it’s change log to be sure it addresses the vulnerability you found.  If not, report the issue to the distributor/programmer.  They might be able to get you a hotfix.  There is also a possibility that you will be rewarded for your efforts.
11. After everything is fixed open your firewall back up to those you offer your service to.
    1. iptables-restore < ipsave

Most intrusions into public facing servers are those that are running vulnerable web applications. If you identify that your web application is to blame you should take it offline to fix the vulnerability that was exploited by your attacker.  Be sure to clean up all files that the intruder left behind.  If you do not you could be leaving in a backdoor that the intruder could use to re-exploit your box.

Logs can be your best friend when dealing with an intrusion.  They tell you what the intruder accessed and to what degree they were able to infiltrate your server.  However, intruders often clear the logs covering their tracks.  For this reason I recommend that you setup your server to push the log files off to a central syslog server.