Chat with Us
Get a Call
This is Part 2 of a GoGrid security blog series on identifying and recovering from a Linux security breach. Part 1 provided general guidelines for conducting a security analysis on a compromised Linux server and forming strategic teams to address and resolve the breach.
In this article, we’ll review some recommended steps for recovering from a breach.
Recovering from the Breach
Lock the doors
Now that you’ve confirmed that there are no intruders logged in and you’ve identified the established connections, it’s time to “lock the doors.” Locking the doors largely depends on who is managing your firewall. Contact GoGrid in the event that we’re managing your firewall or perform the following actions if you manage your firewall:
- Modify your system’s iptables configuration to restrict all remote console connections such as SSH to your office network
- Modify your system’s iptables configuration to block all previously identified suspicious connections from and to your system.
- Modify your system’s iptables to block all other services from the public Internet to your server. Doing so will effectively bring down your website or services, but you want to avoid compromising your customers or web site visitors.
Install and run a rootkit analyzer
Until this point, we’ve have been running common tools provided by the system. Because we really can’t trust a compromised server, we’ll need to install and run a rootkit analyzer. GoGrid recommends Rootkit Hunter.
Download Rootkit Hunter by running :
wget http://sourceforge.net/projects/rkhunter/files/rkhunter/1.4.0/rkhunter-1.4.0.tar.gz/
Next, unzip the files and untar them by running:
tar zxvf index.html
Change directories to rootkit by running:
cd rkhunter-1.4.0/
Make sure that you’re running under root before you run the software installer by running:
su
Now install rkhunter:
./installer.sh –install
Next, update rkhunter:
rkhunter –propupd
Start rkhunter by running:
rkhunter -c
Rkhunter will prompt you to press enter each time it completes a particular security scan You should see a report on your console similar to the following screen capture upon scan completion:
Our scan results suggest there are no rootkits, which is good; however, there are some warnings that may require your attention. Considering reviewing the results for any items that contain “Warning” by running:
cat /var/log/rkhunter.log | more
Back to our analysis
Now that we’ve confirmed we don’t a rootkit, we can continue with our investigation. The following summarizes our current findings:
- A process called “ps” is consuming almost 100% of CPU and is running under some user’s context and its PID is 21104.
It’s time to investigate the hidden process by running:
ps –l 21104
The ps command (not to be confused with the potentially malicious “.ps” process in this example) displays “a snapshot of the current processes.”
Our ps command results display the PPID number, which in this example is 13855 (PPID is the process parent PID or what is responsible for running the ./ps process):
We can find out more about this PPID by running:
ps –o cmd,cwd –p 13855
In doing so, we find another hidden file called ./kas:
Next, change directories to the user’s home directory because that is who we see the process running under (recall the TOP results).
However, simply running the ls –al command doesn’t show us the hidden “kas” file:
So we run the following command:
find . -print
And Eureka, we find it! Using the escape key (\) is a common trick intruders use to try to cover their tracks. In fact, we find an entire hidden directory \ /.smtp:
We can see what’s inside this directory by typing:
cd \ / .smtp/
Once inside the directory, we can run ls –al; however, there are additional hidden files, so we need to run
find . –print
again:
And now we find the rest of the hidden files:
What should you do if you find suspicious folders and files?
That largely depends on what you intent to do with them. As a general rule, you need to proceed with caution before you consider backing up or copying the malicious files to another location to avoid potential cross contamination. Moreover, your organization could be required to perform a thorough security forensic analysis to determine if PII data or PHI data was exposed.
Moving Forward
To recap, you’ve formed two teams as part of your incident management team (IMT) activities. One group, the Recovery Team, has been largely responsible for building a new hardened Linux image that leverages GoGrid’s MyGSI functionality to save as your new base image. The other group, the Forensic Team, has been largely responsible for performing security analysis on the compromised system. Once they’ve finished, your next priority is to re-build your compromised system.
Reconvene the IMTs
Although the Forensic Team has confirmed that the system has been compromised, it hasn’t yet determined the method used to successfully exploit the system. Unfortunately, you may not have days or weeks to perform a thorough analysis due to business service recovery requirements. There are, however, some activities the IMTs can perform to get your website running again with some sense of security.
Turn the lights back on
We recommend dividing personnel into two groups again to reduce your recovery time. One group, the Recovery Team, will create a new set of system user accounts and database accounts. The other group, the Forensic Team, will perform static code analysis. When both teams have successfully completed their respective tasks, you’ll be able to restart your business services.
Recovery Team
This team has hardened the base image, installed and ran rkhunter, installed OSSEC, and protected the server by using some type of firewall services. Now it’s time to finalize their activities.
- Create any new user system accounts, but avoid using the same accounts and passwords from the compromised systems because they may subject to successfully brute force attacks.
- Install any additional applications such as PHP or MySQL, but ensure they are fully patched.
- Ensure you don’t let protected services such as your database be publicly accessible (that should be part of your firewall policies).
- Re-create any database accounts , but avoid using the same database accounts and passwords from the compromised server.
Forensic Team
Members of this team will need considerable code development experience to perform the following steps:
- Obtain a copy of your application’s source code, but don’t use any of the code or configuration files from the compromised systems.
- Perform static code analysis on your source code by running one of the following tools:
- RIPS –A tool that finds PHP vulnerabilities by leveraging static code analysis (it checks your PHP code without executing any of your code). This link takes you to the RIPS website where you can download the tool and relevant documentation.
- OWASP – Although somewhat outdated, the OWASP team has published a list of source code analyzers here.
- Note that you must address identified vulnerabilities before deploying the source code back to the new server image.
Remember, server breaches are extremely serious and often difficult to detect. At GoGrid, we run continuous infrastructure security scans on our customers’ accounts to ensure that no servers are compromised. We do this as a courtesy and to provide a risk-free public cloud. If you receive a security notification from GoGrid, we ask that you immediately review it and begin to take action. If you have any questions, we encourage you to contact us, especially if you believe your GoGrid Cloud Server has become compromised.
Mario Duarte
Latest posts by Mario Duarte (see all)
- How to Recover from a Linux Security Breach – Recovery & Hardening (Part 2) - January 29, 2013
- How to Recover from a Linux Security Breach – Forensics, Analysis, & Building Teams (Part 1) - January 28, 2013
- Security Basics: 4 Steps to Tighten up Linux Security - November 20, 2012
Good guide!
Sometimes it is more useful to restore from older backups and then install updates one by one from trusted source. However, all the log files should be preserved and inspected.