How to Recover from a Linux Security Breach – Forensics, Analysis, & Building Teams (Part 1)

January 28th, 2013 by - 8,081 views

This 2-part GoGrid security blog series provides general guidelines for conducting a security analysis on a compromised Linux server and for recovering from a breach. Before you begin the security analysis, you need to consider two important factors:

1. The type of data your compromised server is storing or transmitting,
2. How important the server’s function is to your business

The data type—Personally Identifiable Information (PII) or Protected Health Information (PHI), for example—is important because your organization could be legally required to notify external parties and local or federal government agencies in the event of a breach. The compromised server’s function is important because its criticality may drive the recovery timeline.

You also may want to consider engaging a third-party that specializes in security forensics.

This series will cover 3 important items:

1) Understanding & assessing the breach
2) Setting up forensics & recovery teams
3) Recovering from the breach

Although this series won’t replace what a competent security firm can accomplish, it does provide an overview of some core processes, procedures, and activities you can do to potentially recover from a breach. And because each incident varies based on your computer system, be sure to conduct additional analysis and consult with experts to double-check your breach identification and resolution plan.

Understanding & Assessing the Breach


  • The compromised system is a Linux OS: CentOS, Debian, RHEL, or Ubuntu
  • You’ll use GoGrid’s MyGSI


You’ve just received a security notification from a third party or from GoGrid that your system is performing some malicious activities against the network environment. A security notification may include intrusion logs such as:

# begin logs
Nov 29 22:22:21.038854 173.x.x.x.36422 > S (src OS: unknown) 3344850667:3344850667(0) win 14600 <mss 1460,sackOK,timestamp 282574261 0,nop,wscale 6> (DF)

What should you do with this information?

Analyze the attack notification

Your first step should be to analyze the attack notification. For this example, we’ve highlighted some relevant elements from the attack notification:

  • The event date and time
  • The source of the attack (your server’s public IP)
  • The target’s IP (organizations often remove their IP for security reasons)
  • The target service (we can see that your system is attacking a remote organization’s SMTP services)


As soon as you become aware of the breach, you need to begin the research and recovery processes.

Setting up Forensics and Recovery Teams

To begin breach recovery, start by ensuring your company is organized.

If you believe your server has been compromised, form two incident management teams: a Forensics Team and a Recovery Team. The Forensics Team will perform the security analysis on the compromised system while the Recovery Team builds a new, hardened virtual machine (VM). The following diagram provides a high-level view of the major activities each team will need to perform.


Recovery Team Activities

This team is largely responsible for creating a new hardened VM image if one doesn’t already exist. This is where GoGrid’s MyGSI (Personal GoGrid Server Image) functionality comes in handy because it lets you make a copy of your new hardened VM for current and future deployments. Your system hardening approach largely depends on your risk tolerance level, but at minimum, we recommend the following:

  • Implement GoGrid’s basic steps for securing your Linux system
    • Be sure you don’t use the same system user accounts and passwords as those in the compromised systems because they could be subject to subsequent brute force attacks.
  • Install a rootkit analyzer on your VM. GoGrid recommends Rootkit Hunter.
  • Modify Iptables to only allow SSH connections from trusted networks such as your office or consider using GoGrid’s firewall services
  • Install a file integrity-monitoring (FIM) tool to monitor and alert when critical files systems have been created or modified or when new system user accounts have been created. GoGrid recommends OSSEC. (Watch for the OSSEC installation and administration guide we plan to publish.)
  • Save your new hardened VM using GoGrid’s MyGSI functionality.

Forensic Team Activities

As we said earlier, GoGrid strongly recommends engaging a professional third party that specializes in security forensics whenever protected data has been exposed.

Performing a security analysis on a compromised system requires considerable technical expertise, tenacity, and a little luck. Our recommendations are basic ones that may not fully confirm if a system has been compromised because intruders often install rootkits that may hide malicious code.

Ultimately, a compromised system’s data and its system binaries cannot be trusted.

Who is home?

Begin by determining who is on your system and what suspicious network connections or processes are running.

This diagram provides a high-level view of major activities you’ll need to perform during this phase.


WARNING: Before you connect to the compromised server, ensure the system you’re using is fully patched and has effective antivirus (AV) protection (real-time protection is enabled and it has updated AV virus definitions).

  • Run the who command once you’ve successfully logged into the compromised server.
  • The who command will display any account that has an open session on the system. Immediately contact GoGrid if you determine that there are unknown or suspicious sessions.
  • Don’t reboot or turn off the system because we want to collect any processes that could be running or identify an established connection as part our security analysis.

Are there any established connections?

  • Run netstat –antp | more
    • By running the netstat command, you’ll be able to identify if there are any existing connections to external systems.

Our netstat example provides the following results:

  • There’s a hidden process called ./ps (hidden processes strongly suggest the occurrence of malicious code).
  • The ./ps process appears to be making multiple remote connections to systems running port 25 (SMTP), which may explain the original attack notification (refer to “Analyze the attack notification” at the beginning of this article).


    • There’s also another process running under the name ./rdp (again, another hidden file). The ./rdp process appears to be making a connection to a remote system via port 21 (FTP), so there’s a strong possibility that something is copying data from your system to a remote system.


  • Next, let’s see what or who is running the ./ps process. We can do so by running the top command because it “provides a dynamic real-time view of a running system.”


The top command provides the following results:

  • A process called ./ps is consuming almost 100% of the CPU and is running under some user’s account (we’ve removed the user account for privacy reasons).
  • Next, search for the process identifier’s (PID) number, which in this case is 21104.


Identifying the details involved in the breach is critical to recovery. In Part 2, we walk through the Recovery process and steps that you can take to make your infrastructure more secure.

The following two tabs change content below.

Mario Duarte

Director of Security at GoGrid
Mario is GoGrid's Director of Security responsible for all security-related activities. He has 14 years of experience as a security professional working in the retail, health care, and financial sectors. He has built and managed security teams, developed and implemented security programs, and managed PCI and HIPAA compliance initiatives for medium and large organizations.

Leave a reply