3 Important Things

Investigating a Suspected Compromise on a Linux System

When responding to a suspected compromise on a Linux system, answering the following key questions systematically helps uncover malicious activities and guide further investigation.

---

1. Is There Malware Actively Running in the System?

Objective: Detect and analyze potentially malicious processes running on the system.

Steps to Identify Malicious Processes:

1. List Active Processes:

   • Use ps or top to review running processes.

     ps aux | less

     Look for:

     • Processes with unusual names (e.g., xmr, random123).

     • Commands running from temporary or suspicious directories like /tmp, /dev/shm.

     • High CPU or memory usage by unexpected processes.

2. Analyze the Process Tree:

   • Use pstree for a hierarchical view of processes.

     pstree -p

     Focus on:

     • Unexpected parent-child relationships (e.g., apache2 spawning bash).

3. Investigate Executable Files:

   • Use lsof to list files opened by a specific process.

     lsof -p <PID>

     Check for:

     • Executables running from non-standard directories.

     • Open network sockets or files.

4. Review Hidden Processes:

   • Hidden processes might evade standard tools. Cross-verify using:

     ps -elf | grep <process-name>

---

2. Is There Any Suspicious Internal or External Communication?

Objective: Detect unusual or unauthorized network activity indicative of C2 communication or data exfiltration.

Steps to Investigate Network Activity:

1. Check Active Network Connections:

   • Use ss or netstat to list open connections and listening services.

     ss -tuln

     Look for:

     • Connections to unknown external IPs.

     • Unusual listening ports (e.g., port 4444 used for reverse shells).

2. Monitor Network Traffic:

   • Capture live traffic with tcpdump.

     tcpdump -i eth0

     Filter for suspicious activity:

     • DNS queries (port 53) for domains resembling C2 communication.

       tcpdump -i eth0 port 53

     • Large outbound transfers:

       tcpdump -i eth0 'tcp[13] & 16 != 0'

3. Analyze Historical Network Activity:

   • Check system logs for past network connections.

     grep "connect" /var/log/syslog

     • Look for unknown IP addresses or repeated failed connection attempts.

4. Inspect DNS Queries:

   • Review DNS query logs (if available) to identify suspicious domain resolutions.

---

3. Is There Any Persistence?

Objective: Detect mechanisms attackers use to maintain long-term access.

Steps to Identify Persistence Mechanisms:

1. Check for Malicious Cron Jobs:

   • List current user's scheduled tasks.

     crontab -l

   • System-wide cron jobs:

     cat /etc/crontab
     ls -la /etc/cron.*

     Indicators:

     • Unknown scripts or commands scheduled to run periodically.

     • Scripts running from temporary locations.

2. Inspect Startup Scripts:

   • Check for suspicious scripts or services configured to start on boot:

     ls -la /etc/init.d/
     ls -la /etc/systemd/system/

   • Review /etc/rc.local for manually added startup commands.

3. Examine SSH Keys:

   • Verify authorized keys in ~/.ssh/authorized_keys.

     cat ~/.ssh/authorized_keys

     Look for:

     • Unauthorized public keys potentially used for backdoor access.

4. Search for Recently Modified Files:

   • Identify files changed in the last few days.

     find / -type f -mtime -5 2>/dev/null

     Check:

     • Key system files like .bashrc, .profile, and system binaries for unauthorized modifications.

---

Findings and Next Steps:

By following the outlined steps, you can determine:

• Active Malware Presence: Identified through anomalous processes and suspicious executable files.

• Suspicious Network Activity: Revealed by unusual external connections, DNS queries, and active network traffic.

• Persistence Mechanisms: Found through malicious cron jobs, startup scripts, or unauthorized SSH keys.

Next Steps:

1. Isolate the affected system to prevent further spread.

2. Collect Evidence: Save logs, memory dumps, and identified malware samples.

3. Remediate: Remove malicious processes, clean persistence mechanisms, and patch vulnerabilities.

4. Monitor: Continuously monitor for signs of re-infection or additional compromises.