Free to read – no sign-up

Incident Response with journald

A server has been compromised. You investigate the incident systematically: filter journald logs, identify suspicious logins and processes, reconstruct a timeline, isolate compromised services, and write an incident report. University level — SOC analyst workflow in practice. 5 tasks, about 90 minutes.

Duration: 90 minLevel: AdvancedExercises: 5

Investigate a server incident

Incident response: investigating a server breach

A server has been compromised — an attacker gained access. As an analyst you must systematically find out: who got in, how, what they did, and how to stop them. This structured process is called incident response and is the daily work of a SOC analyst (Security Operations Center).

Key terms

journald / journalctl: the central logging system of modern Linux servers. journalctl searches these logs.
Persistence: mechanisms an attacker uses to embed themselves permanently (e.g. a hidden cronjob that runs regularly).
Timeline: the chronological reconstruction of the attack — the backbone of any incident report.
Containment: isolating the threat so no further damage occurs.

Your goal

You filter suspicious logins, find the malicious cronjob, build a timeline, isolate the compromised service, and write an incident report — university level.

Exercises

1. Find failed logins
Step 1: who attacked? Failed SSH logins reveal the IP addresses of a brute-force attack. You filter them from the journal and save the attacker IPs.

Create the working folder and extract the source IPs of the failed logins:
```
mkdir -p /home/student/incident
journalctl -u ssh | grep -i "failed password" | grep -oE '([0-9]{1,3}\.){3}[0-9]{1,3}' | sort -u > /home/student/incident/suspicious-ips.txt
```
grep -oE pulls out only the IP addresses, sort -u makes them unique.

Check: /home/student/incident/suspicious-ips.txt is not empty.
2. Identify suspicious cronjob
Step 2: find persistence. Attackers use cronjobs to embed themselves permanently — an entry that regularly starts, say, a reverse shell survives every reboot. You find it by checking all crontab locations.

Search the cron directories and copy the suspicious line into a file:
```
sudo grep -rE "bash|curl|wget|nc|/tmp/" /etc/crontab /etc/cron.d/ /var/spool/cron/ 2>/dev/null
```
Put the malicious line you found into the file (nano /home/student/incident/malicious-cron.txt).

Check: /home/student/incident/malicious-cron.txt is not empty.
3. Reconstruct attack timeline
Step 3: reconstruct the timeline. A chronological timeline shows the entry point, the attacker's actions and the impact in the right order — without it nobody can follow the incident. Use the timestamps from the journal.

Write timestamp + event per line (at least 3 lines) into timeline.txt, e.g.:
```
2026-05-28 02:14  First successful SSH login from attacker IP
2026-05-28 02:17  Malicious cronjob created in /etc/cron.d/
2026-05-28 02:19  Unknown service 'rogue-agent' started
```
Helpful for research: journalctl --since "today" | less.

Check: /home/student/incident/timeline.txt contains at least 3 lines.
4. Isolate compromised service
Step 4: containment. As long as the compromised service runs, the attacker can stay active. You must stop it (immediate damage halted) and disable it (won't restart after reboot — the persistence is gone).

Stop and disable the malicious rogue-agent service:
```
sudo systemctl stop rogue-agent
sudo systemctl disable rogue-agent
```
Check: systemctl is-active rogue-agent reports inactive and systemctl is-enabled rogue-agent reports disabled.
5. Write incident report
Step 5: incident report. A good report starts with a summary (decision-makers often read only that) and ends with recommendations (so the same attack doesn't work again). This closes the loop: lessons learned.

Write a structured report (nano /home/student/incident/report.md) containing at least these sections:
```
# Incident report

## Summary
Briefly: what happened, when, what impact.

## Timeline
(reference timeline.txt)

## Affected systems
...

## Recommendations
- Reset passwords
- Enforce public-key auth, disable password login
- Set up monitoring/fail2ban
```
Check: /home/student/incident/report.md contains a heading "Summary" (or "Zusammenfassung") and "Recommendations" (or "Empfehlungen").

Now practice it yourself

Reading is good – doing is better. Start this course on a real Linux VM, right in your browser. A free account is all it takes.

Start for free

Investigate a server incident

Incident response: investigating a server breach

Key terms

Your goal

Exercises

1. Find failed logins

2. Identify suspicious cronjob

3. Reconstruct attack timeline

4. Isolate compromised service

5. Write incident report

Now practice it yourself