How to track deleted log files in a compromised Linux system

Question

After a suspected security incident, I discovered that certain log files were deleted. I want to determine if there’s a way to recover or track what was deleted. My concerns are:

• How to check for signs of log file deletion using system metadata (e.g., lsof, auditd logs)?
• How to recover deleted log files from disk (e.g., extundelete, foremost)?
• What are the best practices for securing logs against tampering, such as forwarding logs to a remote server?

Any insights into forensic techniques for detecting log tampering would be appreciated.

Answer 1

Dealing with deleted log files on a compromised Linux system requires a methodical approach to detect, recover, and prevent future tampering. Here's how you can address each of your concerns:

1. Detecting Signs of Log File Deletion Using System Metadata

• Using lsof (List Open Files): This command helps identify processes that have files open, including deleted ones.

lsof | grep '(deleted)'

This will list files that are still open by processes but have been deleted from the filesystem.

• Checking auditd Logs: The Linux Auditing System (auditd) can track file deletions if it was configured prior to the incident.

  • View Audit Logs:

    ausearch -k delete

    This searches for events tagged with the key "delete".

  • Set Up Monitoring for Future Events:

    auditctl -w /var/log/ -p wa -k log_monitor

    This command sets a watch on the /var/log/ directory for write and attribute changes, tagging them with "log_monitor".

2. Recovering Deleted Log Files from Disk

• Using extundelete: If your system uses the ext3 or ext4 filesystem, extundelete can help recover deleted files.

  • Unmount the Filesystem:

    umount /dev/sdX#

    Replace /dev/sdX# with the appropriate device identifier.

  • Recover Files:

    extundelete /dev/sdX# --restore-directory /var/log/

    This attempts to restore deleted files in the /var/log/ directory.

• Using foremost: A file carving tool that can recover files based on their headers, footers, and internal data structures.

  • Run foremost:

    foremost -i /dev/sdX# -o /recovery_output/ 

    This scans the device and outputs recovered files to /recovery_output/.

3. Best Practices for Securing Logs Against Tampering

• Remote Log Forwarding: Send logs to a dedicated remote server to ensure they remain intact even if the local system is compromised.

  • Using rsyslog:

    • Configure the Remote Server: On the remote server, enable reception of logs.

      # In /etc/rsyslog.conf or /etc/rsyslog.d/remote.conf
      module(load="imtcp")
      input(type="imtcp" port="514") 

    • Configure the Local Machine: On the local machine, forward logs to the remote server.

      # In /etc/rsyslog.conf or /etc/rsyslog.d/remote.conf
      *.* @@remote-server-ip:514

• Implementing Immutable Log Files: Set log files to be immutable, preventing even the root user from modifying or deleting them.

  • Make Logs Immutable:

    chattr +i /var/log/syslog

    Replace /var/log/syslog with the specific log file you want to protect.

  • Remove Immutability (if necessary):

    chattr -i /var/log/syslog

4. Forensic Techniques for Detecting Log Tampering

• Check for Gaps in Log Timestamps: Inconsistent or missing timestamps can indicate tampering.

• Verify Log Integrity with Checksums: Generate and compare checksums of log files to detect unauthorized changes.

  • Generate a Checksum:

    sha256sum /var/log/syslog

  • Regularly Verify Checksums: Compare the current checksum with a known good value to detect alterations.

• Use Specialized Forensic Tools: Tools like chkrootkit and rkhunter can help detect rootkits and signs of log tampering.

  • Run chkrootkit:

    chkrootkit

  • Run rkhunter:

    rkhunter --check

By implementing these steps, you can enhance the security of your log files and improve your ability to detect and respond to unauthorized deletions or modifications.