Do you Trust your System Logs?

by Alejo Sanchez, <alejo@corest.com>

0. The Problem

A commonly used technique among computer crackers, and experienced thieves as well, is to erase their fingerprints from the crime scene. This usually means erasing or modifying the logs stored on the computer that will expose them if carefully examined. Unprotected logs will make system security checks an impossible task in most cases. When crackers gain complete access to the system, they gain the ability to read, modify or erase any logs.

Let us define "Secure Logging" as the ability to record a given amount of information on a given storage media and be able to check the authenticity of that record later. This is part of the CIA triangle: confidentiality, integrity, and availability.

This definition says nothing about the security of the storage media where the information is recorded, we must assume that anybody can read, modify or erase it.

If a record is erased we will know as a fact that is not authentic, for there will be nothing recorded, and we know that something was.

Something different happens if the record is modified, we should have a way to tell whether it was modified, even if we do not know what information was recorded.

1. PEO and l-PEO A Solution

When an attacker gains absolute control of a system's resources, standard cryptographic techniques are usually compromised. They can always browse through the system memory to retrieve any symmetric or public key used for encryption, and with that information proceed to modify the stored logs. The protocols introduced within this article are oriented to determine if the information logged in a system before an intrusion has been altered. Given a system that appends records periodically to a database, and an attacker that gains access to the system in a given instant of time, an auditor can establish if the data logged before the intrusion has been modified.

These protocols do not guarantee what happens with the information recorded after the intrusion, since at that moment the cracker has control of all the inputs and outputs of the system. The security of these protocols rely on the fact that the state of the system in the instant a record is appended cannot be reproduced with the information present on the system thereafter.

The PEO (from the Spanish "Primer Estado Oculto" which means "Hidden Initial State") algorithm consists of three stages. Stage one is the initial key (K₀) generation; stage two calculates a new key consisting of the hash of the previous one concatenated with the new log line; on stage three, the auditor repeats all the second stage operations for the whole log set. If the integrity was compromised, the last stage will not match.

Stage	Action	Description
0	init K0 = Random()	The auditor generates a random K0 and stores it in a secure place.
1 < i < n	store Ki = H ( Di ^ Ki - 1 )	The source generates the Di (which is a log line); the system stores it and computes Ki using Di and Ki - 1. Ki - 1 is destroyed.
n	verify K'n =? Kn	The auditor verifies if K'n = Kn is true by computing K'n as a function of K'0 = K0 and all the Di

The main concept behind PEO and l-PEO is that information is destroyed once used; that you need a first state (key) to reconstruct each line's state again. This first state must be stored in a secure place (which could even be a written paper in a safe), removed from the system being protected. All the integrity checks must be done on a secure system.

The l-PEO protocol also saves a MAC (Message Authentication Code) of each line, thus you can discover where the log file was changed.

For a comprehensive description of these protocols, read the PEO papers.

2. Modular Syslog

Modular Syslog implements PEO and l-PEO protocols to protect logs and also adds other improvements to the log handling service.

This new design of syslog allows for an easy implementation of input and output modules. The modules that maintain compatibility with its predecessor are included in the standard distribution along with new modules:

• PEO-1 and L-PEO output module

• MySQL output module

• PostgreSQL output module

• regex matching filtering output module

• TCP log transport

• traditional UDP log transport

Modular Syslog runs on the following operating systems: Debian GNU/Linux, RedHat GNU/Linux, OpenBSD, FreeBSD, Solaris, Irix, and AIX. It should also be easy to port to other POSIX compliant platforms.

3. Getting The Software

Modular Syslog comes under the BSD license. Please read the file COPYING within the tarball for more information.

Full source code is available. While downloading, please fill the optional form.

There is also a Sourceforge project, Modular Syslog.

4. Setup

A program called "peochk", distributed with the Modular Syslog daemon, generates the initial state and does the checks. To generate the key for a log file, /var/log/authlog, you have to ensure the following:

1. the file should be present and empty (use "touch" to create it)

2. there is a directory /var/ssyslog

3. syslog is not running

4. klogd is not running

peochk -g -f /var/log/authlog -i authkey0 -m rmd160

Now you'll have a /var/ssyslog/var.log.authlog.key file with the initial key in binary mode, and another file ./authkey0 with the same key translated to ASCII (easier to write down). The hash method used to generate the key is rmd160 (sha1 and md5 algorithms are also available). You should save the contents of ./authkey0 file in a secure place.

Now edit your syslog.conf(5) file to enable PEO output module on that file:

auth.info	%classic /var/log/authlog %peo -m rmd160 -l -k /var/ssyslog/.var.log.authlog.key

With this, the PEO output module receives each message as an text string and calculates its hash key based on the last hash key generated for the previous message. The module removes the last key and writes the new one into the binary keyfile. The -l option is for doing l-PEO, which will write a hash of each line to a file named /var/ssyslog/.var.log.authlog.mac. The -m option tells it to use the rmd160 algorithm.

Now start modular syslog (in this case an OpenBSD box, but on others it is advisable to do it with the appropriate init scripts):

# /usr/sbin/syslogd -i bsd -i unix

5. Checking Logfile Integrity

Export the log files to a secure machine (and the MAC file in the case of l-PEO), and use peochk like this:

peochk -m rmd160 -f /var/log/authlog -i mykey

Here the file mykey is the initial key, and the MAC file should be /var/ssyslog/.var.log.authlog.mac (in this case). If it is broken it will simply tell you so; in case of l-PEO, it will tell you the line number.

6. Things to remember

1. Use a good pseudo random number generator when generating the keys. This may be on another system, you do not need that afterwords.

2. You can rotate logs, but you have to put them back together to make the checks. Specifically, do not erase old logs.

3. Use tabs and not spaces.

4. Take care of key file permissions; the /var/ssyslog directory should be mode 0700 and all files inside 0600.

7. Contacting Us

You can reach the author at alejo@corest.com.

There are two mailing lists for Modular Syslog:

msyslog-usr

This mailing list is for general discussions about the modular syslog system.

msyslog-dev

This mailing list is for technical discussions about the modular syslog system. It's intended for developers or any other person interested in the inner working parts of Modular Syslog.

Send a message to majordomo@corest.com with the text:

subscribe list-name

in the message body, replacing list-name by the name of the chosen list. You can also send an email to the developers only at core.devel.alat@corest.com

8. Thanks

The author would like to thank Kurt Seifried, Bruce Leidl, and Carlos Sarraute for their corrections and improvements to this article. Special thanks go to the DaemonNews editors for all their help.

9. Copyright

The contents of this article are copyright (c) 2001 CORE Security Technologies and may be distributed freely provided that proper credit is given.