Integrity Protected Security Audit Log

ENTERPRISE

Log Signing

There are many ways to fulfill requirements on Log Signing, also known as Signed Log Files.

The built in EJBCA Database Integrity Protection (described here).
Sign logs with external tools as they are being rotated. For more information, see External Log Signing section in EJBCA Security.

The IntegrityProtectedDevice Security Audit Log implementation stores the log entries to the database and relies on the Database Integrity Protection in EJBCA for protecting the authenticity of the log events. Each log event is given an identifier consisting of a "nodeId" and a "sequenceNumber" that is part of the integrity protected data. The nodeId is configurable and must be unique for each JVM in a cluster with shared database access. The "sequenceNumber" is unique per nodeId and starts with 0. The sequenceNumber for a node (JVM) is read when the first log entry is written to this device and then kept in memory for the duration of the JVM's lifetime.

For more technical details, see the Database Integrity Protection in EJBCA Security.

The security of this implementation relies on:

Database integrity protection token.
The sequence number in memory for each node (JVM).

Using a sequence number will prevent that a single log entry is removed without detection being possible. Keeping the sequence number in the JVM will prevent that all log entries up to a previous point in time is removed without detection being possible.

This implementation cannot detect:

Removal of all the latest log entries for a node up to a previous point in time if the node's JVM is not running.
Forged log entries if the database integrity protection token is compromised.

The motivation for this design is that each node (JVM) does not have to wait for other nodes in a (shared database only) cluster. Internally in each JVM the only locking between threads happens when the sequence number is atomically updated. This will allow horizontal scaling to a very high degree without any JVM-to-JVM communication when the shared database supports row-locking.

When using this implementation, the integrity of each fetched log entry is always validated when loaded from the database, but full validation with checks for missing sequenceNumbers are not performed.

Configuration of integrity protected log device is in the files:

conf/cesecore.properties
conf/databaseprotection.properties

It is possible to verify a whole table with the Local Database CLI tool. If any row in the table can't be verified the tool will show it. For the AuditRecordData table it will also be checked with the sequence number that no row is missing.

Verifying Integrity Protected Security Audit Logs

The Local Database CLI can be used to verify database protection both in an online EJBCA instance and in a database where only a Security Audit Log has been exported.

For more information on the Local Database CLI, see Command Line Interfaces.

Repairing Sequence Gaps

The Security Audit Log sequence number per node is a counter. Whenever there is a need to write an Security Audit Log entry, the counter is fetched and incremented (as an atomic operation). If the write to the database fails:

The counter will not be decreased (since multiple threads might be operating on it there is no way of knowing what the missing entry is).
There will be a sequence gap in the log

This could also happen in the case where an attacker has removed certain entries in the database to cover his or her tracks. Since failing to write audit log to the database will roll back the transaction that initiated the security event, the security event will not be performed. Most likely you could correlate this with server logs from the time this happen to find that the database was unavailable for technical reasons. For example, a MariaDB+Galera cluster that is in the process of reforming a quorum could experience a few failed transactions before being operational again.

There is currently no user-friendly way to "repair" such sequence, but it should rather be documented why this happened (or rather that the cause was technical and not malicious).

A non-user-friendly way to repair sequence gaps follows this principle:

In a copy of the database, delete all records except 1 (or as many as you have holes)
Update the record(s), using sql to change sequenceNumber to the ones missing, and primaryKey to something unique
Run ejbca-db-cli export from the copy database. This will result in a dump with only these records, matching the holes in the original database
Run ejbca-db-cli import to import this dump in the original database. This will insert the missing sequenceNumbers, using databaseprotection as configured in your ejbca-db-cli

Starting a Fresh Audit Log

If the audit log gets too large or damaged, you can start a fresh new audit log instead of trying to repair an old one. Since the Audit Log is controlled per EJBCA node, you can do this either globally (for all nodes) or for a single node.

To start a fresh audit log globally, for all nodes, do the following:

Stop all EJBCA nodes.
Archive/Export the existing AuditRecordData table, in order to have the old audit log available off-line (you do not want to just delete it without keeping a copy).
Truncate the AuditRecordData table, removing all existing records.
Start EJBCA nodes again. With AuditRecordData empty, each node will start a fresh new audit log.

Exporting Security Audit Logs

All installations have their specific requirements how audit logs should be handled. There are several ways to export Security Audit Logs.

Rotate system log files (including the audit log) and archive/process the rotated log files
Use a SYSLOG appended to send the System log (including the Audit log) to a central log server that performs monitoring functions.
Export the database audit log using the Local Database CLI
Export the database audit log using database tools
Export a specific view from the Audit Log page in CA UI to an XML file