Maximizing Performance

EJBCA is used to issue billions of certificates under a very high transaction load (hundreds or thousands per second). As with any large scale system you need to design the solution to support such a use case, both with the solution design and the IT infrastructure. For smaller use cases both design and IT is rather easy, but for huge use cases there are many things to take into consideration, and much expertise is needed (as it is for any IT system with huge amounts of data and tough requirements).

General Performance Tips

Normally in a PKI extreme performance is not really an issue. A default installation of EJBCA can easily issue tenths, even up to hundreds, of certificates per second, which is more than enough even for very large installations. For OCSP it is more important, but OCSP is also more lightweight so the default performance there is hundreds of requests per second.

In more specialized PKI installations there can however be the need to issue hundreds of certificates per second. Therefore we provide a list of configuration items that can maximize performance of EJBCA to issue more than 100 certificates per second (depending on hardware etc).

Disable logging to database in cesecore.properties, only use Log4jLogDevice. This removes a database insert and gives a big boost.
Configure Log4j to log only to file, not console, and use only INFO or ERROR logging, not DEBUG. This can give a small improvement.
Disable use of CertReqHistory in CA configuration (Use Certificate Request History). This removes a database insert and gives a big boost. The default setting is to have this disabled
Disable finishUser in CA configuration. Verify that this is ok first by reading the documentation for this feature. This can save a database read and update.
Enable all caches in cache.properties. If you cache indefinitely you can use the CLI command 'bin/ejbca.sh clearcache' if configuration changes are made. This can give a small improvement, see Clear All Caches.
Do not enforce any uniqueness (keys, DN, Subject DN SerialNumber) in CA configuration. This will ensure no extra selects are made and can give a small improvement.
Minimize the number of indexes in the database, only use the ones that are absolutely needed. This makes inserts in the database a bit faster and can give additional 10%.
Use the native JBoss connectors for http and https. This can give an additional 10%.
If you have a lot of certificates (>100M) it might be a good idea to use a separate table for the storage of the encoded certificates. See the database.useSeparateCertificateTable property in ./conf/database.properties.sample

Database Indexes

A prerequisite to get good performance under load is to have your database running fast. Make sure that (at least) the recommended database indexes applied. See doc/sql-scripts for recommended indexes. Check your database performance during a test run to ensure that it operates under optimal conditions.

Solution Design Considerations

In a high speed, high volume environment you don't want to design your solution to synchronize around a single object/row in the database as that will just limit your throughput. Instead you want to use non-conflicting transactions as much as possible.

To maintain speed in parallel use cases, EJBCA (and most other application) use optimistic locking in the database. With optimistic locking using the same object (end entity) for multiple parallel threads is not a good idea as there will be a race condition on this object. While you could configure the database/EJBCA to synchronize harder, but that will only lead to congestion and wait times.

In practice, this means that you should not make parallel requests using the same end entity if you want to have good performance without having to serialize around the end entity object. A better design for maximizing performance is to use:

a unique end entity per thread (serialized to the same end entity), or
a unique end entity per request, and/or
use API calls or enrollment operations that combine end-entity creation and enrollment into one operation

You can use random end entities created per request, enabling very large parallelism.

Adapting EJBCA for Extreme Volumes

Ephemeral Certificates

EJBCA can be configured to function as an Ephemeral Certificate CA. In this mode, EJBCA simply functions as a high speed certificate factory, issuing certificates but not storing any trace of them in the local database.

Using EJBCA in Ephemeral CA-mode curtails some functionality, foremost the following:

You cannot search for issued certificates.
You cannot enable certain limitations such as Enforce unique DN or Enforce unique public keys.
You may revoke certificates. However it requires specific CA configuration and only limited certificate data will be persisted upon revocation. For more information, see Accept Revocations for Non-Existing Entries in CA Fields.

If these limitations are acceptable you can gain some benefits by using Ephemeral CA-mode:

Maximum performance since no database queries are done. Enable to maximum caching and EJBCA will not make a single call to the database.
Minimal storage requirements. Since no issued certificates are stored in the database only EJBCA configuration needs storage.

To use Ephemeral Certificates, turn off Certificate Storage in either the Certificate Profile or in the Certificate Authority.

Database Sharding

To shard your database in order to not save everything in the same physical volume. By setting the database.useSeparateCertificateTable to true in the database.properties file, the certificate body will be stored in the table Base64CertificateData instead of CertificateData.

BASH

database.useSeparateCertificateTable = true

Base64CertificateDatacan then be sharded and placed on a different database volume.

Limiting Database Query Size

Certain types of queries can, whether intentionally or not, generate massive results. In order to not clog up the database driver, EJBCA has a default maximum value of 500 results for certain tables, which affects the following data sets:

Certificates
End Entities
Hard Token Usernames

You can change the query limit under System Configuration. For safety, there is still a hard upper limit of 25,000 entries.

Limiting Database Query Timeout

Certain types of searches can, whether intentionally or not, generate database heavy queries (even full table scans). To not overload the database, EJBCA will try to limit for how long a query is allowed to run for certain searches before aborting. For such limitations to work, both your database and the Java DataBase Connector (JDBC) driver must support this. This setting currently affects the following searches:

RA GUI Search for certificates
RA GUI Search for end entities

You can lower or raise the default of 10000 ms (10 seconds) in the query timeout under System Configuration. Setting the value to 0 means that EJBCA will not try to enforce any query timeout and will default to your application server and database configuration.

CRL Partitioning

If your population of unexpired certificates is large and you rely on CRLs, you might start finding that CRL generation times are beginning to spin out of control and that CRL sizes become unmanageable. EJBCA supports CRL partitioning in accordance with RFC 5280, allowing certificates to be assigned to a specific CRL shard.

CRL partitioning means that instead of a single CRL, the CRL is split into several shards. As the shards grow in size, EJBCA allows you to suspend shards, automatically creating new ones.

Service Pinning

Service pinning currently only functions on EJBCA Solution Platform installations.

In a clustered EJBCA instance, service execution happens at semi-random, the service being run by the first node to activate within the granted service interval. If some services - for example generating CRLs - are taking an inordinate amount of time, you may be experiencing latency in the cluster node executing the service, leading to intermittent delays being experienced while the service is running. The easiest solution is to pin the service to a single node and remove that node from the load balancer's roster, meaning that all service executions will happen on that node only, while enrollment, issuance and revocation operations are processed on the remaining node.

For more information on pinning to specific node(s), see Services.

Publisher Optimization

To optimize the throughput of the Queue Publishing Service, there are two configurations to take into consideration.

Publisher Queue Interval and Max Number of Entries

The publisher queue service worker can be configured by setting both the interval and the number of entries to be processed per run. Given a scenario where the queue is filling up faster than the CA can publish and where the service worker remains idle, these two settings can be modified to optimize throughput for your given environment.

The VA Peer Publisher can be configured to publish only the essential data needed to produce an OCSP response, i.e. issuer, serial number and revocation status. This dramatically reduced the amount of data which needs to be transmitted to the VA.

Infrastructure Considerations

Apart from configurations of EJBCA itself, an infrastructure supporting your performance goals is needed. It can be unexpected things such as network capacity between components leading to unnecessary latency.

Application server pool optimization, database and thread pools optimized for your load pattern.
Fast enough HSMs with low latency
High speed network links between components, like HSM, database, CA-VA, etc.

Optimizing the infrastructure can make the difference between issuing only hundreds of certificates per second, to thousands.

Precompiled OCSP Responses

Each OCSP reply requires an individual signature by the crypto token on the VA. While generated responses are cached by the EJBCA VA, validity times of OCSP replies are commonly short (< one day) and caches are not shared between nodes in a cluster, thus responses still need to be generated anew frequently. The traditional solution to this has been OCSP Stapling, caching the first reply encountered in the HTTP proxy. While this may solve the problem to some extent, it moves the burden of administration of caching the replies over to you.

Instead, EJBCA offers Precompiled OCSP Responses. Also known as Canned OCSP, this functionality allows a VA to generate the full set of expected OCSP responses on a regular schedule within a set timeframe when there are expected lulls in traffic. For any PKI, this will dramatically decrease the latency of the VA infrastructure.

For more information about Precompiled OCSP Responses, see OCSP Response Pre-Production.