Monitoring and Healthcheck

The EJBCA health check service can be used for remote health monitoring. Monitoring the health is essential if running EJBCA in a clustered installation and can help determine whether a node should remain in the cluster or needs to be taken out.

Health checks are designed to indicate if something is not operating as expected so that a monitoring system can send alarms and cluster nodes be taken offline.

An activated CA, for example, cannot remain online if the Crypto Token goes offline, and the health check will issue a warning.

Note, however, that putting a CA in the CA Service State offline will not trigger the health check, since the CA has been deliberately deactivated. Therefore, the following configuration will not result in any health check warnings.

For information on how to set up Monit to monitor an EJBCA host, see Monitor EJBCA host using Monit.

Servlet URL

The servlet is located in the URL: http://localhost:8080/ejbca/publicweb/healthcheck/ejbcahealth

Note that the client (e.g. the load balancer) is responsible for closing the connection to the application server. Failure to do so may result in denial of service, preventing other clients from connecting to EJBCA. For more information, refer to Wikipedia on Slowloris.

URL Query Parameters

The health check URL supports query parameters to further customize what checks are performed during a health check. The following parameters are supported:

Parameter Name	Value Type	Description
ca	String or "none"	Name of a CA to be tested during a health check. Can be specified more than once to test additional CAs. If "none", then no CA health checks are performed. Note that if the CA specified does not have "included in health check" selected in its configuration, it will not be checked. For example `ca=CA1&ca=`My Second CA. If this parameter is not included, then all CAs that have the "included in health check" configuration set will be checked.
ocsp	"true" or "false"	Specify whether or not the OCSP check should be performed.
publishers	"true" or "false"	Specify whether publishers should be checked.
tokenX/keyX	String	By adding "token" and "key" query parameters, crypto tokens can be included in the health check. "X" is a common suffix used to pair up the token and key parameters. For example `token=IssuingToken&key=testKey&token1=RootToken&key1=testKey001` would perform two token tests, one by testing testKey on IssuingToken, the other by testing testKey001 on RootToken.

Configuration

You can set which CAs are checked by the health check service in the EJBCA CA UI, either on the CA Activation page or on the Edit CA page.

The behavior of the servlet can be modified by configuring the below values in conf/ejbca.properties.

General Configuration

The following configuration parameters may be set to configure authorization and what the service checks:

Key	Default	Description
healthcheck.amountfreemem	1	The amount of memory that must be free on the server, in megabytes.
healthcheck.dbquery	select 1	Parameter indicating the string that should be used to do a minimal check that the database is working.
healthcheck.authorizedips	127.0.0.1	Specifies which remote IPs that may call this healthcheck servlet. Multiple IPs may be separated by a semicolon.
healthcheck.catokensigntest	false	Set to true to perform a test signature on each CA token during the check. Otherwise just checks that the token status is active.
healthcheck.publisherconnections	false	Set to true to perform a health test on all active publisher connections.

Maintenance File Properties

Key	Default	Description
healthcheck.maintenancefile		Location of file containing information about maintenance.
healthcheck.maintenancepropertyname	DOWN_FOR_MAINTENANCE	The key of the property value in the maintenance, should be in the following format: `DOWN_FOR_MAINTENANCE=true`.

Servlet Configuration

The following parameters configure what message or HTTP error code the health service returns.

Key	Default	Description
healthcheck.okmessage	ALLOK	Text string used to say that everything is ok with this node. Any properties defined properties value can be used here by inserting it in as a property, e.g: `ALLOK ${httpsserver.hostname} Version ${app.version.number}`
healthcheck.sendservererror	true	Set to true of the HTTP error code 500 should be sent in case of error.
healthcheck.customerrormessage	null	Allows for a custom error message to be configured.

Error Messages

If an error is detected one or several of the following error messages is reported. All errors will be sent with a response code of 500

Error	Description
MEM: Error Virtual Memory is about to run out, currently free memory : number	The JVM is about to run out of memory
DB: Error creating connection to database	JDBC Connection to the database failed, this might occur if DB crashes or network is down.
CA: Error CA Token is disconnected: CAName	This is a sign of hardware problems with one or several of the hard ca tokens in the node.
MAINT: DOWN_FOR_MAINTENANCE	This is reported when the healthcheck.maintenancefile is used and the node is set to be offline.
Error when testing the connection with publisher: PublisherName	This is reported when a test connection to one of the publishers failed.
Could not perform a test signature on the audit log.	Reported when the audit log failed to sign (if database protection is enabled)