Monitor EJBCA host using Monit

The following instructions describe how to set up Monit to monitor EJBCA host variations.

Monit is an open-source monitoring utility that can be used to monitor servers and trigger alerts in case of errors. By integrating EJBCA and Monit, you can monitor the health of your EJBCA application, application server, database management system (DBMS), as well as Apache HTTP Server (HTTPD), if applicable. With Monit, you can view the status of your monitored services in the web interface (dashboard) and configure email notifications for when issues arise.

List of the technology stack used in this guide:

AlmaLinux 9.0
EJBCA version 7.10.0.1
WildFly24
MariaDB version 10.5.16
Apache version 2.4 front-end reverse proxy running on ports 80 and 443

Step 1 - Configure EJBCA health check service

The EJBCA health check servlet offers several application-level points to be monitored, see Monitoring and Healthcheck for more information. This guide outlines how to enable and monitor the CA health check.

Enable CA HealthCheck

In EJBCA, use the CA Activation page or the Edit CA page to enable CA health check for monitoring. Note that a testKey needs to be available in the crypto token, see Certificate Authority Overview.

To enable the CA health check service on the CA Activation page:

In EJBCA CA UI, under CA Functions, click CA Activation.
Select the desired CAs to be monitored.
Point your browser to the following URL to reach the health check status over HTTP:
TEXT
```
http://<ejbca-host>:<port>/ejbca/publicweb/healthcheck/ejbcahealth 
```

Track status output

Add a script to track the output of the health check URL. The output will be processed and displayed in the Monit interface. Polling intervals and alerts can be configured in the control file, see Step 3 - Configure Monit.

To add a bash script for the EJBCA health check:

On your server, create ejbcahealthcheck.sh:
CODE
```
[root@alma90 ~]# vi /opt/ejbcahealthcheck.sh
```

In the editor, add the following contents to ejbcahealthcheck.sh and then save:

CODE

#!/bin/bash
status=$(curl -s http://localhost:8080/ejbca/publicweb/healthcheck/ejbcahealth | sed -e 's/<[^>]*>//g')
if [ "$status" == "ALLOK" ] ; then
        echo "EJBCA is OK!"
        exit 0
else
        echo "NOT OK!"
		echo "${status}"
        exit 1
fi

Add execute permissions:
CODE
```
chmod +x /opt/ejbcahealthcheck.sh
```

Step 2 - Install Monit

To install Monit, you can either build it from source or install it via OS package managers.

Build Monit from source

To build Monit from source, refer to Monit on Bitbucket.

Install Monit using OS package managers

RHEL-based systems (provided via the EPEL release, refer to Extra Packages for Enterprise Linux):

CODE

sudo yum update
sudo yum install monit

MacOS:

CODE

brew install monit

Debian-based systems:

CODE

sudo apt-get update
sudo apt-get install monit

To display version details, run:

TEXT

[root@alma90 ~]# monit --version
This is Monit version 5.32.0
Built with ssl, with ipv6, with compression, with pam and with large files
Copyright (C) 2001-2022 Tildeslash Ltd. All Rights Reserved.

Step 3 - Configure Monit

By default, Monit provides a default configuration control-file named monitrc located at /etc/monitrc (Linux) that contains sample configuration examples.

Follow the steps below to create a new customized configuration file and update the configuration file to align with the services you want to monitor. Depending on your needs, this can range from being minimal without alerts to including additional services and email notifications.

Notes on control file configuration

Run a syntax check for the control file before running using monit -t.
Reload Monit after updating the control file using monit reload (or optionally restart the service via systemd).
Keep a backup copy of your customized file.
The control file permissions should be chmod 700.

For more information on customizing the control file, additional settings, and updating the monitored service details, refer to the Monit documentation.

To create a customized Monit control file, using a minimal or additional services example configuration, follow these steps:

On your server, run the following:
CODE
```
[root@alma90 ~]# vi ~/.monitrc
```

Update your new custom configuration file with one of the following example configurations, and then save the file:

Minimal configuration: The following provides a minimal example configuration:

CODE

# Global
set daemon  30              # check at 30 seconds intervals
set log syslog

# Monit host and port
set httpd port 2812 and
    allow admin:monit      # require user 'admin' with password 'monit'

# EJBCA Healthcheck
check program EJBCA[Healthcheck] with path /opt/ejbcahealthcheck.sh
        if status != 0 then alert

Additional monitored services configuration: The following provides an example configuration including additional services to monitor, and an option to trigger outbound email notifications via SMTP. Note that the example values for memory, usage, and so on are just sample values and may need to be adjusted for your environment and server resources.

CODE

# Global
set daemon  30              # check services at 30 seconds intervals
set log syslog

# Monit host and port
set httpd port 2812 and
#    use address localhost  # only accept connection from localhost (drop if you use M/Monit)
#    allow localhost        # allow localhost to connect to the server and
    allow admin:monit      # require user 'admin' with password 'monit'
    #    with ssl {            # enable SSL/TLS and set path to server certificate
    #    pemfile: /etc/ssl/certs/monit.pem
    #    }
    #

# EJBCA Healthcheck
check program EJBCA[Healthcheck] with path /opt/ejbcahealthcheck.sh
        if status != 0 then alert

# Network interface
check network Interface[Bridge] with interface ens160
        if link down then alert
        if link down then alert
        if changed link then alert
        if saturation > 90% then alert
        if download > 10 MB/s then alert
        if total uploaded > 1 GB in last hour then alert

# System resources
check system localhost[$HOST]
        if loadavg (1min) per core > 2 for 5 cycles then alert
        if loadavg (5min) per core > 1.5 for 10 cycles then alert
        if cpu usage > 95% for 10 cycles then alert
        if memory usage > 75% then alert
        if swap usage > 25% then alert

# MariaDB
check process MariaDB
  with pidfile /var/run/mariadb/mariadb.pid
  start program = "/usr/sbin/service mysql start" with timeout 60 seconds
  stop program = "/usr/sbin/service mysql stop" with timeout 60 seconds
  if totalmem > 400 MB for 5 cycles then alert
  if totalmem > 600 MB for 5 cycles then restart
  if cpu > 60% for 3 cycles then alert
  if cpu > 80% for 3 cycles then restart
  if 2 restarts within 7 cycles then timeout

# httpd
check process HTTPD with pidfile /var/run/httpd/httpd.pid
 group apache
 start program = "/usr/bin/systemctl httpd start"
 stop program = "/usr/bin/systemctl httpd stop"
 if cpu usage > 70% for 10 cycles then restart
 if 5 restarts within 10 cycles then alert

# Wildfly
check process Wildfly matching 'standalone.sh'
  start program = "/usr/bin/systemctl wildfly start"
  stop program = "/usr/bin/systemctl wildfly stop"
  if totalmem > 3 GB for 2 cycles then restart
  if cpu usage > 90% for 5 cycles then restart
  if 5 restarts within 10 cycles then alert

# Filesystem storage
 check filesystem localhost[/boot] with path /dev/nvme0n1p1
  if space usage > 70% for 5 times within 15 cycles then alert
 check filesystem localhost[/root] with path /dev/mapper/almalinux-root
  if space usage > 80% for 5 times within 15 cycles then alert

############################################################################

set eventqueue
     basedir /var/monit  # set the base directory where events will be stored
     slots 99999           # optionally limit the queue size

############################################################################

# SMTP settings
set mailserver smtp.yourdomain.tld
  port 465
  username "noreply@yourdomain.tld" password "pa$$wOrdHere"
  using auto

# Email recipient
set alert name.surname@company.com

# Message settings
set mail-format {
   from:    Monit <noreply@yourdomain.tld>
   subject: monit alert --  $EVENT $SERVICE
   message: $EVENT Service $SERVICE
                 Date:        $DATE
                 Action:      $ACTION
                 Host:        $HOST
                 Description: $DESCRIPTION

            Regards,
            MonitBot
 }

###############################################################################
## Includes
###############################################################################
##
## It is possible to include additional configuration parts from other files or
## directories.
#
include /etc/monit.d/*
#

Update the control file permissions:
CODE
```
chmod 700 /root/.monitrc
```
Once you have configured the services that you want to include for monitoring, and before launching the Monit web interface in the next step, run a syntax check for the control file:
CODE
```
monit -t
```

Step 4 - Start Monit

Once you have configured the control file, you need to start (and enable) the monit service.

To start Monit as a service and enable it to start automatically on boot, run the following:

CODE

<sudo> systemctl start monit
<sudo> systemctl stop monit // to stop
<sudo> systemctl enable monit // to enable on boot

Step 5 - Access monitored data

The monitored data can be reached via the Monit web interface or the command line.

Access Monit web interface

To access the Monit web interface and view the status of your monitored services, navigate to the host URL and port defined in the configuration control file in Step 3 - Configure Monit.

Open a browser and go to the host URL and port:
CODE
```
http://<host-or-IP>:2812
```
Sign in using the default HTTP Basic Authentication username=admin and password=monit.

The following displays examples of monitored data and the information displayed in the Monit Service Manager.

Monitoring examples - Minimal monitored services

Displaying minimal monitored services:
With minimal monitored services and the EJBCA health check in status failed:

Monitoring examples - Additional monitored services

Displaying additional monitored services:
Additional monitored services and the EJBCA health check in status failed:
Click EJBCA[Healthcheck] under Program to view details of the failed event, including native messages returned from the EJBCA health check. Example of Monit Program status:

Monitor services from the command line

You can also monitor the services from the command line and view the status of all monitored services, for example.

To print service status information:

On your server, run:
CODE
```
monit status
```
The output will look something like the following:

Email notification examples

The following provides examples of email notifications:

Status succeeded notification:

CA status failed notification:

Status failed notification (with more than one failed CA):

Useful Monit commands

The following lists some useful Monit commands and options. For a list of all available arguments and options, refer to the Monit documentation.

Print service status information (in the terminal):

CODE

monit status

Reload all daemon(s):

CODE

monit reload

Stop all running daemon(s):

CODE

monit stop all

Run syntax check for the control file:

CODE

monit -t

Start the service using a different control file:

CODE

monit -c /path/to/control-file

Next steps

Monit also allows you to monitor files, directories, and filesystems for changes, such as timestamp changes, checksum changes, or size changes. For more information, refer to the Monit documentation.

To learn more about the EJBCA health check service used for health monitoring, see Monitoring and Healthcheck.

To learn how to get started with EJBCA Community as a Docker Container, you can follow the Quick Start Guide - Start EJBCA Container with Unauthenticated Network Access.

To find out more about EJBCA use cases, see Solution Areas.