On-Prem Multi-Node Installation Guide

This guide describes how to deploy a scalable, high-availability multi-node on-premises cluster with dedicated backend, frontend, and optional scan nodes to support production workloads and large data volumes.

Overview

The multi-node installation creates a distributed cluster with specialized nodes. It is recommended for:

• Production environments

• High availability requirements

• Large data volumes

Backend Nodes

Backend nodes run the following services:

Frontend Nodes

Frontend Nodes run the following services:

Scan Nodes

Scan Nodes are an optional component for scalability needs. The following services run on scan nodes:

Prerequisites

Operating System Requirements

The following Linux operating systems are supported:

• Officially supported: Red Hat Enterprise Linux (RHEL) 9+

• Compatible (expected to work): CentOS 9+, Alma Linux 9+

System requirements:

Ports:

The following defaults ports are used by various internal components. All ports are configurable.

• External traffic type: traffic entering the platform through the load balancer or directly.

• Internal traffic type: service-to-service traffic within the platform nodes.

• Network connectivity between cluster nodes (for multi-node installations), including any firewall rules to allow network traffic on specific ports listed above.

Firewall Rule Requirements

For a four-node cluster (frontend-1, frontend-2, backend-1, backend-2) with an upstream loadbalancer (LB), below is an example of needed ingress firewall rules between nodes.

Note: This assumes any intra-node (within VM) traffic is allowed. This does not include platform-egress firewall rules, e.g. outgoing traffic going to github, gitlab resources, etc., as those are dependent on end user asset scanning preferences.

All traffic is on TCP

Certificate Requirements

The platform uses mutual TLS (mTLS) for secure authenticated communication between all services and server components.

All required certificates can be generated and self-signed, or you can provide your own certificates. See Certificates Management Guide for detailed information and instructions on various options for certificates.

generate_cert.sh generates the following certificates:

It is recommended to run generate_cert.sh to see list of files and locations. Generate the certificates on the first node (backend-1), then copy them to the other nodes, following the filename conventions above.

Domain Name Requirements

A single external FQDN is required. The following format is recommended:

• agilesec.<external domain>

• For example: agilesec.dilithiumbank.com

Installation Steps for a 3-node Cluster

This section covers the steps for installing a three-node cluster with the following node types:

• Frontend 1

• Backend 1

• Backend 2

For instructions on installing an optional Scan node, refer to the HA guide.

The installation can be performed in two modes: Interactive (default) or Non-Interactive (for automated/CI-CD deployments).

In this version, install_analytics.sh introduces the install sub-command for performing fresh installations:

./install_analytics.sh install [OPTIONS]

For backward compatibility, running ./install_analytics.sh [OPTIONS] without the install sub-command continues to work as in previous versions. To see all available install options, run ./install_analytics.sh install --help.

Non-Interactive Mode

All key installation scripts support a --non-interactive (or -n) flag that skips confirmation prompts, enabling fully automated deployments.

Pre-requisites for Non-Interactive Install

The install_analytics.sh script reads the admin password from a .pass file when running in non-interactive mode. To set this up:

1. Copy the example file: cp .pass.example .pass

2. Edit .pass and set admin_password

3. Set secure permissions: chmod 600 .pass

IMPORTANT:

Kafka quorum during installation: When multiple backend nodes are installed in parallel, the Kafka KRaft quorum may go through re-elections as new nodes join. If Kafka stops running on any backend node during installation, manually restart it with ./scripts/manage.sh start kafka to help the quorum form and allow the installation to make progress.

Health check cron: As installation of each node completes, verify the health check cron job is active on all nodes (crontab -l). This cron automatically detects and restarts stopped services, which helps maintain Kafka quorum stability during normal operation.

Step 1: Prepare the Environment

Ensure you have the installer package and your nodes meet the minimum CPU and memory requirements.

For easier file transfers between nodes, we also recommend setting up SSH key-based password-less access from backend-1 to all other nodes, since certificates and other configuration files are generated on backend-1. We also recommend using the same directory names on each node when copying files between machines.

1. Download the installation zip archive from the Keyfactor download portal.

2. Extract the zip archive to your preferred location:

unzip -d <installer_directory> <installer_package>.zip
cd <installer_directory>

3. Ensure the installation script is executable:

cd <installer_directory>
chmod +x install_analytics.sh

4. Environment configuration is required before starting the installation. A sample configuration file for a multi-node installation is available at generate_envs/multi_node_config.conf. Edit this file based on your installation type. At a minimum, you must provide the private IP addresses of all your nodes (be-1,be-2,fe-1). The following additional settings can be overridden based on your environment:

[Optional] Using Port 443 (HTTPS Standard Port)

To run the platform on the standard HTTPS port (443) instead of the default 8443, set analytics_port=443 in generate_envs/multi_node_config.conf before running generate_envs.sh. Then proceed with the normal installation steps.

When using port 443, be aware of the following post-install behavior:

• Since port 443 is a privileged port (< 1024), HAProxy requires root privileges to bind to it. This applies to frontend nodes where HAProxy handles external ingress traffic.

• ./manage.sh start as a regular user will start all services except HAProxy.

• HAProxy must be started or restarted with sudo:

  BASH

  sudo ./manage.sh start haproxy

• However, stopping HAProxy does not require sudo:

  BASH

  ./manage.sh stop haproxy

• The platform URL simplifies to https://<analytics_hostname>.<analytics_domain> (no port number needed, since 443 is the default HTTPS port).

5. Generate config file for multi-node installation.

./generate_envs/generate_envs.sh -t multi-node

Tip: Add --non-interactive to skip overwrite prompts. See Non-Interactive Mode.

The following files will be generated:

• <installer_directory>/generate_envs/generated_envs/env.backend-1

• <installer_directory>/generate_envs/generated_envs/env.backend-2

• <installer_directory>/generate_envs/generated_envs/env.frontend-1

generate_envs.sh will copy the env.backend01 file to <installation_directory>/.env

6. Generate certificates

You can generate and self-sign all required certificates using <installer_directory>/certificates/generate_certs.sh. Alternately, you can use certificates issued by your own CA. For POCs and first-time installations, it is recommended you generate all certificates using generate_certs.sh.

Run the following command to generate and self-sign all required certificates. The .env file is required to generate certificates. By default, the script looks for the .env file under <installation directory>. This command must be run on backend-1.

The script populates the certificate files under the certificates directory and generates an archive named kf-agilesec.internal-certs.tgz, which you must copy to all other nodes. This archive also includes env.backend-2, env.backend-1, and env.frontend-1 in addition to the certificates.

cd certificates/
./generate_certs.sh

Tip: Add --non-interactive (or -n) to skip overwrite prompts. See Non-Interactive Mode.

[Optional] Using your own certificates

If you are using your own certificates, perform following steps:**

• Copy your CA cert chain to <installation_directory>/certificates/ca

• Copy server and client certificates to <installation_directory>/certificates/<analytics_internal_domain> .

• Certificate filenames must match those listed under Certificate Requirements.

7. Copy files to the other nodes from backend-1. For each node, copy kf-agilesec.internal-certs.tgz to <installer_directory>/certificates/.

scp kf-agilesec.internal-certs.tgz $BE-2_IP:<installer_directory>/certificates/
scp kf-agilesec.internal-certs.tgz $FE-1_IP:<installer_directory>/certificates/

Note: scp command is provided as an example. You may use any file transfer method suiting your environment.

Step 2: Install on Backend-1

1. Ensure you have either a DNS entry for <analytics_hostname>.<external domain> (recommended) pointing to frontend-1 IP address or an entry in your /etc/hosts for <analytics_hostname>.<external domain> pointing to frontend-1 IP address.

2. Make sure settings file <installer_directory>/.env is present.

3. Run sudo ./scripts/tune.sh -u <username> to update following:

   1. System settings:
      Sysctl settingRecommended valuevm.max_map_count262144fs.file-max65536

   2. Security settings in /etc/security/limits.conf for file descriptors and number of threads. This is needed by OpenSearch:
      Setting- nofile 65536- nproc 65536soft memlock unlimitedhard memlock unlimited

   3. /etc/hosts entries:
      <private ip> <frontend node 1 hostname>.<analytics_internal_domain>
<private ip> <backend node 1 hostname>.<analytics_internal_domain>
<private ip> <backend node 2 hostname>.<analytics_internal_domain>

   4. Install git binary

Alternately, you can perform the above steps manually

4. Run the following command to start the installation, then follow the prompts:

cd <installer_directory>
./install_analytics.sh install -u <user> -p <installation-dir>

Note 1: The previous functionality of using ./install_analytics.sh -u <user> -p <installation-dir> -v without the install sub-command is still supported.

Note 2: <installation-dir> is a new, separate directory where the installed files will reside.

Tip: Add --non-interactive to skip all prompts (requires .pass file). See Non-Interactive Mode.

To see additional install options, run ./install_analytics.sh install --help. If any required parameters are omitted, the script will prompt you to enter them interactively.

Step 3: Install on Backend-2

1. On Backend 2, follow these steps to unarchive kf-agilesec.internal-certs.tgz and copy .env file to <installer_directory>

cd <installer_directory>/certificates/
tar zxvf kf-agilesec.internal-certs.tgz
cp env.backend-2 ../.env

2. Follow steps 2.1, 2.2 and 2.3 from Step 2 for backend-2 instead of backend-1.

3. Run the following command to start the installation on backend-2, then follow the prompts:

cd <installer_directory>
./install_analytics.sh install -u <user> -p <installation-dir>

Step 4: Install on Frontend-1

1. On Frontend-1, follow these steps to unarchive kf-agilesec.internal-certs.tgz and copy .env file to <installer_directory>

cd <installer_directory>/certificates/
tar zxvf kf-agilesec.internal-certs.tgz
cp env.frontend-1 ../.env

2. Follow steps 2.1, 2.2 and 2.3 from Step 2 for frontend-1 instead of backend-1.

3. Run the following command to start the installation on frontend-1, then follow the prompts:

cd <installer_directory>
./install_analytics.sh install -u <user> -p <installation-dir>

Example: End-to-End Non-Interactive Multi-Node Install

# === On backend-1 ===
cd <installer_directory>

# Generate env configs for all nodes
./generate_envs/generate_envs.sh -t multi-node --non-interactive

# Generate certificates
cd certificates/
./generate_certs.sh --non-interactive
cd ..

# Prepare .pass file
cp .pass.example .pass
# Edit .pass and set admin_password
chmod 600 .pass

# System tuning and install
sudo ./scripts/tune.sh -u <user>
./install_analytics.sh install -u <user> -p <installation-dir> --non-interactive

# Copy certs archive to other nodes
scp certificates/kf-agilesec.internal-certs.tgz <BE-2_IP>:<installer_directory>/certificates/
scp certificates/kf-agilesec.internal-certs.tgz <FE-1_IP>:<installer_directory>/certificates/

# === On backend-2 ===
cd <installer_directory>/certificates/
tar zxvf kf-agilesec.internal-certs.tgz
cp env.backend-2 ../.env
cd ..
cp .pass.example .pass
# Edit .pass and set admin_password
chmod 600 .pass
sudo ./scripts/tune.sh -u <user>
./install_analytics.sh install -u <user> -p <installation-dir> --non-interactive

# === On frontend-1 ===
cd <installer_directory>/certificates/
tar zxvf kf-agilesec.internal-certs.tgz
cp env.frontend-1 ../.env
cd ..
cp .pass.example .pass
# Edit .pass and set admin_password
chmod 600 .pass
sudo ./scripts/tune.sh -u <user>
./install_analytics.sh install -u <user> -p <installation-dir> --non-interactive

At the end of the installation, the installer will display the following access details:

• Web UI access information:

  • Login URL

  • Admin username

  • Admin password (the password you provided during installation will not be displayed here)

• Ingestion service endpoint for the v3 unified sensor

• Ingestion endpoint for v2 sensors

Post-Installation Verification

Verify Service Health

Run ./scripts/manage.sh status on each node to check the status of all services. If any service shows Not running, try restarting it. See managing services for instructions on starting and restarting services.

On Backend 1 you should see 9 services in Running status:

$ ./manage.sh status
SERVICE                   DESCRIPTION                              STATUS                         UPTIME         
------------------------  ---------------------------------------- ------------------------------ ---------------
mongodb                   MongoDB Server                           Running (PID: 6839)            23h 8m 51s      
opensearch                OpenSearch Search Engine                 Running (PID: 6846)            23h 8m 52s      
kafka                     Kafka Server                             Running (PID: 243770)          23m 17s        
td-agent                  Fluentd Data Collector                   Running (2 instances)          23h 8m 31s      
scheduler                 Scheduler Microservice                   Running (PID: 7296)            23h 8m 37s      
analytics-manager         Analytics Manager Microservice           Running (PID: 7318)            23h 8m 37s      
ingestion                 Ingestion Microservice                   Running (PID: 7351)            23h 8m 37s      
indexing                  Indexing Microservice                    Running (PID: 7377)            23h 8m 37s      
sm                        Security Manager Microservice            Running (PID: 7404)            23h 8m 37s      
haproxy                   HAProxy Load Balancer                    Running (PID: 7475)            23h 8m 37s 

On Backend 2 you should see 9 services in Running status:

$ ./manage.sh status
SERVICE                   DESCRIPTION                              STATUS                         UPTIME         
------------------------  ---------------------------------------- ------------------------------ ---------------
mongodb                   MongoDB Server                           Running (PID: 6293)            23h 3m 35s       
opensearch                OpenSearch Search Engine                 Running (PID: 6300)            23h 3m 36s     
kafka                     Kafka Server                             Running (PID: 6722)            23h 3m 22s      
td-agent                  Fluentd Data Collector                   Running (2 instances)          23h 3m 22s        
scheduler                 Scheduler Microservice                   Running (PID: 6752)            23h 3m 22s   
analytics-manager         Analytics Manager Microservice           Running (PID: 6780)            23h 3m 23s      
ingestion                 Ingestion Microservice                   Running (PID: 6813)            23h 3m 23s    
indexing                  Indexing Microservice                    Running (PID: 6846)            23h 3m 24s 
sm                        Security Manager Microservice            Running (PID: 6863)            23h 3m 24s      
haproxy                   HAProxy Load Balancer                    Running (PID: 6946)            23h 3m 15s 

On Frontend 1 you should see 9 services in Running status:

$ ./manage.sh status
SERVICE                   DESCRIPTION                              STATUS                         UPTIME         
------------------------  ---------------------------------------- ------------------------------ ---------------
mongodb                   MongoDB Server                           Running (PID: 5394)            23h 1m 18s        
opensearch                OpenSearch Search Engine                 Running (PID: 5401)            23h 1m 17s      
kafka                     Kafka Server                             Running (PID: 5778)            23h 1m 17s   
sm                        Security Manager Microservice            Running (PID: 5789)            23h 1m 17s   
api                       Web API Microservice                     Running (PID: 32148)           23h 1m 17s  
webui                     Web UI Microservice                      Running (PID: 5864)            23h 1m 17s     
opensearch-dashboards     OpenSearch Dashboards                    Running (PID: 6530)            23h 1m 17s     
cbom                      CBOM Exporter Microservice               Running (PID: 31934)           23h 1m 17s 
haproxy                   HAProxy Load Balancer                    Running (port < 1000, started with root) 23h 1m 10s 

After installation completes, allow approximately 30 seconds for all services to fully initialize before accessing the UI. Attempting to log in immediately may result in a 503 Service Unavailable error. If this occurs, wait 30 seconds and refresh the page.

Access the Platform UI

To log in to the new Web UI, use the URL and login displayed at the end of install:

• Login URL: https://<analytics_hostname>.<analytics_domain>:<analytics_port>

• Username: admin@<analytics_domain>

• Password:

For example, using the default settings:

• Login URL: https://analytics.kf-agilesec.com:8443

• Username: admin@kf-agilesec.com

• Password: HelloWorld123456!

You will see a login screen like this:

After logging in, the Overview Dashboard should show 0 across all charts, as shown in the screenshot below:

Run a Smoke Test

Follow these steps to run a quick smoke test and confirm the platform is working:

Step 1: In the Web UI, go to Sensors -> Network Scan.

Step 2: On the network scan page, enter an HTTPS URL to scan (for example: https://www.google.com), then click Scan to start the scan.

Step 3: While the scan is running, you fill see a screen similar to the following:

Step 4: Once the scan is completed successfully, you will see a screen similar to following:

Step 5: At this point, the scan has completed and the pipeline is waiting for the policy execution to finish. Policy execution can take up to 45 seconds. Until policies run, all findings will show a Pending status under the Findings tab:

Step 6: Once policies have run, the Score column will show a risk score instead of Pending, as shown in the screenshot below:

Step 7: Shortly after policies run successfully, a backend process performs additional analysis on the findings. Once this process completes, you will see Successful statuses as shown in the screenshot below. This confirms the platform is working as expected.

Managing Services

After installation, you can manage services using the unified service manager script at ./scripts/manage.sh.

The manage.sh script provides a centralized way to manage all platform services:

cd /scripts ./manage.sh [action] [options] [services...]

Actions

Options

Available Services

If no specific services are specified, the action will be applied to all installed services.

Usage Examples

• To start all services:

./manage.sh start

• To start only OpenSearch with debug output:

./manage.sh start -d opensearch

• To start multiple specific services:

./manage.sh start opensearch opensearch-dashboards

The script automatically resolves dependencies, starting OpenSearch first (as it's a dependency for other services) before starting any dependent services.

• to stop all services:

# Stop all installed services
./manage.sh stop

• To stop only specific services:

./manage.sh stop haproxy indexing

The script stops services in reverse dependency order to ensure a clean shutdown.

• To Restart all installed services:

# Restart all installed services
./manage.sh restart

• To restart only OpenSearch:

./manage.sh restart opensearch

• To reload configuration:

./manage.sh reload haproxy

• To list status of all services:

./manage.sh status

• To check status of specific services:

./manage.sh status opensearch indexing

• To list all available services:

./manage.sh list

Note: If HAProxy is configured to use a privileged port (< 1000), you will need root privileges to start or stop it. The script will display the appropriate command to run with sudo.

Post Installation Configuration

SAML Setup

For detailed instructions on various SSO integration options see Authentication and Access Control documentation.

Only users with the Platform Admin privilege role for the organization can edit the SAML 2.0 configuration. To access the SAML setup page, do the following:

Step 1: Go to Settings -> Authentication Options.

Step 2: Turn on SAML 2.0 Single Sign-On

Step3: Open Settings for SAML 2.0 Configuration, then configure the SAML settings for your environment. Details for each configurable option are provided in the next section.

Configuration Options

Service Provider Information

Authorization Settings

Note: Only SHA256 signing algorithm is supported for now.

Identity Provider Configuration

Note: Required to select one of the options.

Custom Attribute Configuration

Role Mapping Configuration

Troubleshooting

If you encounter issues during installation or operation:

Installation Issues

• Check the console output for specific error messages.

• Verify all prerequisites are met.

• Ensure all certificate files are correctly placed and have proper permissions and filename.

• Check the .env file to make sure the following are correct:

  • private_ip

  • analytics_hostname

  • analytics_domain

  • analytics_port

  • cluster_frontend_node_ips

  • cluster_backend_node_ips

• Make sure analytics FQDN <analytics_hostname>.<analytics_domain> is reachable on <analytics_port>.

• Verify network connectivity between nodes in a multi-node installation.

Service Issues

• Check service logs in the <installation_path>/logs directory.

• Verify port availability using netstat or ss commands.

• Ensure proper certificate permissions and ownership.

• Check disk space with df -h.

• Verify memory availability with free -h.

Common Errors

• Certificate Issues: Ensure all certificate DNs and CNs match specs from certificates requirements section.

• Port Conflicts: Verify no other services are using the required ports.

• Permission Denied: Check file and directory permissions.

• Memory Errors: Verify you have sufficient memory available (minimum 8GB recommended).