.png){kind=logo}
Upgrade the AgileSec platform to ensure access to latest features and improvements.
Overview
When upgrading AgileSec for on-prem, all steps should be followed in the order described. If a step can be run in parallel across nodes, it is noted.
For Single-Node Installations:
Single-node installations consider the Single Node to be both Frontend and Backend. Perform all listed steps on the Single Node to upgrade. Files do not need to be copied to other non-existent nodes.
New option: systemd
3.6 introduces the ability to use systemd for managing services. This is optional. The default behavior remains binary service management as in 3.5.
systemd may be enabled anytime: either immediately after upgrading (after Step 6 below) or later if customers wish to do so.
See On-Prem Managing Services for instructions on enabling and managing services with systemd post-upgrade.
Upgrade Flow
Upgrades are intended to go sequentially, for example:
-
3.3 → 3.4 → 3.5 → 3.6
Skip-version upgrades may work but have not been tested.
Prerequisites
-
Running AgileSec node(s)
-
The latest AgileSec installer package extracted on all nodes
-
SSH access to all nodes
-
Sudo privileges on all nodes
-
rsyncinstalled on all nodes (used by the upgrade process to sync files).# Install rsync on RHEL sudo dnf check-update sudo dnf install -y rsync # Verify installation rsync --version
Path References
The following placeholders are used throughout this guide. Replace them with the actual paths for your environment.
|
Placeholder |
Description |
Example |
|---|---|---|
|
|
Directory where the old AgileSec is currently installed (and where the new AgileSec version update will be installed to). |
|
|
|
Directory where the old installer package was extracted. |
|
|
|
Directory where the new installer package is extracted. |
|
IMPORTANT REQUIREMENT: Keep the previous installer directory prev_installer_dir until at least the next time you upgrade. This will ensure customers have the ability to roll back if needed.
Set Environment Variables
Set the path references to environment variables to be able to copy+paste commands in this guide.
export AGILESEC_INSTALL_DIR=<agilesec_install_dir>
export PREV_INSTALLER_DIR=<prev_installer_dir>
Pre-Upgrade Checklist (all nodes)
Before starting the upgrade, verify the current installation is healthy and all services are running on all nodes:
# Run on each node:
cd $AGILESEC_INSTALL_DIR
./scripts/manage.sh status
All nodes, Frontend and Backend, should show all services running.
Verify rsync is installed on all nodes:
rsync --version
If not installed, install with:
sudo dnf install -y rsync
Upgrade Steps
IMPORTANT: Steps are listed in the order they should be executed. If a step can be run in parallel across nodes, it is noted.
Step 1: Disable the health check cron job (all nodes)
The health check cron job automatically restarts stopped services. If you have health check cron enabled, it must be disabled before stopping services to prevent restarting services during the upgrade.
Run on each node (can be done in parallel) to disable the health check entry in cron:
crontab -l | sed 's|^\(\*/5 .*health_check\.sh\)$|#DISABLED# \1|' | crontab -
Verify on each node:
crontab -l
Expected output:
#DISABLED# */5 * * * * /bin/bash $AGILESEC_INSTALL_DIR/scripts/health_check.sh
Step 2: Backup existing installation directory (all nodes)
Rollback can only occur if the original installation is backed up. Backing up is technically optional, but if you do not perform a backup, you will not be able to roll back the upgrade if post-upgrade verification fails.
Stop all services on each node prior to backing up the original installation directory (can be done in parallel):
cd $AGILESEC_INSTALL_DIR
./scripts/manage.sh stop
All services on each node should show stopped successfully.
Backup the original installation directory on each node before moving forward. Following rsync command is recommended:
TIMESTAMP=$(date +%Y%m%d-%H%M%S)
sudo rsync --sparse -aAX --numeric-ids --info=progress2 $AGILESEC_INSTALL_DIR $AGILESEC_INSTALL_DIR-$TIMESTAMP.backup
Compare size of current agilesec_install_dir with new backup directory to make sure they are same.
du -hs --apparent-size $AGILESEC_INSTALL_DIR
du -hs --apparent-size $AGILESEC_INSTALL_DIR-$TIMESTAMP.backup
Note: rsync is slower but more reliable compared to cp -r.
Step 3: Generate environment configuration (backend-1 only)
The new installer will merge your existing old multi-node configuration into new version environment files for all nodes.
The new installer will merge your existing old multi-node configuration into new version environment files for all nodes.
Extract new installer package and set environment variable for new installer directory:
unzip <new-installer-package>.zip
export NEW_INSTALLER_DIR=<path-to-new-unzipped-installer-package>
For multi-node, run on backend-1 only:
cd $NEW_INSTALLER_DIR
./generate_envs/generate_envs.sh -t multi-node --old-config $PREV_INSTALLER_DIR/generate_envs/multi_node_config.conf
For single-node, run:
cd $NEW_INSTALLER_DIR
./generate_envs/generate_envs.sh -t single-node --old-config $PREV_INSTALLER_DIR/generate_envs/single_node_config.conf
Note: The --old-config flag tells the generator to:
-
Preserve customer-specific values from the old version (organization name, domain, ports, secrets, IPs, certificates, etc.)
-
Skip version/URL fields (uses new version’s values for sensor URLs, API versions, etc.)
-
Add new version's settings with their defaults (e.g., indexing service config, Kafka replication settings)
This generates .env files for all nodes:
-
For single nodes:
-
$NEW_INSTALLER_DIR/generate_envs/generated_envs/env.single_node
-
-
For multi-node clusters:
-
$NEW_INSTALLER_DIR/generate_envs/generated_envs/env.backend-1 -
.../env.backend-2 -
.../env.frontend-1 -
etc.
-
env.backend-1 or env.single_node is automatically copied to $NEW_INSTALLER_DIR/.env.
For multi-node, distribute env files to other nodes from backend-1.
Copy env file for backend-2:
scp $NEW_INSTALLER_DIR/generate_envs/generated_envs/env.backend-2 \
<user@backend-2-ip>:$NEW_INSTALLER_DIR/.env
Copy env file for frontend-1:
scp $NEW_INSTALLER_DIR/generate_envs/generated_envs/env.frontend-1 \
<user@frontend-1-ip>:$NEW_INSTALLER_DIR/.env
Step 4: Run upgrade precheck (all nodes)
Run on each node (can be done in parallel):
cd $NEW_INSTALLER_DIR
./install_analytics.sh upgrade-precheck -p $AGILESEC_INSTALL_DIR
Expected output on each node:
=====================================
Pre-requisite Checks Summary
=====================================
Checks Passed: 14
Warnings: 0
Checks Failed: 0
The upgrade precheck should show 0 failures on all nodes.
The upgrade precheck also generates $NEW_INSTALLER_DIR/versions.json on each node.
Note: If any checks fail on any node, review the errors and resolve before proceeding.
Step 5: Run upgrade plan and review (all nodes)
Review the upgrade plan on each node. This is a dry run to show all changes without modifying anything. This can be done in parallel on each node.
cd $NEW_INSTALLER_DIR
./install_analytics.sh upgrade-plan -p $AGILESEC_INSTALL_DIR \
--previous-installer-dir $PREV_INSTALLER_DIR
Scan Node Note: If you are doing upgrade-planon scan-node, you must add the -r scan flag to the upgrade-plan command.
Review the planned changes carefully on each node, especially for files marked as customer changed.
IMPORTANT: REVIEW SUMMARY REPORT FOR MANUAL CONFIG CHANGES
Any changes manually defined outside standard configuration files, such as multi_node_config.conf or single_node_config.conf, will be overwritten during the upgrade.
upgrade-plan --previous-installer-dir compares the upgrade plan to the previous installer directory and generates a safety summary report at $NEW_INSTALLER_DIR/tmp/config-review/<00#_timestamp>/UPGRADE_SUMMARY.txt. The summary report location will also be output in the console.
REVIEW THIS SAFETY REPORT. If the safety summary report says ACTION REQUIRED, review the listed diff files, note them, and reapply any custom settings again after the upgrade if necessary.
Safety Summary Report Classifications
The safety summary report produces the following change classifications based on the configured upgrade:
-
product_changes_only: Informational only. No unexpected, manual customer changes were found. Customer configurations will not be affected by product upgrade changes. No customer action is normally required.
-
customer_changes_only: Customer made manual changes outside standard configuration files. These changed files will be overwritten by upgrade to previous default. Manual review is required.
-
customer_and_product_changes: Customer made manual changes outside standard configuration files. The product upgrade will also change these files. Manual review is required.
-
unknown_baseline: The upgrade installer cannot determine the previous baseline, possibly due to a missing or incorrect--previous-installer-dir. Manual review or re-runningupgrade plan --previous-installer-dir <old-installer-dir>may be required.
Example Summary Report
The following is an example UPGRADE_SUMMARY.txt with product_changes_only and customer_and_product_changes:
===================================================================================
Keyfactor AgileSec Analytics Platform -- Upgrade Config Review Summary
===================================================================================
Generated : 2026-07-01 16:57:47 UTC
Status : ACTION REQUIRED
Path : /home/ec2-user/351
===================================================================================
Manual review required: 1 file
ACTION REQUIRED: Configuration files require manual merge
Modified configuration files were detected during upgrade.
The installer applied default settings and preserved previous
versions alongside diff files for comparison.
You must manually merge any previous customer settings before going live.
For each file below, three options are available:
a) Review and edit manually using the saved pre-upgrade file as reference.
b) Review your changes only:
diff <saved-pre-upgrade-file> <upgraded-file>
c) Apply your changes using the provided customer diff:
patch <upgraded-file> <customer-diff>
Then verify the result:
diff <saved-pre-upgrade-file> <upgraded-file>
===================================================================================
Following files require manual merge
===================================================================================
scheduler [scheduler] customer_and_product_changes
Reason : customer changes detected; product template also changed
Upgraded file : /home/ec2-user/351/config_envs/scheduler
Pre-upgrade file: /home/ec2-user/kf_agilesec_unified_installer_rhel9_3.6.0-rc2/tmp/config-review/20260701165742-17407/current/scheduler
Customer changes to re-apply: /home/ec2-user/kf_agilesec_unified_installer_rhel9_3.6.0-rc2/tmp/config-review/20260701165742-17407/diffs/scheduler.a02ba163f3a0.customer.diff
-----------------------------------------------------------------------------------
Product-only changes: 11 files
- analytics-manager [analytics_manager] product_changes_only
- indexing [indexing] product_changes_only
- ingestion [ingestion] product_changes_only
- log4j.properties [kafka] product_changes_only
- kafka [kafka] product_changes_only
- log4j2.properties [opensearch] product_changes_only
- config.yml [opensearch] product_changes_only
- default_tenants.yml [opensearch] product_changes_only
- haproxy.cfg [proxy] product_changes_only
- migrate.json [scheduler] product_changes_only
- sm [sm] product_changes_only
===================================================================================
Step 6: Run the upgrade (all nodes)
After reviewing the upgrade plan and documenting any manual changes, if applicable, proceed to upgrade. Run on each node (can be done in parallel):
cd $NEW_INSTALLER_DIR
./install_analytics.sh upgrade -p $AGILESEC_INSTALL_DIR \
--previous-installer-dir $PREV_INSTALLER_DIR \
--non-interactive --no-restart --no-validate --ack-config-overwrite
The --ack-config-overwrite flag acknowledges you are okay with overwriting any changes in the upgrade plan summary report.
If the upgrade detects a manual configuration overwrite risk, another safety summary report location ($AGILESEC_INSTALL_DIR/backups/<00#_timestamp>/config-review/UPGRADE_SUMMARY.txt) will be output in the console.
Scan Node Note: If you are upgrading a scan node, you must add the -r scan flag to the upgrade command.
|
install_analytics.sh upgrade Flag |
Purpose |
|---|---|
|
|
Skip confirmation prompts. |
|
|
Do not auto-start services after upgrade. |
|
|
Skip post-upgrade validation (services are stopped). |
|
|
Designate a new file path for vars file. By default, |
|
|
Path for the AgileSec install location. Should match the previous AgileSec install location |
|
|
Acknowledge non-standard, manual changes declared in UPGRADE_SUMMARY.txt will be overwritten. If this acknowledgement flag is not included, upgrades with non-standard, manual changes will be blocked. |
|
|
Path to the previous installer directory for comparison to find manual configuration changes for the safety report. |
Expected final output on each node:
...
[SUCCESS] All tiers completed successfully
[INFO] Running post-upgrade hook
[SUCCESS] post-upgrade hook completed successfully
[INFO] Installation version updated to: <new version>
[INFO] Upgrade history recorded: upgrade-20260329-045111
[SUCCESS] Upgrade completed successfully
Note: A backup of configs and binaries is created at <agilesec_install_dir>/backups/<00#_timestamp>/ on each node before changes are applied.
Note: Generated update log files can be found at <new_installer_dir>/logs.
Step 7: Start all services except ingestion, analytics-manager, scheduler, api, webui (all nodes - sequential)
IMPORTANT FOR MULTI-NODE:
Start nodes in order: backend-1 first, then backend-2, then additional Backends sequentially. After starting each Backend node, start frontend-1 and additional Frontend nodes. Frontend nodes may be started in parallel. backend-1 must be running before other nodes start, as it hosts the MongoDB primary and initializes cluster state.
On backend-1:
cd $AGILESEC_INSTALL_DIR
./scripts/manage.sh start --exclude ingestion,analytics-manager,scheduler
./scripts/manage.sh status
On backend-2:
cd $AGILESEC_INSTALL_DIR
./scripts/manage.sh start --exclude ingestion,analytics-manager,scheduler
./scripts/manage.sh status
On frontend-1:
cd $AGILESEC_INSTALL_DIR
./scripts/manage.sh start --exclude api,webui
./scripts/manage.sh status
Note: If HAProxy is configured to run on a privileged port (e.g., 443), you will need sudo to start it separately: sudo ./scripts/manage.sh start haproxy.
For Single Node:
cd $AGILESEC_INSTALL_DIR
./scripts/manage.sh start --exclude ingestion,analytics-manager,scheduler,api,webui
./scripts/manage.sh status
Step 8: Run Metadata Migration (frontend-1 ONLY)
Run on frontend-1 only. MongoDB, OpenSearch, and Kafka must be running on all nodes before running migrate.
cd $NEW_INSTALLER_DIR
./install_analytics.sh migrate -p $AGILESEC_INSTALL_DIR --apply-security -v
Expected output:
...
[SUCCESS] config.yml applied successfully.
[SUCCESS] OpenSearch security configuration applied successfully.
...
> npm run platform-data-init && npm run migrate-mongo:up
...
> node ./src/platform/platform.data.init.js
...
> migrate-mongo up
...
[SUCCESS] Database migration completed successfully.
...
[SUCCESS] API migration script completed successfully.
...
[SUCCESS] v3 Kafka topics created successfully
[SUCCESS] Kafka topics created successfully.
[SUCCESS] Migration completed successfully.
Note: If you see the following messages about Kafka service not reachable, please start Kafka again on all Backend Nodes.
2026-03-31 15:24:38 [INFO] Checking broker backend-1.kf-agilesec.internal:9092...
2026-03-31 15:24:38 [WARN] Kafka broker at backend-1.kf-agilesec.internal:9092 is not reachable (attempt 1).
2026-03-31 15:24:48 [INFO] Waiting 10s for broker to start (attempt 1/10)... ( 0s remaining)
2026-03-31 15:25:03 [WARN] Kafka broker at backend-1.kf-agilesec.internal:9092 is not reachable (attempt 2).
2026-03-31 15:25:14 [INFO] Waiting 10s for broker to start (attempt 2/10)... ( 0s remaining)
2026-03-31 15:25:29 [WARN] Kafka broker at backend-1.kf-agilesec.internal:9092 is not reachable (attempt 3).
2026-03-31 15:25:39 [INFO] Waiting 10s for broker to start (attempt 3/10)... ( 0s remaining)
2026-03-31 15:25:47 [INFO] Kafka broker at backend-1.kf-agilesec.internal:9092 is reachable and mTLS verified.
2026-03-31 15:26:05 [INFO] Checking broker frontend-1.kf-agilesec.internal:9092...
2026-03-31 15:26:05 [WARN] Kafka broker at frontend-1.kf-agilesec.internal:9092 is not reachable (attempt 1)
Step 9: Start all services on all nodes
systemd Note: 3.6 introduces the ability to enable systemd for services management. systemd is optional. To enable systemd prior to starting services, see On-Prem Managing Services.
cd $AGILESEC_INSTALL_DIR
./scripts/manage.sh start --enable-healthcheck
./scripts/manage.sh status
systemd Note: --enable-healthcheck flag is not required when systemd is enabled.
If any service shows Not running, start it individually:
./scripts/manage.sh start <service-name>
The --enable-healthcheck flag re-enables the health check cron job disabled previously. To confirm:
crontab -l
# Expected Output
*/5 * * * * /bin/bash $AGILESEC_INSTALL_DIR/scripts/health_check.sh
Step 10: Upgrade Enrichment Profile Policy to New Version
-
Login to the AgileSec Platform as a Platform Admin user.
Tip: The default admin user admin@org_domain created during platform installation is a Platform Admin.
-
On Left menu, under Platform Management, navigate to: Analytics Configuration -> Profiles.
-
Click on the Create New button.
-
Fill in the form fields:
-
Profile Name: AgileSec Enrichment Policies
v<new version> -
Description: Default Analytics app policies version
<new version> -
Binary Location: policy.bin
-
Execution Type: Scheduled
-
Schedule Frequency: Hourly
-
-
Click Upload the Policy Zip file, then attach the package:
<new version>-policy.zip(provided as an upgrade artifact). -
Click Save to create the profile.
-
Navigate to: Analysis -> App Policies.
-
Locate the newly created profile AgileSec Enrichment Policies
v<new version>. -
Under the vertical 3-Ellipses, Click Action -> Sync to synchronize the policy.
-
After synchronization completes, under the vertical 3-Ellipses, click Action -> Active to make it active.
Post-Upgrade Verification
Step 11: Validate the upgrade
-
Verify all services on all nodes are running:
# On each node:
./scripts/manage.sh status
-
Login to the Web UI at
https://<analytics_hostname>.<analytics_domain>:<analytics_port>using your existing admin credentials. -
Verify existing data is preserved - confirm scan configurations from before the upgrade are still present in the UI.
-
Run a new scan - perform a Network Scan to validate upgraded platform is functioning correctly:
-
Go to Sensors -> Network Scan.
-
Enter an HTTPS URL to scan (e.g.,
https://www.google.com). -
Click Scan and wait for it to complete.
-
Verify the scan completes successfully with findings and risk scores.
-
-
Check the Advanced Dashboard - navigate to the OpenSearch Dashboards and confirm they are loading correctly.
If all of the above checks pass, the upgrade from the new version is complete.
Rollback Procedure
If post-upgrade verification reveals issues, use this procedure to restore the old version’s installation from the backups created in Step 2.
Note: This procedure requires the backup ($AGILESEC_INSTALL_DIR-<date-time-stamp>.backup) to have been previously created on all nodes.
-
Disable the health check cron job (all nodes). Run on each node (can be done in parallel):
crontab -l | sed 's|^\(\*/5 .*health_check\.sh\)$|#DISABLED# \1|' | crontab -
-
Stop all services (all nodes). Run on each node (can be done in parallel):
cd $AGILESEC_INSTALL_DIR
./scripts/manage.sh stop
Note: If HAProxy is configured to run on a privileged port (e.g., 443), you will need sudo to stop it: sudo ./scripts/manage.sh stop haproxy
-
Restore the backup on each node (can be done in parallel). This preserves the new version’s state for investigation and restores the old version’s backup:
# Copy the upgraded state for investigation and debugging
mv $AGILESEC_INSTALL_DIR $AGILESEC_INSTALL_DIR-<new version>-upgrade-state
# Restore the old version's backup
rsync --sparse -aAX --numeric-ids --info=progress2 $AGILESEC_INSTALL_DIR-$TIMESTAMP.backup $AGILESEC_INSTALL_DIR
Note: If disk-space is a concern, use mv instead of rsync to restore the backup:
mv $AGILESEC_INSTALL_DIR-$TIMESTAMP.backup $AGILESEC_INSTALL_DIR
-
Start services (all nodes - sequential)
IMPORTANT
Start nodes in order: backend-1 first, then backend-2, then additional Backends sequentially. After starting each Backend node, start frontend-1 and additional Frontend nodes. Frontend nodes may be started in parallel. backend-1 must be running before other nodes start, as it hosts the MongoDB primary and initializes cluster state.
On backend-1:
cd $AGILESEC_INSTALL_DIR
./scripts/manage.sh start
./scripts/manage.sh status
Note: Wait for all services on backend-1 to show Running before proceeding.
On backend-2:
cd $AGILESEC_INSTALL_DIR
./scripts/manage.sh start
./scripts/manage.sh status
On frontend-1:
cd $AGILESEC_INSTALL_DIR
./scripts/manage.sh start
./scripts/manage.sh status
Note: If HAProxy is configured to run on a privileged port (e.g., 443), you will need sudo access to start it.
-
Re-enable health check cron (all nodes)
Run on each node (can be done in parallel):
crontab -l | sed 's|^#DISABLED# \(.*health_check\.sh\)$|\1|' | crontab -
Verify on each node:
crontab -l
All services should now be running on the old version. Verify by logging into the Web UI and confirming the platform is operational.
Post-Upgrade Validation
See AgileSec On-Prem Validation Checklist for thorough post-upgrade validation checks.