Skip to main content
Skip table of contents

On-Prem Guide to Upgrade to v3.5 (Multi-Node)

These instructions apply to all upgrade paths from v3.4.x to v3.5.1. The examples in these instructions upgrade from v3.4 to v.3.5.0. However, the same steps apply for all the following upgrade paths:

  • v.3.4.x >> v3.5.0

  • v.3.4.x >> v3.5.1

  • v3.5.0 >> v3.5.1

Simply substitute the version-specific paths and installer package for your upgrade path in the instructions.

Prerequisites

  • A running AgileSec v3.4 multi-node installation with at least 2 backends and 1 frontend. This guide covers the example architecture of two backends and one frontend.

  • The v3.5.0 installer package extracted on all nodes.

  • SSH access to all nodes with sudo privileges.

  • rsync installed on all nodes (used by the upgrade process to sync files).

Path References

The following placeholders are used throughout this guide. Replace them with the actual paths for your environment.

Placeholder

Description

Example

<v3.4-installation-dir>

Directory where v3.4 is currently installed

/home/ec2-user/agilesec-341

<v3.4-installer-dir>

Directory where the v3.4 installer package was extracted

/home/ec2-user/installer_rhel9_3.4

<v3.5.0-installer-dir>

Directory where the v3.5.0 installer package is extracted

/home/ec2-user/installer_rhel9_3.5.0

Pre-Upgrade Checklist

Before starting the upgrade, verify the current installation is healthy by running the following on all nodes:

BASH 
# Run on each node:
./scripts/manage.sh status

  • Backend nodes should show all services running.

  • Frontend node should show all services running.

Verify rsync is installed on all nodes:

BASH 
rsync --version

If not installed, install with:

BASH 
sudo dnf install -y rsync

Upgrade Steps

IMPORTANT: Steps are listed in the order they should be executed. If a step can be run in parallel across nodes, it is noted.

Step 1: Disable the health check cron job (all nodes)

The health check cron job automatically restarts stopped services. If you have health check cron enabled, it must be disabled before stopping services to prevent restarting services during the upgrade.

Run on each node (can be done in parallel):

BASH 
crontab -l | sed 's|^\(\*/5 .*health_check\.sh\)$|#DISABLED# \1|' | crontab -

Verify on each node:

BASH 
crontab -l

Expected output:

JAVA 
#DISABLED# */5 * * * * /bin/bash <v3.4-installation-dir>/scripts/health_check.sh

Step 2: Stop all services and perform backup of installation directory on all nodes

Run on each node (can be done in parallel):

BASH 
cd <v3.4-installation-dir>
./scripts/manage.sh stop

All services on each node should show stopped successfully.

For easier rollback, if your machine has extra storage, backup the installation directory before moving forward.

BASH 
du -hs <v3.4-installation-dir>
cp -r <v3.4-installation-dir> <v3.4-installation-dir>.bak

Step 3: Generate environment configuration (backend-1 only)

Note: Pre-Upgrade Configuration Change: org_domain with Subdomains

If you are using subdomains (e.g., company.kf-agilesec.com) in the analytics_domain config value in <v3.4-installer-dir>/generate_envs/multi_node_config.conf, then you will need to make a change to the file <v3.4-installer-dir>/generate_envs/multi_node_config.conf before upgrading.

Change the value of org_domain="${analytics_domain}" to the base domain. For example, if your analytics_domain is company.kf-agilesec.com, change

CODE 
org_domain="${analytics_domain}"

to

CODE 
org_domain="kf-agilesec.com"

This ensures the migration tool, OpenSearch roles, and Kafka topics use the correct base domain matching your existing indices and API expectations.

Run on backend-1 only. The v3.5.0 installer will merge your existing v3.4 multi-node configuration into new v3.5.0 environment files for all nodes.

BASH 
unzip -d <v3.5.0-installer-dir> <v3.5.0-installer-package>.zip
cd <v3.5.0-installer-dir>
./generate_envs/generate_envs.sh -t multi-node --old-config <v3.4-installer-dir>/generate_envs/multi_node_config.conf

The --old-config flag tells the generator to:

  • Preserve customer-specific values from v3.4 (organization name, domain, ports, secrets, IPs, certificates, etc.)

  • Skip version/URL fields (uses new v3.5.0 values for sensor URLs, API versions, etc.)

  • Add new v3.5.0 settings with their defaults (e.g., indexing service config, Kafka replication settings)

This generates .env files for all nodes:

  • <v3.5.0-installer-dir>/generate_envs/generated_envs/env.backend-1

  • <v3.5.0-installer-dir>/generate_envs/generated_envs/env.backend-2

  • <v3.5.0-installer-dir>/generate_envs/generated_envs/env.frontend-1

The env.backend-1 is automatically copied to <v3.5.0-installer-dir>/.env.

Distribute env files to other nodes from backend-1:

BASH 
scp <v3.5.0-installer-dir>/generate_envs/generated_envs/env.backend-2 \
  <backend-2-ip>:<v3.5.0-installer-dir>/.env

scp <v3.5.0-installer-dir>/generate_envs/generated_envs/env.frontend-1 \
  <frontend-1-ip>:<v3.5.0-installer-dir>/.env

Step 4: Run upgrade precheck (all nodes)

Run on each node (can be done in parallel):

BASH 
cd <v3.5.0-installer-dir>
./install_analytics.sh upgrade-precheck -p <v3.4-installation-dir> \
  --vars <v3.4-installation-dir>/scripts/.vars

Expected output on each node:

JAVA 
=====================================
Pre-requisite Checks Summary
=====================================
Checks Passed:   13
Warnings:        1
Checks Failed:   0

The precheck should show 0 failures on all nodes. A warning about "State file missing (reconstruction will be attempted)" is expected when upgrading from v3.4 and can be safely ignored.

This step also generates <v3.5.0-installer-dir>/versions.json on each node.

Note: If any checks fail on any node, review the errors and resolve before proceeding.

Step 5: Run upgrade plan (all nodes)

Note: If you are doing upgrade-planon scan-node, please add an extra flag -r scanto the upgrade-plan command. Otherwise, continue as normal with the following command.

Review the upgrade plan on each node. This is a dry run to show all changes without modifying anything.

BASH 
cd <v3.5.0-installer-dir>
./install_analytics.sh upgrade-plan -p <v3.4-installation-dir> \
  --vars <v3.4-installation-dir>/scripts/.vars --force

The --force flag is required for legacy installations (v3.4) without a state file.

Review the diffs carefully on each node, especially for files marked as USER EDITED.

Step 6: Run the upgrade (all nodes)

Run on each node (can be done in parallel):

BASH 
cd <v3.5.0-installer-dir>
./install_analytics.sh upgrade -p <v3.4-installation-dir> \
  --vars <v3.4-installation-dir>/scripts/.vars \
  --non-interactive --force --no-restart --no-validate

Flag

Purpose

--non-interactive

Skip confirmation prompts

--force

Required for legacy v3.4 installations without state file

--no-restart

Do not auto-start services after upgrade

--no-validate

Skip post-upgrade validation (services are stopped)

The upgrade proceeds in tiers on each node:

JAVA 
Tier 1 (Dependencies):   nodejs, openjdk
Tier 2 (Databases):      kafka, mongodb, opensearch
Tier 3 (Microservices):  varies by node role (backend vs frontend)
Tier 4 (Proxy):          haproxy

Expected final output on each node:

JAVA 
...
[SUCCESS] All tiers completed successfully
[INFO] Running post-upgrade hook
[SUCCESS] post-upgrade hook completed successfully
[INFO] Installation version updated to: 3.5.0
[INFO] Upgrade history recorded: upgrade-20260329-045111
[SUCCESS] Upgrade completed successfully

A backup is created at <v3.4-installation-dir>/backups/<timestamp>/ on each node before changes are applied.

Step 7: Start all services except ingestion, analytics-manager, scheduler, api, webui (all nodes - sequential)

IMPORTANT: Start nodes in order: backend-1 first, then backend-2, then frontend-1. Backend-1 must be running before other nodes start, as it hosts the MongoDB primary and initializes cluster state.

On backend-1:

BASH 
cd <v3.4-installation-dir>
./scripts/manage.sh start --exclude ingestion,analytics-manager,scheduler,api,webui
./scripts/manage.sh status

On backend-2:

BASH 
cd <v3.4-installation-dir>
./scripts/manage.sh start --exclude ingestion,analytics-manager,scheduler,api,webui
./scripts/manage.sh status

On frontend-1:

BASH 
cd <v3.4-installation-dir>
./scripts/manage.sh start --exclude ingestion,analytics-manager,scheduler,api,webui
./scripts/manage.sh status

Note: If HAProxy is configured to run on a privileged port (e.g., 443), you will need sudo to start it: sudo ./scripts/manage.sh start haproxy

Step 8: Run Metadata Migration (frontend-1 ONLY)

Run on frontend-1 only. MongoDB, OpenSearch, and Kafka must be running on all nodes before running migrate.

BASH 
cd <v3.5.0-installer-dir>
./install_analytics.sh migrate -p <v3.4-installation-dir> --apply-security -v

Expected output:

JAVA 
...
2026-03-29 05:08:27 [SUCCESS] config.yml applied successfully.
2026-03-29 05:08:29 [SUCCESS] roles.yml applied successfully.
2026-03-29 05:08:31 [SUCCESS] roles_mapping.yml applied successfully.
2026-03-29 05:08:31 [SUCCESS] OpenSearch security configuration applied successfully.
...
> npm run platform-data-init && npm run migrate-mongo:up
...
> node ./src/platform/platform.data.init.js
...
> migrate-mongo up
...
[SUCCESS] Database migration completed successfully.
...
[SUCCESS] API migration script completed successfully.
...
[SUCCESS] v3 Kafka topics created successfully
[SUCCESS] Kafka topics created successfully.
[SUCCESS] Migration completed successfully.

Note: If you see the following messages about Kafka service not reachable, please start Kafka again on backend-1 and backend-2:

BASH 
2026-03-31 15:24:38 [INFO] Checking broker backend-1.kf-agilesec.internal:9092...
2026-03-31 15:24:38 [WARN] Kafka broker at backend-1.kf-agilesec.internal:9092 is not reachable (attempt 1).
2026-03-31 15:24:48 [INFO] Waiting 10s for broker to start (attempt 1/10)... (  0s remaining)
2026-03-31 15:25:03 [WARN] Kafka broker at backend-1.kf-agilesec.internal:9092 is not reachable (attempt 2).
2026-03-31 15:25:14 [INFO] Waiting 10s for broker to start (attempt 2/10)... (  0s remaining)
2026-03-31 15:25:29 [WARN] Kafka broker at backend-1.kf-agilesec.internal:9092 is not reachable (attempt 3).
2026-03-31 15:25:39 [INFO] Waiting 10s for broker to start (attempt 3/10)... (  0s remaining)
2026-03-31 15:25:47 [INFO] Kafka broker at backend-1.kf-agilesec.internal:9092 is reachable and mTLS verified.
2026-03-31 15:26:05 [INFO] Checking broker frontend-1.kf-agilesec.internal:9092...
2026-03-31 15:26:05 [WARN] Kafka broker at frontend-1.kf-agilesec.internal:9092 is not reachable (attempt 1)

Step 9: Start all services on all nodes

BASH 
cd <v3.4-installation-dir>
./scripts/manage.sh start --enable-healthcheck
./scripts/manage.sh status

The --enable-healthcheck flag re-enables the health check cron job disabled in On-Prem Upgrade Guide: v3.4 to v3.5.0 (Multi-Node).

If any service shows Not running, start it individually:

BASH 
./scripts/manage.sh start <service-name>

Step 10: Upgrade Enrichment Profile Policy to Version 3.5.0

  1. Login to the AgileSec Platform as the default admin user.

  2. On Left menu, navigate to: Platform Management -> Analytics Configuration -> Profiles

  3. Click on the Create New button.

  4. Fill in the form fields:

    • Profile Name: AgileSec Enrichment Policies v3.5.0

    • Description: Default Analytics app policies version 3.5.0

    • Binary Location: policy.bin

    • Execution Type: Scheduled

    • Schedule Frequency: Hourly

  5. Click Upload the Policy Zip file, then attach the package: 3.5.0-policy.zip (provided to you as an upgrade artifact)

  6. Click Save to create the profile.

  7. Navigate to: Analytics -> App Policies

  8. Locate the newly created profile AgileSec Enrichment Policies v3.5.0.

  9. Under the vertical 3-Ellipses, Click Action -> Sync to synchronize the policy.

  10. After synchronization completes, under the vertical 3-Ellipses, click Action -> Active to make it active.

Post-Upgrade Verification

Step 11: Validate the upgrade

  1. Verify services on all nodes are running:

BASH 
# On each node:
./scripts/manage.sh status

  1. Login to the Web UI at https://<analytics_hostname>.<analytics_domain>:<analytics_port> using your existing admin credentials.

  2. Verify existing data is preserved - confirm scan configurations from before the upgrade are still present in the UI.

  3. Run a new scan - perform a Network Scan to validate upgraded platform is functioning correctly:

    • Go to Sensors -> Network Scan.

    • Enter an HTTPS URL to scan (e.g., https://www.google.com).

    • Click Scan and wait for it to complete.

    • Verify the scan completes successfully with findings and risk scores.

  4. Check the Advanced Dashboard - navigate to the OpenSearch Dashboards and confirm they are loading correctly.

If all of the above checks pass, the upgrade from v3.4 to v3.5.0 is complete.

Rollback Procedure

If post-upgrade verification reveals issues, use this procedure to restore the v3.4 installation from the backups created in On-Prem Upgrade Guide: v3.4 to v3.5.0 (Multi-Node).

Note: This procedure requires the backup (<v3.4-installation-dir>.bak) was created on all nodes during On-Prem Upgrade Guide: v3.4 to v3.5.0 (Multi-Node).

  1. Disable the health check cron job (all nodes). Run on each node (can be done in parallel):

BASH 
crontab -l | sed 's|^\(\*/5 .*health_check\.sh\)$|#DISABLED# \1|' | crontab -

  1. Stop all services (all nodes). Run on each node (can be done in parallel):

BASH 
cd <v3.4-installation-dir>
./scripts/manage.sh stop

Note: If HAProxy is configured to run on a privileged port (e.g., 443), you will need sudo to stop it: sudo ./scripts/manage.sh stop haproxy

  1. Restore the backup (all nodes). Run on each node (can be done in parallel). This preserves the v3.5.0 state for investigation and restores the v3.4 backup:

BASH 
mv <v3.4-installation-dir> <v3.4-installation-dir>-3.5.0-upgrade-state
cp -r <v3.4-installation-dir>.bak <v3.4-installation-dir>

  1. Start services (all nodes - sequential)

IMPORTANT: Start nodes in order: backend-1 first, then backend-2, then frontend-1. backend-1 must be running before other nodes start, as it hosts the MongoDB primary.

On backend-1:

BASH 
cd <v3.4-installation-dir>
./scripts/manage.sh start
./scripts/manage.sh status

Wait for all services on backend-1 to show Running before proceeding.

On backend-2:

BASH 
cd <v3.4-installation-dir>
./scripts/manage.sh start
./scripts/manage.sh status

On frontend-1:

BASH 
cd <v3.4-installation-dir>
./scripts/manage.sh start
./scripts/manage.sh status

Note: If HAProxy is configured to run on a privileged port (e.g., 443), you will need sudo access to start it.

  1. Re-enable health check cron (all nodes)

Run on each node (can be done in parallel):

BASH 
crontab -l | sed 's|^#DISABLED# \(.*health_check\.sh\)$|\1|' | crontab -

Verify on each node:

BASH 
crontab -l

All services should now be running on v3.4. Verify by logging into the Web UI and confirming the platform is operational.

JavaScript errors detected

Please note, these errors can depend on your browser setup.

If this problem persists, please contact our support.

Contact Support Close