On-Prem Disaster Recovery and Backup Process Guide

This guide outlines disaster recovery and backup procedures for the On-Prem AgileSec Linux installations. To minimize downtime, this guide provides guidelines for backing up your data and reconstituting the AgileSec Platform from a previous backup in the event of a catastrophic failure.

1. Data Components to Backup

The operational data and scan findings data reside in two datastores. To ensure full disaster recovery, the following data components must be backed up:

• OpenSearch: Stores scan results and provides search and analytics capabilities.

• MongoDB: Stores operational data including configuration state, platform management settings, and policies.

2. Backup Methods

This guide covers backing up crucial data components with the following methods:

• OpenSearch: Native Snapshot & Restore API (supports incremental backups to shared file systems or S3).

• MongoDB: mongodump using full storage-level snapshots.

3. Backup Strategy

In order to recover from disaster, it is crucial to have a robust backup strategy including the following components.

3.1 Recovery Point Objective (RPO) Policy

Define your RPO Policy based on data criticality and your organizational policies. Determine backup frequency and retention periods based on the RPO.

3.2 Verification & Integrity Testing

• Automated Verification: All backup scripts should check exit codes ($?) and verify backup file size/existence.

• Drill Testing: Perform a "Dry Run" recovery quarterly on a non-production environment to verify data integrity.

3.3 Offsite Storage

• Backups MUST NOT reside on the same disk/VM as the production data.

• Ship backup archives to an object storage service (e.g., AWS S3, Azure Blob Storage) or a dedicated NFS/NAS backup share, or a separate disaster recovery site after generation.

export installer_dir=</path/to/installer>
export agilesec_install_dir=</path/to/installation>

4. Backup Cluster Configuration and Certificate Materials

In order to successfully restore the previous AgileSec cluster, backup the configuration file .env and certificate materials.

4.1 Backup Cluster Configuration and Certificate Materials

Backup the required materials on each node.

1. Setup environment variables

   export installer_dir=</path/to/installer>
   export agilesec_install_dir=</path/to/installation>
   

2. Package required .env configuration and certificate materials from previous installation

   tar -zcvf agilesec_config_<node>.tar.gz $installer_dir/.env $agilesec_install_dir/certificates
   

3. Backup agilesec_config_<node>.tar.gz securely to an object storage.

5. Backup OpenSearch Clusters

OpenSearch snapshots are the only supported way to back up an OpenSearch cluster.

5.1 Snapshot Repository Setup

You must register a snapshot repository (e.g., S3 or Shared File System like NFS). If you are running on AWS, we recommend using S3 (Option A), but using a shared file system (fs) is also supported (Option B).

Setup environment variables

• Update agilesec_install_dir with actual path

• Update analytics_internal_domain with AgileSec internal domain. Default is “kf-agilesec.internal"

export agilesec_install_dir=</path/to/installation>

# default: kf-agilesec.internal. Change to match your
# actual domain
export analytics_internal_domain="kf-agilesec.internal"
export CA_CERT="$agilesec_install_dir/certificates/"\
"ca/agilesec-rootca-cert.pem"

export CLIENT_CERT="$agilesec_install_dir/certificates/"\
"$analytics_internal_domain/admin-user-cert.pem"

export CLIENT_KEY="$agilesec_install_dir/certificates/"\
"$analytics_internal_domain/admin-user-key.pem"

# if you are running opensearch on the same host as the backup
# script, otherwise REPLACE OPENSEARCH_HOST with your actual
# host
export OPENSEARCH_HOST="127.0.0.1"

# if you are using the default opensearch port, 
# otherwise REPLACE OPENSEARCH_PORT with your actual port
export OPENSEARCH_PORT="9200"

Option A: Object Storage Service - AWS S3 Repository Example (s3)

To back up OpenSearch to an AWS S3 bucket, install and configure the repository-s3 plugin on ALL OpenSearch nodes.

On each OpenSearch node perform the following steps.

1. Install repository-s3 Plugin.

cd $agilesec_install_dir/services/opensearch/bin
./opensearch-plugin install repository-s3
# Restart OpenSearch service on all nodes
cd $agilesec_install_dir
./scripts/manage.sh restart opensearch

2. Configure Keystore (Credentials). Securely add your AWS credentials to the OpenSearch keystore on ALL nodes.

cd $agilesec_install_dir/services/opensearch/bin
./opensearch-keystore add s3.client.default.access_key
./opensearch-keystore add s3.client.default.secret_key

3. Create an S3 Bucket. Create an S3 bucket if you don’t already have one. To take snapshots, you need permissions to access the bucket. The following IAM policy is an example of those permissions:

{
"Version": "2012-10-17",
"Statement": [{
    "Action": [
        "s3:*"
    ],
    "Effect": "Allow",
    "Resource": [
        "arn:aws:s3:::$OPENSEARCH_SNAPSHOT_S3_BUCKET",
        "arn:aws:s3:::$OPENSEARCH_SNAPSHOT_S3_BUCKET/*"
    ]
}]
}

4. Create snapshot configs and Register the Repo via REST API. Replace the values in the export statements with your actual values.

cd $agilesec_install_dir
# Create snapshot configs
# REPLACE with your actual aws region
export OPENSEARCH_SNAPSHOT_S3_REGION="us-east-1"
export OPENSEARCH_SNAPSHOT_S3_BUCKET="<my-disaster-recovery-backup-bucket>"
export OPENSEARCH_SNAPSHOT_BASEPATH="<opensearch-snapshots-basepath>"
export SNAPSHOT_DIR=$(date -u +"%Y%m%dt%H%M%Sz")
export SNAPSHOT_CONFIG=$(printf \
'{"type":"s3","settings":{"region":"%s","bucket":"%s",'\
'"base_path":"%s"}}' \
"$OPENSEARCH_SNAPSHOT_S3_REGION" \
"$OPENSEARCH_SNAPSHOT_S3_BUCKET" \
"$OPENSEARCH_SNAPSHOT_BASEPATH")
#REPLACE with your actual backup directory
export OS_BACKUP_DIR='my-backup-directory'

Register the repository via REST API.

curl -X PUT -k \
"https://$OPENSEARCH_HOST:$OPENSEARCH_PORT/_snapshot/backup" \
--cacert "$CA_CERT" \
--cert "$CLIENT_CERT" --key "$CLIENT_KEY" \
-H 'Content-Type: application/json' \
-d "$SNAPSHOT_CONFIG" 

Option B: Shared File System Repository (fs)

1. Mount a shared NFS directory on ALL OpenSearch nodes (e.g., /mnt/snapshots).

2. Add path.repo: ["/mnt/snapshots"] to $agilesec_install_dir/services/opensearch/config/opensearch.yml on all nodes and restart OpenSearch.

3. Register the repo via API as follows:

# Set up snapshot location
# - Replace /mnt/snapshots with your actual snapshot
#   location and ensure that the directory exists on all nodes
export SNAPSHOT_LOCATION="/mnt/snapshots"
export SNAPSHOT_DIR=$(date -u +"%Y%m%dt%H%M%Sz")
export SNAPSHOT_CONFIG=\
$(printf '{"type":"fs","settings":{"location":"%s"}}' \
"$SNAPSHOT_LOCATION")

# Register the repo via REST API:
curl -X PUT -k --cacert "$CA_CERT" --cert "$CLIENT_CERT" \
--key "$CLIENT_KEY" \
"https://$OPENSEARCH_HOST:$OPENSEARCH_PORT/_snapshot/backup" \
-H 'Content-Type: application/json' \
-d "$SNAPSHOT_CONFIG"

5.2 Create Backup Snapshot $SNAPSHOT_DIR

Trigger a manual snapshot backup $SNAPSHOT_DIR with the following:

Manually trigger snapshot creation:

# Snapshot Name: $SNAPSHOT_DIR
curl -X PUT \
-k "https://$OPENSEARCH_HOST:$OPENSEARCH_PORT"\
"/_snapshot/backup/$SNAPSHOT_DIR" \
--cacert "$CA_CERT" --cert "$CLIENT_CERT" --key "$CLIENT_KEY" \
-H 'Content-Type: application/json' \
-d "$SNAPSHOT_CONFIG" 

Validate the snapshot status

# Check the status of the snapshot
curl -X GET \
"https://$OPENSEARCH_HOST:$OPENSEARCH_PORT"\
"/_snapshot/backup/$SNAPSHOT_DIR" \
--cacert "$CA_CERT" \
--cert "$CLIENT_CERT" \
--key "$CLIENT_KEY" \
-H 'Content-Type: application/json' \
-d '{ "ignore_unavailable": true }' \
-k

• In the output you should see 

  • state: SUCCESS

  • validate shard.total == shard.successful

Backup OpenSearch security config:

Only set the OS_BACKUP_DIR variable below if you are using the fs based snapshot method (option B).

# RUN and RE-SET this OS_BACKUP_DIR variable only if you are
# using fs (file system) repository based snapshot method.
export OS_BACKUP_DIR=$SNAPSHOT_LOCATION/$SNAPSHOT_DIR

Run the following command to backup OpenSearch security config:

# Backup opensearch security config
export JAVA_HOME=$agilesec_install_dir/bin/java
"$agilesec_install_dir/services/opensearch/plugins/"\
"opensearch-security/tools/securityadmin.sh" \
    -h $OPENSEARCH_HOST \
    -p $OPENSEARCH_PORT \
    -backup $OS_BACKUP_DIR \
    -icl -nhnv \
    -cacert "$CA_CERT" \
    -cert "$CLIENT_CERT" \
    -key "$CLIENT_KEY"

6. Backup MongoDB

For the MongoDB Replica Set, use mongodump to backup mongodb.

6.1 How to Backup

Run the following against the Primary or a Preferred Secondary mongonode.

# Setup Environment Variables
# Replace if taking backup from a different host or using a replica set
export MONGO_URI="mongodb://<mongonode fqdn>:27017"
# Replace if you want to store the backup in a different location.
export MDB_BACKUP_DIR="$HOME/backups/mongo/$(date +%Y%m%d)"

mkdir -p $MDB_BACKUP_DIR
# Replace with actual path
export agilesec_install_dir=</path/to/installation>
# default: kf-agilesec.internal.
# Change it to match your actual domain
export analytics_internal_domain="kf-agilesec.internal"
export CA_CERT="$agilesec_install_dir/certificates/ca/"\
"agilesec-rootca-cert.pem"
export CLIENT_CERT="$agilesec_install_dir/certificates/"\
"$analytics_internal_domain/admin-user-cert.pem"
export CLIENT_KEY="$agilesec_install_dir/certificates/"\
"$analytics_internal_domain/admin-user-key.pem"

CLIENT_PEM="$(mktemp)"
cat "$CLIENT_CERT" "$CLIENT_KEY" > "$CLIENT_PEM"
chmod 600 "$CLIENT_PEM"

# Perform Dump with Oplog 
$agilesec_install_dir/bin/mongodb-tools/bin/mongodump \
  --uri $MONGO_URI \
  --oplog \
  --out $MDB_BACKUP_DIR \
  --ssl \
  --sslCAFile $CA_CERT \
  --sslPEMKeyFile $CLIENT_PEM \
  --authenticationMechanism MONGODB-X509 

rm -f "$CLIENT_PEM"

7. Restore Cluster Configuration and Certificate Materials

Download the backed-up materials and re-run the installation with the saved materials.

Install the AgileSec platform with the backup configuration and certificates materials. Perform the following on each node.

1. Download agilesec_config_<node>.tar.gz from your object storage.

2. Setup environment variable

   export installer_dir=</path/to/installer>
   

3. Unarchive agilesec_config_<node>.tar.gz

   mkdir -p agilesec_config
   tar -xf agilesec_config_<node>.tar.gz -C agilesec_config
   

4. Copy .env from unarchived directory to new installer location

   cp agilesec_config/<OLD_installer_dir>/.env $installer_dir
   

5. Copy certificates and keystore from unarchived directory to new installer location

   cp agilesec_config/<OLD_agilesec_install_dir>/certificates/* $installer_dir/certificates
   

6. Run the installation script on each node following order as applicable to your environment.

   1. backend-1

   2. backend-2

   3. additional backends in parallel

   4. frontend-1 or coordinator-1

   5. additional frontends in parallel

cd $installer_dir
sudo ./scripts/tune.sh -u <username>

./install_analytics.sh install -u <user> -p <installation-dir> --non-interactive

8. Restore OpenSearch Clusters

Restore backed-up OpenSearch snapshots.

Prerequisite: A new cluster must be running with preserved certificates, .env file, and keys (or keystore) (see Step 4).

1. Delete selected empty indices from new cluster

After a disaster, before restoring in a newly created cluster, we first need to delete the empty indices to be restored in the new cluster:

export TARGET="agilesec.*,.kibana_*_*_*,"\
"-.opendistro_security,-.kibana*admintenant*"

# Check which indices will be deleted
curl -k --cacert $CA_CERT --cert $CLIENT_CERT \
--key $CLIENT_KEY \
"https://$OPENSEARCH_HOST:$OPENSEARCH_PORT/_cat/indices/"\
"${TARGET}?v&s=index"
# Delete the indices
curl -k --cacert $CA_CERT --cert $CLIENT_CERT \
--key $CLIENT_KEY -X DELETE \
"https://$OPENSEARCH_HOST:$OPENSEARCH_PORT/${TARGET}?pretty"

2. Restore selected indices from $SNAPSHOT_DIR

Restore selected indices from the snapshot:

curl -X POST "https://$OPENSEARCH_HOST:$OPENSEARCH_PORT"\
"/_snapshot/backup/$SNAPSHOT_DIR/_restore" \
-H 'Content-Type: application/json' \
-d '{"indices": "-.opendistro_security,agilesec.*,'\
'.kibana_*_*_*,-.kibana*admintenant*",'\
'"include_global_state": false}' \
-k --cacert "$CA_CERT" --cert "$CLIENT_CERT" \
--key "$CLIENT_KEY"

3. Restore roles

Export required variable OPENSEARCH_JAVA_HOME before running restore command:

export OPENSEARCH_JAVA_HOME=$agilesec_install_dir/bin/java

Restore roles with OpenSearch securityadmin.sh:

# Restore roles
$agilesec_install_dir/services/opensearch/plugins/opensearch-security/tools/securityadmin.sh \
            -h $OPENSEARCH_HOST -p $OPENSEARCH_PORT \
            -icl -nhnv -t roles -f "$OS_BACKUP_DIR/roles.yml" \
            -cacert "$CA_CERT" \
            -cert "$CLIENT_CERT" \
            -key "$CLIENT_KEY"

4. Restore tenants:

Export required variable OPENSEARCH_JAVA_HOME before running restore command:

export OPENSEARCH_JAVA_HOME=$agilesec_install_dir/bin/java

Restore tenants with OpenSearch securityadmin.sh:

$agilesec_install_dir/services/opensearch/plugins/opensearch-security/tools/securityadmin.sh \
            -h $OPENSEARCH_HOST -p $OPENSEARCH_PORT \
            -icl -nhnv -t tenants \
            -f "$OS_BACKUP_DIR/tenants.yml" \
            -cacert "$CA_CERT" \
            -cert "$CLIENT_CERT" \
            -key "$CLIENT_KEY"

5. Restore roles_mapping:

Export required variable OPENSEARCH_JAVA_HOME before running restore command:

export OPENSEARCH_JAVA_HOME=$agilesec_install_dir/bin/java

Restore roles_mapping with OpenSearch securityadmin.sh:

$agilesec_install_dir/services/opensearch/plugins/\
opensearch-security/tools/securityadmin.sh \
            -h $OPENSEARCH_HOST -p $OPENSEARCH_PORT \
            -icl -nhnv -t rolesmapping \
            -f "$OS_BACKUP_DIR/roles_mapping.yml" \
            -cacert "$CA_CERT" \
            -cert "$CLIENT_CERT" \
            -key "$CLIENT_KEY"

6. Restore internal users

Export required variable OPENSEARCH_JAVA_HOME before running restore command:

export OPENSEARCH_JAVA_HOME=$agilesec_install_dir/bin/java

Restore internal users with OpenSearch securityadmin.sh:

$agilesec_install_dir/services/opensearch/plugins/opensearch-security/tools/securityadmin.sh" \
            -h $OPENSEARCH_HOST -p $OPENSEARCH_PORT \
            -icl -nhnv -t internalusers \
            -f "$OS_BACKUP_DIR/internal_users.yml" \
            -cacert "$CA_CERT" \
            -cert "$CLIENT_CERT" \
            -key "$CLIENT_KEY"

7. Monitor Restore Progress

Monitor recovery progress with the following command:

curl -k -X GET "https://$OPENSEARCH_HOST:$OPENSEARCH_PORT/_cat/recovery?v"

Upstream documentation: https://docs.opensearch.org/2.19/tuning-your-cluster/availability-and-recovery/snapshots/snapshot-restore/.

9. Restore MongoDB Backup

Use mongorestore with the --oplogReplay flag to restore from the MongoDB backup and apply changes captured during the dump.

1. Stop Application Services: Ensure no new writes are coming in.

2. Restore:

$agilesec_install_dir/bin/mongodb-tools/bin/mongorestore \
--uri $MONGO_URI \
--drop \
--nsExclude='admin.*' \
--ssl \
--sslCAFile $CA_CERT \
--sslPEMKeyFile $CLIENT_PEM \
--authenticationMechanism MONGODB-X509 \
$MDB_BACKUP_DIR

Flags: --drop: Drops existing collections before restoring to ensure a clean state.

10. Verify Recovery

1. Check Replica Set status: rs.status() to ensure the restored node syncs correctly if it's the new primary. Use the Troubleshooting guide for more details on how to connect to mongodb and check the status.

2. Login into the application and verify the scan data and scan configuration is there.

   1. If you see data in the 'Overview Dashboard' section, then the recovery of scan-data via OpenSearch was successful.

   2. If you see scan data under 'Scan History' section, then the recovery of configuration data via mongodb was also successful.