On-Prem Disaster Recovery and Backup Process Guide

The following sections describe disaster recovery and backup procedures for the AgileSec Platform (On-Premise), with guidance on protecting critical data and restoring platform operations after a failure.

1. Overview

This guide outlines the disaster recovery (DR) and backup procedures for the AgileSec Platform (On-Premise). To minimize downtime, this guide provides guidelines for backing up your data and reconstituing the AgileSec Platform from a previous backup in the event of a catastrophic failure.

2. Data Components to Backup

The operational data and scan findings data resides in two datastores. To ensure a full recovery, the following stateful data components must be included in your backup strategy.

OpenSearch: Stores scan results and provides search and analytics capabilities.
MongoDB: Stores operational data including configuration state, platform management settings, and policies.

3. Recovery Point Objective (RPO) Policy & Backup Strategy

Define your Recovery Point Objective (RPO) based on data criticality and your organizational policies. Based on the RPO, define the backup frequency and retention period.

3.1 Backup Methods

OpenSearch: Native Snapshot & Restore API (supports incremental backups to shared file systems or S3).
MongoDB: mongodump using full storage-level snapshots.

3.2 Verification & Integrity

Automated Verification: All backup scripts should check exit codes ($?) and verify backup file size/existence.
Drill Testing: Perform a "Dry Run" recovery quarterly on a non-production environment to verify data integrity.

3.3 Offsite Storage

Backups MUST NOT reside on the same disk/VM as the production data.
Ship backup archives to an object storage service (e.g., AWS S3, Azure Blob Storage) or a dedicated NFS/NAS backup share, or a separate DR site after generation.

4. Component Procedures: OpenSearch

OpenSearch snapshots are the only supported way to back up an OpenSearch cluster.

4.1 Snapshot Repository Setup

You must register a snapshot repository (e.g., S3 or Shared File System like NFS). If you are running on AWS, we recommend using S3 but using a shared file system is also supported as Option B below.

Setup environment variables:

#   - Replace </path/to/installation> with actual path
#   - Replace "kf-agilesec.internal" with your actual domain
#     name
export installation_path=</path/to/installation>

# default: kf-agilesec.internal. Change it to match your
# actual domain
export analytics_internal_domain="kf-agilesec.internal"
export CA_CERT="$installation_path/certificates/"\
"ca/agilesec-rootca-cert.pem"

export CLIENT_CERT="$installation_path/certificates/"\
"$analytics_internal_domain/admin-user-cert.pem"

export CLIENT_KEY="$installation_path/certificates/"\
"$analytics_internal_domain/admin-user-key.pem"

# if you are running opensearch on the same host as the backup
# script, otherwise REPLACE OPENSEARCH_HOST with your actual
# host
export OPENSEARCH_HOST="127.0.0.1"

# if you are using the default opensearch port, 
# otherwise REPLACE OPENSEARCH_PORT with your actual port
export OPENSEARCH_PORT="9200"

Option A: Object Storage Service - AWS S3 Repository Example (`s3`)

To back up OpenSearch to an AWS S3 bucket, you must install the repository-s3 plugin on ALL OpenSearch nodes.

Install Plugin:

cd $installation_path/services/opensearch/bin
./opensearch-plugin install repository-s3
# Restart OpenSearch service on all nodes
cd $installation_path
./scripts/manage.sh restart opensearch

Configure Keystore (Credentials): Securely add your AWS credentials to the OpenSearch keystore on ALL nodes.

cd $installation_path/services/opensearch/bin
./opensearch-keystore add s3.client.default.access_key
./opensearch-keystore add s3.client.default.secret_key

Create an S3 Bucket: Create an S3 bucket if you don’t already have one. To take snapshots, you need permissions to access the bucket. The following IAM policy is an example of those permissions:

{
"Version": "2012-10-17",
"Statement": [{
    "Action": [
        "s3:*"
    ],
    "Effect": "Allow",
    "Resource": [
        "arn:aws:s3:::$OPENSEARCH_SNAPSHOT_S3_BUCKET",
        "arn:aws:s3:::$OPENSEARCH_SNAPSHOT_S3_BUCKET/*"
    ]
}]
}

Create snapshot configs and Register the Repo via REST API: Note: Replace the values in the export statements with your actual values.

cd $installation_path
# Create snapshot configs
# REPLACE with your acutal aws region
export OPENSEARCH_SNAPSHOT_S3_REGION="us-east-1"
# REPLACE with your actual bucket name
export OPENSEARCH_SNAPSHOT_S3_BUCKET="my-dr-backup-bucket"
# REPLACE with your actual base path
export OPENSEARCH_SNAPSHOT_BASEPATH="opensearch-snapshots"
export SNAPSHOT_DIR=$(date -u +"%Y%m%dt%H%M%Sz")
export SNAPSHOT_CONFIG=$(printf \
'{"type":"s3","settings":{"region":"%s","bucket":"%s",'\
'"base_path":"%s"}}' \
"$OPENSEARCH_SNAPSHOT_S3_REGION" \
"$OPENSEARCH_SNAPSHOT_S3_BUCKET" \
"$OPENSEARCH_SNAPSHOT_BASEPATH")
#REPLACE with your actual backup directory
export BACKUP_DIR='my-backup-directory'

Register the repository via REST API:

curl -X PUT -k \
"https://$OPENSEARCH_HOST:$OPENSEARCH_PORT/_snapshot/backup" \
--cacert "$CA_CERT" \
--cert "$CLIENT_CERT" --key "$CLIENT_KEY" \
-H 'Content-Type: application/json' \
-d "$SNAPSHOT_CONFIG"

Option B: Shared File System Repository (`fs`)

Mount a shared NFS directory on ALL OpenSearch nodes (e.g., /mnt/snapshots).
Add path.repo: ["/mnt/snapshots"] to $installation_path/services/opensearch/config/opensearch.yml on all nodes and restart OpenSearch.
Register the repo via API as follows:

# Set up snapshot location
# - Replace /mnt/snapshots with your actual snapshot
#   location and ensure that the directory exists on all nodes
export SNAPSHOT_LOCATION="/mnt/snapshots"
export SNAPSHOT_DIR=$(date -u +"%Y%m%dt%H%M%Sz")
export SNAPSHOT_CONFIG=\
$(printf '{"type":"fs","settings":{"location":"%s"}}' \
"$SNAPSHOT_LOCATION")

# Register the repo via REST API:
curl -X PUT -k --cacert "$CA_CERT" --cert "$CLIENT_CERT" \
--key "$CLIENT_KEY" \
"https://$OPENSEARCH_HOST:$OPENSEARCH_PORT/_snapshot/backup" \
-H 'Content-Type: application/json' \
-d "$SNAPSHOT_CONFIG"

4.2 How to Backup (Snapshot)

You can trigger a manual snapshot as follows:

Manual Snapshot:

# Snapshot Name: $SNAPSHOT_DIR
curl -X PUT \
-k "https://$OPENSEARCH_HOST:$OPENSEARCH_PORT"\
"/_snapshot/backup/$SNAPSHOT_DIR" \
--cacert "$CA_CERT" --cert "$CLIENT_CERT" --key "$CLIENT_KEY" \
-H 'Content-Type: application/json' \
-d "$SNAPSHOT_CONFIG"

Validate the status of the snapshot

# Check the status of the snapshot
curl -X GET \
"https://$OPENSEARCH_HOST:$OPENSEARCH_PORT"\
"/_snapshot/backup/$SNAPSHOT_DIR" \
--cacert "$CA_CERT" \
--cert "$CLIENT_CERT" \
--key "$CLIENT_KEY" \
-H 'Content-Type: application/json' \
-d '{ "ignore_unavailable": true }'

In the output you should see state: SUCCESS.
Also, in the output validate that shard.total == shard.successful

Backup OpenSearch security config:

Only set the BACKUP_DIR variable below if you are using the FS (File System) based snapshot method.

# RUN and RE-SET this BACKUP_DIR variable only if you are
# using fs (file system) repository based snapshot method.
export BACKUP_DIR=$SNAPSHOT_LOCATION/$SNAPSHOT_DIR

Run the following command to backup OpenSearch security config:

# Backup opensearch security config
export JAVA_HOME=$installation_path/bin/java
"$installation_path/services/opensearch/plugins/"\
"opensearch-security/tools/securityadmin.sh" \
    -h $OPENSEARCH_HOST \
    -p $OPENSEARCH_PORT \
    -backup $BACKUP_DIR \
    -icl -nhnv \
    -cacert "$CA_CERT" \
    -cert "$CLIENT_CERT" \
    -key "$CLIENT_KEY"

4.3 How to Restore from a Snapshot

Pre-condition: A new cluster must be running, and the repository must be registered.

Restore selected indices from $SNAPSHOT_DIR: Given that you are restoring in a newly created cluster (after the disaster), we first need to delete the indices that you want to restore in the new cluster since these indice do not contain any data:

export TARGET="*-isg-*,.kibana_*_*_*,"\
"-.opendistro_security,-.kibana*admintenant*"

# Check which indices will be deleted
curl -k --cacert $CA_CERT --cert $CLIENT_CERT \
--key $CLIENT_KEY \
"https://$OPENSEARCH_HOST:$OPENSEARCH_PORT/_cat/indices/"\
"${TARGET}?v&s=index"
# Delete the indices
curl -k --cacert $CA_CERT --cert $CLIENT_CERT \
--key $CLIENT_KEY -X DELETE \
"https://$OPENSEARCH_HOST:$OPENSEARCH_PORT/${TARGET}?pretty"

Restore from the snapshot:

curl -XDELETE "https://$OPENSEARCH_HOST:$OPENSEARCH_PORT"\
"/_snapshot/backup/$SNAPSHOT_DIR/_restore" \
-H 'Content-Type: application/json' \
-d '{"indices": "-.opendistro_security,*-isg-*,'\
'.kibana_*_*_*,-.kibana*admintenant*",'\
'"include_global_state": false}' \
-k --cacert "$CA_CERT" --cert "$CLIENT_CERT" \
--key "$CLIENT_KEY"

Restore roles:

$installation_path/services/opensearch/plugins/opensearch-security/tools/securityadmin.sh \
            -h $OPENSEARCH_HOST -p $OPENSEARCH_PORT \
            -icl -nhnv -t roles -f "$BACKUP_DIR/roles.yml" \
            -cacert "$CA_CERT" \
            -cert "$CLIENT_CERT" \
            -key "$CLIENT_KEY"

Restore tenants:

"$installation_path/services/opensearch/plugins/"\
"opensearch-security/tools/securityadmin.sh" \
            -h $OPENSEARCH_HOST -p $OPENSEARCH_PORT \
            -icl -nhnv -t tenants \
            -f "$BACKUP_DIR/tenants.yml" \
            -cacert "$CA_CERT" \
            -cert "$CLIENT_CERT" \
            -key "$CLIENT_KEY"

Restore roles_mapping:

"$installation_path/services/opensearch/plugins/"\
"opensearch-security/tools/securityadmin.sh" \
            -h $OPENSEARCH_HOST -p $OPENSEARCH_PORT \
            -icl -nhnv -t rolesmapping \
            -f "$BACKUP_DIR/roles_mapping.yml" \
            -cacert "$CA_CERT" \
            -cert "$CLIENT_CERT" \
            -key "$CLIENT_KEY"

Restore internal users:

"$installation_path/services/opensearch/plugins/"\
"opensearch-security/tools/securityadmin.sh" \
            -h $OPENSEARCH_HOST -p $OPENSEARCH_PORT \
            -icl -nhnv -t internalusers \
            -f "$BACKUP_DIR/internal_users.yml" \
            -cacert "$CA_CERT" \
            -cert "$CLIENT_CERT" \
            -key "$CLIENT_KEY"

Monitor Restore Progress:

curl -XGET \
"https://$OPENSEARCH_HOST:$OPENSEARCH_PORT/_cat/recovery?v"

Upstream documentation: https://docs.opensearch.org/2.19/tuning-your-cluster/availability-and-recovery/snapshots/snapshot-restore/

5. Component Procedures: MongoDB

For the MongoDB Replica Set, use mongodump for mongodb backup.

5.1 How to Backup

Run the following against the Primary or a Preferred Secondary mongonode.

# Setup Environment Variables
# Replace if taking backup from a different host or using a replica set
export MONGO_URI="mongodb://127.0.0.1:27017"
# Replace if you want to store the backup in a different location.
export BACKUP_DIR="$HOME/backups/mongo/$(date +%Y%m%d)"

mkdir -p $BACKUP_DIR
# Replace with actual path
export installation_path=</path/to/installation>
# default: kf-agilesec.internal.
# Change it to match your actual domain
export analytics_internal_domain="kf-agilesec.internal"
export CA_CERT="$installation_path/certificates/ca/"\
"agilesec-rootca-cert.pem"
export CLIENT_CERT="$installation_path/certificates/"\
"$analytics_internal_domain/admin-user-cert.pem"
export CLIENT_KEY="$installation_path/certificates/"\
"$analytics_internal_domain/admin-user-key.pem"

CLIENT_PEM="$(mktemp)"
cat "$CLIENT_CERT" "$CLIENT_KEY" > "$CLIENT_PEM"
chmod 600 "$CLIENT_PEM"

# Perform Dump with Oplog 
$installation_path/bin/mongodb-tools/bin/mongodump \
  --uri $MONGO_URI \
  --oplog \
  --out $BACKUP_DIR \
  --ssl \
  --sslCAFile $CA_CERT \
  --sslPEMKeyFile $CLIENT_PEM \
  --authenticationMechanism MONGODB-X509

rm -f "$CLIENT_PEM"

5.2 How to Restore from a MongoDB Backup

Use mongorestore with the --oplogReplay flag to apply the changes captured during the dump.

Stop Application Services: Ensure no new writes are coming in.
Restore:

$installation_path/bin/mongodb-tools/bin/mongorestore \
--uri $MONGO_URI \
--drop \
--nsExclude='admin.*' \
--ssl \
--sslCAFile $CA_CERT \
--sslPEMKeyFile $CLIENT_PEM \
--authenticationMechanism MONGODB-X509 \
$BACKUP_DIR

Flags:--drop: Drops existing collections before restoring to ensure a clean state.

5.3 Verifying Recovery

Check Replica Set status: rs.status() to ensure the restored node syncs correctly if it's the new primary. Use the Troubleshooting guide for more details on how to connect to mongodb and check the status.
Login into the application and verify that the scan data and scan configuration is there. If you see data in the 'Overview Dashboard' section, then the recovery of scan-data via OpenSearch was successful. If you also see the data under 'Scan History' section, then the recovery of configuration data via mongodb was successful as well.