On-Prem Troubleshooting Guide

This troubleshooting guide provides detailed procedures for diagnosing and resolving issues with the Keyfactor AgileSec Analytics Platform. It covers both single-node and multi-node deployments with focus on root cause analysis and resolution steps.

1. Introduction

1.1 Platform Architecture Overview

The platform consists of three service tiers:

Infrastructure Layer:

OpenSearch - Search and analytics engine
OpenSearch Dashboards - Web interface for OpenSearch
MongoDB - Operational data store with replica set support
Kafka - Message queue broker (KRaft mode)

Supporting Services:

HAProxy - Load balancer and reverse proxy
Fluentd (td-agent) - Ship data from Kafka to OpenSearch

Application Microservices:

Java Backend Services: ingestion, scheduler, sm (Secrets Manager), analytics-manager, indexing
Node.js Frontend Services: webui, api, cbom

1.2 Deployment Types

Type	Description
Single-Node	All services on one server
Multi-Node	Distributed deployment with PRIMARY_FULL_BACKEND, FULL_BACKEND, FRONTEND, and SCAN nodes

1.3 Service Dependencies

Services Dependencies:

opensearch            : (no dependencies)
opensearch-dashboards : opensearch
mongodb               : (no dependencies)
kafka                 : (no dependencies)
td-agent              : kafka, opensearch
webui                 : api
api                   : kafka, opensearch, opensearch-dashboards, mongodb, sm, analytics-manager, ingestion
cbom                  : kafka, opensearch, mongodb, sm
sm                    : (no dependencies)
analytics-manager     : kafka, mongodb, sm
ingestion             : kafka, analytics-manager
indexing              : kafka, opensearch
scheduler             : kafka, mongodb, sm, ingestion, opensearch
haproxy               : (no dependencies)

2. Log File Locations

2.1 Log Directory Structure

All logs are stored under $installation_path/logs/.

2.2 Infrastructure Service Logs

Quick platform logs: These are the main log files for each infrastructure service.

Service	Log Location	Description
OpenSearch	`$installation_path/logs/opensearch/agilesec-cluster.log`	Cluster logs, slow queries, deprecation warnings
MongoDB	`$installation_path/logs/mongodb.log`	Server logs, query logs
Kafka	`$installation_path/logs/kafka/server.log`	Broker logs, controller logs

Full platform logs: These show detailed logs for each platform.

Service	Log Location	Description
OpenSearch	`$installation_path/logs/opensearch/`	Cluster logs, slow queries, deprecation warnings
MongoDB	`$installation_path/logs/mongod.log`	Server logs, query logs
Kafka	`$installation_path/logs/kafka/`	Broker logs, controller logs

2.3 Application Microservice Logs

Service	Log Location
webui	`$installation_path/logs/webui.log`
api	`$installation_path/logs/api.log`
cbom	`$installation_path/logs/cbom.log`
sm	`$installation_path/logs/sm.log`
analytics-manager	`$installation_path/logs/analytics-manager.log`
ingestion	`$installation_path/logs/ingestion.log`
scheduler	`$installation_path/logs/scheduler.log`
indexing	`$installation_path/logs/indexing.log`

2.4 Supporting Service Logs

Service	Log Location
HAProxy	`$installation_path/logs/haproxy.log`
Fluentd (td-agent)	`$installation_path/logs/td-agent.log`

2.5 Management and Health Check Logs

Log File	Purpose
`$installation_path/logs/health_check_cronjob.log`	Automated health check results and service restart attempts

2.6 Log Archives

Archived logs are stored under: $installation_path/logs/archive

Check Archive Directory

# Important: 
#   - Replace </path/to/installation> with actual path
export installation_path=</path/to/installation>

ls -ld "$installation_path/logs" "$installation_path/logs/archive"

find "$installation_path/logs/archive" -maxdepth 1 -type f \
  -printf '%TY-%Tm-%Td %TH:%TM %10s %f\n' | sort | tail -50

Check Log and Archive Disk Usage

# Important: 
#   - Replace </path/to/installation> with actual path
export installation_path=</path/to/installation>

du -sh "$installation_path/logs" "$installation_path/logs/archive"

find "$installation_path/logs" -maxdepth 1 -type f -name '*.log' \
  -printf '%10s %f\n' | sort -nr

Archive Sources

Log Type	Archive Behavior
OpenSearch	Uses Log4j2 and archives to $installation_path/logs/archive
Java services	Use application-managed Logback archives under $installation_path/logs/archive
Node.js services	Rotated by installer-managed bundled logrotate when present on the node
MongoDB	Rotated by bundled logrotate using USR1; copytruncate is not used
Fluentd / td-agent	Rotated by bundled logrotate from the real td-agent service log
Health check log	Rotated by bundled logrotate when health-check cron is enabled

3. Service Status Inspection Commands

3.1 Using the Unified Management Script

The primary tool for service management is manage.sh:

cd $installation_path
./scripts/manage.sh <action> [options] [service1 service2 ...]

Available Actions:

Action	Description
`start`	Start services
`stop`	Stop services
`restart`	Stop and then start services
`reload`	Reload service configuration (where supported)
`status`	Check status of services
`list`	List available services

Options:

Option	Description
`-d, --debug`	Enable debug mode (show service output in console)
`-s, --silent`	Enable silent mode (no output displayed)

Note: manage.sh does require sudo privileges or root login to start/stop services if systemd management was enabled

3.2 Common Status Commands

# Check all services status
./scripts/manage.sh status

# Check specific service status
./scripts/manage.sh status opensearch
./scripts/manage.sh status mongodb kafka

# Start all services
./scripts/manage.sh start

# Start specific service with debug output
./scripts/manage.sh start -d opensearch

# Stop specific services
./scripts/manage.sh stop haproxy td-agent

# Restart a service
./scripts/manage.sh restart scheduler

# Reload HAProxy configuration
./scripts/manage.sh reload haproxy

3.3 Understanding Status Output

The status command shows:

Service name
Description
Running/Not running status
Uptime

Example Output:

SERVICE                  DESCRIPTION                      STATUS                    UPTIME    
------------------------ -------------------------------- ------------------------- ----------
mongodb                  MongoDB Server                   Running                   10m 39s   
opensearch               OpenSearch Search Engine         Running                   10m 39s   
kafka                    Kafka Server                     Running                   10m 14s   
td-agent                 Fluentd Data Collector           Running (2 procs)         10m 2s    
scheduler                Scheduler Microservice           Running                   10m 14s   
analytics-manager        Analytics Manager Microservice   Running                   10m 14s   
ingestion                Ingestion Microservice           Running                   10m 13s   
indexing                 Indexing Microservice            Running                   10m 13s   
sm                       Secrets Manager Microservice     Running                   10m 12s   
haproxy                  HAProxy Load Balancer            Running (2 procs)         10m 11s

3.4 Systemd Service Status

The platform uses a systemd service for automatic startup:

# Check systemd service status
sudo systemctl status kf_agilesec-*.service
# View service logs
sudo journalctl -u kf_agilesec-*.service-f

4. Common Failure Scenarios and Fixes

Use this section to identify the likely failing component and choose the next diagnostic step. Run commands on the node where the target service is installed.

Before running service-specific checks, set the common variables:

export installation_path=/path/to/install
export installer_path=/path/to/installer
export analytics_internal_domain=kf-agilesec.internal

export mongodb_host=<mongodb-node-hostname> # Example: backend-1.kf-agilesec.internal
export kafka_bootstrap_host=<kafka-broker-hostname> # Example: backend-1.kf-agilesec.internal

4.1 OpenSearch Failures

4.1.1 Service Fails to Start

Symptoms:

OpenSearch process exits immediately after starting
No listening on port 9200
Logs show Unable to lock JVM Memory, OutOfMemoryError, SSLHandshakeException, or Address already in use

Initial diagnostics:

cd "$installation_path"
./scripts/manage.sh status opensearch
ss -tlnp | grep -E ':(9200|9300)\b' || true
ulimit -l
tail -100 "$installation_path/services/opensearch/logs/"*.log

Root Cause Analysis:

Cause	Log Indicator	Resolution
Memory lock failure	"Unable to lock JVM Memory"	Run tune.sh for the install user
Insufficient heap	"OutOfMemoryError"	Review OpenSearch heap settings
Certificate errors	"SSLHandshakeException"	Validate certificate paths and chain
Port already in use	"Address already in use"	Kill conflicting process

Resolution - Memory Lock:

# Check current limits
ulimit -l

# Run tune.sh command to adjust system settings
cd "$installer_path"
sudo ./scripts/tune.sh -u <install_user>

# Important: Logout/Login to refresh session

Resolution - Heap Size:

# Check current heap settings in service environment file
cat $installation_path/services/opensearch/config/jvm.options

# Modify -Xms and -Xmx values
# Example: 
#  -Xms32g 
#  -Xmx32g

4.1.2 OpenSearch Cluster Health Is RED

Symptoms:

API returns cluster health status as "red"
Some indices are unavailable
Write operations failing

Root Cause Analysis:

Unassigned primary shards
OpenSearch node down or unreachable
Disk watermark exceeded
Cluster allocation disabled
Certificate or network issue between OpenSearch nodes

Diagnostic Commands:

# Set certificate variables
# Important: 
#   - Replace </path/to/installation> with actual path
#   - Replace "kf-agilesec.internal" with your actual domain name

export installation_path=</path/to/installation>
export analytics_internal_domain="kf-agilesec.internal" # default: kf-agilesec.internal. Change it to match your actual domain
export CA_CERT=$installation_path/certificates/ca/agilesec-rootca-cert.pem
export CLIENT_CERT=$installation_path/certificates/$analytics_internal_domain/admin-user-cert.pem
export CLIENT_KEY=$installation_path/certificates/$analytics_internal_domain/admin-user-key.pem

# Check cluster health
curl -k --cacert $CA_CERT --cert $CLIENT_CERT --key $CLIENT_KEY \
  'https://127.0.0.1:9200/_cluster/health?pretty'

# Check shard allocation explanation
curl -k --cacert $CA_CERT --cert $CLIENT_CERT --key $CLIENT_KEY \
  'https://127.0.0.1:9200/_cluster/allocation/explain?pretty'

# List unassigned shards
curl -k --cacert $CA_CERT --cert $CLIENT_CERT --key $CLIENT_KEY \
  'https://127.0.0.1:9200/_cat/shards?v&h=index,shard,prirep,state,unassigned.reason'

Resolution:

# Enable shard allocation (if disabled)
curl -k --cacert $CA_CERT --cert $CLIENT_CERT --key $CLIENT_KEY \
  -X PUT 'https://127.0.0.1:9200/_cluster/settings' \
  -H 'Content-Type: application/json' -d '{
    "transient": {
      "cluster.routing.allocation.enable": "all"
    }
  }'

# Reroute stuck shards
curl -k --cacert $CA_CERT --cert $CLIENT_CERT --key $CLIENT_KEY \
  -X POST 'https://127.0.0.1:9200/_cluster/reroute?retry_failed=true'

4.1.3 Certificate Authentication Failure

Symptoms:

"SSLHandshakeException" in logs
"certificate verify failed" errors
Services cannot connect to OpenSearch

Root Cause Analysis:

Certificate expired
Wrong CA certificate
Certificate chain incomplete
DN not in allowed list

Diagnostic Commands:

# Set certificate variables
# Important: 
#   - Replace </path/to/installation> with actual path
#   - Replace "kf-agilesec.internal" with your actual domain name

export installation_path=</path/to/installation>
export analytics_internal_domain="kf-agilesec.internal" # default: kf-agilesec.internal. Change it to match your actual domain
export CA_CERT="$installation_path/certificates/ca/agilesec-rootca-cert.pem"
export CLIENT_CERT="$installation_path/certificates/$analytics_internal_domain/admin-user-cert.pem"

openssl x509 -in "$CLIENT_CERT" -noout -dates
openssl verify -CAfile "$CA_CERT" "$CLIENT_CERT"
openssl x509 -in "$CLIENT_CERT" -noout -subject -issuer

Resolution:

Regenerate certificates if expired
Ensure CA certificate matches the one used to sign client/server certificates
Verify plugins.security.nodes_dn and plugins.security.authcz.admin_dn in opensearch.yml

4.1.4 JVM Heap Exhaustion

Symptoms:

"OutOfMemoryError: Java heap space" in logs
Service becomes unresponsive
Frequent garbage collection pauses

Root Cause Analysis:

Heap size too small for data volume
Large aggregation queries

Resolution:

# Check current heap usage
curl -k --cacert $CA_CERT --cert $CLIENT_CERT --key $CLIENT_KEY \
  'https://127.0.0.1:9200/_nodes/stats/jvm?pretty' | grep -A 10 "heap"

# Check current heap settings in service environment file
cat $installation_path/services/opensearch/config/jvm.options

# Modify -Xms and -Xmx values
# Example: 
#  -Xms32g 
#  -Xmx32g

# Restart OpenSearch
./scripts/manage.sh restart opensearch

4.1.5 Disk Space Exhaustion

Symptoms:

Write operations rejected
"disk watermark exceeded" in logs
Index status becomes read-only

Root Cause Analysis:

Data growth exceeding available disk
Log accumulation
Old indices not cleaned up

Diagnostic Commands:

# Check disk usage
# Important: 
#   - Replace </path/to/installation> with actual path
#   - Replace "kf-agilesec.internal" with your actual domain name

export installation_path=</path/to/installation>
export analytics_internal_domain="kf-agilesec.internal" # default: kf-agilesec.internal. Change it to match your actual domain
export CA_CERT=$installation_path/certificates/ca/agilesec-rootca-cert.pem
export CLIENT_CERT=$installation_path/certificates/$analytics_internal_domain/admin-user-cert.pem
export CLIENT_KEY=$installation_path/certificates/$analytics_internal_domain/admin-user-key.pem

# Check disk size of each nodes
curl -k --cacert $CA_CERT --cert $CLIENT_CERT --key $CLIENT_KEY \
  'https://127.0.0.1:9200/_cat/allocation?v&h=node,shards,disk.percent,disk.used,disk.avail,disk.total'

Resolution:

Increase free disk size if it is too low

4.2 MongoDB Failures

4.2.1 MongoDB Fails to Start

Symptoms:

mongod process exits immediately
Logs show permission, socket, TLS, or lock errors

Diagnostics:

cd "$installation_path"

./scripts/manage.sh status mongodb
ss -tlnp | grep ':27017' || true
pgrep -af mongod || true
tail -100 "$installation_path/logs/mongod.log"

Root Cause Analysis:

Cause	What to check	Recommended action
Port in use	Port 27017 already in use	Kill conflicting process
TLS certificate issues	Certificate or key errors in logs	Check certificate paths and permissions
Lock file exists	Lock file exists after crash	Remove only after confirming MongoDB is stopped
Permission issue	Permission denied in logs	Check ownership and file modes

Resolution - Lock File:

# Remove stale lock file (only if mongod is not running)
rm -f $installation_path/data/mongodb/mongod.lock

4.2.2 MongoDB TLS/mTLS Connection Failure

Symptoms:

"SSL peer certificate validation failed"
"certificate verify failed"
Services cannot connect to MongoDB

Root Cause Analysis:

Client certificate not trusted by server
CA mismatch between client and server
Certificate expired
Wrong certificate key file

Diagnostic Commands:

# Test MongoDB connection with TLS
# Important: 
#   - Replace </path/to/installation> with actual path
#   - Replace "kf-agilesec.internal" with your actual domain name
#   - Replace "<Path to installer-directory>" with actual path
#   - Replace "<mongodb-node-hostname>" with actual internal mongodb hostname

export installation_path=</path/to/installation>
export analytics_internal_domain="kf-agilesec.internal" # default: kf-agilesec.internal. Change it to match your actual domain
export mongodb_host=<mongodb-node-hostname> # Example: backend-1.kf-agilesec.internal
export installer_path=<Path to installer-directory> #Note that <installer_path> is different from <installation_path>
export OPENSSL_CONF="$installer_path/templates/mongodb/openssl-mongosh.cnf"

"$installation_path/bin/mongosh" \
  --tls \
  --tlsCAFile "$installation_path/certificates/ca/agilesec-rootca-cert.pem" \
  --tlsCertificateKeyFile "$installation_path/certificates/$analytics_internal_domain/shared-client-combo-cert-key.pem" \
  --authenticationMechanism MONGODB-X509 \
  --host "$mongodb_host" \
  --port 27017 \
  --quiet \
  --eval 'db.runCommand({ ping: 1 })'

Resolution:

Verify tlsCAFile path in mongod.conf matches the CA used to sign client certificates
Ensure tlsCertificateKeyFile contains both certificate and key
Check certificate expiration dates

4.3 Kafka Failures

4.3.1 Broker Fails to Start

Symptoms:

Kafka process exits immediately
"Address already in use" error
KRaft controller election fails
SSL handshake failures

No space left on device

Root Cause Analysis:

Cause	Log Indicator	Resolution
Port conflict	"Address already in use"	Check ports 9092, 9093, 9094
KRaft init failure	"Cluster ID mismatch"	Check cluster_id.txt
SSL configuration	"SSL handshake failed"	Verify keystore paths
Insufficient disk	"No space left on device"	Free disk space

Diagnostic Commands:

# Important: 
#   - Replace </path/to/installation> with actual path
export installation_path=</path/to/installation>

cd "$installation_path"
./scripts/manage.sh status kafka
ss -tlnp | grep -E ':(9092|9093|9094)\b' || true
tail -100 "$installation_path/services/kafka_"*/logs/server.log
df -h

Resolution - Port Conflict:

# Find and kill process using the port
fuser -k 9092/tcp
fuser -k 9093/tcp
fuser -k 9094/tcp

# Restart Kafka
./scripts/manage.sh start kafka

4.4 Java Microservice Failures

Applies to: scheduler, sm, ingestion, analytics-manager, indexing

4.4.1 Kafka Connection Failure

Symptoms:

Java Microservice logs show the error with below Exception pattern:
- "TimeoutException"
- "DisconnectException"
- "SerializationException"
- "DeserializationException"
- "CommitFailedException"
- "AuthorizationException"
- "SaslAuthenticationException"

Root Cause Analysis:

Kafka broker not running or not healthy
SSL configuration mismatch
CPU/MEM of kafka broker is peak
Network connectivity issues

Resolution:

Verify Kafka is running: ./scripts/manage.sh status kafka
Check kafka is healthy or not ( See Section 5.3 for kafka health checks)
Check Kafka load (CPU, Disk, Memory usage)
Verify SSL certificate match between Kafka and microservice
Check network connectivity between microservice and Kafka brokers

4.4.2 MongoDB Connection Failure

Symptoms:

Java Microservice logs show the error with below Exception pattern:
- "MongoTimeoutException"
- "MongoSocketOpenException"
- "MongoSocketReadException"
- "Timed out while waiting for a server"
- "Connection refused"
- "No server chosen by ReadPreference"
- "MongoSecurityException"
- "Authentication failed"
- "Unauthorized"
- "not authorized on .* to execute"

Resolution:

Verify mongo is running: ./scripts/manage.sh status mongodb
Check mongo is healthy or not ( See Section 5.2 for mongo health checks)
Check MongoDB load (CPU, Disk, Memory usage)
Verify SSL certificate match between MongoDB and microservice
Check network connectivity between microservice and MongoDB cluster

4.4.3 SM Service Keystore Failure

Symptoms:

"Cannot load keystore"
"Keystore was tampered with, or password was incorrect"

Resolution:

# Verify keystore exists and is readable
export analytics_internal_domain="kf-agilesec.internal" # default: kf-agilesec.internal. Change it to match your actual domain
export sm_keystore_pass=$(cat $installation_path/certificates/$analytics_internal_domain/sm-service-keystore.pass)

ls -la $installation_path/certificates/$analytics_internal_domain/sm-service.p12

# Verify password
$installation_path/bin/java/bin/keytool -list -keystore $installation_path/certificates/$analytics_internal_domain/sm-service.p12 -storepass "$sm_keystore_pass"

4.5 Node.js Microservice Failures

Applies to: webui, api, cbom

4.5.1 Kafka Connection Failure

Symptoms:

Nodejs Microservice logs show the error with below Exception pattern:
- "KafkaJSConnectionError"
- "SSL alert number"

Root Cause Analysis:

Kafka broker not running or not healthy
SSL configuration mismatch
CPU/MEM of kafka broker is peak
Network connectivity issues

Resolution:

Verify Kafka is running: ./scripts/manage.sh status kafka
Check kafka is healthy or not ( See Section 5.3 for kafka health checks)
Check Kafka load (CPU, Disk, Memory usage)
Verify SSL certificate match between Kafka and microservice
Check network connectivity between microservice and Kafka brokers

4.5.2 MongoDB Connection Errors

Symptoms:

Nodejs Microservice logs show the error with below Exception pattern:
- "MongooseError"
- "buffering timed out after"

Resolution:

Verify mongo is running: ./scripts/manage.sh status mongodb
Check mongo is healthy or not ( See Section 5.2 for mongo health checks)
Check MongoDB load (CPU, Disk, Memory usage)
Verify SSL certificate match between MongoDB and microservice
Check network connectivity between microservice and MongoDB cluster

4.5.3 Opensearch Connection Errors

Symptoms:

Nodejs Microservice logs show the error with below Exception pattern:
- "Query open-search error"
- "ECONNREFUSED"

Resolution:

Verify OpenSearch is running: ./scripts/manage.sh status opensearch
Check OpenSearch is healthy or not ( See Section 5.1 for opensearch health checks)
Check OpenSearch load (CPU, Disk, Memory usage)
Verify SSL certificate match between OpenSearch and microservice
Check network connectivity between microservice and OpenSearch cluster

5. Cluster Health Diagnostics

5.1 OpenSearch Cluster Health

Using curl with certificate authentication for cluster diagnostics

# Important: 
#   - Replace </path/to/installation> with actual path
#   - Replace "kf-agilesec.internal" with your actual domain name
export installation_path=</path/to/installation>
export analytics_internal_domain="kf-agilesec.internal" # default: kf-agilesec.internal. Change it to match your actual domain

export CA_CERT=$installation_path/certificates/ca/agilesec-rootca-cert.pem
export CLIENT_CERT=$installation_path/certificates/$analytics_internal_domain/admin-user-cert.pem
export CLIENT_KEY=$installation_path/certificates/$analytics_internal_domain/admin-user-key.pem

curl -k --cacert "$CA_CERT" --cert "$CLIENT_CERT" --key "$CLIENT_KEY" \
  "https://127.0.0.1:9200/_cluster/health?pretty"

# Node status
curl -k --cacert $CA_CERT --cert $CLIENT_CERT --key $CLIENT_KEY \
  'https://127.0.0.1:9200/_cat/nodes?v&h=ip,name,role,master,heap.percent,disk.used_percent'

# Cluster statistics
curl -k --cacert $CA_CERT --cert $CLIENT_CERT --key $CLIENT_KEY \
  'https://127.0.0.1:9200/_cluster/stats?pretty'

# Index health
curl -k --cacert $CA_CERT --cert $CLIENT_CERT --key $CLIENT_KEY \
  'https://127.0.0.1:9200/_cat/indices?v'

# Shard health
curl -k --cacert "$CA_CERT" --cert "$CLIENT_CERT" --key "$CLIENT_KEY" \
  "https://127.0.0.1:9200/_cat/shards?v"

Expected result:

Cluster health should normally be green.
yellow usually means replica shards are not fully assigned.
red means at least one primary shard is unavailable and needs immediate investigation.
_cat/nodes should show all expected OpenSearch nodes.
_cat/shards should not show unexpected unassigned primary shards.

OpenSearch Dashboards can also be used for manual diagnostics

Using Opensearch Dashboard for cluster diagnostics
- Opensearch Dashboard's Dev Tools console provides additional cluster diagnostic capabilities
- Use GET /_cat/nodes?v and GET /_cat/indices?v for detailed node and index status
- Use GET /_cluster/health for cluster health status
- Use GET /_cluster/stats for cluster statistics
- Use GET /_cat/shards?v for shard allocation status
- Monitor for yellow/red health indicators which may indicate shard allocation issues
- Pay attention to unassigned shards and disk usage percentages (look for nodes with high disk usage, especially above 80%)
- Check for any nodes showing "UNREACHABLE" or "DISCONNECTED" status
- Look for nodes with high memory usage (above 85%) which may indicate performance issues
- Watch for nodes with high CPU usage (above 90%) which may indicate resource contention
- Check for any nodes with excessive load averages that may indicate system overload

5.2 MongoDB Replica Set Health

# Important: 
#   - Replace </path/to/installation> with actual path
#   - Replace "kf-agilesec.internal" with your actual domain name
#   - Replace "<Path to installer-directory>" with actual path
#   - Replace "<mongodb-node-hostname>" with actual internal mongodb hostname

export installation_path=</path/to/installation>
export analytics_internal_domain="kf-agilesec.internal" # default: kf-agilesec.internal. Change it to match your actual domain
export mongodb_host=<mongodb-node-hostname> # Example: backend-1.kf-agilesec.internal
export installer_path=<Path to installer-directory> #Note that <installer_path> is different from <installation_path>
export OPENSSL_CONF="$installer_path/templates/mongodb/openssl-mongosh.cnf"

# Connect and check replica set status
"$installation_path/bin/mongosh" \
  --tls \
  --tlsCAFile "$installation_path/certificates/ca/agilesec-rootca-cert.pem" \
  --tlsCertificateKeyFile "$installation_path/certificates/$analytics_internal_domain/admin-user-combo-cert-key.pem" \
  --host "$mongodb_host:27017" \
  --quiet \
  --eval 'const s=rs.status(); print("ok:", s.ok); print("members:", s.members.length); print("myState:", s.myState);'

# Check replica set members
"$installation_path/bin/mongosh" \
  --tls \
  --tlsCAFile "$installation_path/certificates/ca/agilesec-rootca-cert.pem" \
  --tlsCertificateKeyFile "$installation_path/certificates/$analytics_internal_domain/admin-user-combo-cert-key.pem" \
  --host "$mongodb_host:27017" \
  --quiet \
  --eval 'const h=db.hello(); print("isWritablePrimary:", h.isWritablePrimary); print("secondary:", h.secondary); printjson(h.hosts);'

Expected result:

ok: 1
Replica set members are listed.
One node should be writable primary.
Other data-bearing nodes should report as secondary.

5.3 Kafka Cluster Health

# Important: 
#   - Replace </path/to/installation> with actual path
#   - Replace "kf-agilesec.internal" with your actual domain name
#   - Replace "<kafka-broker-hostname>" with your actual node domain name
export installation_path=</path/to/installation>
export analytics_internal_domain="kf-agilesec.internal" # default: kf-agilesec.internal. Change it to match your actual domain
export kafka_bootstrap_host=<kafka-broker-hostname> # Example: backend-1.kf-agilesec.internal

export JAVA_HOME="$installation_path/bin/java"
test -d "$installation_path/tmp" || {
  echo "$installation_path/tmp does not exist. Verify the installation path."
  exit 1
}

umask 077
work_dir="$(mktemp -d "$installation_path/tmp/kafka-health.XXXXXX")"
trap 'rm -rf "$work_dir"' EXIT

ssl_properties="$work_dir/ssl.properties"

cat > "$ssl_properties" <<EOF
security.protocol=SSL
ssl.truststore.type=PEM
ssl.truststore.location=$installation_path/certificates/ca/agilesec-rootca-cert.pem
ssl.keystore.type=PEM
ssl.keystore.location=$installation_path/certificates/$analytics_internal_domain/shared-client-combo-cert-key.pem
ssl.endpoint.identification.algorithm=https
EOF

"$installation_path/services/kafka/bin/kafka-topics.sh" \
  --bootstrap-server "$kafka_bootstrap_host:9092" \
  --command-config "$ssl_properties" \
  --list

"$installation_path/services/kafka/bin/kafka-consumer-groups.sh" \
  --bootstrap-server "$kafka_bootstrap_host:9092" \
  --command-config "$ssl_properties" \
  --describe \
  --all-groups

Validation result:

$installation_path/tmp exists and is writable.
Temporary directory was created as 700.
ssl.properties was created as 600.
Kafka topic listing succeeded.
Kafka consumer group describe succeeded.

6. Node Recovery Procedures

6.1 Single-Node Recovery

Complete Recovery Sequence:

# Important: 
#   - Replace </path/to/installation> with actual path
export installation_path=</path/to/installation>

# 1. Stop all services gracefully
./scripts/manage.sh stop

# 2. Verify all processes are stopped
./scripts/manage.sh status
ps aux | grep -E '(opensearch|mongod|kafka|java|node)'

# 3. Verify disk space
df -h

# 4. Start services in order
./scripts/manage.sh start 

# 5. Verify all services
./scripts/manage.sh status

6.2 Multi-Node Recovery

6.2.1 OpenSearch Node Recovery

# On the failed node:
# Important: 
#   - Replace </path/to/installation> with actual path
#   - Replace "kf-agilesec.internal" with your actual domain name

export installation_path=</path/to/installation>
export analytics_internal_domain="kf-agilesec.internal" # default: kf-agilesec.internal. Change it to match your actual domain
export CA_CERT="$installation_path/certificates/ca/agilesec-rootca-cert.pem"
export CLIENT_CERT="$installation_path/certificates/$analytics_internal_domain/admin-user-cert.pem"
export CLIENT_KEY="$installation_path/certificates/$analytics_internal_domain/admin-user-key.pem"

cd "$installation_path"
# 0. Check logs:
tail -100 "$installation_path/logs/opensearch/agilesec-cluster.log"
curl -s -k --cacert "$CA_CERT" --cert "$CLIENT_CERT" --key "$CLIENT_KEY" \
    "https://127.0.0.1:9200/_cluster/health?pretty" | \
    grep -E '"(status|number_of_nodes|relocating_shards|initializing_shards|unassigned_shards)"'

#Only if OpenSearch data is confirmed corrupted and the cluster has recoverable replicas, move the data aside instead of deleting it:
# 1. Stop OpenSearch and check logs:
./scripts/manage.sh stop opensearch

# 2. Check and fix data directory if corrupted. Backup the directory before deleting.
# (Only if necessary, e.g., when recovering from backup - THIS WILL RESULT IN DATA LOSS)
# mv "$installation_path/data/opensearch/nodes" "$installation_path/data/opensearch/nodes.recovery.$(date +%Y%m%d-%H%M%S)"

# 3. Restart OpenSearch
./scripts/manage.sh start opensearch

# 4. Monitor cluster recovery
watch -n 5 "curl -s -k --cacert \$CA_CERT --cert \$CLIENT_CERT --key \$CLIENT_KEY \
  'https://127.0.0.1:9200/_cluster/health?pretty' | grep -E '(status|relocating|initializing)'"

6.2.2 MongoDB Node Recovery

Secondary Node Recovery:

# 1. Stop MongoDB on failed secondary
./scripts/manage.sh stop mongodb

# 2. Optionally resync from primary (if data is corrupted)
# mv "$installation_path/data/mongodb" \
     "$installation_path/data/mongodb.recovery.$(date +%Y%m%d-%H%M%S)"
mkdir -p "$installation_path/data/mongodb"

# 3. Start MongoDB
./scripts/manage.sh start mongodb

# 4. MongoDB will automatically sync from primary

Primary Node Recovery:

# If primary fails, a secondary should automatically become primary
# After recovering the failed node:

# 1. Start MongoDB
./scripts/manage.sh start mongodb

# 2. It will rejoin as secondary and sync
# 3. To force it back to primary (if desired):
# rs.stepDown()  # On current primary

6.2.3 Kafka Broker Recovery

# 1. Stop failed broker
./scripts/manage.sh stop kafka

# 2. Check for log corruption
ls -la $installation_path/data/kafka/

# 3. Start broker
./scripts/manage.sh start kafka

# 4. Verify broker rejoined cluster
$installation_path/services/kafka_*/bin/kafka-dump-log.sh \
  --files $installation_path/data/kafka/__cluster_metadata-0/00000000000000000000.log \
  --cluster-metadata-decoder

6.3 Full Platform Recovery Sequence

For complete platform recovery after major failure:

# Phase 1: Infrastructure (run on all nodes)
./scripts/manage.sh start opensearch mongodb kafka

# Phase 2: Wait for infrastructure to stabilize
sleep 120

# Verify infrastructure health on each node
./scripts/manage.sh status opensearch mongodb kafka

# Phase 3: Supporting services
./scripts/manage.sh start haproxy td-agent

# Phase 4: Application services
./scripts/manage.sh start sm analytics-manager ingestion scheduler
./scripts/manage.sh start webui api cbom

# Phase 5: Final verification
./scripts/manage.sh status

7. Performance Troubleshooting

7.1 Resource Utilization Analysis

# CPU usage by process
top -b -n 1 | head -20

# Memory usage
free -h
ps aux --sort=-%mem | head -10

7.2 OpenSearch Performance Issues

High Query Latency:

# Enable slow query log
curl -k --cacert $CA_CERT --cert $CLIENT_CERT --key $CLIENT_KEY \
  -X PUT 'https://127.0.0.1:9200/_all/_settings' \
  -H 'Content-Type: application/json' -d '{
    "index.search.slowlog.threshold.query.warn": "10s",
    "index.search.slowlog.threshold.query.info": "5s"
  }'

# Check hot threads
curl -k --cacert $CA_CERT --cert $CLIENT_CERT --key $CLIENT_KEY \
  'https://127.0.0.1:9200/_nodes/hot_threads'

# Check pending tasks
curl -k --cacert $CA_CERT --cert $CLIENT_CERT --key $CLIENT_KEY \
  'https://127.0.0.1:9200/_cluster/pending_tasks?pretty'

7.3 MongoDB Performance Issues

# Important: 
#   - Replace </path/to/installation> with actual path
#   - Replace "kf-agilesec.internal" with your actual domain name
#   - Replace "<Path to installer-directory>" with actual path
#   - Replace "<mongodb-node-hostname>" with actual internal mongodb hostname

export installation_path=</path/to/installation>
export analytics_internal_domain="kf-agilesec.internal" # default: kf-agilesec.internal. Change it to match your actual domain
export mongodb_host=<mongodb-node-hostname> # Example: backend-1.kf-agilesec.internal

// Go to mongosh
$installation_path/bin/mongosh \
  --tls \
  --tlsCAFile $installation_path/certificates/ca/agilesec-rootca-cert.pem \
  --tlsCertificateKeyFile $installation_path/certificates/$analytics_internal_domain/admin-user-combo-cert-key.pem \
  --authenticationMechanism MONGODB-X509 \
  --host $mongodb_host" \
  --port 27017

// Enable profiling for slow queries
db.setProfilingLevel(1, { slowms: 100 })

// Check slow queries
db.system.profile.find().sort({ ts: -1 }).limit(10).pretty()

// Check current operations
db.currentOp({ "secs_running": { "$gt": 5 } })

// Server status
db.serverStatus()

7.4 Kafka Performance Issues

# Important: 
#   - Replace </path/to/installation> with actual path
#   - Replace "kf-agilesec.internal" with your actual domain name
#   - Replace "<kafka-broker-hostname>" with your actual node domain name
export installation_path=</path/to/installation>
export analytics_internal_domain="kf-agilesec.internal" # default: kf-agilesec.internal. Change it to match your actual domain
export kafka_bootstrap_host=<kafka-broker-hostname> # Example: backend-1.kf-agilesec.internal

export JAVA_HOME="$installation_path/bin/java"
test -d "$installation_path/tmp" || {
  echo "$installation_path/tmp does not exist. Verify the installation path."
  exit 1
}

umask 077
work_dir="$(mktemp -d "$installation_path/tmp/kafka-health.XXXXXX")"
trap 'rm -rf "$work_dir"' EXIT

ssl_properties="$work_dir/ssl.properties"

cat > "$ssl_properties" <<EOF
security.protocol=SSL
ssl.truststore.type=PEM
ssl.truststore.location=$installation_path/certificates/ca/agilesec-rootca-cert.pem
ssl.keystore.type=PEM
ssl.keystore.location=$installation_path/certificates/$analytics_internal_domain/shared-client-combo-cert-key.pem
ssl.endpoint.identification.algorithm=https
EOF

# Check consumer lag
"$installation_path/services/kafka/bin/kafka-consumer-groups.sh" \
  --bootstrap-server "$kafka_bootstrap_host:9092" \
  --command-config "$ssl_properties" \
  --describe --all-groups

# Check log segment sizes
du -sh $installation_path/data/kafka/*

8. Support Bundle Log Collection

8.1 Run command to generate support bundle logs

Create a support bundle script collect-logs.sh containing all relevant diagnostic information:

Important: Replace /installation/path with your actual installation directory path.

#!/usr/bin/env bash
set -euo pipefail

: "${installation_path:?Set installation_path before running this script}"

umask 077
bundle_parent="$installation_path/tmp"
test -d "$bundle_parent" || { echo "$bundle_parent does not exist"; exit 1; }

bundle_dir="$(mktemp -d "$bundle_parent/support-bundle.XXXXXX")"
archive="${bundle_dir}.tar.gz"

mkdir -p "$bundle_dir/system" "$bundle_dir/logs" "$bundle_dir/config"

{
  echo "date=$(date -Is)"
  echo "hostname=$(hostname -f 2>/dev/null || hostname)"
  uname -a
  cat /etc/os-release
  uptime
  free -h
  df -h "$installation_path"
} > "$bundle_dir/system/system_info.txt" 2>&1

"$installation_path/scripts/manage.sh" list > "$bundle_dir/system/service_list.txt" 2>&1 || true
"$installation_path/scripts/manage.sh" status > "$bundle_dir/system/service_status.txt" 2>&1 || true

while IFS= read -r -d '' log_file; do
  rel="${log_file#$installation_path/}"
  mkdir -p "$bundle_dir/$(dirname "$rel")"
  tail -5000 "$log_file" > "$bundle_dir/$rel" 2>/dev/null || true
done < <(find "$installation_path/logs" -type f \( -name '*.log' -o -name '*.out' \) -not -path '*/archive/*' -print0 2>/dev/null)

sanitize_file() {
  sed -E \
    -e '/-----BEGIN [A-Z ]*PRIVATE KEY-----/,/-----END [A-Z ]*PRIVATE KEY-----/c\<redacted private key>' \
    -e 's/((password|passwd|secret|token|credential|api[_-]?key)[^=:#]*[=:#][[:space:]]*).*/\1<redacted>/Ig' \
    "$1" > "$2" 2>/dev/null || true
}

if [ -d "$installation_path/config_envs" ]; then
  while IFS= read -r -d '' conf; do
    rel="${conf#$installation_path/}"
    mkdir -p "$bundle_dir/$(dirname "$rel")"
    sanitize_file "$conf" "$bundle_dir/$rel"
  done < <(find "$installation_path/config_envs" -maxdepth 1 -type f -print0 2>/dev/null)
fi

[ -f "$installation_path/scripts/.vars" ] && \
  sanitize_file "$installation_path/scripts/.vars" "$bundle_dir/config/scripts.vars.sanitized"

tar -czf "$archive" -C "$(dirname "$bundle_dir")" "$(basename "$bundle_dir")"
rm -rf "$bundle_dir"

echo "Support bundle created: $archive"

Run it:

# Important: 
#   - Replace </path/to/installation> with actual path

chmod 700 collect-logs.sh
export installation_path=</path/to/installation>
./collect-logs.sh

8.2 Files to Include

Get the bundle logs file from the /tmp directory after running the script. It includes:

Category	Files
All Platform Logs	`$installation_path/logs/*.log`
Configuration	`$installation_path/config_envs/*` (sanitized)
System Info	OS version, memory, disk, uptime

8.3 Sanitizing Sensitive Information

Important: Before sharing logs, remove:

Passwords and secrets
API keys and tokens
Private keys and certificates
Personal information
Private IPs

8.4 Submitting Support Bundles

Create the support bundle using the procedure above
Verify no sensitive information is included
Upload to secure file sharing as directed by support
Include ticket number and description of the issue