AgileSec Architecture and Components

High-Level Architecture

AgileSec is an event-driven platform designed for scalability, extensibility, and enterprise security. The architecture is built around a central Kafka Cluster that functions as the primary Data and Events Pipeline, providing loose coupling and high-throughput asynchronous communication between components.

The platform is logically organized into three primary layers, supported by datastores for operational and findings data.

Client Access Layer

This layer provides all external interfaces to the platform, including the Web UI and Web Backend (Node.js), Advanced Dashboards, REST API / API Server, Ingestion Service, Load Balancer, and SSO/SAML integration. It serves as the secure entry point for end users, administrators, and automated clients.

Core Services

The always-on orchestration and intelligence layer consists of the Scheduler, Manager, Secrets Management (KMS/Vault integration), and Indexing services. These services manage job lifecycle and various system events, process scan results, handle data transformation and enrichments, and securely manage encryption keys.

On-Demand and Scheduled Execution Layer

This layer is responsible for the execution of all scanning and processing jobs in a secure and scalable manner. The execution environment — where jobs actually run — varies depending on the deployment model and job type, while scan targets are always external customer systems (cloud assets, on-premises infrastructure, containers, IaC repositories, etc.).

Scan jobs

Scan jobs can be initiated on-demand (via the Web UI, REST API, or remotely using the Unified Sensor CLI) or run automatically on predefined schedules.

• Platform Scans - Executed centrally by the AgileSec platform.

  • In Kubernetes deployments, jobs run in short-lived, isolated sandboxed pods with restricted privileges, network policies, and resource limits for strong security isolation and horizontal scalability.

  • In on-premise deployments, jobs run on backend nodes, or dedicated scan nodes configured within the AgileSec platform.

• API Scans - Also executed centrally by the AgileSec platform, but triggered programmatically via the REST API. They use the same execution environment as Platform Scans.

• Remote Scans - Executed using Unified Sensor framework deployed as CLI in the customer's environment. Jobs run locally against targets. Using this model ideal for air-gapped networks or scenarios where data must not leave the customer's perimeter.

All types of scan jobs leverage the Unified Sensor Framework with pluggable sensors, ensuring consistent output format.

Policies Execution and Findings Processing Jobs

These jobs are always executed centrally on the platform. They run in the same secure platform execution environment as Platform scans, consuming findings from OpenSearch, applying policies, perform de-duplication, transformation, auto-resolution, and publishing enriched results.

For both Kubernetes and on-premises environments, jobs are dispatched to the appropriate execution environment via events from the Scheduler over the Kafka pipeline, enabling resilient, asynchronous, and scalable processing.

Supporting Infrastructure

Kafka Cluster

A durable, high-throughput event bus that serves as the central communication backbone. It carries job requests, execution events, raw findings, processed findings, control messages, and system events, ensuring resilient and asynchronous processing across platform.

Datastores

• Operational Datastore – MongoDB cluster serving as the primary source of truth for all operational and configuration data, including:

  • User accounts and platform-level configuration

  • System administration settings

  • Scheduling information for recurring and on-demand scans

  • Sensor configurations and execution history

  • Scan history and job state

  • Configurable policies

• Findings Datastore - OpenSearch cluster dedicated for storing, indexing and querying enriched findings data.

  • Indexing service – Consumes raw findings from Kafka, performs schema validation, applies transformations, and indexes them into OpenSearch for search and analytics.

  • Advanced Dashboards – SSO-protected OpenSearch Dashboards providing customers with powerful, self-service tools to perform advanced searches, create custom visualizations, and build tailored reports for their findings.

Component Layer Details

This section provides in-depth descriptions of the major components of AgileSec, organized by the primary layers introduced in the high-level architecture.

Client Access Layer

Web UI and Web Backend

The primary browser-based interface for end users, security teams, and administrators. The Web UI offers intuitive navigation for scan management (creating, configuring, and triggering scans), on-demand scan initiation, schedule management, system administration, and interactive findings exploration through pre-built dashboards and exploratory data analysis tools. The Node.js Web Backend handles server-side logic, session management, authentication coordination, and data retrieval for the UI.

Advanced Dashboards

SSO-protected OpenSearch Dashboards instance that provides power users and analysts with self-service capabilities. Users can perform advanced full-text searches, apply complex filters, aggregate data, create fully custom visualizations, save dashboards, and generate reports directly over their indexed findings data.

REST API and API Server

Programmatic interface used by the Web UI and external clients. It provides endpoints for scan submission, status checking, and results retrieval. All endpoints are protected by authentication and authorization, requiring a valid access token.

Ingestion Service

Dedicated high-volume ingestion endpoints designed to receive data streams from sensors. It performs token authentication, tenant identification, and publishes data to the Kafka pipeline for further processing.

Load Balancer

Distributes incoming traffic across service instances, provides TLS termination, and enables high availability and horizontal scaling for the Client Access Layer and all downstream services.

SAML SSO

Enterprise authentication integration with customer Identity Providers (IdP). Enables single sign-on for the Web UI, and Advanced Dashboards, with support for role mapping.

Core Services

The Core Services layer consists of always-on, long-running microservices that form the orchestration and intelligence backbone of the platform.

Scheduler

Always-on orchestration service responsible for initiating jobs and protecting the execution infrastructure from overload.

The Scheduler receives all job requests exclusively through Kafka topics. Job request events are published to Kafka by the Client Access Layer components (Web Backend for UI-triggered jobs, API Server for API-triggered jobs, or by internal recurring schedule logic).

Key responsibilities include:

• Consuming job request events from Kafka and evaluating them for execution.

• Enforcing rate limiting and concurrency controls to prevent overwhelming the underlying execution environment, ensuring stable performance under high load.

• Supporting job prioritization based on configured priority levels (High, Medium, Low) for scan jobs, allowing critical scans to be processed ahead of lower-priority ones.

• Using deployment-specific mechanisms to start jobs:

  • In Kubernetes deployments — Creates short-lived, isolated sandboxed pods for job execution via the Kubernetes API using pre-defined pod templates.

  • In on-premises deployments — Directly assigns and executes jobs on backend nodes or dedicated scan nodes.

Manager

Always-on service responsible for full job lifecycle management and post-scan orchestration across all deployment models.

Key responsibilities include:

• Receiving and processing scan job requests — Consumes scan job request events published to Kafka by the API Server for API-initiated scans or UI-initiated scans.

• Submitting jobs for execution — Publishes execution-ready job events to Kafka for the Scheduler to pick up and initiate.

• Tracking job status — Consumes progress, completion, and error events from Kafka (published via the Ingestion Service from execution environments).

• Persisting job state — Updates and maintains job status, history, and metadata in the Operational Datastore throughout the job lifecycle.

• Invoking API callbacks — Upon job completion or status change, delivers asynchronous status updates and results via customer-configured webhook callbacks.

• Triggering post-scan processing — Upon successful scan completion, evaluates conditions and publishes events to Kafka to initiate Findings Processing jobs and Policy Execution jobs.

Secrets Management

Always-on service responsible for secure key management, token generation and encryption across the platform.

The Secrets Management service ensures that sensitive data is encrypted at rest using envelope encryption, with master keys managed in the deployment-appropriate secure store.

Key responsibilities include:

• Integration with deployment-specific key management systems — Interfaces with the appropriate external or local key store based on the deployment model:

  • AWS EKS deployments — Uses AWS KMS for generating and storing customer master keys.

  • Azure AKS deployments — Uses Azure Key Vault (AKV) for storing customer master keys.

  • On-premises deployments — Uses a local Java Keystore for storing customer master keys.

• Generation and storage of customer master keys — Generates Customer Master Keys (CMKs) in the configured key store and securely references them for platform use. CMKs always stay in keystore.

• Envelope encryption — Uses CMKs to encrypt Data Encryption Keys (DEKs). Encrypted DEKs are stored alongside the data in the Operational Datastore, while plaintext DEKs are never persisted.

• Secure key distribution — Provides plaintext DEKs to authorized services at runtime for encryption/decryption operations, then securely wipes them from memory.

• Key rotation support — Facilitates automated rotation DEKs and re-encryption of associated DEKs.

• Access control and auditing — Enforces least-privilege access to keys and logs all key operations for compliance and security auditing.

The Secrets Management service ensures that sensitive data is encrypted at rest using industry-standard envelope encryption, with master keys managed in the deployment-appropriate secure store to meet customer control and compliance requirements.

Indexing Service

Always-on service that consumes raw findings events from Kafka, performs schema validation, applies final normalization and transformations, and indexes them into the OpenSearch Findings Datastore.