Breadcrumbs

AgileSec Data Flows

High-level architecture and end-to-end data flow of a typical scan job. Scans are a critical function of AgileSec, enabling users to discover, process, inventory, and analyze cryptographic data throughout their environment.

High-Level Scan Lifecycle

A typical AgileSec scan has three main phases:

  1. Execute Scan – A user or automated process triggers a scan via the Web UI, REST API, schedule, or Unified Sensor CLI. The Unified Sensor Framework executes the scan against the target systems and produces raw findings.

  2. Process – The Ingestion Service receives the raw findings. Policies are applied and backend processes run to cleanse, de-duplicate, transform, and enrich the data. The final results are stored in the Findings Database.

  3. Review – Enriched findings can be viewed and analyzed via the AgileSec Web UI with built-in dashboards and explorer, Advanced Dashboards for custom visualizations, or programmatically through REST API queries.


Note: The underlying event-driven architecture uses Kafka to enable asynchronous, scalable processing.

High-Level Architecture: Unified Sensor Framework

AgileSec uses a Unified Sensor framework with individual sensor/connector plugins for consistency, flexibility, and scalability. The modular Unified Sensor framework separates target-specific logic from core scanning capabilities.

  • Sensor Plugin – Handles authentication, data retrieval, and initial extraction for the specific target being scanned. Its job is to only fetch the content for scanning.

  • Unified Sensor Core – Performs standardized scans (binary, source, library detection), enriches raw results with context and labels, and transmits them in a consistent format back to the platform via the Ingestion Service.

The Unified Sensor framework ensures:

  • Consistency – All scans produce uniformly formatted raw findings regardless of target type.

  • Extensibility – New sensor plugins can be added without changing the core.

  • Separation of Concern – Target specific logic is stored in sensor plugins, while core scanners, enrichment and transformations remain centralized in the core.

Scan Data Flow

The flow follows the same path for all Sensors operating as a unified sensor plugin. Connectors follow the same flow except they do not utilize the Host Sensor.

Scan Flow Description

  • Authenticate: Sensor establishes secure connection to target using provided access authentication (token, key, etc.).

  • Enumerate: Queries the target API to list repositories matching include/exclude patterns

  • Clone/Fetch:

    • Initial Scan: Performs a shallow clone of the specified branch to minimize bandwidth and storage

    • Incremental Scan: Fetches only the commits since the last scan and identifies changed files

  • Scan: For Sensors, all files downloaded from the target are passed to the Host Sensor to perform the following types of scans:

    • Binary Scan: Analyzes binary files to discover cryptographic objects.

    • Source Scan: Performs static analysis of source code files to detect cryptographic implementations, API calls, and embedded cryptographic objects.

    • Library Detection: Scans source code and dependencies to identify cryptographic libraries being used.

  • Transform: Normalizes and standardizes scan results into a consistent format for processing.

  • Enrich: Enhances scan result with additional metadata like labels.

  • Transmit: Sends the enriched scan results to Ingestion Service on Keyfactor AgileSec Platform for inventory and policy-based analysis.

Post-Scan Process and Review Data Flow

The findings pipeline processes raw scan results into enriched, actionable insights through an event-driven, asynchronous flow for easy review and consumption.

Post-Scan Findings Flow Description

  • Raw findings ingestion – Scans transmit raw findings to the Ingestion Service, which publishes them as events to the Kafka Cluster. The Kafka Cluster data pipeline distributes raw findings events to downstream consumers (Indexing Service instances) for parallel processing.

  • Transformation and indexing – Indexing service consumes raw findings events from Kafka, performs schema validation, applies final normalization and transformations, and indexes the data into the OpenSearch Cluster for search and analytics. All new findings are initially marked with a pending status.

  • Post-scan or scheduled processing – Findings Processors and Policies run asynchronously, triggered either immediately after a scan completes (post-scan) or on a predefined schedule:

    • Findings Processors read findings from OpenSearch, perform cleansing, de-duplication, enrichment, transformation, and auto-resolution (where configured), and write enriched findings back to OpenSearch.

    • Policies Execution: The AgileSec platform executes user-configured policies on the data stored in OpenSearch. Policies calculate a risk score for each finding and save it alongside the finding data in OpenSearch. At this point, findings are no longer considered pending, and each is assigned a score classification displayed as: compliant, high risk, medium risk, or low risk.

  • Review and consumption – Enriched and indexed findings are available for interactive review through the Client Access layer:

    • Web UI and Web Backend provide built-in dashboards and findings explorer.

    • Advanced Dashboards through OpenSearch Dashboards provides custom visualizations and reporting capabilities.

    • REST API and API Server enable programmatic queries.

Scan Execution Flow

See Scan Execution Flows for scan execution methodology and process flows.