Platform Tuning Guide

This guide walks through tuning AgileSec to handle large data volumes and/or large numbers of sensors.

Test Profiles

For full tuning coverage and validation, tune and validate against both test profiles. Each test profile stresses different parts of the AgileSec data pipeline.

Tuning Prioritization Strategy

OS and HAProxy tuning should be applied as a baseline, regardless of symptoms. Other tuning may be prioritized and applied based on platform symptoms as follows.

Baseline Tuning: OS and HAProxy

Apply OS-Level Kernel and HAProxy tuning preventatively as a baseline, even before problems turn up. Every node should have headroom for file descriptors and connections preventatively.

Node topology

1. OS-level kernel tuning

The following OS tuning steps apply to all frontend nodes (HAProxy) and backend nodes (Fluentd and OpenSearch).

Take the following into account for OS-level tuning:

• vm.max_map_count matters most for OpenSearch (Lucene mmap)

• The file descriptor and backlog settings matter for HAProxy and Fluentd.

• On the backend nodes, size nofile and fs.file-max against Fluentd and OpenSearch running together, not either service alone – see Node topology above.

1. Add the following to /etc/sysctl.conf:

   vm.max_map_count = 262144
   fs.file-max = 2097152
   net.core.somaxconn = 16384
   net.ipv4.tcp_max_syn_backlog = 16384
   

2. Run sysctl -p to apply without a reboot.

3. Add the following to /etc/security/limits.conf:

   <os user> - nofile   262144
   <os user> - nproc    262144
   <os user> - memlock  unlimited
   

4. limits.conf only applies to new login sessions, not already-running processes. systemd services do not always inherit PAM limits from limits.conf. For systemd-managed services (HAProxy, Fluentd, OpenSearch), also set the limit under [Service] in the unit's override file:

   [Service]
   LimitNOFILE=262144
   

2. HAProxy

Update services/haproxy/haproxy.cfg on all frontend and backend nodes.

1. Raise maxconn (both files, global and defaults)

   global
       maxconn     <sized to target connection count – see note below>
       ulimit-n    <2x maxconn>
   
   defaults
       maxconn     <below global, with headroom for other backends>
   

   2. Note: maxconn by default is 4000/3000, process-wide across every frontend and backend in that instance – the first ceiling hit under load. No ulimit-n is set by default.

2. Add L7 health check + timeout overrides to fluentd_backend. Add these lines to the backend fluentd_backend block:

   timeout connect 10s
   timeout check   30s
   timeout server  2m
   timeout client  2m
   option httpchk GET /
   http-check expect status 200
   default-server inter 5s fall 3 rise 2 on-marked-down shutdown-sessions check port 6444
   

   1. Note: This replaces the existing default-server line already present in the block. Without timeouts, 10s timeouts are inherited from defaults, which are too short and can cause 503s on slower VMs or under heavy load.

3. Raise timeouts on opensearch_backend / be_opensearch

   timeout connect 10s
   timeout server  10m
   timeout client  10m
   

1. 1. Note: OpenSearch bulk indexing/search/forcemerge/snapshot ops can run past the 1 minute default under heavy load.

3. Fluentd

Fluentd Tuning Strategy: Increasing Workers vs. Tuning Buffer Settings

Use signals from monitor_agent (port 24230) to decide whether add workers or tune buffer settings.

Before tuning, check both signals from monitor_agent (port 24230) on each backend node.

1. From backend node terminals, check the buffer queue / retry stats:

   curl http://localhost:<FLUENTD_MONITOR_PORT>/api/plugins.json
   

2. Look at the output plugin entry for:

   1. buffer_queue_length – chunks currently queued, waiting to flush. Climbing steadily = buffer backing up.

   2. buffer_total_queued_size – total bytes queued across those chunks.

   3. retry_count – failed flush attempts on the current chunk. Climbing alongside queue length = downstream (Kafka/OpenSearch) rejecting writes, not just a slow flush.

3. To filter to a single plugin instead of the full list:

   curl "http://localhost:<FLUENTD_MONITOR_PORT>/api/plugins.json?type=kafka2"
   

4. To extract only OpenSearch output plugins actually buffering and writing data:

   curl -s http://localhost:<FLUENTD_MONITOR_PORT>/api/plugins.json \
     | jq '.plugins[] | select(.plugin_id | test("opensearch")) | {plugin_id, buffer_queue_length, buffer_total_queued_size, retry_count}'
   

   1. This excludes filter/tag-routing plugins and stdout debug outputs.

Increase Workers

Add workers when a single process cannot accept/process connections fast enough. One worker means one accept loop handling all concurrent connections.

On backend nodes:

1. Uncomment and set workers in <system> in services/td-agent/etc/td-agent.conf:

   <system>
     ...
     workers "#{ENV['FLUENTD_WORKERS'] || 4}"
   </system>
   

2. Move healthcheck and monitor_agent into a <worker 0> block. Pinning sources to worker 0 is required, not optional, before raising worker count:

   <worker 0>
     <source>
       @type http_healthcheck
       @id healthcheck
       bind 0.0.0.0
       port "#{ENV['FLUENTD_HEALTH_CHECK_PORT'] || 24220}"
     </source>
     <source>
       @type monitor_agent
       bind 0.0.0.0
       port "#{ENV['FLUENTD_MONITOR_PORT'] || 24230}"
       tag debug.monitor
       emit_interval "#{ENV['FLUENTD_MONITOR_INTERVAL_SECONDS'] || 1800}"
     </source>
   </worker>
   

   1. By default, sources sit at top level. If sources are not pinned to worker 0, when workers is uncommented, every worker will try to bind the same ports, causing failure to start or other misbehavior.

Tune Buffer Settings

Update buffer settings in buf.conf / buf_new.conf when monitor_agent shows buffer queue length climbing, or when you see any of these downstream signals:

• OpenSearch 429s / circuit breaker trips: check GET _nodes/stats/thread_pool or GET _cat/thread_pool?v for write queue rejections.

• retry_count climbing alongside buffer_queue_length in the monitor_agent/jq output: distinguishes "flush is just slow" (queue grows, retries stay 0) from "downstream is actively rejecting" (both climb).

• td-agent logs showing buffer flush took long time or failed to flush the buffer warnings, which often surface the actual downstream exception.

On backend nodes:

1. Update buf.conf when sending data directly to Fluentd:

   @type "file"
   flush_at_shutdown true
   overflow_action block
   chunk_limit_size          "#{ENV['FLUENTD_BUFFER_CHUNK_LIMIT_SIZE'] || '16M'}"
   total_limit_size          "#{ENV['FLUENTD_BUFFER_TOTAL_LIMIT_SIZE'] || '5GB'}"
   flush_thread_count        "#{ENV['FLUENTD_BUFFER_FLUSH_THREAD_COUNT'] || 2}"
   flush_mode                "#{ENV['FLUENTD_BUFFER_FLUSH_MODE'] || 'interval'}"
   flush_interval            "#{ENV['FLUENTD_BUFFER_FLUSH_INTERVAL'] || '5s'}"
   queued_chunks_limit_size  "#{ENV['FLUENTD_QUEUED_CHUNKS_LIMIT_SIZE'] || 128}"
   retry_type exponential_backoff
   retry_wait                "#{ENV['FLUENTD_BUFFER_RETRY_WAIT'] || '10s'}"
   retry_max_interval        "#{ENV['FLUENTD_RETRY_MAX_INTERVAL'] || '3m'}"
   retry_forever             "#{ENV['FLUENTD_RETRY_FOREVER'] || true}"
   

2. Update buf_new.conf when sending data through ingestion service:

   flush_thread_count "#{ENV['FLUENTD_BUFFER_FLUSH_THREAD_COUNT'] || 2}"
   

4. OpenSearch

Updating OpenSearch’s refresh interval to 30 seconds (30s) delays finding visibility but does not affect policy execution, which forces an explicit refresh.

From AgileSec’s Advanced Dashboards → Dev Tools, update the refresh interval:

PUT kf-agilesec_com-isg-event-*/_settings
{
  "index": { "refresh_interval": "30s" }
}