BYOC Approach 2: BYOC All Certificate Material

The following steps must be performed for CA chain replacement pre-installation:

1. Place the CA chain on each node. Copy agilesec-rootca-cert.pem to $installer_dir/certificates/ca/agilesec-rootca-cert.pem on every node. This is the filename and path the installer expects, so no configuration changes are required.

2. Protect the CA chain with chmod 400:

   chmod 400 agilesec-rootca-cert.pem
   

3. Remove unneeded files generated by generate_certs.sh. The installer-generated CA private key (agilesec-rootca-key.pem) and serial file (agilesec-rootca-cert.srl) are not required once the Root CA is replaced. Delete both files from $installer_dir/certificates/ca after completing CA replacement.

4. Install the CA into the system trust store with tune.sh. On each node, run tune.sh to install the updated root CA into the node trust store:

   cd $installer_dir
   sudo ./scripts/tune.sh -u <user>
   

Perform these steps on all backend nodes, primary frontend (frontend-1), scan nodes, and the coordinator node:

1. Place the updated CA chain in AgileSec’s installed location $agilesec_installation_dir/certificates/ca/agilesec-rootca-cert.pem.

2. Copy the updated CA chain to the installer directory (tune.sh reads from this location) at $installer_dir/certificates/ca/agilesec-rootca-cert.pem.

3. Protect both copies with chmod 400.

4. The installer-generated CA private key (agilesec-rootca-key.pem) and serial file (agilesec-rootca-cert.srl) are not required once the Root CA is replaced. Remove these unneeded files, if present.

5. Install the CA into the system trust store with tune.sh. On each node, run tune.sh to install the updated root CA into the node trust store:

   cd $installer_dir
   sudo ./scripts/tune.sh -u <user>

6. Update the Root CA trust bundle copy maintained by OpenSearch under its service directory $agilesec_installation_dir/services/opensearch/config/certs/agilesec-rootca-cert.pem.

7. Update the JVM default truststore. tune.sh only updates the OS-level trust store, not the JVM's default cacerts truststore. Java service relying on the default JVM truststore for TLS trust will not pick up the new Root CA unless this update is done manually:

   cd $agilesec_installation_dir
   ./bin/java/bin/keytool -delete -alias "root.<analytics_internal_domain>" \
     -keystore bin/java/lib/security/cacerts \
     -storepass changeit
     
   ./bin/java/bin/keytool -noprompt -importcert -trustcacerts \
     -alias "root.<analytics_internal_domain>" \
     -file certificates/ca/agilesec-rootca-cert.pem \
     -keystore bin/java/lib/security/cacerts \
     -storepass changeit
   

8. After completing all post-install BYOC certificate replacements, restart the cluster.

When use_single_client_cert=true, generate_certs.sh generates a single shared client certificate used by all platform services for mTLS client authentication.

When use_single_client_cert=false, generate_certs.sh generates a separate client certificate per service. This provides stronger isolation and enables independent rotation of a single service credential without impacting others.

1. Add the indexing services' client certificate DN to roles_mapping.ymlon backend-1:

   1. Pre-install BYOC: Update $installer_dir/templates/opensearch/roles_mapping.yml

   2. Post-Install BYOC: Update the post-install OpenSearch roles mapping, not the installer directory, $agilesec_installation_dir/services/opensearch/config/opensearch-security/roles_mapping.yml

   3. Add the following to roles_mapping.yml:

      agilesec_ingest_role:
        reserved: false
        backend_roles:
          - "agilesec_ingest_role"
        # List of client cert CNs
        users:
        - "CN=indexing-service-client.<analytics_internal_domain>,<client_certificate_subject>"
      {{OPENSEARCH_INGESTION_USERS}}
        description: "Data ingestion"
      

2. Create a monitor only role in roles.yml on backend-1:

   1. Pre-Install BYOC: Update $installer_dir/templates/opensearch/roles.yml

   2. Post-Install BYOC: Update the post-install OpenSearch roles, not the installer directory, $agilesec_installation_dir/services/opensearch/config/opensearch-security/roles.yml

   3. Add the following to roles.yml:

        monitor_only:
          reserved: false
          cluster_permissions:
            - "cluster:monitor/nodes/info"
            - "cluster:monitor/main"
          index_permissions: []
      

3. Map HAProxy’s health-check identity to the monitor-only role at the end of roles_mapping.yml on backend-1:

   1. Pre-install BYOC: Update $installer_dir/templates/opensearch/roles_mapping.yml

   2. Post-Install BYOC: Update the post-install OpenSearch roles mapping, not the installer directory, $agilesec_installation_dir/services/opensearch/config/opensearch-security/roles_mapping.yml

   3. Add the following to roles_mapping.yml:

      monitor_only:
        reserved: false
        backend_roles: []
        hosts: []
        users:
          - "CN=haproxy-client.<analytics_internal_domain>,<client_certificate_subject>"
      

   4. Replace <client_certificate_subject> with the value set in multi_node_config.conf or single_node_config.conf.

4. If OpenSearch configuration steps above were updated post-install, run the following on backend-1:

   export AGILESEC_INSTALLATION_DIR=$agilesec installation dir
   export JAVA_HOME=$AGILESEC_INSTALLATION_DIR/bin/java
   export OPENSEARCH_HOME=$AGILESEC_INSTALLATION_DIR/services/opensearch
   
   $OPENSEARCH_HOME/plugins/opensearch-security/tools/securityadmin.sh -f $OPENSEARCH_HOME/config/opensearch-security/roles.yml \
      -t roles \
      -icl \
      -key $AGILESEC_INSTALLATION_DIR/certificates/<analytics_internal_domain>/opensearch-admin-user-key.pem \
      -cert $AGILESEC_INSTALLATION_DIR/certificates/<analytics_internal_domain>/opensearch-admin-user-cert.pem \
      -cacert $AGILESEC_INSTALLATION_DIR/certificates/ca/agilesec-rootca-cert.pem \
      -h backend-1.<analytics_internal_domain> -p 9200 -nhnv
   
   $OPENSEARCH_HOME/plugins/opensearch-security/tools/securityadmin.sh -f $OPENSEARCH_HOME/config/opensearch-security/roles_mapping.yml \
      -t rolesmapping \
      -icl \
      -key $AGILESEC_INSTALLATION_DIR/certificates/<analytics_internal_domain>/opensearch-admin-user-key.pem \
      -cert $AGILESEC_INSTALLATION_DIR/certificates/<analytics_internal_domain>/opensearch-admin-user-cert.pem \
      -cacert $AGILESEC_INSTALLATION_DIR/certificates/ca/agilesec-rootca-cert.pem \
      -h backend-1.<analytics_internal_domain> -p 9200 -nhnv
   

   1. Replace <analytics_internal_domain> with value from multi_node_config.conf or single_node_config.conf.

5. If performing pre-install BYOC, complete AgileSec installation before proceeding with the next steps.

6. Replace HAProxy frontend section fe_opensearch with the following on all backend and scan nodes:

   frontend fe_opensearch
       bind <backend | scan host>.<analytics_internal_domain>:50443
       mode tcp
       option tcplog
       default_backend be_opensearch
   

7. Replace HAProxy backend section be_opensearch with the following on all backend and scan nodes:

   backend be_opensearch
       mode tcp
       timeout connect 10s
       timeout server  10m
       timeout client  10m
       default-server inter 5s fastinter 2s downinter 10s fall 3 rise 2 on-marked-down shutdown-sessions
       server os1 backend-1.<analytics_internal_domain>:9200 track be_opensearch_healthcheck/ois1
       server os2 backend-2.<analytics_internal_domain>:9200 track be_opensearch_healthcheck/ois2 backup
   
   # Health-check-only backend - HAProxy authenticates as haproxy-client, never carries real traffic
   backend be_opensearch_healthcheck
       mode http
       balance roundrobin
       option httpchk
       http-check send meth GET uri /_nodes/_local
       http-check expect status 200
       default-server inter 5s fastinter 2s downinter 10s fall 3 rise 2 ca-file $agilesec_installation_dir/certificates/ca/agilesec-rootca-cert.pem crt $agilesec_installation_dir/certificates/<analytics_internal_domain>/haproxy-client-combo-cert-key.pem
       server ois1 backend-1.<analytics_internal_domain>:9200 ssl verify none sni str(backend-1.<analytics_internal_domain>) check
       server ois2 backend-2.<analytics_internal_domain>:9200 ssl verify none sni str(backend-2.<analytics_internal_domain>) check
   

   1. If you have more than two backend nodes, add additional server entries to be_opensearch and be_opensearch_healthcheck (see lines 18-19). Under be_opensearch, additional server entries for all non-local nodes must have backup at the end.

   2. Warning: If you have updated names for backend-1 and backend-2, make sure they are updated accordingly.

8. Restart HAProxy on all backend and scan nodes. (See On-Prem Managing Services.)

When use_single_client_cert=true, generate_certs.sh generates a single shared client certificate used by all platform services for administrative authentication to both MongoDB and OpenSearch.

When use_single_client_cert=false, generate_certs.sh generates a separate client certificate per service. This model provides stronger isolation and enables independent rotation of a single service credential without impacting others.

If performing post-install BYOC, perform the following steps:

1. Extract the certificate as a single-line base64 blob value (no PEM headers):

   openssl x509 -in idp-cert.pem -outform DER | openssl base64 -A
   

2. On all backend and frontend nodes running OpenSearch, update the file $agilesec_installation_dir/services/opensearch/config/opensearch-security/idp-metadata.xml and replace the certificate value in <ds:X509Certificate> with the base64 value extracted in step 1. Leave all other values unchanged.

   <EntityDescriptor xmlns="urn:oasis:names:tc:SAML:2.0:metadata" entityID="ui_idp">
       <IDPSSODescriptor protocolSupportEnumeration="urn:oasis:names:tc:SAML:2.0:protocol">
           <KeyDescriptor use="signing">
               <ds:KeyInfo xmlns:ds="http://www.w3.org/2000/09/xmldsig#">
               <ds:X509Data>
                   <ds:X509Certificate><---value from step 1---></ds:X509Certificate>
               </ds:X509Data>
               </ds:KeyInfo>
           </KeyDescriptor>
           <SingleSignOnService Binding="urn:oasis:names:tc:SAML:2.0:bindings:HTTP-Redirect" Location="https://agilesec.us-west-1.kf-agilesec.com/sso"/>
           <SingleLogoutService Binding="urn:oasis:names:tc:SAML:2.0:bindings:HTTP-Redirect" Location="https://agilesec.us-west-1.kf-agilesec.com/signout"/>
       </IDPSSODescriptor>
   </EntityDescriptor>
   

3. On the frontend node running API, replace the file $agilesec_installation_dir/services/api/dist/modules/v1/saml/config/privatekey.pemwith idp-enc-key.pem

4. On the frontend node running API, set the password for the encrypted private key idp-enc-key.pem in $agilesec_installation_dir/config_envs/api in the SAML_PASSPHRASE parameter.

5. Run the following commands on backend-1 to update idp-metadata config stored in MongoDB:

   METADATA_ESCAPED=$(sed -e 's/\\/\\\\/g' -e 's/"/\\"/g' $agilesec_installation_dir/services/opensearch/config/opensearch-security/idp-metadata.xml | awk '{printf "%s\\n", $0}')
   $agilesec_installation_dir/bin/mongosh "mongodb://backend-1.<agilesec_internal_domain>:27017/platform" \
     --tls \
     --tlsCAFile $agilesec_installation_dir/certificates/ca/agilesec-rootca-cert.pem \
     --tlsCertificateKeyFile $agilesec_installation_dir/certificates/<agilesec_internal_domain/[admin-user-combo-cert-key.pem | mongodb-admin-user-combo-cert-key.pem] \
     --authenticationMechanism MONGODB-X509 \
     --authenticationDatabase '$external' \
     --eval 'db.settings.updateOne({ name: "idp-metadata" }, { $set: { value: "'"${METADATA_ESCAPED}"'" } })'
   
   

6. View idp-metadata in MongoDB and confirm updates were effective:

   $agilesec_installation_dir/bin/mongosh "mongodb://backend-1.<agilesec_internal_domain>:27017/platform" \
     --tls \
     --tlsCAFile $agilesec_installation_dir/certificates/ca/agilesec-rootca-cert.pem \
     --tlsCertificateKeyFile $agilesec_installation_dir/certificates/<agilesec_internal_domain/[admin-user-combo-cert-key.pem | mongodb-admin-user-combo-cert-key.pem] \
     --authenticationMechanism MONGODB-X509 \
     --authenticationDatabase '$external' \
     --eval 'printjson(db.settings.findOne({ name: "idp-metadata" }))'
   

7. Restart OpenSearch on frontend and backend nodes. (See On-Prem Managing Services.)

8. Restart API service on frontend nodes. (See On-Prem Managing Services.)