Peer Systems Operations

ENTERPRISE

The following provides information on managing Peer Connectors using the Peer Systems overview (System Functions → Peer Systems).

For more conceptual information about Peer Systems, see Peer Systems Overview.

Overview

In the Peer Systems Overview you can:

• Modify global settings like enabling or disabling incoming connections.
• View a list of configured known EJBCA Peer Systems that this instance can connect to (Peer Connectors) and their current connection status.
  • Connection URL, i.e. https://remotehost:8443/ejbca/peer/v1
  • Connection pool status, connections in use / ready / max / queued
  • Synchronization status, if peer connection is started
  • Client and server TLS certificate information
  • Role used for incoming connection, if enabled
• View a list of systems that have connected to this instance recently (Incoming Connections).

Additionally, links are available to the relevant authentication settings for outgoing connections (AuthenticationKeyBinding) and incoming connections (Administrator Role).

Outgoing Connections 

Outgoing connections are allowed by default and can be disabled by administrators authorized to /peer/modify recursively.

A Peer Connector is a representation of a remote EJBCA instance and can be referenced by other components like Publishers and Services. Each Peer Connector maintains a pool of outgoing re-usable (HTTP Keep-Alive) connections and new connections are only created when needed.

To authenticate to a peer, you need to configure an Remote Authenticator. The Remote Authenticator is the EJBCA instance identity and consists of a client SSL X509 certificate and a key pair in one of the instance's Crypto Tokens. Ensure that the Remote Authenticator is configured to trust the peer's server SSL certificate or the connection will fail. Remote Authenticator settings are only read when a Peer Connector's connection pool is started and if the client certificate is updated this will not take effect until a connection pool is Reset.

For each configured Peer Connector, you can see the human-readable name, connection endpoint and current connection status. You can Ping any of the peers to check the connectivity and the result is the round-trip time from and to the application over the secure connection (giving a more accurate view of the actual round trip time than a network ping using the ICMP protocol).

Only the first connection to a peer using the same client certificate will be subject to a full authentication check and subsequent requests will share the same credential. This means that an initial Ping request to a peer system will be significantly slower than the next one.

Setting up an Outgoing Peer Connection

For more information on how to add an Outgoing Peer Connection, see:

→ How to set up an Outgoing Peer Connection

Setting up an outgoing peer connection assumes that an Remote Authenticator has already been set up for this outgoing connection. For instructions, see Setting up a Remote Authenticator.

Edit, Clone or Delete a Peer Connector 

From the Peer Systems overview, click Edit for an existing Peer Connector. In the view of the Peer Connector, you can modify and save the existing name, URL or state. In the Edit view, you can also clone or delete the current object.

Incoming Connections 

Incoming connections are by default not allowed and can be enabled by administrators authorized to /peer/modify recursively.

In the list of incoming connections, you can see systems that have successfully connected to the EJBCA instance with a client certificate trusted by the application server's SSL configuration. If the client certificate presented by the connecting system is already part of an Administrator Role, the name of this role is shown.

Authorization of incoming connections 

• By default, EJBCA requires an Admin TLS certificate to be present in the database. If you are setting up an external VA or RA, you probably want to be able to connect a Peer from the CA to the VA/RA without having to import the CAs TLS client certificate (from an Authentication Key Binding) into the VA/RA. This is done by setting web.reqcertindb=false in conf/web.properties. See OCSP Responder Management.
• If no Administrator Role exists for the connecting system's client certificate, click Create new role to setup a new Administrator Role for the incoming client certificate and a relevant set of access rules. You can also select to add the incoming client certificate to an existing Administrator Role.
• If an Administrator Role exists for the connecting system's client certificate, click Modify role to change the relevant set of access rules for the Administrator Role.

This provides a simplified view of EJBCA's normal authentication and authorization management, and the created Administrator Role can be edited just like any other Administrator Role.

Management Operations for an EJBCA Peer System 

Once an EJBCA instance has been authorized to connect and manage another EJBCA instance, operations on the peer can be performed through the Admin GUI of the authorized instance.

The administrator initiating these operations needs to be authorized to the /peer/view /peer/manage access rules and additionally any relevant CA.

Publishing using Peer Connectors 

The Publishers Overview is used to propagate certificate or CRL information to a peer system. The PeerPublisher implementation allows this information to be pushed to a configured Peer Connector.

The connecting system needs to be authorized to the /peerincoming /peerpublish/writecert /peerpublish/writecrl /ca/[CAName] access rules to be able to push both certificate and CRL data.

Certificate Synchronization to a VA

In a setup where an EJBCA CA instance (or a cluster) uses external EJBCA VA/OCSP responders, revocation information needs to be propagated from CA to VA. Information about newly issued certificates and revocation updates are normally sent using a (Peer)Publisher, but the first time a new VA is connected the current state of all previously issued certificates needs to be pushed to the VA.

In the overview of Peer Connectors:

1. Click Manage for the peer connector representing the VA and select the Certificate Data Synchronization tab. 
2. Configure the relevant subset of information to synchronize.
3. Click Start to initiate the synchronization as a background task. 
   The progress can be followed either in this view or in the Peer Systems overview.

The subset of revocation information to send affects how database queries are performed. Depending on your database it might be faster to only synchronize everything or only one subset of data at the time.

The connecting system needs to be authorized to the /peerincoming /peerpublish/readcert /peerpublish/writecert /ca/\[CAName\] access rules to be able to check synchronization data and push missing or outdated certificate entries.

Troubleshoot Synchronization Problems

Once the synchronization has completed, you are proposed to download a report containing all items which could not be synced. The example below shows what the file can look like if the CA was not authorized to write to the VA's database.

implementation: org.ejbca.peerconnector.PeerSyncReport
creationDate: 2019-08-28 13:52
failureCount: 1
failures:
-   fingerprint: da28b65061b753f0059ac85c4917332d27f14f0d
    subjectDn: CN=OCSPTest,O=AnaTom,C=SE
    issuerDn: CN=TEST
    errorType: org.cesecore.authorization.AuthorizationDeniedException
    errorMessage: Not authorized to publish certificates for CN=TEST.

Renewal of Remote Internal Key Bindings 

In a setup where an EJBCA CA instance (or a cluster) uses external EJBCA VA/OCSP responders, a CA delegates signing of OCSP responses to an OCSP signing certificate (configured as a OcspKeyBinding) at the VA. The OCSP signing certificate should be short-lived and to make renewal easier, this is available as a remote management operation on the CA.

In the overview of peer connectors:

1. Click Manage for the peer connector representing the VA and select the Remote Key Bindings tab.
2. Click Renew to generate a new certificate. Optionally you can also select to generate a new key pair for the next OCSP signing certificate.

Renewal of remote Internal Key Bindings can be automated using a Remote Internal Key Binding Updater.

The connecting system needs to be authorized to the /peerincoming /internalkeybinding/view/[Internal Key Binding]/internalkeybinding/modify/[Internal Key Binding Name] and /cryptotoken/view/[Crypto Token Name] access rules to be able to renew an Internal Key Binding certificate. Additionally /cryptotoken/keys/generate/[Crypto Token Name] is required if key renewal should be allowed.

When renewing the signing certificate, deactivate publishing in the OCSP certificate profiles in order to avoid overlaps during renewal.

Serving Registration Authority Requests via Peer Connections 

The EJBCA Registration Authority (RA) can be run either locally or remotely using long hanging Peer Connections from CA to RA.

The CA will connect to the RA and listen for requests, grab either the first pending or wait for one, and return the request to the CA. Once the CA is done processing the request, it will reconnect to the same RA and deliver the result and then wait for a new request.

Configuring the Peer Connector to Fetch Requests from the RA 

When setting up an outgoing Peer Connector on the CA, pay attention to the "Incoming requests" options.

• Process Incoming requests: Enabling this option will make the CA establish long hanging connections to the RA and fetch requests.
• Minimum parallel requests: Minimum number of long hanging connections that will wait for the requests on the RA when the RA is idle.
• Maximum parallel requests: Maximum number of long hanging connections that will process requests from the RA when the RA is fully utilized.

The CA will automatically throttle up and down the number of long hanging connections to each RA based on utilization. If you want rapid responses to bursts of requests after a long period of idle time, you should increase the minimum. If you want to limit the load the an RA can inflict on a CA when the RA is fully utilized, you should lower the maximum.

Authorizing which Types of Requests from the RA to Server

Once the CA is able to connect to the RA, there is an option in the outgoing Peer Connector list to Authorize requests. The RA authenticates to the CA with its server side TLS certificate and using Authorize requests will allow you to create a new role (suitable action for first RA) or assign this certificate to an existing administrator role (suitable action for the rest of the RAs).

Once the server side TLS certificate belongs to a role, you are shown a simplified view of access rules relevant to RA requests processing. Since requests from the RA over the long hanging Peer Connections are authorized with a common subset of access rules from an adminstrator authenticated to the RA and the RA itself, this will limit the maximum access any administrator can have when performing requests over the RA. Think of it as context aware authorization.

Authorizing the CA to Fetch Requests from the RA 

Similar to authorization in Authorization of incoming connections, you need to grant the CA rights on the RA to fetch pending requests. Modify the matching role and ensure that Accept long hanging connections (External RA) is selected.

Load Balancing Considerations

The Peer Connector configuration is stored in the database shared by all nodes in the CA cluster. Each node where the Authentication Key Binding is usable (e.g. the Crypto Token with the private key is active) will try to set up connections to the RA end-point.

A random CA node that has connected to the RA will pick up a new request to ensure even distribution of load. When all CA nodes have the same number of open long hanging connections, there is an increased probability that CA nodes with more idle long hanging connections will pick up the next request.

When a CA node uses all current connections to an RA, more connections will be created until "Maximum number of long hanging connections" is reached. If the deployment has asymmetric network bandwidth between CA nodes and an RA, high latency nodes will end up keeping more connections open to achieve the same throughput as low latency nodes at the cost of increased response times.