Tutorial - Deploy EJBCA container to issue certificates to an Istio service mesh
Learn how to use a service mesh to issue mutual TLS certificates with EJBCA running in Kubernetes.
This tutorial shows you how to use EJBCA to issue mutual TLS certificates to a service mesh such as Istio. By integrating Istio with EJBCA, you get a PKI that complies with policy and supports PKCS11 for protecting the certificate authority (CA) key. EJBCA can provide certificates to a service mesh using the EJBCA Certificate Signing Request Proxy for K8s (EJBCA CSR Signer container).
Using the EJBCA CSR Signer is an alternative to using the cert-manager which EJBCA also integrates with, see Issuing Certificates to Kubernetes Services using cert-manager.
In this tutorial, you will learn how to:
- Deploy the EJBCA CSR Signer container
- Integrate the EJBCA CSR Signer with the EJBCA REST API
- Deploy an Istio service mesh
- Deploy the sample Istio Bookinfo application
- Check logs to validate certificates were issued
Prerequisites
- Before you begin, you need to create roles in EJBCA and set up an RA credential to be used by the EJBCA Certificate Signing Request Proxy for K8s (ejbca-csr-signer container) to authenticate to the EJBCA REST API. To learn how to configure EJBCA roles, follow the tutorial
- Additionally, you should have a basic understanding of how to use the Kubernetes command line tool kubectl.
Step 1 - Prepare the EJBCA CSR Signer container deployment
Deploy the EJBCA CSR Signer to integrate with the Kubernetes CSR-Signer-API which sends CSRs over to EJBCA using the REST API. CSRs from a service mesh are routed to the Kubernetes CSR-Signer-API to then get sent over to EJBCA for signing. Manual CSRs signing can also be done in the Kubernetes cluster.
To prepare for the EJBCA CSR Signer deployment, follow these steps:
- SSH to the MicroK8s test host that has EJBCA deployed and configured.
Create the csr-api directory to organize all the files for this tutorial:
CODE$ mkdir csr-api
Change directory to the csr-api directory:
CODE$ cd csr-api
Download the EJBCA CSR Signer release from GitHub which contains the Helm chart to deploy the container:
CODE$ curl -L https://github.com/Keyfactor/ejbca-k8s-csr-signer/releases/download/ejbca-csr-signer-1.0.3/ejbca-csr-signer-1.0.3.tgz -o ejbca-csr-signer-1.0.3.tgz
Unpack the archive file:
CODE$ tar xf ejbca-csr-signer-1.0.3.tgz
Using a text editor such as vim, create the credentials.yaml file:
CODE$ vim credentials.yaml
Add the following to the file:
CODE# Hostname to EJBCA server hostname: "ejbca-internal.ejbca-k8s" # Password used to protect private key, if it's encrypted according to RFC 1423. Leave blank if private key # is not encrypted. keyPassword: "" # EJBCA username used if the proxy was configured to use EST for enrollment. To enable EST, set useEST to true in values.yaml. ejbcaUsername: "" # EJBCA password used if the proxy was configured to use EST for enrollment. ejbcaPassword: ""
Save and close the file:
CODE:wq
Create a new namespace for the EJBCA CSR Signer:
CODE$ kubectl create namespace ejbca-csr-signer
Create a Kubernetes secret for the credentials.yaml file:
CODE$ kubectl -n ejbca-csr-signer create secret generic ejbca-credentials --from-file ./credentials.yaml
Create a Kubernetes ConfigMap that contains the Management CA certificate. This is required because the Management CA issued a TLS certificate to Apache HTTPD to terminate TLS.
CODE$ kubectl -n ejbca-csr-signer create configmap ejbca-ca-cert --from-file ../ca.crt
Change directory to the ejbca-csr-signer directory:
CODE$ cd ejbca-csr-signer
Create the ejbca-csr-signer-helm directory to stage the EJBCA CSR Signer Helm chart:
CODE$ mkdir ejbca-csr-signer-helm
Move the Chart.yaml file and templates directory into the ejbca-csr-signer-helm directory:
CODE$ mv Chart.yaml templates ejbca-csr-signer-helm
- Open the k8-RA.pem in a text editor. This file was created following the Tutorial - Create roles in EJBCA.
Using a text editor such as vim, create the client.pem file:
CODE$ vim ../client.pem
Add the k8-RA certificate to the file. Example of the k8-RA certificate:
CODE-----BEGIN CERTIFICATE----- MIIEcjCCAlqgAwIBAgIUDi3zw8usjEWgwVkqe+zaAfIdzNAwDQYJKoZIhvcNAQEL BQAwQjEVMBMGA1UEAwwMTWFuYWdlbWVudENBMRwwGgYDVQQKDBNLZXlmYWN0b3Ig Q29tbXVuaXR5MQswCQYDVQQGEwJTRTAeFw0yMzAxMjkxODIzNDhaFw0yNDAxMjUx ODIzNDdaMDsxCzAJBgNVBAYTAlNFMRwwGgYDVQQKDBNLZXlmYWN0b3IgQ29tbXVu aXR5MQ4wDAYDVQQDDAVrOC1SQTCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoC ggEBAJjuJsk1jqhuaZJo3bkHcW9Ka/FxH32A15/J7dGOuKNizVcmtQCvT0eA2K2d NfbGcpnQ+yKTS3xV5ixJOtz6PXczqq+ZLtaV70uMOSt/9YovnjiQZD6MZCzVsxjH 42UkCL63VrmuslAdFmcT0YKxV4LtH+lJY6Ct//VDG7hoRotxfl+clvWVWvwJ/iHL X0pcHDsRakbetZZl0tM5HqVpoBvYpqRNHq0b+zrn1wI6TmtxFAoNhmkkMMdBLwhA X/C6RiLsOFbpOXI5rmy150+WfPbIUM0M2ZZ8QXpoEJX5CPPTQZSYW2ATwAiJh95Y 6NhseoYExS9tgsa0WeIs13tGn2cCAwEAAaNnMGUwHwYDVR0jBBgwFoAU1/4xlENJ PF+hu17+BYAELATzoNIwEwYDVR0lBAwwCgYIKwYBBQUHAwIwHQYDVR0OBBYEFAAz DQA2UVgs382BCgnwBzI7X4E/MA4GA1UdDwEB/wQEAwIHgDANBgkqhkiG9w0BAQsF AAOCAgEATUsPC26wDxlM/D8gzaYYZiii+gLzPR51ii9xN5sOyMLLa3disonQVRVV AkvvrQZHudEJkPPqEvBTPEweOnkmvION9XFoJSzhzY1wgxpuuYVtdKii8X+kz2v2 gBV0H4tfG7XUBAXLskKk0Z0Hhhh50wNNvOhmcPxjmiPAVgT438ERKWY4mbl3cr3W mHrYy0C4FU6mWLiRDoScAjSyR/hKqb6zQcIGaFqunzwI+oHUC4icIJH/z6IUl2uz O3vSNJBC0rcjt8I48FV1hapBGUHMM6xAOB4+/kMyKCnJh9n7Mo/7czjw7Mdhb/fW Ybck/86dJ5AIHqhWk8EFuJ7aqwIW4NtCt3uleh9pow6VVeqv8xgS2NFWA/rEyeUy JYSZJ2xyMCK742XcaOpQw1vV7oGyN9UaWbBlSMuWnp65mWvGy3t4GnJArbZeFW8E WJyXNYaWwC4X7bjuDSjegh1fF4hn0jTGqKvD8CrLKLLYsjbwbhcb4jtBvXY2258U HCfZwJ07Rjx0T6/R6qPASmyVnlJYlTOm3O8hsbquMixfm+MgvnO1weCoCh/mOl5D grUpz9BXGBt1IXbYt78+v6wkjnEgGgIOA7lpUYWiaKlyFed33wiWcFUJMcf2BIPr F9JKuXvnw7WtRDqGKoDXN3CWL3uJN5364kGEnDboOLSGCHLASOE= -----END CERTIFICATE-----
Save and close the file:
CODE:wq
Using a text editor such as vim, create the client.key file:
CODE$ vim ../client.key
Add the k8-RA private key to the file. Example of the k8-RA private key:
CODE-----BEGIN PRIVATE KEY----- MIIEvQIBADANBgkqhkiG9w0BAQEFAASCBKcwggSjAgEAAoIBAQCY7ibJNY6obmmS aN25B3FvSmvxcR99gNefye3RjrijYs1XJrUAr09HgNitnTX2xnKZ0Psik0t8VeYs STrc+j13M6qvmS7Wle9LjDkrf/WKL544kGQ+jGQs1bMYx+NlJAi+t1a5rrJQHRZn E9GCsVeC7R/pSWOgrf/1Qxu4aEaLcX5fnJb1lVr8Cf4hy19KXBw7EWpG3rWWZdLT OR6laaAb2KakTR6tG/s659cCOk5rcRQKDYZpJDDHQS8IQF/wukYi7DhW6TlyOa5s tedPlnz2yFDNDNmWfEF6aBCV+Qjz00GUmFtgE8AIiYfeWOjYbHqGBMUvbYLGtFni LNd7Rp9nAgMBAAECggEAO6dKAdqeVx0amT3Gn1JD8UF6cafKvM3xTicaWU/uvezg ZEp4+Fdp+V5NJwvX7Pbj5RQbohUKsOlg6411JJWIPGMvBWgfWR0LRtDfzBQR12FT uoS4VZ21xbdmMRhnnyA7OQmTDsMSUyXFg1e7tdsvY6bTd9BkyFyXJziSK5ChU+rK 0yS2+85wArC3XYgO4VrXba6z8qIaprvV35LtDD2UnpWh5zSaEj1BgD6ffsWevvde qprGb6rR68ghHE3kL+q2RCUG6nX7U2mX4QynwSQn8me7982aQLgz7efAaupA3q7e dy5eU8hMmc8/HEBa5ZE1LGcCvflDdUqrilFWdc/NgQKBgQDSkHtuRGKqTS1xM9Ej 0XsFbKvoXU+WD83atHtFz7e66Jii4KlbbkUf4oHkHBitIlnsaGHWMYItpv5sjd+J P7ntFuePw++3XYMJY6H+DoXRlZY//L6j8U2Nb1ao9VnZLjJjZwVnLlh9gREp+nV2 my8cMVjEWGHy5jyHPvtVZEivcQKBgQC57fo+MnxQxuNW6UGwTWahCQ7E6cdjsXgU C5YvXWEiFYu3Xk/peIbycVoigw+W57E8M2GS3yLHA/Xh9T20ThLAe9zP1Z6P92xO pcXVVAM9W/3O9zl61t4kvZoshIP9knBjYoUozoaU++pbnKx1GqFIFVdK7zL2jkBL 4MqOPEMAVwKBgQClb5U63onyqe6RKZghHz4b1fT+/QlBqqsfMXxFLl15gbQjDIaj anDvC0Tol1af+QRT5PMxmfZgrfrqCVHfAO2wpLVM1DIsjFEe+GPXO0vSjkfdgFO8 dSNsg1TALPzp0Q0P4mpxVg16lgSJSdouVODfsrm+kn5qnJBj5o0L213sUQKBgFy+ s8wgvNhCTZbF5el+wonjjcV14+r71K0TFohr6Q7qdnYyimQopg/7sP10KOuaiVNB QhPUUHG7rQRYo730D/CKGJxnr5+aySD2GhgOv0r1P0blFXwMAGWNWoGIXJq5WGyK 8WdolcNtYfruzSvg68CcPJ35cY+BZ9sxt3h54OYjAoGAcbj1X5aB20/2dxYmqCfw 6DzUR7XxtTO4sJP3dkzJhBUDMfr20XKcV77DGsh8GIVQq5JX5NmWtguroEY3McDS rjJ8Hq2T+rErbd5dOsXvpMAEDuZbWg61aPPDtnZbg3Qt4vL1iRlceiWfKlRtLpYx Kc85tPjOyld1611tqwLBtgw= -----END PRIVATE KEY-----
Save and close the file:
CODE:wq
Create a Kubernetes secret that contains the client certificate and key:
CODE$ kubectl -n ejbca-csr-signer create secret tls ejbca-client-cert --cert=../client.pem --key=../client.key
Using a text editor such as vim, create the values.yaml file:
CODE$ vim ejbca-csr-signer-helm/values.yaml
Add the following to the file:
CODEreplicaCount: 1 imagePullSecrets: [] nameOverride: "" fullnameOverride: "" ejbca: image: repository: keyfactor/ejbca-k8s-csr-signer pullPolicy: IfNotPresent tag: 1.0.3 useEST: false healthcheckPort: 5354 defaultESTAlias: "" defaultCertificateProfileName: "ShortLivedProfile" defaultEndEntityProfileName: "ShortLivedProfile" defaultCertificateAuthorityName: "MyPKISubCA-G1" credsSecretName: ejbca-credentials clientCertSecretName: ejbca-client-cert caCertConfigmapName: "ejbca-ca-cert" chainDepth: 4 signerNames: - "keyfactor.com/*" vault: enabled: false roleName: ejbca-cred vaultSecretPath: secret/data/ejbca serviceAccount: # Specifies whether a service account should be created create: true # Annotations to add to the service account annotations: {} # The name of the service account to use. # If not set and create is true, a name is generated using the fullname template name: "ejbca-k8s" podSecurityContext: {} # fsGroup: 2000 securityContext: {} # capabilities: # drop: # - ALL # readOnlyRootFilesystem: true # runAsNonRoot: true # runAsUser: 1000 resources: {} # We usually recommend not to specify default resources and to leave this as a conscious # choice for the user. This also increases chances charts run on environments with little # resources, such as Minikube. If you do want to specify resources, uncomment the following # lines, adjust them as necessary, and remove the curly braces after 'resources:'. # limits: # cpu: 100m # memory: 128Mi # requests: # cpu: 100m # memory: 128Mi autoscaling: enabled: true minReplicas: 1 maxReplicas: 100 targetCPUUtilizationPercentage: 80 # targetMemoryUtilizationPercentage: 80 nodeSelector: {} tolerations: [] affinity: {}
Save and close the file:
CODE:wq
Package the Helm chart:
CODE$ helm package ejbca-csr-signer-helm
The Helm chart is packaged and the configuration files to deploy EJBCA CSR Signer are complete. Continue to the next section to deploy the EJBCA CSR Signer.
Step 2 - Deploy the EJBCA CSR Signer
After getting the staging of the EJBCA CSR Signer configuration files complete and packing the Helm chart the deployment can begin.
To deploy the EJBCA CSR Signer, follow these steps:
Use Helm to deploy the EJBCA CSR Signer Helm chart:
CODE$ helm install ejbca-csr-signer-helm ./ejbca-csr-signer-1.0.3.tgz --namespace ejbca-csr-signer
Review what is running in the MicroK8s ejbca-csr-signer namespace:
CODE$ kubectl get all,ingress,secret,no -n ejbca-csr-signer
Tail the EJBCA CSR Signer pod logs to verify the container started successfully:
CODE$ kubectl -n ejbca-csr-signer logs $(kubectl get pods --template '{{range .items}}{{.metadata.name}}{{end}}' -n ejbca-csr-signer) -f
The output is similar to the following:
CODEINFO[2023-04-29T00:46:19Z] Getting configuration from ./config/config.yaml scope=Config TRAC[2023-04-29T00:46:19Z] ./config/config.yaml exists and contains 211 bytes: useEST: false defaultESTAlias: defaultCertificateProfileName: ShortLivedProfile defaultEndEntityProfileName: ShortLivedProfile defaultCertificateAuthorityName: MyPKISubCA-G1 healthcheckPort: 5354 chainDepth: 4 scope=Config INFO[2023-04-29T00:46:19Z] Successfully retrieved configuration: &config.ServerConfig{HealthCheckPort:"5354", DefaultCertificateProfileName:"ShortLivedProfile", DefaultEndEntityProfileName:"ShortLivedProfile", DefaultCertificateAuthorityName:"MyPKISubCA-G1", UseEST:false, DefaultESTAlias:"", ChainDepth:4} scope=Config INFO[2023-04-29T00:46:19Z] Getting credentials from credentials.yaml scope=Credential TRAC[2023-04-29T00:46:19Z] credentials.yaml exists and contains 452 bytes scope=Credential INFO[2023-04-29T00:46:19Z] Successfully retrieved credentials. scope=Credential INFO[2023-04-29T00:46:19Z] Looking in /clientcert/ for client certificates scope=Credential WARN[2023-04-29T00:46:19Z] read /clientcert/..data: is a directory scope=Credential INFO[2023-04-29T00:46:19Z] tls.crt exists and contains 1602 bytes scope=Credential INFO[2023-04-29T00:46:19Z] tls.key exists and contains 1704 bytes scope=Credential INFO[2023-04-29T00:46:19Z] Successfully retrieved client certificate scope=Credential DEBU[2023-04-29T00:46:19Z] Creating EJBCA client (useEST=false) scope=Main 2023/04/29 00:46:19 [INFO] Building new EJBCA client 2023/04/29 00:46:19 [TRACE] Reading client certificate from certkey.pem 2023/04/29 00:46:19 [TRACE] Found CERTIFICATE 2023/04/29 00:46:19 [TRACE] Private key is not protected 2023/04/29 00:46:19 [TRACE] Found PRIVATE KEY 2023/04/29 00:46:19 [DEBUG] Found client certificate and private key 2023/04/29 00:46:19 [TRACE] System Cert Pool contains 1 certificates 2023/04/29 00:46:19 0: 0B10U ManagementCA10U Keyfactor Community1 0 USE 2023/04/29 00:46:19 [INFO] Getting EJBCA V1 Certificate Endpoint Status 2023/04/29 00:46:19 [INFO] Preparing a GET request to path 'https://ejbca-internal.ejbca-k8s/ejbca/ejbca-rest-api/v1/certificate/status' 2023/04/29 00:46:19 [DEBUG] GET succeeded with response code 200 2023/04/29 00:46:19 [INFO] Connected to instance EJBCA 7.11.0 Community (8d14e27cda0b32eba35a1fd1423f8e6a31d1ed8e) with status OK
EJBCA CSR Signer is deployed using a Helm chart. Continue to the next step to deploy the Istio service mesh.
Step 3 - Deploy Istio service mesh
The EJBCA CSR Signer is deployed and configured to issue certificates from EJBCA and next is to deploy a service mesh with Istio.
To deploy Istio, follow these steps:
Download the Istio release archive from GitHub:
CODE$ curl -L https://github.com/istio/istio/releases/download/1.16.3/istio-1.16.3-linux-amd64.tar.gz -o istio-1.16.3-linux-amd64.tar.gz
Unpack the Istio archive file:
CODE$ tar xf istio-1.16.3-linux-amd64.tar.gz
Change the directory to the Istio directory:
CODE$ cd istio-1.16.3
Add the Istio bin directory to the PATH environment variable:
CODE$ export PATH=$PWD/bin:$PATH
Install Istio on the MicroK8s cluster:
CODE$ istioctl install --set profile=demo -y
Change to the previous directory ejbca-csr-signer:
CODE$ cd ../
Istio is installed into the MicroK8s cluster. Continue to the next step to configure Istio to use the EJBCA CSR Signer.
Step 4 - Configure Istio to use EJBCA CSR Signer
Up to this point, Istio is installed and if certificates are issued they would not be from EJBCA. Next, configure Istio to use EJBCA to issue certificates.
To configure Istio to issue certificates from EJBCA, follow these steps:
Get the external IP address used for EJBCA:
CODE$ export theIP="$(kubectl -n ingress get services -o json | jq -r '.items[] |.status.loadBalancer?|.ingress[]?|.ip ' | cut -d : -f 2)"
Update the hosts file with the IP Address from the previous step to resolve to ejbca-node1:
CODE$ sudo bash -c 'echo '"${theIP} ejbca-node1"' >> /etc/hosts'
Download the Elliptic CA chain from EJBCA RA Web using cURL with mutual TLS:
CODE$ surl -X GET --cert ../client.pem --key ../client.key --cacert ../../ca.crt "https://ejbca-node1/ejbca/ra/cert?caid=-1419783344&chain=true&format=pem" -H "accept: */*" -o cacerts.pem
Convert the CA chain file to base 64:
CODE$ export CA64ECODE="$(base64 cacerts.pem | tr -d \\n)"
Create the external-ca-secret.yaml:
CODE$ cat<<EOF>external-ca-secret.yaml apiVersion: v1 kind: Secret metadata: name: external-ca-cert namespace: istio-system data: root-cert.pem: $CA64ECODE EOF
Apply the external-ca-secret.yaml file:
CODE$ kubectl apply -f external-ca-secret.yaml
Using a text editor such as vim, create the istio.yaml file:
CODE$ vim istio.yaml
Add the following to the file:
YMLapiVersion: install.istio.io/v1alpha1 kind: IstioOperator spec: meshConfig: defaultConfig: proxyMetadata: ECC_SIGNATURE_ALGORITHM: ECDSA components: pilot: k8s: env: # Indicate to Istiod that we use an external signer - name: EXTERNAL_CA value: ISTIOD_RA_KUBERNETES_API # Indicate to Istiod the external k8s Signer Name - name: K8S_SIGNER #value: example.com/foo value: keyfactor.com/kubernetes-integration overlays: # Amend ClusterRole to add permission for istiod to approve certificate signing by custom signer - kind: ClusterRole name: istiod-clusterrole-istio-system patches: - path: rules[-1] value: | apiGroups: - certificates.k8s.io resourceNames: - example.com/foo resources: - signers verbs: - approve - kind: Deployment name: istiod patches: - path: spec.template.spec.containers[0].volumeMounts[-1] value: | # Mount external CA certificate into Istiod name: external-ca-cert mountPath: /etc/external-ca-cert readOnly: true - path: spec.template.spec.volumes[-1] value: | name: external-ca-cert secret: secretName: external-ca-cert optional: true
Save and close the file:
CODE:wq
Update Istio to use the updated istio.yaml configuration file:
CODE$ istioctl install --set profile=demo -f ./istio.yaml -y
Istio is configured to issue certificates from EJBCA using the EJBCA CSR Signer and configured with the Elliptic curve CA chain. Continue to the next step to deploy the Istio Bookinfo application.
Step 5 - Deploy the Istio Bookinfo application
Next, to issue some mutual TLS certificates to the Istio service mesh you will use the Istio Bookinfo application used to demonstrate various Istio features. Deploying the sample Istio Bookinfo application triggers certificate enrollment for the mutual TLS certificates.
To test mutual TLS certificate issuance from EJBCA using the Istio Bookinfo application, follow these steps:
Create the bookinfo namespace:
CODE$ kubectl create ns bookinfo
Enable Istio injection for the default and bookinfo namespaces:
CODE$ kubectl label namespace default istio-injection=enabled $ kubectl label namespace bookinfo istio-injection=enabled
Verify the names are updated to allow Istio injection:
CODE$ kubectl get namespace -L istio-injection
The output is similar to the following:
CODENAME STATUS AGE ISTIO-INJECTION kube-system Active 41d kube-public Active 41d kube-node-lease Active 41d ingress Active 41d metallb-system Active 41d ejbca-k8s Active 40d ejbca-csr-signer Active 34d istio-system Active 34d default Active 41d enabled bookinfo Active 34d enabled
Deploy the Istio Bookinfo sample application:
CODE$ kubectl apply -f <(istioctl kube-inject -f istio-1.16.3/samples/bookinfo/platform/kube/bookinfo.yaml) -n bookinfo
Deploy the Istio Bookinfo application gateway:
CODE$ kubectl apply -f <(istioctl kube-inject -f istio-1.16.3/samples/bookinfo/networking/bookinfo-gateway.yaml) -n bookinfo
Tail the logs for the EJBCA CSR Signer pod and watch certificates get issued for mutual TLS:
CODE$ kubectl -n ejbca-csr-signer logs $(kubectl get pods --template '{{range .items}}{{.metadata.name}}{{end}}' -n ejbca-csr-signer) -f
EJBCA issued certificates for the Istio sample Bookinfo application.
The Istio Bookinfo application is now deployed and you have successfully tested mutual TLS certificate issuance from EJBCA using the Istio Bookinfo application.
Next steps
In this tutorial series, you learned how to set up EJBCA and integrated with Istio to create mutual TLS certificates for your service mesh.
Next, you can browse our other video tutorials on the Keyfactor Community YouTube channel.
To read more about issuing and managing PKI credentials and machine identities for applications in DevOps, see Managing PKI Credentials and Machine Identities for Applications.