Tutorial - Deploy EJBCA container to issue certificates to an Istio service mesh

Learn how to use a service mesh to issue mutual TLS certificates with EJBCA running in Kubernetes.

This tutorial shows you how to use EJBCA to issue mutual TLS certificates to a service mesh such as Istio. By integrating Istio with EJBCA, you get a PKI that complies with policy and supports PKCS11 for protecting the certificate authority (CA) key. EJBCA can provide certificates to a service mesh using the EJBCA Certificate Signing Request Proxy for K8s (EJBCA CSR Signer container).

Using the EJBCA CSR Signer is an alternative to using the cert-manager which EJBCA also integrates with, see Issuing Certificates to Kubernetes Services using cert-manager.

In this tutorial, you will learn how to:

Deploy the EJBCA CSR Signer container
Integrate the EJBCA CSR Signer with the EJBCA REST API
Deploy an Istio service mesh
Deploy the sample Istio Bookinfo application
Check logs to validate certificates were issued

Prerequisites

Before you begin, you need to create roles in EJBCA and set up an RA credential to be used by the EJBCA Certificate Signing Request Proxy for K8s (ejbca-csr-signer container) to authenticate to the EJBCA REST API. To learn how to configure EJBCA roles, follow the tutorial Create roles in EJBCA.
You also need to configure EJBCA by following the previous tutorials in this tutorial series Get started with EJBCA and Istio.
Additionally, you should have a basic understanding of how to use the Kubernetes command line tool kubectl.

Step 1 - Prepare the EJBCA CSR Signer container deployment

Deploy the EJBCA CSR Signer to integrate with the Kubernetes CSR-Signer-API which sends CSRs over to EJBCA using the REST API. CSRs from a service mesh are routed to the Kubernetes CSR-Signer-API to then get sent over to EJBCA for signing. Manual CSRs signing can also be done in the Kubernetes cluster.

To prepare for the EJBCA CSR Signer deployment, follow these steps:

SSH to the MicroK8s test host that has EJBCA deployed and configured.
Create the csr-api directory to organize all the files for this tutorial:
CODE
```
$ mkdir csr-api
```
Change directory to the csr-api directory:
CODE
```
$ cd csr-api
```

Download the EJBCA CSR Signer release from GitHub which contains the Helm chart to deploy the container:

CODE

$ curl -L https://github.com/Keyfactor/ejbca-k8s-csr-signer/releases/download/ejbca-csr-signer-1.0.3/ejbca-csr-signer-1.0.3.tgz -o ejbca-csr-signer-1.0.3.tgz

Unpack the archive file:
CODE
```
$ tar xf ejbca-csr-signer-1.0.3.tgz
```
Using a text editor such as vim, create the credentials.yaml file:
CODE
```
$ vim credentials.yaml
```

Add the following to the file:

CODE

# Hostname to EJBCA server
hostname: "ejbca-internal.ejbca-k8s"

# Password used to protect private key, if it's encrypted according to RFC 1423. Leave blank if private key
# is not encrypted.
keyPassword: ""

# EJBCA username used if the proxy was configured to use EST for enrollment. To enable EST, set useEST to true in values.yaml.
ejbcaUsername: ""

# EJBCA password used if the proxy was configured to use EST for enrollment.
ejbcaPassword: ""

Save and close the file:
CODE
```
:wq
```
Create a new namespace for the EJBCA CSR Signer:
CODE
```
$ kubectl create namespace ejbca-csr-signer
```

Create a Kubernetes secret for the credentials.yaml file:

CODE

$ kubectl -n ejbca-csr-signer create secret generic ejbca-credentials --from-file ./credentials.yaml

Create a Kubernetes ConfigMap that contains the Management CA certificate. This is required because the Management CA issued a TLS certificate to Apache HTTPD to terminate TLS.
CODE
```
$ kubectl -n ejbca-csr-signer create configmap ejbca-ca-cert --from-file ../ca.crt
```
Change directory to the ejbca-csr-signer directory:
CODE
```
$ cd ejbca-csr-signer 
```
Create the ejbca-csr-signer-helm directory to stage the EJBCA CSR Signer Helm chart:
CODE
```
$ mkdir ejbca-csr-signer-helm
```
Move the Chart.yaml file and templates directory into the ejbca-csr-signer-helm directory:
CODE
```
$ mv Chart.yaml templates ejbca-csr-signer-helm
```
Open the k8-RA.pem in a text editor. This file was created following the Tutorial - Create roles in EJBCA.
Using a text editor such as vim, create the client.pem file:
CODE
```
$ vim ../client.pem
```

Add the k8-RA certificate to the file. Example of the k8-RA certificate:

CODE

Save and close the file:
CODE
```
:wq
```
Using a text editor such as vim, create the client.key file:
CODE
```
$ vim ../client.key
```

Add the k8-RA private key to the file. Example of the k8-RA private key:

CODE

Save and close the file:
CODE
```
:wq
```

Create a Kubernetes secret that contains the client certificate and key:

CODE

$ kubectl -n ejbca-csr-signer create secret tls ejbca-client-cert --cert=../client.pem --key=../client.key

Using a text editor such as vim, create the values.yaml file:
CODE
```
$ vim ejbca-csr-signer-helm/values.yaml
```

Add the following to the file:

CODE

replicaCount: 1

imagePullSecrets: []
nameOverride: ""
fullnameOverride: ""

ejbca:
  image:
    repository: keyfactor/ejbca-k8s-csr-signer
    pullPolicy: IfNotPresent
    tag: 1.0.3
  useEST: false
  healthcheckPort: 5354
  defaultESTAlias: ""
  defaultCertificateProfileName: "ShortLivedProfile"
  defaultEndEntityProfileName: "ShortLivedProfile"
  defaultCertificateAuthorityName: "MyPKISubCA-G1"
  credsSecretName: ejbca-credentials
  clientCertSecretName: ejbca-client-cert
  caCertConfigmapName: "ejbca-ca-cert"
  chainDepth: 4
  signerNames:
    - "keyfactor.com/*"
  vault:
    enabled: false
    roleName: ejbca-cred
    vaultSecretPath: secret/data/ejbca

serviceAccount:
  # Specifies whether a service account should be created
  create: true
  # Annotations to add to the service account
  annotations: {}
  # The name of the service account to use.
  # If not set and create is true, a name is generated using the fullname template
  name: "ejbca-k8s"

podSecurityContext: {}
  # fsGroup: 2000

securityContext: {}
  # capabilities:
  #   drop:
  #   - ALL
  # readOnlyRootFilesystem: true
  # runAsNonRoot: true
  # runAsUser: 1000

resources: {}
  # We usually recommend not to specify default resources and to leave this as a conscious
  # choice for the user. This also increases chances charts run on environments with little
  # resources, such as Minikube. If you do want to specify resources, uncomment the following
  # lines, adjust them as necessary, and remove the curly braces after 'resources:'.
  # limits:
  #   cpu: 100m
  #   memory: 128Mi
  # requests:
  #   cpu: 100m
  #   memory: 128Mi

autoscaling:
  enabled: true
  minReplicas: 1
  maxReplicas: 100
  targetCPUUtilizationPercentage: 80
  # targetMemoryUtilizationPercentage: 80

nodeSelector: {}

tolerations: []

affinity: {}

Save and close the file:
CODE
```
:wq
```
Package the Helm chart:
CODE
```
$ helm package ejbca-csr-signer-helm
```

The Helm chart is packaged and the configuration files to deploy EJBCA CSR Signer are complete. Continue to the next section to deploy the EJBCA CSR Signer.

Step 2 - Deploy the EJBCA CSR Signer

After getting the staging of the EJBCA CSR Signer configuration files complete and packing the Helm chart the deployment can begin.

To deploy the EJBCA CSR Signer, follow these steps:

Use Helm to deploy the EJBCA CSR Signer Helm chart:

CODE

$ helm install ejbca-csr-signer-helm ./ejbca-csr-signer-1.0.3.tgz --namespace ejbca-csr-signer

Review what is running in the MicroK8s ejbca-csr-signer namespace:
CODE
```
$ kubectl get all,ingress,secret,no -n ejbca-csr-signer
```

Tail the EJBCA CSR Signer pod logs to verify the container started successfully:

CODE

$ kubectl -n ejbca-csr-signer logs $(kubectl get pods --template '{{range .items}}{{.metadata.name}}{{end}}' -n ejbca-csr-signer) -f

The output is similar to the following:

CODE

INFO[2023-04-29T00:46:19Z] Getting configuration from ./config/config.yaml  scope=Config
TRAC[2023-04-29T00:46:19Z] ./config/config.yaml exists and contains 211 bytes:
 useEST: false
defaultESTAlias:
defaultCertificateProfileName: ShortLivedProfile
defaultEndEntityProfileName: ShortLivedProfile
defaultCertificateAuthorityName: MyPKISubCA-G1
healthcheckPort: 5354
chainDepth: 4  scope=Config
INFO[2023-04-29T00:46:19Z] Successfully retrieved configuration:
 &config.ServerConfig{HealthCheckPort:"5354", DefaultCertificateProfileName:"ShortLivedProfile", DefaultEndEntityProfileName:"ShortLivedProfile", DefaultCertificateAuthorityName:"MyPKISubCA-G1", UseEST:false, DefaultESTAlias:"", ChainDepth:4}  scope=Config
INFO[2023-04-29T00:46:19Z] Getting credentials from credentials.yaml     scope=Credential
TRAC[2023-04-29T00:46:19Z] credentials.yaml exists and contains 452 bytes  scope=Credential
INFO[2023-04-29T00:46:19Z] Successfully retrieved credentials.           scope=Credential
INFO[2023-04-29T00:46:19Z] Looking in /clientcert/ for client certificates  scope=Credential
WARN[2023-04-29T00:46:19Z] read /clientcert/..data: is a directory       scope=Credential
INFO[2023-04-29T00:46:19Z] tls.crt exists and contains 1602 bytes        scope=Credential
INFO[2023-04-29T00:46:19Z] tls.key exists and contains 1704 bytes        scope=Credential
INFO[2023-04-29T00:46:19Z] Successfully retrieved client certificate     scope=Credential
DEBU[2023-04-29T00:46:19Z] Creating EJBCA client (useEST=false)          scope=Main
2023/04/29 00:46:19 [INFO] Building new EJBCA client
2023/04/29 00:46:19 [TRACE] Reading client certificate from certkey.pem
2023/04/29 00:46:19 [TRACE] Found CERTIFICATE
2023/04/29 00:46:19 [TRACE] Private key is not protected
2023/04/29 00:46:19 [TRACE] Found PRIVATE KEY
2023/04/29 00:46:19 [DEBUG] Found client certificate and private key
2023/04/29 00:46:19 [TRACE] System Cert Pool contains 1 certificates
2023/04/29 00:46:19 0:
0B10U

     ManagementCA10U

Keyfactor Community1
                    0	USE
2023/04/29 00:46:19 [INFO] Getting EJBCA V1 Certificate Endpoint Status
2023/04/29 00:46:19 [INFO] Preparing a GET request to path 'https://ejbca-internal.ejbca-k8s/ejbca/ejbca-rest-api/v1/certificate/status'
2023/04/29 00:46:19 [DEBUG] GET succeeded with response code 200
2023/04/29 00:46:19 [INFO] Connected to instance EJBCA 7.11.0 Community (8d14e27cda0b32eba35a1fd1423f8e6a31d1ed8e) with status OK

EJBCA CSR Signer is deployed using a Helm chart. Continue to the next step to deploy the Istio service mesh.

Step 3 - Deploy Istio service mesh

The EJBCA CSR Signer is deployed and configured to issue certificates from EJBCA and next is to deploy a service mesh with Istio.

To deploy Istio, follow these steps:

Download the Istio release archive from GitHub:

CODE

$ curl -L https://github.com/istio/istio/releases/download/1.16.3/istio-1.16.3-linux-amd64.tar.gz -o istio-1.16.3-linux-amd64.tar.gz

Unpack the Istio archive file:

CODE

$ tar xf istio-1.16.3-linux-amd64.tar.gz

Change the directory to the Istio directory:
CODE
```
$ cd istio-1.16.3
```
Add the Istio bin directory to the PATH environment variable:
CODE
```
$ export PATH=$PWD/bin:$PATH
```
Install Istio on the MicroK8s cluster:
CODE
```
$ istioctl install --set profile=demo -y
```
Change to the previous directory ejbca-csr-signer:
CODE
```
$ cd ../
```

Istio is installed into the MicroK8s cluster. Continue to the next step to configure Istio to use the EJBCA CSR Signer.

Step 4 - Configure Istio to use EJBCA CSR Signer

Up to this point, Istio is installed and if certificates are issued they would not be from EJBCA. Next, configure Istio to use EJBCA to issue certificates.

To configure Istio to issue certificates from EJBCA, follow these steps:

Get the external IP address used for EJBCA:

CODE

$ export theIP="$(kubectl -n ingress get services -o json | jq -r '.items[] |.status.loadBalancer?|.ingress[]?|.ip ' | cut -d : -f 2)"

Update the hosts file with the IP Address from the previous step to resolve to ejbca-node1:
CODE
```
$ sudo bash -c 'echo '"${theIP} ejbca-node1"' >> /etc/hosts'
```

Download the Elliptic CA chain from EJBCA RA Web using cURL with mutual TLS:

CODE

$ surl -X GET --cert ../client.pem --key ../client.key --cacert ../../ca.crt "https://ejbca-node1/ejbca/ra/cert?caid=-1419783344&chain=true&format=pem" -H  "accept: */*" -o cacerts.pem

Convert the CA chain file to base 64:

CODE

$ export CA64ECODE="$(base64 cacerts.pem | tr -d \\n)"

Create the external-ca-secret.yaml:

CODE

$ cat<<EOF>external-ca-secret.yaml
apiVersion: v1
kind: Secret
metadata:
  name: external-ca-cert
  namespace: istio-system
data:
  root-cert.pem: $CA64ECODE

EOF

Apply the external-ca-secret.yaml file:
CODE
```
$ kubectl apply -f external-ca-secret.yaml
```
Using a text editor such as vim, create the istio.yaml file:
CODE
```
$ vim istio.yaml
```

Add the following to the file:

YML

apiVersion: install.istio.io/v1alpha1
kind: IstioOperator
spec:
  meshConfig:
    defaultConfig:
      proxyMetadata:
        ECC_SIGNATURE_ALGORITHM: ECDSA
  components:
    pilot:
      k8s:
        env:
          # Indicate to Istiod that we use an external signer
          - name: EXTERNAL_CA
            value: ISTIOD_RA_KUBERNETES_API
          # Indicate to Istiod the external k8s Signer Name
          - name: K8S_SIGNER
            #value: example.com/foo
            value: keyfactor.com/kubernetes-integration
        overlays:
          # Amend ClusterRole to add permission for istiod to approve certificate signing by custom signer
          - kind: ClusterRole
            name: istiod-clusterrole-istio-system
            patches:
              - path: rules[-1]
                value: |
                  apiGroups:
                  - certificates.k8s.io
                  resourceNames:
                  - example.com/foo
                  resources:
                  - signers
                  verbs:
                  - approve
          - kind: Deployment
            name: istiod
            patches:
              - path: spec.template.spec.containers[0].volumeMounts[-1]
                value: |
                  # Mount external CA certificate into Istiod
                  name: external-ca-cert
                  mountPath: /etc/external-ca-cert
                  readOnly: true
              - path: spec.template.spec.volumes[-1]
                value: |
                  name: external-ca-cert
                  secret:
                    secretName: external-ca-cert
                    optional: true

Save and close the file:
CODE
```
:wq
```
Update Istio to use the updated istio.yaml configuration file:
CODE
```
$ istioctl install --set profile=demo -f ./istio.yaml -y
```

Istio is configured to issue certificates from EJBCA using the EJBCA CSR Signer and configured with the Elliptic curve CA chain. Continue to the next step to deploy the Istio Bookinfo application.

Step 5 - Deploy the Istio Bookinfo application

Next, to issue some mutual TLS certificates to the Istio service mesh you will use the Istio Bookinfo application used to demonstrate various Istio features. Deploying the sample Istio Bookinfo application triggers certificate enrollment for the mutual TLS certificates.

To test mutual TLS certificate issuance from EJBCA using the Istio Bookinfo application, follow these steps:

Create the bookinfo namespace:
CODE
```
$ kubectl create ns bookinfo
```

Enable Istio injection for the default and bookinfo namespaces:

CODE

$ kubectl label namespace default istio-injection=enabled
$ kubectl label namespace bookinfo istio-injection=enabled

Verify the names are updated to allow Istio injection:
CODE
```
$ kubectl get namespace -L istio-injection
```

The output is similar to the following:

CODE

NAME               STATUS   AGE   ISTIO-INJECTION
kube-system        Active   41d
kube-public        Active   41d
kube-node-lease    Active   41d
ingress            Active   41d
metallb-system     Active   41d
ejbca-k8s          Active   40d
ejbca-csr-signer   Active   34d
istio-system       Active   34d
default            Active   41d   enabled
bookinfo           Active   34d   enabled

Deploy the Istio Bookinfo sample application:

CODE

$ kubectl apply -f <(istioctl kube-inject -f istio-1.16.3/samples/bookinfo/platform/kube/bookinfo.yaml) -n bookinfo

Deploy the Istio Bookinfo application gateway:

CODE

$ kubectl apply -f <(istioctl kube-inject -f istio-1.16.3/samples/bookinfo/networking/bookinfo-gateway.yaml) -n bookinfo

Tail the logs for the EJBCA CSR Signer pod and watch certificates get issued for mutual TLS:

CODE

$ kubectl -n ejbca-csr-signer logs $(kubectl get pods --template '{{range .items}}{{.metadata.name}}{{end}}' -n ejbca-csr-signer) -f

EJBCA issued certificates for the Istio sample Bookinfo application.

The Istio Bookinfo application is now deployed and you have successfully tested mutual TLS certificate issuance from EJBCA using the Istio Bookinfo application.

Next steps

In this tutorial series, you learned how to set up EJBCA and integrated with Istio to create mutual TLS certificates for your service mesh.

Next, you can browse our other video tutorials on the Keyfactor Community YouTube channel.

To read more about issuing and managing PKI credentials and machine identities for applications in DevOps, see Managing PKI Credentials and Machine Identities for Applications.