Chapter 3. Configuring certificates

Procedure

1. Create a config map that includes only the root CA certificate that is used to sign the wildcard certificate:

   $ oc create configmap custom-ca \
        --from-file=ca-bundle.crt=</path/to/example-ca.crt> \
        -n openshift-config

   where

   </path/to/example-ca.crt>

   The path to the root CA certificate file on your local file system. For example, /etc/pki/ca-trust/source/anchors.

2. Update the cluster-wide proxy configuration with the newly created config map:

   $ oc patch proxy/cluster \
        --type=merge \
        --patch='{"spec":{"trustedCA":{"name":"custom-ca"}}}'

   Note

   If you update only the trusted CA for your cluster, the MCO updates the /etc/pki/ca-trust/source/anchors/openshift-config-user-ca-bundle.crt file and the Machine Config Controller (MCC) applies the trusted CA update to each node so that a node reboot is not required. However, with these changes, the Machine Config Daemon (MCD) restarts critical services on each node, such as kubelet and CRI-O. These service restarts cause each node to briefly enter the NotReady state until the service is fully restarted.

   If you change any other parameter in the openshift-config-user-ca-bundle.crt file, such as noproxy, the MCO reboots each node in your cluster.

3. Create a secret that contains the wildcard certificate chain and key:

   $ oc create secret tls <secret> \
        --cert=</path/to/cert.crt> \
        --key=</path/to/cert.key> \
        -n openshift-ingress

   where:

   <secret>

   Specifies the name of the secret that will contain the certificate chain and private key.

   </path/to/cert.crt>

   Specifies the path to the certificate chain on your local file system.

   </path/to/cert.key>

   Specifies the path to the private key associated with this certificate.

4. Update the Ingress Controller configuration with the newly created secret:

   $ oc patch ingresscontroller.operator default \
        --type=merge -p \
        '{"spec":{"defaultCertificate": {"name": "<secret>"}}}' \
        -n openshift-ingress-operator

   • <secret>:: Specifies the name used for the secret. Replace <secret> with the name used for the secret.

Procedure

1. Log in to the CLI as the kubeadmin user:

   $ oc login -u kubeadmin -p <password> https://<fqdn>:6443

   where:

   <password>

   Specifies your cluster administrative password.

   <fqdn>

   Specifies the fully qualified domain name of the internal cluster API endpoint.

2. Create a secret that contains the certificate chain and private key in the openshift-config namespace:

   $ oc create secret tls <secret_name> \
        --cert=<path_to_certificate_file> \
        --key=<path_to_private_key_file> \
        -n openshift-config

   where:

   <secret_name>

   Specifies the name of the new secret resource that will contain the cryptographic key pair.

   <path_to_certificate_file>

   Specifies the absolute local path to your custom certificate chain file.

   <path_to_private_key_file>

   Specifies the absolute local path to the unencrypted private key file associated with the certificate.

3. Update the API server to reference the created secret resource:

   $ oc patch apiserver cluster --type=merge -p '
   {
     "spec": {
       "servingCerts": {
         "namedCertificates": [
           {
             "names": ["<fqdn>"],
             "servingCertificate": {
               "name": "<secret_name>"
             }
           }
         ]
       }
     }
   }'

   where:

   <fqdn>

   Specifies the fully qualified domain name for which the API server serves this custom certificate. Do not include a port number.

   <secret_name>

   Specifies the name of the secret you created in the previous step.

4. Verify that a new revision of the Kubernetes API server rolls out by checking the operator status:

   $ oc get clusteroperators kube-apiserver

   Note

   The PROGRESSING status column will change to True while the API server operator deploys the new pod revision configured with your custom certificate. Do not interrupt the process or apply additional configuration updates while the rollout is underway. Continue only after the status returns to False and AVAILABLE reads True.

Procedure

1. Log in to the CLI as the kubeadmin user.

2. Update the existing secret resource in the openshift-config namespace with the newly issued certificate and private key:

   $ oc create secret tls <existing_secret_name> \
        --cert=<path_to_new_cert>.crt \
        --key=<path_to_new_key>.key \
        -n openshift-config \
        --dry-run=client -o yaml | oc replace -f -

   where:

   <existing_secret_name>

   Specifies the target name of the existing active secret that you are replacing.

   <path_to_new_cert>.crt

   Specifies the absolute local file system path to the renewed certificate chain file.

   <path_to_new_key>.key

   Specifies the absolute local file system path to the corresponding unencrypted private key file.

3. Verify that the kube-apiserver pods successfully hot-reload the updated assets without initiating a new cluster deployment revision:

   $ oc get clusteroperators kube-apiserver

   Confirm that the PROGRESSING status column remains False. If the status changes to True, verify that your underlying apiserver/cluster resource parameters were not modified structural layout changes during the substitution.

Procedure

1. Annotate the service with service.beta.openshift.io/serving-cert-secret-name:

   $ oc annotate service <service_name> \
        service.beta.openshift.io/serving-cert-secret-name=<secret_name>

   • Replace <service_name> with the name of the service to secure.

   • <secret_name> will be the name of the generated secret containing the certificate and key pair.

     Note

     For convenience, it is recommended that this value be the same as <service_name>.

     For example, use the following command to annotate the service test1:

     $ oc annotate service test1 service.beta.openshift.io/serving-cert-secret-name=test1

2. Examine the service to confirm that the annotations are present:

   $ oc describe service <service_name>

   Example output

   ...
   Annotations:              service.beta.openshift.io/serving-cert-secret-name: <service_name>
                             service.beta.openshift.io/serving-cert-signed-by: openshift-service-serving-signer@1556850837
   ...

3. After the cluster generates a secret for your service, your Pod spec can mount it, and the pod will run after it becomes available.

Procedure

1. Annotate the config map with the service.beta.openshift.io/inject-cabundle=true annotation by entering the following command:

   $ oc annotate configmap <config_map_name> \
        service.beta.openshift.io/inject-cabundle=true

   • Replace <config_map_name> with the name of the config map to annotate.

     Note

     Explicitly referencing the service-ca.crt key in a volume mount prevents a pod from starting until the config map has been injected with the CA bundle. You can override this behavior by setting the optional parameter to true in the serving certificate configuration of the volume.

2. View the config map to ensure that the service CA bundle has been injected:

   $ oc get configmap <config_map_name> -o yaml

   The CA bundle is displayed as the value of the service-ca.crt key in the YAML output:

   apiVersion: v1
   data:
     service-ca.crt: |
       -----BEGIN CERTIFICATE-----
   ...

3. Mount the config map as a volume to each container that exists in a pod by configuring your Deployment object.

   Example Deployment object that defines the volume for the mounted config map

   apiVersion: apps/v1
   kind: Deployment
   metadata:
     name: my-example-custom-ca-deployment
     namespace: my-example-custom-ca-ns
   spec:
     ...
       spec:
         ...
         containers:
           - name: my-container-that-needs-custom-ca
             volumeMounts:
             - name: trusted-ca
               mountPath: /etc/pki/ca-trust/extracted/pem
               readOnly: true
         volumes:
         - name: trusted-ca
           configMap:
             name: <config_map_name>
             items:
               - key: ca-bundle.crt
                 path: tls-ca-bundle.pem
   # ...

   where:

   <config_map_name>

   Specifies the name of the config map that you annotated in an earlier step of the procedure.

   ca-bundle.crt

   Specifies the ConfigMap key. This is required.

   tls-ca-bundle.pem

   Specifies the ConfigMap path. This is required.

Procedure

1. Annotate the API service with service.beta.openshift.io/inject-cabundle=true:

   $ oc annotate apiservice <api_service_name> \
        service.beta.openshift.io/inject-cabundle=true

   • Replace <api_service_name> with the name of the API service to annotate.

     For example, use the following command to annotate the API service test1:

     $ oc annotate apiservice test1 service.beta.openshift.io/inject-cabundle=true

2. View the API service to ensure that the service CA bundle has been injected:

   $ oc get apiservice <api_service_name> -o yaml

   The CA bundle is displayed in the spec.caBundle field in the YAML output:

   apiVersion: apiregistration.k8s.io/v1
   kind: APIService
   metadata:
     annotations:
       service.beta.openshift.io/inject-cabundle: "true"
   ...
   spec:
     caBundle: <CA_BUNDLE>
   ...

Procedure

1. Annotate the CRD with service.beta.openshift.io/inject-cabundle=true:

   $ oc annotate crd <crd_name> \
        service.beta.openshift.io/inject-cabundle=true

   • Replace <crd_name> with the name of the CRD to annotate.

     For example, use the following command to annotate the CRD test1:

     $ oc annotate crd test1 service.beta.openshift.io/inject-cabundle=true

2. View the CRD to ensure that the service CA bundle has been injected:

   $ oc get crd <crd_name> -o yaml

   The CA bundle is displayed in the spec.conversion.webhook.clientConfig.caBundle field in the YAML output:

   apiVersion: apiextensions.k8s.io/v1
   kind: CustomResourceDefinition
   metadata:
     annotations:
       service.beta.openshift.io/inject-cabundle: "true"
   ...
   spec:
     conversion:
       strategy: Webhook
       webhook:
         clientConfig:
           caBundle: <CA_BUNDLE>
   ...

Procedure

1. Annotate the mutating webhook configuration with service.beta.openshift.io/inject-cabundle=true:

   $ oc annotate mutatingwebhookconfigurations <mutating_webhook_name> \
        service.beta.openshift.io/inject-cabundle=true

   • Replace <mutating_webhook_name> with the name of the mutating webhook configuration to annotate.

     For example, use the following command to annotate the mutating webhook configuration test1:

     $ oc annotate mutatingwebhookconfigurations test1 service.beta.openshift.io/inject-cabundle=true

2. View the mutating webhook configuration to ensure that the service CA bundle has been injected:

   $ oc get mutatingwebhookconfigurations <mutating_webhook_name> -o yaml

   The CA bundle is displayed in the clientConfig.caBundle field of all webhooks in the YAML output:

   apiVersion: admissionregistration.k8s.io/v1
   kind: MutatingWebhookConfiguration
   metadata:
     annotations:
       service.beta.openshift.io/inject-cabundle: "true"
   ...
   webhooks:
   - myWebhook:
     - v1beta1
     clientConfig:
       caBundle: <CA_BUNDLE>
   ...

Procedure

1. Annotate the validating webhook configuration with service.beta.openshift.io/inject-cabundle=true:

   $ oc annotate validatingwebhookconfigurations <validating_webhook_name> \
        service.beta.openshift.io/inject-cabundle=true

   • Replace <validating_webhook_name> with the name of the validating webhook configuration to annotate.

     For example, use the following command to annotate the validating webhook configuration test1:

     $ oc annotate validatingwebhookconfigurations test1 service.beta.openshift.io/inject-cabundle=true

2. View the validating webhook configuration to ensure that the service CA bundle has been injected:

   $ oc get validatingwebhookconfigurations <validating_webhook_name> -o yaml

   The CA bundle is displayed in the clientConfig.caBundle field of all webhooks in the YAML output:

   apiVersion: admissionregistration.k8s.io/v1
   kind: ValidatingWebhookConfiguration
   metadata:
     annotations:
       service.beta.openshift.io/inject-cabundle: "true"
   ...
   webhooks:
   - myWebhook:
     - v1beta1
     clientConfig:
       caBundle: <CA_BUNDLE>
   ...

Procedure

1. Examine the service to determine the secret containing the certificate. This is found in the serving-cert-secret-name annotation, as seen below.

   $ oc describe service <service_name>

   Example output

   ...
   service.beta.openshift.io/serving-cert-secret-name: <secret>
   ...

2. Delete the generated secret for the service. This process will automatically recreate the secret.

   $ oc delete secret <secret>

   • Replace <secret> with the name of the secret from the previous step.

3. Confirm that the certificate has been recreated by obtaining the new secret and examining the AGE.

   $ oc get secret <service_name>

   Example output

   NAME              TYPE                DATA   AGE
   <service.name>    kubernetes.io/tls   2      1s

Procedure

1. View the expiration date of the current service CA certificate by using the following command.

   $ oc get secrets/signing-key -n openshift-service-ca \
        -o template='{{index .data "tls.crt"}}' \
        | base64 --decode \
        | openssl x509 -noout -enddate

2. Manually rotate the service CA. This process generates a new service CA which will be used to sign the new service certificates.

   $ oc delete secret/signing-key -n openshift-service-ca

3. To apply the new certificates to all services, restart all the pods in your cluster. This command ensures that all services use the updated certificates.

   $ for I in $(oc get ns -o jsonpath='{range .items[*]} {.metadata.name}{"\n"} {end}'); \
         do oc delete pods --all -n $I; \
         sleep 1; \
         done

   Warning

   This command will cause a service interruption, as it goes through and deletes every running pod in every namespace. These pods will automatically restart after they are deleted.

Procedure

1. Create a config map that includes the root CA certificate used to sign the wildcard certificate:

   $ oc create configmap custom-ca \
        --from-file=ca-bundle.crt=</path/to/example-ca.crt> \
        -n openshift-config

   </path/to/example-ca.crt> is the path to the CA certificate bundle on your local file system.

2. Update the cluster-wide proxy configuration with the newly created config map:

   $ oc patch proxy/cluster \
        --type=merge \
        --patch='{"spec":{"trustedCA":{"name":"custom-ca"}}}'