Understanding service serving certificates

Service serving certificates are intended to support complex middleware applications that require encryption. These certificates are issued as TLS web server certificates.

The service-ca controller uses the x509.SHA256WithRSA signature algorithm to generate service certificates.

The generated certificate and key are in PEM format, stored in tls.crt and tls.key respectively, within a created secret. The certificate and key are automatically replaced when they get close to expiration.

The service CA certificate, which issues the service certificates, is valid for 26 months and is automatically rotated when there is less than six months validity left. After rotation, the previous service CA configuration is still trusted until its expiration. This allows a grace period for all affected services to refresh their key material before the expiration. If you do not upgrade your cluster during this grace period, which restarts services and refreshes their key material, you might need to manually restart services to avoid failures after the previous service CA expires.

You can use the following command to manually restart all Pods in the cluster. Be aware that running this command causes a service interruption, because it deletes every running Pod in every namespace. These Pods will automatically restart after they are deleted.

$ for I in $(oc get ns -o jsonpath='{range .items[*]} {.metadata.name}{"\n"} {end}'); \
      do oc delete pods --all -n $I; \
      sleep 1; \
      done

Add a service certificate

To secure communication to your service, generate a signed serving certificate and key pair into a secret in the same namespace as the service.

The generated certificate is only valid for the internal service DNS name <service.name>.<service.namespace>.svc, and are only valid for internal communications.

Prerequisites:
  • You must have a service defined.

Procedure
  1. Annotate the service with service.beta.openshift.io/serving-cert-secret-name:

    $ oc annotate service <service-name> \(1)
         service.beta.openshift.io/serving-cert-secret-name=<secret-name> (2)
    1 Replace <service-name> with the name of the service to secure.
    2 <secret-name> will be the name of the generated secret containing the certificate and key pair. For convenience, it is recommended that this be the same as <service-name>.

    For example, use the following command to annotate the service foo:

    $ oc annotate service foo service.beta.openshift.io/serving-cert-secret-name=foo
  2. Examine the service to confirm that the annotations are present:

    $ oc describe service <service-name>

    Verify that the annotations are listed in the output:

    ...
    Annotations:              service.beta.openshift.io/serving-cert-secret-name: <service-name>
                              service.beta.openshift.io/serving-cert-signed-by: openshift-service-serving-signer@1556850837
    ...
  3. After the cluster generates a secret for your service, your PodSpec can mount it, and the Pod will run after it becomes available.

Add the service CA bundle to a ConfigMap

A Pod can access the service CA certificate by mounting a ConfigMap that is annotated with service.beta.openshift.io/inject-cabundle=true. Once annotated, the cluster automatically injects the service CA certificate into the service-ca.crt key on the ConfigMap. Access to this CA certificate allows TLS clients to verify connections to services using service serving certificates.

After adding this annotation to a ConfigMap all existing data in it is deleted. It is recommended to use a separate ConfigMap to contain the service-ca.crt, instead of using the same ConfigMap that stores your Pod’s configuration.

Procedure
  1. Annotate the ConfigMap with service.beta.openshift.io/inject-cabundle=true:

    $ oc annotate configmap <configmap-name> \(1)
         service.beta.openshift.io/inject-cabundle=true
    1 Replace <configmap-name> with the name of the ConfigMap to annotate.

    Explicitly referencing the service-ca.crt key in a volumeMount will prevent a Pod from starting until the ConfigMap has been injected with the CA bundle. This behavior can be overridden by setting the optional field to true for the volume’s serving certificate configuration.

    For example, use the following command to annotate the ConfigMap foo:

    $ oc annotate configmap foo service.beta.openshift.io/inject-cabundle=true
  2. View the ConfigMap to ensure that the service CA bundle has been injected:

    $ oc get configmap <configmap-name> -o yaml

    The CA bundle is displayed as the value of the service-ca.crt key in the YAML output:

    apiVersion: v1
    data:
      service-ca.crt: |
        -----BEGIN CERTIFICATE-----
    ...

Add the service CA bundle to an APIService

You can annotate an APIService with service.beta.openshift.io/inject-cabundle=true to have its spec.caBundle field populated with the service CA bundle. This allows the Kubernetes API server to validate the service CA certificate used to secure the targeted endpoint.

Procedure
  1. Annotate the APIService with service.beta.openshift.io/inject-cabundle=true:

    $ oc annotate apiservice <apiservice-name> \(1)
         service.beta.openshift.io/inject-cabundle=true
    1 Replace <apiservice-name> with the name of the APIService to annotate.

    For example, use the following command to annotate the APIService foo:

    $ oc annotate apiservice foo service.beta.openshift.io/inject-cabundle=true
  2. View the APIService to ensure that the service CA bundle has been injected:

    $ oc get apiservice <apiservice-name> -o yaml

    The CA bundle is displayed in the spec.caBundle field in the YAML output:

    apiVersion: apiregistration.k8s.io/v1
    kind: APIService
    metadata:
      annotations:
        service.beta.openshift.io/inject-cabundle: "true"
    ...
    spec:
      caBundle: <CA_BUNDLE>
    ...

Add the service CA bundle to a Custom Resource Definition

You can annotate a Custom Resource Definition (CRD) with service.beta.openshift.io/inject-cabundle=true to have its spec.conversion.webhook.clientConfig.caBundle field populated with the service CA bundle. This allows the Kubernetes API server to validate the service CA certificate used to secure the targeted endpoint.

The service CA bundle will only be injected into the CRD if the CRD is configured to use a webhook for conversion. It is only useful to inject the service CA bundle if a CRD’s webhook is secured with a service CA certificate.

Procedure
  1. Annotate the CRD with service.beta.openshift.io/inject-cabundle=true:

    $ oc annotate crd <crd-name> \(1)
         service.beta.openshift.io/inject-cabundle=true
    1 Replace <crd-name> with the name of the CRD to annotate.

    For example, use the following command to annotate the CRD foo:

    $ oc annotate crd foo service.beta.openshift.io/inject-cabundle=true
  2. View the CRD to ensure that the service CA bundle has been injected:

    $ oc get crd <crd-name> -o yaml

    The CA bundle is displayed in the spec.conversion.webhook.clientConfig.caBundle field in the YAML output:

    apiVersion: apiextensions.k8s.io/v1
    kind: CustomResourceDefinition
    metadata:
      annotations:
        service.beta.openshift.io/inject-cabundle: "true"
    ...
    spec:
      conversion:
        strategy: Webhook
        webhook:
          clientConfig:
            caBundle: <CA_BUNDLE>
    ...

Add the service CA bundle to a MutatingWebhookConfiguration

You can annotate a MutatingWebhookConfiguration with service.beta.openshift.io/inject-cabundle=true to have the clientConfig.caBundle field of each webhook populated with the service CA bundle. This allows the Kubernetes API server to validate the service CA certificate used to secure the targeted endpoint.

Do not set this annotation for admission webhook configurations that need to specify different CA bundles for different webhooks. If you do, then the service CA bundle will be injected for all webhooks.

Procedure
  1. Annotate the MutatingWebhookConfiguration with service.beta.openshift.io/inject-cabundle=true:

    $ oc annotate mutatingwebhoookconfigurations <mutatingwebhook-name> \(1)
         service.beta.openshift.io/inject-cabundle=true
    1 Replace <mutatingwebhook-name> with the name of the MutatingWebhookConfiguration to annotate.

    For example, use the following command to annotate the MutatingWebhookConfiguration foo:

    $ oc annotate mutatingwebhoookconfigurations foo service.beta.openshift.io/inject-cabundle=true
  2. View the MutatingWebhookConfiguration to ensure that the service CA bundle has been injected:

    $ oc get mutatingwebhookconfigurations <mutatingwebhook-name> -o yaml

    The CA bundle is displayed in the clientConfig.caBundle field of all webhooks in the YAML output:

    apiVersion: admissionregistration.k8s.io/v1
    kind: MutatingWebhookConfiguration
    metadata:
      annotations:
        service.beta.openshift.io/inject-cabundle: "true"
    ...
    webhooks:
    - myWebhook:
      - v1beta1
      clientConfig:
        caBundle: <CA_BUNDLE>
    ...

Add the service CA bundle to a ValidatingWebhookConfiguration

You can annotate a ValidatingWebhookConfiguration with service.beta.openshift.io/inject-cabundle=true to have the clientConfig.caBundle field of each webhook populated with the service CA bundle. This allows the Kubernetes API server to validate the service CA certificate used to secure the targeted endpoint.

Do not set this annotation for admission webhook configurations that need to specify different CA bundles for different webhooks. If you do, then the service CA bundle will be injected for all webhooks.

Procedure
  1. Annotate the ValidatingWebhookConfiguration with service.beta.openshift.io/inject-cabundle=true:

    $ oc annotate validatingwebhookconfigurations <validatingwebhook-name> \(1)
         service.beta.openshift.io/inject-cabundle=true
    1 Replace <validatingwebhook-name> with the name of the ValidatingWebhookConfiguration to annotate.

    For example, use the following command to annotate the ValidatingWebhookConfiguration foo:

    $ oc annotate validatingwebhookconfigurations foo service.beta.openshift.io/inject-cabundle=true
  2. View the ValidatingWebhookConfiguration to ensure that the service CA bundle has been injected:

    $ oc get validatingwebhookconfigurations <validatingwebhook-name> -o yaml

    The CA bundle is displayed in the clientConfig.caBundle field of all webhooks in the YAML output:

    apiVersion: admissionregistration.k8s.io/v1
    kind: ValidatingWebhookConfiguration
    metadata:
      annotations:
        service.beta.openshift.io/inject-cabundle: "true"
    ...
    webhooks:
    - myWebhook:
      - v1beta1
      clientConfig:
        caBundle: <CA_BUNDLE>
    ...

Manually rotate the generated service certificate

You can rotate the service certificate by deleting the associated secret. Deleting the secret results in a new one being automatically created, resulting in a new certificate.

Prerequisites
  • A secret containing the certificate and key pair must have been generated for the service.

Procedure
  1. Examine the service to determine the secret containing the certificate. This is found in the serving-cert-secret-name annotation, as seen below.

    $ oc describe service <service-name>
    ...
    service.beta.openshift.io/serving-cert-secret-name: <secret>
    ...
  2. Delete the generated secret for the service. This process will automatically recreate the secret.

    $ oc delete secret <secret> (1)
    1 Replace <secret> with the name of the secret from the previous step.
  3. Confirm that the certificate has been recreated by obtaining the new secret and examining the AGE.

    $ oc get secret <service-name>
    
    NAME              TYPE                DATA   AGE
    <service.name>    kubernetes.io/tls   2      1s

Manually rotate the service CA certificate

The service CA is valid for 26 months and is automatically refreshed when there is less than six months validity left.

If necessary, you can manually refresh the service CA by using the following procedure.

A manually-rotated service CA does not maintain trust with the previous service CA. You might experience a temporary service disruption until the Pods in the cluster are restarted, which ensures that Pods are using service serving certificates issued by the new service CA.

Prerequisites
  • You must be logged in as a cluster admin.

Procedure
  1. View the expiration date of the current service CA certificate by using the following command.

    $ oc get secrets/signing-key -n openshift-service-ca \
         -o template='{{index .data "tls.crt"}}' \
         | base64 -d \
         | openssl x509 -noout -enddate
  2. Manually rotate the service CA. This process generates a new service CA which will be used to sign the new service certificates.

    $ oc delete secret/signing-key -n openshift-service-ca
  3. To apply the new certificates to all services, restart all the Pods in your cluster. This command ensures that all services use the updated certificates.

    $ for I in $(oc get ns -o jsonpath='{range .items[*]} {.metadata.name}{"\n"} {end}'); \
          do oc delete pods --all -n $I; \
          sleep 1; \
          done

    This command will cause a service interruption, as it goes through and deletes every running pod in every namespace. These pods will automatically restart after they are deleted.