Overview

This topic reviews how to check cluster certificate expiration dates, back up the certificates, and redeploy them using Ansible playbooks. This method fixes common certificate errors. Possible use cases include:

  • The installer detected the wrong host names and the issue was identified too late.

  • The certificates are expired and you need to update them.

  • You have a new CA and would like to create certificates using it instead

Checking Certificate Expirations

You can use the installer to warn you about any certificates expiring within a configurable window of days and notify you about any certificates that have already expired. Certificate expiry playbooks use the Ansible role openshift_certificate_expiry.

Certificates examined by the role include:

  • Master and node service certificates

  • Router and registry service certificates from etcd secrets

  • Master, node, router, registry, and kubeconfig files for cluster-admin users

  • etcd certificates (including embedded)

Role Variables

The openshift_certificate_expiry role uses the following variables:

Table 1. Core Variables
Variable Name Default Value Description

openshift_certificate_expiry_config_base

/etc/origin

Base OpenShift Enterprise configuration directory.

openshift_certificate_expiry_warning_days

30

Flag certificates that will expire in this many days from now.

openshift_certificate_expiry_show_all

no

Include healthy (non-expired and non-warning) certificates in results.

Table 2. Optional Variables
Variable Name Default Value Description

openshift_certificate_expiry_generate_html_report

no

Generate an HTML report of the expiry check results.

openshift_certificate_expiry_html_report_path

/tmp/cert-expiry-report.html

The full path for saving the HTML report.

openshift_certificate_expiry_save_json_results

no

Save expiry check results as a JSON file.

openshift_certificate_expiry_json_results_path

/tmp/cert-expiry-report.json

The full path for saving the JSON report.

Running Certificate Expiration Playbooks

The OpenShift Enterprise installer provides a set of example certificate expiration playbooks, using different sets of configuration for the openshift_certificate_expiry role.

Just like the redeploying certificates playbook, these playbooks must be used with an inventory that is representative of the cluster. For best results, run ansible-playbook with the -v option.

Using the easy-mode.yaml example playbook, you can try the role out before tweaking it to your specifications as needed. This playbook:

  • Produces JSON and stylized HTML reports in /tmp/.

  • Sets the warning window very large, so you will almost always get results back.

  • Includes all certificates (healthy or not) in the results.

easy-mode.yaml Playbook
- name: Check cert expirys
  hosts: nodes:masters:etcd
  become: yes
  gather_facts: no
  vars:
    openshift_certificate_expiry_warning_days: 1500
    openshift_certificate_expiry_save_json_results: yes
    openshift_certificate_expiry_generate_html_report: yes
    openshift_certificate_expiry_show_all: yes
  roles:
    - role: openshift_certificate_expiry

To run the easy-mode.yaml playbook:

$ ansible-playbook -v -i <inventory_file> \
    /usr/share/ansible/openshift-ansible/playbooks/certificate_expiry/easy-mode.yaml

Other Example Playbooks

The other example playbooks are also available to run directly out of the /usr/share/ansible/openshift-ansible/playbooks/certificate_expiry/ directory.

Table 3. Other Example Playbooks
File Name Usage

default.yaml

Produces the default behavior of the openshift_certificate_expiry role.

html_and_json_default_paths.yaml

Generates HTML and JSON artifacts in their default paths.

longer_warning_period.yaml

Changes the expiration warning window to 1500 days.

longer-warning-period-json-results.yaml

Changes the expiration warning window to 1500 days and saves the results as a JSON file.

To run any of these example playbooks:

$ ansible-playbook -v -i <inventory_file> \
    /usr/share/ansible/openshift-ansible/playbooks/certificate_expiry/<playbook>

Output Formats

As noted above, there are two ways to format your check report. In JSON format for machine parsing, or as a stylized HTML page for easy skimming.

HTML Report

An example of an HTML report is provided with the installer. You can open the following file in your browser to view it:

/usr/share/ansible/openshift-ansible/roles/openshift_certificate_expiry/examples/cert-expiry-report.html

JSON Report

There are two top-level keys in the saved JSON results: data and summary.

The data key is a hash where the keys are the names of each host examined and the values are the check results for the certificates identified on each respective host.

The summary key is a hash that summarizes the total number of certificates:

  • examined on the entire cluster

  • that are OK

  • expiring within the configured warning window

  • already expired

For an example of the full JSON report, see /usr/share/ansible/openshift-ansible/roles/openshift_certificate_expiry/examples/cert-expiry-report.json.

The summary from the JSON data can be easily checked for warnings or expirations using a variety of command-line tools. For example, using grep you can look for the word summary and print out the two lines after the match (-A2):

$ grep -A2 summary /tmp/cert-expiry-report.json
    "summary": {
        "warning": 16,
        "expired": 0

If available, the jq tool can also be used to pick out specific values. The first two examples below show how to select just one value, either warning or expired. The third example shows how to select both values at once:

$ jq '.summary.warning' /tmp/cert-expiry-report.json
16

$ jq '.summary.expired' /tmp/cert-expiry-report.json
0

$ jq '.summary.warning,.summary.expired' /tmp/cert-expiry-report.json
16
0

Running the Certificate Redeploy Playbook

Running the certificate redeploy playbook will redeploy OpenShift Enterprise certificates that exist on systems (master, node, etcd):

$ ansible-playbook -i <inventory> playbooks/byo/openshift-cluster/redeploy-certificates.yml

This playbook must be run with an inventory that is representative of the cluster. The inventory must specify or override all host names and IP addresses set via openshift_hostname, openshift_public_hostname, openshift_ip, openshift_public_ip, openshift_master_cluster_hostname, or openshift_master_cluster_public_hostname such that they match the current cluster configuration.

By default, the redeploy playbook does not redeploy the OpenShift Enterprise CA. New certificates are created using the original OpenShift Enterprise CA.

To redeploy all certificates including the OpenShift Enterprise CA, specify openshift_certificates_redeploy_ca=true.

For example:

$ ansible-playbook -i <inventory> playbooks/byo/openshift-cluster/redeploy-certificates.yml \
--extra-vars "openshift_certificates_redeploy_ca=true"

This also adds a custom CA certificate:

# Configure custom ca certificate
# NOTE: CA certificate will not be replaced with existing clusters.
# This option may only be specified when creating a new cluster or
# when redeploying cluster certificates with the redeploy-certificates
# playbook.
#openshift_master_ca_certificate={'certfile': '/path/to/ca.crt', 'keyfile': '/path/to/ca.key'}

All pods using service accounts to communicate with the OpenShift Enterprise API must be redeployed when the OpenShift Enterprise CA is replaced so the certificate redeploy playbook will serially evacuate all nodes in the cluster when this variable is set.