Dynamically issuing certificates using the cert-manager Operator on ROSA | Tutorials

Prerequisites
- Setting up your environment
Preparing your AWS account
Installing the cert-manager Operator
Creating a custom domain ingress controller
Configuring dynamic certificates for custom domain routes
Deploying a sample application
Debugging

While wildcard certificates provide simplicity by securing all first-level subdomains of a given domain with a single certificate, other use cases may require the use of individual certificates per domain. This tutorial guides you through using the cert-manager Operator for Red Hat OpenShift and Let’s Encrypt to dynamically issue certificates for routes created using a custom domain.

Prerequisites

A ROSA cluster
You have access to the OpenShift CLI (oc)
You have access to the AWS CLI (aws)
A unique domain, such as *.apps.<company_name>.io
An Amazon Route 53 public hosted zone for the above domain

Setting up your environment

Configure the following environment variables:

$ export DOMAIN=apps.<company_name>.io (1)
$ export EMAIL=<youremail@company_name.io> (2)
$ export AWS_PAGER=""
$ export CLUSTER_NAME=$(oc get infrastructure cluster -o=jsonpath="{.status.infrastructureName}"  | sed 's/-[a-z0-9]\{5\}$//')
$ export OIDC_ENDPOINT=$(oc get authentication.config.openshift.io cluster -o json | jq -r .spec.serviceAccountIssuer | sed  's|^https://||')
$ export REGION=$(oc get infrastructure cluster -o=jsonpath="{.status.platformStatus.aws.region}")
$ export AWS_ACCOUNT_ID=$(aws sts get-caller-identity --query Account --output text)
$ export SCRATCH="/tmp/${CLUSTER_NAME}/dynamic-certs"
$ mkdir -p ${SCRATCH}
$ echo "Cluster: ${CLUSTER_NAME}, Region: ${REGION}, OIDC Endpoint: ${OIDC_ENDPOINT}, AWS Account ID: ${AWS_ACCOUNT_ID}"

1	The custom domain.
2	The e-mail Let’s Encrypt will use to send notifications about your certificates.

Preparing your AWS account

When cert-manager requests a certificate from Let’s Encrypt (or another ACME certificate issuer), Let’s Encrypt servers validate that you control the domain name in that certificate using “challenges". For this tutorial, you are using a DNS-01 challenge that proves that you control the DNS for your domain name by putting a specific value in a TXT record under that domain name. This is all done automatically by cert-manager. To allow cert-manager permission to modify the Amazon Route 53 public hosted zone for your domain, you need to create an IAM role with specific policy permissions and a trust relationship to allow access to the pod.

The public hosted zone that is used in this tutorial is in the same account as the ROSA cluster. If your public hosted zone is in a different account, a few additional steps for Cross Account Access are required.

Retrieve the Amazon Route 53 public hosted zone ID:

This command will look for a public hosted zone that matches the custom domain you specified earlier. You can manually specify the Amazon Route 53 public hosted zone by running export ZONE_ID=<zone_ID> and replacing <zone_ID> with your specific Amazon Route 53 public hosted zone ID.

$ export ZONE_ID=$(aws route53 list-hosted-zones-by-name --output json \
  --dns-name "${DOMAIN}." --query 'HostedZones[0]'.Id --out text | sed 's/\/hostedzone\///')

Create an AWS IAM policy document for the cert-manager Operator that provides the ability to update only the specified public hosted zone:

$ cat <<EOF > "${SCRATCH}/cert-manager-policy.json"
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": "route53:GetChange",
      "Resource": "arn:aws:route53:::change/*"
    },
    {
      "Effect": "Allow",
      "Action": [
        "route53:ChangeResourceRecordSets",
        "route53:ListResourceRecordSets"
      ],
      "Resource": "arn:aws:route53:::hostedzone/${ZONE_ID}"
    },
    {
      "Effect": "Allow",
      "Action": "route53:ListHostedZonesByName",
      "Resource": "*"
    }
  ]
}
EOF

Create the IAM Policy using the file you created in the previous step:

$ POLICY_ARN=$(aws iam create-policy --policy-name "${CLUSTER_NAME}-cert-manager-policy" \
  --policy-document file://${SCRATCH}/cert-manager-policy.json \
  --query 'Policy.Arn' --output text)

Create an AWS IAM trust policy for the cert-manager Operator:

$ cat <<EOF > "${SCRATCH}/trust-policy.json"
{
 "Version": "2012-10-17",
 "Statement": [
 {
 "Effect": "Allow",
 "Condition": {
   "StringEquals" : {
     "${OIDC_ENDPOINT}:sub": "system:serviceaccount:cert-manager:cert-manager"
   }
 },
 "Principal": {
   "Federated": "arn:aws:iam::$AWS_ACCOUNT_ID:oidc-provider/${OIDC_ENDPOINT}"
 },
 "Action": "sts:AssumeRoleWithWebIdentity"
 }
 ]
}
EOF

Create an IAM Role for the cert-manager Operator using the trust policy you created in the previous step:

$ ROLE_ARN=$(aws iam create-role --role-name "${CLUSTER_NAME}-cert-manager-operator" \
   --assume-role-policy-document "file://${SCRATCH}/trust-policy.json" \
   --query Role.Arn --output text)

Attach the permissions policy to the role:

$ aws iam attach-role-policy --role-name "${CLUSTER_NAME}-cert-manager-operator" \
  --policy-arn ${POLICY_ARN}

Installing the cert-manager Operator

Create a project to install the cert-manager Operator into:

$ oc new-project cert-manager-operator

Do not attempt to use more than one cert-manager Operator in your cluster. If you have a community cert-manager Operator installed in your cluster, you must uninstall it before installing the cert-manager Operator for Red Hat OpenShift.

Install the cert-manager Operator for Red Hat OpenShift:

$ cat << EOF | oc apply -f -
apiVersion: operators.coreos.com/v1
kind: OperatorGroup
metadata:
  name: openshift-cert-manager-operator-group
  namespace: cert-manager-operator
spec:
  targetNamespaces:
  - cert-manager-operator
---
apiVersion: operators.coreos.com/v1alpha1
kind: Subscription
metadata:
  name: openshift-cert-manager-operator
  namespace: cert-manager-operator
spec:
  channel: stable-v1
  installPlanApproval: Automatic
  name: openshift-cert-manager-operator
  source: redhat-operators
  sourceNamespace: openshift-marketplace
EOF

It will take a few minutes for this Operator to install and complete its set up.

Verify that the cert-manager Operator is running:

$ oc -n cert-manager-operator get pods

Example output

NAME                                                        READY   STATUS    RESTARTS   AGE
cert-manager-operator-controller-manager-84b8799db5-gv8mx   2/2     Running   0          12s

Annotate the service account used by the cert-manager pods with the AWS IAM role you created earlier:
```
$ oc -n cert-manager annotate serviceaccount cert-manager eks.amazonaws.com/role-arn=${ROLE_ARN}
```
Restart the existing cert-manager controller pod by running the following command:
```
$ oc -n cert-manager delete pods -l app.kubernetes.io/name=cert-manager
```

Patch the Operator’s configuration to use external nameservers to prevent DNS-01 challenge resolution issues:

$ oc patch certmanager.operator.openshift.io/cluster --type merge \
  -p '{"spec":{"controllerConfig":{"overrideArgs":["--dns01-recursive-nameservers-only","--dns01-recursive-nameservers=1.1.1.1:53"]}}}'

Create a ClusterIssuer resource to use Let’s Encrypt by running the following command:

$ cat << EOF | oc apply -f -
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: letsencrypt-production
spec:
  acme:
    server: https://acme-v02.api.letsencrypt.org/directory
    email: ${EMAIL}
    # This key doesn't exist, cert-manager creates it
    privateKeySecretRef:
      name: prod-letsencrypt-issuer-account-key
    solvers:
    - dns01:
        route53:
         hostedZoneID: ${ZONE_ID}
         region: ${REGION}
         secretAccessKeySecretRef:
           name: ''
EOF

Check the ClusterIssuer resource to confirm it is ready:

$ oc get clusterissuer.cert-manager.io/letsencrypt-production

Example output

NAME                     READY   AGE
letsencrypt-production   True    47s

Creating a custom domain ingress controller

Create a new project:
```
$ oc new-project custom-domain-ingress
```
Create a certificate resource to provision a certificate for the custom domain Ingress Controller:

The following example uses a single domain certificate. SAN and wildcard certificates are also supported.

Configure the certificate:

$ cat << EOF | oc apply -f -
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: custom-domain-ingress-cert
  namespace: custom-domain-ingress
spec:
  secretName: custom-domain-ingress-cert-tls
  issuerRef:
     name: letsencrypt-production
     kind: ClusterIssuer
  commonName: "${DOMAIN}"
  dnsNames:
  - "${DOMAIN}"
EOF

Verify the certificate has been issued:

It will take a few minutes for this certificate to be issued by Let’s Encrypt. If it takes longer than 5 minutes, run oc -n custom-domain-ingress describe certificate.cert-manager.io/custom-domain-ingress-cert to see any issues reported by cert-manager.

$ oc -n custom-domain-ingress get certificate.cert-manager.io/custom-domain-ingress-cert

Example output

NAME                         READY   SECRET                           AGE
custom-domain-ingress-cert   True    custom-domain-ingress-cert-tls   9m53s

Create a new CustomDomain custom resource (CR):

$ cat << EOF | oc apply -f -
apiVersion: managed.openshift.io/v1alpha1
kind: CustomDomain
metadata:
  name: custom-domain-ingress
spec:
  domain: ${DOMAIN}
  scope: External
  loadBalancerType: NLB
  certificate:
    name: custom-domain-ingress-cert-tls
    namespace: custom-domain-ingress
EOF

Verify that your custom domain Ingress Controller has been deployed and is ready:

$ oc get customdomains

Example output

NAME                    ENDPOINT                                                               DOMAIN            STATUS
custom-domain-ingress   tfoxdx.custom-domain-ingress.cluster.1234.p1.openshiftapps.com         example.com       Ready

Prepare a document with the necessary DNS changes to enable DNS resolution for your custom domain Ingress Controller:

$ INGRESS=$(oc get customdomain.managed.openshift.io/custom-domain-ingress --template={{.status.endpoint}})
$ cat << EOF > "${SCRATCH}/create-cname.json"
{
  "Comment":"Add CNAME to custom domain endpoint",
  "Changes":[{
      "Action":"CREATE",
      "ResourceRecordSet":{
        "Name": "*.${DOMAIN}",
      "Type":"CNAME",
      "TTL":30,
      "ResourceRecords":[{
        "Value": "${INGRESS}"
      }]
    }
  }]
}
EOF

Submit your changes to Amazon Route 53 for propagation:

$ aws route53 change-resource-record-sets \
  --hosted-zone-id ${ZONE_ID} \
  --change-batch file://${SCRATCH}/create-cname.json

While the wildcard CNAME record avoids the need to create a new record for every new application you deploy using the custom domain Ingress Controller, the certificate that each of these applications use is not a wildcard certificate.

Configuring dynamic certificates for custom domain routes

At this stage, you can expose cluster applications on any first-level subdomains of the specified domain, but the connection will not be secured with a TLS certificate that matches the domain of the application. To ensure these cluster applications have valid certificates for each domain name, configure cert-manager to dynamically issue a certificate to every new route created under this domain.

Create the necessary OpenShift resources cert-manager requires to manage certificates for OpenShift routes:

This step creates a new deployment (and therefore a pod) that specifically monitors annotated routes in the cluster. If the issuer-kind and issuer-name annotations are found in a new route, it requests the Issuer (ClusterIssuer in this case) for a new certificate that is unique to this route and which will honor the hostname that was specified while creating the route.

If the cluster does not have access to GitHub, you can save the raw contents locally and run oc apply -f localfilename.yaml -n cert-manager.
```
$ oc -n cert-manager apply -f https://github.com/cert-manager/openshift-routes/releases/latest/download/cert-manager-openshift-routes.yaml
```
The following additional OpenShift resources are also created in this step:
- ClusterRole - grants permissions to watch and update the routes across the cluster
- ServiceAccount - uses permissions to run the newly created pod
- ClusterRoleBinding - binds these two resources

Ensure that the new pod cert-manager-openshift-routes is running successfully by running the following command:

$ oc -n cert-manager get pods

Example result

NAME                                             READY   STATUS    RESTARTS   AGE
cert-manager-866d8f788c-9kspc                    1/1     Running   0          4h21m
cert-manager-cainjector-6885c585bd-znws8         1/1     Running   0          4h41m
cert-manager-openshift-routes-75b6bb44cd-f8kd5   1/1     Running   0          6s
cert-manager-webhook-8498785dd9-bvfdf            1/1     Running   0          4h41m

Deploying a sample application

Create a new project for your sample application:
```
$ oc new-project hello-world
```

Deploy a hello world application:

$ oc -n hello-world new-app --image=docker.io/openshift/hello-openshift

Create a route to expose the application from outside the cluster:

$ oc -n hello-world create route edge --service=hello-openshift hello-openshift-tls --hostname hello.${DOMAIN}

Verify the certificate for the route is untrusted:

$ curl -I https://hello.${DOMAIN}

Example output

curl: (60) SSL: no alternative certificate subject name matches target host name 'hello.example.com'
More details here: https://curl.se/docs/sslcerts.html

curl failed to verify the legitimacy of the server and therefore could not
establish a secure connection to it. To learn more about this situation and
how to fix it, please visit the web page mentioned above.

Annotate the route to trigger cert-manager to provision a certificate for the custom domain:
```
$ oc -n hello-world annotate route hello-openshift-tls cert-manager.io/issuer-kind=ClusterIssuer cert-manager.io/issuer-name=letsencrypt-production
```
It will take a 2-3 minutes for the certificate to be created. The renewal of the certificate will automatically be managed by the cert-manager Operator as it approaches expiration.

Verify the certificate for the route is now trusted:

$ curl -I https://hello.${DOMAIN}

Example output

HTTP/2 200
date: Thu, 05 Oct 2023 23:45:33 GMT
content-length: 17
content-type: text/plain; charset=utf-8
set-cookie: 52e4465485b6fb4f8a1b1bed128d0f3b=68676068bb32d24f0f558f094ed8e4d7; path=/; HttpOnly; Secure; SameSite=None
cache-control: private

Debugging

The validation process usually takes 2-3 minutes to complete while creating certificates.

If you face issues during the certificate create step, run oc describe against each of the certificate,certificaterequest,order, and challenge resources to view the events or reasons that will help identify the cause of the issue.

$ oc get certificate,certificaterequest,order,challenge

This is a very helpful guide in debugging certificates as well.

You can also use the cmctl CLI tool for various certificate management activities, such checking the status of certificates and testing renewals.

Tutorial: Dynamically issuing certificates using the cert-manager Operator on ROSA

Prerequisites

Setting up your environment

Preparing your AWS account

Installing the cert-manager Operator

Creating a custom domain ingress controller

Configuring dynamic certificates for custom domain routes

Deploying a sample application

Debugging