OpenShift docs are moving and will soon only be available at docs.redhat.com, the home of all Red Hat product documentation. Explore the new docs experience today.
  1. Documentation
    2. history bug_report picture_as_pdf
×

Tutorial: Dynamically issuing certificates using the cert-manager Operator on ROSA

edit

While wildcard certificates provide simplicity by securing all first-level subdomains of a given domain with a single certificate, other use cases can require the use of individual certificates per domain.

Learn how to use the cert-manager Operator for Red Hat OpenShift and Let’s Encrypt to dynamically issue certificates for routes created using a custom domain.

edit

Prerequisites

  • A ROSA cluster (HCP or Classic)

  • A user account with cluster-admin privileges

  • The OpenShift CLI (oc)

  • The Amazon Web Services (AWS) CLI (aws)

  • A unique domain, such as *.apps.example.com

  • An Amazon Route 53 public hosted zone for the above domain

Setting up your environment

  1. Configure the following environment variables:

    $ export DOMAIN=apps.example.com (1)
$ export EMAIL=email@example.com (2)
$ export AWS_PAGER=""
$ export CLUSTER=$(oc get infrastructure cluster -o=jsonpath="{.status.infrastructureName}"  | sed 's/-[a-z0-9]\{5\}$//')
$ export OIDC_ENDPOINT=$(oc get authentication.config.openshift.io cluster -o json | jq -r .spec.serviceAccountIssuer | sed  's|^https://||')
$ export REGION=$(oc get infrastructure cluster -o=jsonpath="{.status.platformStatus.aws.region}")
$ export AWS_ACCOUNT_ID=$(aws sts get-caller-identity --query Account --output text)
$ export SCRATCH="/tmp/${CLUSTER}/dynamic-certs"
$ mkdir -p ${SCRATCH}
    1 Replace with the custom domain you want to use for the IngressController.
    2 Replace with the e-mail you want Let’s Encrypt to use to send notifications about your certificates.

  2. Ensure all fields output correctly before moving to the next section:

    $ echo "Cluster: ${CLUSTER}, Region: ${REGION}, OIDC Endpoint: ${OIDC_ENDPOINT}, AWS Account ID: ${AWS_ACCOUNT_ID}"

    The "Cluster" output from the previous command may be the name of your cluster, the internal ID of your cluster, or the cluster’s domain prefix. If you prefer to use another identifier, you can manually set this value by running the following command:

    $ export CLUSTER=my-custom-value

Preparing your AWS account

When cert-manager requests a certificate from Let’s Encrypt (or another ACME certificate issuer), Let’s Encrypt servers validate that you control the domain name in that certificate using challenges. For this tutorial, you are using a DNS-01 challenge that proves that you control the DNS for your domain name by putting a specific value in a TXT record under that domain name. This is all done automatically by cert-manager. To allow cert-manager permission to modify the Amazon Route 53 public hosted zone for your domain, you need to create an Identity Access Management (IAM) role with specific policy permissions and a trust relationship to allow access to the pod.

The public hosted zone that is used in this tutorial is in the same AWS account as the ROSA cluster. If your public hosted zone is in a different account, a few additional steps for Cross Account Access are required.

  1. Retrieve the Amazon Route 53 public hosted zone ID:

    This command looks for a public hosted zone that matches the custom domain you specified earlier as the DOMAIN environment variable. You can manually specify the Amazon Route 53 public hosted zone by running export ZONE_ID=<zone_ID>, replacing <zone_ID> with your specific Amazon Route 53 public hosted zone ID.

    		 
    $ export ZONE_ID=$(aws route53 list-hosted-zones-by-name --output json \
  --dns-name "${DOMAIN}." --query 'HostedZones[0]'.Id --out text | sed 's/\/hostedzone\///')

  2. Create an AWS IAM policy document for the cert-manager Operator that provides the ability to update only the specified public hosted zone:

    $ cat <<EOF > "${SCRATCH}/cert-manager-policy.json"
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": "route53:GetChange",
      "Resource": "arn:aws:route53:::change/*"
    },
    {
      "Effect": "Allow",
      "Action": [
        "route53:ChangeResourceRecordSets",
        "route53:ListResourceRecordSets"
      ],
      "Resource": "arn:aws:route53:::hostedzone/${ZONE_ID}"
    },
    {
      "Effect": "Allow",
      "Action": "route53:ListHostedZonesByName",
      "Resource": "*"
    }
  ]
}
EOF

  3. Create the IAM policy using the file you created in the previous step:

    $ POLICY_ARN=$(aws iam create-policy --policy-name "${CLUSTER}-cert-manager-policy" \
  --policy-document file://${SCRATCH}/cert-manager-policy.json \
  --query 'Policy.Arn' --output text)

  4. Create an AWS IAM trust policy for the cert-manager Operator:

    $ cat <<EOF > "${SCRATCH}/trust-policy.json"
{
 "Version": "2012-10-17",
 "Statement": [
 {
 "Effect": "Allow",
 "Condition": {
   "StringEquals" : {
     "${OIDC_ENDPOINT}:sub": "system:serviceaccount:cert-manager:cert-manager"
   }
 },
 "Principal": {
   "Federated": "arn:aws:iam::$AWS_ACCOUNT_ID:oidc-provider/${OIDC_ENDPOINT}"
 },
 "Action": "sts:AssumeRoleWithWebIdentity"
 }
 ]
}
EOF

  5. Create an IAM role for the cert-manager Operator using the trust policy you created in the previous step:

    $ ROLE_ARN=$(aws iam create-role --role-name "${CLUSTER}-cert-manager-operator" \
   --assume-role-policy-document "file://${SCRATCH}/trust-policy.json" \
   --query Role.Arn --output text)

  6. Attach the permissions policy to the role:

    $ aws iam attach-role-policy --role-name "${CLUSTER}-cert-manager-operator" \
  --policy-arn ${POLICY_ARN}

Installing the cert-manager Operator

  1. Create a project to install the cert-manager Operator into:

    $ oc new-project cert-manager-operator

    Do not attempt to use more than one cert-manager Operator in your cluster. If you have a community cert-manager Operator installed in your cluster, you must uninstall it before installing the cert-manager Operator for Red Hat OpenShift.

  2. Install the cert-manager Operator for Red Hat OpenShift:

    $ cat << EOF | oc apply -f -
apiVersion: operators.coreos.com/v1
kind: OperatorGroup
metadata:
  name: openshift-cert-manager-operator-group
  namespace: cert-manager-operator
spec:
  targetNamespaces:
  - cert-manager-operator
---
apiVersion: operators.coreos.com/v1alpha1
kind: Subscription
metadata:
  name: openshift-cert-manager-operator
  namespace: cert-manager-operator
spec:
  channel: stable-v1
  installPlanApproval: Automatic
  name: openshift-cert-manager-operator
  source: redhat-operators
  sourceNamespace: openshift-marketplace
EOF

    It takes a few minutes for this Operator to install and complete its set up.

  3. Verify that the cert-manager Operator is running:

    $ oc -n cert-manager-operator get pods
    Example output
    NAME                                                        READY   STATUS    RESTARTS   AGE
cert-manager-operator-controller-manager-84b8799db5-gv8mx   2/2     Running   0          12s

  4. Annotate the service account used by the cert-manager pods with the AWS IAM role you created earlier:

    $ oc -n cert-manager annotate serviceaccount cert-manager eks.amazonaws.com/role-arn=${ROLE_ARN}

  5. Restart the existing cert-manager controller pod by running the following command:

    $ oc -n cert-manager delete pods -l app.kubernetes.io/name=cert-manager

  6. Patch the Operator’s configuration to use external nameservers to prevent DNS-01 challenge resolution issues:

    $ oc patch certmanager.operator.openshift.io/cluster --type merge \
  -p '{"spec":{"controllerConfig":{"overrideArgs":["--dns01-recursive-nameservers-only","--dns01-recursive-nameservers=1.1.1.1:53"]}}}'

  7. Create a ClusterIssuer resource to use Let’s Encrypt by running the following command:

    $ cat << EOF | oc apply -f -
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: letsencrypt-production
spec:
  acme:
    server: https://acme-v02.api.letsencrypt.org/directory
    email: ${EMAIL}
    # This key doesn't exist, cert-manager creates it
    privateKeySecretRef:
      name: prod-letsencrypt-issuer-account-key
    solvers:
    - dns01:
        route53:
         hostedZoneID: ${ZONE_ID}
         region: ${REGION}
         secretAccessKeySecretRef:
           name: ''
EOF

  8. Verify the ClusterIssuer resource is ready:

    $ oc get clusterissuer.cert-manager.io/letsencrypt-production
    Example output
    NAME                     READY   AGE
letsencrypt-production   True    47s

Creating a custom domain Ingress Controller

  1. Create and configure a certificate resource to provision a certificate for the custom domain Ingress Controller:

    The following example uses a single domain certificate. SAN and wildcard certificates are also supported.

    		 
    $ cat << EOF | oc apply -f -
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: custom-domain-ingress-cert
  namespace: openshift-ingress
spec:
  secretName: custom-domain-ingress-cert-tls
  issuerRef:
     name: letsencrypt-production
     kind: ClusterIssuer
  commonName: "${DOMAIN}"
  dnsNames:
  - "${DOMAIN}"
EOF

  2. Verify the certificate has been issued:

    It takes a few minutes for this certificate to be issued by Let’s Encrypt. If it takes longer than 5 minutes, run oc -n openshift-ingress describe certificate.cert-manager.io/custom-domain-ingress-cert to see any issues reported by cert-manager.

    		 
    $ oc -n openshift-ingress get certificate.cert-manager.io/custom-domain-ingress-cert
    Example output
    NAME                         READY   SECRET                           AGE
custom-domain-ingress-cert   True    custom-domain-ingress-cert-tls   9m53s

  3. Create a new IngressController resource:

    $ cat << EOF | oc apply -f -
apiVersion: operator.openshift.io/v1
kind: IngressController
metadata:
  name: custom-domain-ingress
  namespace: openshift-ingress-operator
spec:
  domain: ${DOMAIN}
  defaultCertificate:
    name: custom-domain-ingress-cert-tls
  endpointPublishingStrategy:
    loadBalancer:
      dnsManagementPolicy: Unmanaged
      providerParameters:
        aws:
          type: NLB
        type: AWS
      scope: External
    type: LoadBalancerService
EOF

    This IngressController example will create an internet accessible Network Load Balancer (NLB) in your AWS account. To provision an internal NLB instead, set the .spec.endpointPublishingStrategy.loadBalancer.scope parameter to Internal before creating the IngressController resource.

  4. Verify that your custom domain IngressController has successfully created an external load balancer:

    $ oc -n openshift-ingress get service/router-custom-domain-ingress
    Example output
    NAME                           TYPE           CLUSTER-IP      EXTERNAL-IP                                                                     PORT(S)                      AGE
router-custom-domain-ingress   LoadBalancer   172.30.174.34   a309962c3bd6e42c08cadb9202eca683-1f5bbb64a1f1ec65.elb.us-east-1.amazonaws.com   80:31342/TCP,443:31821/TCP   7m28s

  5. Prepare a document with the necessary DNS changes to enable DNS resolution for your custom domain Ingress Controller:

    $ INGRESS=$(oc -n openshift-ingress get service/router-custom-domain-ingress -ojsonpath="{.status.loadBalancer.ingress[0].hostname}")
$ cat << EOF > "${SCRATCH}/create-cname.json"
{
  "Comment":"Add CNAME to custom domain endpoint",
  "Changes":[{
      "Action":"CREATE",
      "ResourceRecordSet":{
        "Name": "*.${DOMAIN}",
      "Type":"CNAME",
      "TTL":30,
      "ResourceRecords":[{
        "Value": "${INGRESS}"
      }]
    }
  }]
}
EOF

  6. Submit your changes to Amazon Route 53 for propagation:

    $ aws route53 change-resource-record-sets \
  --hosted-zone-id ${ZONE_ID} \
  --change-batch file://${SCRATCH}/create-cname.json

    While the wildcard CNAME record avoids the need to create a new record for every new application you deploy using the custom domain Ingress Controller, the certificate that each of these applications use is not a wildcard certificate.

Configuring dynamic certificates for custom domain routes

Now you can expose cluster applications on any first-level subdomains of the specified domain, but the connection will not be secured with a TLS certificate that matches the domain of the application. To ensure these cluster applications have valid certificates for each domain name, configure cert-manager to dynamically issue a certificate to every new route created under this domain.

  1. Create the necessary OpenShift resources cert-manager requires to manage certificates for OpenShift routes.

    This step creates a new deployment (and therefore a pod) that specifically monitors annotated routes in the cluster. If the issuer-kind and issuer-name annotations are found in a new route, it requests the Issuer (ClusterIssuer in this case) for a new certificate that is unique to this route and which will honor the hostname that was specified while creating the route.

    If the cluster does not have access to GitHub, you can save the raw contents locally and run oc apply -f localfilename.yaml -n cert-manager.

    		 
    $ oc -n cert-manager apply -f https://github.com/cert-manager/openshift-routes/releases/latest/download/cert-manager-openshift-routes.yaml

    The following additional OpenShift resources are also created in this step:

    • ClusterRole - grants permissions to watch and update the routes across the cluster

    • ServiceAccount - uses permissions to run the newly created pod

    • ClusterRoleBinding - binds these two resources

  2. Ensure that the new cert-manager-openshift-routes pod is running successfully:

    $ oc -n cert-manager get pods
    Example result
    NAME                                             READY   STATUS    RESTARTS   AGE
cert-manager-866d8f788c-9kspc                    1/1     Running   0          4h21m
cert-manager-cainjector-6885c585bd-znws8         1/1     Running   0          4h41m
cert-manager-openshift-routes-75b6bb44cd-f8kd5   1/1     Running   0          6s
cert-manager-webhook-8498785dd9-bvfdf            1/1     Running   0          4h41m

Deploying a sample application

Now that dynamic certificates are configured, you can deploy a sample application to confirm that certificates are provisioned and trusted when you expose a new route.

  1. Create a new project for your sample application:

    $ oc new-project hello-world

  2. Deploy a hello world application:

    $ oc -n hello-world new-app --image=docker.io/openshift/hello-openshift

  3. Create a route to expose the application from outside the cluster:

    $ oc -n hello-world create route edge --service=hello-openshift hello-openshift-tls --hostname hello.${DOMAIN}

  4. Verify the certificate for the route is untrusted:

    $ curl -I https://hello.${DOMAIN}
    Example output
    curl: (60) SSL: no alternative certificate subject name matches target host name 'hello.example.com'
More details here: https://curl.se/docs/sslcerts.html

curl failed to verify the legitimacy of the server and therefore could not
establish a secure connection to it. To learn more about this situation and
how to fix it, please visit the web page mentioned above.

  5. Annotate the route to trigger cert-manager to provision a certificate for the custom domain:

    $ oc -n hello-world annotate route hello-openshift-tls cert-manager.io/issuer-kind=ClusterIssuer cert-manager.io/issuer-name=letsencrypt-production

    It takes 2-3 minutes for the certificate to be created. The renewal of the certificate will automatically be managed by the cert-manager Operator as it approaches expiration.

  6. Verify the certificate for the route is now trusted:

    $ curl -I https://hello.${DOMAIN}
    Example output
    HTTP/2 200
date: Thu, 05 Oct 2023 23:45:33 GMT
content-length: 17
content-type: text/plain; charset=utf-8
set-cookie: 52e4465485b6fb4f8a1b1bed128d0f3b=68676068bb32d24f0f558f094ed8e4d7; path=/; HttpOnly; Secure; SameSite=None
cache-control: private

Troubleshooting dynamic certificate provisioning

The validation process usually takes 2-3 minutes to complete while creating certificates.

If annotating your route does not trigger certificate creation during the certificate create step, run oc describe against each of the certificate,certificaterequest,order, and challenge resources to view the events or reasons that can help identify the cause of the issue.

$ oc get certificate,certificaterequest,order,challenge

For troubleshooting, you can refer to this helpful guide in debugging certificates.

You can also use the cmctl CLI tool for various certificate management activities, such as checking the status of certificates and testing renewals.