Chapter 2. Single-cluster deployments

Use this setup to deploy Red Hat build of Keycloak to an OpenShift cluster.

The behaviour and high-availability performance of the Red Hat build of Keycloak deployment are ultimately determined by the configuration of the OpenShift cluster. Typically, OpenShift clusters are deployed on a single availability-zone, however in order to increase fault-tolerance, it is possible to deploy the cluster across multiple availability-zones.

The Red Hat build of Keycloak Operator defines the following topology spread constraints by default to prefer that Red Hat build of Keycloak pods are deployed on distinct nodes and distinct availability-zones when possible:

      topologySpreadConstraints:
        - maxSkew: 1
          topologyKey: "topology.kubernetes.io/zone"
          whenUnsatisfiable: "ScheduleAnyway"
          labelSelector:
            matchLabels:
              app: "keycloak"
              app.kubernetes.io/managed-by: "keycloak-operator"
              app.kubernetes.io/instance: "keycloak"
              app.kubernetes.io/component: "server"
        - maxSkew: 1
          topologyKey: "kubernetes.io/hostname"
          whenUnsatisfiable: "ScheduleAnyway"
          labelSelector:
            matchLabels:
              app: "keycloak"
              app.kubernetes.io/managed-by: "keycloak-operator"
              app.kubernetes.io/instance: "keycloak"
              app.kubernetes.io/component: "server"

Deploying Red Hat build of Keycloak on a single cluster in a single zone, or across multiple availability-zones, or data centers with the required network latency and database configuration, changes the high-availability characteristics significantly, therefore we consider these architectures independently.

Continue reading in the Building blocks single-cluster deployments chapter to find blueprints for the different building blocks.

Red Hat build of Keycloak requires a low latency network connection for the synchronous replication of data by the database and Red Hat build of Keycloak clustering.

A round-trip latency of less than 5 ms is suggested and below 10 ms is required, together with a reliable network between the zones to avoid unexpected problems with latency, throughput or connectivity.

Network latency and latency spikes amplify in the response time of the service and can lead to queued requests, timeouts, and failed requests. Networking problems can cause downtimes until the failure detection isolates problematic nodes.

Suggested setup: OpenShift cluster consisting of two or more AWS Availability Zones within the same AWS Region.

Not considered: OpenShift clusters spread across multiple regions on the same or different continents, as it would increase the latency and the likelihood of network failures. Synchronous replication of databases as services with Aurora Regional Deployments on AWS is only available within the same region.

A synchronously replicated database available across all availability-zones.

A clustered deployment of Red Hat build of Keycloak with pods distributed across availability-zones.

Blueprint: Deploying Red Hat build of Keycloak across multiple availability-zones with the Operator.

Creating new database connections is expensive as it takes time. Creating them when a request arrives will delay the response, so it is good to have them created before the request arrives. It can also contribute to a stampede effect where creating a lot of connections in a short time makes things worse as it slows down the system and blocks threads. Closing a connection also invalidates all server side statements caching for that connection.

For the best performance, the values for the initial, minimal and maximum database connection pool size should all be equal. This avoids creating new database connections when a new request comes in which is costly.

Keeping the database connection open for as long as possible allows for server side statement caching bound to a connection. In the case of PostgreSQL, to use a server-side prepared statement, a query needs to be executed (by default) at least five times.

See the PostgreSQL docs on prepared statements for more information.

Summary:

Recommendations:

Red Hat build of Keycloak, which by default stores user sessions in the database, requires the following resources for optimal performance on an Aurora PostgreSQL multi-AZ database:

For every 100 login/logout/refresh requests per second:

The vCPU requirement is given as a range, as with an increased CPU saturation on the database host the CPU usage per request decreases while the response times increase. A lower CPU quota on the database can lead to slower response times during peak loads. Choose a larger CPU quota if fast response times during peak loads are critical. See below for an example.

Aurora database clusters consist of multiple Aurora database instances, with one instance designated as the primary writer and all others as backup readers. To ensure high availability in the event of availability zone failures, Aurora allows database instances to be deployed across multiple zones in a single AWS region. In the event of a failure on the availability zone that is hosting the Primary database instance, Aurora automatically heals itself and promotes a reader instance from a non-failed availability zone to be the new writer instance.

See the AWS Aurora documentation for more details on the semantics provided by Aurora databases.

This documentation follows AWS best practices and creates a private Aurora database that is not exposed to the Internet. To access the database from a ROSA cluster, establish a peering connection between the database and the ROSA cluster.

The following procedure contains two sections:

The simplest way to verify that a connection is possible between a ROSA cluster and an Aurora DB cluster is to deploy psql on the OpenShift cluster and attempt to connect to the writer endpoint.

The following command creates a pod in the default namespace and establishes a psql connection with the Aurora cluster if possible. Upon exiting the pod shell, the pod is deleted.

USER=keycloak 

PASSWORD=secret99 

DATABASE=keycloak 

HOST=$(aws rds describe-db-clusters \
  --db-cluster-identifier keycloak-aurora \

  --query 'DBClusters[*].Endpoint' \
  --region eu-west-1 \
  --output text
)
oc run -i --tty --rm debug --image=postgres:15 --restart=Never -- psql postgresql://${USER}:${PASSWORD}@${HOST}/${DATABASE}

After successful deployment of the Aurora database continue with Deploying Red Hat build of Keycloak across multiple availability-zones with the Operator

CloudNativePG is an opensource operator that manages PostgreSQL clusters on OpenShift. It is designed to operate one primary writer instance and optional reader instances.

As this setup uses standard a PostgreSQL image that does not support Transparent Data Encryption (TDE), all storage devices for the database and write-ahead logs (WAL) should be placed on encrypted storage. See Configuring the database for more information.

After successful deployment of the CloudNativePG database continue with Deploying Red Hat build of Keycloak across multiple availability-zones with the Operator

As this setup does not support encrypted backups, the S3 bucket should be encrypted. Newly created AWS S3 buckets are encrypted by default. See Configuring the database for more information.

Both the CloudNativePG operator and the Barman Cloud plugin are required in order to enable backup and restore operations using an S3-compatible object store.

oc apply --filename=https://github.com/cert-manager/cert-manager/releases/download/v1.20.0/cert-manager.yaml

oc rollout status deployment --namespace=cert-manager cert-manager-webhook
oc rollout status deployment --namespace=cert-manager cert-manager

deployment "cert-manager-webhook" successfully rolled out
deployment "cert-manager" successfully rolled out

oc apply --filename=https://github.com/cloudnative-pg/plugin-barman-cloud/releases/download/v0.11.0/manifest.yaml

oc rollout status deployment --namespace=cnpg-system barman-cloud

deployment "barman-cloud" successfully rolled out

This blueprint uses AWS S3 as the object store. Refer to the Barman Cloud plugin object stores documentation for details on how to configure other cloud providers.

This blueprint uses AWS access keys for authentication. Other authentication methods, such as IAM Roles for Service Accounts (IRSA), are also supported. Refer to the Barman Cloud plugin AWS S3 documentation for more details.

Create a Kubernetes secret with the AWS credentials required to access the S3 bucket. The secret must be created in the same namespace as the CloudNativePG cluster.

oc create secret generic aws-creds \
  --namespace cnpg-keycloak \
  --from-literal=ACCESS_KEY_ID=<access_key> \
  --from-literal=ACCESS_SECRET_KEY=<secret_key> \
  --from-literal=REGION=<region>

The ObjectStore resource defines the S3 bucket destination and encryption settings for backups and WAL archiving. The example below uses AWS S3, but the Barman Cloud plugin also supports Azure Blob Storage and Google Cloud Storage. For details on configuring other cloud providers, refer to the Barman Cloud plugin object stores documentation.

Create the CloudNativePG Cluster resource to enable backup and WAL archiving to the object store using the Barman Cloud plugin.

The ScheduledBackup resource enables automatic periodic backups of the CloudNativePG cluster. Scheduled backups are the recommended way to implement a reliable backup strategy.

After successful configuration of the backup, continue with Deploying Red Hat build of Keycloak across multiple availability-zones with the Operator.

For details of how to restore a CloudNativePG cluster from a backup, see Recovering a CloudNativePG cluster from an S3 backup.

For more information about backup and recovery operations, refer to the CloudNativePG documentation.

In a CloudNativePG cluster deployed in high availability mode, standby instances are critical for both data durability and failover readiness. Monitoring standby health helps detect replication issues early and ensures a safe promotion candidate is available when needed.

To see the status on the command line:

To monitor the status via metrics and dashboards:

Verify standby health in the Streaming Replication status table.

The following fields help determine whether standbys are healthy and replication is working:

A healthy standby setup typically shows:

The following are indicators that a standby requires attention:

If one or more standby instances show these symptoms, investigate using the following commands:

For possible troubleshooting scenarios refer to the CloudNativePG documentation.

The recovery process creates a new CloudNativePG cluster by bootstrapping it from an existing backup in the object store. Instead of using bootstrap.initdb to create a fresh database, the bootstrap.recovery section instructs CloudNativePG to restore from a backup source.

Point-in-Time Recovery allows restoring the database to a specific moment in time rather than to the latest available state. This is useful in scenarios such as recovering from a corrupted database state, reverting changes caused by a failed Red Hat build of Keycloak upgrade, or undoing an unintended data modification.

PITR relies on continuous WAL archiving, which is already configured when using the Barman Cloud plugin as described in Deploying CloudNativePG with scheduled backups to S3. CloudNativePG automatically selects the base backup closest to the specified target time and replays the WAL logs up to that point.

date -u +"%Y-%m-%dT%H:%M:%SZ"

To perform a Point-in-Time Recovery, add a recoveryTarget section to the bootstrap.recovery configuration. The following example recovers the cluster to a specific timestamp:

For more details on Point-in-Time Recovery options, refer to the CloudNativePG PITR documentation.

After a recovery, any database modifications made after the last archived WAL segment or the PITR target time are lost. Some Red Hat build of Keycloak tables require attention because stale data may affect cluster behavior or security.

TRUNCATE jgroups_ping;

After successful recovery of the CloudNativePG cluster, continue with Deploying Red Hat build of Keycloak across multiple availability-zones with the Operator.

For more information about recovery operations, refer to the CloudNativePG recovery documentation.

Depending on the database you deployed use the relevant procedure for deploying Red Hat build of Keycloak.

Confirm that the Red Hat build of Keycloak deployment is ready.

oc wait --for=condition=Ready keycloaks.k8s.keycloak.org/keycloak
oc wait --for=condition=RollingUpdate=False keycloaks.k8s.keycloak.org/keycloak

To enable load shedding, limit the number of queued requests.

spec:
  additionalOptions:
    - name: http-max-queued-requests
      value: "1000"

All exceeding requests are served with an HTTP 503.

You might consider limiting the value for http-pool-max-threads further because multiple concurrent threads will lead to throttling by OpenShift once the requested CPU limit is reached.

See the Concepts for configuring thread pools chapter about load shedding for details.

When running on OpenShift and the default passthrough Ingress setup as provided by the Red Hat build of Keycloak Operator, the load balancing done by HAProxy is done by using sticky sessions based on the IP address of the source. When running load tests, or when having a reverse proxy in front of HAProxy, you might want to disable this setup to avoid receiving all requests on a single Red Hat build of Keycloak Pod.

Add the following supplementary configuration under the spec in the Red Hat build of Keycloak Custom Resource to disable sticky sessions.

spec:
  ingress:
    enabled: true
    annotations:
      # When running load tests, disable sticky sessions on the OpenShift HAProxy router
      # to avoid receiving all requests on a single Red Hat build of Keycloak Pod.
      haproxy.router.openshift.io/balance: roundrobin
      haproxy.router.openshift.io/disable_cookies: 'true'

The CloudNativePG operator is designed to perform an automated switchover in situations such as changing a cluster configuration that requires a rolling update (for example imageName, resources, or certain postgresql.paramaters), an operator upgrade, or a OpenShift node maintenance (Pod eviction).

This procedure can be used to perform a switchover manually, outside of these situations, for example to test how Red Hat build of Keycloak behaves during switchovers.

In order to minimize service disruptions it is recommended to perform this procedure during a period of minimal load. As long as the primary database node is shut down gracefully, no committed data should be lost.

For possible troubleshooting scenarios refer to the CloudNativePG documentation.