Red Hat OpenStack Services on OpenShift (RHOSO) deploys MariaDB Galera clusters to provide highly available database services for the control plane. By default, the following two Galera clusters are deployed:

• openstack, the cluster hosting the databases for all RHOSO services.
• openstack-cell1, the cluster hosting the databases specific to the Compute service (nova) for cell1.

Galera Custom Resources (CRs) configure both clusters. You can monitor the startup of the Galera pods for the required Galera CR.

You can use the oc get galera command to retrieve more information about these Galera CRs, as shown in the following example:

$ oc get galera
NAME            READY   MESSAGE
openstack       True    Setup complete
openstack-cell1 True    Setup complete

The Message and Ready columns show the startup state and the service availability of the Galera CR, specified by the Name column. When the Ready condition is True, the pods are started and ready to accept traffic.

The Galera service Operator mariadb-operator performs the following Galera cluster operations:

• Creates the pods that host the mysqld servers.
• Runs the logic for bootstrapping a Galera cluster. For example, the mariadb-operator starts the cluster using the most recent copy of the Galera database.
• Monitors the running Galera pods.
• Restarts the pods when the pods fail their health check.

The mariadb-operator creates an OpenShift service object called openstack, which provides the IP address that the OpenStack services use to access the database. You can use the following command to reveal this IP address:

$ oc get service -l mariadb/name=openstack
NAME        TYPE        CLUSTER-IP     EXTERNAL-IP    PORT(S)    AGE
openstack   ClusterIP   10.217.5.210   <none>         3306/TCP   7h

The mariadb-operator also creates a 'headless' DNS service for the Galera pods, called openstack-galera that is only used for the internal Galera cluster communication. OpenStack pods never access this service.

RHOSO marks a Galera pod as available based on its Readiness health check. When the Galera cluster is bootstrapped by the mariadb-operator, the first available Galera pod receives all the incoming database traffic. The other available Galera pods remain in a state of hot standby.

When the active pod is disconnected from the Galera cluster, for any reason, such as pod stop or pod restart, failover occurs. At this time, the first available Galera pod in hot standby becomes the active pod, and the openstack service is updated accordingly.

You can use the following commands to troubleshoot the Galera services:

When you analyze the openstack service, IP field specifies the IP address assigned to the service and Endpoints field specifies the IP address of the currently active Galera pod that is receiving all the traffic.

$ oc describe service openstack
Name: openstack
Namespace: openstack
Labels: app=mariadb cr=mariadb-openstack mariadb/name=openstack mariadb/namespace=openstack mariadb/uid=796b4c64-fb39-4144-817f-34d2b309eb30 owner=mariadb-operator
Annotations: <none>
Selector: app=galera,cr=galera-openstack,statefulset.kubernetes.io/pod-name=openstack-galera-2
Type: ClusterIP
IP Family Policy: SingleStack
IP Families: IPv4
IP: 172.30.133.220
IPs: 172.30.133.220
Port: database 3306/TCP
TargetPort: 3306/TCP
Endpoints: 192.168.56.28:3306
Session Affinity: None
Events: <none>

When you analyze the headless openstack-galera service, the IP field is not specified but the Endpoints field specify the IP addresses of all the Galera pods, of which there are typically 3.

$ oc describe service openstack-galera
Name: openstack-galera
Namespace: openstack
Labels: <none>
Annotations: <none>
Selector: app=galera,cr=galera-openstack
Type: ClusterIP
IP Family Policy: SingleStack
IP Families: IPv4
IP: None
IPs: None
Port: mysql 3306/TCP
TargetPort: 3306/TCP
Endpoints: 192.168.48.52:3306,192.168.52.43:3306,192.168.56.23:3306
Session Affinity: None
Events: <none>