Cluster notifications
Cluster notifications are messages about the status, health, or performance of your cluster.
Cluster notifications are the primary way that Red Hat Site Reliability Engineering (SRE) communicates with you about the health of your managed cluster. SRE may also use cluster notifications to prompt you to perform an action in order to resolve or prevent an issue with your cluster.
Cluster owners and administrators must regularly review and action cluster notifications to ensure clusters remain healthy and supported.
You can view cluster notifications in the Red Hat Hybrid Cloud Console, in the Cluster history tab for your cluster. By default, only the cluster owner receives cluster notifications as emails. If other users need to receive cluster notification emails, add each user as a notification contact for your cluster.
What to expect from cluster notifications
As a cluster administrator, you need to be aware of when and why cluster notifications are sent, as well as their types and severity levels, in order to effectively understand the health and administration needs of your cluster.
Cluster notification policy
Cluster notifications are designed to keep you informed about the health of your cluster and high impact events that affect it.
Most cluster notifications are generated and sent automatically to ensure that you are immediately informed of problems or important changes to the state of your cluster.
In certain situations, Red Hat Site Reliability Engineering (SRE) creates and sends cluster notifications to provide additional context and guidance for a complex issue.
Cluster notifications are not sent for low-impact events, low-risk security updates, routine operations and maintenance, or minor, transient issues that are quickly resolved by SRE.
Red Hat services automatically send notifications when:
-
Remote health monitoring or environment verification checks detect an issue in your cluster, for example, when a worker node has low disk space.
-
Significant cluster life cycle events occur, for example, when scheduled maintenance or upgrades begin, or cluster operations are impacted by an event, but do not require customer intervention.
-
Significant cluster management changes occur, for example, when cluster ownership or administrative control is transferred from one user to another.
-
Your cluster subscription is changed or updated, for example, when Red Hat makes updates to subscription terms or features available to your cluster.
SRE creates and sends notifications when:
-
An incident results in a degradation or outage that impacts your cluster’s availability or performance, for example, your cloud provider has a regional outage. SRE sends subsequent notifications to inform you of incident resolution progress, and when the incident is resolved.
-
A security vulnerability, security breach, or unusual activity is detected on your cluster.
-
Red Hat detects that changes you have made are creating or may result in cluster instability.
-
Red Hat detects that your workloads are causing performance degradation or instability in your cluster.
Cluster notification severity levels
Each cluster notification has an associated severity level to help you identify notifications with the greatest impact to your business. You can filter cluster notifications according to these severity levels in the Red Hat Hybrid Cloud Console, in the Cluster history tab for your cluster.
Red Hat uses the following severity levels for cluster notifications, from most to least severe:
- Critical
-
Immediate action is required. One or more key functions of a service or cluster is not working, or will stop working soon. A critical alert is important enough to page on-call staff and interrupt regular workflows.
- Major
-
Immediate action is strongly recommended. One or more key functions of the cluster will soon stop working. A major issue may lead to a critical issue if it is not addressed in a timely manner.
- Warning
-
Action is required as soon as possible. One or more key functions of the cluster are not working optimally and may degrade further, but do not pose an immediate danger to the functioning of the cluster.
- Info
-
No action necessary. This severity does not describe problems that need to be addressed, only important information about meaningful or important life cycle, service, or cluster events.
- Debug
-
No action necessary. Debug notifications provide low-level information about less important lifecycle, service, or cluster events to aid in debugging unexpected behavior.
Cluster notification types
Each cluster notification has an associated notification type to help you identify notifications that are relevant to your role and responsibilities. You can filter cluster notifications according to these types in the Red Hat Hybrid Cloud Console, in the Cluster history tab for your cluster.
Red Hat uses the following notification types to indicate notification relevance.
- Capacity management
-
Notifications for events related to updating, creating, or deleting node pools, machine pools, compute replicas or quotas (load balancer, storage, etc.).
- Cluster access
-
Notifications for events related to adding or deleting groups, roles or identity providers, for example, when SRE cannot access your cluster because STS credentials have expired, when there is a configuration problem with your AWS roles, or when you add or remove identity providers.
- Cluster add-ons
-
Notifications for events related to add-on management or upgrade maintenance for add-ons, for example, when an add-on is installed, upgraded, or removed, or cannot be installed due to unmet requirements.
- Cluster configuration
-
Notifications for cluster tuning events, workload monitoring, and inflight checks.
- Cluster lifecycle
-
Notifications for cluster or cluster resource creation, deletion, and registration, or change in cluster or resource status (for example, ready or hibernating).
- Cluster networking
-
Notifications related to cluster networking, including HTTP/S proxy, router, and ingress state.
- Cluster ownership
-
Notifications related to cluster ownership transfer from one user to another.
- Cluster scaling
-
Notifications related to updating, creating, or deleting node pools, machine pools, compute replicas or quota.
- Cluster security
-
Events related to cluster security, for example, an increased number of failed access attempts, updates to trust bundles, or software updates with security impact.
- Cluster subscription
-
Cluster expiration, trial cluster notifications, or switching from free to paid.
- Cluster updates
-
Anything relating to upgrades, such as upgrade maintenance or enablement.
- Customer support
-
Updates on support case status.
- General notification
-
The default notification type. This is only used for notifications that do not have a more specific category.
Viewing cluster notifications using the Red Hat Hybrid Cloud Console
Cluster notifications provide important information about the health of your cluster. You can view notifications that have been sent to your cluster in the Cluster history tab on the Red Hat Hybrid Cloud Console.
-
You are logged in to the Hybrid Cloud Console.
-
Navigate to the Clusters page of the Hybrid Cloud Console.
-
Click the name of your cluster to go to the cluster details page.
-
Click the Cluster history tab.
Cluster notifications appear under the Cluster history heading.
-
Optional: Filter for relevant cluster notifications
Use the filter controls to hide cluster notifications that are not relevant to you, so that you can focus on your area of expertise or on resolving a critical issue. You can filter notifications based on text in the notification description, severity level, notification type, when the notification was received, and which system or person triggered the notification.
Cluster notification emails
By default, when a cluster notification is sent to the cluster, it is also sent as an email to the cluster owner. You can configure additional recipients for notification emails to ensure that all appropriate users remain informed about the state of the cluster.
Adding notification contacts to your cluster
Notification contacts receive emails when cluster notifications are sent to the cluster. By default, only the cluster owner receives cluster notification emails. You can configure other cluster users as additional notification contacts in your cluster support settings.
-
Your cluster is deployed and registered to the Red Hat Hybrid Cloud Console.
-
You are logged in to the Hybrid Cloud Console as the cluster owner or as a user with the cluster editor role.
-
The intended notification recipient has a Red Hat Customer Portal account associated with the same organization as the cluster owner.
-
Navigate to the Clusters page of the Hybrid Cloud Console.
-
Click the name of your cluster to go to the cluster details page.
-
Click the Support tab.
-
On the Support tab, find the Notification contacts section.
-
Click Add notification contact.
-
In the Red Hat username or email field, enter the email address or the user name of the new recipient.
-
Click Add contact.
-
The "Notification contact added successfully" message displays.
- The Add notification contact button is disabled
-
This button is disabled for users who do not have permission to add a notification contact. Log in to an account with the cluster owner, cluster editor, or cluster administrator role and try again.
- Error:
Could not find any account identified by <username>or<email-address> -
This error occurs when the intended notification recipient is not part of the same Red Hat account organization as the cluster owner. Contact your organization administrator to add the intended recipient to the relevant organization and try again.
Removing notification contacts from your cluster
Notification contacts receive emails when cluster notifications are sent to the cluster.
You can remove notification contacts in your cluster support settings to prevent them from receiving notification emails.
-
Your cluster is deployed and registered to the Red Hat Hybrid Cloud Console.
-
You are logged in to the Hybrid Cloud Console as the cluster owner or as a user with the cluster editor role.
-
Navigate to the Clusters page of the Hybrid Cloud Console.
-
Click the name of your cluster to go to the cluster details page.
-
Click the Support tab.
-
On the Support tab, find the Notification contacts section.
-
Click the options menu (⚙) beside the recipient you want to remove.
-
Click Delete.
-
The "Notification contact deleted successfully" message displays.
Troubleshooting
-
Ensure that emails sent from
@redhat.comaddresses are not filtered out of your email inbox. -
Ensure that your correct email address is listed as a notification contact for the cluster.
-
Ask the cluster owner or administrator to add you as a notification contact: Cluster notification emails.
-
Ensure that your cluster can access resources at
api.openshift.com. -
Ensure that your firewall is configured according to the documented prerequisites: AWS firewall prerequisites