$ oc adm must-gather --image=registry.redhat.io/openshift-service-mesh/istio-must-gather-rhel7
Red Hat OpenShift Service Mesh is a platform that provides behavioral insight and operational control over the service mesh, providing a uniform way to connect, secure, and monitor microservice applications.
The term service mesh describes the network of microservices that make up applications in a distributed microservice architecture and the interactions between those microservices. As a service mesh grows in size and complexity, it can become harder to understand and manage.
Based on the open source Istio project, Red Hat OpenShift Service Mesh adds a transparent layer on existing distributed applications without requiring any changes to the service code. You add Red Hat OpenShift Service Mesh support to services by deploying a special sidecar proxy throughout your environment that intercepts all network communication between microservices. You configure and manage the service mesh using the control plane features.
Red Hat OpenShift Service Mesh provides an easy way to create a network of deployed services that provides discovery, load balancing, service-to-service authentication, failure recovery, metrics, and monitoring. A service mesh also provides more complex operational functionality, including A/B testing, canary releases, rate limiting, access control, and end-to-end authentication.
If you experience difficulty with a procedure described in this documentation, visit the Red Hat Customer Portal. Through the Customer Portal, you can:
Search or browse through the Red Hat Knowledgebase of technical support articles about Red Hat products.
Submit a support case to Red Hat Support.
When submitting a support case, it is recommended to provide the following information about your cluster to Red Hat Support to aid in troubleshooting:
|
Access other product documentation.
If you have a suggestion for improving this documentation or have found an error, please submit a Bugzilla report against the OpenShift Container Platform product for the Documentation component. Please provide specific details, such as the section name and OpenShift Container Platform version.
When opening a support case, it is helpful to provide debugging information about your cluster to Red Hat Support.
The must-gather
tool enables you to collect diagnostic information about your
OpenShift Container Platform cluster, including virtual machines and other data related to
Red Hat OpenShift Service Mesh.
For prompt support, supply diagnostic information for both OpenShift Container Platform and Red Hat OpenShift Service Mesh.
The oc adm must-gather
CLI command collects the information from your cluster
that is most likely needed for debugging issues, such as:
Resource definitions
Audit logs
Service logs
You can specify one or more images when you run the command by including the
--image
argument. When you specify an image, the tool collects data related to
that feature or product.
When you run oc adm must-gather
, a new Pod is created on the cluster. The data
is collected on that Pod and saved in a new directory that starts with
must-gather.local
. This directory is created in the current working
directory.
Access to the cluster as a user with the cluster-admin
role.
The OpenShift Container Platform CLI (oc
) installed.
You can use the oc adm must-gather
CLI command to collect information about your cluster, including features and objects associated with Red Hat OpenShift Service Mesh.
To collect Red Hat OpenShift Service Mesh data with must-gather
, you must specify the Red Hat OpenShift Service Mesh image:
$ oc adm must-gather --image=registry.redhat.io/openshift-service-mesh/istio-must-gather-rhel7
The following are the only supported configurations for the Red Hat OpenShift Service Mesh:
Red Hat OpenShift Container Platform version 4.x.
OpenShift Online and OpenShift Dedicated are not supported for Red Hat OpenShift Service Mesh. |
The deployment must be contained to a single OpenShift Container Platform cluster that is not federated.
This release of Red Hat OpenShift Service Mesh is only available on OpenShift Container Platform x86_64.
This release only supports configurations where all Service Mesh components are contained in the OpenShift cluster in which it operates. It does not support management of microservices that reside outside of the cluster, or in a multi-cluster scenario.
This release only supports configurations that do not integrate external services such as virtual machines.
Red Hat OpenShift Service Mesh provides a number of key capabilities uniformly across a network of services:
Traffic Management - Control the flow of traffic and API calls between services, make calls more reliable, and make the network more robust in the face of adverse conditions.
Service Identity and Security - Provide services in the mesh with a verifiable identity and provide the ability to protect service traffic as it flows over networks of varying degrees of trustworthiness.
Policy Enforcement - Apply organizational policy to the interaction between services, ensure access policies are enforced and resources are fairly distributed among consumers. Policy changes are made by configuring the mesh, not by changing application code.
Telemetry - Gain understanding of the dependencies between services and the nature and flow of traffic between them, providing the ability to quickly identify issues.
Component | Version |
---|---|
Istio |
1.4.8 |
Jaeger |
1.17.4 |
Kiali |
1.12.7 |
3scale Istio Adapter |
1.0.0 |
This release of Red Hat OpenShift Service Mesh addresses Common Vulnerabilities and Exposures (CVEs) and bug fixes.
This release of Red Hat OpenShift Service Mesh addresses Common Vulnerabilities and Exposures (CVEs) and bug fixes.
This release of Red Hat OpenShift Service Mesh addresses Common Vulnerabilities and Exposures (CVEs) and bug fixes.
This release of Red Hat OpenShift Service Mesh addresses Common Vulnerabilities and Exposures (CVEs) and bug fixes.
This release of Red Hat OpenShift Service Mesh addresses Common Vulnerabilities and Exposures (CVEs) and bug fixes.
This release of Red Hat OpenShift Service Mesh addresses Common Vulnerabilities and Exposures (CVEs) and bug fixes.
This release also added support for configuring cipher suites.
This release of Red Hat OpenShift Service Mesh addresses Common Vulnerabilities and Exposures (CVEs) and bug fixes.
There are manual steps that must be completed to address CVE-2020-8663. |
The fix for CVE-2020-8663: envoy: Resource exhaustion when accepting too many connections
added a configurable limit on downstream connections. The configuration option for this limit must be configured to mitigate this vulnerability.
These manual steps are required to mitigate this CVE whether you are using the 1.1 version or the 1.0 version of Red Hat OpenShift Service Mesh. |
This new configuration option is called overload.global_downstream_max_connections
, and it is configurable as a proxy runtime
setting. Perform the following steps to configure limits at the Ingress Gateway.
Create a file named bootstrap-override.json
with the following text to force the proxy to override the bootstrap template and load runtime configuration from disk:
{ "runtime": { "symlink_root": "/var/lib/istio/envoy/runtime" } }
Create a secret from the bootstrap-override.json
file, replacing <SMCPnamespace> with the namespace where you created the service mesh control plane (SMCP):
$ oc create secret generic -n <SMCPnamespace> gateway-bootstrap --from-file=bootstrap-override.json
Update the SMCP configuration to activate the override.
apiVersion: maistra.io/v1
kind: ServiceMeshControlPlane
spec:
istio:
gateways:
istio-ingressgateway:
env:
ISTIO_BOOTSTRAP_OVERRIDE: /var/lib/istio/envoy/custom-bootstrap/bootstrap-override.json
secretVolumes:
- mountPath: /var/lib/istio/envoy/custom-bootstrap
name: custom-bootstrap
secretName: gateway-bootstrap
To set the new configuration option, create a secret that has the desired value for the overload.global_downstream_max_connections
setting. The following example uses a value of 10000
:
$ oc create secret generic -n <SMCPnamespace> gateway-settings --from-literal=overload.global_downstream_max_connections=10000
Update the SMCP again to mount the secret in the location where Envoy is looking for runtime configuration:
apiVersion: maistra.io/v1
kind: ServiceMeshControlPlane
spec:
template: default
#Change the version to "v1.0" if you are on the 1.0 stream.
version: v1.1
istio:
gateways:
istio-ingressgateway:
env:
ISTIO_BOOTSTRAP_OVERRIDE: /var/lib/istio/envoy/custom-bootstrap/bootstrap-override.json
secretVolumes:
- mountPath: /var/lib/istio/envoy/custom-bootstrap
name: custom-bootstrap
secretName: gateway-bootstrap
# below is the new secret mount
- mountPath: /var/lib/istio/envoy/runtime
name: gateway-settings
secretName: gateway-settings
When updating from Elasticsearch 5 to Elasticsearch 6, you must delete your Jaeger instance, then recreate the Jaeger instance because of an issue with certificates. Re-creating the Jaeger instance triggers creating a new set of certificates. If you are using persistent storage the same volumes can be mounted for the new Jaeger instance as long as the Jaeger name and namespace for the new Jaeger instance are the same as the deleted Jaeger instance.
Determine the name of your Jaeger custom resource file:
$ oc get jaeger -n istio-system
You should see something like the following:
NAME AGE
jaeger 3d21h
Copy the generated custom resource file into a temporary directory:
$ oc get jaeger jaeger -oyaml -n istio-system > /tmp/jaeger-cr.yaml
Delete the Jaeger instance:
$ oc delete jaeger jaeger -n istio-system
Recreate the Jaeger instance from your copy of the custom resource file:
$ oc create -f /tmp/jaeger-cr.yaml -n istio-system
Delete the copy of the generated custom resource file:
$ rm /tmp/jaeger-cr.yaml
Before you begin, create a copy of your Jaeger custom resource file.
Delete the Jaeger instance by deleting the custom resource file:
$ oc delete -f <jaeger-cr-file>
For example:
$ oc delete -f jaeger-prod-elasticsearch.yaml
Recreate your Jaeger instance from the backup copy of your custom resource file:
$ oc create -f <jaeger-cr-file>
Validate that your Pods have restarted:
$ oc get pods -n jaeger-system -w
This release of Red Hat OpenShift Service Mesh addresses Common Vulnerabilities and Exposures (CVEs) and bug fixes.
This release of Red Hat OpenShift Service Mesh addresses a security vulnerability.
This release of Red Hat OpenShift Service Mesh adds support for a disconnected installation.
This release of Red Hat OpenShift Service Mesh adds support for Istio 1.4.6 and Jaeger 1.17.1.
If you are updating from Red Hat OpenShift Service Mesh 1.0 to 1.1, you must update the ServiceMeshControlPlane
resource to update the control plane components to the new version.
In the web console, click the Red Hat OpenShift Service Mesh Operator.
Click the Project menu and choose the project where your ServiceMeshControlPlane
is deployed from the list, for example istio-system
.
Click the name of your control plane, for example basic-install
.
Click YAML and add a version field to the spec:
of your ServiceMeshControlPlane
resource. For example, to update to Red Hat OpenShift Service Mesh 1.1.0, add version: v1.1
.
spec: version: v1.1 ...
The version field specifies the version of ServiceMesh to install and defaults to the latest available version.
This release of Red Hat OpenShift Service Mesh addresses Common Vulnerabilities and Exposures (CVEs) and bug fixes.
There are manual steps that must be completed to address CVE-2020-8663. See instructions above. |
This release of Red Hat OpenShift Service Mesh addresses Common Vulnerabilities and Exposures (CVEs).
This release of Red Hat OpenShift Service Mesh addresses Common Vulnerabilities and Exposures (CVEs).
This release of Red Hat OpenShift Service Mesh addresses compatibility issues with OpenShift Container Platform 4.4. You must upgrade Red Hat OpenShift Service Mesh to 1.0.8 before you upgrade from OpenShift Container Platform 4.3 to OpenShift Container Platform 4.4.
This release of Red Hat OpenShift Service Mesh addresses Common Vulnerabilities and Exposures (CVEs).
This release of Red Hat OpenShift Service Mesh adds support for Kiali 1.0.9, and addresses Common Vulnerabilities and Exposures (CVEs).
This release of Red Hat OpenShift Service Mesh adds support for Kiali 1.0.8, and addresses Common Vulnerabilities and Exposures (CVEs).
This release of Red Hat OpenShift Service Mesh adds support for Istio 1.1.17, Jaeger 1.13.1, Kiali 1.0.7, and the 3scale Istio Adapter 1.0 and OpenShift Container Platform 4.2.
Unresolved directive in modules/ossm-rn-new-features-1x.adoc - include::modules/ossm-document-attributes.adoc[] :context: installing-ossm toc::[]
Updating Red Hat OpenShift Service Mesh requires extra steps before you update OpenShift Container Platform to version 4.2. You must upgrade Red Hat OpenShift Service Mesh to 1.0.2 before upgrading OpenShift Container Platform 4.1.x to 4.2.
Red Hat OpenShift Service Mesh version 1.0.1
OpenShift Container Platform version 4.1
Configure existing SMCP resource requests by running the following oc patch
command. Replace
the <smcp_namespace> and <smcp_name> with your specific names:
$ oc patch -n <smcp_namespace> smcp <smcp_name> \(1) --type=merge -p \ '{"spec": {"istio": {"global": {"defaultResources": {"requests": {"cpu": "10m","memory": "128Mi"},"limits":{}},"proxy": {"resources": {"requests": {"cpu": "10m","memory": "128Mi"},"limits":{}}},"defaultPodDisruptionBudget": {"enabled": false}},"security": {"resources": {"requests": {"cpu": "10m","memory": "128Mi"}}},"galley": {"resources": {"requests": {"cpu": "10m","memory": "128Mi"}}},"pilot": {"resources": {"requests": {"cpu": "10m","memory": "128Mi"}}},"mixer": {"telemetry": {"resources": {"requests": {"cpu": "10m","memory": "128Mi"}}}},"gateways": {"istio-egressgateway": {"resources": {"requests": {"cpu": "10m","memory": "128Mi"}}},"istio-ingressgateway": {"resources": {"requests": {"cpu": "10m","memory": "128Mi"}}}},"prometheus": {"resources": {"requests": {"cpu": "10m","memory": "128Mi"}}}}}}'
1 | For example, basic-install . |
After running this command, wait until all SMCP Pods are replaced in the SMCP namespace.
After the Pods are running in the SMCP namespace, redeploy your Data Plane applications, such as bookinfo
.
Log in as a cluster-admin
user such as kubeadmin
, and then run the following command to delete the CNI istio-node
DaemonSet.
Replace openshift-operators
if your Red Hat OpenShift Service Mesh Operator was not installed in the default openshift-operators
namespace:
$ oc delete -n openshift-operators daemonset istio-node
Upgrade Red Hat OpenShift Service Mesh Operator and SMCP to 1.0.2. After all Pods are running in the SMCP namespace, patch Data Plane applications by running the following command for each deployment:
$ oc patch -n <data_plane_namespace> deployment/<deployment_name> -p \ '{"spec":{"template":{"metadata":{"annotations":{"kubectl.kubernetes.io/restartedAt": "'`date -Iseconds`'"}}}}}'
Upgrade OpenShift Container Platform using the OpenShift Container Platform web console.
This release of Red Hat OpenShift Service Mesh adds support for Istio 1.1.11, Jaeger 1.13.1, Kiali 1.0.6, and the 3scale Istio Adapter 1.0 and OpenShift Container Platform 4.1.
This release of Red Hat OpenShift Service Mesh adds support for Istio 1.1.11, Jaeger 1.13.1, Kiali 1.0.5, and the 3scale Istio Adapter 1.0 and OpenShift Container Platform 4.1.
Other notable changes in this release include the following:
The Kubernetes Container Network Interface (CNI) plug-in is always on.
The control plane is configured for multitenancy by default. Single tenant, cluster-wide control plane configurations are deprecated.
The Elasticsearch, Jaeger, Kiali, and Service Mesh Operators are installed from OperatorHub.
You can create and specify control plane templates.
Automatic route creation was removed from this release.
Some features available in previous releases have been deprecated or removed.
Deprecated functionality is still included in OpenShift Container Platform and continues to be supported; however, it will be removed in a future release of this product and is not recommended for new deployments.
The following custom resources are deprecated in this release and will be removed in a future release.
Policy
- The Policy
resource is deprecated and will be replaced by the PeerAuthentication
resource in a future release.
MeshPolicy
- The MeshPolicy
resource is deprecated and will be replaced by the PeerAuthentication
resource in a future release.
v1alpha1
RBAC API -The v1alpha1 RBAC policy is deprecated by the v1beta1 AuthorizationPolicy
. RBAC (Role Based Access Control) defines ServiceRole
and ServiceRoleBinding
objects.
ServiceRole
ServiceRoleBinding
RbacConfig
- RbacConfig
implements the Custom Resource Definition for controlling Istio RBAC behavior.
ClusterRbacConfig
(versions prior to Red Hat OpenShift Service Mesh 1.0)
ServiceMeshRbacConfig
(Red Hat OpenShift Service Mesh version 1.0 and later)
In Kiali, the login
and LDAP
strategies are deprecated. A future version will introduce authentication using OpenID providers.
The following components are also deprecated in this release and will be replaced by the Istiod component in a future release.
Mixer - access control and usage policies
Pilot - service discovery and proxy configuration
Citadel - certificate generation
Galley - configuration validation and distribution
These limitations exist in Red Hat OpenShift Service Mesh:
Red Hat OpenShift Service Mesh does not support IPv6, as it is not supported by the upstream Istio project, nor fully supported by OpenShift.
Graph layout - The layout for the Kiali graph can render differently, depending on your application architecture and the data to display (number of graph nodes and their interactions). Because it is difficult if not impossible to create a single layout that renders nicely for every situation, Kiali offers a choice of several different layouts. To choose a different layout, you can choose a different Layout Schema from the Graph Settings menu.
The first time you access related services such as Jaeger and Grafana, from the Kiali console, you must accept the certificate and re-authenticate using your OpenShift Container Platform login credentials. This happens due to an issue with how the framework displays embedded pages in the console.
These are the known issues in Red Hat OpenShift Service Mesh:
Maistra-1502 As a result of CVEs fixes in version 1.0.10, the Istio dashboards are not available from the Home Dashboard menu in Grafana. The Istio dashboards still exist. To access them, click the Dashboard menu in the navigation panel and select the Manage tab.
Bug 1821432 Toggle controls in OpenShift Container Platform Control Resource details page do not update the CR correctly. UI Toggle controls in the Service Mesh Control Plane (SMCP) Overview page in the OpenShift Container Platform web console sometimes update the wrong field in the resource. To update a SMCP, edit the YAML content directly or update the resource from the command line instead of clicking the toggle controls.
Jaeger/Kiali Operator upgrade blocked with operator pending When upgrading the Jaeger or Kiali Operators with Service Mesh 1.0.x installed, the operator status shows as Pending. There is a solution in progress and a workaround. See the linked Knowledge Base article for more information.
Istio-14743 Due to limitations in the version of Istio that this release of Red Hat OpenShift Service Mesh is based on, there are several applications that are currently incompatible with Service Mesh. See the linked community issue for details.
MAISTRA-858 The following Envoy log messages describing deprecated options and configurations associated with Istio 1.1.x are expected:
[2019-06-03 07:03:28.943][19][warning][misc] [external/envoy/source/common/protobuf/utility.cc:129] Using deprecated option 'envoy.api.v2.listener.Filter.config'. This configuration will be removed from Envoy soon.
[2019-08-12 22:12:59.001][13][warning][misc] [external/envoy/source/common/protobuf/utility.cc:174] Using deprecated option 'envoy.api.v2.Listener.use_original_dst' from file lds.proto. This configuration will be removed from Envoy soon.
MAISTRA-806 Evicted Istio Operator Pod causes mesh and CNI not to deploy.
If the istio-operator
pod is evicted while deploying the control pane, delete the evicted istio-operator
pod.
MAISTRA-681 When the control plane has many namespaces, it can lead to performance issues.
MAISTRA-465 The Maistra Operator fails to create a service for operator metrics.
MAISTRA-453 If you create a new project and deploy pods immediately, sidecar injection does not occur. The operator fails to add the maistra.io/member-of
before the pods are created, therefore the pods must be deleted and recreated for sidecar injection to occur.
MAISTRA-193 Unexpected console info messages are visible when health checking is enabled for citadel.
MAISTRA-158 Applying multiple gateways referencing the same hostname will cause all gateways to stop functioning.
These are the known issues in Kiali:
KIALI-2206 When you are accessing the Kiali console for the first time, and there is no cached browser data for Kiali, the “View in Grafana” link on the Metrics tab of the Kiali Service Details page redirects to the wrong location. The only way you would encounter this issue is if you are accessing Kiali for the first time.
KIALI-507 Kiali does not support Internet Explorer 11. This is because the underlying frameworks do not support Internet Explorer. To access the Kiali console, use one of the two most recent versions of the Chrome, Edge, Firefox or Safari browser.
These limitations exist in Jaeger:
While Kafka publisher is included as part of Jaeger, it is not supported.
Apache Spark is not supported.
Only self-provisioned Elasticsearch instances are supported. External Elasticsearch instances are not supported in this release.
These are the known issues in Jaeger:
TRACING-1166 It is not currently possible to use the Jaeger streaming strategy within a disconnected environment. When a Kafka cluster is being provisioned, it results in a error: Failed to pull image registry.redhat.io/amq7/amq-streams-kafka-24-rhel7@sha256:f9ceca004f1b7dccb3b82d9a8027961f9fe4104e0ed69752c0bdd8078b4a1076
.
TRACING-809 Jaeger Ingester is incompatible with Kafka 2.3. When there are two or more instances of the Jaeger Ingester and enough traffic it will continuously generate rebalancing messages in the logs. This is due to a regression in Kafka 2.3 that was fixed in Kafka 2.3.1. For more information, see Jaegertracing-1819.
The following issues been resolved in the current release:
MAISTRA-1352 Cert-manager Custom Resource Definitions (CRD) from the control plane installation have been removed for this release and future releases. If you have already installed Red Hat OpenShift Service Mesh, the CRDs must be removed manually if cert-manager is not being used.
To remove the CRDs, run the following commands:
$ oc delete crd clusterissuers.certmanager.k8s.io
$ oc delete crd issuers.certmanager.k8s.io
$ oc delete crd certificates.certmanager.k8s.io
$ oc delete crd orders.certmanager.k8s.io
$ oc delete crd challenges.certmanager.k8s.io
MAISTRA-1649 Headless services conflict when in different namespaces. When deploying headless services within different namespaces the endpoint configuration is merged and results in invalid Envoy configurations being pushed to the sidecars.
MAISTRA-1541 Panic in kubernetesenv when the controller is not set on owner reference. If a pod has an ownerReference which does not specify the controller, this will cause a panic within the kubernetesenv cache.go
code.
TRACING-1300 Failed connection between Agent and Collector when using Istio sidecar. An update of the Jaeger Operator enabled TLS communication by default between a Jaeger sidecar agent and the Jaeger Collector.
TRACING-1208 Authentication "500 Internal Error" when accessing Jaeger UI. When trying to authenticate to the UI using OAuth, I get a 500 error because oauth-proxy sidecar doesn’t trust the custom CA bundle defined at installation time with the additionalTrustBundle.
OSSM-99 Workloads generated from direct Pod without labels may crash Kiali.
OSSM-93 IstioConfigList can’t filter by two or more names.
OSSM-92 Cancelling unsaved changes on the VS/DR YAML edit page does not cancel the changes.
OSSM-90 Traces not available on the service details page.
MAISTRA-1001 Closing HTTP/2 connections could lead to segmentation faults in istio-proxy
.
MAISTRA-932 Added the requires
metadata to add dependency relationship between Jaeger operator and Elasticsearch operator. Ensures that when the Jaeger operator is installed, it automatically deploys the Elasticsearch operator if it is not available.
MAISTRA-862 Galley dropped watches and stopped providing configuration to other components after many namespace deletions and re-creations.
MAISTRA-833 Pilot stopped delivering configuration after many namespace deletions and re-creations.
MAISTRA-684 The default Jaeger version in the istio-operator
is 1.12.0, which does not match Jaeger version 1.13.1 that shipped in Red Hat OpenShift Service Mesh 0.12.TechPreview.
MAISTRA-622 In Maistra 0.12.0/TP12, permissive mode does not work. The user has the option to use Plain text mode or Mutual TLS mode, but not permissive.
MAISTRA-572 Jaeger cannot be used with Kiali. In this release Jaeger is configured to use the OAuth proxy, but is also only configured to work through a browser and does not allow service access. Kiali cannot properly communicate with the Jaeger endpoint and it considers Jaeger to be disabled. See also TRACING-591.
MAISTRA-357 In OpenShift 4 Beta on AWS, it is not possible, by default, to access a TCP or HTTPS service through the ingress gateway on a port other than port 80. The AWS load balancer has a health check that verifies if port 80 on the service endpoint is active. Without a service running on port 80, the load balancer health check fails.
MAISTRA-348 OpenShift 4 Beta on AWS does not support ingress gateway traffic on ports other than 80 or 443. If you configure your ingress gateway to handle TCP traffic with a port number other than 80 or 443, you have to use the service hostname provided by the AWS load balancer rather than the OpenShift router as a workaround.
KIALI-3239 If a Kiali Operator pod has failed with a status of “Evicted” it blocks the Kiali operator from deploying. The workaround is to delete the Evicted pod and redeploy the Kiali operator.
KIALI-3118 After changes to the ServiceMeshMemberRoll, for example adding or removing projects, the Kiali pod restarts and then displays errors on the Graph page while the Kiali pod is restarting.
KIALI-3096 Runtime metrics fail in Service Mesh. There is an OAuth filter between the Service Mesh and Prometheus, requiring a bearer token to be passed to Prometheus before access is granted. Kiali has been updated to use this token when communicating to the Prometheus server, but the application metrics are currently failing with 403 errors.
KIALI-3070 This bug only affects custom dashboards, not the default dashboards. When you select labels in metrics settings and refresh the page, your selections are retained in the menu but your selections are not displayed on the charts.
KIALI-2686 When the control plane has many namespaces, it can lead to performance issues.