Red Hat is committed to replacing problematic language in our code, documentation, and web properties. We are beginning with these four terms: master, slave, blacklist, and whitelist. Because of the enormity of this endeavor, these changes will be implemented gradually over several upcoming releases. For more details, see our CTO Chris Wright’s message.
Red Hat OpenShift Service Mesh provides a number of key capabilities uniformly across a network of services:
Traffic Management - Control the flow of traffic and API calls between services, make calls more reliable, and make the network more robust in the face of adverse conditions.
Service Identity and Security - Provide services in the mesh with a verifiable identity and provide the ability to protect service traffic as it flows over networks of varying degrees of trustworthiness.
Policy Enforcement - Apply organizational policy to the interaction between services, ensure access policies are enforced and resources are fairly distributed among consumers. Policy changes are made by configuring the mesh, not by changing application code.
Telemetry - Gain understanding of the dependencies between services and the nature and flow of traffic between them, providing the ability to quickly identify issues.
This release of Red Hat OpenShift Service Mesh adds new features and enhancements, and is supported on OpenShift Container Platform 4.9 and 4.10.
| Component | Version |
|---|---|
Istio |
1.12.7 |
Envoy Proxy |
1.20.4 |
Jaeger |
1.34.1 |
Kiali |
1.48.0.16 |
WasmPlugin APIThis release adds support for the WasmPlugin API and deprecates the ServiceMeshExtention API.
This release introduces service mesh support for Red Hat OpenShift on AWS (ROSA), including multi-cluster federation.
istio-node DaemonSet renamedThis release, the istio-node DaemonSet is renamed to istio-cni-node to match the name in upstream Istio.
Istio 1.10 updated Envoy to send traffic to the application container using eth0 rather than lo by default.
This release marks the end of support for Service Mesh Control Planes based on Service Mesh 1.1 for all platforms.
Service Mesh 2.2 is based on Istio 1.12, which brings in new features and product enhancements. While many Istio 1.12 features are supported, the following unsupported features should be noted:
AuthPolicy Dry Run is a tech preview feature.
gRPC Proxyless Service Mesh is a tech preview feature.
Telemetry API is a tech preview feature.
Discovery selectors is not a supported feature.
External control plane is not a supported feature.
Gateway injection is not a supported feature.
Kubernetes Gateway API is a technology preview feature that is disabled by default.
To enable the feature, set the following environment variables for the Istiod container in ServiceMeshControlPlane:
spec:
runtime:
components:
pilot:
container:
env:
PILOT_ENABLE_GATEWAY_API: true
PILOT_ENABLE_GATEWAY_API_STATUS: true
# and optionally, for the deployment controller
PILOT_ENABLE_GATEWAY_API_DEPLOYMENT_CONTROLLER: trueRestricting route attachment on Gateway API listeners is possible using the SameNamespace or All settings. Istio ignores usage of label selectors in listeners.allowedRoutes.namespaces and reverts to the default behavior (SameNamespace).
This release of Red Hat OpenShift Service Mesh addresses Common Vulnerabilities and Exposures (CVEs) and bug fixes.
This release of Red Hat OpenShift Service Mesh addresses Common Vulnerabilities and Exposures (CVEs) and bug fixes.
This release of Red Hat OpenShift Service Mesh addresses Common Vulnerabilities and Exposures (CVEs) and bug fixes.
This release of Red Hat OpenShift Service Mesh addresses Common Vulnerabilities and Exposures (CVEs) and bug fixes.
With this release, the Red Hat OpenShift distributed tracing platform Operator is now installed to the openshift-distributed-tracing namespace by default. Previously the default installation had been in the openshift-operator namespace.
This release of Red Hat OpenShift Service Mesh addresses Common Vulnerabilities and Exposures (CVEs) and bug fixes.
This release also adds the ability to disable the automatic creation of network policies.
| Component | Version |
|---|---|
Istio |
1.9.9 |
Envoy Proxy |
1.17.1 |
Jaeger |
1.24.1 |
Kiali |
1.36.7 |
Red Hat OpenShift Service Mesh automatically creates and manages a number of NetworkPolicies resources in the control plane and application namespaces. This is to ensure that applications and the control plane can communicate with each other.
If you want to disable the automatic creation and management of NetworkPolicies resources, for example to enforce company security policies, you can do so. You can edit the ServiceMeshControlPlane to set the spec.security.manageNetworkPolicy setting to false
|
When you disable |
In the OpenShift Container Platform web console, click Operators → Installed Operators.
Select the project where you installed the control plane, for example istio-system, from the Project menu.
Click the Red Hat OpenShift Service Mesh Operator. In the Istio Service Mesh Control Plane column, click the name of your ServiceMeshControlPlane, for example basic-install.
On the Create ServiceMeshControlPlane Details page, click YAML to modify your configuration.
Set the ServiceMeshControlPlane field spec.security.manageNetworkPolicy to false, as shown in this example.
apiVersion: maistra.io/v2
kind: ServiceMeshControlPlane
spec:
security:
trust:
manageNetworkPolicy: falseClick Save.
This release of Red Hat OpenShift Service Mesh adds support for Istio 1.9.8, Envoy Proxy 1.17.1, Jaeger 1.24.1, and Kiali 1.36.5 on OpenShift Container Platform 4.6 EUS, 4.7, 4.8, 4.9, along with new features and enhancements.
| Component | Version |
|---|---|
Istio |
1.9.6 |
Envoy Proxy |
1.17.1 |
Jaeger |
1.24.1 |
Kiali |
1.36.5 |
New Custom Resource Definitions (CRDs) have been added to support federating service meshes. Service meshes may be federated both within the same cluster or across different OpenShift clusters. These new resources include:
ServiceMeshPeer - Defines a federation with a separate service mesh, including gateway configuration, root trust certificate configuration, and status fields. In a pair of federated meshes, each mesh will define its own separate ServiceMeshPeer resource.
ExportedServiceMeshSet - Defines which services for a given ServiceMeshPeer are available for the peer mesh to import.
ImportedServiceSet - Defines which services for a given ServiceMeshPeer are imported from the peer mesh. These services must also be made available by the peer’s ExportedServiceMeshSet resource.
Service Mesh Federation is not supported between clusters on Red Hat OpenShift Service on AWS (ROSA), Azure Red Hat OpenShift (ARO), or OpenShift Dedicated (OSD).
The OVN-Kubernetes Container Network Interface (CNI) was previously introduced as a Technology Preview feature in Red Hat OpenShift Service Mesh 2.0.1 and is now generally available in Red Hat OpenShift Service Mesh 2.1 and 2.0.x for use on OpenShift Container Platform 4.7.32, OpenShift Container Platform 4.8.12, and OpenShift Container Platform 4.9.
The ServiceMeshExtensions Custom Resource Definition (CRD), first introduced in 2.0 as Technology Preview, is now generally available. You can use CRD to build your own plugins, but Red Hat does not provide support for the plugins you create.
Mixer has been completely removed in Service Mesh 2.1. Upgrading from a Service Mesh 2.0.x release to 2.1 will be blocked if Mixer is enabled. Mixer plugins will need to be ported to WebAssembly Extensions.
With Mixer now officially removed, OpenShift Service Mesh 2.1 does not support the 3scale mixer adapter. Before upgrading to Service Mesh 2.1, remove the Mixer-based 3scale adapter and any additional Mixer plugins. Then, manually install and configure the new 3scale WebAssembly adapter with Service Mesh 2.1+ using a ServiceMeshExtension resource.
3scale 2.11 introduces an updated Service Mesh integration based on WebAssembly.
Service Mesh 2.1 is based on Istio 1.9, which brings in a large number of new features and product enhancements. While the majority of Istio 1.9 features are supported, the following exceptions should be noted:
Virtual Machine integration is not yet supported
Kubernetes Gateway API is not yet supported
Remote fetch and load of WebAssembly HTTP filters are not yet supported
Custom CA Integration using the Kubernetes CSR API is not yet supported
Request Classification for monitoring traffic is a tech preview feature
Integration with external authorization systems via Authorization policy’s CUSTOM action is a tech preview feature
The amount of time Red Hat OpenShift Service Mesh uses to prune old resources at the end of every ServiceMeshControlPlane reconciliation has been reduced. This results in faster ServiceMeshControlPlane deployments, and allows changes applied to existing SMCPs to take effect more quickly.
Kiali 1.36 includes the following features and enhancements:
Service Mesh troubleshooting functionality
Control plane and gateway monitoring
Proxy sync statuses
Envoy configuration views
Unified view showing Envoy proxy and application logs interleaved
Namespace and cluster boxing to support federated service mesh views
New validations, wizards, and distributed tracing enhancements
This release of Red Hat OpenShift Service Mesh addresses Common Vulnerabilities and Exposures (CVEs) and bug fixes.
This release of Red Hat OpenShift Service Mesh addresses Common Vulnerabilities and Exposures (CVEs) and bug fixes.
This release of Red Hat OpenShift Service Mesh addresses bug fixes.
This release of Red Hat OpenShift Service Mesh addresses Common Vulnerabilities and Exposures (CVEs).
Red Hat OpenShift Service Mesh contains a remotely exploitable vulnerability, CVE-2021-39156, where an HTTP request with a fragment (a section in the end of a URI that begins with a # character) in the URI path could bypass the Istio URI path-based authorization policies. For instance, an Istio authorization policy denies requests sent to the URI path /user/profile. In the vulnerable versions, a request with URI path /user/profile#section1 bypasses the deny policy and routes to the backend (with the normalized URI path /user/profile%23section1), possibly leading to a security incident.
You are impacted by this vulnerability if you use authorization policies with DENY actions and operation.paths, or ALLOW actions and operation.notPaths.
With the mitigation, the fragment part of the request’s URI is removed before the authorization and routing. This prevents a request with a fragment in its URI from bypassing authorization policies which are based on the URI without the fragment part.
To opt-out from the new behavior in the mitigation, the fragment section in the URI will be kept. You can configure your ServiceMeshControlPlane to keep URI fragments.
|
Disabling the new behavior will normalize your paths as described above and is considered unsafe. Ensure that you have accommodated for this in any security policies before opting to keep URI fragments. |
ServiceMeshControlPlane modificationapiVersion: maistra.io/v2
kind: ServiceMeshControlPlane
metadata:
name: basic
spec:
techPreview:
meshConfig:
defaultConfig:
proxyMetadata: HTTP_STRIP_FRAGMENT_FROM_PATH_UNSAFE_IF_DISABLED: "false"Istio generates hostnames for both the hostname itself and all matching ports. For instance, a virtual service or Gateway for a host of "httpbin.foo" generates a config matching "httpbin.foo and httpbin.foo:*". However, exact match authorization policies only match the exact string given for the hosts or notHosts fields.
Your cluster is impacted if you have AuthorizationPolicy resources using exact string comparison for the rule to determine hosts or notHosts.
You must update your authorization policy rules to use prefix match instead of exact match. For example, replacing hosts: ["httpbin.com"] with hosts: ["httpbin.com:*"] in the first AuthorizationPolicy example.
apiVersion: security.istio.io/v1beta1
kind: AuthorizationPolicy
metadata:
name: httpbin
namespace: foo
spec:
action: DENY
rules:
- from:
- source:
namespaces: ["dev"]
to:
- operation:
hosts: [“httpbin.com”,"httpbin.com:*"]apiVersion: security.istio.io/v1beta1
kind: AuthorizationPolicy
metadata:
name: httpbin
namespace: default
spec:
action: DENY
rules:
- to:
- operation:
hosts: ["httpbin.example.com:*"]This release of Red Hat OpenShift Service Mesh addresses Common Vulnerabilities and Exposures (CVEs) and bug fixes.
Red Hat OpenShift Service Mesh is now supported through Red Hat OpenShift Dedicated and Microsoft Azure Red Hat OpenShift.
This release of Red Hat OpenShift Service Mesh addresses Common Vulnerabilities and Exposures (CVEs) and bug fixes.
This release of Red Hat OpenShift Service Mesh addresses Common Vulnerabilities and Exposures (CVEs) and bug fixes.
This release of Red Hat OpenShift Service Mesh addresses Common Vulnerabilities and Exposures (CVEs) and bug fixes.
|
There are manual steps that must be completed to address CVE-2021-29492 and CVE-2021-31920. |
Istio contains a remotely exploitable vulnerability where an HTTP request path with multiple slashes or escaped slash characters (%2F or %5C) could potentially bypass an Istio authorization policy when path-based authorization rules are used.
For example, assume an Istio cluster administrator defines an authorization DENY policy to reject the request at path /admin. A request sent to the URL path //admin will NOT be rejected by the authorization policy.
According to RFC 3986, the path //admin with multiple slashes should technically be treated as a different path from the /admin. However, some backend services choose to normalize the URL paths by merging multiple slashes into a single slash. This can result in a bypass of the authorization policy (//admin does not match /admin), and a user can access the resource at path /admin in the backend; this would represent a security incident.
Your cluster is impacted by this vulnerability if you have authorization policies using ALLOW action + notPaths field or DENY action + paths field patterns. These patterns are vulnerable to unexpected policy bypasses.
Your cluster is NOT impacted by this vulnerability if:
You don’t have authorization policies.
Your authorization policies don’t define paths or notPaths fields.
Your authorization policies use ALLOW action + paths field or DENY action + notPaths field patterns. These patterns could only cause unexpected rejection instead of policy bypasses. The upgrade is optional for these cases.
|
The Red Hat OpenShift Service Mesh configuration location for path normalization is different from the Istio configuration. |
Istio authorization policies can be based on the URL paths in the HTTP request. Path normalization, also known as URI normalization, modifies and standardizes the incoming requests' paths so that the normalized paths can be processed in a standard way. Syntactically different paths may be equivalent after path normalization.
Istio supports the following normalization schemes on the request paths before evaluating against the authorization policies and routing the requests:
| Option | Description | Example | Notes |
|---|---|---|---|
|
No normalization is done. Anything received by Envoy will be forwarded exactly as-is to any backend service. |
|
This setting is vulnerable to CVE-2021-31920. |
|
This is currently the option used in the default installation of Istio. This applies the |
|
This setting is vulnerable to CVE-2021-31920. |
|
Slashes are merged after the BASE normalization. |
|
Update to this setting to mitigate CVE-2021-31920. |
|
The strictest setting when you allow all traffic by default. This setting is recommended, with the caveat that you must thoroughly test your authorization policies routes. Percent-encoded slash and backslash characters ( |
|
Update to this setting to mitigate CVE-2021-31920. This setting is more secure, but also has the potential to break applications. Test your applications before deploying to production. |
The normalization algorithms are conducted in the following order:
Percent-decode %2F, %2f, %5C and %5c.
The RFC 3986 and other normalization implemented by the normalize_path option in Envoy.
Merge slashes.
|
While these normalization options represent recommendations from HTTP standards and common industry practices, applications may interpret a URL in any way it chooses to. When using denial policies, ensure that you understand how your application behaves. |
Ensuring Envoy normalizes request paths to match your backend services' expectations is critical to the security of your system.
The following examples can be used as a reference for you to configure your system.
The normalized URL paths, or the original URL paths if NONE is selected, will be:
Used to check against the authorization policies.
Forwarded to the backend application.
| If your application… | Choose… |
|---|---|
Relies on the proxy to do normalization |
|
Normalizes request paths based on RFC 3986 and does not merge slashes. |
|
Normalizes request paths based on RFC 3986 and merges slashes, but does not decode percent-encoded slashes. |
|
Normalizes request paths based on RFC 3986, decodes percent-encoded slashes, and merges slashes. |
|
Processes request paths in a way that is incompatible with RFC 3986. |
|
To configure path normalization for Red Hat OpenShift Service Mesh, specify the following in your ServiceMeshControlPlane. Use the configuration examples to help determine the settings for your system.
spec:
techPreview:
global:
pathNormalization: <option>In some environments, it may be useful to have paths in authorization policies compared in a case insensitive manner.
For example, treating https://myurl/get and https://myurl/GeT as equivalent.
In those cases, you can use the EnvoyFilter shown below.
This filter will change both the path used for comparison and the path presented to the application. In this example, istio-system is the name of the control plane project.
Save the EnvoyFilter to a file and execute the following command:
$ oc create -f <myEnvoyFilterFile>apiVersion: networking.istio.io/v1alpha3
kind: EnvoyFilter
metadata:
name: ingress-case-insensitive
namespace: istio-system
spec:
configPatches:
- applyTo: HTTP_FILTER
match:
context: GATEWAY
listener:
filterChain:
filter:
name: "envoy.filters.network.http_connection_manager"
subFilter:
name: "envoy.filters.http.router"
patch:
operation: INSERT_BEFORE
value:
name: envoy.lua
typed_config:
"@type": "type.googleapis.com/envoy.extensions.filters.http.lua.v3.Lua"
inlineCode: |
function envoy_on_request(request_handle)
local path = request_handle:headers():get(":path")
request_handle:headers():replace(":path", string.lower(path))
endThis release of Red Hat OpenShift Service Mesh addresses Common Vulnerabilities and Exposures (CVEs) and bug fixes.
In addition, this release has the following new features:
Added an option to the must-gather data collection tool that gathers information from a specified control plane namespace. For more information, see OSSM-351.
Improved performance for control planes with hundreds of namespaces
This release of Red Hat OpenShift Service Mesh adds support for IBM Z and IBM Power Systems. It also addresses Common Vulnerabilities and Exposures (CVEs) and bug fixes.
This release of Red Hat OpenShift Service Mesh addresses Common Vulnerabilities and Exposures (CVEs) and bug fixes.
This release of Red Hat OpenShift Service Mesh adds support for Istio 1.6.5, Jaeger 1.20.0, Kiali 1.24.2, and the 3scale Istio Adapter 2.0 and OpenShift Container Platform 4.6.
In addition, this release has the following new features:
Simplifies installation, upgrades, and management of the control plane.
Reduces the control plane’s resource usage and startup time.
Improves performance by reducing inter-control plane communication over networking.
Adds support for Envoy’s Secret Discovery Service (SDS). SDS is a more secure and efficient mechanism for delivering secrets to Envoy side car proxies.
Removes the need to use Kubernetes Secrets, which have well known security risks.
Improves performance during certificate rotation, as proxies no longer require a restart to recognize new certificates.
Adds support for Istio’s Telemetry v2 architecture, which is built using WebAssembly extensions. This new architecture brings significant performance improvements.
Updates the ServiceMeshControlPlane resource to v2 with a streamlined configuration to make it easier to manage the Control Plane.
Introduces WebAssembly extensions as a Technology Preview feature.
Some features in this release are currently in Technology Preview. These experimental features are not intended for production use.
|
Technology Preview features are not supported with Red Hat production service level agreements (SLAs) and might not be functionally complete. Red Hat does not recommend using them in production. These features provide early access to upcoming product features, enabling customers to test functionality and provide feedback during the development process. For more information about the support scope of Red Hat Technology Preview features, see the Technology Preview Support Scope. |
In the table, features are marked with the following statuses:
TP: Technology Preview
GA: General Availability
Note the following scope of support on the Red Hat Customer Portal for these features:
| Feature | Istio Version | Support Status | Description |
|---|---|---|---|
holdApplicationUntilProxyStarts |
1.7 |
TP |
Blocks application container startup until proxy is running |
DNS capture |
1.8 |
GA |
Enabled by default |
Some features available in previous releases have been deprecated or removed.
Deprecated functionality is still included in OpenShift Container Platform and continues to be supported; however, it will be removed in a future release of this product and is not recommended for new deployments.
Removed functionality no longer exists in the product.
The ServiceMeshExtension API is deprecated as of release 2.2 and will be removed in a future release. While ServiceMeshExtension API is still supported in release 2.2, customers should start moving to the new WasmPlugin API.
This release marks the end of support for service mesh control planes based on Service Mesh 1.1 for all platforms.
In Service Mesh 2.1, the Mixer component is removed. Bug fixes and support is provided through the end of the Service Mesh 2.0 life cycle.
Upgrading from a Service Mesh 2.0.x release to 2.1 will not proceed if Mixer plugins are enabled. Mixer plugins must be ported to WebAssembly Extensions.
The Mixer component was deprecated in release 2.0 and will be removed in release 2.1. While using Mixer for implementing extensions was still supported in release 2.0, extensions should have been migrated to the new WebAssembly mechanism.
The following resource types are no longer supported in Red Hat OpenShift Service Mesh 2.0:
Policy (authentication.istio.io/v1alpha1) is no longer supported. Depending on the specific configuration in your Policy resource, you may have to configure multiple resources to achieve the same effect.
Use RequestAuthentication (security.istio.io/v1beta1)
Use PeerAuthentication (security.istio.io/v1beta1)
ServiceMeshPolicy (maistra.io/v1) is no longer supported.
Use RequestAuthentication or PeerAuthentication, as mentioned above, but place in the control plane namespace.
RbacConfig (rbac.istio.io/v1alpha1) is no longer supported.
Replaced by AuthorizationPolicy (security.istio.io/v1beta1), which encompasses behavior of RbacConfig, ServiceRole, and ServiceRoleBinding.
ServiceMeshRbacConfig (maistra.io/v1) is no longer supported.
Use AuthorizationPolicy as above, but place in control plane namespace.
ServiceRole (rbac.istio.io/v1alpha1) is no longer supported.
ServiceRoleBinding (rbac.istio.io/v1alpha1) is no longer supported.
In Kiali, the login and LDAP strategies are deprecated. A future version will introduce authentication using OpenID providers.
These limitations exist in Red Hat OpenShift Service Mesh:
Red Hat OpenShift Service Mesh does not yet support IPv6, as it is not yet fully supported by the upstream Istio project. As a result, Red Hat OpenShift Service Mesh does not support dual-stack clusters.
Graph layout - The layout for the Kiali graph can render differently, depending on your application architecture and the data to display (number of graph nodes and their interactions). Because it is difficult if not impossible to create a single layout that renders nicely for every situation, Kiali offers a choice of several different layouts. To choose a different layout, you can choose a different Layout Schema from the Graph Settings menu.
The first time you access related services such as distributed tracing platform and Grafana, from the Kiali console, you must accept the certificate and re-authenticate using your OpenShift Container Platform login credentials. This happens due to an issue with how the framework displays embedded pages in the console.
The Bookinfo sample application cannot be installed on IBM Z and IBM Power.
WebAssembly extensions are not supported on IBM Z and IBM Power.
LuaJIT is not supported on IBM Power.
These are the known issues in Red Hat OpenShift Service Mesh:
OSSM-1668
jwksResolverCA field is missing in SMCP.
If you upgrade from Service Mesh operator 2.1.3 to Service Mesh operator 2.2, then the jwksResolverCA field is not supported. You must use the techPreview jwksResolverExtraRootCA field to enable additional JWKS CA certificates.
OSSM-1655 Kiali dashboard shows error after enabling mTLS in SMCP.
After enabling the spec.security.controlPlane.mtls setting in the SMCP, the Kiali console displays the following error message "No subsets defined".
OSSM-1211 Configuring Federated service meshes for failover does not work as expected.
The Istiod pilot log displays the following error: envoy connection [C289] TLS error: 337047686:SSL routines:tls_process_server_certificate:certificate verify failed
There is no workaround at this time.
Istio-14743 Due to limitations in the version of Istio that this release of Red Hat OpenShift Service Mesh is based on, there are several applications that are currently incompatible with Service Mesh. See the linked community issue for details.
OSSM-1396 If a gateway resource contains the spec.externalIPs setting, instead of being recreated when the ServiceMeshControlPlane is updated, the gateway is removed and never recreated.
OSSM-1168 When service mesh resources are created as a single YAML file, the Envoy proxy sidecar is not reliably injected into pods. When the SMCP, SMMR, and Deployment resources are created individually, the deployment works as expected.
OSSM-1052 When configuring a Service ExternalIP for the ingressgateway in the service mesh control plane, the service is not created. The schema for the SMCP is missing the parameter for the service. The workaround for this issue is to disable the gateway creation in the SMCP spec and manage the gateway deployment entirely manually (including Service, Role and RoleBinding).
OSSM-882 Namespace is in the accessible_namespace list but does not appear in Kiali UI. By default, Kiali will not show any namespaces that start with "kube" because these namespaces are typically internal-use only and not part of a mesh.
For example, if you create a namespace called 'akube-a' and add it to the Service Mesh member roll, then the Kiali UI does not display the namespace. For defined exclusion patterns, the software excludes namespaces that start with or contain the pattern.
The workaround is to change the Kiali Custom Resource setting so it prefixes the setting with a carat (^). For example:
api:
namespaces:
exclude:
- "^istio-operator"
- "^kube-.*"
- "^openshift.*"
- "^ibm.*"
- "^kiali-operator"MAISTRA-2735 The resources that the Service Mesh Operator deletes when reconciling the SMCP have changed. Previously, the Operator deleted a resource with the following labels:
maistra.io/owner
app.kubernetes.io/version
Now, the Operator ignores resources that don’t also include the app.kubernetes.io/managed-by=maistra-istio-operator label. If you create your own resources, you should not add the app.kubernetes.io/managed-by=maistra-istio-operator label to them.
MAISTRA-2692 With Mixer removed, custom metrics that have been defined in Service Mesh 2.0.x cannot be used in 2.1. Custom metrics can be configured using EnvoyFilter. Red Hat is unable to support EnvoyFilter configuration except where explicitly documented. This is due to tight coupling with the underlying Envoy APIs, meaning that backward compatibility cannot be maintained.
MAISTRA-2648 ServiceMeshExtensions are currently not compatible with meshes deployed on IBM Z Systems.
MAISTRA-2411 When the Operator creates a new ingress gateway using spec.gateways.additionaIngress in the ServiceMeshControlPlane, Operator is not creating a NetworkPolicy for the additional ingress gateway like it does for the default istio-ingressgateway. This is causing a 503 response from the route of the new gateway. The workaround for this issue is to manually create the NetworkPolicy in the <istio-system> namespace.
MAISTRA-1959 Migration to 2.0 Prometheus scraping (spec.addons.prometheus.scrape set to true) does not work when mTLS is enabled. Additionally, Kiali displays extraneous graph data when mTLS is disabled.
This problem can be addressed by excluding port 15020 from proxy configuration, for example,
spec:
proxy:
networking:
trafficControl:
inbound:
excludedPorts:
- 15020MAISTRA-1947 Technology Preview Updates to ServiceMeshExtensions are not applied. The workaround is to remove and recreate the ServiceMeshExtensions.
MAISTRA-1314 Red Hat OpenShift Service Mesh does not yet support IPv6.
MAISTRA-806 Evicted Istio Operator Pod causes mesh and CNI not to deploy.
If the istio-operator pod is evicted while deploying the control pane, delete the evicted istio-operator pod.
MAISTRA-681 When the control plane has many namespaces, it can lead to performance issues.
MAISTRA-465 The Maistra Operator fails to create a service for operator metrics.
MAISTRA-453 If you create a new project and deploy pods immediately, sidecar injection does not occur. The operator fails to add the maistra.io/member-of before the pods are created, therefore the pods must be deleted and recreated for sidecar injection to occur.
MAISTRA-158 Applying multiple gateways referencing the same hostname will cause all gateways to stop functioning.
|
New issues for Kiali should be created in the OpenShift Service Mesh project with the |
These are the known issues in Kiali:
KIALI-2206 When you are accessing the Kiali console for the first time, and there is no cached browser data for Kiali, the “View in Grafana” link on the Metrics tab of the Kiali Service Details page redirects to the wrong location. The only way you would encounter this issue is if you are accessing Kiali for the first time.
KIALI-507 Kiali does not support Internet Explorer 11. This is because the underlying frameworks do not support Internet Explorer. To access the Kiali console, use one of the two most recent versions of the Chrome, Edge, Firefox or Safari browser.
These limitations exist in Red Hat OpenShift distributed tracing:
Apache Spark is not supported.
The streaming deployment via AMQ/Kafka is unsupported on IBM Z and IBM Power Systems.
These are the known issues for Red Hat OpenShift distributed tracing:
TRACING-2057 The Kafka API has been updated to v1beta2 to support the Strimzi Kafka Operator 0.23.0. However, this API version is not supported by AMQ Streams 1.6.3. If you have the following environment, your Jaeger services will not be upgraded, and you cannot create new Jaeger services or modify existing Jaeger services:
Jaeger Operator channel: 1.17.x stable or 1.20.x stable
AMQ Streams Operator channel: amq-streams-1.6.x
To resolve this issue, switch the subscription channel for your AMQ Streams Operator to either amq-streams-1.7.x or stable.
The following issues been resolved in the current release:
OSSM-1099
The Kiali console displayed the message Sorry, there was a problem. Try a refresh or navigate to a different page.
OSSM-1074 Pod annotations defined in SMCP are not injected in the pods.
OSSM-999 Kiali retention did not work as expected. Calendar times were greyed out in the dashboard graph.
OSSM-797 Kiali Operator pod generates CreateContainerConfigError while installing or updating the operator.
OSSM-722
Namespace starting with kube is hidden from Kiali.
OSSM-569 There is no CPU memory limit for the Prometheus istio-proxy container. The Prometheus istio-proxy sidecar now uses the resource limits defined in spec.proxy.runtime.container.
OSSM-449 VirtualService and Service causes an error "Only unique values for domains are permitted. Duplicate entry of domain."
OSSM-419 Namespaces with similar names will all show in Kiali namespace list, even though namespaces may not be defined in Service Mesh Member Role.
OSSM-296 When adding health configuration to the Kiali custom resource (CR) is it not being replicated to the Kiali configmap.
OSSM-291 In the Kiali console, on the Applications, Services, and Workloads pages, the "Remove Label from Filters" function is not working.
OSSM-289 In the Kiali console, on the Service Details pages for the 'istio-ingressgateway' and 'jaeger-query' services there are no Traces being displayed. The traces exist in Jaeger.
OSSM-287 In the Kiali console there are no traces being displayed on the Graph Service.
OSSM-285 When trying to access the Kiali console, receive the following error message "Error trying to get OAuth Metadata". The workaround is to restart the Kiali pod.
MAISTRA-2687 Red Hat OpenShift Service Mesh 2.1 federation gateway does not send the full certificate chain when using external certificates. The Service Mesh federation egress gateway only sends the client certificate. Because the federation ingress gateway only knows about the root certificate, it cannot verify the client certificate unless you add the root certificate to the federation import ConfigMap.
MAISTRA-2635 Replace deprecated Kubernetes API. To remain compatible with OpenShift Container Platform 4.8, the apiextensions.k8s.io/v1beta1 API was deprecated as of Red Hat OpenShift Service Mesh 2.0.8.
MAISTRA-2631 The WASM feature is not working because podman is failing due to nsenter binary not being present. Red Hat OpenShift Service Mesh generates the following error message: Error: error configuring CNI network plugin exec: "nsenter": executable file not found in $PATH. The container image now contains nsenter and WASM works as expected.
MAISTRA-2534 When istiod attempted to fetch the JWKS for an issuer specified in a JWT rule, the issuer service responded with a 502. This prevented the proxy container from becoming ready and caused deployments to hang. The fix for the community bug has been included in the Service Mesh 2.0.7 release.
MAISTRA-2401 CVE-2021-3586 servicemesh-operator: NetworkPolicy resources incorrectly specified ports for ingress resources. The NetworkPolicy resources installed for Red Hat OpenShift Service Mesh did not properly specify which ports could be accessed. This allowed access to all ports on these resources from any pod. Network policies applied to the following resources are affected:
Galley
Grafana
Istiod
Jaeger
Kiali
Prometheus
Sidecar injector
MAISTRA-2378 When the cluster is configured to use OpenShift SDN with ovs-multitenant and the mesh contains a large number of namespaces (200+), the OpenShift Container Platform networking plugin is unable to configure the namespaces quickly. Service Mesh times out causing namespaces to be continuously dropped from the service mesh and then reenlisted.
MAISTRA-2370 Handle tombstones in listerInformer. The updated cache codebase was not handling tombstones when translating the events from the namespace caches to the aggregated cache, leading to a panic in the go routine.
MAISTRA-2117 Add optional ConfigMap mount to operator. The CSV now contains an optional ConfigMap volume mount, which mounts the smcp-templates ConfigMap if it exists. If the smcp-templates ConfigMap does not exist, the mounted directory is empty. When you create the ConfigMap, the directory is populated with the entries from the ConfigMap and can be referenced in SMCP.spec.profiles. No restart of the Service Mesh operator is required.
Customers using the 2.0 operator with a modified CSV to mount the smcp-templates ConfigMap can upgrade to Red Hat OpenShift Service Mesh 2.1. After upgrading, you can continue using an existing ConfigMap, and the profiles it contains, without editing the CSV. Customers that previously used ConfigMap with a different name will either have to rename the ConfigMap or update the CSV after upgrading.
MAISTRA-2010 AuthorizationPolicy does not support request.regex.headers field. The validatingwebhook rejects any AuthorizationPolicy with the field, and even if you disable that, Pilot tries to validate it using the same code, and it does not work.
MAISTRA-1979 Migration to 2.0 The conversion webhook drops the following important fields when converting SMCP.status from v2 to v1:
conditions
components
observedGeneration
annotations
Upgrading the operator to 2.0 might break client tools that read the SMCP status using the maistra.io/v1 version of the resource.
This also causes the READY and STATUS columns to be empty when you run oc get servicemeshcontrolplanes.v1.maistra.io.
MAISTRA-1983 Migration to 2.0 Upgrading to 2.0.0 with an existing invalid ServiceMeshControlPlane cannot easily be repaired. The invalid items in the ServiceMeshControlPlane resource caused an unrecoverable error. The fix makes the errors recoverable. You can delete the invalid resource and replace it with a new one or edit the resource to fix the errors. For more information about editing your resource, see [Configuring the Red Hat OpenShift Service Mesh installation].
Maistra-1502 As a result of CVEs fixes in version 1.0.10, the Istio dashboards are not available from the Home Dashboard menu in Grafana. The Istio dashboards still exist. To access them, click the Dashboard menu in the navigation panel and select the Manage tab.
MAISTRA-1399 Red Hat OpenShift Service Mesh no longer prevents you from installing unsupported CNI protocols. The supported network configurations has not changed.
MAISTRA-1089 Migration to 2.0 Gateways created in a non-control plane namespace are automatically deleted. After removing the gateway definition from the SMCP spec, you need to manually delete these resources.
MAISTRA-858 The following Envoy log messages describing deprecated options and configurations associated with Istio 1.1.x are expected:
[2019-06-03 07:03:28.943][19][warning][misc] [external/envoy/source/common/protobuf/utility.cc:129] Using deprecated option 'envoy.api.v2.listener.Filter.config'. This configuration will be removed from Envoy soon.
[2019-08-12 22:12:59.001][13][warning][misc] [external/envoy/source/common/protobuf/utility.cc:174] Using deprecated option 'envoy.api.v2.Listener.use_original_dst' from file lds.proto. This configuration will be removed from Envoy soon.
MAISTRA-193 Unexpected console info messages are visible when health checking is enabled for citadel.
Bug 1821432 Toggle controls in OpenShift Container Platform Control Resource details page do not update the CR correctly. UI Toggle controls in the Service Mesh Control Plane (SMCP) Overview page in the OpenShift Container Platform web console sometimes update the wrong field in the resource. To update a SMCP, edit the YAML content directly or update the resource from the command line instead of clicking the toggle controls.
TRACING-2337 Jaeger is logging a repetitive warning message in the Jaeger logs similar to the following:
{"level":"warn","ts":1642438880.918793,"caller":"channelz/logging.go:62","msg":"[core]grpc: Server.Serve failed to create ServerTransport: connection error: desc = \"transport: http2Server.HandleStreams received bogus greeting from client: \\\"\\\\x16\\\\x03\\\\x01\\\\x02\\\\x00\\\\x01\\\\x00\\\\x01\\\\xfc\\\\x03\\\\x03vw\\\\x1a\\\\xc9T\\\\xe7\\\\xdaCj\\\\xb7\\\\x8dK\\\\xa6\\\"\"","system":"grpc","grpc_log":true}This issue was resolved by exposing only the HTTP(S) port of the query service, and not the gRPC port.
TRACING-2009 The Jaeger Operator has been updated to include support for the Strimzi Kafka Operator 0.23.0.
TRACING-1907 The Jaeger agent sidecar injection was failing due to missing config maps in the application namespace. The config maps were getting automatically deleted due to an incorrect OwnerReference field setting and as a result, the application pods were not moving past the "ContainerCreating" stage. The incorrect settings have been removed.
TRACING-1725 Follow-up to TRACING-1631. Additional fix to ensure that Elasticsearch certificates are properly reconciled when there are multiple Jaeger production instances, using same name but within different namespaces. See also BZ-1918920.
TRACING-1631 Multiple Jaeger production instances, using same name but within different namespaces, causing Elasticsearch certificate issue. When multiple service meshes were installed, all of the Jaeger Elasticsearch instances had the same Elasticsearch secret instead of individual secrets, which prevented the OpenShift Elasticsearch Operator from communicating with all of the Elasticsearch clusters.
TRACING-1300 Failed connection between Agent and Collector when using Istio sidecar. An update of the Jaeger Operator enabled TLS communication by default between a Jaeger sidecar agent and the Jaeger Collector.
TRACING-1208 Authentication "500 Internal Error" when accessing Jaeger UI. When trying to authenticate to the UI using OAuth, I get a 500 error because oauth-proxy sidecar doesn’t trust the custom CA bundle defined at installation time with the additionalTrustBundle.
TRACING-1166 It is not currently possible to use the Jaeger streaming strategy within a disconnected environment. When a Kafka cluster is being provisioned, it results in a error: Failed to pull image registry.redhat.io/amq7/amq-streams-kafka-24-rhel7@sha256:f9ceca004f1b7dccb3b82d9a8027961f9fe4104e0ed69752c0bdd8078b4a1076.
TRACING-809 Jaeger Ingester is incompatible with Kafka 2.3. When there are two or more instances of the Jaeger Ingester and enough traffic it will continuously generate rebalancing messages in the logs. This is due to a regression in Kafka 2.3 that was fixed in Kafka 2.3.1. For more information, see Jaegertracing-1819.
BZ-1918920/LOG-1619 The Elasticsearch pods does not get restarted automatically after an update. As a workaround, restart the pods manually.