1. Documentation
    2. history bug_report picture_as_pdf
×

Service Mesh Release Notes

Making open source more inclusive

Red Hat is committed to replacing problematic language in our code, documentation, and web properties. We are beginning with these four terms: master, slave, blacklist, and whitelist. Because of the enormity of this endeavor, these changes will be implemented gradually over several upcoming releases. For more details, see our CTO Chris Wright’s message.

New features and enhancements

This release adds improvements related to the following components and concepts.

New features Red Hat OpenShift Service Mesh version 2.2.2

This release of Red Hat OpenShift Service Mesh addresses Common Vulnerabilities and Exposures (CVEs), bug fixes, and is supported on OpenShift Container Platform 4.9 or later.

Component versions included in Red Hat OpenShift Service Mesh version 2.2.2

Component Version

Istio

1.12.7

Envoy Proxy

1.20.6

Jaeger

1.36

Kiali

1.48.2-1

Copy route labels

With this enhancement, in addition to copying annotations, you can copy specific labels for an OpenShift route. Red Hat OpenShift Service Mesh copies all labels and annotations present in the Istio Gateway resource (with the exception of annotations starting with kubectl.kubernetes.io) into the managed OpenShift Route resource.

New features Red Hat OpenShift Service Mesh version 2.2.1

This release of Red Hat OpenShift Service Mesh addresses Common Vulnerabilities and Exposures (CVEs), bug fixes, and is supported on OpenShift Container Platform 4.9 or later.

Component versions included in Red Hat OpenShift Service Mesh version 2.2.1

Component Version

Istio

1.12.7

Envoy Proxy

1.20.6

Jaeger

1.34.1

Kiali

1.48.2-1

New features Red Hat OpenShift Service Mesh 2.2

This release of Red Hat OpenShift Service Mesh adds new features and enhancements, and is supported on OpenShift Container Platform 4.9 or later.

Component versions included in Red Hat OpenShift Service Mesh version 2.2

Component Version

Istio

1.12.7

Envoy Proxy

1.20.4

Jaeger

1.34.1

Kiali

1.48.0.16

WasmPlugin API

This release adds support for the WasmPlugin API and deprecates the ServiceMeshExtention API.

ROSA support

This release introduces service mesh support for Red Hat OpenShift on AWS (ROSA), including multi-cluster federation.

istio-node DaemonSet renamed

This release, the istio-node DaemonSet is renamed to istio-cni-node to match the name in upstream Istio.

Envoy sidecar networking changes

Istio 1.10 updated Envoy to send traffic to the application container using eth0 rather than lo by default.

Service Mesh Control Plane 1.1

This release marks the end of support for Service Mesh Control Planes based on Service Mesh 1.1 for all platforms.

Istio 1.12 Support

Service Mesh 2.2 is based on Istio 1.12, which brings in new features and product enhancements. While many Istio 1.12 features are supported, the following unsupported features should be noted:

  • AuthPolicy Dry Run is a tech preview feature.

  • gRPC Proxyless Service Mesh is a tech preview feature.

  • Telemetry API is a tech preview feature.

  • Discovery selectors is not a supported feature.

  • External control plane is not a supported feature.

  • Gateway injection is not a supported feature.

Kubernetes Gateway API

Kubernetes Gateway API is a technology preview feature that is disabled by default.

To enable the feature, set the following environment variables for the Istiod container in ServiceMeshControlPlane:

spec:
  runtime:
    components:
      pilot:
        container:
          env:
            PILOT_ENABLE_GATEWAY_API: true
            PILOT_ENABLE_GATEWAY_API_STATUS: true
            # and optionally, for the deployment controller
            PILOT_ENABLE_GATEWAY_API_DEPLOYMENT_CONTROLLER: true

Restricting route attachment on Gateway API listeners is possible using the SameNamespace or All settings. Istio ignores usage of label selectors in listeners.allowedRoutes.namespaces and reverts to the default behavior (SameNamespace).

New features Red Hat OpenShift Service Mesh 2.1.5

This release of Red Hat OpenShift Service Mesh addresses Common Vulnerabilities and Exposures (CVEs), bug fixes, and is supported on OpenShift Container Platform 4.9 or later.

Component versions included in Red Hat OpenShift Service Mesh version 2.1.5

Component Version

Istio

1.9.9

Envoy Proxy

1.17.1

Jaeger

1.36

Kiali

1.36.12-1

New features Red Hat OpenShift Service Mesh 2.1.4

This release of Red Hat OpenShift Service Mesh addresses Common Vulnerabilities and Exposures (CVEs) and bug fixes.

Component versions included in Red Hat OpenShift Service Mesh version 2.1.4

Component Version

Istio

1.9.9

Envoy Proxy

1.17.1

Jaeger

1.30.2

Kiali

1.36.12-1

New features Red Hat OpenShift Service Mesh 2.1.3

This release of Red Hat OpenShift Service Mesh addresses Common Vulnerabilities and Exposures (CVEs) and bug fixes.

Component versions included in Red Hat OpenShift Service Mesh version 2.1.3

Component Version

Istio

1.9.9

Envoy Proxy

1.17.1

Jaeger

1.30.2

Kiali

1.36.10-2

New features Red Hat OpenShift Service Mesh 2.1.2.1

This release of Red Hat OpenShift Service Mesh addresses Common Vulnerabilities and Exposures (CVEs) and bug fixes.

Component versions included in Red Hat OpenShift Service Mesh version 2.1.2.1

Component Version

Istio

1.9.9

Envoy Proxy

1.17.1

Jaeger

1.30.2

Kiali

1.36.9

New features Red Hat OpenShift Service Mesh 2.1.2

This release of Red Hat OpenShift Service Mesh addresses Common Vulnerabilities and Exposures (CVEs) and bug fixes.

With this release, the Red Hat OpenShift distributed tracing platform Operator is now installed to the openshift-distributed-tracing namespace by default. Previously the default installation had been in the openshift-operator namespace.

Component versions included in Red Hat OpenShift Service Mesh version 2.1.2

Component Version

Istio

1.9.9

Envoy Proxy

1.17.1

Jaeger

1.30.1

Kiali

1.36.8

New features Red Hat OpenShift Service Mesh 2.1.1

This release of Red Hat OpenShift Service Mesh addresses Common Vulnerabilities and Exposures (CVEs) and bug fixes.

This release also adds the ability to disable the automatic creation of network policies.

Component versions included in Red Hat OpenShift Service Mesh version 2.1.1

Component Version

Istio

1.9.9

Envoy Proxy

1.17.1

Jaeger

1.24.1

Kiali

1.36.7

Disabling network policies

Red Hat OpenShift Service Mesh automatically creates and manages a number of NetworkPolicies resources in the Service Mesh control plane and application namespaces. This is to ensure that applications and the control plane can communicate with each other.

If you want to disable the automatic creation and management of NetworkPolicies resources, for example to enforce company security policies, you can do so. You can edit the ServiceMeshControlPlane to set the spec.security.manageNetworkPolicy setting to false

When you disable spec.security.manageNetworkPolicy Red Hat OpenShift Service Mesh will not create any NetworkPolicy objects. The system administrator is responsible for managing the network and fixing any issues this might cause.
Procedure

  1. In the OpenShift Container Platform web console, click Operators → Installed Operators.

  2. Select the project where you installed the Service Mesh control plane, for example istio-system, from the Project menu.

  3. Click the Red Hat OpenShift Service Mesh Operator. In the Istio Service Mesh Control Plane column, click the name of your ServiceMeshControlPlane, for example basic-install.

  4. On the Create ServiceMeshControlPlane Details page, click YAML to modify your configuration.

  5. Set the ServiceMeshControlPlane field spec.security.manageNetworkPolicy to false, as shown in this example.

    apiVersion: maistra.io/v2
kind: ServiceMeshControlPlane
spec:
  security:
      trust:
      manageNetworkPolicy: false

  6. Click Save.

New features and enhancements Red Hat OpenShift Service Mesh 2.1

This release of Red Hat OpenShift Service Mesh adds support for Istio 1.9.8, Envoy Proxy 1.17.1, Jaeger 1.24.1, and Kiali 1.36.5 on OpenShift Container Platform 4.6 EUS, 4.7, 4.8, 4.9, along with new features and enhancements.

Component versions included in Red Hat OpenShift Service Mesh version 2.1

Component Version

Istio

1.9.6

Envoy Proxy

1.17.1

Jaeger

1.24.1

Kiali

1.36.5

Service Mesh Federation

New Custom Resource Definitions (CRDs) have been added to support federating service meshes. Service meshes may be federated both within the same cluster or across different OpenShift clusters. These new resources include:

  • ServiceMeshPeer - Defines a federation with a separate service mesh, including gateway configuration, root trust certificate configuration, and status fields. In a pair of federated meshes, each mesh will define its own separate ServiceMeshPeer resource.

  • ExportedServiceMeshSet - Defines which services for a given ServiceMeshPeer are available for the peer mesh to import.

  • ImportedServiceSet - Defines which services for a given ServiceMeshPeer are imported from the peer mesh. These services must also be made available by the peer’s ExportedServiceMeshSet resource.

Service Mesh Federation is not supported between clusters on Red Hat OpenShift Service on AWS (ROSA), Azure Red Hat OpenShift (ARO), or OpenShift Dedicated (OSD).

OVN-Kubernetes Container Network Interface (CNI) generally available

The OVN-Kubernetes Container Network Interface (CNI) was previously introduced as a Technology Preview feature in Red Hat OpenShift Service Mesh 2.0.1 and is now generally available in Red Hat OpenShift Service Mesh 2.1 and 2.0.x for use on OpenShift Container Platform 4.7.32, OpenShift Container Platform 4.8.12, and OpenShift Container Platform 4.9.

Service Mesh WebAssembly (WASM) Extensions

The ServiceMeshExtensions Custom Resource Definition (CRD), first introduced in 2.0 as Technology Preview, is now generally available. You can use CRD to build your own plug-ins, but Red Hat does not provide support for the plug-ins you create.

Mixer has been completely removed in Service Mesh 2.1. Upgrading from a Service Mesh 2.0.x release to 2.1 will be blocked if Mixer is enabled. Mixer plug-ins will need to be ported to WebAssembly Extensions.

3scale WebAssembly Adapter (WASM)

With Mixer now officially removed, OpenShift Service Mesh 2.1 does not support the 3scale mixer adapter. Before upgrading to Service Mesh 2.1, remove the Mixer-based 3scale adapter and any additional Mixer plugins. Then, manually install and configure the new 3scale WebAssembly adapter with Service Mesh 2.1+ using a ServiceMeshExtension resource.

3scale 2.11 introduces an updated Service Mesh integration based on WebAssembly.

Istio 1.9 Support

Service Mesh 2.1 is based on Istio 1.9, which brings in a large number of new features and product enhancements. While the majority of Istio 1.9 features are supported, the following exceptions should be noted:

  • Virtual Machine integration is not yet supported

  • Kubernetes Gateway API is not yet supported

  • Remote fetch and load of WebAssembly HTTP filters are not yet supported

  • Custom CA Integration using the Kubernetes CSR API is not yet supported

  • Request Classification for monitoring traffic is a tech preview feature

  • Integration with external authorization systems via Authorization policy’s CUSTOM action is a tech preview feature

Improved Service Mesh operator performance

The amount of time Red Hat OpenShift Service Mesh uses to prune old resources at the end of every ServiceMeshControlPlane reconciliation has been reduced. This results in faster ServiceMeshControlPlane deployments, and allows changes applied to existing SMCPs to take effect more quickly.

Kiali updates

Kiali 1.36 includes the following features and enhancements:

  • Service Mesh troubleshooting functionality

    • Control plane and gateway monitoring

    • Proxy sync statuses

    • Envoy configuration views

    • Unified view showing Envoy proxy and application logs interleaved

  • Namespace and cluster boxing to support federated service mesh views

  • New validations, wizards, and distributed tracing enhancements

New features Red Hat OpenShift Service Mesh 2.0.11

This release of Red Hat OpenShift Service Mesh addresses Common Vulnerabilities and Exposures (CVEs), bug fixes, and is supported on OpenShift Container Platform 4.9 or later.

Component versions included in Red Hat OpenShift Service Mesh version 2.0.11

Component Version

Istio

1.6.14

Envoy Proxy

1.14.5

Jaeger

1.36

Kiali

1.24.16-1

New features Red Hat OpenShift Service Mesh 2.0.10

This release of Red Hat OpenShift Service Mesh addresses Common Vulnerabilities and Exposures (CVEs) and bug fixes.

Component versions included in Red Hat OpenShift Service Mesh version 2.0.10

Component Version

Istio

1.6.14

Envoy Proxy

1.14.5

Jaeger

1.28.0

Kiali

1.24.16-1

New features Red Hat OpenShift Service Mesh 2.0.9

This release of Red Hat OpenShift Service Mesh addresses Common Vulnerabilities and Exposures (CVEs) and bug fixes.

Component versions included in Red Hat OpenShift Service Mesh version 2.0.9

Component Version

Istio

1.6.14

Envoy Proxy

1.14.5

Jaeger

1.24.1

Kiali

1.24.11

New features Red Hat OpenShift Service Mesh 2.0.8

This release of Red Hat OpenShift Service Mesh addresses bug fixes.

New features Red Hat OpenShift Service Mesh 2.0.7.1

This release of Red Hat OpenShift Service Mesh addresses Common Vulnerabilities and Exposures (CVEs).

Change in how Red Hat OpenShift Service Mesh handles URI fragments

Red Hat OpenShift Service Mesh contains a remotely exploitable vulnerability, CVE-2021-39156, where an HTTP request with a fragment (a section in the end of a URI that begins with a # character) in the URI path could bypass the Istio URI path-based authorization policies. For instance, an Istio authorization policy denies requests sent to the URI path /user/profile. In the vulnerable versions, a request with URI path /user/profile#section1 bypasses the deny policy and routes to the backend (with the normalized URI path /user/profile%23section1), possibly leading to a security incident.

You are impacted by this vulnerability if you use authorization policies with DENY actions and operation.paths, or ALLOW actions and operation.notPaths.

With the mitigation, the fragment part of the request’s URI is removed before the authorization and routing. This prevents a request with a fragment in its URI from bypassing authorization policies which are based on the URI without the fragment part.

To opt-out from the new behavior in the mitigation, the fragment section in the URI will be kept. You can configure your ServiceMeshControlPlane to keep URI fragments.

Disabling the new behavior will normalize your paths as described above and is considered unsafe. Ensure that you have accommodated for this in any security policies before opting to keep URI fragments.
Example ServiceMeshControlPlane modification
apiVersion: maistra.io/v2
kind: ServiceMeshControlPlane
metadata:
  name: basic
spec:
  techPreview:
    meshConfig:
      defaultConfig:
        proxyMetadata: HTTP_STRIP_FRAGMENT_FROM_PATH_UNSAFE_IF_DISABLED: "false"

Required update for authorization policies

Istio generates hostnames for both the hostname itself and all matching ports. For instance, a virtual service or Gateway for a host of "httpbin.foo" generates a config matching "httpbin.foo and httpbin.foo:*". However, exact match authorization policies only match the exact string given for the hosts or notHosts fields.

Your cluster is impacted if you have AuthorizationPolicy resources using exact string comparison for the rule to determine hosts or notHosts.

You must update your authorization policy rules to use prefix match instead of exact match. For example, replacing hosts: ["httpbin.com"] with hosts: ["httpbin.com:*"] in the first AuthorizationPolicy example.

First example AuthorizationPolicy using prefix match
apiVersion: security.istio.io/v1beta1
kind: AuthorizationPolicy
metadata:
  name: httpbin
  namespace: foo
spec:
  action: DENY
  rules:
  - from:
    - source:
        namespaces: ["dev"]
    to:
    - operation:
        hosts: [“httpbin.com”,"httpbin.com:*"]
Second example AuthorizationPolicy using prefix match
apiVersion: security.istio.io/v1beta1
kind: AuthorizationPolicy
metadata:
  name: httpbin
  namespace: default
spec:
  action: DENY
  rules:
  - to:
    - operation:
        hosts: ["httpbin.example.com:*"]

New features Red Hat OpenShift Service Mesh 2.0.7

This release of Red Hat OpenShift Service Mesh addresses Common Vulnerabilities and Exposures (CVEs) and bug fixes.

Red Hat OpenShift Service Mesh on Red Hat OpenShift Dedicated and Microsoft Azure Red Hat OpenShift

Red Hat OpenShift Service Mesh is now supported through Red Hat OpenShift Dedicated and Microsoft Azure Red Hat OpenShift.

New features Red Hat OpenShift Service Mesh 2.0.6

This release of Red Hat OpenShift Service Mesh addresses Common Vulnerabilities and Exposures (CVEs) and bug fixes.

New features Red Hat OpenShift Service Mesh 2.0.5

This release of Red Hat OpenShift Service Mesh addresses Common Vulnerabilities and Exposures (CVEs) and bug fixes.

New features Red Hat OpenShift Service Mesh 2.0.4

This release of Red Hat OpenShift Service Mesh addresses Common Vulnerabilities and Exposures (CVEs) and bug fixes.

There are manual steps that must be completed to address CVE-2021-29492 and CVE-2021-31920.

Manual updates required by CVE-2021-29492 and CVE-2021-31920

Istio contains a remotely exploitable vulnerability where an HTTP request path with multiple slashes or escaped slash characters (%2F or %5C) could potentially bypass an Istio authorization policy when path-based authorization rules are used.

For example, assume an Istio cluster administrator defines an authorization DENY policy to reject the request at path /admin. A request sent to the URL path //admin will NOT be rejected by the authorization policy.

According to RFC 3986, the path //admin with multiple slashes should technically be treated as a different path from the /admin. However, some backend services choose to normalize the URL paths by merging multiple slashes into a single slash. This can result in a bypass of the authorization policy (//admin does not match /admin), and a user can access the resource at path /admin in the backend; this would represent a security incident.

Your cluster is impacted by this vulnerability if you have authorization policies using ALLOW action + notPaths field or DENY action + paths field patterns. These patterns are vulnerable to unexpected policy bypasses.

Your cluster is NOT impacted by this vulnerability if:

  • You don’t have authorization policies.

  • Your authorization policies don’t define paths or notPaths fields.

  • Your authorization policies use ALLOW action + paths field or DENY action + notPaths field patterns. These patterns could only cause unexpected rejection instead of policy bypasses. The upgrade is optional for these cases.

The Red Hat OpenShift Service Mesh configuration location for path normalization is different from the Istio configuration.

Updating the path normalization configuration

Istio authorization policies can be based on the URL paths in the HTTP request. Path normalization, also known as URI normalization, modifies and standardizes the incoming requests' paths so that the normalized paths can be processed in a standard way. Syntactically different paths may be equivalent after path normalization.

Istio supports the following normalization schemes on the request paths before evaluating against the authorization policies and routing the requests:

Table 1. Normalization schemes
Option Description Example Notes

NONE

No normalization is done. Anything received by Envoy will be forwarded exactly as-is to any backend service.

../%2Fa../b is evaluated by the authorization policies and sent to your service.

This setting is vulnerable to CVE-2021-31920.

BASE

This is currently the option used in the default installation of Istio. This applies the normalize_path option on Envoy proxies, which follows RFC 3986 with extra normalization to convert backslashes to forward slashes.

/a/../b is normalized to /b. \da is normalized to /da.

This setting is vulnerable to CVE-2021-31920.

MERGE_SLASHES

Slashes are merged after the BASE normalization.

/a//b is normalized to /a/b.

Update to this setting to mitigate CVE-2021-31920.

DECODE_AND_MERGE_SLASHES

The strictest setting when you allow all traffic by default. This setting is recommended, with the caveat that you must thoroughly test your authorization policies routes. Percent-encoded slash and backslash characters (%2F, %2f, %5C and %5c) are decoded to / or \, before the MERGE_SLASHES normalization.

/a%2fb is normalized to /a/b.

Update to this setting to mitigate CVE-2021-31920. This setting is more secure, but also has the potential to break applications. Test your applications before deploying to production.

The normalization algorithms are conducted in the following order:

  1. Percent-decode %2F, %2f, %5C and %5c.

  2. The RFC 3986 and other normalization implemented by the normalize_path option in Envoy.

  3. Merge slashes.

While these normalization options represent recommendations from HTTP standards and common industry practices, applications may interpret a URL in any way it chooses to. When using denial policies, ensure that you understand how your application behaves.

Path normalization configuration examples

Ensuring Envoy normalizes request paths to match your backend services' expectations is critical to the security of your system. The following examples can be used as a reference for you to configure your system. The normalized URL paths, or the original URL paths if NONE is selected, will be:

  1. Used to check against the authorization policies.

  2. Forwarded to the backend application.

Table 2. Configuration examples
If your application…​ Choose…​

Relies on the proxy to do normalization

BASE, MERGE_SLASHES or DECODE_AND_MERGE_SLASHES

Normalizes request paths based on RFC 3986 and does not merge slashes.

BASE

Normalizes request paths based on RFC 3986 and merges slashes, but does not decode percent-encoded slashes.

MERGE_SLASHES

Normalizes request paths based on RFC 3986, decodes percent-encoded slashes, and merges slashes.

DECODE_AND_MERGE_SLASHES

Processes request paths in a way that is incompatible with RFC 3986.

NONE

Configuring your SMCP for path normalization

To configure path normalization for Red Hat OpenShift Service Mesh, specify the following in your ServiceMeshControlPlane. Use the configuration examples to help determine the settings for your system.

SMCP v2 pathNormalization
spec:
  techPreview:
    global:
      pathNormalization: <option>

Configuring for case normalization

In some environments, it may be useful to have paths in authorization policies compared in a case insensitive manner. For example, treating https://myurl/get and https://myurl/GeT as equivalent. In those cases, you can use the EnvoyFilter shown below. This filter will change both the path used for comparison and the path presented to the application. In this example, istio-system is the name of the Service Mesh control plane project.

Save the EnvoyFilter to a file and run the following command:

$ oc create -f <myEnvoyFilterFile>
apiVersion: networking.istio.io/v1alpha3
kind: EnvoyFilter
metadata:
  name: ingress-case-insensitive
  namespace: istio-system
spec:
  configPatches:
  - applyTo: HTTP_FILTER
    match:
      context: GATEWAY
      listener:
        filterChain:
          filter:
            name: "envoy.filters.network.http_connection_manager"
            subFilter:
              name: "envoy.filters.http.router"
    patch:
      operation: INSERT_BEFORE
      value:
        name: envoy.lua
        typed_config:
            "@type": "type.googleapis.com/envoy.extensions.filters.http.lua.v3.Lua"
            inlineCode: |
              function envoy_on_request(request_handle)
                local path = request_handle:headers():get(":path")
                request_handle:headers():replace(":path", string.lower(path))
              end

New features Red Hat OpenShift Service Mesh 2.0.3

This release of Red Hat OpenShift Service Mesh addresses Common Vulnerabilities and Exposures (CVEs) and bug fixes.

In addition, this release has the following new features:

  • Added an option to the must-gather data collection tool that gathers information from a specified Service Mesh control plane namespace. For more information, see OSSM-351.

  • Improved performance for Service Mesh control planes with hundreds of namespaces