×

You can configure a cluster-wide proxy during cluster installation or after the cluster has been installed.

If you use a cluster-wide proxy, you are responsible for the following:

  • Maintaining the availability of the proxy to the cluster.

  • Understanding that if the proxy becomes unavailable, then it may impact the health and supportability of the cluster.

Cluster-wide proxy is a functionally-complete feature and suitable for production workloads. There are additional considerations that need to be added to documentation, and until then, this feature is considered a Technology Preview.

Prerequisites for configuring a cluster-wide proxy

To configure a cluster-wide proxy, you must meet the following requirements. These requirements are valid for both fresh installation and post installation proxy configuration.

General requirements

  • You are the cluster owner.

  • Your account has sufficient privileges.

  • You have added the ec2.<region>.amazonaws.com, elasticloadbalancing.<region>.amazonaws.com, and s3.<region>.amazonaws.com endpoints to your virtual private cloud (VPC) endpoint. These endpoints are required to complete requests from the nodes to the AWS EC2 API. Because the proxy works on the container level, not the node level, you must route these requests to the AWS EC2 API through the AWS private network. Adding the public IP address of the EC2 API to your allowlist in your proxy server is not sufficient.

  • You must have a Customer Cloud Subscription (CCS) cluster with a VPC that the proxy can access.

  • You have the ocm CLI installed and configured.

Network requirements

  • If your proxy re-encyrpts egress traffic, you must create exclusions to the domain and port combinations. The following table offers guidance into these exceptions.

    • Allowlist the following OpenShift URLs for re-encryption.

      Address Protocol/Port Function

      observatorium-mst.api.openshift.com

      https/443

      Required. Used for Managed OpenShift-specific telemetry.

      sso.redhat.com

      https/443

      The https://cloud.redhat.com/openshift site uses authentication from sso.redhat.com to download the cluster pull secret and use Red Hat SaaS solutions to facilitate monitoring of your subscriptions, cluster inventory, and chargeback reporting.

    • Allowlist the following site reliability engineering (SRE) and management URLs for re-encryption.

      Address Protocol/Port Function

      *.osdsecuritylogs.splunkcloud.com

      OR

      inputs1.osdsecuritylogs.splunkcloud.com inputs2.osdsecuritylogs.splunkcloud.com inputs4.osdsecuritylogs.splunkcloud.com inputs5.osdsecuritylogs.splunkcloud.com inputs6.osdsecuritylogs.splunkcloud.com inputs7.osdsecuritylogs.splunkcloud.com inputs8.osdsecuritylogs.splunkcloud.com inputs9.osdsecuritylogs.splunkcloud.com inputs10.osdsecuritylogs.splunkcloud.com inputs11.osdsecuritylogs.splunkcloud.com inputs12.osdsecuritylogs.splunkcloud.com inputs13.osdsecuritylogs.splunkcloud.com inputs14.osdsecuritylogs.splunkcloud.com inputs15.osdsecuritylogs.splunkcloud.com

      tcp/9997

      Used by the splunk-forwarder-operator as a log forwarding endpoint to be used by Red Hat SRE for log-based alerting.

      http-inputs-osdsecuritylogs.splunkcloud.com

      https/443

      Used by the splunk-forwarder-operator as a log forwarding endpoint to be used by Red Hat SRE for log-based alerting.

Additional Resources

For more information, see Getting started with OpenShift Dedicated for a basic cluster installation workflow.

The use of a proxy server to perform TLS re-encryption is currently not supported if the server is acting as a transparent forward proxy where it is not configured on-cluster via the --http-proxy or --https-proxy arguments.

A transparent forward proxy intercepts the cluster’s traffic, but it is not actually configured on the cluster itself.

Configuring a cluster-wide proxy during installation

You can add a proxy during cluster installation. Prior to installation, however, you should verify that the proxy is accessible from the intended cluster virtual private cloud (VPC) and its private subnets.

Only cluster system egress traffic is proxied, including calls to the AWS API. A system-wide proxy does not affect user workloads. It only affects system components.

Procedure
  • To create a cluster with a proxy, run the following command:

    $ ocm create cluster \
     <other_arguments_here> \
     --additional-trust-bundle-file <path_to_CA_bundle_file> \   (1) (2) (3)
     --http-proxy http://<username>:<pswd>@<ip>:<port> \  (1) (4)
     --https-proxy http(s)://<username>:<pswd>@<ip>:<port> (4)
    1 The http-proxy, https-proxy, and additional-trust-bundle-file arguments are all optional.
    2 If you use the additional-trust-bundle-file option without an http(s)-proxy argument, the passed additional trust bundle is set on the cluster, but it is not configured to be used with the proxy.
    3 The additional-trust-bundle-file argument is a file path pointing to a bundle of PEM-encoded X.509 certificates, which are all concatenated together. The additionalTrustBundle parameter is required unless the proxy’s identity certificate is signed by an authority from the RHCOS trust bundle. If you use an MITM transparent proxy network that does not require additional proxy configuration but requires additional CAs, you must provide the MITM CA certificate.
    4 The http-proxy and https-proxy arguments must point to a valid URL.

Configuring or updating your cluster-wide proxy after installation

As the cluster owner, you may wish to add a proxy to your created cluster after installation, or you may wish to make changes to your proxy that you configured during cluster installation. The ocm CLI provides some options for adding a proxy to your cluster or modifying an existing proxy on your cluster.

You may need to perform these actions if:

  • the cluster-wide proxy is configured after installation,

  • the proxy’s network address needs to be updated, and/or

  • any of the proxy’s certificate authorities have expired and the additional trust bundle needs to be replaced.

The cluster applies the configuration to the cluster’s control plane and worker nodes. This process results in each node in the cluster temporarily being placed into an unschedulable state and drained of its workloads while applying the configuration. Each node will be restarted as part of this process.

Procedure
  • To edit a cluster, run the following command:

    $ ocm edit cluster \
     --cluster $CLUSTER_NAME \
     --additional-trust-bundle-file $CA_BUNDLE_FILE \
     --http-proxy $HTTP_PROXY \
     --https-proxy $HTTPS_PROXY

    While the additional-trust-bundle-file, http-proxy, and https-proxy arguments are optional, if you set a additional-trust-bundle-file without either an http-proxy or https-proxy argument, then the additional trust bundle will still be used for verifying cluster system egress traffic.

  • You can verify that the proxy and certificate authority configuration updates have been successfully applied to your cluster by:

    • All of the MachineConfigPools are updated. Run the following command to see their status:

      $ oc get machineconfigpools
      Sample Output
      NAME     CONFIG                                             UPDATED   UPDATING   DEGRADED   MACHINECOUNT   READYMACHINECOUNT   UPDATEDMACHINECOUNT   DEGRADEDMACHINECOUNT   AGE
      master   rendered-master-d9a03f612a432095dcde6dcf44597d90   True      False      False      3              3                   3                     0                      31h
      worker   rendered-worker-f6827a4efe21e155c25c21b43c46f65e   True      False      False      6              6                   6                     0                      31h
    • As the cluster owner, the following command displays the proxy status:

      $ oc get proxy cluster -o yaml
      Sample Output
      apiVersion: config.openshift.io/v1
      kind: Proxy
      spec:
        httpProxy: http://proxy.host.domain:<port>
        httpsProxy: https://proxy.host.domain:<port>
        <...more...>
      status:
        httpProxy: http://proxy.host.domain:<port>
        httpsProxy: https://proxy.host.domain:<port>
        <...more...>

You should not attempt to change the proxy or additional trust bundle configuration on the cluster itself. These changes should always be done via the ocm command-line tools. Any changes that are made directly to the cluster will be reverted automatically.

Responsibilites for additional trust bundles

If you supplied an additional trust bundle file, you are responsible for the following:

  • Ensuring that the contents of the additional trust bundle are valid,

  • Ensuring that the certificates, including intermediary certificates, contained in the additional trust bundle have not expired, and

  • Tracking the expiry and performing any necessary renewals for certificates contained in the additional trust bundle, and subsequently updating the cluster’s configuration with the updated additional trust bundle.