Chapter 1. Troubleshoot the MCP gateway

Procedure

1. Check the general Gateway object configuration by running the following command:

   $ oc get gateway -A

   This command returns general information about all Gateway objects in the cluster. If the Gateway object you are troubleshooting exists, the command returns the following information:

   • The gatewayClassName is it using, whether or not it has an IP address or hostname assigned
   • The status, such as Ready, Programmed, or Pending

2. Check the full metadata and status history for a specific Gateway object that is stuck in Pending by running the following command:

   $ oc describe gateway <gateway_name> -n <gateway_system>

   • Replace <gateway_name> with the name of the Gateway object.
   • Replace <gateway_system> with the namespace where the Gateway object is applied.
3. Check for port conflicts and verify that the SSL/TLS certificates are correctly attached to Listener CRs.

4. Verify the Listener CR configuration by running the following command:

   $ oc get gateway <gateway_name> -n <gateway_system> -o yaml | grep -A 10 listeners

   • Replace <gateway_name> with the name of the Gateway object.
   • Replace <gateway_system> with the namespace where the Gateway object is applied.

5. Check all of your Listener CR configurations at the same time by running the following command:

   $ oc get gateway <gateway_name> -n <gateway_system> -o jsonpath='{range .spec.listeners[*]}{.name}{"\t"}{.hostname}{"\t"}{.port}{"\n"}{end}'

   • Replace <gateway_name> with the name of the Gateway object.
   • Replace <gateway_system> with the namespace where the Gateway object is applied.

6. Check that the Istio gateway pod is running by using the following command:

   $ oc get pods -n <gateway_system> -l gateway.istio.io/managed=istio.io-gateway-controller

   • Replace <gateway_system> with the name of your Gateway object deployment.
   • This command checks the status of Envoy-proxy pods and returns pod, traffic flow, and policy errors.

7. Verify that the port you are trying to use is not already in use by running the following command:

   $ oc get gateway -A -o yaml | grep "port:"

Procedure

1. Check the HTTPRoute general custom resource (CR) status by running the following command:

   $ oc get httproute -A

   • This command returns general information about all HTTPRoute objects in the cluster.
2. Check for the Accepted condition in the HTTPRoute CR status fields.

3. Check the full metadata and status history for one specific HTTPRoute object by running the following command:

   $ oc describe httproute <route_name> -n <namespace>

   • Replace <route_name> with the name of the HTTPRoute object.
   • Replace <namespace> with the namespace where the HTTPRoute object is applied.
4. Verify that the hostnames value in the HTTPRoute CR matches the gateway Listener CR hostname.
5. If the HTTPRoute status shows Accepted: False, then the Gateway object is not using the route.
6. If the condition is ResolvedRefs: False:, the route is accepted through the Gateway object, but it cannot find the backend MCP service. There might be either a mismatch in the CR metadata.name: field, or the MCP service is in a namespace the Gateway object cannot access.

7. Verify the parent reference by running the following command:

   $ oc get httproute <route_name> -n <namespace> -o yaml | grep -A 5 parentRefs

   • Replace <route_name> with the name of the HTTPRoute object.
   • Replace <namespace> with the namespace where the HTTPRoute object is applied.
8. Ensure that the retrieved parentRefs value matches your Gateway CR name and namespace exactly.

9. Check that the allowedRoutes.namespaces value in the Gateway CR allows the HTTPRoute namespace by running the following command:

   $ oc get gateway <gateway_name> -n <gateway_namespace> -o jsonpath='{range .spec.listeners[*]}{.name}{": "}{.allowedRoutes.namespaces.from}{"\n"}{end}'

   • Replace <gateway_name> with the name of the Gateway object.
   • Replace <gateway_namespace> with the namespace where the Gateway object is applied.

Procedure

1. Ensure that the Gateway object exists and is in the expected namespace by checking the general Gateway object configuration by running the following command:

   $ oc get gateway -A

   This command returns general information about all Gateway objects in the cluster. If the Gateway object you are troubleshooting exists, the command returns the gatewayClassName is it using, whether or not it has an IP address or hostname assigned, and a status, such as Ready, Programmed, or Pending.

2. Verify that the MCPGatewayExtension CR is Ready by running the following command:

   $ oc get mcpgatewayextension -A

3. Verify that a ReferenceGrant CR exists if the MCPGatewayExtension CR is in a different namespace than the Gateway object by running the following command:

   $ oc get referencegrant <referencegrant_name> -n <gateway_system> -o yaml

   • Replace <referencegrant_name> with the names of the ReferenceGrant CR.
   • Replace <gateway_system> with the namespace where the Gateway object is applied.

4. Check the HTTPRoute CR by running the following command:

   $ oc describe httproute <httproute_name> -n <httproute_namespace>

   • Replace <httproute_namespace> with the namespace where the HTTPRoute CR is applied.
   • Replace <httproute_name> with the names of the HTTPRoute CR.
   • If the HTTPRoute CR is not Accepted by the Gateway object, the route is not programmed into Envoy, causing a 404.
   • The conditions Status.Parents.Conditions: Accepted: True and Programmed: True show that the route is correct.

5. If any of the CRs you just checked are not Ready, check the controller logs for EnvoyFilter creation errors by running the following command:

   $ oc logs -n <mcp_system> deployment/mcp-gateway-controller

   • Replace <mcp_system> with the name of your MCP gateway deployment.
   • This step verifies that the MCP controller is successfully generating the underlying Istio configurations.

6. Check that the EnvoyFilter exists in the Gateway namespace by running the following command:

   $ oc get envoyfilter -n <gateway_namespace> -l app.kubernetes.io/managed-by=mcp-gateway-controller

   • Replace <gateway_namespace> with the namespace where the Gateway object is applied.

7. Verify the EnvoyFilter configuration by running the following command:

   $ oc describe envoyfilter -n <gateway_namespace> -l app.kubernetes.io/managed-by=mcp-gateway-controller

   • Replace <gateway_namespace> with the namespace where the Gateway object is applied.
   • The workloadSelector labels must match your Gateway pods, or your policies are bypassed.

8. Compare the EnvoyFilter labels against your pod labels by running the following command:

   $ oc get pods -n <gateway_namespace> --show-labels

   • Replace <gateway_namespace> with the namespace where the Gateway object is applied.

9. Identify the port that the Gateway object is configured to use by running the following command:

   $ oc get gateway <gateway_name> -n <gateway_namespace> -o jsonpath='{range .spec.listeners[*]}{.name}{": "}{.port}{"\n"}{end}'

   • Replace <gateway_name> with the name of the Gateway object.
   • Replace <gateway_namespace> with the namespace where the Gateway object is applied.

10. Use the following commands to verify the EnvoyFilter chain binding:

    1. Create a secure tunnel between a port on your computer and a port on the pod by running the following command:

       $ oc port-forward deploy/<gateway_deployment_name> -n <gateway_system> 15000:15000

       • Replace <gateway_deployment_name> with the name of the Deployment related to the Gateway object. Typically this value is either <<gateway_name>-istio or <gateway_name>-openshift-default, depending on the ingress you used.

       • Replace <gateway_system> with the namespace where the Gateway object is applied.

         1. From a different terminal window, access localhost:15000 by running the following command:

            $ curl -s localhost:15000/listeners

       • If this command returns empty, the EnvoyFilter CR is not active on this pod. Traffic is bypassing your policies.

11. Restart the Istio gateway to force a configuration reload by running the following command:

    $ oc rollout restart deployment/<gateway_name>-istio -n <gateway_namespace>

    • Replace <gateway_namespace> with the namespace where the Gateway object is applied.
    • Replace <gateway_name> with the names of the Gateway object.