Chapter 2. Troubleshoot MCP gateway on-premise MCP servers

Procedure

1. If tools from your on-premise MCP server do not display in tools/list, check that the MCP server is properly discovered by MCP gateway components by running the following commands:

   • List your MCPServerRegistration CRs by running the following command:

     $ oc get mcpsr -A

   • Get the detailed status and configuration of the MCPServerRegistration CR that has missing tools by running the following command:

     $ oc describe mcpserverregistration <mcpsr_name> -n <mcpsr_namespace>

     • Replace <mcpsr_name> with the name of your MCP server.
     • Replace <mcpsr_namespace> with the namespace where your MCPServerRegistration CR is applied.

2. Verify that the MCPServerRegistration CR targetRef points to correct HTTPRoute name and namespace by running the following commands:

   1. Get the name and namespace of the HTTPRoute CR that the MCPServerRegistration CR is attempting to attach to by running the following command:

      $ oc get mcpserverregistration <mcpsr_name> -n <mcpsr_namespace> -o jsonpath='{.spec.targetRef.name}{"\n"}{.spec.targetRef.namespace}{"\n"}'

      • Replace <mcpsr_name> with the name of your MCP server.
      • Replace <mcpsr_namespace> with the namespace where your MCPServerRegistration CR is applied.
   2. If the MCPServerRegistration CR targets the HTTPRoute CR that is applied in the same namespace, the .spec.targetRef.namespace is not needed. No output is displayed.

   3. Cross-reference the HTTPRoute details from of the previous command with your HTTPRoute CRs by verifying that a route actually exists with those exact details by running the following command:

      $ oc get httproute <httproute_name> -n <httproute_namespace>

      • Replace <httproute_name> with the value from the output of the earlier step.
      • Replace <httproute_namespace> with the value from the output of the earlier step.
   4. If you get a NotFound server Error from the earlier step, correct your MCPServerRegistration CR.

3. Verify that the MCP gateway controller component successfully bound the MCPGatewayExtension and HTTPRoute CR together by checking the status with the following command:

   $ oc get mcpserverregistration <mcpsr_name> -n <mcpsr_namespace> -o jsonpath='{.status.conditions[?(@.type=="Accepted")]}'

   • Replace <mcpsr_name> with the name of your MCP server.

   • Replace <mcpsr_namespace> with the namespace where your MCPServerRegistration CR is applied.

     1. If you get a status: False output, the MCP gateway controller component found the route but could not use it. Check that you created a ReferenceGrant CR if the MCPGatewayExtension and HTTPRoute CRs are in different namespaces.

4. Check that the backend MCP server is running by entering the following command:

   $ oc get pods -n <mcpsr_namespace>

   Replace <mcpsr_namespace> with the namespace where your MCPServerRegistration CR is applied.

5. If your MCP server pods are crashing, check for the following conditions:

   • CrashLoopBackOff: This generally means that your MCP server is starting but then crashing. This problem can be caused by a missing environment variable or by permission issues.
   • Pending: This might mean that the pod has not started because of a resource issue in the cluster.
   • ErrImagePull: Either the cluster cannot reach the registry, or you do not have permission to pull the image. Check your OpenShift Container Platform settings and network configuration.
   • If your pod has a Running status with a high restart count, your MCP server might be unstable because of memory constraints or connectivity issues. Check your OpenShift Container Platform settings and network configuration.

6. Verify that the targeted application of your MCPServerRegistration CR actually exists and has a valid entry point by running the following command:

   $ oc get svc -n <namespace> <service_name>

   • Replace <namespace> with the namespace where your application runs.
   • Replace <service_name> with the name of your application.
7. Check that your application has a standard ClusterIP assigned to handle load balancing across pods.
8. Compare the ports to your MCPServerRegistration CR to make sure that they match and that either the TCP or SCTP protocol is used.

9. Check that the selectors on the Service CR for your application match the labels on your pods by running the following command:

   $ oc get endpoints <service_name> -n <namespace>

   • Replace <service_name> with the name of your application.
   • Replace <namespace> with the namespace where your application runs.

10. Check that the attached HTTPRoute CR has valid backend reference by running the following command:

    $ oc describe httproute <route_name>

Procedure

1. Check the MCP broker component router logs for errors by running the following command:

   $ oc logs -n <mcp_system> -l app.kubernetes.io/name=<mcp_gateway>

   • Replace <mcp_system> with the namespace of your MCP gateway deployment.
   • Replace <mcp_gateway> with the name of your MCP gateway deployment.
2. If you see errors, check for typos in the targetRef parameter value and ensure that you are pointing to an existing Service CR.
3. Verify that the backend MCP server is implementing the tools/list method correctly. Using the MCP Inspector is the easiest way to check.
4. Check your backend MCP server logs for errors.
5. Ensure that your backend MCP server is returning valid MCP protocol responses. Using the MCP Conformance Test Framework is the easiest way to catch protocol errors.
6. Verify that the prefix entry in the MCPServerRegistration CR is valid, meaning that there are no spaces or special characters.

7. Start a debug session based on your existing MCP server deployment to initiate a backend server test by running the following command:

   $ oc debug deployment/<mcpsr_name> -it

   Replace <mcpsr_name> with the name of your MCP server.

8. Test whether your backend MCP server is functional by running the following command:

   sh-4.4# curl -X POST https://localhost:<port>/mcp \
     -H "Content-Type: application/json" \
     -d '{"jsonrpc": "2.0", "id": 1, "method": "tools/list"}'

   If the command returns a list of tools, your backend is healthy.

9. If your backend is healthy but tools are missing in the Gateway CR, your MCPServerRegistration CR is not correctly mapping the service. Check the logs to see why the backend rejected the request by running the following command:

   $ oc logs -l app=<mcp_server_label>

   Replace <mcp_server_label> with the application defined in the metadata.labels.app: section of your MCP server Pod or Deployment CR.

Procedure

1. Check the MCPServerRegistration custom resource (CR) by running the following command:

   $ oc get mcpsr <mcpsr_name> -n <mcpsr_namespace> -o yaml | grep prefix

   • Replace <mcpsr_name> with the name of your MCP server.
   • Replace <mcpsr_namespace> with the namespace where your MCPServerRegistration CR is applied.
2. Verify that the prefix entry in the MCPServerRegistration CR is valid, meaning that there are no spaces or special characters.

3. Check the MCP gateway controller component logs for problems with your MCPServerRegistration CR by running the following command:

   $ oc logs -n <mcp_system> deployment/mcp-gateway-controller | grep prefix

   Replace <mcp_system> with the namespace of your MCP gateway deployment.

   1. If you see failed to generate prefix, then there is an error in your MCPServerRegistration CR metadata or there is a conflict.
   2. If there is no output at all, then either the controller is not detecting your MCPServerRegistration CR, or the request is failing before it gets to the routing stage.