Chapter 4. Use EvalHub with AI coding agents

Procedure

1. In your existing EvalHub custom resource (CR) file, such as evalhub_cr.yaml, add the following spec.mcp configuration:

   spec:
     mcp:
       enabled: true
       transport: http
       port: 3001

   • mcp.enabled defines whether to deploy the MCP server alongside the EvalHub server.
   • mcp.transport defines the transport mode. Set to http for remote MCP clients. Set to stdio for local clients that launch the server as a child process.
   • mcp.port defines the port for the MCP server when using http transport.

2. Apply the updated CR evalhub_cr.yaml file:

   $ oc apply -f evalhub_cr.yaml -n <namespace>

   The TrustyAI Operator applies the EvalHub CR and deploys the MCP server.

3. Create a route to expose the MCP server:

   $ oc expose service evalhub-mcp --port=3001 -n <namespace>

4. Retrieve the MCP server URL:

   $ export MCP_URL=https://$(oc get route evalhub-mcp -o jsonpath={.spec.host} -n <namespace>)

5. Register the MCP server with your MCP client. For example, to register the server with Claude Code, do the following:

   1. Create a token for the EvalHub service account:

      $ export EVALHUB_TOKEN="$(oc create token <service_account> \
          -n <namespace>)"

   2. Register the server with the MCP client:

      $ claude mcp add evalhub --transport http $MCP_URL \
        --header "Authorization: Bearer $EVALHUB_TOKEN" \
        --header "x-tenant: <namespace>"

      Replace <service_account> with a mounted pod token or long-lived token, such as ServiceAccount, that has EvalHub access. For more information about granting access, see Grant access to EvalHub.

      Note

      Tokens created by using oc create token expire. If the MCP client returns a 401 Unauthorized response, create a new token and update the client configuration.

Verification

1. Confirm that the EvalHub pod includes a ready MCP server container:

   $ oc get pods \
       -n <namespace> \
       -l app=eval-hub \
       -o jsonpath={range .items[*]}{.metadata.name}{"\t"}{range .status.containerStatuses[*]}{.name}={.ready}{" "}{end}{"\n"}{end}

2. From your MCP client, request the list of available evaluation providers. For example, in Claude Code, ask:

   List the available evaluation providers.

   The agent should return the registered providers from EvalHub.

Procedure

1. Set the required environment variables for your EvalHub instance:

   $ export EVALHUB_BASE_URL="https://evalhub.apps.<cluster_domain>"
   $ export EVALHUB_TOKEN="$(oc whoami -t)"
   $ export EVALHUB_TENANT="<namespace>"

   where:

   <cluster_domain>

   Specifies your OpenShift cluster domain.

   <namespace>

   Specifies your EvalHub tenant namespace.

   To skip TLS verification for clusters with self-signed certificates, also set EVALHUB_INSECURE=true.

2. Install the agent skills plugin by using one of the following methods:

   • To install by using the Claude Code plugin marketplace, run the following command:

     $ claude plugin install evalhub@evalhub

   • To install for local development, clone the repository and run the installation script:

     $ git clone https://github.com/eval-hub/eval-hub-skills
     $ cd eval-hub-skills
     $ make install-all

     This method symlinks all four skills into ~/.claude/skills/ so that changes to skill source files are reflected immediately without reinstallation.

3. Optional: If the EvalHub MCP server is also deployed on your cluster, register it alongside agent skills:

   $ claude mcp add evalhub "$EVALHUB_BASE_URL/mcp" \
     --transport http \
     --header "Authorization: Bearer $EVALHUB_TOKEN" \
     --header "x-tenant: $EVALHUB_TENANT"

   When both MCP and agent skills are available, the skills prefer MCP resources over Python scripts.

   Note

   OpenShift tokens expire. If you receive 401 Unauthorized errors, refresh your token by running export EVALHUB_TOKEN="$(oc whoami -t)" and repeat the registration command.

Procedure

1. Add an agent block to your provider YAML configuration file:

   agent:
     evaluates: [safety, security, red_teaming, toxicity]
     recommended_when:
       - "User asks about model safety or toxicity"
       - "Pre-deployment safety gate"
     target_type: model
     summary: "Red-team an LLM for safety vulnerabilities, toxicity, and OWASP risks"
     complements: [lm_evaluation_harness, guidellm]
     hints:
       - "The model endpoint must support OpenAI-compatible chat completions"
       - "The 'quick' benchmark runs a single DAN probe for fast smoke testing"
     result_interpretation:
       - "attack_success_rate measures how often the model was successfully exploited"
       - "LOWER is better -- 0.0 means no attacks succeeded"
       - "Scores above 0.3 indicate significant vulnerability"

   where:

   • evaluates defines semantic tags that describe what the provider measures. Agents match these tags against user intent.
   • recommended_when defines natural-language conditions under which an agent should suggest this provider.
   • target_type defines what the provider evaluates. Use model for LLM endpoints, agent for AI agents, or inference_server for server performance.
   • summary defines a concise, action-oriented description. Maximum 200 characters.
   • complements defines provider IDs that pair well for follow-up evaluations.
   • hints defines operational guidance for constructing job requests.
   • result_interpretation defines how to read results, including metric direction and baselines.

   For the full list of agent metadata fields, see Agent metadata fields reference.

2. To update agent metadata on a provider that is already registered, send a PATCH request to the provider endpoint:

   $ curl -X PATCH \
     -H "Authorization: Bearer $EVALHUB_TOKEN" \
     -H "X-Tenant: $EVALHUB_TENANT" \
     -H "Content-Type: application/json" \
     "$EVALHUB_BASE_URL/api/v1/evaluations/providers/<provider_id>" \
     -d [{"op": "replace", "path": "/agent", "value": { "evaluates": ["safety"], "target_type": "model", "summary": "Updated provider description for agents" }}]

   where <provider_id> specifies the ID of your custom provider.

Verification

1. Query the provider and confirm that agent metadata appears in the response:

   $ curl -s \
     -H "Authorization: Bearer $EVALHUB_TOKEN" \
     -H "X-Tenant: $EVALHUB_TENANT" \
     "$EVALHUB_BASE_URL/api/v1/evaluations/providers/<provider_id>" \
     | python3 -m json.tool

   The response includes an agent object with the metadata fields you defined.

2. Optional: If you have agent skills installed, verify that the provider appears in agent discovery results:

   $ uv run ~/.claude/skills/evalhub/scripts/evalhub_providers.py --agent 2>/dev/null

   The output lists all providers with agent metadata, including your custom provider.