SupportConsoleDevelopersStart a trialContact

Chapter 5. Generate and use evaluation cards

Generate standardized evaluation cards for evaluation runs in EvalHub to document what the evaluation tested, how it performed, and what infrastructure it used. EvalHub generates an evaluation card when the job request includes an MLflow experiment or OCI export configuration.

5.1. Evaluation cards overview
Copy link

An evaluation card is a structured JSON document that EvalHub generates when you specify an MLflow experiment or OCI export configuration in the evaluation job request.

Each evaluation card captures the full context of an evaluation: what model was tested, which benchmarks and datasets the evaluation used, and the results and pass or fail outcomes. EvalHub generates one evaluation card per job, regardless of whether the job evaluates a collection or a list of individual benchmarks.

Evaluation cards address three challenges in AI evaluation:

Reproducibility
Evaluation results are difficult to reproduce without a record of the exact configuration, infrastructure, and framework versions used. Evaluation cards capture this context so that teams can repeat evaluations with confidence.
Accessibility
Without a standardized format, evaluation documentation scatters across MLflow logs, code comments, and informal records. Evaluation cards consolidate this information into a single, discoverable artifact.
Governance
Regulatory frameworks such as the EU AI Act and FedRAMP require auditable records of AI system evaluations. Evaluation cards provide a versioned, schema-validated record suitable for compliance workflows.

5.1.1. When EvalHub generates evaluation cards
Copy link

EvalHub generates an evaluation card as a post-processing step when the evaluation job includes either of the following configurations:

  • MLflow experiment configuration.
  • OCI export configuration.

No evaluation card is generated if neither field is specified.

Each generated card includes metadata, evaluation context, and per-benchmark results with thresholds.

5.2. Configure evaluation card generation
Copy link

Enable evaluation card generation for an evaluation job so you can get a comprehensive record of the evaluation for compliance, auditing, and reproducibility. EvalHub generates the evaluation card as a post-processing step after it commits the evaluation results.

You can enable evaluation card generation one of the following ways: 1. By specifying an MLflow experiment 2. By setting an OCI export configuration in the job request

If neither an MLflow experiment nor an OCI export is configured, EvalHub does not generate an evaluation card.

Prerequisites

  • You have a running EvalHub instance. For details, see Deploy EvalHub with the TrustyAI Operator.
  • You have a running MLflow instance accessible from the EvalHub deployment, or access to an OCI-compatible container registry.

Procedure

  1. To generate an evaluation card by configuring an MLflow experiment, include an experiment block in the job submission:

    • To use the REST API, run: 

      "benchmarks": [
  {
    "provider_id": "lm_evaluation_harness",
    "benchmark_id": "mmlu"
  }
],
"experiment": {
  "name": "my-model-v2-eval"
}

      For the full job submission request, see Submit an evaluation job.

    • To use the CLI, include the experiment field in your YAML config file: 

      experiment:
  name: my-model-v2-eval
       
      $ evalhub eval run --config eval-with-mlflow.yaml

    • To use the Python SDK, pass an ExperimentConfig to the JobSubmissionRequest: 

      from evalhub.models import ExperimentConfig

experiment = ExperimentConfig(name="my-model-v2-eval")

  2. To generate an evaluation card by configuring an OCI export, include an exports block in the job submission: 

    "benchmarks": [
  {
    "provider_id": "lm_evaluation_harness",
    "benchmark_id": "mmlu"
  }
],
"exports": {
  "oci": {
    "coordinates": {
      "oci_host": "quay.io",
      "oci_repository": "my-org/eval-results"
    },
    "k8s": {
      "connection": "oci-registry-credentials"
    }
  }
}

    For details on OCI export configuration, see Export evaluation results to an OCI registry.

Verification

  1. After the job completes, verify that EvalHub generated the evaluation card: 

    $ curl -s -H "Authorization: Bearer $TOKEN" -H "X-Tenant: <namespace>" \
    $EVALHUB_URL/api/v1/evaluations/jobs/<job_id> | jq .results.evalcard

    The response includes the evaluation card metadata and the artifact path.

    Note

    Evaluation card generation is a post-processing step that runs after EvalHub commits evaluation results to MLflow. If card generation fails, the evaluation results are still available. EvalHub logs card generation errors but does not fail the evaluation job.

5.3. Retrieve and interpret evaluation cards
Copy link

After an evaluation completes, retrieve the evaluation card to review results, share findings with stakeholders, or pass the card into audit tools, CI pipelines, or dashboards.

Depending on the job request configuration, EvalHub stores evaluation cards as MLflow run artifacts, OCI artifacts, or both. You can access them through the MLflow UI, an OCI registry, or programmatically.

Prerequisites

  • You have a completed evaluation job with an evaluation card generated. For details, see Configure evaluation card generation.
  • You have access to the MLflow instance configured for EvalHub.

Procedure

  1. Identify the MLflow run ID for your evaluation job: 

    $ curl -s -H "Authorization: Bearer $TOKEN" -H "X-Tenant: <namespace>" \
    $EVALHUB_URL/api/v1/evaluations/jobs/<job_id> | jq .results.mlflow_run_id

  2. Retrieve the evaluation card by using one of the following methods:

    • To use the MLflow UI, navigate to the experiment run and open the Artifacts tab. EvalHub stores the evaluation card as evalcard.json in the run’s artifact root.

    • To use the MLflow Python client, run: 

      import json
import mlflow

local_path = mlflow.artifacts.download_artifacts(
    run_id="<mlflow_run_id>",
    artifact_path="evalcard.json"
)

with open(local_path) as f:
    evalcard = json.load(f)

print(f"Model: {evalcard['evaluation_context']['model']['name']}")
print(f"Schema version: {evalcard['schema_version']}")

    • To use the CLI, run: 

      $ evalhub eval evalcard <job_id> --format json

      The --format flag supports json and table.

Verification

  1. Review the download evaluation card contents.

  2. Confirm that the downloaded card validates against the evaluation card schema by checking the schema_version field and verifying that all required sections are present: 

    required_sections = ["metadata", "evaluation_context", "results"]
for section in required_sections:
    assert section in evalcard, f"Missing section: {section}"
print("Evaluation card structure is valid.")

5.4. Evaluation card schema reference
Copy link

An evaluation card conforms to a versioned JSON schema that EvalHub validates before storing.

5.4.1. Top-level fields
Copy link

Every evaluation card contains the following top-level fields.

Table 5.1. Top-level evaluation card fields
FieldTypeRequiredDescription

card_version

String

Yes

The version of this card instance, for example 1.0.0.

schema_version

String

Yes

The version of the evaluation card schema that this card conforms to, for example 1.0.0.

metadata

Object

Yes

Card metadata including timestamps and generator identity. See Section 5.4.2, “Metadata fields”.

evaluation_context

Object

Yes

The evaluation configuration including model, datasets, tasks, and framework. See Section 5.4.3, “Evaluation context fields”.

results

Object

Yes

Benchmark metrics, scores, and pass/fail outcomes. See Section 5.4.4, “Results fields”.

5.4.2. Metadata fields
Copy link

The metadata section records when and how EvalHub generated the card.

Table 5.2. Metadata fields
FieldTypeRequiredDescription

generated_at

String (ISO 8601)

Yes

Timestamp when the card was generated.

generator

String

Yes

The identity and version of the generator, for example eval-hub/1.5.0.

authors

Array of strings

No

Identifiers of the users or service accounts that submitted the evaluation job.

tags

Array of strings

No

User-defined tags for card classification and filtering.

5.4.3. Evaluation context fields
Copy link

The evaluation_context section describes what was evaluated and how the evaluation was configured.

Table 5.3. Evaluation context fields
FieldTypeRequiredDescription

model

Object

Yes

The model that was evaluated. Contains name (string) and url (string).

datasets

Array of objects

No

The datasets used in the evaluation. Each entry contains name (string) and source (string).

tasks

Array of objects

No

Descriptions of the evaluation tasks. Each entry contains name (string), description (string), and provider_id (string).

framework

Object

Yes

The evaluation framework metadata. Contains name (string), version (string), and configuration (object, optional).

collection

Object

No

The collection used, if the evaluation was a collection run. Contains name (string) and version (string).

5.4.4. Results fields
Copy link

The results section captures benchmark scores and overall pass or fail outcomes.

Table 5.4. Results fields
FieldTypeRequiredDescription

benchmarks

Array of objects

Yes

Per-benchmark results. See Section 5.4.5, “Benchmark result fields”.

overall_pass

Boolean

No

Whether the evaluation passed overall, based on collection-level or job-level thresholds.

overall_score

Number

No

The weighted average score across all benchmarks.

summary_statistics

Object

No

Aggregated statistics for collection evaluations, including total_benchmarks (integer), passed (integer), and failed (integer).

5.4.5. Benchmark result fields
Copy link

The benchmarks array describes the results for each benchmark in the evaluation job.

Table 5.5. Benchmark result fields
FieldTypeRequiredDescription

benchmark_id

String

Yes

The benchmark identifier, for example mmlu.

provider_id

String

Yes

The provider that ran the benchmark, for example lm_evaluation_harness.

metrics

Object

Yes

Key-value pairs of metric names and their scores, for example {"acc": 0.65, "acc_norm": 0.68}.

primary_score

Number

Yes

The primary score used for pass/fail evaluation.

primary_metric

String

Yes

The name of the primary metric, for example acc_norm.

threshold

Number

No

The pass/fail threshold applied to this benchmark.

pass

Boolean

No

Whether the benchmark passed based on the threshold and lower_is_better setting.

lower_is_better

Boolean

No

If true, the benchmark passes when the score is less than or equal to the threshold. Defaults to false.

weight

Number

No

The weight of this benchmark in the overall score calculation.

5.4.6. Evaluation card example
Copy link

The following example shows a complete evaluation card for a collection run with two benchmarks. 

{
  "card_version": "1.0.0",
  "schema_version": "1.0.0",
  "metadata": {
    "generated_at": "2026-07-15T14:30:00Z",
    "generator": "eval-hub/1.5.0",
    "authors": [
      "system:serviceaccount:my-namespace:eval-sa"
    ],
    "tags": [
      "production",
      "quarterly-review"
    ]
  },
  "evaluation_context": {
    "model": {
      "name": "my-model",
      "url": "http://my-model.my-namespace.svc.cluster.local:8080/v1"
    },
    "datasets": [
      { "name": "mmlu", "source": "built-in" },
      { "name": "hellaswag", "source": "built-in" }
    ],
    "tasks": [
      { "name": "mmlu", "description": "Multitask language understanding", "provider_id": "lm_evaluation_harness" },
      { "name": "hellaswag", "description": "Commonsense reasoning", "provider_id": "lm_evaluation_harness" }
    ],
    "framework": {
      "name": "lm_evaluation_harness",
      "version": "0.4.8",
      "configuration": {
        "num_fewshot": 5,
        "batch_size": 16
      }
    },
    "collection": {
      "name": "leaderboard-v2",
      "version": "1.0.0"
    }
  },
  "results": {
    "benchmarks": [
      {
        "benchmark_id": "mmlu",
        "provider_id": "lm_evaluation_harness",
        "metrics": { "acc": 0.65, "acc_norm": 0.68 },
        "primary_score": 0.68,
        "primary_metric": "acc_norm",
        "threshold": 0.60,
        "pass": true,
        "lower_is_better": false,
        "weight": 1
      },
      {
        "benchmark_id": "hellaswag",
        "provider_id": "lm_evaluation_harness",
        "metrics": { "acc": 0.72, "acc_norm": 0.75 },
        "primary_score": 0.75,
        "primary_metric": "acc_norm",
        "threshold": 0.70,
        "pass": true,
        "lower_is_better": false,
        "weight": 1
      }
    ],
    "overall_pass": true,
    "overall_score": 0.715,
    "summary_statistics": {
      "total_benchmarks": 2,
      "passed": 2,
      "failed": 0
    }
  }
}
Red Hat logoGithubRedditYoutubeTwitter

Learn

Try, buy, & sell

Communities

About Red Hat

We deliver hardened solutions that make it easier for enterprises to work across platforms and environments, from the core datacenter to the network edge.

Making open source more inclusive

Red Hat is committed to replacing problematic language in our code, documentation, and web properties. For more details, see the Red Hat Blog.

About Red Hat Documentation

We help Red Hat users innovate and achieve their goals with our products and services with content they can trust. Explore our recent updates.

Legal Notice

Theme

© 2026 Red HatBack to top