Chapter 3. Evaluate your system by using the OpenShift AI dashboard

Procedure

1. In the OpenShift AI dashboard, click Develop & train Evaluations.
2. Optional: From the Project dropdown menu, select the project namespace with the model you deployed. This filters the evaluations list to show only evaluations from the selected project.

3. Click the Start evaluation run button and select the benchmarks to run:

   • Benchmark: Select an individual benchmark to evaluate a specific capability or task.

   • Benchmark suite: Select a predefined collection of related benchmarks to evaluate multiple aspects of model performance. Benchmark suites in the dashboard correspond to collections as described in EvalHub built-in collections.

     You can select multiple benchmarks or benchmark suites. Each selection appears as a tag below the field.

4. Configure the basic evaluation parameters:

   • Evaluation name: Enter a unique name for the evaluation job. By default, the name is the evaluation timestamp. This name identifies the job in the evaluations list and in MLflow experiments.
   • MLflow experiment: Select an existing MLflow experiment from the dropdown menu. Evaluation results are logged as runs within the selected experiment.
   • Source: Select the source to evaluate from the dropdown menu. You can choose a model, an agent, or pre-recorded responses as a source.
   • Enter additional parameters, such as the source name, Endpoint URL, or Dataset URL, depending on the selected source.

5. Optional: Change the evaluation threshold values by dragging the slider to select a value between 0 and 100. This value is equivalent to the 0.0-1.0 threshold used in the API.

   Alternatively, click the numeric input field to the right of the slider and type a value.

   Thresholds define pass or fail criteria for evaluation results. If you set a threshold, the evaluation results include a pass or fail status based on whether the overall score meets the specified value.

6. Click Benchmark parameters to expand the section and configure benchmark-specific parameters. For example, if you evaluate with the Basic science Q&A benchmark, add the following parameters:

     {
       "num_examples": 10,
       "limit": 5,
       "tokenizer": "google/flan-t5-small"
     }

7. Click Evaluate.

   The evaluation job is submitted and the page returns to the evaluations list.

Verification

1. On the Evaluations page, locate your evaluation job in the list.
2. Verify that the Status column shows Running while the job executes. The job progresses to Completed when all benchmarks finish successfully, or Failed if an error occurs.
3. When the job completes, click the evaluation name to view detailed results, including benchmark scores and pass or fail status if you configured a threshold.