> ## Documentation Index
> Fetch the complete documentation index at: https://docs.vijil.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Run Evaluations

> Start and inspect behavioral and Red Team evaluations with the CLI, MCP, REST API, or Python SDK.

Diamond supports two evaluation types. Both run against a registered [Agent](/tutorials/manage-agents) and use an `evaluation_id`, but they accept different settings and produce different evidence.

| Evaluation Type | Inputs | Output |
| - | - | - |
| Behavioral | Standard or custom Harnesses and an optional sample size | Trust Score and Harness results |
| Red Team | Optional wave and attack settings | Attack evidence and a Red Team report |

Red Team does not accept baseline, Harness, sample-size, or local-mode settings.

## Prerequisites

* An authenticated CLI or SDK client, or a bearer token for REST requests. See [SDK Setup](/developer-guide/sdk/setup).
* A registered Agent and its ID or alias.

## Start an Evaluation

<Tabs>
  <Tab title="CLI">
    Run the standard Trust Score Harnesses:

    ```bash theme={null}
    vijil evaluate "$AGENT_ID" --baseline --sample-size 50
    ```

    Run a Red Team evaluation:

    ```bash theme={null}
    vijil evaluate "$AGENT_ID" \
      --type redteam \
      --redteam '{"max_waves": 3}'
    ```

    The command waits for a terminal status. Add `--no-wait` to return after creation, or place `--output json` before `evaluate` for JSON output.
  </Tab>

  <Tab title="MCP">
    With the [Vijil MCP server](/developer-guide/agentic/quickstart) configured, ask Claude Code in natural language:

    <Prompt description="Run a safety and security evaluation on agent a1b2c3d4-... with sample size 50, and wait for it to complete">
      Run a safety and security evaluation on agent a1b2c3d4-... with sample size 50, and wait for it to complete
    </Prompt>

    Claude calls `eval_run` with `wait=True` and reports back Trust Scores when the evaluation finishes.
  </Tab>

  <Tab title="SDK">
    `client.evaluate()` creates an evaluation and polls until it reaches a terminal status:

    ```python theme={null}
    from vijil import Vijil

    client = Vijil()

    behavioral = client.evaluate("<agent-id>", baseline=True, sample_size=50)
    print(behavioral.trust_score)

    redteam = client.evaluate(
        "<agent-id>",
        type="redteam",
        redteam={"max_waves": 3},
    )
    print(redteam.evaluation_id, redteam.status)
    ```

    To start without polling, use the evaluation resource:

    ```python theme={null}
    behavioral = client.evaluations.create(
        agent_id="<agent-id>",
        baseline=True,
        sample_size=50,
    )
    redteam = client.evaluations.create(
        agent_id="<agent-id>",
        type="redteam",
        redteam={"max_waves": 3},
    )
    ```
  </Tab>

  <Tab title="REST API">
    Create a behavioral evaluation:

    ```bash theme={null}
    curl -s -X POST "$VIJIL_URL/v1/evaluations/" \
      -H "Authorization: Bearer $TOKEN" \
      -H "Content-Type: application/json" \
      -d "{
        \"agent_id\": \"$AGENT_ID\",
        \"team_id\": \"$TEAM_ID\",
        \"type\": \"behavioral\",
        \"harness_names\": [\"safety\", \"security\"],
        \"sample_size\": 50
      }"
    ```

    Create a Red Team evaluation through the same endpoint:

    ```bash theme={null}
    curl -s -X POST "$VIJIL_URL/v1/evaluations/" \
      -H "Authorization: Bearer $TOKEN" \
      -H "Content-Type: application/json" \
      -d "{
        \"agent_id\": \"$AGENT_ID\",
        \"team_id\": \"$TEAM_ID\",
        \"type\": \"redteam\",
        \"redteam\": {\"max_waves\": 3}
      }"
    ```

    Both requests return `202 Accepted` with an `evaluation_id` and initial status.
  </Tab>
</Tabs>

## Run Compliance Harnesses

Compliance Harnesses test an Agent against a regulatory or standards framework. They are standard Harnesses, so you start them like `safety` or `security`, by name:

| Framework | Harness name |
| - | - |
| EU AI Act | `eu_ai_act_agent_policy` |
| ISO/IEC 42001 | `iso42001_agent_policy` |
| NIST AI RMF | `nist_ai_rmf_agent_policy` |
| OSFI E-23 | `osfi_e23_agent_policy` |
| PIPEDA | `pipeda_agent_policy` |
| Taiwan AI Basic Act | `taiwan_ai_basic_act_agent_policy` |
| Singapore PDPA | `sg_pdpa_agent_policy` |

See [Compliance Harnesses](/owner-guide/simulate-environment/harnesses/compliance) for what each framework tests. A run that contains only Compliance Harnesses produces a Compliance Trust Report. Mixing them with `safety`, `security`, or `reliability` in one run does not, so start the Trust Score as a separate evaluation.

<Tabs>
  <Tab title="MCP">
    <Prompt description="Run the EU AI Act and ISO 42001 compliance Harnesses on agent a1b2c3d4-… with sample size 50, and wait for it to complete">
      Run the EU AI Act and ISO 42001 compliance Harnesses on agent a1b2c3d4-… with sample size 50, and wait for it to complete
    </Prompt>

    Claude calls `harness_list` to resolve the framework names, then `eval_run` with those `harness_names`.
  </Tab>

  <Tab title="SDK">
    Pass the names with `harness_type="standard"`. Without it, the SDK treats any name outside `safety`, `security`, and `reliability` as a custom Harness:

    ```python theme={null}
    evaluation = client.evaluations.create(
        agent_id="<agent-id>",
        harness_names=["eu_ai_act_agent_policy", "iso42001_agent_policy"],
        harness_type="standard",
        sample_size=50,
    )
    ```
  </Tab>

  <Tab title="REST API">
    ```bash theme={null}
    curl -s -X POST "$VIJIL_URL/v1/evaluations/" \
      -H "Authorization: Bearer $TOKEN" \
      -H "Content-Type: application/json" \
      -d "{
        \"agent_id\": \"$AGENT_ID\",
        \"team_id\": \"$TEAM_ID\",
        \"type\": \"behavioral\",
        \"harness_names\": [\"eu_ai_act_agent_policy\", \"iso42001_agent_policy\"],
        \"harness_type\": \"standard\",
        \"sample_size\": 50
      }"
    ```
  </Tab>
</Tabs>

<Note>
  `vijil evaluate` does not select Compliance Harnesses yet: `--baseline` runs only the Trust Score set, and `--bespoke` expects a custom Harness. Use MCP, the SDK, or the REST API instead.
</Note>

## Check Status

Behavioral evaluations can move through `starting`, `pending`, `running`, `completed`, and `saving` before reaching `saved`. They can also end as `succeeded`, `failed`, or `cancelled`.

<CodeGroup>
  ```bash title="CLI" theme={null}
  vijil evaluations show "$EVALUATION_ID"
  ```

  ```python title="SDK" theme={null}
  evaluation = client.evaluations.show("<evaluation-id>")
  print(evaluation.status)
  ```

  ```bash title="REST" theme={null}
  curl -s "$VIJIL_URL/v1/evaluations/$EVALUATION_ID" \
    -H "Authorization: Bearer $TOKEN"
  ```
</CodeGroup>

Red Team uses its own progress response and terminal statuses: `succeeded`, `failed`, `cancelled`, or `degraded`.

<CodeGroup>
  ```bash title="CLI" theme={null}
  vijil evaluations redteam-status "$EVALUATION_ID"
  ```

  ```python title="SDK" theme={null}
  status = client.evaluations.redteam_status("<evaluation-id>")
  print(status.phase, status.current_wave, status.status)
  ```

  ```bash title="REST" theme={null}
  curl -s "$VIJIL_URL/v1/evaluations/$EVALUATION_ID/redteam?team_id=$TEAM_ID" \
    -H "Authorization: Bearer $TOKEN"
  ```
</CodeGroup>

A `degraded` Red Team evaluation completed with some attacker failures. Inspect its error fields and wave breakdown before using its report.

## Retrieve Results

### Behavioral Results

Behavioral evaluations produce a Trust Score and a hierarchy of Harness, Scenario, and Probe results.

<CodeGroup>
  ```bash title="CLI" theme={null}
  vijil scores show "$AGENT_ID"
  ```

  ```python title="SDK" theme={null}
  score = client.scores.get("<evaluation-id>")
  results = client.scores.results("<evaluation-id>")
  print(score.trust_score, score.reliability, score.security, score.safety)
  ```

  ```bash title="REST" theme={null}
  curl -s "$VIJIL_URL/v1/evaluation-results/$EVALUATION_ID/results?team_id=$TEAM_ID" \
    -H "Authorization: Bearer $TOKEN"
  ```
</CodeGroup>

Use `client.scores.results()` or the corresponding REST endpoint when you need the complete nested result hierarchy. Use the response listing below when you need filterable, paginated prompt-level evidence.

### Prompt-Level Responses

Retrieve individual prompt and Agent response rows from a completed behavioral evaluation. The method supports both standard and custom Harness evaluations.

<CodeGroup>
  ```python title="SDK" theme={null}
  page = client.evaluations.responses(
      "<evaluation-id>",
      status="failed",
      limit=50,
      offset=0,
  )

  for response in page.items:
      print(response.prompt_text)
      print(response.response_text)
      print(response.probe_score)
  ```

  ```bash title="REST" theme={null}
  curl -sG "$VIJIL_URL/v1/evaluations/$EVALUATION_ID/responses" \
    -H "Authorization: Bearer $TOKEN" \
    --data-urlencode "team_id=$TEAM_ID" \
    --data-urlencode "status=failed" \
    --data-urlencode "limit=50" \
    --data-urlencode "offset=0"
  ```
</CodeGroup>

Each row identifies its parent Harness, Scenario, and Probe and includes the prompt, Agent response, parent Probe score, and raw Detector results. A `failed` row belongs to a Probe that scored below the evaluation's pass threshold. Filter by `harness_id`, `scenario_id`, or `probe_id` to narrow the results further.

The CLI does not currently expose this response listing. Red Team evaluations do not support it; use the Red Team evidence methods below instead. See the [`client.evaluations` SDK reference](/developer-guide/sdk/resources#prompt-level-behavioral-responses) for all arguments and the return type.

### Red Team Evidence

Red Team evaluations expose seeds, attacks, judgments, and reflections under the evaluation ID:

<CodeGroup>
  ```bash title="CLI" theme={null}
  vijil evaluations redteam-seeds "$EVALUATION_ID"
  vijil evaluations redteam-attacks "$EVALUATION_ID"
  vijil evaluations redteam-judgments "$EVALUATION_ID"
  vijil evaluations redteam-reflections "$EVALUATION_ID"
  ```

  ```python title="SDK" theme={null}
  seeds = client.evaluations.redteam_seeds("<evaluation-id>")
  attacks = client.evaluations.redteam_attacks("<evaluation-id>")
  judgments = client.evaluations.redteam_judgments("<evaluation-id>")
  reflections = client.evaluations.redteam_reflections("<evaluation-id>")
  ```

  ```bash title="REST" theme={null}
  curl -s "$VIJIL_URL/v1/evaluations/$EVALUATION_ID/redteam/attacks?team_id=$TEAM_ID" \
    -H "Authorization: Bearer $TOKEN"
  ```
</CodeGroup>

Use `--wave` to filter the CLI seed, attack, or judgment commands. The attack and judgment commands also accept `--seed` with `--wave`. See the [Red Team CLI guide](/developer-guide/cli/red-team) for the complete workflow.

## Retrieve Reports

Download the behavioral evaluation PDF:

<CodeGroup>
  ```bash title="CLI" theme={null}
  vijil reports download "$EVALUATION_ID"
  ```

  ```python title="SDK" theme={null}
  pdf_bytes = client.reports.download("<evaluation-id>")
  with open("report.pdf", "wb") as report_file:
      report_file.write(pdf_bytes)
  ```

  ```bash title="REST" theme={null}
  curl -s "$VIJIL_URL/v1/evaluations/$EVALUATION_ID/pdf?team_id=$TEAM_ID" \
    -H "Authorization: Bearer $TOKEN" \
    -o report.pdf
  ```
</CodeGroup>

Retrieve the Red Team report as structured JSON, Markdown, or PDF:

<CodeGroup>
  ```bash title="CLI" theme={null}
  vijil evaluations redteam-report "$EVALUATION_ID" --format md
  ```

  ```python title="SDK" theme={null}
  report = client.evaluations.redteam_report("<evaluation-id>")
  markdown = client.evaluations.redteam_report_markdown("<evaluation-id>")
  pdf_bytes = client.evaluations.redteam_report_pdf("<evaluation-id>")
  ```

  ```bash title="REST" theme={null}
  curl -s "$VIJIL_URL/v1/evaluations/$EVALUATION_ID/redteam/report?team_id=$TEAM_ID" \
    -H "Authorization: Bearer $TOKEN"
  ```
</CodeGroup>

## List and Manage Evaluations

<CodeGroup>
  ```bash title="CLI" theme={null}
  vijil evaluations list --agent "$AGENT_ID"
  ```

  ```python title="SDK" theme={null}
  page = client.evaluations.list(agent_id="<agent-id>")
  for evaluation in page.items:
      print(evaluation.id, evaluation.status)

  client.evaluations.delete("<evaluation-id>")
  ```

  ```bash title="REST" theme={null}
  curl -s "$VIJIL_URL/v1/evaluations/" \
    -H "Authorization: Bearer $TOKEN"

  curl -s -X POST "$VIJIL_URL/v1/evaluations/$EVALUATION_ID/cancel" \
    -H "Authorization: Bearer $TOKEN"

  curl -s -X DELETE "$VIJIL_URL/v1/evaluations/$EVALUATION_ID" \
    -H "Authorization: Bearer $TOKEN"
  ```
</CodeGroup>

The current CLI supports listing and showing evaluations. Use the SDK or REST API when you need to delete an evaluation, and use REST to cancel one.

## Next Steps

<CardGroup cols={2}>
  <Card title="Understand Results" icon="chart-bar" href="/developer-guide/evaluate/understanding-results">
    Interpret behavioral Trust Scores and Red Team evidence
  </Card>

  <Card title="Red Team CLI" icon="shield" href="/developer-guide/cli/red-team">
    Inspect waves, attacks, judgments, and reports
  </Card>

  <Card title="Custom Harnesses" icon="wrench" href="/developer-guide/evaluate/custom-harnesses">
    Create targeted behavioral evaluations
  </Card>

  <Card title="Local Agents" icon="laptop" href="/developer-guide/evaluate/local-agent">
    Run behavioral evaluations against local Agents
  </Card>
</CardGroup>


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.