Run Evaluations

Diamond supports two evaluation types. Both run against a registered Agent and use an evaluation_id, but they accept different settings and produce different evidence.

Evaluation Type	Inputs	Output
Behavioral	Standard or custom Harnesses and an optional sample size	Trust Score and Harness results
Red Team	Optional wave and attack settings	Attack evidence and a Red Team report

Red Team does not accept baseline, Harness, sample-size, or local-mode settings.

Prerequisites

An authenticated CLI or SDK client, or a bearer token for REST requests. See SDK Setup.
A registered Agent and its ID or alias.

Start an Evaluation

CLI
MCP
SDK
REST API

Run the standard Trust Score Harnesses:

vijil evaluate "$AGENT_ID" --baseline --sample-size 50

Run a Red Team evaluation:

vijil evaluate "$AGENT_ID" \
  --type redteam \
  --redteam '{"max_waves": 3}'

The command waits for a terminal status. Add --no-wait to return after creation, or place --output json before evaluate for JSON output.

With the Vijil MCP server configured, ask Claude Code in natural language:

Run a safety and security evaluation on agent a1b2c3d4-... with sample size 50, and wait for it to complete

Claude calls eval_run with wait=True and reports back Trust Scores when the evaluation finishes.

client.evaluate() creates an evaluation and polls until it reaches a terminal status:

from vijil import Vijil

client = Vijil()

behavioral = client.evaluate("<agent-id>", baseline=True, sample_size=50)
print(behavioral.trust_score)

redteam = client.evaluate(
    "<agent-id>",
    type="redteam",
    redteam={"max_waves": 3},
)
print(redteam.evaluation_id, redteam.status)

To start without polling, use the evaluation resource:

behavioral = client.evaluations.create(
    agent_id="<agent-id>",
    baseline=True,
    sample_size=50,
)
redteam = client.evaluations.create(
    agent_id="<agent-id>",
    type="redteam",
    redteam={"max_waves": 3},
)

Create a behavioral evaluation:

curl -s -X POST "$VIJIL_URL/v1/evaluations/" \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d "{
    \"agent_id\": \"$AGENT_ID\",
    \"team_id\": \"$TEAM_ID\",
    \"type\": \"behavioral\",
    \"harness_names\": [\"safety\", \"security\"],
    \"sample_size\": 50
  }"

Create a Red Team evaluation through the same endpoint:

curl -s -X POST "$VIJIL_URL/v1/evaluations/" \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d "{
    \"agent_id\": \"$AGENT_ID\",
    \"team_id\": \"$TEAM_ID\",
    \"type\": \"redteam\",
    \"redteam\": {\"max_waves\": 3}
  }"

Both requests return 202 Accepted with an evaluation_id and initial status.

Run Compliance Harnesses

Compliance Harnesses test an Agent against a regulatory or standards framework. They are standard Harnesses, so you start them like safety or security, by name:

Framework	Harness name
EU AI Act	`eu_ai_act_agent_policy`
ISO/IEC 42001	`iso42001_agent_policy`
NIST AI RMF	`nist_ai_rmf_agent_policy`
OSFI E-23	`osfi_e23_agent_policy`
PIPEDA	`pipeda_agent_policy`
Taiwan AI Basic Act	`taiwan_ai_basic_act_agent_policy`
Singapore PDPA	`sg_pdpa_agent_policy`

See Compliance Harnesses for what each framework tests. A run that contains only Compliance Harnesses produces a Compliance Trust Report. Mixing them with safety, security, or reliability in one run does not, so start the Trust Score as a separate evaluation.

MCP
SDK
REST API

Run the EU AI Act and ISO 42001 compliance Harnesses on agent a1b2c3d4-… with sample size 50, and wait for it to complete

Claude calls harness_list to resolve the framework names, then eval_run with those harness_names.

Pass the names with harness_type="standard". Without it, the SDK treats any name outside safety, security, and reliability as a custom Harness:

evaluation = client.evaluations.create(
    agent_id="<agent-id>",
    harness_names=["eu_ai_act_agent_policy", "iso42001_agent_policy"],
    harness_type="standard",
    sample_size=50,
)

curl -s -X POST "$VIJIL_URL/v1/evaluations/" \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d "{
    \"agent_id\": \"$AGENT_ID\",
    \"team_id\": \"$TEAM_ID\",
    \"type\": \"behavioral\",
    \"harness_names\": [\"eu_ai_act_agent_policy\", \"iso42001_agent_policy\"],
    \"harness_type\": \"standard\",
    \"sample_size\": 50
  }"

vijil evaluate does not select Compliance Harnesses yet: --baseline runs only the Trust Score set, and --bespoke expects a custom Harness. Use MCP, the SDK, or the REST API instead.

Check Status

Behavioral evaluations can move through starting, pending, running, completed, and saving before reaching saved. They can also end as succeeded, failed, or cancelled.

vijil evaluations show "$EVALUATION_ID"

evaluation = client.evaluations.show("<evaluation-id>")
print(evaluation.status)

curl -s "$VIJIL_URL/v1/evaluations/$EVALUATION_ID" \
  -H "Authorization: Bearer $TOKEN"

Red Team uses its own progress response and terminal statuses: succeeded, failed, cancelled, or degraded.

vijil evaluations redteam-status "$EVALUATION_ID"

status = client.evaluations.redteam_status("<evaluation-id>")
print(status.phase, status.current_wave, status.status)

curl -s "$VIJIL_URL/v1/evaluations/$EVALUATION_ID/redteam?team_id=$TEAM_ID" \
  -H "Authorization: Bearer $TOKEN"

A degraded Red Team evaluation completed with some attacker failures. Inspect its error fields and wave breakdown before using its report.

Retrieve Results

Behavioral Results

Behavioral evaluations produce a Trust Score and a hierarchy of Harness, Scenario, and Probe results.

vijil scores show "$AGENT_ID"

score = client.scores.get("<evaluation-id>")
results = client.scores.results("<evaluation-id>")
print(score.trust_score, score.reliability, score.security, score.safety)

curl -s "$VIJIL_URL/v1/evaluation-results/$EVALUATION_ID/results?team_id=$TEAM_ID" \
  -H "Authorization: Bearer $TOKEN"

Use client.scores.results() or the corresponding REST endpoint when you need the complete nested result hierarchy. Use the response listing below when you need filterable, paginated prompt-level evidence.

Prompt-Level Responses

Retrieve individual prompt and Agent response rows from a completed behavioral evaluation. The method supports both standard and custom Harness evaluations.

page = client.evaluations.responses(
    "<evaluation-id>",
    status="failed",
    limit=50,
    offset=0,
)

for response in page.items:
    print(response.prompt_text)
    print(response.response_text)
    print(response.probe_score)

curl -sG "$VIJIL_URL/v1/evaluations/$EVALUATION_ID/responses" \
  -H "Authorization: Bearer $TOKEN" \
  --data-urlencode "team_id=$TEAM_ID" \
  --data-urlencode "status=failed" \
  --data-urlencode "limit=50" \
  --data-urlencode "offset=0"

Each row identifies its parent Harness, Scenario, and Probe and includes the prompt, Agent response, parent Probe score, and raw Detector results. A failed row belongs to a Probe that scored below the evaluation’s pass threshold. Filter by harness_id, scenario_id, or probe_id to narrow the results further. The CLI does not currently expose this response listing. Red Team evaluations do not support it; use the Red Team evidence methods below instead. See the client.evaluations SDK reference for all arguments and the return type.

Red Team Evidence

Red Team evaluations expose seeds, attacks, judgments, and reflections under the evaluation ID:

vijil evaluations redteam-seeds "$EVALUATION_ID"
vijil evaluations redteam-attacks "$EVALUATION_ID"
vijil evaluations redteam-judgments "$EVALUATION_ID"
vijil evaluations redteam-reflections "$EVALUATION_ID"

seeds = client.evaluations.redteam_seeds("<evaluation-id>")
attacks = client.evaluations.redteam_attacks("<evaluation-id>")
judgments = client.evaluations.redteam_judgments("<evaluation-id>")
reflections = client.evaluations.redteam_reflections("<evaluation-id>")

curl -s "$VIJIL_URL/v1/evaluations/$EVALUATION_ID/redteam/attacks?team_id=$TEAM_ID" \
  -H "Authorization: Bearer $TOKEN"

Use --wave to filter the CLI seed, attack, or judgment commands. The attack and judgment commands also accept --seed with --wave. See the Red Team CLI guide for the complete workflow.

Retrieve Reports

Download the behavioral evaluation PDF:

vijil reports download "$EVALUATION_ID"

pdf_bytes = client.reports.download("<evaluation-id>")
with open("report.pdf", "wb") as report_file:
    report_file.write(pdf_bytes)

curl -s "$VIJIL_URL/v1/evaluations/$EVALUATION_ID/pdf?team_id=$TEAM_ID" \
  -H "Authorization: Bearer $TOKEN" \
  -o report.pdf

Retrieve the Red Team report as structured JSON, Markdown, or PDF:

vijil evaluations redteam-report "$EVALUATION_ID" --format md

report = client.evaluations.redteam_report("<evaluation-id>")
markdown = client.evaluations.redteam_report_markdown("<evaluation-id>")
pdf_bytes = client.evaluations.redteam_report_pdf("<evaluation-id>")

curl -s "$VIJIL_URL/v1/evaluations/$EVALUATION_ID/redteam/report?team_id=$TEAM_ID" \
  -H "Authorization: Bearer $TOKEN"

List and Manage Evaluations

vijil evaluations list --agent "$AGENT_ID"

page = client.evaluations.list(agent_id="<agent-id>")
for evaluation in page.items:
    print(evaluation.id, evaluation.status)

client.evaluations.delete("<evaluation-id>")

curl -s "$VIJIL_URL/v1/evaluations/" \
  -H "Authorization: Bearer $TOKEN"

curl -s -X POST "$VIJIL_URL/v1/evaluations/$EVALUATION_ID/cancel" \
  -H "Authorization: Bearer $TOKEN"

curl -s -X DELETE "$VIJIL_URL/v1/evaluations/$EVALUATION_ID" \
  -H "Authorization: Bearer $TOKEN"

The current CLI supports listing and showing evaluations. Use the SDK or REST API when you need to delete an evaluation, and use REST to cancel one.

Next Steps

Understand Results

Interpret behavioral Trust Scores and Red Team evidence

Red Team CLI

Inspect waves, attacks, judgments, and reports

Custom Harnesses

Create targeted behavioral evaluations

Local Agents

Run behavioral evaluations against local Agents

Get Started

Evaluate Agents

Red Team

Protect Agents

Reference

Run Evaluations

Prerequisites

Start an Evaluation

Run Compliance Harnesses

Check Status

Retrieve Results

Behavioral Results

Prompt-Level Responses

Red Team Evidence

Retrieve Reports

List and Manage Evaluations

Next Steps

Understand Results

Red Team CLI

Custom Harnesses

Local Agents

​Prerequisites

​Start an Evaluation

​Run Compliance Harnesses

​Check Status

​Retrieve Results

​Behavioral Results

​Prompt-Level Responses

​Red Team Evidence

​Retrieve Reports

​List and Manage Evaluations

​Next Steps

Understand Results

Red Team CLI

Custom Harnesses

Local Agents

Prerequisites

Start an Evaluation

Run Compliance Harnesses

Check Status

Retrieve Results

Behavioral Results

Prompt-Level Responses

Red Team Evidence

Retrieve Reports

List and Manage Evaluations

Next Steps