Run Evaluations

Vijil Evaluate is a quality assurance framework that automates the testing of LLM applications. An Evaluation in Vijil is an automated test run where you select one or more AI agents and a test Harness (covering Security, Reliability, and Safety) to systematically assess the quality, safety, and reliability of LLM applications.

Prerequisites

CLI configured and authenticated, or a Bearer token from POST /auth/jwt/login — see Console API
A registered Agent and its UUID

Start an Evaluation

vijil eval run \
  --agent-id "$AGENT_ID" \
  --harness-names '["safety", "security"]' \
  --sample-size 50 \
  --wait

Flag	Description	Required
`--agent-id`	UUID of the Agent to evaluate	Yes
`--harness-names`	JSON array of Harness names	Yes
`--sample-size`	Probes per Harness (1–1000); omit to run all
`--harness-type`	`standard` (default) or `custom`
`--wait`	Block until the evaluation completes
`--json`	Output as JSON

Use --sample-size 10 for fast iteration during development. Run the full Harness before releasing to production.

The API returns 202 Accepted immediately. Use the returned evaluation_id to poll for status.

curl -s -X POST "$VIJIL_URL/evaluations/" \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d "{
    \"agent_id\": \"$AGENT_ID\",
    \"team_id\": \"$TEAM_ID\",
    \"harness_names\": [\"safety\", \"security\"],
    \"sample_size\": 50
  }"

Response:

{
  "evaluation_id": "e5f6a7b8-...",
  "status": "starting"
}

Save the evaluation ID:

export EVAL_ID="e5f6a7b8-..."

Field	Description	Required
`agent_id`	UUID of the Agent to evaluate	Yes
`team_id`	UUID of your team	Yes
`harness_names`	Array of Harness names	Yes
`sample_size`	Probes per Harness (1–1000); omit to run all
`harness_type`	`"standard"` (default) or `"custom"`

With the Vijil MCP server configured, ask Claude Code in natural language:

Run a safety and security evaluation on agent a1b2c3d4-... with sample size 50, and wait for it to complete

Claude calls eval_run with wait=True and reports back Trust Scores when the evaluation finishes.

Check Status

Status progresses through: starting → pending → running → completed → saving → saved. It may also be failed or canceled.

vijil eval status <evaluation_id>

When the status reaches completed, the response includes per-Harness Trust Scores:

{
  "evaluation_id": "e5f6a7b8-...",
  "status": "completed",
  "scores": {
    "safety": 0.82,
    "security": 0.67
  },
  "completed_at": 1712506200
}

Retrieve Results

Get the full per-Probe breakdown once the evaluation status is saved.

vijil eval results-detail <evaluation_id> --json | jq '.scores'

Generate a Report

Produce a Trust Report for sharing or archiving.

vijil eval report <evaluation_id>

List and Cancel Evaluations

vijil eval list --agent-id "$AGENT_ID" --status completed

Use GET /evaluations/ and POST /evaluations/{evaluation_id}/cancel for the REST API equivalents.

Next Steps

Understand Results

Deep dive into scores and failures

Custom Harnesses

Create targeted Evaluations

Cloud Providers

Configure provider integrations

​Prerequisites

​Start an Evaluation

​Check Status

​Retrieve Results

​Generate a Report

​List and Cancel Evaluations

​Next Steps

Understand Results

Custom Harnesses

Cloud Providers

Prerequisites

Start an Evaluation

Check Status

Retrieve Results

Generate a Report

List and Cancel Evaluations

Next Steps