Your Agent works in demos. Your unit tests pass. But how do you know it will not hallucinate facts, leak customer data, or comply with malicious instructions when it encounters inputs you did not anticipate?
Diamond is Vijil’s evaluation platform. Choose a behavioral evaluation for a Trust Score and Probe-level findings, or a Red Team evaluation for deeper exploration of vulnerabilities and successful attack strategies.
A behavioral evaluation sends hundreds of adversarial Probes to your agent (prompt injections, jailbreak attempts, data exfiltration payload, etc.) and measures how your agent responds. You get a quantified Trust Score and specific findings you can fix before deployment.
How Evaluation Works
Behavioral Evaluations
Diamond sends test Probes from a Harness to your Agent and analyzes the responses:Red Team Evaluations
Red Team uses the registered Agent’s context to generate attack seeds, run attackers, judge transcripts, and use reflections to guide later waves. Its report groups vulnerabilities and successful attack strategies. See Red Team in Diamond for the wave loop.Behavioral Results: Trust Score
Behavioral evaluations produce a Trust Score, a composite metric (0 to 100) based on three pillars:
Higher scores indicate more trustworthy behavior. Use scores to:
- Set deployment gates (e.g., require Trust Score ≥ 70)
- Compare Agent versions
- Track improvements over time
- Identify specific vulnerabilities
Evaluation Status
Both evaluation types use anevaluation_id, but they have different status vocabularies and result endpoints. The SDK treats these as terminal statuses:
A
degraded Red Team run has finished with some attackers lost along the way. Check the findings and failure details before relying on its report.
Evaluation Options
Cloud-Hosted Agents
Evaluate Agents deployed on supported cloud platforms such as OpenAI, Anthropic, AWS Bedrock, Google Vertex AI, DigitalOcean, and any OpenAI-compatible endpoint.Local Agents
Run behavioral evaluations against Agents running locally without deployment. This creates a temporary authenticated tunnel (via ngrok) for Vijil to communicate with your local Agent. Red Team requires a registered Agent and cannot run in local mode.Behavioral Harnesses
Add
_Small suffix (e.g., security_Small) for faster iterations during development.
Evaluation Workflow
1
Choose your integration method
Use cloud provider integration for deployed agents, or LocalAgentExecutor for local development
2
Select an evaluation type
Choose
behavioral and a Harness for a Trust Score, or redteam and optional wave settings for adaptive attacks3
Run evaluation
Execute via the CLI, Python SDK, REST API, or Console
4
Track status
Use the evaluation ID to follow the status for the selected type until it reaches a terminal state
5
Analyze results
Review Trust Score and Probe findings for behavioral evaluations, or attack evidence and the report for Red Team evaluations
6
Remediate issues
Fix vulnerabilities, improve prompts, add Guardrails
7
Re-evaluate
Confirm fixes and track improvement
Behavioral Sample Size
Use a smaller Probe sample during behavioral evaluations to avoid overwhelming your Agent or hitting API limits. Red Team uses wave, seed, and parallel attacker limits instead.
Use the
_Small Harness variants (e.g., security_Small) for faster iteration runs that sample a representative subset of Probes.
Work in Progress
The programmatic evaluation capabilities are currently in private preview and subject to change.
Next Steps
Run Evaluations
Execute evaluations and monitor progress
Understand Results
Interpret scores and failures
Cloud Providers
Configure cloud platform integrations
Custom Harnesses
Create targeted evaluation Scenarios