Skip to main content

Documentation Index

Fetch the complete documentation index at: https://docs.vijil.ai/llms.txt

Use this file to discover all available pages before exploring further.

Agents’ evaluation is at the core of Vijil.
NOTE: This section is currently under development.
Your agent works in demos. Your unit tests pass. But how do you know it will not hallucinate facts, leak customer data, or comply with malicious instructions when it encounters inputs you did not anticipate? Diamond is Vijil’s evaluation platform. It sends hundreds of adversarial Probes to your agent (prompt injections, jailbreak attempts, data exfiltration payload, etc.) and measures how your agent responds. You get a quantified Trust Score and specific findings you can fix before deployment.

​
How Evaluation Works

Diamond sends test Probes to your agent and analyzes the responses: Diamond evaluation flow showing Harness, Scenarios, Probes, agent, Detectors, and Trust Score
ComponentPurpose
HarnessCollection of Scenarios to run (e.g., trust_score, security)
ScenarioTesting context with personas and policies
ProbeIndividual test case sent to your agent
DetectorAnalyzes agent responses to identify failures

​
Trust Score

The Trust Score is a composite metric (0.0 to 1.0) based on three pillars:
DimensionWhat It Measures
ReliabilityHallucination resistance, consistency, accuracy
SecurityPrompt injection, data leakage, jailbreak resistance
SafetyHarmful content, policy compliance, ethical behavior
Higher scores indicate more trustworthy behavior. Use scores to:
  • Set deployment gates (e.g., require Trust Score ≥ 0.70)
  • Compare agent versions
  • Track improvements over time
  • Identify specific vulnerabilities

​
Evaluation Options

​
Cloud-Hosted Agents

Evaluate agents deployed on supported cloud platforms such as OpenAI, Anthropic, AWS Bedrock, Google Vertex AI, DigitalOcean, and any OpenAI-compatible endpoint.

​
Local Agents

Evaluate agents running locally without deployment. This creates a temporary authenticated tunnel (via ngrok) for Vijil to communicate with your local agent.

​
Available Harnesses

HarnessDescriptionUse Case
trust_scoreComprehensive evaluation across all dimensionsPre-deployment validation
securityPrompt injection, jailbreaks, data leakageSecurity review
reliabilityHallucination, consistency, accuracyQuality assurance
safetyHarmful content, ethics, policy complianceSafety review
owasp-llm-top-10OWASP Top 10 for LLM ApplicationsCompliance
Add _Small suffix (e.g., security_Small) for faster iterations during development.

​
Evaluation Workflow

1

Choose your integration method

Use cloud provider integration for deployed agents, or LocalAgentExecutor for local development
2

Select harnesses

Start with trust_score for comprehensive coverage, or specific Harnesses for targeted testing
3

Run evaluation

Execute via Python client, REST API, or console
4

Analyze results

Review Trust Score, dimension scores, and individual failures
5

Remediate issues

Fix vulnerabilities, improve prompts, add Guardrails
6

Re-evaluate

Confirm fixes and track improvement

​
Rate Limiting

Control the evaluation pace to avoid overwhelming your agent or hitting API limits during evaluations.

Work in Progress

The programmatic evaluation capabilities are currently in private preview and subject to change.

​
Next Steps

Run Evaluations

Execute evaluations and monitor progress

Understand Results

Interpret scores and failures

Cloud Providers

Configure cloud platform integrations

Custom Harnesses

Create targeted evaluation Scenarios
Last modified on April 28, 2026