Run Evaluations

Evaluations test your AI agent against targeted Scenarios to measure its trustworthiness. Diamond, Vijil’s evaluation engine, sends Probes to your Agent and analyzes responses to produce a Trust Score. For deeper adversarial testing, Diamond also runs Red Team campaigns against a registered Agent.

​

Diamond Evaluations

Navigate to Verify in the sidebar to open Diamond Evaluations. The page has three sections:

• Create Evaluation: Configure and launch new Evaluations
• Evaluation Results: Track progress and access completed Evaluations
• Terminated Evaluations: A list of canceled or failed Evaluations

​

Creating an Evaluation

Start by clicking Create Evaluation on the Diamond Evaluations page.

​

Select Agent

The Agent table shows all registered Agents in your workspace: Select the Agent you want to evaluate by clicking its row. Only Agents with status Active can be evaluated.

​

Select Harness

In the Test Configuration panel, you can choose an Evaluation type:

1. Trust Score: Standard Evaluation type. Measures Agent trustworthiness across reliability, security, and safety. Configure it under Baseline or Bespoke tabs
2. Compliance: Framework Evaluation type. Tests the Agent against regulatory and standards frameworks such as the EU AI Act and NIST AI RMF. Configure it under the Compliance tab
3. Red Team: Adaptive Evaluation type. Configure it under the Adaptive tab

Baseline tab: Configure standard Trust Score Evaluation across three dimensions:

• Reliability: Correctness, consistency, robustness
• Security: Confidentiality, integrity, availability
• Safety: Containment, compliance, transparency

Each dimension has a toggle. Enable all three for comprehensive Evaluation, or select specific dimensions to focus on particular concerns. Compliance tab: Choose one or more regulatory or standards frameworks, such as the EU AI Act, ISO/IEC 42001, NIST AI RMF, OSFI E-23, PIPEDA, the Taiwan AI Basic Act, and Singapore PDPA. Each framework has a toggle, with Select all and Deselect all to change them together. A run that contains only Compliance Harnesses produces a Compliance Trust Report. See Compliance Harnesses for what each framework tests. Bespoke tab: Configure custom Trust Score Evaluation based on Custom Harnesses that combine specific personas and policies. Custom Harnesses appear in this tab when you have created them in the Harness Registry with status Active. Adaptive tab: Configure a Red Team campaign, which explores the Agent over multiple waves of attacks instead of measuring it against a fixed Harness. The tab is only visible on Red Team-enabled deployments. See Run a Red Team Campaign for the settings and their tradeoffs.

​

Run Evaluation

Once you have selected an Agent and configured your Harness:

1. Verify your Agent selection in the left panel
2. Confirm configuration settings in the right panel
3. Click Run Evaluation

The Evaluation starts immediately. Diamond sends Probes to your Agent based on the selected configuration and records responses for analysis.

​

Running a Red Team Campaign

Red Team is designed for deeper adversarial exploration than a standard Trust Score or custom Harness Evaluation. It is useful when:

• The Agent handles sensitive data, regulated workflows, or privileged actions
• The Agent uses tools, MCP servers, delegated Agents, or external data stores
• A Trust Score or custom Harness finding needs deeper investigation
• A release needs security, safety, or risk-owner review before deployment
• You want to validate whether previous fixes reduced exploitable behavior

Red Team does not replace Trust Score Evaluations. Use Trust Score for reproducible readiness evidence, then use Red Team to search for harder-to-find vulnerabilities and successful attack strategies.

​

Before You Start

For best results, make sure the selected Agent is Active and has as much context as you can safely provide.

​

Launch Red Team

1. Open Create Evaluation in the Verify page in the Console
2. Choose the registered Agent you want to test
3. Select the Adaptive tab in the Test Configuration panel
4. Configure Red Team settings
5. Start the campaign

Each campaign follows the same loop: Taxonomy -> Attack seeds -> Attackers -> Judgments -> Reflections -> Report The taxonomy defines the risk areas to explore. Attack seeds convert those risks into concrete adversarial goals. Attackers execute the seeds against the Agent. Judges review transcripts for harmful behavior, policy violations, and leaked artifacts. Reflections summarize what worked and guide the next wave. The final report clusters the most important findings.

​

Red Team Settings

Optionally, you can add Personas and Policies to help improve the Evaluation: If Policies are missing, Red Team can still run, but judgments may rely more heavily on general safety and security expectations.

​

Advanced Settings

Use advanced settings when you understand the cost and runtime impact of the campaign:

​

Monitoring Progress

Running Evaluations appear in the Evaluation Results table:

​

Evaluation Status

Evaluations typically complete in 5-30 minutes depending on the Harness size and your Agent’s rate limits. Red Team campaigns take longer and are monitored through their own campaign panel. See Run a Red Team Campaign.

​

Viewing Results

When an Evaluation completes, access the results through the Actions column:

• View (eye icon) — Opens the Trust Report in a new tab
• Download (download icon) — Downloads results as a file

The Trust Report provides:

• Overall Trust Score with pass/fail status
• Per-dimension breakdown
• Detailed findings for each Probe category
• Deployment recommendations

See Understand Results for detailed guidance on interpreting Trust Score findings, and Understand Red Team Results for waves, judgments, and the final campaign report.

​

Evaluation Considerations

​

Rate Limits

Diamond respects the rate limit you configured during Agent registration. Higher rate limits enable faster Evaluations but may exceed your provider’s quotas. If Evaluations fail with timeout errors:

• Verify your Agent URL is accessible
• Check that your API credentials are valid
• Consider reducing the rate limit in Agent settings

​

Agent Availability

Your Agent must remain available throughout the Evaluation. If your Agent goes offline or becomes unresponsive, the Evaluation may fail or produce incomplete results. For production Agents behind load balancers, ensure sufficient capacity to handle Evaluation traffic alongside normal usage.

​

Re-running Evaluations

You can run multiple Evaluations against the same Agent. Each Evaluation creates a new entry in the results table, allowing you to:

• Track Trust Score changes over time
• Compare results before and after Agent modifications
• Verify fixes for previously identified issues

​

Best Practices

Evaluate before deployment: Run a Trust Score Evaluation on every Agent before it reaches production. The results provide evidence of baseline trustworthiness. Test after changes: Any modification to your agent, such as prompt updates, model changes, or tool additions, can affect behavior. Re-evaluate to verify. Use appropriate Harnesses: The Trust Score Harness tests general behaviors. For domain-specific requirements, create custom Harnesses with relevant personas and policies. Use Red Team for deeper security review: Run a Red Team campaign after baseline Evaluation, before major releases, and after changes to tools, prompts, policies, or access controls. Monitor for regressions: Compare Trust Scores across Evaluations. A declining score indicates problems introduced by recent changes.

​

Next Steps