Supported Providers
| Provider | Hub ID | Supported Targets |
|---|---|---|
| OpenAI | openai | GPT-4o, GPT-4, GPT-3.5 |
| Anthropic | anthropic | Claude 3 family |
| AWS Bedrock | bedrock / bedrockAgents | Foundation models, Bedrock Agents |
| Google Vertex AI | vertex | Gemini family |
| DigitalOcean | digitalocean | GenAI Platform agents |
| Custom | custom | Any OpenAI-compatible endpoint |
OpenAI
Store your OpenAI credentials via the API or dashboard before running an evaluation. Supported models includegpt-4o, gpt-4-turbo, gpt-4, and gpt-3.5-turbo.
Anthropic
Store your Anthropic credentials before running an evaluation. Supported models include theclaude-3 family.
AWS Bedrock
Bedrock supports both foundation models and custom Bedrock Agents.Foundation Models
Store your Bedrock access keys and specify the region to evaluate Foundation Models.For Bedrock foundation models, prepend
us. to the model ID from AWS’s supported models list.Bedrock Agents
For Bedrock Agents, securely supply youragent_id, agent_alias_id, and AWS access configurations.
Google Vertex AI
Authenticate with Google Cloud and retrieve your application default credentials. Store the providedclient_id, client_secret, and refresh_token in Vijil before targeting models from the gemini-1.5 or gemini-1.0 families.
DigitalOcean
Follow DigitalOcean’s guide to get youragent_id and agent_key, and securely provide them to Vijil.
Custom Endpoints
Evaluate any agent with an OpenAI-compatible API by storing your custom endpoint’s API key and providing the URL of the endpoint.For local agents not yet deployed, use LocalAgentExecutor instead.
Rate Limiting
Control evaluation pace to avoid API throttling. You can set limits for the number of requests per interval when registering credentials.Recommended Rate Limits
| Provider | Requests/min | Notes |
|---|---|---|
| OpenAI | 60-100 | Depends on tier |
| Anthropic | 60-100 | Depends on tier |
| Bedrock | 30-60 | Model-dependent |
| Vertex AI | 60-120 | Project quotas apply |
Work in Progress
The programmatic evaluation capabilities are currently in private preview and subject to change.
Next Steps
Running Evaluations
Execute and monitor evaluations
Custom Agents
Evaluate local agents
Understanding Results
Interpret evaluation results
Configure Protection
Set up Dome guardrails