View Custom Harnesses
You can view previously created custom Harnesses by navigating to Harnesses in the left sidebar. To view the prompts in a custom Harness, click on its row in the Harnesses table.Create a Custom Harness
- In the left sidebar, navigate to Harnesses and click Create Harness.
- Enter a Harness name and a description.
- Select Prompts
- Generate Prompts
- Select an existing prompt, or create a new one by pressing Add Prompt.
- Press Create.
Create a Custom Harness via API
You can programmatically create custom harnesses using the Vijil client API.Policy Document(s)
You can create a custom policy adherence Harness that checks whether your model adheres to its system prompt or an organizational policy. To do this, you need a system prompt specified as a string, and an optional organizational policy provided as a.txt or .pdf file. If you do not provide a policy file, Vijil will create a harness based only on the provided system prompt. To specify that you want a policy adherence harness, you need to specify the category argument as ["AGENT_POLICY"].
You can use the harnesses.create API function to create a Harness to test adherence against frameworks like NIST AI RMF. After creation, you can check the status of the harness creation job. The harness_config_version starts at 1.0.0 for any new Harness name. If you create another Harness with the same name, Vijil automatically increments the Harness version.
Once the Harness is created, you can pass its ID when running an evaluation with the API.
Knowledge Base (Enterprise Only)
If you are developing a RAG agent and would like to generate a custom test Harness to evaluate generation and retrieval capabilities based on a set of document chunks, upload the documents that you would like to base the evaluation on into a GCP storage bucket. When creating the harness via the API, set thecategory parameter to ["KNOWLEDGE_BASE"].
Tool Calling Agent (Enterprise Only)
To evaluate a tool calling agent, you need to supply input and output schemas for a function that you want to generate test prompts based on, as well as an endpoint to call that function. To specify that you want a tool calling Harness, set thecategory parameter to ["FUNCTION_ROUTE"].
In addition, provide the function_route variable which stores the URL required to call the function with the arguments and return the result.
Custom Harness with Multiple Components
You can also create a Harness that contains multiple components. For example, you can create a Harness that contains a knowledge base component and a tool calling agent component, or any of those components together with a policy adherence component. To do this, specify multiple values in thecategory list when making the API call.
Work in Progress
The programmatic evaluation capabilities are currently in private preview and subject to change.
Next Steps
Run Evaluations
Execute custom harness evaluations
Understand Results
Analyze custom harness results
Personas
Learn more about personas
Policies
Learn more about policies