AWS Bedrock¶

You can evaluate LLMs, accessed as RESTful APIs on Amazon Bedrock, using Vijil Evaluate. To do so, first set up a Bedrock Access Gateway.

You can then add an API key using the UI or the python client.

Run an Evaluation¶

To run an evaluation from the UI, simply select Bedrock as the Model Hub, then paste your custom endpoint as the Model URL, and select a *Model Name from the dropdown.

To run an evaluation using the Python client, use the following code pattern, with your custom endpoint as model_url, model name as model_name, and a harness of your choice.

Python

client.evaluations.create(
    model_hub="bedrock",
    model_url="https://your_model_url",
    model_name="anthropic.claude-v2",
    model_params={"temperature": 0},
    harnesses=["hallucination"],
)