Local Agent Evaluation

Use this workflow when your Agent runs on your computer or does not support the request format that Vijil expects. You will place a small proxy in front of the Agent and expose that proxy through an ngrok tunnel.

TL;DR: Start an Agent locally, expose it through an authenticated ngrok tunnel, then run a behavioral or Red Team Evaluation with the Python SDK. Vijil services send requests to the local proxy through the public tunnel, so keep the proxy and tunnel running until the Evaluation completes.

A proxy changes Vijil requests into a format your Agent understands. It then changes the Agent’s response into a format Vijil can process. The tunnel makes the proxy reachable at a public HTTPS URL while it runs. This guide uses a restricted version of the Market Researcher from the Anthropic financial services examples. You can use the same pattern with another Agent by changing the function that calls Claude Code.

Evaluation Probes are adversarial. Use test data and test credentials. Disable write and destructive tools. If the Agent can access files or external systems, run it in a sandbox. Do not give it access to production resources.The tunnel URL is public, but the example protects /v1/chat/completions with a random API key. Run the proxy only while you test the Agent. Enterprise deployments can also restrict tunnel traffic to Vijil.

Understand How Requests Move

Vijil sends each Probe to the public tunnel URL. The tunnel forwards the request to the proxy on your computer. The proxy calls the Agent under test and returns its response to Vijil. You do not need to send Probes one at a time. After you register the proxy URL, you start the Evaluation in the same way as any other SDK Evaluation. Vijil sends the Probes, collects the responses, and records the results.

Install the Tools

Create a Python 3.12 virtual environment and install the packages used by the proxy:

python3.12 -m venv .venv
source .venv/bin/activate

python -m pip install --upgrade pip
python -m pip install "vijil-sdk>=0.1.61" fastapi uvicorn pyngrok

Set Up the Agent Under Test

This example uses Claude Code to run Market Researcher. Install Claude Code, then confirm that the claude command is available:

claude --version

Clone the financial services examples from Anthropic and save the absolute path to the Market Researcher plugin:

git clone https://github.com/anthropics/financial-services.git
cd financial-services

export MARKET_RESEARCHER_PLUGIN="$PWD/plugins/agent-plugins/market-researcher"

The example reads Market Researcher’s system prompt and loads its sector-overview Skill. It creates a temporary Agent definition that has no runtime tools. The Agent cannot read, write, or edit files, run shell commands, invoke more Skills, or use MCP connectors. Claude Code still reads the plugin prompt and Skill during startup. The --no-session-persistence flag prevents it from saving the conversation for later use.

Configure Credentials

Export the values used by the example:

export VIJIL_GATEWAY_URL="<console-gateway-base-url>"
export VIJIL_API_KEY="<vijil-access-token>"
export ANTHROPIC_API_KEY="<test-only-anthropic-api-key>"
export MARKET_RESEARCHER_PLUGIN="<absolute-path-to-market-researcher-plugin>"

Keep these values in the terminal where you run the proxy. Do not put them in the Python file or commit them to source control. Claude Code uses ANTHROPIC_API_KEY to run Market Researcher. The key stays on your computer. The script generates a separate key to protect the proxy.

Create the Proxy

The integration has three parts. The complete file at the end of this section puts them together.

Call the Agent

run_market_researcher() is the Agent-specific part. Replace this function when you use another Agent.

def run_market_researcher(messages: list[Message]) -> str:
    plugin_dir = Path(os.environ["MARKET_RESEARCHER_PLUGIN"]).resolve()
    source = (plugin_dir / "agents" / "market-researcher.md").read_text()
    market_researcher_prompt = source.split("---", maxsplit=2)[2].strip()

    # Build the restricted Agent definition and labelled prompt
    agent_definition = {"vijil-market-researcher": {...}}
    prompt = "\n\n".join(
        f"{message.role.upper()}:\n{message.content}"
        for message in messages
        if message.content
    )

    # Run the Agent and return its final text response
    result = subprocess.run(
        ["claude", "--agents", json.dumps(agent_definition), "-p", prompt],
        check=True,
        capture_output=True,
        text=True,
    )
    return json.loads(result.stdout)["result"]

The complete script includes the flags that load the Skill and remove runtime file, shell, Skill, and MCP tools.

Add the Vijil Endpoint

The proxy accepts an OpenAI-compatible request and returns the Agent response in choices[0].message.content. It rejects requests that do not contain the generated API key.

PROXY_API_KEY = secrets.token_urlsafe(32)


class Message(BaseModel):
    role: str
    content: str | None = None


class ChatCompletionRequest(BaseModel):
    messages: list[Message] = Field(default_factory=list)


@app.post("/v1/chat/completions")
def chat_completions(
    request: ChatCompletionRequest,
    authorization: str | None = Header(default=None),
):
    expected = f"Bearer {PROXY_API_KEY}"
    if authorization is None or not secrets.compare_digest(authorization, expected):
        raise HTTPException(status_code=401, detail="Invalid API key")

    content = run_market_researcher(request.messages)
    return {
        "model": MODEL_NAME,
        "choices": [
            {"message": {"role": "assistant", "content": content}}
        ],
    }

Run the Evaluation

Start the proxy, open the tunnel, and register the generated proxy key with Console. Keep the tunnel open until client.evaluate() returns.

# 1. Fetch tunnel credentials from Vijil
tunnel = client.agent_tunnel.get()

# 2. Start the local proxy
threading.Thread(target=start_server, daemon=True).start()

# 3. Open the ngrok tunnel
connection = ngrok.connect(LOCAL_PORT, "http", domain=tunnel.binding_url)

try:
    # 4. Register the Agent in Vijil Console
    agent = client.agents.create(
        name="market-researcher-local",
        url=connection.public_url,
        model_name=MODEL_NAME,
        hub="custom",
        api_key=PROXY_API_KEY,
        rate_limit_requests_per_minute=5,
    )

    # 5. Run the Evaluation and poll until it finishes
    evaluation = client.evaluate(
        agent.id,
        harness_id="security",
        sample_size=5,
    )
finally:
    # 6. Close the tunnel
    ngrok.disconnect(connection.public_url)

View the Complete Script

import json
import os
import secrets
import subprocess
import threading
import time
import uuid
from pathlib import Path

import uvicorn
from fastapi import FastAPI, Header, HTTPException
from pydantic import BaseModel, Field
from pyngrok import conf, ngrok

from vijil import Vijil


LOCAL_PORT = 8322
MODEL_NAME = "claude-code-market-researcher"
PROXY_API_KEY = secrets.token_urlsafe(32)


class Message(BaseModel):
    role: str
    content: str | None = None


class ChatCompletionRequest(BaseModel):
    model: str = ""
    messages: list[Message] = Field(default_factory=list)
    temperature: float | None = None
    max_tokens: int | None = None


app = FastAPI()


def run_market_researcher(messages: list[Message]) -> str:
    plugin_dir = Path(os.environ["MARKET_RESEARCHER_PLUGIN"]).expanduser().resolve()
    source = (plugin_dir / "agents" / "market-researcher.md").read_text()
    market_researcher_prompt = source.split("---", maxsplit=2)[2].strip()

    agent_definition = {
        "vijil-market-researcher": {
            "description": "A text-only Market Researcher",
            "prompt": market_researcher_prompt
            + """

Work only in text-only sector-overview mode. Do not access files, run
commands, call connectors, or invoke other Skills. Mark every unsupported
figure as [UNSOURCED].
""",
            "skills": ["market-researcher:sector-overview"],
        }
    }
    conversation = "\n\n".join(
        f"{message.role.upper()}:\n{message.content}"
        for message in messages
        if message.content
    )
    prompt = (
        "Respond to this labelled conversation. Return only the final response.\n\n"
        + conversation
    )

    command = [
        "claude",
        "--bare",
        "--plugin-dir",
        str(plugin_dir),
        "--agents",
        json.dumps(agent_definition),
        "--agent",
        "vijil-market-researcher",
        "--permission-mode",
        "dontAsk",
        "--tools",
        "",
        "--strict-mcp-config",
        "--no-session-persistence",
        "--output-format",
        "json",
        "-p",
        prompt,
    ]

    try:
        result = subprocess.run(
            command,
            check=True,
            capture_output=True,
            text=True,
            timeout=180,
        )
    except FileNotFoundError as exc:
        raise HTTPException(
            status_code=502,
            detail="Claude Code is not installed or is not on PATH",
        ) from exc
    except subprocess.SubprocessError as exc:
        raise HTTPException(
            status_code=502,
            detail="Claude Code could not run the Agent",
        ) from exc

    return json.loads(result.stdout)["result"]


@app.post("/v1/chat/completions")
def chat_completions(
    request: ChatCompletionRequest,
    authorization: str | None = Header(default=None),
):
    expected = f"Bearer {PROXY_API_KEY}"
    if authorization is None or not secrets.compare_digest(authorization, expected):
        raise HTTPException(status_code=401, detail="Invalid API key")

    content = run_market_researcher(request.messages)
    return {
        "id": f"chatcmpl-{uuid.uuid4().hex[:8]}",
        "object": "chat.completion",
        "created": int(time.time()),
        "model": MODEL_NAME,
        "choices": [
            {
                "index": 0,
                "message": {"role": "assistant", "content": content},
                "finish_reason": "stop",
            }
        ],
    }


@app.get("/health")
def health():
    return {"status": "ok"}


def start_server() -> None:
    uvicorn.run(app, host="127.0.0.1", port=LOCAL_PORT, log_level="warning")


def main() -> None:
    client = Vijil(
        api_key=os.environ["VIJIL_API_KEY"],
        gateway=os.environ["VIJIL_GATEWAY_URL"],
    )

    # 1. Fetch tunnel credentials from Vijil
    print("Fetching tunnel credentials")
    tunnel = client.agent_tunnel.get()

    # 2. Start the local proxy
    server_thread = threading.Thread(target=start_server, daemon=True)
    server_thread.start()
    time.sleep(1)

    conf.get_default().auth_token = tunnel.token

    # 3. Open the ngrok tunnel
    print("Opening the ngrok tunnel")
    connection = ngrok.connect(
        LOCAL_PORT,
        "http",
        domain=tunnel.binding_url,
    )
    public_url = connection.public_url
    print(f"Tunnel URL: {public_url}")

    try:
        # 4. Register the Agent in Vijil Console
        print("Registering the Agent")
        agent = client.agents.create(
            name="market-researcher-local",
            url=public_url,
            model_name=MODEL_NAME,
            hub="custom",
            api_key=PROXY_API_KEY,
            rate_limit_requests_per_minute=5,
        )
        print(f"Agent ID: {agent.id}")

        # 5. Run the Evaluation
        print("Running a small security Evaluation")
        evaluation = client.evaluate(
            agent.id,
            harness_id="security",
            sample_size=5,
            on_poll=lambda current: print(f"Status: {current.status}"),
        )

        print(f"Evaluation ID: {evaluation.id}")
        print(f"Status: {evaluation.status}")
        if evaluation.trust_score is not None:
            print(f"Trust Score: {evaluation.trust_score}")
        if evaluation.dimensions:
            print(f"Security: {evaluation.dimensions.security}")
        print("Results: https://evaluate.vijil.ai/evaluations")
    finally:
        # 6. Close the tunnel
        ngrok.disconnect(public_url)
        print("Tunnel closed")


if __name__ == "__main__":
    main()

The proxy converts each role and text value into a labelled section of one Claude Code prompt. It does not preserve native message-role semantics, and it ignores temperature and max_tokens. The example starts with a limit of five requests per minute. Increase rate_limit_requests_per_minute only after you confirm that the Agent and its model provider can handle more traffic.

Run the Complete Example

Save the complete script as local_agent_evaluation.py. Run it from the terminal where you exported the credentials:

python local_agent_evaluation.py

Keep the process running until the Evaluation finishes. The script:

Fetches your tunnel credentials from Vijil.
Starts the proxy on your computer.
Opens the public ngrok tunnel.
Registers the tunnel URL and generated proxy key in Vijil Console.
Starts a security Evaluation and polls until it finishes.
Closes the tunnel.

The sample_size=5 setting selects five test prompts for each Probe in the security Harness. This small sample helps you check the connection. It does not provide a complete security assessment. client.evaluate() polls until the Evaluation finishes and returns the final result. The script prints the Agent ID, Evaluation ID, status, Trust Score, Security score, and the Console results URL.

Run Adaptive Red Teaming

Because the tunnel provides a public endpoint, this workflow supports Red Team Evaluations. The SDK’s separate no-tunnel --local mode supports behavioral Evaluations only. To run adaptive Red Teaming against the same Agent, add this code inside the existing try block, after the standard Evaluation and before finally:

red_team = client.evaluate(agent.id, type="redteam")
print(f"Red Team Evaluation ID: {red_team.evaluation_id}")
print(f"Red Team status: {red_team.status}")
if red_team.degraded:
    print(f"Red Team warning: {red_team.degradation_reason}")
print("Results: https://evaluate.vijil.ai/evaluations")

client.evaluate(..., type="redteam") polls until the Red Team Evaluation finishes. Keep the proxy and tunnel running for the entire Evaluation. Adaptive Red Teaming can take longer than a standard Evaluation because it runs successive waves of attacks. Use this standalone version when you want to run adaptive Red Teaming instead of the standard Evaluation:

View the Complete Red Teaming Script

import json
import os
import secrets
import subprocess
import threading
import time
import uuid
from pathlib import Path

import uvicorn
from fastapi import FastAPI, Header, HTTPException
from pydantic import BaseModel, Field
from pyngrok import conf, ngrok

from vijil import Vijil


LOCAL_PORT = 8322
MODEL_NAME = "claude-code-market-researcher"
PROXY_API_KEY = secrets.token_urlsafe(32)


class Message(BaseModel):
    role: str
    content: str | None = None


class ChatCompletionRequest(BaseModel):
    model: str = ""
    messages: list[Message] = Field(default_factory=list)
    temperature: float | None = None
    max_tokens: int | None = None


app = FastAPI()


def run_market_researcher(messages: list[Message]) -> str:
    plugin_dir = Path(os.environ["MARKET_RESEARCHER_PLUGIN"]).expanduser().resolve()
    source = (plugin_dir / "agents" / "market-researcher.md").read_text()
    market_researcher_prompt = source.split("---", maxsplit=2)[2].strip()

    agent_definition = {
        "vijil-market-researcher": {
            "description": "A text-only Market Researcher",
            "prompt": market_researcher_prompt
            + """

Work only in text-only sector-overview mode. Do not access files, run
commands, call connectors, or invoke other Skills. Mark every unsupported
figure as [UNSOURCED].
""",
            "skills": ["market-researcher:sector-overview"],
        }
    }
    conversation = "\n\n".join(
        f"{message.role.upper()}:\n{message.content}"
        for message in messages
        if message.content
    )
    prompt = (
        "Respond to this labelled conversation. Return only the final response.\n\n"
        + conversation
    )

    command = [
        "claude",
        "--bare",
        "--plugin-dir",
        str(plugin_dir),
        "--agents",
        json.dumps(agent_definition),
        "--agent",
        "vijil-market-researcher",
        "--permission-mode",
        "dontAsk",
        "--tools",
        "",
        "--strict-mcp-config",
        "--no-session-persistence",
        "--output-format",
        "json",
        "-p",
        prompt,
    ]

    try:
        result = subprocess.run(
            command,
            check=True,
            capture_output=True,
            text=True,
            timeout=180,
        )
    except FileNotFoundError as exc:
        raise HTTPException(
            status_code=502,
            detail="Claude Code is not installed or is not on PATH",
        ) from exc
    except subprocess.SubprocessError as exc:
        raise HTTPException(
            status_code=502,
            detail="Claude Code could not run the Agent",
        ) from exc

    return json.loads(result.stdout)["result"]


@app.post("/v1/chat/completions")
def chat_completions(
    request: ChatCompletionRequest,
    authorization: str | None = Header(default=None),
):
    expected = f"Bearer {PROXY_API_KEY}"
    if authorization is None or not secrets.compare_digest(authorization, expected):
        raise HTTPException(status_code=401, detail="Invalid API key")

    content = run_market_researcher(request.messages)
    return {
        "id": f"chatcmpl-{uuid.uuid4().hex[:8]}",
        "object": "chat.completion",
        "created": int(time.time()),
        "model": MODEL_NAME,
        "choices": [
            {
                "index": 0,
                "message": {"role": "assistant", "content": content},
                "finish_reason": "stop",
            }
        ],
    }


@app.get("/health")
def health():
    return {"status": "ok"}


def start_server() -> None:
    uvicorn.run(app, host="127.0.0.1", port=LOCAL_PORT, log_level="warning")


def main() -> None:
    client = Vijil(
        api_key=os.environ["VIJIL_API_KEY"],
        gateway=os.environ["VIJIL_GATEWAY_URL"],
    )

    # 1. Fetch tunnel credentials from Vijil
    print("Fetching tunnel credentials")
    tunnel = client.agent_tunnel.get()

    # 2. Start the local proxy
    server_thread = threading.Thread(target=start_server, daemon=True)
    server_thread.start()
    time.sleep(1)

    conf.get_default().auth_token = tunnel.token

    # 3. Open the ngrok tunnel
    print("Opening the ngrok tunnel")
    connection = ngrok.connect(
        LOCAL_PORT,
        "http",
        domain=tunnel.binding_url,
    )
    public_url = connection.public_url
    print(f"Tunnel URL: {public_url}")

    try:
        # 4. Register the Agent in Vijil Console
        print("Registering the Agent")
        agent = client.agents.create(
            name="market-researcher-local",
            url=public_url,
            model_name=MODEL_NAME,
            hub="custom",
            api_key=PROXY_API_KEY,
            rate_limit_requests_per_minute=5,
        )
        print(f"Agent ID: {agent.id}")

        # 5. Run adaptive Red Teaming
        print("Running adaptive Red Teaming")
        red_team = client.evaluate(agent.id, type="redteam")
        print(f"Red Team Evaluation ID: {red_team.evaluation_id}")
        print(f"Red Team status: {red_team.status}")
        if red_team.degraded:
            print(f"Red Team warning: {red_team.degradation_reason}")
        print("Results: https://evaluate.vijil.ai/evaluations")
    finally:
        # 6. Close the tunnel
        ngrok.disconnect(public_url)
        print("Tunnel closed")


if __name__ == "__main__":
    main()

Reuse the Agent

The script closes the ngrok connection after each run. The public binding becomes unreachable when the tunnel closes. It does not revoke your tunnel credentials or archive the Agent in Console. You can run an Evaluation for this local Agent in Console only while its proxy and user-specific tunnel are running. client.agent_tunnel.get() returns the same tunnel credentials on later runs, so you can open the same binding again. Keep the Agent ID if you want to compare future Evaluations. If you rerun the full script, it registers another Agent record.

Get Started

Evaluate Agents

Red Team

Protect Agents

Reference

Local Agent Evaluation

Understand How Requests Move

Install the Tools

Set Up the Agent Under Test

Configure Credentials

Create the Proxy

Call the Agent

Add the Vijil Endpoint

Run the Evaluation

Run the Complete Example

Run Adaptive Red Teaming

Reuse the Agent

​Understand How Requests Move

​Install the Tools

​Set Up the Agent Under Test

​Configure Credentials

​Create the Proxy

​Call the Agent

​Add the Vijil Endpoint

​Run the Evaluation

​Run the Complete Example

​Run Adaptive Red Teaming

​Reuse the Agent

Understand How Requests Move

Install the Tools

Set Up the Agent Under Test

Configure Credentials

Create the Proxy

Call the Agent

Add the Vijil Endpoint

Run the Evaluation

Run the Complete Example

Run Adaptive Red Teaming

Reuse the Agent