> ## Documentation Index > Fetch the complete documentation index at: https://docs.vijil.ai/llms.txt > Use this file to discover all available pages before exploring further. # Client > Discover the `client.py` docs ## APIKeys Class for managing model hub API keys, which are required to query models. parameters: * `client` (VijilClient): The Vijil client instance. ### \_register\_cache\_refresh\_callback Register a callback to be called when API keys are modified. parameters: * `callback` (Callable\[\[], None]): A callable that will refresh the API key cache. ### \_notify\_cache\_refresh Notify all registered callbacks that the API key cache should be refreshed. ### list List all stored model hub API keys. Returns list of dictionaries. each dictionary contains information about an api key as `List(dict)`. ### get\_id\_by\_name Get the ID of an API key by its name. Used by other functions to get the ID of an API key. Returns the id of the api key as `str`. parameters: * `name` (str): The name of the API key. ### check\_model\_hub Used by other functions to check that the model hub is valid and the key name is unique. parameters: * `model_hub` (str): The name of the model hub. ### name\_exists Check whether the API key name already exists. Returns true if the name exists among the stored api keys, false otherwise as `bool`. parameters: * `name` (str): The name of the API key. ### check\_hub\_config Check that the model hub configuration is valid, i.e. that it has any fields required for that hub. parameters: * `model_hub` (str): The name of the model hub. * `hub_config` (dict): The configuration of the model hub. * `api_key` (str): The name of the API key. ### create Create a new model hub API key. Returns response to the api request as `dict`. parameters: * `name` (str): Name for the API key. This must be unique. * `model_hub` (str): Name of the model hub. Current supported values are 'openai', 'together', 'digitalocean', 'mistral', 'fireworks', 'nvidia', 'bedrock', 'azure', 'custom', 'digitalocean', 'openrouter', 'bedrockAgents' * `rate_limit_per_interval` (int, optional): The maximum amount of times Vijil will query the model hub in the specified rate\_limit\_interval, defaults to 60 * `rate_limit_interval` (int, optional): The size of the interval (in seconds) defining maximum queries to model hub in said interval. For example, if rate\_limit\_per\_interval is 60 and rate\_limit\_interval is 10, then Vijil will query the model hub at most 60 times in 10 seconds. Defaults to 10 * `api_key` (str, optional): The API key. * `hub_config` (dict, optional): A dictionary containing additional configuration for the model hub. Defaults to None. ### rename Rename a stored API key. Returns response to the api request that renames the key as `dict`. parameters: * `name` (str): The current name of the key. * `new_name` (str): The new name of the key. ### modify Modify model hub, key, or rate limits of a stored API key. Cannot be used to rename key. Returns response to the api request that modifies the key or model hub configuration as `dict`. parameters: * `name` (str): The name of the key you want to modify. * `model_hub` (str, optional): Name of the model hub. Current supported values are 'openai', 'together', 'octo'. * `api_key` (str, optional): The API key. * `rate_limit_per_interval` (int, optional): The maximum amount of times Vijil will query the model hub in the specified rate\_limit\_interval, defaults to 60 * `rate_limit_interval` (int, optional): The size of the interval (in seconds) defining maximum queries to model hub in said interval. For example, if rate\_limit\_per\_interval is 60 and rate\_limit\_interval is 10, then Vijil will query the model hub at most 60 times in 10 seconds. Defaults to 10 ### delete Delete the API key with the specified name. Returns response to the api request that deletes the key as `dict`. parameters: * `name` (str): The name of the key you want to delete ## Harnesses Class for handling Harnesses API requests. parameters: * `client` (VijilClient): The VijilClient instance. ### calculate\_md5\_base64 Calculate the MD5 hash of a custom Harness policy file and return it as a base64 string. Returns md5 hash of the file as a base64 string as `str`. parameters: * `file_path` (str): Path to the file. ### calculate\_file\_size Calculate the size of a custom Harness policy file in bytes. Returns size of the file in bytes as `int`. parameters: * `file_path` (str): Path to the file. ### list List all Harnesses. Returns list of dicts where each dict contains the metadata for a Harness, or a pandas dataframe if format is "dataframe" as `List(dict) or pandas.DataFrame`. parameters: * `type` (Optional\[str], optional): Type of Harness to list. Current supported values are "benchmark", "audit", "dimension", "custom". Defaults to None, in which case all Harnesses are listed. * `format` (str, optional): Format of the returned list. Current supported values are 'dataframe', 'list'. Defaults to "dataframe". ### create Create a custom Harness from a system prompt and an optional policy file. Returns the specified Harness name, the Harness id, and the status of the Harness creation process as `dict`. parameters: * `name` (any): The name of the Harness. * `system_prompt` (any): The system prompt for the model you are testing. * `category` (any): The category of the Harness. Options are "AGENT\_POLICY", "KNOWLEDGE\_BASE", "FUNCTION\_ROUTE", "PERSONA" * `policy_file_path` (any): The path to the policy document (pdf or txt). Applicable to an agent policy Harness. Defaults to "". * `kb_bucket` (any): The bucket name for the knowledge base. Must be specified if you want to include a knowledge base Harness. Defaults to "". * `input_schema` (any): The input schema to be used for Harness creation. Applicable to a tool-calling agent. Defaults to {}. * `output_schema` (any): The output schema to be used for Harness creation. Applicable to a tool-calling agent. Defaults to {}. * `function_route` (any): The function route to be used for Harness creation. Applicable to a tool-calling agent. Defaults to "". * `persona_ids` (any): The persona IDs to be used for Harness creation. Applicable to a persona Harness. Defaults to \[]. ### get\_status Get the status of a Harness. Returns the status of the custom Harness as `dict`. parameters: * `harness_id` (any): The ID of the Harness. ## AnalysisReports AnalysisReports class for handling analysis reports. parameters: * `client` (any): The VijilClient instance. * `evaluation_id` (any): The ID of the evaluation. * `evaluation_metadata` (any): The metadata of the evaluation. ### \_list\_reports List all the reports for an evaluation. Returns a list of report ids as `list`. parameters: * `status` (any): The status of the reports to list. Defaults to "CREATED". ### \_get\_analysis\_report\_by\_id Get the report given eval ID and report ID. Returns the report as `dict`. parameters: * `report_id` (any): The ID of the report to get. ### \_request\_analysis\_report Request an analysis report for the evaluation. ### \_save\_report Save the report content to a file in the specified format. parameters: * `report_content` (any): The content of the report. * `save_file` (any): The file path to save the report. * `format` (any): The format of the report ('html' or 'pdf'). ### generate Generates an analysis report for the evaluation. First checks to see if a report already exists, if so, it fetches the most recent report. Otherwise, a request is sent to create a report. If wait\_till\_completion is true, we wait till the report generation process is completed. Returns none if the report was generated successfully, otherwise the error message as `None | str`. parameters: * `save_file` (any): The file path to save the report. If not, a default file name formed from the evaluation ID and format is used. * `wait_till_completion` (any): Whether to wait till the report generation process is completed. Defaults to True. * `poll_frequency` (any): The frequency to poll for the report generation process. Defaults to 5 seconds. * `format` (any): The format of the report ('html' or 'pdf'). Defaults to 'html'. ## Evaluations Class for handling evaluations API requests. parameters: * `client` (VijilClient): The VijilClient instance. ### \_refresh\_api\_proxy\_dict Refresh the API proxy dictionary cache. ### list List all valuations. Will return only 10 evaluations unless specified. Returns list of evaluations as `list`. parameters: * `limit` (int, optional): The number of evaluations to return, defaults to 10. ### list\_harnesses\_for\_type List all Harnesses of a given type(s). Returns list of Harnesses as `list`. parameters: * `harness_types` (List\[str]): List of Harness types to list. * `latest_version` (bool, optional): If True, will return only the latest version of each Harness, defaults to True. ### get\_harness\_tags Given the list of Harnesses, ensure they belong to the same tag group and get the tag group. This is to ensure they are all on the correct UI page. Returns tag group of the Harnesses as `str`. parameters: * `harness_names` (List\[str]): List of Harness names to get the tag group for. ### create Create a new evaluation. Returns api response containing evaluation id of the newly created evaluation as `dict`. parameters: * `model_hub` (str): The model hub you want to use. Supported options are "openai", "together", "digitalocean", "custom". * `harness_version` (str): The version of the Harness you want to use. * `model_name` (str, optional): The name of the model you want to use. Check the model hub's API documentation to find valid names. * `name` (str, optional): The name of the evaluation. If not specified, model hub will be concatenated with model name. * `api_key_name` (str, optional): The name of the model hub API key you want to use. If not specified, will use the first key we find for the specified model\_hub. * `model_url` (str, optional): The URL of the model you want to use. Only required for custom model hub. Defaults to None * `model_params` (dict, optional): A dictionary specifying inference parameters like temperature and top\_p. If none are specified, model hub defaults will be used. Defaults to {} * `harness_params` (dict, optional): Set optional parameters like is\_lite, defaults to {} * `harnesses` (List\[str], optional): A list of Harnesses you want to include in the evaluation, defaults to \[] ### get\_status Retrieve the status of an evaluation. Returns a dict with the id, status, and other metadata of the evaluation as `dict`. parameters: * `evaluation_id` (str): The unique ID of the evaluation ### get\_metadata Get the metadata for an evaluation ID, including tag information. Returns a dict with the id, status, and other metadata of the evaluation as `dict`. parameters: * `evaluation_id` (str): The unique ID of the evaluation ### get\_tree Retrieve the tree of an evaluation. Returns for each Probe, information about which Harness and Scenario it came from as `dict`. parameters: * `evaluation_id` (str): The unique ID of the evaluation ### \_get\_ancestry Retrieve the ancestry of a node in the tree. Returns a dict with each ancestor as a value and ancestor types as the keys as `dict`. parameters: * `tree` (dict): The tree of the evaluation * `node_id` (str): The unique ID of the node ### summarize Return summary dataframe of the evaluation results, aggregated at every level (overall evaluation, dimension, Scenario, Probe). Returns a dataframe with the level, level\_name, and score of the evaluation as `pandas.DataFrame`. parameters: * `evaluation_id` (str): The unique ID of the evaluation ### describe Return either a list or a dataframe of prompt-level metadata and evaluation results, with metadata and evaluation scores for each prompt/response in the given evaluation id. Returns a list or dataframe of prompt-level metadata and evaluation results as `list or pandas.DataFrame`. parameters: * `evaluation_id` (str): The unique ID of the evaluation * `limit` (int, optional): The maximum number of prompts to include in description. Defaults to 1000. * `format` (str, optional): The format of the output. Defaults to "dataframe". Options are "dataframe" and "list". * `prettify` (bool, optional): If True, will remove the "Vijil.Probes." prefix from the Probe names to make it more readable. Defaults to True. * `hits_only` (bool, optional): If True, will only return prompts that had undesirable responses (according to our Detectors). Defaults to False. ### export Exports output logs from describe() into csv, jsonl, json, or parquet. Returns success message with the filepath where the report was exported as `str`. parameters: * `evaluation_id` (str): The unique ID of the evaluation * `limit` (int, optional): The maximum number of prompts to include in the report. Defaults to 1000000. * `format` (str, optional): The format of the output. Defaults to "csv". Options are "csv", "parquet", "json" and "jsonl" * `output_dir` (str, optional): The directory to save the report. Defaults to the current directory. * `prettify` (bool, optional): If True, will remove the "Vijil.Probes." prefix from the Probe names to make it more readable. Defaults to True. * `hits_only` (bool, optional): If True, will only return prompts that had undesirable responses (according to our Detectors). Defaults to False. ### cancel Cancels an in-progress evaluation. parameters: * `evaluation_id` (str): The unique ID of the evaluation ### delete Deletes an evaluation. parameters: * `evaluation_id` (str): The unique ID of the evaluation ### get\_probes Get all Probes and Probe metadata for a specific evaluation. Returns a dict with keys results, count. Results array contains Probes and count indicates number of Probes. ### get\_probes\_info Get metadata for all Probes in a specific evaluation. Returns a list of dicts with keys: Probe, name, description, scoring\_function. parameters: * `evaluation_id` (str): The unique ID of the evaluation ### get\_scenario\_info Get metadata for all Scenarios in a specific evaluation. Returns a list of dicts with keys: Scenarios, name, description. parameters: * `evaluation_id` (str): The unique ID of the evaluation ### get\_harness\_info Get metadata for all Harnesses in a specific evaluation. Returns a list of dicts with keys: Harness, name, description. parameters: * `full` (bool, optional): If True, returns all Harness info. If False, returns only Harness, name, description. Defaults to False. ### report ## Detectors Class for handling API requests to get Detector metadata. parameters: * `client` (VijilClient): The VijilClient instance ### get\_detector\_info Gets Detector metadata for a specific Detector id. Returns the Detector metadata as `dict`. parameters: * `detector_id` (str): The unique ID of the Detector * `version` (Optional\[str], optional): The version of the Detector metadata to get. Defaults to None. ### list Lists all available Detectors and their metadata. Returns the Detector metadata as `dict`. parameters: * `version` (Optional\[str], optional): The version of the Detector metadata to get. Defaults to the latest version. ## Detections Class for handling requests to the detections API. ### \_refresh\_api\_proxy\_dict Refresh the API proxy dictionary cache. ### list\_detectors Lists all available Detectors. Returns the list of available Detectors as `list`. ### create Create a new detection. Returns the response from the api. if the detection creation was successful, this is a dictionary with the following format: `{'id': your_guid, 'status': 'created'}` as `dict`. parameters: * `detector_id` (str): The unique ID of the Detector * `detector_inputs` (List\[dict]): Input payload to the Detector * `detector_params` (dict): Optional parameters to be passed for the Detector ### get\_status Retrieve the status of a detection. Returns the response from the api as `dict`. parameters: * `detection_id` (str): The unique ID of the detection ### describe Describe a detection. Returns the response from the api as `dict`. parameters: * `detection_id` (str): The unique ID of the detection ## Agents ### \_check\_agent\_name\_exists Check if an agent name already exists. parameters: * `agent_name` (str): The agent name to check. * `exclude_agent_id` (str, optional): Optional agent ID to exclude from the check (for updates). ### \_find\_agent\_by\_name Find an agent by name and return the agent object. Returns the agent object as `dict`. parameters: * `agent_name` (str): The agent name to find. * `include_archived` (bool, optional): Whether to include archived agents in the search. ### create Create a new agent. If api\_key\_name is specified, use the API key with that name. Otherwise, create a new API key with the specified API key value. Returns the response from the api showing the created agent configuration as `dict`. parameters: * `agent_name` (str): The name of the agent. * `hub` (str): The hub of the agent. * `api_key_name` (str): The name of an existing API key to use. If not specified, we will create a new API key with a random name using the other fields in the request. * `agent_id` (str): The ID of the agent. Used only for certain hubs. * `agent_alias_id` (str): The alias ID of the agent. Used only for Bedrock Agents. * `model_name` (str): The name of the model. * `agent_system_prompt` (str): The system prompt of the agent. * `api_key_value` (str): The value of the API key to use. Must be empty if api\_key\_name is specified. * `rate_limit_interval` (int, optional): The size of the interval (in seconds) defining maximum queries to model hub in said interval. For example, if rate\_limit\_per\_interval is 60 and rate\_limit\_interval is 10, then Vijil will query the model hub at most 60 times in 10 seconds. Defaults to 10 * `rate_limit_per_interval` (int, optional): The maximum amount of times Vijil will query the model hub in the specified rate\_limit\_interval, defaults to 60 * `hub_config` (Optional\[dict], optional): The hub config of the agent, defaults to None. This is required for certain hubs. ### update Update an existing agent configuration by name. Returns response to the api request, showing the updated agent configuration as `dict`. parameters: * `agent_name` (str): The current name of the agent to update. * `new_agent_name` (str, optional): The new name for the agent (if renaming). * `model_name` (str, optional): The new model name. * `agent_url` (str, optional): The new URL of the agent. * `api_key_name` (str, optional): The name of the API key to use. * `hub` (str, optional): The hub of the agent. * `agent_system_prompt` (str, optional): The new system prompt of the agent. ### list List agent configurations. Returns list of agent configurations as `List[dict]`. parameters: * `include_archived` (bool, optional): Whether to include archived (deleted) agents in the list. ### delete Archive (delete) an agent by name. Updates the agent's status to 'archived'. Returns response to the api request containing the configuration of the deleted agent as `dict`. parameters: * `agent_name` (str): The name of the agent to archive. ## LocalAgents Class for local agent execution and evaluation. parameters: * `base_url` (str): The base URL of the Vijil API. * `evaluation_client` (Evaluations): The Evaluations object. * `api_key_client` (APIKeys): The APIKeys object. ### register Register a local agent with the Vijil API. Used to interact with agents that are not OpenAI-compliant. Interactions occur via an ngrok proxy. Returns a tuple containing the localserver instance and the api key name created for the agent as `tuple[LocalServer, str]`. parameters: * `agent_name` (str): The name of the agent. * `evaluator` (LocalAgentExecutor): The local agent executor to use for evaluation. * `rate_limit` (int, optional): The maximum number of requests to the model hub per rate\_limit\_interval seconds. Defaults to None. * `rate_limit_interval` (int, optional): The interval (in seconds) over which the rate limit is applied. Defaults to None. ### deregister Deregister a local agent with the Vijil API. parameters: * `server` (LocalServer): The local server instance to deregister. * `api_key_name` (str): The name of the API key to delete. ### create ### evaluate Evaluate a local agent. Returns none as `None`. parameters: * `agent_name` (str): The name of the agent. * `evaluation_name` (str): The name of the evaluation. * `agent` (LocalAgentExecutor): The local agent executor instance. * `harnesses` (list): The list of Harnesses to use for evaluation. * `harness_parameters` (dict): The parameters to pass to the Harnesses. * `rate_limit` (int, optional): The maximum number of requests to the model hub per rate\_limit\_interval seconds. Defaults to None. * `rate_limit_interval` (int, optional): The interval (in seconds) over which the rate limit is applied. Defaults to None. * `poll_interval` (float, optional): The interval (in seconds) over which the evaluation status is polled. Defaults to 5.0. * `keep_alive` (bool, optional): If True, the system will be kept awake to allow the evaluation to run. Defaults to False. * `tags` (List\[str], optional): The tags to apply to the evaluation. Defaults to None. ## DomeConfigs :param client: The Vijil client instance. ### get\_config Get the Dome config for a specific agent. Returns the Dome config for the agent as `dict`. parameters: * `agent_id` (str): The unique ID of the agent ### get\_default\_config Get the default Dome config. Returns the default Dome config as `dict`. ### update\_dome\_config Update the Dome config for a specific agent. Returns none as `None`. parameters: * `agent_id` (str): The unique ID of the agent * `config` (dict): The Dome config to set for the agent ### delete\_dome\_config Delete a Dome config by its ID. Returns none as `None`. parameters: * `dome_config_id` (str): The ID of the Dome config to delete ## Vijil Base class for the Vijil API client. parameters: * `base_url` (str): The base URL for the Vijil API * `api_key` (str): The API key for the Vijil API