Build your own custom policy by writing a prompt.
You can create custom policies from the Policy Catalog page. When you create a new custom policy you will see the configuration page, where you can define the prompt and any additional configuration:
When configuring custom policies, you can choose to use either “simple” or “advanced” configuration (for more control over the final results).
Either way, you must select a target
and a modality
for your policy.
target
is either prompt
or response
, and determines if the policy should run on prompts or responses, respectively. Note that if any of the extractions in the evaluation instructions or system prompt run on the response, then the policy target must also be response
modality
is either legit
or violate
, and determines how the response from the LLM (which is always TRUE
or FALSE
) will be interpreted. In legit
modality, a TRUE
response means the message is legitimate and there are no issues, while a FALSE
response means there is an issue with the checked message. In violate
modality, the opposite is true.In simple mode, you must specify evaluation instructions that will be appended to a system prompt provided by Aporia. Extractions can be used to refer to parts of the message the policy is checking, but only the {question}
, {context}
and {answer}
extractions are supported.
Extractions in the evaluation instructions should be used as though they were regular words (unlike advanced mode, in which extractions are replaced by the extracted content at runtime).
In advanced mode, you must specify a full system prompt that will be sent to the LLM.
TRUE
or FALSE
.{extraction}
tag will be replaced with the actual content extracted from the message that is being checked.Additionally, you may select the temperature
and top_p
for the LLM.
To use an extraction in a custom policy, use the following syntax in the evaluation instructions or system prompt: {extraction_descriptor}
, where extraction_descriptor
can be any extraction that is configured for your projects (e.g. {question}
, {answer}
).
If you want the text to contain the string {extraction_descriptor}
without being treated as an extraction, you can escape it as follows: {{extraction_descriptor}}