AI Gateway integration
LiteLLM integration
LiteLLM is an open-source AI gateway. For more information on integrating Aporia with AI gateways, see this guide.
Integration Guide
Installation
To configure LiteLLM with Aporia, start by installing LiteLLM:
pip install 'litellm[proxy]'
For more details, visit LiteLLM - Getting Started guide.
Use LiteLLM AI Gateway with Aporia Guardrails
In this tutorial we will use LiteLLM Proxy with Aporia to detect PII in requests.
1. Setup guardrails on Aporia
Pre-Call: Detect PII
Add the PII - Prompt
to your Aporia project.
2. Define Guardrails on your LiteLLM config.yaml
- Define your guardrails under the
guardrails
section and setpre_call_guardrails
model_list:
- model_name: gpt-3.5-turbo
litellm_params:
model: openai/gpt-3.5-turbo
api_key: os.environ/OPENAI_API_KEY
guardrails:
- guardrail_name: "aporia-pre-guard"
litellm_params:
guardrail: aporia # supported values: "aporia", "lakera"
mode: "during_call"
api_key: os.environ/APORIA_API_KEY_1
api_base: os.environ/APORIA_API_BASE_1
Supported values for mode
pre_call
Run before LLM call, on inputpost_call
Run after LLM call, on input & outputduring_call
Run during LLM call, on input
3. Start LiteLLM Gateway
litellm --config config.yaml --detailed_debug
4. Test request
Expect this to fail since since ishaan@berri.ai
in the request is PII
curl -i http://localhost:4000/v1/chat/completions \
-H "Content-Type: application/json" \
-H "Authorization: Bearer sk-npnwjPQciVRok5yNZgKmFQ" \
-d '{
"model": "gpt-3.5-turbo",
"messages": [
{"role": "user", "content": "hi my email is ishaan@berri.ai"}
],
"guardrails": ["aporia-pre-guard"]
}'
Expected response on failure
{
"error": {
"message": {
"error": "Violated guardrail policy",
"aporia_ai_response": {
"action": "block",
"revised_prompt": null,
"revised_response": "Aporia detected and blocked PII",
"explain_log": null
}
},
"type": "None",
"param": "None",
"code": "400"
}
}
5. Control Guardrails per Project (API Key)
Use this to control what guardrails run per project. In this tutorial we only want the following guardrails to run for 1 project (API Key)
guardrails
: [“aporia-pre-guard”, “aporia”]
Step 1 Create Key with guardrail settings
curl -X POST 'http://0.0.0.0:4000/key/generate' \
-H 'Authorization: Bearer sk-1234' \
-H 'Content-Type: application/json' \
-D '{
"guardrails": ["aporia-pre-guard", "aporia"]
}
}'
Step 2 Test it with new key
curl --location 'http://0.0.0.0:4000/chat/completions' \
--header 'Authorization: Bearer sk-jNm1Zar7XfNdZXp49Z1kSQ' \
--header 'Content-Type: application/json' \
--data '{
"model": "gpt-3.5-turbo",
"messages": [
{
"role": "user",
"content": "my email is ishaan@berri.ai"
}
]
}'