Skip to content

Commit db7c3b8

Browse files
vmmadathilVisakh Madathil
vmmadathil
and
Visakh Madathil
authored
Add third-party (DeepEval/AutoEval) evaluators sample (#1978)
Add a 3p-evals sample under llm-as-a-judge-evaluation showing how to run DeepEval and AutoEval metrics with AgentCore Evaluations, both as managed evaluators (evaluatorType=ThirdParty) and as a custom-derived evaluator that runs a base metric on a Bedrock model of your choice. The sample reuses the shared HR Assistant agent and covers discovery via ListEvaluators, on-demand evaluation mixing managed 3p / derived / built-in metrics, and an online config using reference-free 3p metrics. Link it from the parent README's Next Steps. Co-authored-by: Visakh Madathil <visakhm@amazon.com>
1 parent a647f7f commit db7c3b8

5 files changed

Lines changed: 675 additions & 0 deletions

File tree

‎01-features/06-observe-evaluate-optimize-your-agent/02-evaluate/llm-as-a-judge-evaluation/3p-evals/.gitignore‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,2 @@
1+
results/
2+
__pycache__/

‎01-features/06-observe-evaluate-optimize-your-agent/02-evaluate/llm-as-a-judge-evaluation/3p-evals/README.md‎

Lines changed: 183 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,183 @@
1+
# Third-party evaluators
2+
3+
AgentCore Evaluations can run metrics from the open-source **DeepEval** and **AutoEval** libraries. This sample builds on the [parent LLM-as-a-judge sample](../README.md) and reuses the same HR Assistant agent.
4+
5+
> **Evaluator quality:** AgentCore built-in evaluators are tested and benchmarked. DeepEval and AutoEval are open-source, and AWS makes no claims about their quality.
6+
7+
## Two ways to use them
8+
9+
| Approach | `evaluatorType` | Who runs the model | You configure |
10+
|---|---|---|---|
11+
| **Managed** | `ThirdParty` | AWS (service capacity) | Just the ID |
12+
| **Custom-derived** | `CustomDerived` | You (your account + credentials) | The Bedrock model + inference config |
13+
14+
- **Managed** — reference an evaluator by ID (e.g. `ThirdParty.DeepEval.TaskCompletion`). There's no model field and no version to pick; the service runs the library version it has validated. A managed ID works wherever a `Builtin.*` ID does: on-demand, online, or batch.
15+
- **Custom-derived** — reuse a base evaluator's prompt and scoring (a built-in or a managed 3p evaluator) but run it on a model you choose. The base owns the prompt and scale; you supply the model. LLM-based evaluators only.
16+
17+
## Evaluator identity
18+
19+
Every evaluator is one `(evaluatorType, provider)` pair:
20+
21+
| Evaluator | `evaluatorType` | `provider` |
22+
|---|---|---|
23+
| Managed built-in | `Builtin` | `AWS` |
24+
| Managed third-party | `ThirdParty` | `DeepEval` or `AutoEval` |
25+
| Derived from a built-in | `CustomDerived` | `AWS` |
26+
| Derived from a third-party | `CustomDerived` | `DeepEval` or `AutoEval` |
27+
28+
Managed third-party IDs follow `ThirdParty.<Provider>.<Metric>` (for example `ThirdParty.DeepEval.TaskCompletion` or `ThirdParty.AutoEval.Security`), mirroring the `Builtin.<Metric>` format.
29+
30+
## Available metrics (initial set)
31+
32+
**DeepEval**
33+
34+
| Metric | What it checks |
35+
|---|---|
36+
| `Bias` | Gender, political, racial, or geographical bias |
37+
| `Toxicity` | Attacks, mockery, hate, or threats |
38+
| `PIILeakage` | Whether the response exposes personal information |
39+
| `Summarization` | Whether a summary is faithful and comprehensive |
40+
| `TaskCompletion` | Whether the agent accomplished the user's goal |
41+
| `ConversationCompleteness` | Whether all requests across the conversation were addressed |
42+
| `KnowledgeRetention` | Whether the agent remembered information shared earlier |
43+
| `TurnRelevancy` | Whether each reply stays relevant to prior turns |
44+
| `GoalAccuracy` | Whether goals were achieved across a multi-turn conversation |
45+
| `ToolUse` | Whether the agent picked the right tool with correct arguments |
46+
47+
**AutoEval**
48+
49+
| Metric | What it checks |
50+
|---|---|
51+
| `Security` | Whether the response is malicious |
52+
| `Humor` | Whether the response is funny |
53+
| `Possible` | Whether the agent attempted a solution or declared the task impossible |
54+
55+
Step 1 of the script lists what's active in your account with `ListEvaluators`.
56+
57+
---
58+
59+
## Prerequisites
60+
61+
**1. Enable CloudWatch Transaction Search** (once per account/region). On-demand evaluation reads *span documents* from the `aws/spans` log group, which only exists once Transaction Search is on. Without it, `Evaluate` fails with `no span documents … ensure that transaction search is enabled`. Enable it in the CloudWatch console (Application Signals → Transaction Search), or via API:
62+
63+
```bash
64+
# X-Ray needs a Logs resource policy to write span documents into aws/spans
65+
aws logs put-resource-policy \
66+
--policy-name TransactionSearchAccess \
67+
--policy-document '{"Version":"2012-10-17","Statement":[{"Effect":"Allow","Principal":{"Service":"xray.amazonaws.com"},"Action":"logs:PutLogEvents","Resource":["arn:aws:logs:<region>:<account>:log-group:aws/spans:*","arn:aws:logs:<region>:<account>:log-group:/aws/application-signals/data:*"],"Condition":{"StringEquals":{"aws:SourceAccount":"<account>"},"ArnLike":{"aws:SourceArn":"arn:aws:xray:<region>:<account>:*"}}}]}'
68+
69+
aws xray update-trace-segment-destination --destination CloudWatchLogs
70+
aws xray update-indexing-rule --name Default --rule '{"Probabilistic":{"DesiredSamplingPercentage":100.0}}'
71+
```
72+
73+
Enabling takes a few minutes to go `ACTIVE`, and span indexing lags emission — right after enabling, allow extra time before evaluating a fresh session (this sample waits 180s).
74+
75+
**2. Deploy the shared HR Assistant agent** (runs once for all `02-evaluate/` samples):
76+
77+
```bash
78+
cd ../../utils
79+
python deploy.py
80+
```
81+
82+
This writes `utils/agent_config.json`, which `evaluate.py` reads automatically.
83+
84+
## Run the evaluation
85+
86+
```bash
87+
pip install -r requirements.txt
88+
python evaluate.py
89+
```
90+
91+
Optional flags:
92+
93+
```bash
94+
python evaluate.py --region us-west-2
95+
python evaluate.py --config /path/to/agent_config.json
96+
```
97+
98+
## What the script does
99+
100+
### Step 1 — Discover third-party evaluators
101+
102+
Calls `ListEvaluators`, filters to `evaluatorType == "ThirdParty"`, and prints them grouped by provider. It also captures each evaluator's **level** (`TRACE` / `SESSION`) — needed later because the SDK cannot resolve the level of AWS-managed global evaluators via `GetEvaluator`.
103+
104+
### Step 2 — Create a custom evaluator derived from a 3p base
105+
106+
Creates a `CustomDerived` evaluator from `ThirdParty.DeepEval.TaskCompletion` that runs on a Bedrock model you choose:
107+
108+
```python
109+
_cp.create_evaluator(
110+
evaluatorName="MyTaskCompletion_<suffix>",
111+
level="TRACE", # must match the base evaluator's level
112+
evaluatorConfig={
113+
"derived": {
114+
"baseEvaluatorId": "ThirdParty.DeepEval.TaskCompletion",
115+
"modelConfig": {
116+
"bedrockEvaluatorModelConfig": {
117+
"modelId": "us.amazon.nova-lite-v1:0",
118+
"inferenceConfig": {"temperature": 0.0, "topP": 1.0, "maxTokens": 2048},
119+
}
120+
},
121+
}
122+
},
123+
)
124+
```
125+
126+
Don't set `instructions` or `ratingScale`: the base evaluator owns both, and `provider` comes from the base (`DeepEval` here). Note that although the docs describe `level` as derived from the base, the API validates it as required, so pass the base evaluator's level (the script reuses the level discovered in step 1).
127+
128+
> **Inference ownership:** a derived evaluator's model runs in your account with your credentials (the caller's for on-demand, the execution role for online), rather than on service capacity as a managed evaluator does. Since you pick the model, evaluation quality reflects that choice.
129+
130+
### Step 3 — Invoke the HR Assistant
131+
132+
Sends three turns to the deployed HR Assistant to generate a CloudWatch session, then waits for span ingestion.
133+
134+
### Step 4 — On-demand evaluation (`EvaluationClient`)
135+
136+
Managed 3p, derived, and built-in evaluators run in one call. All metrics chosen here are **reference-free**, so no ground truth is needed:
137+
138+
| Evaluator | Type | Level |
139+
|---|---|---|
140+
| `ThirdParty.DeepEval.TaskCompletion` | managed 3p | discovered |
141+
| `ThirdParty.DeepEval.Toxicity` | managed 3p | discovered |
142+
| `ThirdParty.AutoEval.Security` | managed 3p | discovered |
143+
| `MyTaskCompletion` (derived) | `CustomDerived` | inherited from base |
144+
| `Builtin.Helpfulness` | built-in | `TRACE` |
145+
146+
Results are saved to `results/on_demand_results.json`.
147+
148+
### Step 5 — Online evaluation with reference-free 3p metrics
149+
150+
Creates an online config using `ThirdParty.DeepEval.Toxicity` and `ThirdParty.AutoEval.Security` against live traffic.
151+
152+
> **Why these go online but the parent sample's custom evaluators can't:** the parent sample's evaluators use reference-input placeholders (`{expected_response}`, `{assertions}`) that need ground truth, which live traffic doesn't have, so they stay on-demand only. Many managed 3p metrics (Toxicity, Bias, PIILeakage, `AutoEval.Security`) need no reference input, so they can go on an online config.
153+
154+
Configuration details are saved to `results/online_eval_config.json`.
155+
156+
## Results files
157+
158+
| File | Contents |
159+
|---|---|
160+
| `results/discovered_evaluators.json` | Third-party evaluators returned by `ListEvaluators` |
161+
| `results/on_demand_results.json` | Managed + derived + built-in scores for the session |
162+
| `results/online_eval_config.json` | Online config ID, ARN, and evaluators |
163+
164+
## Managing the online evaluation config
165+
166+
```bash
167+
# Disable
168+
aws bedrock-agentcore-control update-online-evaluation-config \
169+
--online-evaluation-config-id <config-id-from-results/online_eval_config.json> \
170+
--enable-config false
171+
172+
# Delete when no longer needed
173+
aws bedrock-agentcore-control delete-online-evaluation-config \
174+
--online-evaluation-config-id <config-id>
175+
```
176+
177+
---
178+
179+
## Additional resources
180+
181+
- [Third-party evaluators — Developer Guide](https://docs.aws.amazon.com/bedrock-agentcore/latest/devguide/third-party-evaluators.html)
182+
- [Parent sample: LLM-as-a-judge evaluation](../README.md)
183+
- [Amazon Bedrock AgentCore Developer Guide](https://docs.aws.amazon.com/bedrock-agentcore/latest/devguide/)

0 commit comments

Comments
 (0)