Learning exercise
DeepEval — agent and RAG evaluation
An evaluation lab covering agent task completion, tool correctness, retrieval quality, grounded answers, and multi-turn conversation behavior.
Learning exercise focused on evaluation design and tracing. Metrics and example cases are implemented; benchmark scores are not claimed.
System architecture
On smaller screens, scroll the diagram horizontally or read the text version below.
Solid arrows: application and data flow.
Read components and connections as text
Golden inputs and expected behavior feed evaluation runners. Separate support-agent, RAG-agent, and multi-turn chatbot paths exercise tools, policy retrieval, and conversation history. Agent and RAG callbacks capture traces; chatbot tests capture turns. Metric suites use this evidence and configured graders to produce per-case scores and reasons, with no benchmark result claimed here.
- Golden test cases: Order / refund / policy inputs. Small example datasets.
- Evaluation runners: Dataset evaluation iterator. Conversation evaluate().
- Expected behavior: Expected tool names. Reference answers for RAG.
- Support agent: LangChain + OpenAI. Instrumented tool-calling agent.
- RAG support agent: LangChain + OpenAI. search_policies tool.
- Multi-turn chatbot: OpenAI SDK tool loop. Keeps history between turns.
- Support tools: Order status + refund lookups. MCP shipping-options tool.
- Policy retrieval: 9 policies · OpenAI embeddings. InMemoryVectorStore · top 3.
- Conversation history: User / assistant messages. Tool calls and tool results.
- Captured evidence: Callbacks + observe → traces. Chat turns + retrieval context.
- OpenAI judge models: LLM-based metric grading. Used by configured metrics.
- DeepEval metric suites: Task / tools · RAG quality. Conversation quality.
- Evaluation output: Per-case scores + reasons. Configured metric thresholds.
- Golden test cases → Evaluation runners: test inputs.
- Expected behavior → Evaluation runners: expectations.
- Evaluation runners → Support agent: agent cases.
- Evaluation runners → RAG support agent: RAG cases.
- Evaluation runners → Multi-turn chatbot: conversation turns.
- Support agent ↔ Support tools: tool calls + results.
- RAG support agent ↔ Policy retrieval: semantic search.
- Multi-turn chatbot ↔ Conversation history: retain turns.
- Support agent → Captured evidence: model + tool spans.
- RAG support agent → Captured evidence: reply + context.
- Multi-turn chatbot → Captured evidence: captured turns.
- Captured evidence → DeepEval metric suites: actual + expected.
- DeepEval metric suites ↔ OpenAI judge models: LLM grading.
- DeepEval metric suites → Evaluation output: scores + reasons.
Evaluation focus
A plausible response does not establish that an agent completed the task or selected the correct tool. This learning lab separates those questions and extends evaluation to retrieved context, grounded answers, and multi-turn conversations.
Agent tracing and tools
Plain and instrumented support agents show how tracing is added around model and tool calls. Golden cases specify expected tool names for order-status and refund-policy questions. DeepEval's Task Completion and Tool Correctness metrics assess the resulting traces as separate evaluation targets.
RAG and conversation evaluation
The RAG evaluation script defines Contextual Precision, Contextual Recall, Answer Relevancy, and Faithfulness metrics against policy questions with expected answers. A separate multi-turn example evaluates Turn Relevancy, Knowledge Retention, and Conversation Completeness across an order-support conversation.
Evidence and next steps
The repository demonstrates evaluation wiring and small example datasets. The inspected agent-tracing and RAG scripts each contain two golden cases, so they do not establish broad reliability. Next steps are a larger reviewed dataset, calibrated grading criteria, and saved results comparing two configurations. No measured quality improvement or passing score is claimed.