Claude Code Integration

User: Analyze trace tr_abc123 and tell me what went wrong

Claude: I'll analyze that trace for you.

[Uses trace_explain tool]

The trace shows a customer support interaction where the agent
failed to provide accurate information about the refund policy.

Key findings:
- Groundedness score: 0.45 (low - response not well supported)
- Hallucination detected: Fabricated 30-day return policy
- Root cause: Missing context about product-specific policies

Recommended fix: Add product category to the retrieval query
to get the correct policy document.

User: What are the most common failure patterns for my support agent?

Claude: Let me check the failure patterns.

[Uses trace_patterns tool]

I found 5 failure clusters for your support agent:

1. **Missing Policy Information** (47 traces)
   - Users asking about policies get incomplete answers
   - Suggested fix: Improve policy document retrieval

2. **Authentication Guidance Errors** (32 traces)
   - Password reset instructions are incorrect
   - Suggested fix: Update authentication knowledge base

3. **Shipping Estimate Hallucinations** (28 traces)
   - Agent fabricates delivery dates
   - Suggested fix: Add real-time shipping API integration

...

User: Generate a fix for the authentication guidance cluster

Claude: I'll generate a fix for that cluster.

[Uses fix_generate tool]

Here's a proposed fix for the Authentication Guidance Errors cluster:

**Problem**: The agent provides incorrect password reset steps,
missing the 2FA verification step for users with MFA enabled.

**Proposed Fix**:
1. Update the knowledge base document "password-reset.md"
2. Add conditional logic check for MFA status
3. Include step: "If 2FA is enabled, verify with your authenticator app"

**Expected Impact**:
- Reduce authentication failures by ~40%
- Improve CSAT for auth-related queries

Would you like me to create a pull request with this fix?

User: What's the overall quality of my RAG pipeline?

Claude: Let me analyze your RAG pipeline quality.

[Uses quality_get_scores tool]

**RAG Quality Report for support-agent (Last 30 days)**

| Metric | Score | Trend |
|--------|-------|-------|
| Groundedness | 0.78 | +0.05 |
| Faithfulness | 0.82 | +0.03 |
| Citation Accuracy | 0.65 | -0.02 |
| Context Relevance | 0.71 | +0.08 |

**Summary**: Overall quality is good but citation accuracy
is declining. Recommend reviewing the citation prompt template.

**Top Issues**:
1. Missing citations in 23% of responses
2. Incorrect page numbers in PDF citations
3. Outdated document references

User: What's the revenue impact of improving our agent?

Claude: Let me calculate the business impact.

[Uses roi_calculate_impact tool]

**ROI Analysis for 10% Quality Improvement**

Current State:
- Success rate: 78%
- Avg ticket cost: $45
- Monthly tickets: 10,000

Projected Impact:
- New success rate: 85.8%
- Tickets deflected: 780/month
- Monthly savings: $35,100
- Annual savings: $421,200

Break-even time: 2.3 months

The improvements would pay for themselves in under 3 months.

# Good - specific trace or agent
User: Analyze trace tr_abc123

# Less helpful - too vague
User: Analyze my traces

User: Show me failure patterns for the last week
User: What's the quality trend over the past month?

User: Find the biggest failure cluster, analyze a sample trace,
and generate a fix

Claude: I'll do all three...
[Uses cluster_list, trace_explain, fix_generate]

User: That fix looks good. Can you show me the actual traces
that would be affected?

Claude: [Uses trace_search to find related traces]

User: Use the quality-report prompt for agent support-agent

User: Analyze all failed traces from yesterday and summarize
the common issues

Claude: [Uses trace_batch_analyze and cluster_list]

User: Export the quality metrics as a CSV

Claude: [Uses quality_get_scores and formats as CSV]