GuidesDrift Monitoring

Drift Monitoring Guide

Track quality changes and detect drift in your AI agents over time.

What is Drift?

Drift occurs when an AI agent’s behavior changes unexpectedly:

  • Model Drift: Changes in underlying model performance
  • Data Drift: Changes in input data patterns
  • Concept Drift: Changes in the relationship between inputs and outputs

Types of Drift

Performance Drift

Quality metrics declining over time:

  • Decreasing success rates
  • Increasing hallucinations
  • Lower groundedness scores

Latency Drift

Response times changing:

  • Increased p95 latency
  • More timeouts
  • Slower retrieval

Topic Drift

Changes in query patterns:

  • New question types
  • Seasonal variations
  • Product changes

Setting Up Monitoring

// Get drift analysis
const drift = await api.get('/api/v1/drift/analyze', {
  params: {
    agentId: 'agent_123',
    baselinePeriod: '7d',
    comparisonPeriod: '1d'
  }
});
 
console.log(drift.data);
// {
//   detected: true,
//   metrics: {
//     successRate: { baseline: 0.85, current: 0.78, change: -0.07 },
//     groundedness: { baseline: 0.82, current: 0.75, change: -0.07 },
//     latencyP95: { baseline: 2000, current: 2800, change: +0.40 }
//   },
//   alerts: [
//     'Success rate dropped 7% below baseline',
//     'Groundedness declined significantly'
//   ]
// }

Alert Configuration

// Configure drift alerts
await api.post('/api/v1/drift/alerts', {
  agentId: 'agent_123',
  thresholds: {
    successRate: { minChange: -0.05 },
    groundedness: { minChange: -0.10 },
    latencyP95: { maxChange: 0.50 }
  },
  webhook: 'https://your-app.com/alerts'
});

Responding to Drift

1. Investigate Root Cause

  • Check for model updates
  • Review recent data changes
  • Analyze failed traces

2. Compare Time Periods

const comparison = await api.get('/api/v1/drift/compare', {
  params: {
    agentId: 'agent_123',
    period1Start: '2024-01-01',
    period1End: '2024-01-07',
    period2Start: '2024-01-08',
    period2End: '2024-01-14'
  }
});

3. Take Action

  • Update knowledge base
  • Retrain models
  • Adjust prompts
  • Roll back changes
⚠️

Monitor These Metrics

  • Success/failure rate
  • RAG quality scores
  • Hallucination frequency
  • Response latency
  • Token usage patterns

Next Steps

Hallucination DetectionShadow Testing