Cases & Fixes

⚠️

Migration Notice: This guide uses the legacy Cases API (/api/v1/cases). The Cases API has been renamed to the Issues & Fixes API (/api/v2/issues). See the Issues & Fixes API Reference for current endpoints. The concepts and workflows below still apply — only the endpoint paths have changed.

ThinkHive automatically clusters similar agent failures into Cases (now called Issues) and generates AI-powered Fix proposals that you can validate through shadow testing before deploying.

How Cases Work

Automatic Case Creation

When ThinkHive detects a cluster of similar failures, it automatically creates a case:

Failure Detection — Traces are analyzed for errors, low-quality responses, or evaluation failures
Clustering — Similar failures are grouped using semantic similarity
Case Creation — A case is created with a description, severity, and representative traces
Pattern Extraction — Common patterns across the clustered traces are identified

Cases are created automatically when the failure pattern reaches a configurable threshold (default: 3 similar failures).

Viewing Cases

In the Dashboard

Navigate to Cases in the sidebar to see all open cases. Each case shows:

Title — Auto-generated description of the failure pattern
Severity — critical, high, medium, or low
Trace Count — Number of affected traces
Status — open, investigating, fix_proposed, resolved
First/Last Seen — Time range of affected traces

Via API

# List all cases
curl "https://app.thinkhive.ai/api/v1/cases" \
  -H "Authorization: Bearer thk_your_api_key"
 
# Get a specific case with its traces
curl "https://app.thinkhive.ai/api/v1/cases/case_abc123" \
  -H "Authorization: Bearer thk_your_api_key"

Response:

{
  "id": "case_abc123",
  "title": "Incomplete password reset instructions for MFA users",
  "severity": "high",
  "status": "open",
  "traceCount": 47,
  "pattern": "Agent omits 2FA reset step when user has MFA enabled",
  "firstSeen": "2025-01-10T08:00:00Z",
  "lastSeen": "2025-01-15T14:30:00Z",
  "representativeTraces": ["trace_1", "trace_2", "trace_3"],
  "suggestedFix": {
    "id": "fix_def456",
    "description": "Add conditional step for MFA users in password reset flow",
    "confidence": 0.87
  }
}

Case Lifecycle

Status	Description	Actions Available
`open`	New case, not yet reviewed	Assign, investigate, generate fix
`investigating`	Under active review	Add notes, link traces, generate fix
`fix_proposed`	AI-generated fix available	Review fix, run shadow test
`testing`	Shadow test in progress	Monitor test results
`resolved`	Fix deployed and verified	Close, reopen if recurring
`dismissed`	Not a real issue	Reopen if needed

Working with Fixes

AI-Generated Fixes

ThinkHive analyzes the clustered failure traces and proposes fixes:

# Generate a fix for a case
curl -X POST "https://app.thinkhive.ai/api/v1/cases/case_abc123/fixes" \
  -H "Authorization: Bearer thk_your_api_key" \
  -H "Content-Type: application/json" \
  -d '{
    "strategy": "auto"
  }'

Fix strategies:

Strategy	Description
`auto`	Let ThinkHive choose the best approach
`prompt_update`	Suggest prompt modifications
`context_enrichment`	Recommend additional context or retrieval
`guardrail`	Add validation or safety checks

Fix Response

{
  "id": "fix_def456",
  "caseId": "case_abc123",
  "strategy": "prompt_update",
  "description": "Add conditional step for MFA users in password reset flow",
  "changes": [
    {
      "type": "prompt_addition",
      "content": "If the user has MFA/2FA enabled, include steps to reset their authenticator app or backup codes.",
      "position": "after_step_3"
    }
  ],
  "confidence": 0.87,
  "evidence": [
    "47 traces show users with MFA enabled receiving incomplete instructions",
    "All affected traces lack mention of authenticator reset"
  ]
}

Validating Fixes with Shadow Testing

Before deploying a fix, validate it with shadow testing:

Create a shadow test

curl -X POST "https://app.thinkhive.ai/api/v1/shadow-tests" \
  -H "Authorization: Bearer thk_your_api_key" \
  -H "Content-Type: application/json" \
  -d '{
    "fixId": "fix_def456",
    "caseId": "case_abc123",
    "traceIds": ["trace_1", "trace_2", "trace_3"],
    "evaluationSuiteId": "suite_xyz"
  }'

Monitor the test

curl "https://app.thinkhive.ai/api/v1/shadow-tests/st_789" \
  -H "Authorization: Bearer thk_your_api_key"

Review results

The shadow test compares the original agent behavior against the fixed version:

{
  "id": "st_789",
  "status": "completed",
  "results": {
    "baseline": { "averageScore": 2.1, "passRate": 0.15 },
    "withFix": { "averageScore": 4.3, "passRate": 0.89 },
    "improvement": "+74% pass rate"
  }
}

Deploy or iterate

If the shadow test passes, mark the case as resolved. If not, iterate on the fix.

SDK Integration

import { ThinkHive } from 'thinkhive-js';
 
const th = new ThinkHive({
  apiKey: process.env.THINKHIVE_API_KEY,
  endpoint: 'https://app.thinkhive.ai',
  serviceName: 'my-agent'
});
 
// List open cases
const cases = await th.cases.list({ status: 'open' });
 
// Get case details
const caseDetail = await th.cases.get('case_abc123');
 
// Generate a fix
const fix = await th.cases.generateFix('case_abc123', {
  strategy: 'auto'
});

Best Practices

Review cases regularly — check the Cases dashboard daily to catch emerging issues early
Set severity thresholds — configure auto-escalation for critical and high-severity cases
Use shadow testing — always validate fixes before deploying to production
Link to evaluation suites — connect cases to ThinkEval criteria for automated quality gates
Track resolution time — monitor mean time to resolution (MTTR) as a platform KPI

Next Steps

Shadow Testing — Detailed shadow testing guide
ThinkEval — Evaluation framework for quality measurement
Drift Monitoring — Detect quality regressions proactively
API Reference: Issues & Fixes — Full endpoint documentation

ThinkEval RAG Evaluation