Issues & Fixes API

⚠️

Migration Notice: This API replaces the deprecated Cases API. If you are still using /api/v2/issues through the old Cases interface, your requests will continue to work, but we recommend updating your integration to follow the patterns documented here. The old Cases API page will be removed in a future release.

Create and manage issues that cluster related agent failures, propose fixes with prompt or configuration changes, and validate those fixes through shadow test execution.

Issues

List Issues

GET /api/v2/issues

Headers:

Authorization: Bearer thk_your_api_key

Query Parameters:

Parameter	Type	Description
`agentId`	string	Filter by agent ID
`status`	string	`open`, `proposed_fix`, `testing`, `closed`
`severity`	string	`low`, `medium`, `high`, `critical`
`page`	number	Page number (default: 1)
`limit`	number	Items per page (default: 20, max: 100)
`sortBy`	string	Field to sort by (`createdAt`, `severity`, `traceCount`)
`sortOrder`	string	`asc` or `desc` (default: `desc`)

Example:

GET /api/v2/issues?agentId=agent_abc123&severity=high&status=open&limit=10

Response:

{
  "success": true,
  "data": [
    {
      "id": "iss_8f3k2m",
      "title": "Incorrect refund policy citations",
      "description": "Agent cites outdated refund policy when customers ask about returns after 30 days",
      "severity": "high",
      "status": "open",
      "traceCount": 47,
      "agentId": "agent_abc123",
      "agentName": "Customer Support Bot",
      "tags": ["retrieval", "policy", "refunds"],
      "createdAt": "2025-08-12T14:30:00Z",
      "updatedAt": "2025-08-14T09:15:00Z"
    },
    {
      "id": "iss_9x7p4n",
      "title": "Hallucinated product features",
      "description": "Agent invents product capabilities not present in the knowledge base",
      "severity": "critical",
      "status": "proposed_fix",
      "traceCount": 23,
      "agentId": "agent_abc123",
      "agentName": "Customer Support Bot",
      "tags": ["hallucination", "product-info"],
      "createdAt": "2025-08-10T08:00:00Z",
      "updatedAt": "2025-08-13T16:45:00Z"
    }
  ],
  "pagination": {
    "page": 1,
    "limit": 10,
    "total": 2,
    "hasMore": false
  }
}

Get Issue

GET /api/v2/issues/:id

Response:

{
  "success": true,
  "data": {
    "id": "iss_8f3k2m",
    "title": "Incorrect refund policy citations",
    "description": "Agent cites outdated refund policy when customers ask about returns after 30 days",
    "severity": "high",
    "status": "open",
    "traceCount": 47,
    "agentId": "agent_abc123",
    "agentName": "Customer Support Bot",
    "tags": ["retrieval", "policy", "refunds"],
    "traceIds": ["tr_a1b2c3", "tr_d4e5f6", "tr_g7h8i9"],
    "fixes": [],
    "metadata": {
      "firstSeenAt": "2025-08-10T11:22:00Z",
      "lastSeenAt": "2025-08-14T09:15:00Z",
      "affectedCustomers": 38,
      "clusterConfidence": 0.92
    },
    "createdAt": "2025-08-12T14:30:00Z",
    "updatedAt": "2025-08-14T09:15:00Z"
  }
}

Create Issue

POST /api/v2/issues

Request Body:

{
  "title": "Shipping estimate miscalculations",
  "description": "Agent provides incorrect delivery date estimates for international orders, consistently understating transit time by 3-5 days",
  "agentId": "agent_abc123",
  "severity": "medium",
  "traceIds": ["tr_x1y2z3", "tr_m4n5o6", "tr_p7q8r9"],
  "tags": ["shipping", "international", "estimates"]
}

Response:

{
  "success": true,
  "data": {
    "id": "iss_4r7t2w",
    "title": "Shipping estimate miscalculations",
    "description": "Agent provides incorrect delivery date estimates for international orders, consistently understating transit time by 3-5 days",
    "severity": "medium",
    "status": "open",
    "traceCount": 3,
    "agentId": "agent_abc123",
    "tags": ["shipping", "international", "estimates"],
    "createdAt": "2025-08-15T10:00:00Z",
    "updatedAt": "2025-08-15T10:00:00Z"
  }
}

Update Issue

PATCH /api/v2/issues/:id

Request Body:

{
  "severity": "high",
  "status": "proposed_fix",
  "tags": ["shipping", "international", "estimates", "escalated"]
}

All fields are optional. Only the provided fields will be updated.

Response:

{
  "success": true,
  "data": {
    "id": "iss_4r7t2w",
    "title": "Shipping estimate miscalculations",
    "severity": "high",
    "status": "proposed_fix",
    "tags": ["shipping", "international", "estimates", "escalated"],
    "updatedAt": "2025-08-15T11:30:00Z"
  }
}

Delete Issue

DELETE /api/v2/issues/:id

Response:

{
  "success": true,
  "message": "Issue deleted successfully"
}

Deleting an issue does not delete associated traces. Linked fixes and shadow tests will be disassociated but not removed.

Fixes

List Fixes

GET /api/fixes

Query Parameters:

Parameter	Type	Description
`issueId`	string	Filter by parent issue
`status`	string	`draft`, `proposed`, `testing`, `applied`, `rejected`
`type`	string	`prompt_update`, `config_change`, `retrieval_update`, `guardrail`
`page`	number	Page number (default: 1)
`limit`	number	Items per page (default: 20, max: 100)

Example:

GET /api/fixes?issueId=iss_8f3k2m&status=proposed

Response:

{
  "success": true,
  "data": [
    {
      "id": "fix_3a9c7e",
      "issueId": "iss_8f3k2m",
      "type": "prompt_update",
      "status": "proposed",
      "description": "Add explicit instruction to cross-reference refund policy effective dates before citing",
      "changes": {
        "promptDiff": "- Answer refund questions using the knowledge base.\n+ Answer refund questions using the knowledge base. Always verify the policy effective date matches the customer's purchase date before citing specific terms."
      },
      "author": "user_abc123",
      "createdAt": "2025-08-13T16:00:00Z",
      "updatedAt": "2025-08-13T16:00:00Z"
    }
  ],
  "pagination": {
    "page": 1,
    "limit": 20,
    "total": 1,
    "hasMore": false
  }
}

Get Fix

GET /api/fixes/:id

Response:

{
  "success": true,
  "data": {
    "id": "fix_3a9c7e",
    "issueId": "iss_8f3k2m",
    "type": "prompt_update",
    "status": "testing",
    "description": "Add explicit instruction to cross-reference refund policy effective dates before citing",
    "changes": {
      "promptDiff": "- Answer refund questions using the knowledge base.\n+ Answer refund questions using the knowledge base. Always verify the policy effective date matches the customer's purchase date before citing specific terms."
    },
    "shadowTests": [
      {
        "id": "st_7k2m9p",
        "status": "completed",
        "improvementRate": 0.85
      }
    ],
    "author": "user_abc123",
    "createdAt": "2025-08-13T16:00:00Z",
    "updatedAt": "2025-08-14T10:30:00Z"
  }
}

Create Fix

POST /api/fixes

Request Body:

{
  "issueId": "iss_9x7p4n",
  "type": "prompt_update",
  "description": "Add grounding instruction to prevent hallucinated feature claims",
  "changes": {
    "promptDiff": "- Help customers understand our product capabilities.\n+ Help customers understand our product capabilities. Only describe features explicitly listed in the provided product documentation. If a customer asks about a feature not in your context, say you are not sure and offer to connect them with a product specialist."
  }
}

Field	Type	Required	Description
`issueId`	string	Yes	The issue this fix addresses
`type`	string	Yes	One of: `prompt_update`, `config_change`, `retrieval_update`, `guardrail`
`description`	string	Yes	Human-readable explanation of the fix
`changes`	object	Yes	The actual changes to apply (structure varies by type)

Response:

{
  "success": true,
  "data": {
    "id": "fix_6b4d8f",
    "issueId": "iss_9x7p4n",
    "type": "prompt_update",
    "status": "draft",
    "description": "Add grounding instruction to prevent hallucinated feature claims",
    "changes": {
      "promptDiff": "- Help customers understand our product capabilities.\n+ Help customers understand our product capabilities. Only describe features explicitly listed in the provided product documentation. If a customer asks about a feature not in your context, say you are not sure and offer to connect them with a product specialist."
    },
    "createdAt": "2025-08-14T12:00:00Z",
    "updatedAt": "2025-08-14T12:00:00Z"
  }
}

Update Fix

PATCH /api/fixes/:id

Request Body:

{
  "status": "applied",
  "description": "Updated grounding instruction with stricter guardrail language"
}

All fields are optional.

Response:

{
  "success": true,
  "data": {
    "id": "fix_6b4d8f",
    "status": "applied",
    "description": "Updated grounding instruction with stricter guardrail language",
    "updatedAt": "2025-08-16T09:00:00Z"
  }
}

Shadow Tests

Shadow tests replay historical traces against a proposed fix to measure improvement before applying changes to production.

List Shadow Tests

GET /api/shadow-tests

Query Parameters:

Parameter	Type	Description
`fixId`	string	Filter by fix ID
`status`	string	`pending`, `running`, `completed`, `failed`
`page`	number	Page number (default: 1)
`limit`	number	Items per page (default: 20, max: 100)

Response:

{
  "success": true,
  "data": [
    {
      "id": "st_7k2m9p",
      "fixId": "fix_3a9c7e",
      "status": "completed",
      "traceCount": 10,
      "config": {
        "comparisonMode": "side_by_side"
      },
      "results": {
        "improved": 8,
        "unchanged": 1,
        "regressed": 1,
        "improvementRate": 0.8
      },
      "createdAt": "2025-08-14T10:00:00Z",
      "completedAt": "2025-08-14T10:25:00Z"
    }
  ],
  "pagination": {
    "page": 1,
    "limit": 20,
    "total": 1,
    "hasMore": false
  }
}

Get Shadow Test

GET /api/shadow-tests/:id

Response:

{
  "success": true,
  "data": {
    "id": "st_7k2m9p",
    "fixId": "fix_3a9c7e",
    "status": "completed",
    "traceCount": 10,
    "config": {
      "comparisonMode": "side_by_side"
    },
    "results": {
      "improved": 8,
      "unchanged": 1,
      "regressed": 1,
      "improvementRate": 0.8
    },
    "traceResults": [
      {
        "traceId": "tr_a1b2c3",
        "verdict": "improved",
        "originalScore": 0.35,
        "fixedScore": 0.88,
        "explanation": "Fixed response correctly cites the current refund policy effective January 2025"
      },
      {
        "traceId": "tr_d4e5f6",
        "verdict": "regressed",
        "originalScore": 0.60,
        "fixedScore": 0.45,
        "explanation": "Fixed response is overly cautious and declines to answer a valid refund question"
      }
    ],
    "createdAt": "2025-08-14T10:00:00Z",
    "completedAt": "2025-08-14T10:25:00Z"
  }
}

Create Shadow Test

POST /api/shadow-tests

Request Body:

{
  "fixId": "fix_3a9c7e",
  "traceIds": ["tr_a1b2c3", "tr_d4e5f6", "tr_g7h8i9"],
  "config": {
    "comparisonMode": "side_by_side"
  }
}

Field	Type	Required	Description
`fixId`	string	Yes	The fix to test
`traceIds`	string[]	Yes	Historical traces to replay
`config`	object	No	Test configuration
`config.comparisonMode`	string	No	`side_by_side` (default) or `automated`

Response:

{
  "success": true,
  "data": {
    "id": "st_2n5q8s",
    "fixId": "fix_3a9c7e",
    "status": "pending",
    "traceCount": 3,
    "config": {
      "comparisonMode": "side_by_side"
    },
    "createdAt": "2025-08-15T14:00:00Z"
  }
}

Update Shadow Test

PATCH /api/shadow-tests/:id

Request Body:

{
  "status": "failed",
  "results": {
    "error": "Upstream model returned 429 during replay"
  }
}

Shadow Execution

Trigger a one-off shadow execution that replays a single trace against a fix and returns the result synchronously.

POST /api/shadow-execution

Request Body:

{
  "fixId": "fix_3a9c7e",
  "traceId": "tr_a1b2c3"
}

Field	Type	Required	Description
`fixId`	string	Yes	The fix to apply during replay
`traceId`	string	Yes	The historical trace to replay

Response:

{
  "success": true,
  "data": {
    "traceId": "tr_a1b2c3",
    "fixId": "fix_3a9c7e",
    "originalOutput": "Our refund policy allows returns within 60 days...",
    "fixedOutput": "Our current refund policy (effective January 2025) allows returns within 30 days of purchase...",
    "verdict": "improved",
    "originalScore": 0.35,
    "fixedScore": 0.91,
    "explanation": "The fixed response accurately cites the current 30-day refund window instead of the outdated 60-day policy",
    "latencyMs": 2400,
    "executedAt": "2025-08-15T14:05:00Z"
  }
}

Shadow execution is synchronous and subject to a 30-second timeout. For batch testing across multiple traces, use the Shadow Tests endpoints instead.

Error Responses

All endpoints return errors in a consistent format:

{
  "success": false,
  "error": {
    "code": "NOT_FOUND",
    "message": "Issue iss_nonexistent not found"
  }
}

Common error codes:

HTTP Status	Code	Description
400	`VALIDATION_ERROR`	Invalid request body or query parameters
401	`UNAUTHORIZED`	Missing or invalid API key
404	`NOT_FOUND`	Resource does not exist
409	`CONFLICT`	Resource state conflict (e.g., deleting an issue with active shadow tests)
429	`RATE_LIMITED`	Too many requests
500	`INTERNAL_ERROR`	Unexpected server error

Next Steps

Evaluation & Grading - Run evaluations against your agents
Guardrails - Configure input/output scanning policies
Webhooks & Notifications - Get notified when issues are created or fixes are validated
Shadow Testing Guide - End-to-end walkthrough of the shadow testing workflow

Traces & Spans Guardrails