Changelog

Guardrails API

New Features

• Guardrails API: Real-time content scanning with 6 scanner types (PII, secrets, keywords, regex, topic, tool_call)
• Guardrail Policies: Named, reusable scan configurations with per-scanner settings
• Guardrail Analytics: Aggregate analytics dashboard for scan activity, top findings, and latency metrics
• SDK Support: Guardrails module added to both JavaScript and Python SDKs

Improvements

• Multi-scanner orchestration with configurable action priority (block > redact > flag > pass)
• Aho-Corasick algorithm for efficient keyword matching
• Configurable timeout, fail-open/fail-closed, and short-circuit options

Platform Expansion

Major Release: Comprehensive expansion of the platform with 18+ new feature areas.

New Features

• Business Metrics: Record and track business outcomes linked to AI agent performance
• ROI Analytics: Configure ROI analysis, view per-agent ROI, predicted impact, and correlation analysis
• Quality Metrics: RAG quality scoring, hallucination detection, and batch evaluation
• API Keys v2: Enhanced key management with rotation, testing, usage stats, and agent scoping
• Issues v2: Replaces “Cases” terminology with improved issue management and clustering
• Analyzer v2: Enhanced trace analysis with cost estimation
• Deep Dive: Structured investigation workflows for complex issues
• Error Analysis: Automated error categorization and trend tracking
• Human Review Queue: Reviewer assignment, calibration sets, skip/reassign workflows
• Nondeterminism Detection: Pass@k analysis, consistency testing, reliability scoring
• Eval Health Monitoring: Health snapshots, regression detection, saturation analysis
• Deterministic Graders: Rule-based evaluation (length, keywords, json_valid, regex, no_pii, response_time)
• Conversation Eval: Multi-turn conversation evaluation
• Transcript Patterns: Pattern detection for escalation, PII exposure, sentiment shifts
• Compliance Scanning: Compliance scan rules and automated scanning
• Flagged Traces: Manual and automated trace flagging for review
• Departments: Organize agents into team-level departments
• Notification Channels: Slack, email, and webhook notification channels with configurable rules

SDK Updates

• JavaScript SDK v4.0.0 with 16 new API modules
• Python SDK v4.0.1 with guardrails and analysis support

Breaking Changes

• Cases API renamed to Issues API (v2) — old /api/v1/cases endpoints deprecated
• Webhook endpoints moved under /api/v1/explainer/webhooks

Bug Report System & Security Hardening

New Features

• In-App Bug Reports: Users can submit bug reports with automatic breadcrumb capture, console logs, and network state
• Admin Bug Report View: Admin dashboard for reviewing, triaging, and managing user-submitted bug reports
• Enhanced PII Redaction: 37 detection patterns for automatic PII redaction across bug reports and traces

Improvements

• Hardened JSONB sanitization pipeline for stored data
• Improved rate limiting on bug report submissions
• Fixed double HTML-encoding issue on JSONB fields containing URLs

ThinkEval Enhancements & Documentation Overhaul

New Features

• ThinkEval Wizard: Guided setup for creating evaluation suites in the dashboard
• Jury Mode: Multi-model evaluation with weighted consensus scoring
• Deterministic Graders: Rule-based evaluation checks (regex, JSON validity, length, PII detection)
• Composite Criteria: Weighted combinations of evaluation criteria

Improvements

• Comprehensive documentation site overhaul with 18 new guides
• Expanded API reference documentation with inline endpoint examples
• Improved evaluation result visualization and trend tracking

SDK Updates

• JavaScript SDK: Added evaluate() method for running evaluation suites
• Python SDK: Added evaluation suite support with decorator-based tracing

Run-Centric Architecture

Major Release: Complete architecture redesign for better AI agent analysis.

New Features

• Runs API: Track complete agent executions from start to finish
• Claims API: Extract and categorize facts vs. inferences from agent outputs
• Calibration API: Measure prediction confidence accuracy with Brier scores
• Ticket Linking: 7 deterministic methods to link runs to support tickets
• Customer Context: Time-series customer state tracking for ROI analysis

SDK Updates

• JavaScript SDK v3.0.0 with new V3 APIs (runs, claims, calibration)
• Python SDK v3.0.0 with decorator-based tracing
• MCP Server v3.0.0 with 25+ tools for Claude Code integration

API Changes

• New endpoints under /api/v1/explainer/
• Run-based analysis replaces trace-based workflows
• Enhanced webhook system with circuit breakers

Breaking Changes

• Trace API endpoints moved from /api/traces to /api/v1/traces
• Run-based analysis replaces trace-based analysis
• SDK initialization requires serviceName parameter

Explainability Engine

Enhanced AI analysis capabilities for production workloads.

New Features

• RAG Evaluation: Groundedness, faithfulness, and citation accuracy metrics
• Hallucination Detection: 9 types of hallucinations with confidence scores
• Counterfactual Analysis: What-if scenarios for debugging
• Batch Analysis: Analyze multiple traces in a single request

Improvements

• 40% faster analysis response times
• Improved clustering accuracy
• Better support for streaming responses

Enterprise Features

Production-ready features for enterprise deployments.

New Features

• Auth0 SSO: Enterprise authentication with MFA support
• HIPAA Compliance: PII redaction, audit logging, data retention policies
• Drift Detection: Automated quality monitoring with alerts
• Shadow Testing: Validate fixes before deploying to production
• Credit System: Usage-based billing with Stripe integration

API Changes

• New authentication system with API keys
• Rate limiting by tier
• Webhook delivery with retry logic

Quality Metrics

Comprehensive quality tracking for AI agents.

New Features

• Quality Dashboard: Real-time metrics visualization
• SLO Monitoring: Define and track service level objectives
• Alert Thresholds: Configurable alerts for quality degradation
• ROI Analytics: Connect AI performance to business outcomes

Initial Release

First public release of ThinkHive.

Features

• Trace capture and storage via OTLP
• Basic explainability analysis
• Case clustering for failure patterns
• Fix suggestions with AI
• JavaScript SDK for Node.js applications
• Python SDK with decorators

Integrations

• OpenAI auto-instrumentation
• LangChain integration
• OpenTelemetry compatibility