Circuit Breaker Open

Overview

This alert triggers when a circuit breaker opens due to repeated failures to an external dependency. The circuit breaker pattern prevents cascading failures by stopping requests to unhealthy services.

What Circuit Breakers Protect

Diagnostic Steps

# Check logs for circuit breaker events
gcloud logging read 'textPayload=~"circuit" OR textPayload=~"breaker"' \
  --limit 30 \
  --freshness=30m

# Look for the failures that triggered the circuit
gcloud logging read 'severity>=ERROR' --limit 50 --freshness=30m

# Test connectivity from Cloud Run
# Deploy a test container or check Cloud Run logs for timeout patterns
gcloud logging read 'textPayload=~"ETIMEDOUT" OR textPayload=~"ECONNREFUSED"' --limit 20

Remediation by Service

# Restart service to reset circuit
gcloud run services update thinkhive-demo \
  --region us-central1 \
  --update-env-vars CIRCUIT_RESET=$(date +%s)

Circuit Breaker States

CLOSED (normal)
    ↓ failures exceed threshold
OPEN (blocking requests)
    ↓ timeout period expires
HALF-OPEN (testing)
    ↓ success → CLOSED
    ↓ failure → OPEN

Manual Circuit Reset

If the dependency is confirmed healthy but circuit remains open:

# Force restart to reset all circuits
gcloud run services update thinkhive-demo \
  --region us-central1 \
  --update-env-vars FORCE_RESTART=$(date +%s)

Graceful Degradation

When circuits open, the service should:

• Return cached data if available
• Show user-friendly error messages
• Log detailed failure information
• Not cascade to other services

Prevention

• Set appropriate timeout values
• Configure retry with exponential backoff
• Monitor dependency health proactively
• Implement fallback responses
• Use multiple providers where possible
• Regular dependency health checks

Related Runbooks

• High Error Rate
• Service Down
• Database Slow