High Memory Usage

⚠️

Severity: Medium | Alert Threshold: Memory usage > 80% for 5+ minutes

Overview

This alert triggers when service memory utilization exceeds 80% of allocated memory for an extended period.

Impact Assessment

Memory Usage	Risk Level	Action
80-85%	Medium	Monitor, prepare to scale
85-95%	High	Investigate, consider scaling
> 95%	Critical	Immediate action required

Diagnostic Steps

Check Current Memory Usage

# View Cloud Run metrics
gcloud run services describe thinkhive-demo \
  --region us-central1 \
  --format "value(status)"
 
# Check logs for memory warnings
gcloud logging read 'textPayload=~"memory" OR textPayload=~"heap"' --limit 20

Identify Memory-Heavy Operations

Common memory consumers in ThinkHive:

Large trace processing
Batch evaluation runs
LLM response caching
Session storage

Check for Memory Leaks

# Look for OOM patterns
gcloud logging read 'textPayload=~"out of memory" OR textPayload=~"OOM"' --limit 10

Review Recent Changes

New features processing large data?
Changes to caching behavior?
Increased batch sizes?

Remediation

When: Legitimate workload requires more memory

# Increase memory allocation
gcloud run services update thinkhive-demo \
  --region us-central1 \
  --memory 1Gi
 
# For heavy workloads
gcloud run services update thinkhive-demo \
  --region us-central1 \
  --memory 2Gi

When: Workload can be distributed

# Reduce concurrency per instance
gcloud run services update thinkhive-demo \
  --region us-central1 \
  --concurrency 40
 
# Increase max instances
gcloud run services update thinkhive-demo \
  --region us-central1 \
  --max-instances 20

When: In-memory caches grown too large

# Restart service to clear in-memory caches
gcloud run services update thinkhive-demo \
  --region us-central1 \
  --update-env-vars CACHE_CLEAR=$(date +%s)

Quick Mitigation

# Immediate: Increase memory and reduce concurrency
gcloud run services update thinkhive-demo \
  --region us-central1 \
  --memory 1Gi \
  --concurrency 50

Prevention

Set appropriate memory limits based on workload
Implement memory monitoring in application
Use streaming for large data operations
Implement cache eviction policies
Regular load testing with production-like data
Profile memory usage in development

Node.js Specific Tips

// Expose memory metrics endpoint
app.get('/debug/memory', (req, res) => {
  const usage = process.memoryUsage();
  res.json({
    heapUsed: Math.round(usage.heapUsed / 1024 / 1024) + 'MB',
    heapTotal: Math.round(usage.heapTotal / 1024 / 1024) + 'MB',
    external: Math.round(usage.external / 1024 / 1024) + 'MB',
    rss: Math.round(usage.rss / 1024 / 1024) + 'MB'
  });
});

Database Slow Memory Critical