Basic Monitoring & Alerting

Concept#

Health-Check Endpoints: Liveness vs Readiness#

To automate container deployments (like Kubernetes or AWS ECS) and keep traffic flowing to healthy instances, applications expose health-check endpoints:

• Liveness probe (/health/live): Confirms if the application container is running. If it returns an error or fails to respond, the orchestrator kills the container and starts a fresh one.
• Readiness probe (/health/ready): Confirms if the application is fully booted and ready to receive traffic (e.g. database connections are open, caches are warmed). If it fails, the balancer stops routing traffic to this container but keeps it running.

// Express readiness check route
app.get('/health/ready', async (req, res) => {
  try {
    await db.$queryRaw`SELECT 1`; // Test database connection
    res.status(200).send('Ready');
  } catch (error) {
    res.status(503).send('Database connection failed');
  }
});

Core Performance Metrics#

1. CPU Usage: Spikes indicate infinite rendering loops, intensive calculations, or memory leak collection thrashing.
2. Memory Usage (Heap): Continuous growth indicates memory leaks.
3. Latency (p95 / p99): The 95th and 99th percentile response times, showing how slow queries impact outliers.
4. Error Rate (5xx status codes): Spikes indicate unhandled server crashes or API outages.

Common Mistakes#

1. Connecting health check endpoints to heavy query paths#

Running a complex, slow aggregate query inside /health/live executes on every balance probe check (e.g. every 5 seconds). This floods the database with redundant queries, degrading production performance. Keep health check queries trivial (SELECT 1).

2. Setting alert thresholds too low (Alert Fatigue)#

Configuring notifications to ping developer Slack channels on every minor, transient CPU spike (e.g. alert if CPU > 80% for 1 second) leads to alert fatigue. Developers start ignoring alerts, missing actual outages. Set alerting rules to trigger only on persistent regressions (e.g., CPU > 80% for 5 minutes).

Best Practices#

• Differentiate Probes: Build separate /health/live and /health/ready endpoints to manage orchestrator restarts.
• Run Simple Test Queries: Use SELECT 1 or ping replica clusters to verify connection sanity without load.
• Enforce Anomaly Thresholds: Configure alerts based on sustained deviations (e.g., error rates exceeding 2% over 5 minutes) rather than instant events.