Observability & Monitoring

Concept#

The beginner framing: a server running correctly on your laptop tells you almost nothing about whether it's healthy in production, under real traffic, at 3am, when nobody's watching a terminal. Observability is the set of practices that make production behavior visible without needing to reproduce the problem locally first.

The precise mental model: this topic covers four distinct, complementary mechanisms, structured logging (what happened), health checks (is it currently working), event-loop-delay monitoring (is it responsive), and request-scoped context (correlating everything above for one specific request).

Structured logging: why console.log doesn't scale#

console.log("User", userId, "failed login at", new Date()); // ❌ unstructured text

console.log(JSON.stringify({           // ✅ structured, machine-parseable
  level: "warn",
  event: "login_failed",
  userId,
  timestamp: new Date().toISOString(),
}));

A plain text log line is fine when a human is tail-ing a single server's output directly, it falls apart at any real scale. Once logs are aggregated across many server instances into a log-search system, unstructured text can only be searched by loose string matching. Structured (JSON) logs are queryable precisely: "show me every login_failed event for userId: 42" is a real, fast query against structured fields, not a hopeful grep. Libraries like pino build on this same JSON-log idea, adding performance optimizations and log-level filtering on top, but the core shift is this one: emit structured data, not prose.

Health checks: liveness vs. readiness#

router.add("GET", "/healthz/live", (req, res) => {
  res.writeHead(200).end("ok"); // "is the process alive and responding at all?"
});
 
router.add("GET", "/healthz/ready", async (req, res) => {
  try {
    await db.ping(); // "is it ready to actually SERVE traffic right now?"
    res.writeHead(200).end("ready");
  } catch {
    res.writeHead(

These answer genuinely different questions, and conflating them causes real production problems. Liveness asks "is this process alive and able to respond at all?", a failing liveness check typically means "restart this instance," since something is fundamentally broken (deadlocked, crashed internals). Readiness asks "is this instance currently able to serve real traffic?", a failing readiness check means "stop routing new traffic here, but don't necessarily restart" (e.g., a database connection is still establishing at startup, or a downstream dependency is temporarily unavailable). Using one check for both purposes causes either unnecessary restarts (treating a temporary readiness failure as a liveness failure) or traffic being routed to an instance that can't actually serve it (treating a readiness concern as if liveness alone were sufficient).

diagnostics_channel: instrumentation decoupled from application code#

const diagnostics_channel = require("diagnostics_channel");
const channel = diagnostics_channel.channel("myapp:db-query");
 
// application code publishes an event, with NO KNOWLEDGE of who's listening:
channel.publish({ query: sql, durationMs: elapsed });
 
// separately, monitoring/APM code subscribes, entirely decoupled:
channel.subscribe((message) => {
  metrics.record("db_query_duration", message.durationMs);
});

Confirmed stable in this app's installed Node v23.11.0: diagnostics_channel provides a pub/sub mechanism specifically designed for instrumentation, application code publishes events without needing to know whether anything is listening, and monitoring tooling (APM agents, custom metrics collectors) subscribes without modifying the application code that publishes. This decoupling is exactly what lets APM tools instrument Node internals and popular libraries without invasive patching.

Event-loop-delay monitoring: the metric that catches blocked loops#

const { monitorEventLoopDelay } = require("perf_hooks");
 
const histogram = monitorEventLoopDelay({ resolution: 20 });
histogram.enable();
 
setInterval(() => {
  console.log(`event loop delay, mean: ${histogram.mean / 1e6}ms, max: ${histogram.max / 1e6}ms`);
  histogram.reset();
}, 10000);

Confirmed working directly against this runtime: monitorEventLoopDelay measures actual event-loop responsiveness by sampling the delay between scheduled and actual timer callback execution, the exact metric that would have flagged the blocked-loop incidents described in Event Loop & libuv (a long synchronous operation) and Security (a ReDoS-triggering regex) as they happened, in production, before a user complaint arrived. A consistently elevated .mean (or .max) is the direct, quantified signal that something is blocking the main thread.

const fs = require('fs');
 
setTimeout(() => console.log('timer'), 0);
setImmediate(() => console.log('immediate'));
fs.readFile(__filename, () => {
  console.log('poll: I/O callback');
  setTimeout(() => console.log('timer (from I/O)'), 0);
  setImmediate(() => console.log('immediate (from I/O)'));
});

Request IDs via AsyncLocalStorage: tying it all together#

requestContext.run({ requestId }, () => {
  logWithContext("handling request");        // structured log includes requestId
  channel.publish({ requestId, query: sql }); // diagnostics event includes it too
});

The AsyncLocalStorage mechanism from Error Handling & Async Context is what makes structured logs, diagnostics events, and error reports all correlate to the same requestId across an entire request's async lifecycle, without threading that ID through every function signature by hand.

Try It#

Predict the outcome before checking the solution.

A service's /healthz/ready check queries the database, and its /healthz/live check just returns 200 OK unconditionally. The database becomes temporarily unreachable for 30 seconds due to a network blip, then recovers on its own.

Implement It Yourself#

Build a minimal structured logger with request-context correlation, combining this topic's pieces:

const { AsyncLocalStorage } = require("async_hooks");
const requestContext = new AsyncLocalStorage();
 
function log(level, event, extra = {}) {
  const context = requestContext.getStore() ?? {};
  console.log(JSON.stringify({
    level,
    event,
    timestamp: new Date().toISOString(),
    ...context,
    ...extra,
  }));
}
 
function withRequestContext(

Every log() call automatically includes the current requestId, the exact combination of structured JSON output and AsyncLocalStorage-based correlation that production logging setups rely on.

Under the Hood#

Event-loop-delay monitoring measures the same phase-transition timing covered in Event Loop & libuv, it's a direct, quantified production instrument for the exact mental model that topic builds conceptually. diagnostics_channel and request-ID correlation both build on the AsyncLocalStorage mechanism established in Error Handling & Async Context. And CPU/memory profiling from Performance & Profiling is the natural next step once event-loop-delay monitoring flags that something is wrong, profiling is how you find what.

Common Mistakes#

1. Using the same endpoint/logic for both liveness and readiness#

router.add("GET", "/health", async (req, res) => {
  await db.ping(); // ❌ a DB blip now triggers unnecessary RESTARTS, not just traffic pausing
  res.writeHead(200).end("ok");
});

Conflating the two causes an orchestrator to restart a perfectly healthy process over a transient downstream issue that should have only paused traffic routing.

2. Logging unstructured strings in a production service#

console.log(`Request from ${ip} failed: ${err.message}`); // ❌ not queryable at scale

This works fine locally but becomes effectively unsearchable once logs are aggregated across many instances, structured JSON fields are what make production log search actually usable.

3. Never monitoring event-loop delay, only reacting to user-reported slowness#

// no monitorEventLoopDelay anywhere, the first signal of a blocking issue
// is a support ticket, not a metric

Without this metric, a blocking operation (an accidental readFileSync in a hot path, a ReDoS-vulnerable regex) is invisible until someone notices the symptom, by which point it's already been affecting real users.

Best Practices#

• Keep liveness and readiness checks genuinely separate, with different failure semantics (restart vs. pause-traffic).
• Emit structured (JSON) logs, not plain text, for anything running in production at more than trivial scale.
• Monitor event-loop delay continuously, not just when investigating a reported problem, it's the leading indicator, not the confirming one.
• Correlate logs, diagnostics events, and error reports with a request ID via AsyncLocalStorage, so a single request's full story can be reconstructed after the fact.

Performance Tips#

• monitorEventLoopDelay's own overhead is intentionally small (it's sampling, not intercepting every operation), the entire point is a monitoring mechanism cheap enough to run continuously in production without itself becoming a performance problem.
• Structured JSON logging has a small serialization cost per log line compared to a raw string, for extremely hot logging paths, libraries like pino are specifically optimized to minimize this, which is part of why a dedicated logging library often outperforms hand-rolled JSON.stringify calls at high log volume.