Cost, Latency & Safety Considerations

Concept#

Everything covered elsewhere in this domain, streaming, tool use, RAG, agents, eventually has to run in production against real traffic, real budgets, and real users who can send genuinely adversarial or unexpected input. This topic covers the three operational dimensions that decide whether an AI feature is actually production-ready: what it costs, how fast it feels, and what happens when things go wrong.

Measuring cost, don't guess, count tokens#

Every response carries exact token usage; never estimate cost from character counts or a rough heuristic.

const response = await client.messages.create({
  model: "claude-opus-4-8",
  max_tokens: 1024,
  messages: [{ role: "user", content: "..." }],
});
 
console.log(response.usage.input_tokens);               // full-price input tokens
console.log(response.usage.output_tokens);               // output tokens generated
console.log(response.usage.cache_read_input_tokens);      // served from cache, ~0.1x cost
console.log(response.usage.cache_creation_input_tokens);   // written to cache, ~1.25x cost

Before sending a large or repeated request, count_tokens gives an exact pre-flight estimate without actually generating a response, useful for cost dashboards, budget checks, or deciding whether a prompt needs trimming before it's sent:

const count = await client.messages.countTokens({
  model: "claude-opus-4-8",
  messages: [{ role: "user", content: largeDocumentText }],
});
console.log(count.input_tokens); // exact count for THIS model, never approximate with a generic tokenizer

Token counts are model-specific, a generic tokenizer library will give a wrong number for Claude specifically, sometimes by a large margin. Always count against the actual model you're going to call.

The cost/latency/quality triangle, concrete levers#

// A cost/latency-sensitive route: lower effort, a smaller max_tokens ceiling
const response = await client.messages.create({
  model: "claude-opus-4-8",
  max_tokens: 512,
  output_config: { effort: "low" },
  messages: [{ role: "user", content: "Classify this support ticket's category." }],
});
 
// A quality-sensitive, latency-tolerant route: higher effort, more room to think
const response2 = await client.messages.create({
  model: "claude-opus-4-8",
  max_tokens: 8192,
  thinking: { type: "adaptive" },
  output_config: { effort: "high" },
  messages: [{ role: "user", content: "Review this pull request for bugs." }],

The right lever setting is route-specific, not global, a simple classification endpoint and a deep code-review endpoint have genuinely different cost/latency/quality requirements, and applying one blanket configuration across an entire application usually means either overpaying on the simple routes or under-delivering on the demanding ones.

Rate limits, handle them, don't just hope they don't happen#

Production traffic will eventually hit a rate limit (requests-per-minute or tokens-per-minute), and the correct response depends on distinguishing it from other failures:

import Anthropic from "@anthropic-ai/sdk";
 
try {
  const response = await client.messages.create({ /* ... */ });
} catch (error) {
  if (error instanceof Anthropic.RateLimitError) {
    // The SDK already retries 429s automatically up to max_retries, if one still
    // surfaces here, the retry budget was exhausted. Back off further or queue.
    const retryAfter = error.headers?.["retry-after"];
    // surface a "we're busy, try again shortly" state to the user, not a generic error
  }
}

A rate limit surfacing as an error to the user (rather than being silently absorbed by the SDK's built-in retries) usually means either sustained traffic genuinely exceeds the account's current tier, or a client-side bug is issuing far more requests than intended (a retry loop with no backoff, a duplicate-submission bug), both are worth distinguishing, since the fix differs (request a tier increase vs. fix a bug).

Safety-relevant response shapes an application must handle explicitly#

stop_reason: "refusal" is not an error, it's a valid, successful HTTP response where the model or a safety classifier declined the request. Code that only checks for end_turn / tool_use / max_tokens will mishandle this silently.

if (response.stop_reason === "refusal") {
  // A genuinely different outcome from a network/API error, present it as such,
  // and do NOT automatically retry the identical request; it will refuse again.
  return { type: "refusal" as const, message: "I'm not able to help with that." };
}

The same discipline applies to tool results on the way back into the model: a failed tool execution should be reported with is_error: true, not silently dropped and not disguised as a successful result, feeding the model a fabricated "success" for a call that actually failed is its own safety-relevant failure mode, since it can lead the model to state something false with full confidence.

// ❌ never disguise a real failure as a successful result
const toolResult = { type: "tool_result" as const, tool_use_id: block.id, content: "Task completed." };
// when the tool actually threw an exception
 
// ✅ report the failure honestly
const toolResult = {
  type: "tool_result" as const,
  tool_use_id: block.id,
  content: `Error: ${err.message}`,
  is_error: true,
};

Input validation and prompt injection awareness#