Which Collector version is this runbook written against?

The configs are validated against the OpenTelemetry Collector contrib distribution v0.161.0. The Collector is pre-1.0, so the runbook tells you to re-check the changelog and validate your merged config before every upgrade.

How should I size the Collector queue?

Not from defaults. The default queue holds 1,000 requests, which says nothing about spans or bytes. The runbook derives capacity from your outage objective, drain rate, and retry budget, and recommends an items-based sizer for predictable sizing.

Agent, gateway, or tiered - which topology should I run?

Match the topology to your failure domain. Agents stay light and close to the workload. A gateway centralizes policy and scaling. Tiered deployments isolate stateful work like tail sampling onto consistently routed tiers. The runbook walks through each pattern and its trade-offs.

OpenTelemetry Collector Production Runbook

Q: Does enabling retries and the sending queue make the pipeline lossless?

No. The runbook treats backpressure as a chain: receiver, processor, queue, and backend each fail differently. Retries plus a queue reduce loss, but a queue that fills past its capacity still drops data, so capacity is derived from an outage objective.

Q: Is the persistent file queue safe to rely on?

Only with durable storage. The runbook covers the file storage extension and warns that ephemeral or node-bound volumes lose queued data on node replacement. Persistent queue storage must be durable, monitored, and tested across restarts.

Key ideas

Six numbers that change how you run the Collector

4 links

Backpressure is a chain

Receiver, processor, queue, backend - each link refuses, drops, or delays differently. Retries enabled is not lossless.

1,000

The default queue is requests

A queue measured in requests says nothing about spans or bytes. Use an items-based sizer and derive capacity from your outage objective.

80%

GOMEMLIMIT below the hard limit

memory_limiter goes first in every pipeline, coordinated with the pod limit and GOMEMLIMIT - the worked example uses 640 of 800 MiB.

5s → 30s

Retry with backoff, then stop

retry_on_failure starts at 5s and caps at 30s. Retrying forever just hides a sick backend and fills the queue.

60–70%

Scale on queue occupancy

Sustained occupancy around 60–70% is the upstream scale-up signal - a starting point, not an SLO. Tune against your own drain rate.

1 of 10

L4 balancing pins gRPC

One long-lived connection through an L4 balancer leaves a single replica hot while nine idle. Use L7/gRPC-aware balancing.

The centerpiece

Watch the Collector with the signals it exposes about itself

The queue is the earliest place trouble shows up. These internal metrics catch silent drops before they become missing dashboards - and the runbook shows how to route them on an independent path, so a sick pipeline never hides its own symptoms.

Signal	What it tells you	Starting alert
otelcol_exporter_queue_size / queue_capacity	Backend is slower than ingress	Warning above 60% for 10m; page higher
otelcol_exporter_enqueue_failed_*	Queue full - data is being dropped now	Any sustained rise pages
otelcol_exporter_send_failed_*	Export path failing; retries burning budget	Ratio to sent, by signal type
otelcol_receiver_refused_*	Memory limiter shedding load	Any sustained rise warns
memory / CPU / restarts	Limiter miscoordination, OOMKill risk	Against pod limit and GOMEMLIMIT

“A queue that silently fills, a tail sampler that sees half a trace, or a rollout that looked green while receivers refused data is not resilience. It is delayed discovery.”

Contents

Eight chapters, a checklist, and the sources

Topology patternsAgent, gateway, and tiered deployments - and the failure domain each one actually gives you.

Load balancing and routingWhy L4 pins gRPC connections, and how to route consistently for stateful processors.

The baseline configA complete, validated starting pipeline: memory_limiter first, batch, retry, queue.

Queue sizingCapacity from an outage objective - drain rate, spike headroom, retry budget, sizer choice.

Memory disciplinememory_limiter, pod limits, and GOMEMLIMIT coordinated so the Collector sheds load before the kernel kills it.

Self-monitoringThe internal metrics that matter, starting alert thresholds, and an independent telemetry path.

Zero-drop upgradesValidate, canary with measurable stop conditions, roll by zone, drain stateful pools in parallel.

Failure-pattern fingerprintsSymptom → likely cause → first checks, from queue climb to tenant misrouting.

✓

Production checklistThirteen items to walk through before you call a Collector deployment done.

SourcesAll 15 primary sources: OpenTelemetry docs, collector and contrib READMEs, the releases changelog.

Who it's for

Built for the people who own the telemetry path

Platform and SRE teams

You run centralized Collector gateways and need sizing, alerting, and upgrade discipline you can defend in review.

Teams moving from agent to gateway

You are consolidating collectors and need the topology trade-offs and routing rules written down before the migration.

On-call engineers

You get paged for missing telemetry. The fingerprint table maps symptoms to likely causes and first checks.

Engineering leads

You are planning Collector upgrades and want a zero-drop rollout pattern with measurable stop conditions.

Methodology & disclosure

How this runbook was built

Primary sources onlyEvery claim traces to OpenTelemetry's own documentation, the collector and contrib component READMEs, or the releases changelog - 15 sources, all listed in the PDF.

Validated configsEvery YAML block was validated against otelcol-contrib v0.161.0. The Collector is pre-1.0, so re-validate before each upgrade.

No invented benchmarksNumbers in this runbook are documented defaults or worked examples, and the text says so. Replay representative load against your own pipelines.

About KloudMateKloudMate is an AI-powered observability and SRE-Ops platform. This guide is vendor-neutral; the product is mentioned only here and in the footer.

FAQ

Common questions

Which Collector version is this written against?

Configs are validated against the contrib distribution v0.161.0. The Collector is pre-1.0, so the runbook builds re-validation and changelog review into the upgrade path itself.

Does enabling retries and the sending queue make the pipeline lossless?

No. Retries plus a queue reduce loss, but a queue that fills past capacity still drops data. The runbook treats backpressure as a chain and derives queue capacity from an outage objective instead of defaults.

How should I size the queue?

From your outage objective: how long the backend can be down, your drain rate, and spike headroom. The default 1,000 requests is not capacity planning - an items-based sizer makes sizing predictable.

Agent, gateway, or tiered - where do I start?

Match topology to failure domain. Agents stay light and close to the workload. A gateway centralizes policy and scaling. Tiered isolates stateful work like tail sampling onto consistently routed pools. Chapter 1 walks each pattern.

Is the persistent file queue safe to rely on?

Only with durable storage. Ephemeral or node-bound volumes lose queued data on node replacement. The checklist requires persistent queue storage to be durable, monitored, and tested across restarts.

Selected sources

Where the evidence comes from

OpenTelemetry, “Scaling the Collector” - opentelemetry.io/docs/collector/scaling/
OpenTelemetry, “Resiliency” - opentelemetry.io/docs/collector/resiliency/
OpenTelemetry, “Internal telemetry” - opentelemetry.io/docs/collector/internal-telemetry/
OpenTelemetry Collector, memory limiter and batch processor READMEs - github.com/open-telemetry/opentelemetry-collector
OpenTelemetry Collector Contrib, file storage, load-balancing exporter, tail-sampling processor READMEs - github.com/open-telemetry/opentelemetry-collector-contrib
OpenTelemetry Collector releases changelog - github.com/open-telemetry/opentelemetry-collector-releases

The full 15-source list, with URLs, is on the sources page of the PDF.