Sampling
Sampling controls how many traces an instrumented service keeps. Keeping every request gives you the most detail but costs the most to ingest. Keeping a fraction lowers the cost but drops some detail. The agent lets you set one global default and override it for individual services. Sampling applies to the opt-in application traces. eBPF monitoring metrics are collected separately.
Global default and per-service overrides
Section titled “Global default and per-service overrides”- Global default: a single sampling rate applies to every service you instrument, so you set the trade-off once.
- Per-service override: raise or lower the rate for one service. For example, keep every trace on a low-traffic, high-value service, and sample a noisy one more aggressively.
The agent applies the rate to each instrumented service through the standard OpenTelemetry (OTel) sampler settings, so the behavior matches what the language agents document.
Parent-based sampling keeps traces whole
Section titled “Parent-based sampling keeps traces whole”The agent uses parent-based ratio sampling. When a service continues a trace that an upstream service already started, it follows the upstream decision instead of making its own. This keeps a sampled trace whole from end to end, so you do not get traces that are kept on one hop and dropped on the next.
Choosing a rate
Section titled “Choosing a rate”- Start with a moderate default and adjust based on your traffic volume and ingestion budget.
- Increase the rate on services where you need complete traces for debugging or service-level objectives.
- Decrease the rate on very high-volume services where a representative sample is enough.