Engineers who run production systems keep asking the same practical questions about the tooling that watches those systems: who looks after it, what it costs, what breaks, and how big it needs to be. This FAQ answers them in plain terms. It leans on the Ops Atlas write-up on ObservabilityOps, which defines the subject as operating the observability stack itself, and on two real-world cases it summarises. Each answer is short enough to paste into a team wiki.
Where are the edges of ObservabilityOps?
The scope is the machinery: instrumentation standards, the pipelines that move logs and traces, the back ends that store metrics, logs, traces and profiles, plus limits on cardinality, sampling and retention, dashboard and alert conventions, monitoring of the monitoring, and cost governance. It deliberately leaves out reliability targets and error-budget policy, incident roles, machine-learning correlation of events, security telemetry and the cloud invoice. Those belong to neighbouring disciplines. If a task is about using the data to fix a live outage, it is not this one.
Is this a recognised industry standard?
No. The Atlas is frank that its searches found no standards body, certification or vendor programme using the label. It also warns against confusing it with a similarly named commercial product. What does exist is the practice, under names such as an observability team or a centre of excellence. Treat the term as a handy label, not a credential.
Why do metrics bills explode so suddenly?
Because each distinct combination of label values creates a separate time series. A harmless-looking label with thousands of possible values, a customer or request identifier for instance, multiplies storage. The Atlas guidance is to keep variety on any single metric under roughly ten and to investigate beyond one hundred. For logs, stream labels should stay in the tens. When limits on a scrape are breached, the entire scrape can be rejected, which means data disappears rather than merely being trimmed.
Can you show a real example of controlling that growth?
The Atlas summarises a case from a large network company in March 2023. It ran 916 Prometheus instances holding about 4.9 billion series. Instead of letting an oversized scrape fail completely, the engineers patched the software to cap the number of series per server. Their rule, quoted by the Atlas, was that if the automated checks pass then there is capacity for what a team needs. The lesson is that a limit enforced in the pipeline is friendlier than an outage discovered by a dashboard going blank.
If you are designing a limit for your own estate, the prompts in this complete guide on the points to settle ahead of a custom web build work as a pre-flight list, because capacity assumptions belong in requirements, not in postmortems.
How small can the team be?
Smaller than most people expect. The Atlas describes a social network with roughly twenty staff handling around ten million requests per minute, where a single engineer runs the observability setup. They deploy one collector per namespace through an operator managed by a GitOps tool, send data to a commercial back end and keep about 0.1 percent of successful traces along with every error trace. The Atlas records no collector problems in about two years. The trick is standardisation and a narrow remit; the engineer is not answering every team’s dashboard request.
Should one central team own all alerts?
No. The Atlas cites DORA research on this very point: watching a system is a shared job, not something to lock inside one person or team. A platform owner maintains pipelines, defaults and limits; service owners decide what deserves a page for their service. Mixing the two produces a ticket queue and a lot of blame.
How do I know the alerting path itself still works?
Use a heartbeat, sometimes called a dead-man’s switch. A test alert fires all the time, and a separate check outside your stack raises an alarm when the heartbeat stops. The Atlas adds that one black-box test of the full alert path beats a collection of per-component checks, because it exercises the whole chain as a user of alerts would.
Sampling: head or tail?
Head sampling decides at the start of a request. It is cheap but cannot guarantee that failing requests are retained. Tail sampling waits until a trace is complete, so it can keep the errors and the slow ones, but it is stateful and, in the words the Atlas quotes, difficult to operate: all spans of one trace must reach the same instance, usually by routing on the trace identifier. OpenTelemetry guidance begins to recommend sampling once volume reaches roughly 1,000 traces each second. Most teams adopt tail sampling for their single largest producer first.
Where do traces usually break?
At message queues, at proxies that discard the trace header, and in estates that mix two propagation formats. Queues are normally handled with span links. Another subtle risk: baggage, the key-value data that travels with a request, can leak to third-party services unless you allow-list keys and strip them at the boundary. Start a new trace at your public edge instead of trusting incoming headers.
What should I measure about the platform?
- Export success rate. The Atlas works an example where 9,950,000 successes against 40,000 failures and 10,000 dropped items gives 99.5 percent, which misses a 99.9 percent target.
- Freshness at the 95th percentile, using a synthetic canary event.
- Share of unused signals over ninety days.
- Portion of telemetry whose service name is unknown, ideally near zero.
- Unit cost, such as spend per million requests.
A caution on the first item: count retries only after they are exhausted, otherwise the success rate looks flattering.
Is running it ourselves the cheaper route?
Not automatically. The Atlas files the idea that hosting it yourself costs nothing under anti-patterns. What you save in licence fees comes back as version upgrades, sizing work and somebody carrying the pager for the data stores. A related point on licences: several popular back ends changed to AGPL terms, so involve whoever reviews legal obligations early.
What does a bad week look like in practice?
Typically it starts with a deploy that adds a new label. Within hours the series count climbs, queries slow down, and a dashboard that engineers rely on times out. Someone raises the memory limit, which postpones the problem by a day. Then the storage layer starts rejecting writes and a second team notices that its alerts went quiet, not because things are healthy but because data is no longer arriving. A heartbeat and a pre-merge series estimate would each have prevented the escalation, and both cost less than the overtime spent afterwards.
The remedy is rarely heroic. Roll back the label, write down the cause, add the check, and tell the other teams what changed. Repeating that small loop several times a year is what maturity looks like in this field.
Where should a new team start?
Follow the ladder the Atlas sets out. At the first stage, name an owner, take an inventory and add a heartbeat with an outside probe. Next, publish an instrumentation and log standard, apply gateway limits and redaction, and schedule the first alert review. Later, put configuration in version control with tests, prune unused signals and report unit cost. If your project documentation is still thin, the engineering-minded deep dive on gathering web project requirements is a sound place to record these decisions.
What is the most common mistake?
Assuming that the work is finished once the software development kit is installed. Installation gets you data, and not necessarily connected, bounded or affordable data. The unglamorous follow-up of limits, ownership and reviews is where the value is.
Where does artificial intelligence fit?
Nowhere by default. Claims about AI-driven observability and about pipeline products are, per the Atlas, marketing until you verify them using your own data. Spend first on naming, limits and ownership, and test any vendor promise with a bounded trial.

