When Observability Becomes Noise

Based on a LinkedIn post originally published on 16 January 2026

A few years ago, I joined a large start-up to manage a DevOps team.

One of the first areas I examined was the observability environment: metrics, probes, collectors, dashboards, alerts and data-retention practices.

The scale was impressive.

There were probes throughout the environment and large quantities of telemetry were being produced. At first glance, this appeared to indicate a mature operational capability.

A closer examination produced a different picture.

A substantial proportion of the telemetry appeared to fall into one or more of these categories:

• data was collected but never analysed;

• signals were measured without supporting a decision;

• dashboards existed but were rarely consulted;

• some data was not retained long enough to support meaningful analysis;

• alerts were not connected to an operational response;

• and nobody could clearly explain the purpose of particular probes.

The environment was highly instrumented, but instrumentation and understanding are not the same thing.

The rational technical response

My initial response was technical and direct.

I began identifying probes with no evident consumer and disabling those that appeared to provide no operational value. At the same time, I enabled or improved signals that more accurately reflected system behaviour and risk.

From my perspective, a probe should have a reason to exist.

It should help answer a question, support a decision, identify a meaningful change or trigger an appropriate action.

Otherwise, it consumes resources without producing a useful outcome.

Telemetry is not free. It consumes:

• processor time;

• memory;

• network bandwidth;

• storage capacity;

• database and indexing resources;

• licensing or cloud-service expenditure;

• engineering effort;

• and human attention.

Removing clearly unused telemetry therefore appeared to be an uncontroversial improvement.

It was not.

“What will replace the probes?”

The SRE Director contacted me and asked what I intended to implement to compensate for the probes I had disabled.

The question revealed that we were approaching observability from different assumptions.

I believed the burden of proof belonged to the probe:

What question does this signal answer, and who acts on it?

The opposing assumption appeared to be that existing telemetry should remain unless another mechanism replaced it.

Our conversation made little progress.

I could not persuade him that collecting data without an identified purpose was wasteful. He could not persuade me that every existing probe represented a capability that required replacement.

At the time, I saw this principally as a disagreement about engineering efficiency.

Looking back, it was more complicated.

Observability should begin with questions

A useful observability system should help engineers answer questions about the behaviour of a service.

Examples include:

• Is the service available?

• Are users receiving the expected result?

• Is latency increasing?

• Are errors concentrated in one dependency or region?

• Is a resource approaching exhaustion?

• Did behaviour change after a deployment?

• Is the system meeting its reliability objectives?

• What happened immediately before an incident?

• Is recovery progressing as expected?

The telemetry should exist because it contributes to answering one or more of these questions.

This does not mean every metric must trigger an alert. Some information supports investigation, capacity planning, trend analysis or security review.

It does mean that the purpose should be understood.

A metric without an immediate consumer may still be valuable if it supports a documented analytical or forensic requirement. A signal retained for only a few minutes may be appropriate for immediate control behaviour. Long-term retention may be justified for capacity trends, compliance evidence or incident analysis.

The problem is not that telemetry lacks an alert.

The problem is that nobody can explain why it exists.

Metrics, dashboards and alerts serve different purposes

Observability components are often discussed as if they were interchangeable.

They are not.

Metrics

Metrics summarise system behaviour over time.

They can show rates, counts, durations, saturation and other measurable characteristics. They are useful for identifying trends and comparing current behaviour with established baselines.

Logs

Logs provide contextual records of events.

They can explain what the system attempted to do, which inputs or states were involved and where a sequence failed.

Traces

Distributed traces show how a request moves between services.

They are particularly valuable when latency or failure emerges from interactions across multiple components.

Dashboards

Dashboards organise selected information for a particular audience and purpose.

A dashboard should answer a recognised set of questions. A collection of charts is not automatically a useful dashboard.

Alerts

Alerts demand attention.

They should therefore indicate a condition requiring investigation or action. An alert with no expected response is merely an automated interruption.

Confusing these purposes contributes to observability noise.

Dashboards nobody uses

Dashboards are easy to create and difficult to retire.

An engineer builds one during an incident. A team creates another for a migration. A vendor supplies several default dashboards. A new manager requests a high-level view. Over time, the organisation accumulates dozens or hundreds of them.

The existence of a dashboard is then mistaken for visibility.

A useful dashboard should have:

• an identified audience;

• a defined operational or analytical purpose;

• an owner;

• known data sources;

• an understood refresh rate;

• and a review or retirement process.

If nobody knows who uses a dashboard or which decision it supports, retaining it indefinitely creates clutter.

During an incident, clutter increases the time required to find the authoritative view.

Alerts without action

Alerts are particularly expensive because they consume human attention.

Every alert creates an implicit request:

Stop what you are doing and investigate this condition.

If the condition does not require action, the alert should probably not exist in that form.

Repeated non-actionable alerts teach engineers to ignore the monitoring system. This creates the dangerous situation in which a genuinely important signal arrives through a channel already associated with noise.

A good alert should normally have:

• a meaningful condition;

• an appropriate severity;

• a clear owner;

• sufficient context;

• an expected response;

• and a connection to a service or operational objective.

Where appropriate, it should also reference a runbook or diagnostic procedure.

Alert volume is not evidence of operational maturity. A quieter system with well-designed alerts may provide far better protection.

Retention without purpose

Telemetry-retention decisions should reflect how the data will be used.

Keeping everything forever is expensive and unnecessary. Retaining data for too little time makes it useless for long-term analysis.

Different information may require different retention periods:

• high-resolution operational metrics for immediate diagnosis;

• aggregated metrics for capacity and trend analysis;

• logs for incident investigation;

• security records for threat detection or audit;

• traces for short-term performance analysis;

• and business-level service indicators for longer-term reporting.

The correct retention period depends on the question.

A data-retention policy should therefore be based on operational, security, analytical and compliance requirements—not simply on default settings.

The missing architectural picture

There was another important part of the context.

In the same environment, I had discovered that nobody appeared to hold a complete architectural view of the platform.

Individual engineers understood their own areas, but the complete relationship between applications, infrastructure, data flows and dependencies was unclear.

This raised a deeper problem.

If the system itself was not well understood, how could the organisation know whether its telemetry represented the system’s most important behaviour?

Instrumentation without an architectural model can become opportunistic. Teams measure the components they know about, using the signals their tools make readily available.

Important end-to-end relationships may remain invisible, while less meaningful local details produce enormous quantities of data.

A useful observability strategy should follow the architecture:

• user journeys;

• service boundaries;

• dependencies;

• data flows;

• trust boundaries;

• failure domains;

• and operational ownership.

Without that connection, the organisation may be observing many things without observing the system.

The organisational dimension

At the time, I treated the removal of unused probes as a straightforward engineering decision.

With hindsight, I understand that observability is also organisational.

Telemetry, dashboards and tools may be connected to:

• team responsibilities;

• vendor relationships;

• budgets;

• staffing levels;

• historical incidents;

• executive reporting;

• compliance commitments;

• and professional identities.

Changing the observability environment can therefore challenge more than a technical configuration.

It may call into question earlier decisions or alter the perceived scope of a team. It may affect a commercial contract or remove something created in response to a serious incident whose history is no longer widely remembered.

I later wondered whether some resistance reflected concern about organisational scope, team justification or established ownership. That remains an interpretation, not something I could prove.

The important lesson is that technical changes can affect human systems even when the engineering rationale appears obvious.

What I would do differently

I still believe that telemetry without a defined purpose should be challenged.

What I would change is the process.

Before removing a probe, dashboard or alert, I would create a structured inventory containing:

• the signal or artefact;

• its owner;

• its consumers;

• the question it answers;

• the action it supports;

• its retention requirement;

• its operational cost;

• and the consequence of removing it.

I would then review the inventory with the relevant SRE, platform, application, security and business stakeholders.

The possible outcomes would be explicit:

1. Retain it because its purpose is valid.

2. Modify it because the signal is useful but poorly implemented.

3. Replace it with a better measure.

4. Reduce its resolution or retention.

5. Convert an alert into a dashboard or analytical signal.

6. Retire it because no justified consumer or requirement exists.

This turns a unilateral cleanup exercise into a governed observability revi

AI Assistance Disclosure

This article is based on my original ideas, experience, analysis and conclusions. Artificial intelligence tools were subsequently used as editorial and research assistants to review grammar and wording, improve structure and presentation, organise some arguments into clearer logical sections, and help review references to legal, regulatory and technical concepts.

Where relevant, factual and regulatory references were checked against the sources cited in the article. AI assistance does not replace professional legal, regulatory, financial or technical advice, and the final selection, interpretation, opinions and conclusions presented here remain my own.