Why most engineering teams overpay for observability and how to fix the bill
Observability is a necessity, not a luxury. Yet most teams overspend on it because they don’t understand the tradeoffs between cost and value. The problem isn’t a lack of tools—it’s a lack of discipline in applying them. Teams buy expensive SaaS platforms, pile on agents, and retain data they’ll never query, all while missing the signals that would actually prevent outages.
I’ve seen this pattern repeatedly: a team spends $50K/month on Datadog, only to realize they’re paying for metrics they don’t need. Or a company with 10 microservices runs Prometheus at scale, only to find they’re alerting on noise rather than root causes. The root cause? They bought the most expensive tool they could find, not the one that solved their problem.
01. The observability tax
Observability isn’t free. The real cost isn’t just the software—it’s the hidden costs of complexity. Every additional data source, every extra agent, every retained metric adds to the operational overhead. Teams often don’t account for:
- The cost of maintaining the infrastructure to store and query the data
- The time engineers spend tuning alerts and dashboards
- The cognitive load of interpreting signals in a noisy environment
For example, a team running OpenTelemetry with 100 services might generate 1TB of traces per day. Storing that data costs $20K/month on cloud storage alone. But if they’re only debugging 5% of those traces, they’re paying for something they’ll never use.
02. The "good enough" trap
Most teams don’t need full-stack observability. They need the right signals at the right time. The problem is that vendors sell observability as a monolith—logs, metrics, traces, and everything in between. But most teams don’t need all of it.
Consider a team running a high-traffic API. They might need:
- Latency metrics for SLOs
- Error logs for debugging
- Trace sampling for critical requests
But they don’t need full-body profiling or synthetic monitoring. Yet they’ll buy a tool that does all of it, even if they only use 20% of its features.
03. The alerting illusion
Alerts are the most expensive part of observability. Every alert requires:
- A signal to monitor
- A threshold to set
- A team to respond
Teams often set up alerts for everything, from "CPU > 80%" to "HTTP 500 errors." But most of these alerts are either:
- False positives (e.g., a transient spike)
- Noise (e.g., a known issue)
- Unactionable (e.g., a metric that never leads to a fix)
For example, a team might alert on "disk space < 10%," only to find that the alert fires every time a backup runs. They’ve spent $5K/month on an alert that doesn’t help.
04. How to fix the bill
The solution isn’t to cut observability—it’s to make it more efficient. Here’s how:
04a. Start with the signals, not the tools
Before buying anything, ask:
- What signals will prevent outages?
- What data do we need to debug incidents?
- What alerts are actually actionable?
For example, a team running a payment service might need:
- Latency percentiles for SLOs
- Error rates for critical endpoints
- Trace sampling for failed transactions
They don’t need logs for every request or metrics for every microservice.
04b. Use open-source tools for the basics
Open-source tools like Prometheus, Grafana, and OpenTelemetry are often cheaper and more flexible than SaaS. They let you:
- Run only what you need
- Scale infrastructure yourself
- Avoid vendor lock-in
For example, a team with 50 services might run Prometheus on a single node, costing $1K/month instead of $10K/month for a managed service.
04c. Tune alerts ruthlessly
Alerts should be:
- Based on real incidents
- Actionable within 5 minutes
- Tied to a runbook
For example, instead of alerting on "CPU > 80%," alert on "CPU > 90% for 5 minutes" and include a runbook for scaling.
04d. Retire unused data
Most teams retain data forever. But they rarely query it. Use tools like Prometheus’s retention policies or S3 lifecycle rules to delete old data.
For example, a team might keep 7 days of high-cardinality metrics and 30 days of low-cardinality ones, reducing storage costs by 50%.
05. The bottom line
Observability shouldn’t be a cost center. It should be an enabler. The key is to:
- Start with the signals, not the tools
- Use open-source where possible
- Tune alerts ruthlessly
- Retire unused data
If you do this, you’ll spend less on observability and get more value from it.
Next step: Audit your current observability setup. Identify which signals are actually used and which alerts are actionable. Then, adjust your tools and processes accordingly.
Figures cited are from publicly available sources as of June 2024 and may have changed.