Teams that move workloads to bare metal for performance, cost, or sovereignty reasons often drag their old SaaS observability contracts along with them. The result is a single-tenant, high-performance server shipping gigabytes of telemetry to a vendor billing by ingestion volume, with per-seat fees layered on top. The economics stop making sense fast, and none of that data ever needed to leave your own racks.
A self-hosted, production-grade bare metal observability stack is not a compromise. The Prometheus, Loki, Tempo, and OpenTelemetry ecosystem has matured to the point where the tooling covers metrics, logs, and traces with the same depth you get from managed services, at a fraction of the running cost. The engineering investment is real, but it is front-loaded, not recurring.
This article walks through the architecture decisions that actually matter: how to size and structure each layer, where the sharp edges are, and how to wire the three signals together so they are useful rather than just present.
Starting with the collection layer: OpenTelemetry as the common pipe
Before you pick backends, settle the collection strategy. OpenTelemetry is the right foundation for a bare metal observability stack because it is vendor-neutral: you instrument once and swap backends without touching application code. The OTel Collector runs as a lightweight agent on each host, receives telemetry from your services, and forwards it to whichever storage backend you choose.
Signal maturity is worth understanding before you commit. The tracing specification is fully stable and covered by long-term support. The metrics and logging signals have reached stable status in the specification, but the OTel Collector itself has mixed stability across its components, so pin your Collector version and test upgrades in staging. One thing to note: the Jaeger propagator has been deprecated in the OTel specification, so if you are migrating from a Jaeger-based setup, plan the propagation format change explicitly rather than assuming compatibility.
The OTel Configuration Schema is still not stable after several years of work, which means programmatic configuration management (Helm values, Ansible templates, and similar) is more reliable right now than relying on the schema for validation. Design your Collector pipelines as code from day one.
Metrics: sizing Prometheus for bare metal reality
Prometheus is the de facto standard for cloud-native metrics, but its resource profile on bare metal is different from what teams experience on managed Kubernetes. It is CPU- and memory-intensive, particularly with high-cardinality label sets. For large environments, multiple cores and at least 32 GB of RAM are the practical baseline. Use NVMe or high-IOPS SSDs: Prometheus’s TSDB runs compaction and query operations that will stall on spinning disk.
Default retention is 15 days, configurable with –storage.tsdb.retention.time. You can also cap disk usage with –storage.tsdb.retention.size, but be aware that Prometheus deletes entire blocks rather than individual samples, so actual storage consumption can slightly exceed the configured cap depending on block boundaries. Plan your disk allocation with a buffer.
Configuring remote write does not reduce local cost. Samples still pass through the local ingestion path, WAL, and head block before being forwarded. If you are adding remote write for long-term storage, budget for the full local resource footprint regardless.
For long-term retention on bare metal, the current actively maintained path is Grafana Mimir (the successor to Cortex, which was also referenced alongside Thanos as a long-term storage backend). Thanos remains a solid alternative, particularly if you want a sidecar model that keeps each Prometheus instance independent. Mimir is better suited to teams who want a single horizontally scalable write path. Choose based on your operational model, not on which project has the louder community at any given moment.
At scale, split scrape responsibilities across multiple Prometheus instances rather than running a single monolith. Federation works but adds latency; separate instances with distinct targets is cleaner for most bare metal clusters.
Logs: Loki over Elasticsearch for bare metal economics
Elasticsearch is powerful, but its operational surface on bare metal is substantial: shard discipline, mapping management, index lifecycle policies, and heap tuning all require ongoing attention. For equivalent log volumes, Elasticsearch typically costs five to ten times more than Loki at scale, and the operational overhead compounds that difference.
Loki’s design philosophy matches bare metal constraints well. It indexes only labels, not log content, and stores compressed chunks in object storage. On bare metal, that object storage can be MinIO running on your own disks, keeping the entire log pipeline within your infrastructure. The multi-component Loki stack (distributor, ingester, querier, query-frontend, compactor, and ruler) looks complex on paper, but each component is stateless or has well-defined state boundaries, which makes it straightforward to operate with standard configuration management tooling.
One practical note: Loki does not handle high-cardinality labels efficiently. Keep your label sets disciplined. Use structured logging in your applications and push cardinality into log content rather than labels. Teams that migrate from Elasticsearch and try to replicate its full-text indexing behavior through Loki labels will hit performance walls quickly.
Traces: Grafana Tempo and the object-storage model
Grafana Tempo is the natural complement to Loki and Prometheus in a self-hosted stack. It requires only object storage to operate, ingests OpenTelemetry, Jaeger, and Zipkin formats, and integrates natively with Grafana for visualization. On bare metal, MinIO handles the storage backend here too, keeping the dependency count low.
The important architectural constraint with Tempo is that it stores traces with no indexing. Finding a specific trace requires a trace ID, which you obtain either from Prometheus exemplars or from a correlated log line in Loki. This is not a limitation to work around; it is the design. Wire your three signals together from the start: emit trace IDs in structured logs, attach exemplars to Prometheus metrics, and the lack of a trace index becomes invisible in normal use. Thoughtworks placed Tempo in the Trial ring in April 2025, noting its Parquet block format and TraceQL query language as the features making it production-ready for serious deployments.
Wiring the stack together: Grafana as the single pane
Grafana connects all three backends and is where the observability stack pays off. Correlating a latency spike in a Prometheus metric with a spike in error logs in Loki and then jumping to the offending trace in Tempo is the workflow that replaces the expensive managed-service dashboards. This works out of the box when you configure Grafana data sources for all three and instrument your services to emit consistent trace IDs across signals.
For alerting, the archival of the open-source Grafana OnCall project in early 2025 means free self-hosted on-call routing through that path is no longer viable. The current practical options are Alertmanager (which ships with Prometheus and handles routing, grouping, and silencing well), or integrating with an external on-call platform. Plan this before you go to production, not after your first 3 AM incident.
If you want to reduce the operational surface further, SigNoz is worth evaluating: it replaces the entire Loki, Grafana, Tempo, and Mimir stack with a single open-source binary. The trade-off is less flexibility and a smaller ecosystem, but for teams who want observability running quickly without managing five separate component upgrade cycles, it is a legitimate alternative.
Running this on bare metal at DIVERSITY
On our Bare Metal platform, we provision servers with the NVMe storage profiles and memory configurations that Prometheus’s TSDB actually needs, and we support MinIO deployments as the object storage layer for both Loki and Tempo, keeping the entire telemetry pipeline within a single-tenant environment. The noisy-neighbor problem that plagues shared-cloud observability backends simply does not exist on dedicated hardware: your TSDB compaction does not compete with another tenant’s query load.
The self-hosted metrics, logs, and traces setup described here is not a cost-cutting shortcut. It is what production observability on bare metal should look like: full visibility, no per-GB ingestion bill, and telemetry that stays exactly where your workloads run.
