Prometheus & Grafana Requirements — Per Target Growth Model

Version 1.6.9 | Updated: August 08, 2026

Prometheus & Grafana Requirements — Per Target Growth Model

How to size the stack when adding servers and exporters.

Every number below is measured, on a named environment, on a stated date. Where a figure is an extrapolation rather than a measurement, it says so — the distinction matters, because the two deployments we run differ by more than 2× in total series and by 8× on individual exporters.

Versions this describes: Prometheus v3.13.1, Grafana 12.3.0, both pinned in prometheus-grafana-stack/.env.example, which is the runtime truth.

1. The two reference deployments

Measured 05.08.2026.

On-prem (grafrd) — VMs and Compose, ~10 monitored hosts:

  • 38 targets, 86.6k active series

  • Prometheus: 0.8 GB resident, 28.9 GB on disk, 10.7k samples/s

  • Retention 90 days, and the TSDB holds a full 90.1 days — so the disk figure is a steady state, not a database still filling up

  • Host: 12 vCPU, 62 GB RAM, 436 GB disk

Kubernetes (ocp-frdly, namespace acs) — 3 nodes:

  • 36 targets, 220.6k active series

  • Prometheus: 1.2 GB resident, 6.9k samples/s

Scrape interval is 10s globally, with 5s on two jobs and 30s on several others (prometheus.yml). Earlier guidance in this file said 60s, which understates ingestion — and therefore disk — by about six times.

2. Series per target — measured, both environments

count by (job) ({name=~".+"}) divided by that job’s target count. A dash means the job does not exist in that environment.

Exporter / Job Kubernetes On-prem Notes

kube-state-metrics

~26.3k

 — 

Cluster object inventory; scales with objects, not nodes

cAdvisor

~20.7k

~0.78k

Per-container. Dominates in Kubernetes, a rounding error on VMs — it scales with pods per node

MySQL Exporter

~18.2k

~27.7k

The largest single exporter on-prem. Grows with tables and performance_schema breadth, never with the device fleet

JmxExporter

~10.6k

~2.6k

Hazelcast + WildFly MBeans. Fixed set per member

process-exporter

~8.4k

~2.6k

Grows with matched process groups

node-exporter

~7.2k

~3.2k

Varies with filesystem and interface count

Longhorn

~2.8k

 — 

Only where Longhorn is the storage provider

ClickHouse

~1.9k

~0.82k

Native endpoint

Prometheus (self)

~1.3k

~1.6k

PostgreSQL

~0.77k

~0.70k

Alertmanager

~0.54k

~0.48k

Northbound API

~0.53k

~1.5k

Provision API

~0.48k

~0.41k

FTACS / acs-metrics

~0.48k

~0.19k

ft-system-metrics

~0.46k

~0.76k

UI Backend

~0.39k

~0.65k

Service API

~0.38k

~0.32k

FT Device Network

~0.15k

~0.18k

AI Agent

~0.06k

~0.09k

Nginx

~0.06k

~0.06k

kubelet

~0.01k

 — 

Three things to take from this.

Size for the infrastructure exporters, not the applications. In Kubernetes, kube-state-metrics plus cAdvisor, process-exporter and node-exporter are 64% of the total; add MySQL and Longhorn and it is 77%. On-prem, MySQL alone is a third. Meanwhile all nine application and API jobs together come to ~4.2k in Kubernetes and ~4.3k on-prem — under 2% either way.

The same exporter is not the same size in both. JmxExporter reports 4× more per target in Kubernetes, process-exporter 3× more, node-exporter 2× more — different hosts, different process lists, different filesystems. Use the column that matches where you are deploying.

These are orders of magnitude, not constants. A node exporter on a host with many filesystems reports considerably more than one with few.

Re-measure rather than trusting this table when it matters:

# on-prem
curl -sG --data-urlencode 'query=count by (job) ({__name__=~".+"})' \
     http://localhost:9090/api/v1/query
# Kubernetes
kubectl -n <ns> exec deploy/ft-prometheus -- \
  wget -qO- 'http://localhost:9090/api/v1/query?query=count+by+(job)+({__name__=~".%2B"})'

3. Storage

Derived from the on-prem measurement — 86.6k series, 10s scrape, a full 90 days on disk:

~0.33 GB per 1k active series.

Retention and scrape interval both move that linearly: halving the interval doubles it, and so does doubling retention. At the default 90 days and 10s:

Active series Disk at 90d / 10s

100k

~33 GB

250k

~83 GB

500k

~165 GB

1M

~330 GB

2M

~660 GB

Leave headroom above these. Compaction needs room to write a new block before dropping the old one, and a full disk stops ingestion rather than slowing it.

4. Growth calculation

  1. List the exporters the new node will run.

  2. Take the series-per-target values from §2, from the right column. If a database exporter or cAdvisor is among them, expect it to dominate.

  3. Sum them and add to the current total (prometheus_tsdb_head_series).

  4. Check §5 for CPU and memory, §3 for disk.

5. Scaling thresholds

  • <2M active series → 4 vCPU / 16 GB is sufficient.

  • 2—​4M → 8 vCPU / 32 GB, SSD 1 TB.

  • 4—​8M → 16 vCPU / 64—​96 GB, SSD 2—​4 TB.

  • >8M → 32 vCPU / 96—​124 GB, SSD 4—​8 TB.

Rules of thumb, not measurements: the largest deployment measured here is 220k series, so every row but the first is extrapolation. What the measurements do support is that the first row is not tight — at 220k series Prometheus holds 1.2 GB resident, so 16 GB has a great deal of room before memory is the limit.

Grafana load is driven by users and dashboards rather than by series count.

6. Examples

A monitored host with a database, on-prem: node-exporter ~3.2k
process-exporter ~2.6k + cAdvisor ~0.8k + MySQL ~27.7k = ~34k series, four fifths of it the database exporter. About 11 GB of disk at 90d/10s. The same host without a database is ~7k, about 2 GB.

A Kubernetes worker node: cAdvisor ~20.7k + process-exporter ~8.4k
node-exporter ~7.2k = ~36k series before a single application pod, plus ~0.1—​2k for each pod scheduled on it. About 12 GB of disk. Note that kube-state-metrics (~26k) is per cluster, not per node — it is paid once.

Both stay far inside the <2M baseline.

branch 1.6.9 · commit a862b45d92590285f229f80a6b5d2efabcb3b1b0 · page generated August 08, 2026