Prometheus & Grafana Requirements — Per Target Growth Model

Version 1.6.9 | Updated: September 04, 2026

Prometheus & Grafana Requirements — Per Target Growth Model

How to size the stack when adding servers and exporters.

Every number below is measured, on a named environment, on a stated date. Where a figure is an extrapolation rather than a measurement, it says so — the distinction matters, because the two deployments we run differ by more than 2× in total series and by 8× on individual exporters.

Versions this describes: Prometheus v3.13.1, Grafana 12.3.0, both pinned in prometheus-grafana-stack/.env.example, which is the runtime truth.

1. The two reference deployments

Measured 05.08.2026.

On-prem (grafrd) — VMs and Compose, ~10 monitored hosts:

  • 38 targets, 86.6k active series

  • Prometheus: 0.8 GB resident, 28.9 GB on disk, 10.7k samples/s

  • Retention 90 days, and the TSDB holds a full 90.1 days — so the disk figure is a steady state, not a database still filling up

  • Host: 12 vCPU, 62 GB RAM, 436 GB disk

Kubernetes (ocp-frdly, namespace acs) — 3 nodes:

  • 36 targets, 220.6k active series

  • Prometheus: 1.2 GB resident, 6.9k samples/s

Scrape interval is 10s globally, with 5s on two jobs and 30s on several others (prometheus.yml). Earlier guidance in this file said 60s, which understates ingestion — and therefore disk — by about six times.

2. Series per target — measured, both environments

count by (job) ({name=~".+"}) divided by that job’s target count. A dash means the job does not exist in that environment.

Exporter / Job Kubernetes On-prem Notes

kube-state-metrics

~26.3k

 — 

Cluster object inventory; scales with objects, not nodes

cAdvisor

~20.7k

~0.78k

Per-container. Dominates in Kubernetes, a rounding error on VMs — it scales with pods per node

MySQL Exporter

~18.2k

~27.7k

The largest single exporter on-prem. Grows with tables and performance_schema breadth, never with the device fleet

JmxExporter

~10.6k

~2.6k

Hazelcast + WildFly MBeans. Fixed set per member

process-exporter

~8.4k

~2.6k

Grows with matched process groups

node-exporter

~7.2k

~3.2k

Varies with filesystem and interface count

Longhorn

~2.8k

 — 

Only where Longhorn is the storage provider

ClickHouse

~1.9k

~0.82k

Native endpoint

Prometheus (self)

~1.3k

~1.6k

PostgreSQL

~0.77k

~0.70k

Alertmanager

~0.54k

~0.48k

Northbound API

~0.53k

~1.5k

Provision API

~0.48k

~0.41k

FTACS / acs-metrics

~0.48k

~0.19k

ft-system-metrics

~0.46k

~0.76k

UI Backend

~0.39k

~0.65k

Service API

~0.38k

~0.32k

FT Device Network

~0.15k

~0.18k

AI Agent

~0.06k

~0.09k

Nginx

~0.06k

~0.06k

kubelet

~0.01k

 — 

Three things to take from this.

Size for the infrastructure exporters, not the applications. In Kubernetes, kube-state-metrics plus cAdvisor, process-exporter and node-exporter are 64% of the total; add MySQL and Longhorn and it is 77%. On-prem, MySQL alone is a third. Meanwhile all nine application and API jobs together come to ~4.2k in Kubernetes and ~4.3k on-prem — under 2% either way.

The same exporter is not the same size in both. JmxExporter reports 4× more per target in Kubernetes, process-exporter 3× more, node-exporter 2× more — different hosts, different process lists, different filesystems. Use the column that matches where you are deploying.

These are orders of magnitude, not constants. A node exporter on a host with many filesystems reports considerably more than one with few.

Re-measure rather than trusting this table when it matters:

# on-prem
curl -sG --data-urlencode 'query=count by (job) ({__name__=~".+"})' \
     http://localhost:9090/api/v1/query
# Kubernetes
kubectl -n <ns> exec deploy/ft-prometheus -- \
  wget -qO- 'http://localhost:9090/api/v1/query?query=count+by+(job)+({__name__=~".%2B"})'

3. Storage

Derived from the on-prem measurement — 86.6k series, 10s scrape, a full 90 days on disk:

~0.33 GB per 1k active series.

Retention and scrape interval both move that linearly: halving the interval doubles it, and so does doubling retention. At the default 90 days and 10s:

Active series Disk at 90d / 10s

100k

~33 GB

250k

~83 GB

500k

~165 GB

1M

~330 GB

2M

~660 GB

Leave headroom above these. Compaction needs room to write a new block before dropping the old one, and a full disk stops ingestion rather than slowing it.

4. Growth calculation

  1. List the exporters the new node will run.

  2. Take the series-per-target values from §2, from the right column. If a database exporter or cAdvisor is among them, expect it to dominate.

  3. Sum them and add to the current total (prometheus_tsdb_head_series).

  4. Check §5 for CPU and memory, §3 for disk.

5. Scaling thresholds

  • <2M active series → 4 vCPU / 16 GB is sufficient.

  • 2—​4M → 8 vCPU / 32 GB, SSD 1 TB.

  • 4—​8M → 16 vCPU / 64—​96 GB, SSD 2—​4 TB.

  • >8M → 32 vCPU / 96—​124 GB, SSD 4—​8 TB.

Rules of thumb, not measurements: the largest deployment measured here is 220k series, so every row but the first is extrapolation. What the measurements do support is that the first row is not tight — at 220k series Prometheus holds 1.2 GB resident, so 16 GB has a great deal of room before memory is the limit.

Grafana load is driven by users and dashboards rather than by series count.

6. Container memory limits

Every service in this stack runs with an explicit mem_limit. A container without one is bounded only by the machine: a single leaking service can consume all host memory and take every other service down with it, including Prometheus itself. MemoryLimit=0 on an FT container is a defect, not a default.

The limit is a hard ceiling — the kernel OOM-kills the container on reaching it. That makes an over-tight limit worse than none at all: it converts a slow problem into an abrupt kill of a healthy service, usually under the load that made it grow. Size from what the service uses on your host, not from a round number.

Shipped defaults, and where each comes from:

Service Default Basis

prometheus

4g

Grows with retention and series count — see §3 and §5. At 220k series it holds 1.2 GB resident.

grafana

1g

Grows with dashboards and concurrent users, not with series.

cadvisor

512m

Scales with the number of containers on the host.

process-exporter

256m

Scales with the number of processes on the host.

oracledb-exporter

512m

The Oracle instant client is heavier than the others.

jmx-exporter-*

512m

JVM processes: heap plus JVM overhead.

mysqld-exporter, postgres-exporter, alertmanager, mailpit, snmp-catcher, alertmanager-config-ui

256m

Fixed footprint; measured at 28—​49 MB steady state.

node-exporter, nginx-exporter, nginx_proxy, snmp-notifier

128m

Fixed footprint; measured at 7—​29 MB steady state.

The sidecars sit an order of magnitude below their limits by design. The headroom is there so that a burst does not kill a working service, and so that genuine growth becomes visible as a rising fraction of the limit before it becomes an outage.

Every limit is an environment variable. Raise one in .env — for example PROMETHEUS_MEM_LIMIT=8g — rather than editing docker-compose.yml, which an upgrade overwrites. Each .env.example lists the variables for its services.

Two limits deserve a second look after any expansion: PROMETHEUS_MEM_LIMIT when servers are added, and CADVISOR_MEM_LIMIT on a host that runs many containers.

Watching usage against the limit

docker stats shows current usage and the limit side by side. In Prometheus, container_memory_rss / container_spec_memory_limit_bytes gives the fraction used, and ContainerHighMemory alerts above 80% of it.

Note that the cAdvisor job drops most series for our own infra and exporter containers to control cardinality — but container_memory_rss, container_memory_working_set_bytes and container_spec_memory_limit_bytes are deliberately exempt for them. Without that exemption the containers we ship limits for would be the only ones nobody could watch approach those limits.

7. Examples

A monitored host with a database, on-prem: node-exporter ~3.2k
process-exporter ~2.6k + cAdvisor ~0.8k + MySQL ~27.7k = ~34k series, four fifths of it the database exporter. About 11 GB of disk at 90d/10s. The same host without a database is ~7k, about 2 GB.

A Kubernetes worker node: cAdvisor ~20.7k + process-exporter ~8.4k
node-exporter ~7.2k = ~36k series before a single application pod, plus ~0.1—​2k for each pod scheduled on it. About 12 GB of disk. Note that kube-state-metrics (~26k) is per cluster, not per node — it is paid once.

Both stay far inside the <2M baseline.

tag 1.6.9 · commit a4c0a5e133c2cba0a8b461039e9634af33e3dd9b · page generated September 04, 2026