How do you monitor streaming data infrastructure at scale?

To monitor streaming data infrastructure at scale, alert on a small set of signals that mean data is at risk or late: replica health, controller state, request latency, handler saturation and consumer lag. Keep the rest as dashboards. Apache Kafka's documentation gives expected values for most of these, so you can alert on deviation.

Why does streaming need different monitoring?

A batch job fails loudly. A stream degrades quietly. A broker can lose a replica while producers keep sending, and the first symptom may be a consumer that is an hour behind. Monitoring has to catch the early signals, before users feel the delay.

This post uses Apache Kafka as the worked example because its documentation lists metrics with normal values. The same thinking carries over to other streaming systems: protect durability first, then freshness, then capacity.

Which signals protect your data?

Kafka's monitoring documentation names several metrics with an expected value of zero or a specific shape:

• UnderReplicatedPartitions: partitions whose in-sync replica set is smaller than the full replica set. Expected 0.
• UnderMinIsrPartitionCount and AtMinIsrPartitionCount: partitions below or exactly at the minimum in-sync replicas. Expected 0. Below the minimum, writes that require acknowledgment can fail.
• OfflineReplicaCount: expected 0.
• UncleanLeaderElectionsPerSec: expected 0. An unclean election can lose data.
• ActiveControllerCount: exactly one broker in the cluster should report 1.
• IsrShrinksPerSec and IsrExpandsPerSec: expected 0 except when a broker goes down or comes back and catches up.

Page on the first group. A nonzero value in any of them means redundancy is already reduced.

Which signals show delay?

Consumer lag is the one users notice. Kafka's consumer metrics include records-lag-max, the number of messages a consumer is behind the producer, and it is published by the consumer, not the broker. Alert on lag that is growing, not on a single high value, because a short spike after a deploy is normal.

For brokers, the documentation lists request timing metrics, including total time and its parts: time waiting in the request queue, time processed at the leader, time waiting for the follower, and time in the response queue. When total time rises, the breakdown tells you where to look. It also gives idle percent metrics for network processors and request handler threads, between 0 and 1 and ideally above 0.3. A drop below that means threads are saturated and latency is about to climb.

How do you keep alert volume sane at scale?

Hundreds of topics and thousands of partitions will produce noise if every metric pages someone. Three habits help:

1. Alert on symptoms tied to a person's action. Under-replicated partitions are actionable. Individual partition counts are not.
2. Aggregate. Alert on the number of topics with growing lag, not on each topic, and route to the owning team by topic name.
3. Check balance as a dashboard. The documentation says partition counts and leader counts should be mostly even across brokers, which is a capacity-planning signal, not a page.

Constructed example: a cluster serving payment events shows lag growing on one consumer group, while request handler idle percent on two brokers has dropped to 0.15. The on-call engineer sees both in one view, concludes two brokers are overloaded, moves leadership off them and watches lag shrink.

Is the monitoring path itself secure?

Easy to forget. Kafka's documentation says remote JMX is disabled by default, and that if you enable it for production you must enable security so unauthorized users cannot monitor or control your broker or the platform it runs on. It notes authentication is disabled for JMX by default and that security settings must be overridden for production deployments. A metrics port open on a flat network is a control channel.

Protect it with authentication and TLS, restrict who can reach it, and prefer an exporter that runs on the broker host over opening remote access.

How to check or apply this

1. List the metrics that mean data loss: under-replicated, under min ISR, offline replicas, unclean elections.
2. Page on those, with the documented expected value as the threshold.
3. Add consumer lag trend alerts per consumer group, routed to the owning team.
4. Track request time breakdown and idle percent to catch saturation early.
5. Secure JMX or any metrics endpoint, and keep it off untrusted networks.
6. Test an alert by stopping one broker in a non-production cluster and confirming the right people are notified.

Quick check

Sources

• Apache Kafka documentation, Monitoring (version 4.1)

Last reviewed: 18 October 2026.