Know Your Cluster

01 · Problem

Problem

Our OpenShift cluster was stable. Monitoring was configured, alerts were quiet and the dashboards were green. Yet our confidence was low. We relied on consultants and vendor support to tell us what we were looking at.

Banks operate under a standard called Know Your Customer. Its core is simple: you must genuinely understand what you are responsible for. Confirming an identity is the floor, and a bank can’t discharge the obligation by noting that the account looks fine. Change one word and it describes our problem. Know Your Cluster: verify what is running, understand what it does, learn what normal looks like, and notice when it changes. A green dashboard is our version of an account that looks fine.

Kubernetes heals itself. Pods restart, workloads reschedule, and probes fail and recover, often without anyone noticing. Out-of-memory kills, probe flaps and CPU throttling resolve before they become outages. Services stay available and the signals disappear into the background. The same capabilities that make a platform resilient make it easy to stop paying attention to.

02 · Constraints

Constraints

• Nothing new to buy. Monitoring, logging, Loki and the Cluster Observability Operator come with OpenShift. The cost was engineering time and attention.
• Defaults are a minimum, not a production setup. Out of the box, metrics sit on ephemeral storage, retention is measured in days, user workload monitoring is off, alerts have nowhere to go and logs vanish with their pods.
• Much of the software wasn’t ours. A significant share of the problems we found sat in components we didn’t write, so every claim had to be backed by evidence.

03 · Decision

Decision

First, earn stability. We made metrics persistent, extended retention, switched on user workload monitoring, gave Alertmanager somewhere to send alerts and deployed Loki to keep logs off-cluster. Then we worked through every alert the cluster raised until there were none left standing.

Then, alert on the event, not on availability. A self-healing platform will restart a container and keep serving traffic, and an availability check sees nothing wrong. We weren’t willing to treat recovery as the same thing as safety, so we correlated restarts with their termination reasons:

- alert: PodOOMKilled
  expr: kube_pod_container_status_last_terminated_reason{reason="OOMKilled"}
  labels:
    severity: warning

The alert is deliberately sticky. It stays up until the pod is replaced, which is exactly what happens when the underlying limit is fixed. A transient event becomes a standing signal that has to be dealt with.

Then, prove it works. We codified routine checks into a daily Platform Health Check covering storage, backup and restore, and cluster health. We keep a set of workloads that fail on purpose, so every rule is tested against the failure it was written for. We even take the monitoring stack down deliberately, because an alert that has never fired is an assumption, not a control.

Then, make it tell us where to look. An automated check runs every morning and reports what matters: failed and crashed pods, node status and the control plane version. The point of observability isn’t to give people more to watch. It’s to tell them where to look.

Finally, define what good looks like. Perses dashboards, delivered through the Cluster Observability Operator, cover three pillars: cluster health, resource utilisation and security profile. Vendor defaults are sized for a lab; tuning them is the job of the people running the platform, using evidence from the platform itself.

04 · Outcome

Outcome

0

chronic alerts; when one fires, the team acts

30+

dashboards across three pillars

Daily

automated platform health check

$0

new licences

• Outages became avoidable rather than survivable. Transient events are pre-incidents. Clearing them took months, one component at a time, until none were left.
• Upgrades became engineering rather than hope. We capture a baseline before every upgrade and run the same checks after. Without one, triage becomes archaeology: is this new, or was it always there?
• Capacity decisions rest on evidence. We found containers throttled at 100% and others holding resources they never used, and sized them against what they actually did.
• Vendor conversations changed. “The service is available” is hard to argue with until you can prove the impact. Evidence moves a support case from opinion to fact.
• Knowledge outlives individuals. Checks live in software, not in whoever happens to be on shift.

Observability is not the goal. Understanding is. You can’t be accountable for something you don’t genuinely understand.

For your team

Everything here ships with OpenShift. The investment is attention, not licences. Start by alerting on events, not availability.

My role

I set the approach, led the platform team through it, and took the evidence to vendors.