Case study 04 · Platform architecture

Private cloud behind the air gap

Designing a private cloud on bare metal OpenShift for an air-gapped data centre, where change takes months and nothing can be fetched from outside.

My role
Solution architect and platform lead
Context
An air-gapped digital engineering platform on OpenShift and IBM Fusion HCI
Stack
IBM Fusion HCI, Bare metal OpenShift, OpenShift Virtualization, Fusion Backup & Restore

Problem

The platform had to run inside a high-security, air-gapped data centre. Nothing can be downloaded, every component arrives through an approved supply chain, and change is measured in months, not days. Sometimes it isn’t possible at all. A design mistake can’t be quietly fixed next sprint.

Building our own from separate servers, storage and switches looked cheaper on paper. In practice it brings considerable overheads: integrating the components, keeping hardware and software aligned, and keeping access to replacement parts. Those costs erode the saving quickly, and they land on the team with the least room to absorb them.

The organisation wanted what a public cloud offers: self-service, automation, and room to run modern and legacy workloads side by side. It had to get there without the public cloud. So the question wasn’t how to get the most capacity for the money. It was how to build a private cloud that keeps running when part of it fails, with no way to reach outside for help.

Constraints

  • Change is slow and costly. Every component is sourced through approved supply chains and verified before it reaches the site. Some choices, like network ranges, can only be changed with a full reinstall.
  • Resilience has to be designed in. Nothing can be pulled from the internet on the day, so the platform has to absorb failures on its own.
  • Parts get harder to find. Late in the hardware lifecycle, replacement and expansion parts may be constrained.
  • The data stays inside. Data science and AI workloads can’t send data to cloud services.
  • A small team runs it. The design had to keep the operational burden low.

Decision

Buy an integrated platform, not a collection of parts. We selected IBM Fusion HCI, an appliance optimised for OpenShift and proven to work as one system. It is supplied with redundancy across nodes, power and storage, an orchestrated install that shortens deployment, and a single console for the whole stack.

Run OpenShift on bare metal. With no hypervisor, workloads have direct access to the hardware. For our platform we put the gain at more than 10% in performance: an estimate for this design, not a figure for every workload. Bare metal also opened up OpenShift Virtualization, so existing virtual machines could move onto the platform as they were, run beside modern workloads, and be refactored into microservices at our own pace.

Scale out, not up. Microservices and a full software delivery toolchain need a lot of general-purpose compute. Four dense nodes, each with 64 cores and a terabyte of memory, would have been compact, but losing one would take away a quarter of the cluster. We chose twice as many nodes at half the size. They still run large workloads, give far more room to balance them, and when one fails the others absorb its load.

4 nodes · 64 cores · 1 TB8 nodes · 32 cores · ½ TB one fails: a quarter gone one fails: others absorb it
Four dense nodes lose a quarter of the cluster when one fails. Eight medium nodes lose an eighth, and the others absorb the load.

Treat storage quorum as its own requirement. Node sizing decides how much compute a failure takes away. Storage resilience is a separate question. In our original minimum storage configuration, losing one storage node was enough to break quorum. Workloads that depended on that storage could not reschedule until quorum was restored. For future acquisitions, we recommend specifying a storage layout that retains quorum after the loss of one node, and verifying that the proposed configuration meets that requirement before accepting it.

Keep a safety net off the platform. However resilient the cluster, a platform-wide issue is always possible. Application backups, taken with Fusion Backup & Restore, and the cluster logs go to an external, fully encrypted flash array that performs on par with the integrated storage. If the platform has a bad day, there is still a secure, fast way back. Proving that an application can be restored from those backups is its own piece of work: see Making Db2 Recovery Fit Production.

original minimum configuration one storage node lost: quorum broken recommended for future acquisitions one lost: quorum kept · verify before accepting off-platformbackups & logs
In our original minimum storage configuration, losing one storage node broke quorum and dependent workloads could not reschedule. For future acquisitions, specify a layout that keeps quorum after losing a node, and verify it before accepting. Backups and logs live off the platform.

Size the network once. Pod, service and storage networks are set at install. They have to be large enough for the workloads and must not overlap enterprise ranges, because changing them later means a full reinstall. We used all four high-speed ports for redundancy, and lined up the patching work and the data centre’s standard switches early.

Bring acceleration inside the boundary. A GPU server gives data scientists accelerated workspaces and lets the team run machine learning and generative AI models locally, so the data never leaves the network.

Outcome

8
medium nodes instead of 4 dense ones
0
downtime when a node’s hardware failed
  • When a node’s hardware failed, operations continued: no downtime and no loss of capability while replacement parts were sourced and fitted.
  • Virtual machines moved onto the platform without changing how application teams worked, and the infrastructure underneath them was consolidated.
  • Machine learning and generative AI run inside the air gap, on work that was previously out of reach.
  • The storage lesson is now our recommendation for future acquisitions: a layout that keeps quorum after losing a node, checked before it is accepted.

Security defines the boundaries. Architecture unlocks the possibilities.

For your team

Behind an air gap, the cheapest design is the one you never have to change. Decide node size, storage layout and network ranges as if they were permanent, because there they nearly are. Size compute for failure, and separately check that storage keeps quorum when a node is lost.

My role

I led the platform design and the team that runs it, and presented the architecture at IBM TechXchange 2025 in Orlando.

Designing a platform behind an air gap? Let’s compare notes.

Connect on LinkedIn