Problem
Cloud-native platforms move fast. To stay secure and supported, an operations team needs to plan for at least two OpenShift upgrades a year, because that is the cadence at which security patches and improvements arrive.
A connected cluster downloads what it needs on demand. Ours can’t. In an air-gapped environment every release image, operator and dependency has to be gathered outside, carried in and loaded into an internal registry before an upgrade can even begin. Without a practice behind it, the speed of cloud-native releases quickly outpaces the platform.
Constraints
- The mirror is the single source of truth. Every workload and operator depends on it. If it slows down, the cluster feels it; if it fails, nothing moves.
- Every transfer step adds time and risk. Content is built in a connected environment, moved on encrypted media, validated and pushed in. A single corrupted package can cost days.
- Support has to be kept. We could only move to versions supported by both IBM Fusion and OpenShift, along a path Red Hat had validated.
- Our own people. The aim was a practice the in-house team could run without outside help.
Decision
Plan before mirroring a single image. Each cycle starts with three checks, in order: the Fusion support matrix sets the highest version we may reach, the OpenShift upgrade graph sets the exact path and its known issues, and the operator lifecycle policy tells us which operators to update before and after.
Move between extended-support releases with control-plane-only updates. Red Hat supports pausing worker nodes while the control plane passes through the intermediate release, then updating workers once. That avoids a round of node drains and reboots, shortens the upgrade and skips mirroring some very large operator catalogues for the intermediate step.
Mirror for an air gap, not for a data centre.
- Filesystem layout: shared image layers are stored once and linked, so a release takes about a third of the space. We package with tar, because zip breaks the links and corrupts the mirror.
- Targeted content: only the operators, channels and versions we need, which keeps the registry lean and transfers short.
- One operator at a time: if a catalogue is corrupted, the blast radius is one operator, not the whole set.
- Permissions: catalogue pods run as arbitrary users, so we set a permissive umask before mirroring. Restrictive defaults in a hardened environment otherwise stop the catalogue from loading.
Treat the registry as a product. It holds over one hundred thousand artefacts. Consistent structure, naming and archiving keep it navigable, and a stable virtual endpoint lets us add and remove content without touching every node’s configuration.
Rehearse first. Before an upgrade reaches the air-gapped cluster, we prove the procedure on a disconnected cloud cluster that can only pull from the mirror.
Outcome
- 2+
- upgrades planned every year
- 1/3
- of the disk space when mirroring to the filesystem
- 1
- operator mirrored at a time, to contain failures
- Upgrades are a regular, predictable process instead of a project, and the platform stays inside vendor support.
- The work is done by our own team, with documented, repeatable procedures and an audit trail of every mirror run.
- Premium hardware and software support meant that when a node failed in the middle of an upgrade, replacement parts and technicians were organised quickly.
Assumptions aren’t shortcuts. They are risks that can cost months.
For your team
If you run disconnected, start with the support matrix and the upgrade graph, not the mirror. Most of the time saved in an air-gapped upgrade is saved before the first image is pulled.
My role
I led the platform design and the team that runs it, and wrote up the practice so others in similar environments can use it.