Speaker notes; captures the intent of the talk rather than an exact copy of the recording. Lightly edited for reading. Two terms replaced in [square brackets] under the site’s OPSEC rules.
Introduction
Good morning everyone, thank you for taking the time to join me today. Before we dive in, a quick intro. My name is Nick Draper I’m a Solution Architect within the Australian Department of Defence, where security and resilience are core principles. I have spent the last decade leading deployments like the one I will present today. I will share our success and the opportunities for improvement, openly and honestly. Please hold your questions for the Q&A section at the end of the session
This session follows the path we took, from first principles to a fully operational platform. Here’s how it unfolds. First, I’ll set the scene why this deployment had to be fully air-gapped, and how strict security requirements shaped every decision we made. Then we’ll dive into the architecture, exploring key design decisions that shaped the platform. After that, we’ll focus on the registry. In an air-gapped world, it’s not just a service it’s your lifeline.
From there, we’ll shift our focus to the deployment and configuration of platform services. And finally, we’ll talk about operations. From support to managing the platform lifecycle. At each step, I’ll share what worked, what didn’t, and what I’d do differently.
Why the platform is air-gapped
Defence’s mission remains constant, clear, unwavering, and essential to the nation. But what does it take to deliver on that? Defence is one of the nation’s largest and most complex organisations, with over 100,000 personnel, and an operating budget that represents 2% of the nation’s GDP. With that scale comes enormous responsibility. Defence isn’t just about war fighting. It’s also about safeguarding information and maintaining readiness, where the stakes are nothing less than national security. And that’s why, for us, security isn’t a feature it’s the foundation. Every system, and every process, has to live up to that mission.
We deployed the platform inside a high-security, air-gapped data centre. An environment completely isolated from the outside world, where every action is tightly controlled. Security is never a single barrier. It’s layer upon layer, each designed to slow, stop, and detect. This is defence in depth. No one control is enough on its own, but together they make compromise extremely difficult. It begins with people: comprehensive background checks followed by ongoing vetting, recognising that insider threats remain a serious risk.
Next is multi-factor authentication, such as passwords, physical tokens, and biometrics. Device control adds another layer, Every component must be sourced through approved supply chains, verified and validated before it ever reaches the site. Within secure areas, only approved, encrypted devices are permitted. And finally, physical safeguards reinforce the digital controls, with zone-by-zone approvals that strictly regulate who crosses each boundary. Unlike commercial settings, change is costly. You can’t just swap a device or update an access list. Each adjustment demands careful consideration and approvals Change is measured in months, not days, and sometimes it isn’t possible at all. That’s why detailed planning is everything. These constraints shaped every architectural decision.
Picture this: you’re at an [air-gapped] site. No internet. No external devices. And yet, you’re expected to deliver an agile platform that is resilient and secure. This was the challenge we faced. We have been conditioned by the convenience of connected environments, the ability to pull images from public registries, and iterate rapidly.
At an [air-gapped] site, none of those assumptions hold true. Instead, every patch, and every update, has to be moved through secure pipelines. In this session, I’ll take you through the deployment of the DSLab platform. The D and the S in DSLab is multifaceted. It represents how our team integrates systems, transforms data, and presents actionable insights to decision-makers. Our platform delivers predictive capabilities for Defence that reduce cost, optimises resource utilisation, and helps maintain readiness. I can share our approach and architectural principles, but the delivered solutions remain classified. This platform is just one element within Defence’s vast analytics network, but it shows how modern approaches can succeed in the most constrained environments.
Architecture: a strong foundation
Security defines the boundaries. Architecture unlocks the possibilities. So how do you design a platform that operates within these constraints It starts with the strong foundation…
Custom-built infrastructure, may look cheaper on paper. But in reality, it introduces considerable overheads: integrating components, keeping hardware and software aligned, and maintaining access to replacement parts. These factors quickly erode the initial cost savings. We needed an integrated platform that was proven to work as a unified system, not just a collection of parts. That’s why we selected Fusion, an appliance optimised for OpenShift and built for resilience.
The platform is supplied with redundancy across nodes, power, and storage, ensuring operations continue without interruption. Next we considered implementation speed. Fusion provides an orchestrated install, that streamlines provisioning and accelerates deployment. Saving valuable time and reducing complexity. And last but not least, it offers a unified view via the Fusion Management console. A single pane of glass to manage the entire stack, reducing the operational burden. Fusion gave us a solid foundation The next step was to acquire an OpenShift subscription for the platform.
The platform was designed for Bare Metal OpenShift. This model simplifies licensing by covering the entire node regardless of virtual CPUs. And the cost difference is dramatic. For example, with a 32-core node, license costs are reduced by 81% Scale that up to 64 cores, and the savings are even greater. That means fewer licenses to manage, lower subscription costs, and a much better return on investment.
But it isn’t just about cost. Bare metal provides more than a 10% boost in performance. With no hypervisor, workloads have direct access to the hardware, preserving valuable resources. The system feels immediate and fluid, delivering a responsive experience for developers and end users. So, when you combine the cost savings with the performance gain, the model was perfect.
It also opened up new opportunities OpenShift Virtualization is only available on bare metal. This operator allowed us to convert Virtual Machines into containers, and run them side-by-side with our modern workloads. Our approach followed three stages. We first evaluated applications by effort, risk, and impact to identify the best migration candidates. Next came a light-touch, lift-and-shift migration. By running VMs natively on OpenShift, application teams kept their familiar processes and tooling, while we consolidated the underlying infrastructure.
The migrated workloads could now be managed as code with version-controlled configuration of resources and machine state, just like modern workloads. We then progressively refactored the applications into microservices. This phased approach simplified management, lowered operational cost, and enabled us to modernisation at our own pace. Now we turn our attention to the hardware, the compute that will power our workloads
Compute and acceleration
A critical design decision we faced was the hardware configuration. This modern microservices architecture brings clear advantages, such as independent scaling and faster releases, but it also increases the resource footprint. Separate runtimes, side processes, and additional management layers. All raise compute and memory requirements for the same function. The platform also had to support the entire software-engineering delivery pipeline. This included every tool required to build, release and maintain applications. To meet these demands, we needed a robust, resilient base of general-purpose compute. Should we deploy a dense setup with high-capacity nodes, or scale out with twice as many medium-sized nodes? Consider four nodes, each with 64 cores and a terabyte of RAM. Dense nodes like these Deliver massive compute in a compact footprint. The trade-off? Less flexibility in workload distribution. And if one node fails, you immediately lose 25% of your cluster capacity. Now consider doubling the cluster size and halving node resources to 32 cores and half a terabyte of RAM. This setup still supports large workloads, but it provides much greater flexibility in balancing them across nodes. It also offers stronger resiliency, if one node fails, the others can absorb the load. That resiliency paid off. When we experienced a hardware failure. Thanks to our scale-out configuration, operations continued without disruption.
No downtime. No loss of capability. The system stayed fully operational until the replacement parts were sourced and fitted. Now just one final consideration. If you leave nodes vacant for future expansion, remember that sourcing components late in the lifecycle may be constrained by part availability.
Next, we added a GPU server to the cluster This enables us to support workloads that were previously out of reach. And in an air-gapped environment, those capabilities are even more valuable. Starting with, accelerated workspaces This means data scientists can iterate faster, experiment freely, and deliver insights all within a secure, reproducible container. It also enables Machine Learning. We can leverage compute-intensive workloads such as Natural Language Processing, and Computer Vision. that were previously impractical.
And finally, everyone’s favourite, Generative AI. In most enterprises, this means cloud services. But in Defence, we simply cannot send our data outside the wire. By running models locally, we can explore cutting-edge AI applications. This lets us innovate without ever sacrificing security.
Storage
Now that we’ve covered compute, let’s talk about storage. Fusion supports a variety of storage options, so you’ve got flexibility. Choose the most suitable option based on your storage and resiliency requirements What I want to focus on is drive configuration. Imagine a cluster with eight nodes. Depending on your needs, the minimum configuration will vary. For basic object storage, we only need three nodes, so lets add 10 flash drives to these storage nodes.
But if you need advanced data protection, the minimum jumps to six nodes. Looks efficient, right? But here’s where we ran into a problem. In our first deployment, we accepted the default minimum configuration, which consolidated storage onto the minimum set of nodes. On paper, it satisfied the requirement. But in practice, it created a critical point of failure. Regardless of which storage service you choose, I would recommend distributing storage across all nodes. Consolidating offers no benefit, and it undermines resilience. Balance is what keeps the platform stable. It’s not just good practice, it’s risk mitigation.
No matter how resilient we make it, there is always the potential for a platform-wide issue. That’s why external storage is critical. It’s our safety net. We selected the IBM FlashSystem to safeguard our application backups and logs. We utilise the Fusion Backup & Restore service for namespaces and Loki for cluster logs. The storage array is equipped with high-performance flash storage that is on par with the integrated Fusion storage. And importantly, the device is fully encrypted, so the information remains protected at rest.
In short, external storage isn’t optional. It completes the resiliency strategy. So if problems occur, we have a secure, high-performance recovery path. Now, with the storage locked in, we shift our focus to networking.
Networking
Planning is critical, even minor oversights can lead to issues down the line. Taking the time to perform detailed validation will help avoid costly delays. Fusion recommends default network ranges for the pod, service, and storage networks. But you need to ensure that these are sized to suit your workload requirements. For example, the pod network directly limits the number of concurrent pods. You also need to ensure that any external-facing ranges, do not overlap with enterprise networks.
Otherwise, you’ll run into routing conflicts and connectivity issues. Just remember, updating network ranges requires a full re-install. Don’t ask how I know. Next is throughput. Fusion supports a pair of 100-Gigabit links on each of the high speed switches. However, your data centre may only support 40, 25, or even 10-Gigabit links. That’s where splitter cables, transceivers, and adapters come in. They help you match port speeds, connect reliably, and ensure the cluster delivers optimal throughput. Regardless of your final configuration, ensure you use all four ports, if possible. It’s the best configuration for maximum redundancy.
Then there’s data centre patching work. Something easy to miss, so line this up early. We had to fit a patch panel into the top two rack units, which required equipment and coordination with external service providers. And finally, compliance. Your data centre may standardise on switch vendors, meaning Fusion’s integrated switches may not align. We had to route them through in-pattern switches to stay aligned with enterprise standards. So make sure you confirm switch suitability early.
The registry server
This slide is the most important of the deployment story. In a fully disconnected environment, success depends entirely on having the right local services in place. Without them, the cluster simply cannot function. First, we needed a mirror registry. Not just for installation, it’s a critical service for ongoing operations. In an air-gapped environment, every image must be staged locally. Treat the registry as a first-class service, not an afterthought. Alongside it, we hosted a private Git server for our source code and configuration repositories. Together, they form the backbone of our disconnected workflow. Critically, this server also functioned as an administration node. It gave us a control point to initialise, patch, and manage the platform. And when incidents occur, it serves as the primary hub for connecting to hardware and running diagnostics. When we deployed Fusion, it only supported block and file storage natively. To bridge the gap, we installed MinIO on the registry server. This service presents block storage as the S3-compliant object storage that modern workloads expect. To complete the solution, we connected the storage array to the registry server via a pair of high-speed Fibre Channel cards.
And finally, this server also enabled us to validate the air-gapped deployment. We added the core services every cluster relies on, DNS, DHCP, and NTP, all hosted locally on the registry server. This standalone network enabled the project team to complete the install, before it ever touched the enterprise network Together, the registry server formed the backbone of our air-gapped platform. Enabling administration, independent deployment, object storage, and secure delivery of artefacts and code.
All these decisions came together to form a unified platform built to deliver performance, resilience, and security. Bare-metal OpenShift, Scale-out compute with acceleration. Resilient storage on and off the cluster. And a secure private registry. This platform supports everything from software engineering to advanced analytics and emerging workloads.
The mirror registry
Now, Let’s take a look at the registry, the cluster’s trusted source for container images. I’ll outline good practice for mirroring content in an air-gapped environment. Then, I’ll discuss how to manage the registry efficiently to ensue secure and reliable image delivery.
Think of the registry as the heartbeat of an air-gapped cluster. Every workload, starts with an image pull from the registry, services, operators, and applications. Because we have no external connections, this is our single source of truth. If it slows down, the entire cluster feels it. If it fails, nothing moves. Keeping it in close proximity to the cluster, and keeping it well-organised, is what keeps the whole environment stable. Mirroring content for an air-gapped cluster is never quick. We can’t just push images from an external network directly into the registry. Instead, the task requires multiple steps We build the mirror in a connected environment using trusted sources, then download and transfer the content to encrypted media. That media is then transported to the secure site, and uploaded for validation, before pushing it into the internal registry. Every step adds time, and risk of corruption. A single mistake can be costly, and that’s why we need a process that is efficient, repeatable, and auditable.
To achieve this, we focus on four key practices. First, the Process. oc mirror is the tool that creates and maintains a local mirror for air-gapped environments. An Image Set Configuration file defines the images and metadata that are downloaded. We follow standardised workflows that preserve the commands executed, the configuration assets, and the console output. This isn’t just housekeeping, it provides traceability for validation, and assists with troubleshooting. In a secure environment, that audit trail is non-negotiable. When you need to move content to an air-gapped network, I recommend utilising the filesystem option. This option reduces duplication by linking common layers which significantly minimises the disk space required. For example, a 60-gigabyte OpenShift release is only 20 Gigabytes when mirrored to the filesystem. Just be mindful when packaging for transfer, use a method that preserves symbolic links like tar. Standard zip will break those links and result in a corrupted package. Third, be targeted. Only mirror the operator packages, and channels you need. You can refine it even further by using min and max Versions. That keeps the registry lean, conserves storage, and reduces mirroring time.
We only mirror and package one operator at a time. This ensures the blast radius of a corruption is minimised. And finally, catalog permissions. When you mirror an operator, oc mirror creates a pruned catalog index. that curated catalog contains the images downloaded. This catalog is first written to local disk, where file permissions are inherited from the user’s profile. In high security environments, files are not created with world-read access. These restrictive permissions will prevent the OpenShift catalog pods that run under arbitrary Users IDs from reading the catalog. To avoid this, set a permissive umask of 0022 for the session.
oc mirror also generates the Image Digest Mirror Set or IDMS. This configuration redirects the cluster to pull operator and release images from your registry. The mirror entries are prefix-based mappings, so only the parent repository path is specified, This allows redirects for all descendant image digests. I want to highlight the significance of a digest when using a mirror registry A digest is a cryptographic hash of the image manifest, immutable and verifiable. So the cluster always pulls the trusted image.
Whereas Tags are mutable labels that can be reassigned to a different image, creating a risk of pulling unverified or tampered content. This immutability guarantees that every deployment uses verified images. Once the IDMS is applied, the Image Registry Operator creates a new machine configuration, and the Machine Config Operator rolls it out to every node. Every time the IDMS changes, the cluster triggers a full machine-config rollout, so it’s critical to manage changes. Inserts are typically quick, but updates and deletes require node drains and reboots. So plan ahead to avoid unnecessary rollouts across the cluster.
Even after the IDMS is configured, we need the ability to modify the registry for upgrades, and to manage registry bloat. This is where the idea of a virtual endpoint helps. It gives us a single stable reference, that eliminates the need for machine config updates. We can add or remove images from the registry, Without the need to touch the machine config.
From a technology standpoint, you can implement this in several ways. Some registries, provide native virtual-repository features, if yours doesn’t, the same result can be achieved with a reverse proxy. Either method will enable you to add or remove content efficiently.
Our mirror registry now holds, over one hundred thousand artefacts. And that’s with active curation. Without deliberate organisation, that sheer volume quickly turns into chaos. Troubleshooting becomes painful, upgrades slow down, and even simple tasks take longer. That’s why organising the registry isn’t optional, it’s mandatory. By structuring repositories, using consistent naming, and archiving what’s obsolete, we keep the environment predictable and upgrades manageable. A well-organised registry is what allows us to scale, audit, and troubleshoot with confidence. With all the content loaded, we now need to ensure our registry is trusted. In an air-gapped environment, identities are signed by an internal Certificate Authority or CA. Every endpoint, from the registry server through to cluster ingress points, needs signed certificates issued by the Enterprise CA. By default, applications trust only the public CAs. To enable secure communication, the Enterprise CAs must added to the trust stores of the applications on the registry server and the cluster. And don’t forget to plan certificate renewals. Operations teams need to track expiration dates and renew certificates well in advance. Expired certificates cause service outages.
Deployment
Let’s we shift our focus to deployment. Moving from cloud validation to full operation. And wrapping up, with a look at our internal developer platform. A capability that lets us do more with less.
Before we attempt an air-gapped deployment, we rehearse using a cloud cluster. Importantly, this cluster is disconnected from the cloud catalogues. Configured to only source images from the mirror registry. Effectively simulating our air-gapped network. Early in the lifecycle, we identify, evaluate and mitigate security vulnerabilities. In addition, we also installed and configured our core operators. This process enabled us to refine and prove our procedures
To serve as a suitable disconnected upstream, the deployment needed to be self-managed OpenShift. While the managed offerings are convenient, They unfortunately hide the management nodes and critical machine configurations. With a self-managed cluster, we have full access to the control-plane nodes and every internal component. We have the freedom to tune all OpenShift configurations and even schedule upgrades on our own timeline. This combination, self-managed control and disconnected validation, gave us the confidence for our air-gapped deployment.
All that remains is the pre-deployment readiness check. The mirror registry is fully loaded and validated. Our network design is locked in and double checked We have acquired the Enterprise CAs and our signed certificates for all services. Lastly, a final hardware check to verify that each component is ready for deployment. With the right preparation and planning, the actual deployment will be smooth. We’ve already done all the hard work.
Platform services
Once the cluster is running, the next priority is configuring the core services that keep it healthy and resilient for the long term. Monitoring is the first line of defence for both the platform and the applications you deploy. OpenShift includes a built-in monitoring stack that tracks services and issues alerts. By default, it focuses on cluster health only, the application workloads are not tracked. So, we configured User Workload Monitoring which enables Prometheus and Alertmanager to track the full cluster.
We also created cluster wide custom alert rules, Such as tracking Out of Memory pod kills. This allowed us to surface these silent issues and enable the ops team to address the cause. A key part of our setup is Alert routing. Alerts are only present while firing, so recording them is essential. We configured Alertmanager to send email notifications. This captures brief or transient events that may go unnoticed. The next key layer is centralised logging. We deployed and configured Vector to collect logs from every pod. Vector then forwards those logs to Loki. This operator persists the logs off-cluster in S3 buckets on our Storage Array. This ensures that in the event of a cluster failure, our complete log history remains secure and accessible for audits and investigations.
When we needed in-depth monitoring, we deployed the Cluster Observability Operator. It provides custom monitoring stacks scoped to specific namespaces. This enables teams to optimise or troubleshoot key capabilities And finally, for quality of life, we utilise Keycloak for Single Sign-On. One login for all cluster applications.
The internal developer platform
With the core services in place, the platform is ready for the real goal: empowering application teams. This is where our Internal Developer Platform or IDP, takes centre stage. At its foundation is our registry server which supplies the packages, Git repositories, and container images, everything a developer needs to build and release software inside an air-gapped environment. Above that foundation, four operators power the workflow: OpenShift Dev Spaces, Pipelines, GitOps, and Advanced Cluster Security. By leveraging built-in automation, we enable small teams to do more with fewer resources, accelerating delivery while reducing manual effort. The platform is secure by design. Starting with the foundation, all our custom container workloads are built on Universal Base Images. These images are hardened by Red Hat and benefit from regular vulnerability patches and security advisories. Next is Quality. Our pipelines are more than just automation; they enforce unit tests, code coverage, and coding standards. Ensuring every commit meets a baseline of reliability and maintainability.
Red Hat Advanced Cluster Security delivers the next two pillars: Scan and Detect. An ACS scan has been integrated into the promotion pipeline for new images. This task identifies known vulnerabilities and flags insecure configurations like running containers as the root user. If critical issues are found, promotion is blocked and developers are notified. This proactive approach ensures that insecure images never reach the cluster. At runtime, ACS continuously monitors the cluster for anomalous processes, suspicious network activity, and privilege escalations. It doesn’t just detect, it enforces. By blocking policy violations in real time, ACS ensures the cluster remains secure. So when we say secure by design, we mean it. Again, Security is not just a single layer, it’s a discipline that is enforced at every stage.
Our developers love OpenShift Dev Spaces because it provides a cloud-native development environment with everything they need on demand. With just a browser, developers can start coding in minutes in identical, reproducible environments, with no local setup required. Using a Devfile, each workspace is defined as code, specifying all containers and resources needed to build and test an application. This approach accelerates onboarding, and eliminates the “it works on my machine” problem, By ensuring every workspace runs from approved images and with consistent configurations.
As developers finish features, they commit to the Quality Assurance branch. That action triggers the validation streams within the Tekton pipeline. Parallel execution means a shorter feedback loop, giving developers the insights they need without relying on specialists. Once validated, the image manifest is committed to the configuration repository. This triggers Argo CD to roll out the new image to the cluster, allowing the team to complete System Integration and User Acceptance Testing. The result is a promotion process that is consistent, reliable, and trusted. With it, we can release to production with confidence. That was a very a brief look at our IDP. Now, here why is matters. It delivers security. Every image is built on a secure foundation, And our registry only contains verified, trusted artefacts. It delivers speed. A developer can be productive in minutes, not days. One-click environments means lower on-boarding costs, faster delivery, and more time spent building capability. It also delivers continuous assurance, Automated pipelines enforce testing, the building, and scanning, so by the time we’re ready to release, the software is already proven. And finally, it delivers confidence. Because the cluster is managed by GitOps, every deployment is consistent, automatic, and free from human error.
Operations
Now it’s time to move onto Operations. Behind every resilient platform stands a strong operational team. They respond fast, minimise disruptions, and maintain pace with the platform lifecycle. This is the important work that keeps everything secure and stable.
Collaborating with vendor support teams is a core discipline for any operational team. The incident resolution time is dependant on sharing the best information to diagnose the issue. If your team is not prepared, you lose precious time, exactly when you can least afford it. That’s why we follow a simple checklist. First, collect. Everyone on the team needs to know how to run core diagnostics, such as an SOS report or a must-gather.
Next, clean. Logs contain sensitive information such as hostnames and IP addresses. In a [secure] domain, this sensitive information must be redacted. Red Hat provides tools to clean their logs, but with other vendors you will need to develop your own. Third, capture. The support team needs the complete picture. Describe the problem, explain the impact, and provide full the context including versions and site configurations Don’t assume they have all the details, they assist many customers and rely on the information you provide. Finally, condition the operations team. Document the steps, practice them, and make log collection and sanitising routine, so an incident triggers a rapid response. The worst time to learn how is in the middle of a Sev-1.
Serious problems rarely wait for business hours. They show up at midnight, on weekends, and at inconvenient times when the experts have gone home. That’s why premium support is essential. It provides us 24/7 coverage So our incident can be prioritised and routed to the right technical team. This keeps downtime to a minimum.
Basic support is simply not adequate, We cannot afford to spend valuable time with inexperienced Level one staff. During a Sev-1, we need access to resources with in-depth product knowledge. And importantly, we secured coverage for the hardware. This enabled us to rapidly organise replacement parts and technicians when a node failed in the middle of an upgrade.
Cloud-native platforms move at a rapid pace. This slide highlights our journey for OpenShift and Fusion over the last few years. To stay secure and supported, I recommend planning for at least two upgrades every year. That cadence aligns with how cloud-native services evolve. They deliver frequent security patches and feature improvements.
Each upgrade cycle needs early planning. Dependencies between operators must be checked, known bugs reviewed, and deprecations assessed. Without preparation, the speed of cloud-native releases can quickly outpace the platform. By treating upgrades as a regular, predictable process, we keep the platform stable, secure, and fully supported.
Closing
If I can leave you with just one message: success in an air-gapped, high-security environment is never about luck. It’s takes deliberate planning, rigorous validation, and an unwavering commitment to security at every layer. From first principles and architecture through to operations, the details matter. Validate each step before you move on. Assumptions aren’t shortcuts, they are risks that can cost months.
Carry that discipline, back to your own environments. Build processes that catch issues early, document your procedures, and leave nothing to chance. That is how you deploy a cloud-native platform in an air-gapped data centre. I hope this session gave you some practical insights, or maybe confirmed the strengths of your own approach. I’d be glad to hear your questions Thank you