Case study 04 · Backup and recovery

Db2 Backup & Restore

Adapting a sample backup recipe to a production namespace layout, so a Db2 database can be restored where it is needed and the restore can be trusted.

My role
Solution architect and platform lead
Context
A digital engineering platform on OpenShift and IBM Fusion HCI
Stack
IBM Fusion Backup & Restore, Db2, OpenShift, GitOps

Problem

Backing up the platform isn’t the same as being able to restore an application. Our first Db2 backups ran on Fusion’s default recipe, which has no Db2 steps, so they most likely weren’t application-consistent.

The IBM Fusion team publishes a sample Db2 recipe that adds those steps. It is written for a lab, where the Db2 operator and the database share a namespace. Our production layout doesn’t work that way. The operator runs in its own namespace, deployed and upgraded through GitOps, and each database instance lives in a project of its own.

We also wanted to restore a database somewhere other than where it came from: into a development or test namespace, or beside production, without touching the source.

Constraints

  • Back up the instance, not the operator. The operator is a platform component. It is redeployed from configuration, not recovered from an application backup. Each backup had to capture its own database instance and nothing else.
  • Don’t reach into the operator. The sample pauses the operator by running a command inside the operator’s pod. With the operator in another namespace and managed by GitOps, the recipe shouldn’t depend on it.
  • A failed backup mustn’t leave production stuck. A backup that fails while Db2 is suspended could leave the database unable to accept writes.
  • A restored copy mustn’t look like production. A backup carries production’s settings with it, including where the database may run and the labels that say which environment it belongs to.

Decision

Split the recipe into a parent and a child. Fusion supports dynamic recipes. The parent is a harness with no hooks of its own; the child holds every hook and step. A namespace variable resolves to the source on backup and to the target on restore, so neither recipe is tied to one namespace. Every database instance shares the same pair, and each keeps its own schedule and retention.

Name what is backed up. IBM’s sample excludes some resource types and backs up everything else. I changed it to an allow-list: the instance’s volumes, its configuration and secrets, and the database instance itself. A new resource type stays out until someone adds it, so the scope fails closed. The trade-off is that the list has to be reviewed against each deployment.

Pause the operator with annotations, not commands. Db2 has a maintenance annotation that stops the operator acting on the database during a snapshot. The child sets and clears it with Fusion’s annotation hooks, which Fusion applies through the API, so no hook ever runs in the operator’s namespace.

Undo completed steps if a later one fails. A backup pauses the operator, suspends writes, snapshots the volumes and then resumes. The pause and suspend steps each declare an inverse operation. If a later step fails, Fusion attempts to undo the completed steps, newest first. It is a safeguard against prolonged disruption, not a guarantee of recovery.

Rewrite what production carries with it. Mapping the namespace isn’t enough on its own. A resource transform replaces the environment in the instance’s node placement and labels as it is restored, so the copy runs on non-production nodes. It also marks the restored objects, so nobody mistakes the copy for live data.

operator · GitOps productionbackupinstancetransform development or test volumes, config, secrets and the instance, nothing else
The operator sits outside the backup. Each backup holds one database instance; a restore with a transform puts a copy into another namespace, where the operator reconciles it.

Outcome

1
parent recipe shared by every database instance
3
resource groups in the backup scope, and nothing else
0
hooks run in the operator’s namespace
  • We used cross-namespace restores to refresh development and test environments from production backups, without affecting the source database.
  • Restoring a copy beside production let us check its integrity and confirm the restore process works, so a backup is proven rather than assumed.
  • The recipe, manifests and a deployment guide are public in the recipe repository under the Apache 2.0 licence. I wrote it up as a walk-through on Medium and for the IBM TechXchange Community.
  • I have proposed the recipe upstream, as a production variant alongside IBM’s original sample.

For your team

If your operator and your databases live in separate namespaces, don’t adapt a sample in place. Start from an explicit backup scope, keep the operator out of it, and restore into an empty namespace before you need to. A restore you have run tells you more than a backup you have scheduled.

My role

I adapted IBM’s sample for our namespace layout, tested the backup and restore on our clusters, and published the recipe, its deployment guide and both write-ups. The IBM Fusion team wrote the sample this builds on, and the features it uses (dynamic recipes, annotation hooks, inverse operations and resource transforms) are Fusion’s own. The recipe builds on community articles by Ashish Gupta, Sandeep Prajapati and Jim Smith. IBM Expert Labs consultant Prasoon Sinha worked with me over several months to combine those features into a recipe for a production namespace layout.

It is part of my IBM Fusion work.

Restoring databases on OpenShift? Let’s compare notes.

Connect on LinkedIn