AL+

RO EN

Email us

Cloud redundancy is not a backup: keep a copy outside your provider

In May 2024 Google Cloud deleted UniSuper's private cloud, replicas included. Keep one tested copy of your data where your main cloud provider cannot reach it.

. By . Architecture and infrastructure

Citiți articolul în română

Redundancy across zones and regions protects you from a failed data centre. It does not protect you from the provider deleting what you run there, and it does not protect you from your own mistakes. For those, a business needs at least one restorable copy of its data in an account the primary provider cannot touch, and a restore it has actually tested. The UniSuper incident of May 2024 is still the clearest illustration of the difference.

What happened

UniSuper is an Australian pension fund. In early May 2024, more than half a million of its members went over a week without access to their accounts (The Guardian, 9 May 2024). On 8 May, UniSuper's CEO and Google Cloud's CEO, Thomas Kurian, issued a joint statement: an "inadvertent misconfiguration during provisioning" had "ultimately resulted in the deletion of UniSuper's Private Cloud subscription".

The sentence that matters for everyone else came next. UniSuper "had duplication in two geographies as a protection against outages and loss", but the deletion "caused deletion across both of these geographies". What saved the fund was that it "had backups in place with an additional service provider", which "minimised data loss, and significantly improved the ability of UniSuper and Google Cloud to complete the restoration." The Register summed it up the same day: without the backups at another provider, "a bad situation could have been oh so much worse."

Two weeks later Google published its own review (dated 24 May 2024 on Google’s page; UniSuper’s preface dates the additional information to 25 May). When Google operators deployed one of UniSuper's Google Cloud VMware Engine private clouds in early 2023 with an internal tool, they left one input parameter blank. The system filled it with a then-unknown default: a fixed one-year term, followed by automatic deletion. A year later the private cloud was deleted, with no notice to the customer, because the deletion did not come from a customer request.

What the later review changed

Google's account is narrower than the headlines that called it an account deletion. It describes one customer, one region, and one of the customer's several private clouds "across two zones". It says the customer's Google account, projects and other private clouds were untouched, and that "data backups stored in Google Cloud Storage (GCS) in the same region were not impacted by the deletion" and, together with third-party backup software, "were instrumental in aiding the rapid restoration."

So the story is less dramatic than "Google deleted a pension fund" (note the gap between "two geographies" in the joint statement and "two zones" in Google's review), and more instructive. The deletion followed the boundary of a single subscription. Everything inside that boundary went, both copies included, because replication copies deletions as faithfully as it copies data. What survived was whatever sat outside it: storage in a different service, and backups held by another provider.

The customer did not choose that boundary. A blank field in an internal tool did. You cannot know in advance where the boundary of the next incident will fall, which is the argument for putting at least one copy where no single provider-side event can reach it.

Three failures, three protections

It helps to separate the ways data actually disappears, because each needs a different protection.

A zone or region fails. Power, cooling or network takes a location down. Replication across zones or regions handles this well; it is what cloud redundancy is designed for.

The provider removes your resources. A deletion like UniSuper's, a suspended account, a billing problem. In the Hacker News discussion of the incident, one commenter described a company whose Google Cloud projects stopped within minutes after HR deleted the account of a departed employee to which the payment profile was attached. That is an anecdote, not a statistic, but it shows how ordinary the trigger can be. Replication inside the provider does nothing here. The only protection is a copy in a different account with separate billing and credentials, ideally at a different provider.

You make the mistake. A migration drops a table, a script empties a storage bucket, someone with administrator rights is phished. Replication copies the damage within seconds. The protection is versioned, point-in-time copies kept for weeks, with credentials that the production system does not hold, so that whatever deletes production cannot also delete the backups.

For most small businesses the third failure is, in our view, far more likely than the second. The good news is that the protection for the second also covers most of the third, if it is set up properly.

A recovery plan a small business can test

None of this requires UniSuper's budget. For a typical web application, agree the maximum acceptable data loss (RPO) and time to restore service (RTO) first. The following is an example plan, not a universal minimum:

  1. Know what cannot be regenerated. Usually the database and the files users have uploaded. Code lives in a repository, servers can be rebuilt. Write down the configuration and secrets needed to run the system, and keep them in a password manager the company controls.
  2. Create consistent recovery points outside the primary provider. Coordinate database backups and uploaded files so that restored records refer to files that exist at the same recovery point. Use database-supported backup methods and a compatible file snapshot or retained file versions; two unrelated copies are not enough. A daily export is an example only: even successful daily jobs permit roughly a day of data loss, and failed jobs or export lag can increase it. Choose frequency from the agreed RPO. Keep the copy in a business-owned account with separate credentials and billing.
  3. Separate retention from versioning. See the S3 Object Lock documentation for one implementation. Versioning helps recover older objects but does not itself stop an authorised or compromised identity deleting versions. Use retention or object-lock controls appropriate to the threat, with deletion authority separately controlled and unavailable to production. Check which administrators can bypass retention; do not call a copy immutable merely because versioning is enabled.
  4. Test against the agreed objectives. For example, run a quarterly restore and repeat after material architecture changes. Restore database and files together into a clean environment, start the application and check business operations. Record the recovery point and elapsed time until usable service returns, including credentials, infrastructure and verification. A schedule does not establish RTO; measured restoration does.

If a second provider is too much for now, a separate account at the same provider is a reasonable first step. AWS, for example, documents copying backups to another account in the same organization so that a backup deleted in one account can be recovered from the other. That covers your own mistakes and a compromised account. It does not cover the provider acting on your organization as a whole.

Who owns the restore test

The engineer, contractor or agency who runs the system should perform the restore test. The business should own the outcome. That means the backup account is registered to the company rather than to a developer's personal email, the restore report goes to someone who can decide, and the test is on a calendar rather than in someone's good intentions. A backup that nobody has restored is a hope. This is the same point as in software needs an owner: the job is not done because a tool is configured, but because someone is answerable for the result.

If you pay for software maintenance, ask to see the last restore report. If there isn't one, that is the first thing to fix.

The strongest objection

Google's own review is the best counterargument. It says this was the first incident of its kind, that it was not systemic, and that UniSuper's same-region backups in Google Cloud Storage survived and were used in the recovery. On that reading, a well-built single-provider setup would have been enough, and a second provider adds cost, a second set of credentials and encryption keys to protect, data transfer charges, and one more system to monitor.

Part of that is right. The risk is small, and a second provider set up carelessly, with keys stored next to production or a restore nobody has tried, adds work without adding safety. But the objection rests on hindsight about where the deletion stopped. The point of an independent copy is that it does not depend on the next incident having the same scope as the last one. And for a small database, an encrypted daily export to another provider is likely to cost much less than a single week without access to your data.

Where this does not apply

The advice fits less well in three cases.

  • Software-as-a-service tools such as an accounting package or an online shop builder, where the data lives in the vendor's system. There, the protection is a regular export in a format you can read without the vendor, and knowing what the vendor's contract says about deletion.
  • Data you can rebuild, such as caches, search indexes or reports generated from other data. Backing them up is optional.
  • Data with residency or contractual limits. If personal or client data has to stay in a particular jurisdiction or with approved processors, the second provider has to meet the same conditions. That is a reason to choose the second provider carefully, not a reason to skip it.

Two and a half years on

The UniSuper incident is mostly remembered as Google's mistake, and Google did change the tool and the default that caused it. The lesson that lasts is about UniSuper's decision, made long before May 2024, to keep backups somewhere else. Redundancy keeps you running when a location fails. A tested copy outside your provider is what lets you recover when the provider, a colleague or a script removes the thing that was redundant.

Sources