Mastering Disaster Recovery Testing for Modern Apps

At 3 AM, nobody cares that your status page says “degraded service” instead of “outage”. Users can't sign in, writes are failing, your mobile app is timing out, and the one person who remembers how the backup job works is asleep or offline.

If you run on Supabase, Firebase, or a similar BaaS or PaaS stack, the failure mode is rarely just “the database is down”. It's usually a messier chain: auth tokens are still being issued but profile reads fail because a policy changed, storage is up but signed URL generation is broken, an Edge Function depends on a secret that only exists in the primary project, or your backup restores data but not the security model wrapped around it.

That's why disaster recovery testing has to be treated as an engineering discipline, not an annual checkbox. Backups matter. Restore proof matters more.

Your App Is Down What Happens Next

The first useful question in an outage isn't “do we have backups?”. It's “how fast can we restore service, and how much data can we afford to lose?”. Those are your Recovery Time Objective (RTO) and Recovery Point Objective (RPO).

For a small SaaS app, RTO is the maximum acceptable downtime. If your team says users must be able to log in again within the agreed recovery window, that's the target your recovery plan has to meet. RPO is the acceptable gap between the last recoverable state and the moment things broke. If your last good copy is too old, recovery can still be technically successful and operationally unacceptable.

Backups aren't the same as recovery

Modern app teams often stop at backup configuration. They turn on snapshots, export Firestore collections, or rely on managed platform features and assume that's enough. It isn't.

A restore only counts if you can bring back a usable service. On BaaS platforms that means more than rows and documents. You also need to recover things like:

• Authentication context such as provider settings, redirect flows, and service credentials
• Authorisation logic including RLS policies, claims mapping, and security rules
• Runtime dependencies such as secrets, webhooks, scheduled jobs, and function config
• Application expectations like bucket names, RPC behaviour, and environment variables baked into clients

Practical rule: If a restored environment can't safely serve production traffic, you haven't proved recovery. You've only proved data extraction.

A lot of teams still test too rarely. A UK benchmark cited by Flexential shows that only 37% of organisations test disaster recovery once a year, while 21% test more than once a year. The same source warns that a low testing cadence raises the risk that the plan looks complete on paper but fails in a real outage (Flexential's disaster recovery testing overview).

Panic is a process problem

When recovery goes badly, the root issue usually isn't effort. It's ambiguity. Nobody knows which system is authoritative, which credentials still work, whether the backup includes schema changes, or who is allowed to make the call to restore into a fresh environment.

That's where incident discipline overlaps with recovery discipline. If your team needs a tighter operational structure for escalations, comms, and ownership, this practical guide to incident management is worth reviewing before your next DR exercise.

The useful mindset shift is simple: stop asking whether you have a plan. Ask whether you can prove the plan works under pressure.

Scoping Your Test and Setting Objectives

Most failed disaster recovery tests are scoped badly before they're executed badly. Teams say they're testing “the app”, but what they really mean is one database restore in isolation.

For Supabase, Firebase, and similar platforms, scope has to follow the actual request path. A user action touches more than storage. It may involve auth, policy evaluation, a function call, an external API, object storage, and client-side assumptions about config values. If any one of those is missing, your test result is misleading.

Map the stack you actually run

Start with user-facing capabilities, not infrastructure labels. “Checkout works”, “staff can sign in”, and “uploads can be retrieved” are better starting points than “Postgres”, “Functions”, and “Storage”.

Then map the components behind each capability:

• Data layer: primary database, backups, replicas, document stores, file buckets
• Access layer: auth provider, JWT settings, claims, roles, service accounts
• Security layer: RLS policies, Firebase Security Rules, RPC permissions, API key exposure
• Execution layer: Edge Functions, Cloud Functions, background jobs, cron tasks
• Integration layer: payment gateways, email services, analytics sinks, webhook consumers

This is also where the shared responsibility model becomes painfully real. Your platform provider may cover underlying infrastructure resilience. You still own your project configuration, your data model, your policies, your secrets, and the way your application behaves after restore.

A lot of startup teams blur backup scope with service scope. They're not the same thing.

Set objectives that reflect business pain

RTO and RPO need to be set per capability, not as one vague target for the whole product. Your admin dashboard doesn't need the same recovery objective as your customer login flow. A file preview service may tolerate delay. Payment write paths usually can't.

Databarracks gives a clean way to think about RPO: a single daily backup creates a maximum RPO of 24 hours, while backing up or replicating every 4 hours reduces the maximum RPO to 4 hours. It also defines RTO as the elapsed time from recovery start to service restoration (Databarracks on DR metrics).

That matters because many BaaS teams discover their “we have backups” story implies a data loss window they'd never accept if they said it out loud.

Include non-technical failure in scope

A narrow DR test asks whether systems recover. A realistic one asks whether the team can recover them under constraints.

Guidance on resilience and equity argues that recovery plans are often tested against classic technical outages while ignoring climate-driven, multi-hazard, and resource-constrained scenarios. It also notes that a DR plan can pass a technical failover test yet still fail operationally if staff or suppliers are disrupted at the same time (resilience planning guidance from Local Housing Solutions).

That lands directly in app teams. Test at least one awkward scenario:

• A key engineer is unavailable
• Your password manager is reachable but one secret isn't
• A vendor dependency is degraded during restore
• Your primary comms channel is down
• You can restore data, but not to the same project or region

Recovery plans fail in the hand-offs. The restore command is usually the easy bit.

For a practical planning companion, this guide on protecting your business with IT recovery is useful for pressure-testing assumptions around continuity. It pairs well with a more formal business continuity planning checklist if you need to connect technical recovery to business operations.

Choosing the Right Disaster Recovery Test

Not every team needs a full failover exercise straight away. In fact, jumping there too early usually wastes time and creates noise. The better approach is to choose the test type that exposes the next most likely weakness.

A solo founder with one production database and a mobile app needs a different test from a team running regional replicas, background queues, and several third-party integrations. The trade-off is always the same: more realism gives you better evidence, but it also costs more effort and carries more risk.

Four useful test types

| Test Type | Realism | Cost & Effort | Disruption Risk | Best For | |---|---|---|---|---| | Tabletop exercise | Low | Low | Low | Early-stage teams, role clarity, dependency discovery | | Simulation | Medium | Medium | Low | Rehearsing alerts, comms, and restore decisions without touching production | | Partial failover | High for one component | Medium to high | Medium | Validating one risky layer such as database, auth, or storage | | Full restore to parallel environment | High end-to-end | High | Low to medium if isolated well | Proving recoverability of the whole stack without routing live traffic immediately |

When each option works

Tabletop exercises are underrated. Walk the team through a realistic outage and force explicit answers to simple questions: who declares the incident, who approves restore, where are the credentials, what happens if auth comes back before data, and who verifies success? These exercises surface ownership gaps fast.

Simulations are the next step when you want pressure without production impact. Trigger alerts, use the actual runbook, and make people fetch the actual artefacts, but don't change live traffic. These exercises expose weak documentation.

If the team can't execute the runbook calmly in a simulation, they won't do it cleanly in production.

Partial failovers are where BaaS-specific issues start to show. Restore only the database into a parallel project. Recreate storage buckets. Reapply policies from code. Point a staging app at the recovered stack and see what breaks. This gives you strong signal without gambling the whole service.

Full restores are the closest thing to proof. Not because they're dramatic, but because they force you to validate sequence. Data first, then auth alignment, then policies, then functions, then smoke tests, then selective traffic. Hidden dependencies are often discovered here, especially around secrets and webhooks.

What to choose for a modern app

If you're on Supabase or Firebase, a useful progression looks like this:

1. Tabletop the incident path
2. Run a simulation using the actual runbook
3. Do a component restore for your most critical stateful service
4. Perform a full restore into an isolated environment
5. Only then consider production failover patterns

What doesn't work is treating disaster recovery testing as a single annual event. Recovery maturity comes from layering tests, not from one heroic exercise.

For most startup teams, the best first serious test isn't a failover. It's a clean rebuild of the app's critical path in a recovery environment using backups, infrastructure code, secrets management, and policy-as-code. That's where fantasy ends and evidence begins.

Building Your Modern Pre-Test Checklist and Runbook

Good recovery tests are won before the test starts. If the team discovers missing secrets, outdated IAM permissions, or undocumented manual steps during the exercise, the test still has value, but you've turned validation into archaeology.

For BaaS stacks, the pre-test checklist must cover the layers traditional DR lists often miss: RLS, service-role usage, client config drift, and code paths that depend on managed platform metadata.

Pre-test checklist for Supabase and Firebase teams

Use this before any simulation, partial restore, or full restore exercise.

• Verify recovery access: The people running the test need working access to cloud accounts, project consoles, secret stores, CI systems, and any break-glass credentials. Don't assume SSO will be available in every scenario.

• Confirm backups are restorable: The backup job being green isn't enough. You need to know what artefacts exist, where they are, what they contain, and whether the format matches your current schema and tooling.

• Store policies as code: If your RLS policies or Firebase Security Rules only exist in the live platform, recovery becomes guesswork. Version them and make reapplication repeatable.

• Export schema and function definitions: Tables are only part of the service. Capture migrations, triggers, stored procedures, RPCs, scheduled jobs, and extension dependencies.

• Check secrets separation: Recovery keys, service-role credentials, API tokens, and webhook signing secrets can't live only inside the environment that might be down. Put them in an external secret manager with documented retrieval steps.

• Map external dependencies: Payment providers, email senders, auth redirects, object storage consumers, and mobile app config all affect whether a restored system is usable.

• Define go or no-go criteria: Agree in advance what counts as success, what pauses the exercise, and who can stop it if data integrity or safety is in doubt.

What a runbook should include

A DR runbook shouldn't read like a pile of shell history. It needs enough structure that another engineer can execute it under pressure without interpretation.

Include these parts:

| Runbook Part | What it needs to contain | |---|---| | Incident trigger | What event starts the DR process and who declares it | | Scope | Which services, data stores, and integrations are included | | Roles | Incident lead, recovery operator, comms owner, verifier | | Preconditions | Required access, artefacts, approvals, and environment readiness | | Recovery steps | Ordered actions with exact references to scripts, jobs, or console tasks | | Verification | Smoke tests, data checks, auth checks, policy checks, and rollback conditions | | Communications | Internal updates, customer status messaging, and vendor escalation paths | | Evidence | Timestamps, screenshots, logs, and remediation notes |

Here's the bit many teams skip: verification must be explicit. “Database restored” is not a verification step. “Test user can sign in, read permitted records, cannot read forbidden records, upload file, and trigger one critical function successfully” is.

Runbook example: Restore database into isolated project, apply latest schema migrations that are not part of backup artefact, reapply RLS policies from repository, inject recovery environment secrets from external store, run smoke suite for auth, read, write, storage, and one RPC, then compare expected versus actual access boundaries before any traffic cutover.

BaaS-specific traps worth documenting

These are common enough that they deserve a named step in the runbook rather than a footnote:

• JWT and auth mismatch: restored data may reference roles or claims that the recovered auth configuration doesn't issue
• Service-role overreach: internal scripts may pass in test because they bypass policy checks, while user flows still fail
• Storage policy drift: buckets restore, but object access rules don't
• Environment leakage: mobile or web clients still point at the old project
• Function config gaps: scheduled jobs and webhooks come back partially because secrets weren't restored

If you build for regulated workloads, your runbook also needs a compliance lens. Teams working around healthcare data often use operational controls that are broader than the restore itself, and AppStarter's definitive HIPAA guide is a useful reference for thinking about security obligations around application behaviour, not just infrastructure.

For the incident command side of the document, a reusable incident response plan template helps keep recovery notes from collapsing into ad hoc chat messages and half-finished tasks.

Automating Recovery Tests in Your CI/CD Pipeline

Manual disaster recovery testing is a solid start. It's not enough for a stack that changes every week.

Modern apps ship new migrations, new functions, new secrets, and new client behaviours constantly. If your recovery proof depends on a calendar reminder and a heroic engineer, it will drift out of date. The fix is to automate the boring parts and reserve human attention for judgement calls.

What to automate first

The best early automation pattern is simple:

1. Provision a temporary recovery environment
2. Restore from the latest approved backup artefact
3. Apply infrastructure and policy code
4. Inject secrets from the external secret manager
5. Run smoke tests against critical paths
6. Destroy the environment
7. Save evidence and failures

This works well with Terraform or Pulumi for environment definition, CI runners for orchestration, and API smoke tests for validation. If your team already keeps application checks in Postman or similar tooling, connect those directly into the workflow. A guide on testing APIs with Postman is a practical way to structure those smoke checks around sign-in, reads, writes, and permission boundaries.

Conceptual GitHub Actions flow

A lightweight workflow might look like this in practice:

• On schedule or after major schema change
• Create isolated project or namespace
• Restore backup
• Run migrations and policy sync
• Seed minimal verification users
• Execute smoke suite
• Upload logs and test report
• Notify team on failure
• Tear down environment

The point isn't to automate every form of DR exercise. It's to make restore validation routine enough that surprises get caught close to the change that introduced them.

Automated restore tests won't replace a full exercise. They will catch the drift that makes full exercises fail.

For BaaS teams, this is especially valuable because shared platforms encourage hidden manual setup. CI-based recovery tests force you to codify project config, policy state, and secret dependencies. That alone improves resilience, even before you have a polished full-scale DR programme.

Measuring Success and Learning From Failure

The test ends. The restore completed. The app loads. Then a support user signs in and sees data they should never see because an RLS policy was missed during recovery. That is a failed DR test, even if the database came back on time.

A useful review starts with the targets already set earlier and compares them with what happened in this run. Record the observed recovery time, the observed data gap, and the exact minute each user-critical path became usable. For BaaS and PaaS stacks, that usually means separating infrastructure recovery from application recovery. A Supabase project can be "up" before storage buckets, edge functions, JWT settings, or RLS policy state match production. A Firebase project can restore data while auth configuration, service accounts, or security rules still block live users or expose too much.

Measure the recovery users actually get

Start with the evidence from the run:

• Actual recovery time against the target
• Actual data loss against the target
• Time to restore auth, secrets, and external integrations
• Time until the first successful user read and write
• Time until admin paths and background jobs worked
• Policy parity after restore, including RLS and security rules
• Manual steps, approvals, and one-person dependencies

That last pair matters more than teams expect. In modern managed stacks, the hard part is often not the database restore itself. It is the hidden config around it. Missing environment secrets, stale OAuth callbacks, unsynced storage policies, or a service role key stored in one engineer's password manager will stretch a test long after the platform reports healthy status.

Security parity deserves its own check. If the restored system accepts traffic with the wrong RLS rules, incorrect Firebase Security Rules, or drifted API keys, recovery is incomplete. The service is running, but the platform is no longer behaving like production. I have seen restores pass smoke tests and still fail in practice because background jobs used old secrets or because policy migrations were applied in the wrong order.

Run the review like an incident postmortem

Skip broad statements like "restore succeeded with minor issues." Build a timeline and force specificity. Who did what, in what order, with which credentials, and where did the runbook stop matching reality?

Good review prompts include:

• What step consumed the most time?
• Which dependency was undocumented?
• Where did the team need privileged access that was not ready?
• Which checks passed even though the environment was still unsafe?
• What failed only because this was a Supabase or Firebase style stack, not a single database restore?
• What would break if the same test ran during leave, after hours, or during a provider-side incident?

The output should become engineering work, not a PDF nobody reads. Update the runbook. Codify the missing config in Terraform, Pulumi, or deployment scripts. Move secrets into managed storage with audited access. Add smoke tests for RLS, storage access, scheduled functions, and privilege boundaries. If a human had to click through console settings to finish the restore, treat that as a defect until proven otherwise.

A failed test with clear fixes is productive. A "successful" test that ignores drift is how teams end up discovering, during a real outage, that backups were the easy part and platform state was the recovery problem.

If you run on Supabase, Firebase, or a mobile backend with complex auth and policy logic, AuditYour.App helps you find the misconfigurations that make recovery harder and riskier. It scans for exposed RLS rules, unprotected RPCs, leaked secrets, and backend weaknesses that often stay hidden until an outage or restore test forces them into view.