Failover and recovery

Install on Fly.io sets up an installation that keeps answering through the failures that happen most often, with nothing to do at the time: two Machines, a database that fails over by itself, and deploys that switch to new Machines only once they are healthy. Bouée keeps nothing that matters in a Machine's memory. Sessions, the outgoing mail queue and attachments live in the database and the bucket, so either Machine can answer any request, and losing one loses nothing.

What it survives

Background jobs (retries, expiry, auto-close, backups) run on every Machine and are safe to: each claims its work in the database, or does work that is the same done twice. bouee.yaml is applied under a database lock, one Machine at a time.

Deploys that stay up

Two versions run side by side for a minute during every deploy: the new one against the migrated database, and the old one still serving until the new one is healthy. So a release never removes what the release before it uses. It adds what it needs, and a later release removes what nothing uses any more. Bouée's own tests hold every migration to that. Keep it in mind for a fork of your own.

A release that fails its health check never takes traffic. To go back after one that started but misbehaves, deploy the version before it: git checkout <previous tag>, then fly deploy.

A whole region

Managed Postgres keeps its standby in its own region, so losing the region takes the database with it. What brings it back is Bouée's own database backups: encrypted, every hour, into the Tigris bucket, which keeps them in several regions. bin/bouee backups lists them. Recovering takes about half an hour, and loses at most the work since the last backup; DATABASE_BACKUP_EVERY_MINUTES=15 narrows that to 15 minutes.

1. Create a database in a region that is up:

   fly mpg create --name your-bouee-db-2 --region ewr --plan Basic --pg-major-version 17

2. Restore the newest backup into it, from a temporary Machine in that region. Paste the new cluster's pooled connection string, from its Connection tab, when asked:

   printf 'New DATABASE_URL: '; read -rs NEW_DATABASE_URL; echo
   fly console --region ewr -e DATABASE_URL="$NEW_DATABASE_URL" -C "/app/bin/restore --latest"

   It needs the installation's SECRET_ENCRYPTION_KEY, which the Machine has from the app's secrets: the backups are encrypted with it.

3. Point the app at the new database, and start its Machines there:

   fly secrets set --stage DATABASE_URL="$NEW_DATABASE_URL"
   fly scale count 2 --region ewr
   fly scale count 0 --region yyz

   Set primary_region to the new region in fly.toml for the next fly deploy.

4. Check it: bin/bouee status. DNS needs no change, since the app's addresses are the same in every region, and the bucket and the secrets belong to the app, not a region.

Rehearse it once while nothing is wrong: restore into a new cluster with fly console, check it with fly mpg connect, then destroy it. /app/bin/restore --list shows every backup, and /app/bin/restore <key> restores an older one.

Watching it

• An uptime check on https://support.example.com/api/health, which answers 200 only while the database does.
• bin/bouee status --json from a scheduled job: it exits 1 when mail stops going out or coming in.
• fly logs, and the Machines' metrics on Fly's dashboard.