Upgrading self-hosted Flagsmith on Kubernetes: a tested upgrade and rollback
I ran a real upgrade, a rollback and a database restore on a local cluster to answer the questions raised in Flagsmith issue #7429: where do migration logs live, and does helm rollback work cleanly?
After review feedback I revised the runbook and retested it on a fresh kind install with the same version pair. The rollback failure and the restore result were the same as described here. The findings below describe the original test. Section 8 now shows the revised commands, and the pull request has the full runbook.
- On a default install there is no migration Job. Migrations run in an init container named
migrate-dbinside the API pod. - Those logs disappear when the pod is replaced, so they have to be read straight after the upgrade.
- The 0.80.0 to 0.85.0 upgrade applied 91 migrations, one of which drops a table the old app needs.
helm rollbackrestores the old app but not the database, and the old app then breaks.- Restoring a
pg_dumptaken before the upgrade fixed it. The cost is that changes made after the backup are lost.
1. Why I ran this test
Issue #7429 points out that the docs have no upgrade runbook for self-hosted Flagsmith: no Helm upgrade procedure, no rollback story and no list of gotchas. Rather than write that page from assumptions, I wanted to run the upgrade myself and record what actually happens, so every statement in the runbook has evidence behind it.
2. Test setup
| Item | Value |
|---|---|
| Cluster | kind, single node, on a laptop |
| Starting version | Helm chart 0.80.0, Flagsmith app 2.222.0 |
| Target version | Helm chart 0.85.0, Flagsmith app 2.286.0 |
| Configuration | Default values, with the chart’s bundled PostgreSQL |
| Test data | One organisation, one project, two feature flags (one on, one off) |
3. What I did
- Installed chart 0.80.0 on a fresh kind cluster and created the test data.
- Took a
pg_dumpbackup of the database, checked it was readable, and copied it out of the cluster. - Upgraded to chart 0.85.0 and read the
migrate-dblogs immediately. - Created a third flag,
created_after_backup, to test what a restore costs. - Rolled back with
helm rollbackand recorded what broke. - Stopped the API, recreated the database, restored the backup, and started the API again.
- Checked the version, the flags and the database tables.
4. Findings
1There is no migration Job by default
The chart ships with jobs.migrateDb.enabled: false and enableMigrateDbInitContainer: true. The API pod lists two containers, flagsmith-api and migrate-db (init), and kubectl get jobs returned nothing. Migrations therefore run before the API starts, inside the pod.
What it means: guidance to read the migrateDb Job logs only applies when that Job is enabled. On a default install the command is kubectl logs <api-pod> -c migrate-db.
2Migration logs vanish when the pod is replaced
After a later rollout replaced the API pod, asking for the old pod’s migrate-db logs returned a NotFound error. The logs only exist as long as the pod does.
What it means: operators should capture the logs right after the upgrade, or ship them to a log system, because they are the only record of what the migrations did.
3The upgrade applied 91 migrations, including one that drops a table
The first upgrade log shows 91 migrations applied. One of them, code_references.0003_introduce_per_feature_scanned_references, runs DROP TABLE "code_references_featureflagcodereferencesscan" CASCADE, as shown by sqlmigrate. Running the upgrade again on a database that was already migrated applied nothing and finished normally.
What it means: an upgrade across this many releases changes the schema in ways the old version cannot read, and re-running migrations is safe in this case.
4helm rollback restores the app, not the database
Helm records a rollback as a new revision (revision 3, “Rollback to 1”) and the old pods start normally. But the 2.222.0 app then fails on the features list with relation "code_references_featureflagcodereferencesscan" does not exist, because the upgrade dropped that table. In the browser this appears as a message saying you do not have permission to view the project or environment, which is misleading, because the real cause is a failed database query.
What it means: for this version pair, rolling back safely needs more than Helm. The database has to go back too.
5Restoring a pre-upgrade backup fixes the rollback
I scaled the API to zero, dropped and recreated the database, restored the dump, and scaled the API back up. The app came back on 2.222.0 with the two original flags intact, and the table list showed only the original code_references_featureflagcodereferencesscan table. The flag created after the backup was gone.
created_after_backup.What it means: a restore works like a time machine. It returns the system to the moment of the backup and discards everything saved since. A runbook should say so plainly. I recreate the database rather than restoring over it, because tables created by the new version would otherwise be left behind and a later upgrade would try to create them again.
6The bundled InfluxDB disappears on upgrade
The 0.80.0 install warned that the in-chart InfluxDB is deprecated. In 0.85.0 it is removed: the InfluxDB pod vanished on upgrade and came back on rollback. The upgrade notes explain the replacement options, but they are printed only in the helm upgrade terminal output.
What it means: anyone upgrading through CI or a GitOps tool may never see that notice, so the runbook should repeat it. I did not test what happens to existing analytics data.
5. What went wrong along the way
- First install timed out at 10 minutes. The PostgreSQL image took over five minutes to download and the migration step restarted while waiting for the database. Pods were healthy shortly after Helm gave up. A longer
--timeoutavoids this on slow connections. - I looked for a migration Job that does not exist. That led to finding 1.
- I lost the first upgrade’s logs because I read them too late. I repeated the whole run on a fresh cluster to capture them, which led to finding 2.
- My downtime check used the wrong path.
/health/returned 404 on 2.286.0, while the pod’s own probes use/health/liveness/and/health/readiness/. I therefore have no downtime measurement.
6. What this means for the upgrade runbook
These results suggest the runbook should cover four things.
- Before upgrading. Read the release notes for the whole version gap, note the current Helm revision, take a database backup, check it is readable, and store it outside the cluster.
- During the upgrade. Use a generous
--timeout, then read themigrate-dbinit container logs immediately and confirm the version in the UI. - If you need to go back. Do not rely on
helm rollbackalone. Roll back, stop the API, recreate the database from the backup, then start the API. Say clearly that changes made since the backup are lost. - Where to look. Point operators to the init container logs on a default install and to the Job logs when
jobs.migrateDb.enabledis true.
7. Limits of this test
- The database held almost no data, so I cannot say anything about how real code reference data behaves during that migration.
- One version pair, one cluster type, default values only.
- The bundled PostgreSQL is a development convenience. Production setups use an external database, where the backup and restore commands will differ.
- I did not test the Job variant (
jobs.migrateDb.enabled: true), the analytics migration job, a restricted database role, or what happens to InfluxDB data. - I ran the restore twice, once in the original test and once after revising the commands. It worked the same way both times.
- I did not test autoscaling, the task processor, or database level settings such as connection limits and privileges.
8. Commands used (revised after review)
These are the commands from the retest. The original test used --no-owner on the restore and ran the upgrade without --reset-then-reuse-values. Each command is a single line, so it can be pasted into PowerShell as is.
Before upgrading, note the Helm revision and the API replica count:
helm history -n flagsmith flagsmith kubectl -n flagsmith get deployment flagsmith-api
Back up (bundled PostgreSQL, user postgres), then copy the backup out of the cluster:
kubectl -n flagsmith exec flagsmith-dev-postgresql-0 -- df -h kubectl -n flagsmith exec flagsmith-dev-postgresql-0 -- sh -c 'PGPASSWORD=$POSTGRES_PASSWORD pg_dump -U postgres -Fc -f /tmp/backup.dump $POSTGRES_DB' kubectl cp flagsmith/flagsmith-dev-postgresql-0:/tmp/backup.dump ./backup.dump
Verify the backup by restoring it into a scratch database, compare table counts, then remove the scratch database:
kubectl -n flagsmith exec flagsmith-dev-postgresql-0 -- sh -c 'PGPASSWORD=$POSTGRES_PASSWORD createdb -U postgres flagsmith_verify && PGPASSWORD=$POSTGRES_PASSWORD pg_restore -U postgres -d flagsmith_verify --single-transaction /tmp/backup.dump' kubectl -n flagsmith exec flagsmith-dev-postgresql-0 -- sh -c 'PGPASSWORD=$POSTGRES_PASSWORD psql -U postgres -d flagsmith_verify -c \\dt | grep -c table' kubectl -n flagsmith exec flagsmith-dev-postgresql-0 -- sh -c 'PGPASSWORD=$POSTGRES_PASSWORD psql -U postgres -d $POSTGRES_DB -c \\dt | grep -c table' kubectl -n flagsmith exec flagsmith-dev-postgresql-0 -- sh -c 'PGPASSWORD=$POSTGRES_PASSWORD dropdb -U postgres flagsmith_verify'
Run the last command only if the scratch database was created by the first one. In the test, both table counts were 145.
Upgrade, then read the migration logs straight away:
helm upgrade -n flagsmith flagsmith flagsmith/flagsmith --version 0.85.0 --reset-then-reuse-values --wait --timeout 20m kubectl -n flagsmith logs <api-pod> -c migrate-db
Roll back, then restore the backup:
helm rollback flagsmith 1 -n flagsmith --wait --timeout 20m kubectl -n flagsmith scale deployment flagsmith-api --replicas=0 kubectl cp ./backup.dump flagsmith/flagsmith-dev-postgresql-0:/tmp/backup.dump kubectl -n flagsmith exec flagsmith-dev-postgresql-0 -- sh -c 'PGPASSWORD=$POSTGRES_PASSWORD dropdb -U postgres $POSTGRES_DB' kubectl -n flagsmith exec flagsmith-dev-postgresql-0 -- sh -c 'PGPASSWORD=$POSTGRES_PASSWORD createdb -U postgres $POSTGRES_DB' kubectl -n flagsmith exec flagsmith-dev-postgresql-0 -- sh -c 'PGPASSWORD=$POSTGRES_PASSWORD pg_restore -U postgres -d $POSTGRES_DB --single-transaction /tmp/backup.dump' kubectl -n flagsmith scale deployment flagsmith-api --replicas=1
Scale back to the replica count you noted before the upgrade. It was 1 in this test.
9. Next step
I would like to turn this into the “Upgrading self-hosted Flagsmith” page requested in issue #7429, with the logs and commands above as its evidence, and submit it as a pull request. I am happy to take a section, test someone else’s draft, or rerun any part of this on a different version pair or database setup.