Repository navigation
Keep live data when a backup restore fails - #444
Merged
Merged
Conversation
Restoring a backup with more than about 10,000 chunks failed while pg_restore rebuilt the IVFFlat index: 1536-dimension vectors need 65 MB of maintenance_work_mem and PostgreSQL defaults to 64 MB. verify ran with 256 MB, so it passed backups that restore then rejected. Because restore dropped the live database before pg_restore, the failure left the deployment empty. Restore and verify now both use 512 MB (MORPHIK_RESTORE_MAINTENANCE_WORK_MEM overrides it). Restore loads into a morphik_restore database, checks it against the manifest, and only then renames it over the live database in one transaction, so a failed pg_restore leaves the current data as it was. The Docker test now seeds 10,500 vectors at 1536 dimensions and checks that a pg_restore failure leaves the live database and files unchanged.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Restoring a backup with more than about 10,000 chunks failed while pg_restore rebuilt the IVFFlat index (1536-dimension vectors need 65 MB of
maintenance_work_mem, PostgreSQL defaults to 64 MB), and because restore had already dropped the live database, the deployment came back empty. A customer hit this on a 359k-chunk install.Restore and verify now both run with 512 MB (
MORPHIK_RESTORE_MAINTENANCE_WORK_MEMoverrides it), so verify can no longer pass a backup that restore rejects.Restore loads into a
morphik_restoredatabase, checks it against the manifest, and only then renames it over the live database in one transaction, so a failed pg_restore leaves the current data as it was.scripts/test_backup_restore.shnow seeds 10,500 vectors at 1536 dimensions and checks the failure path; it fails onmainwith the customer's exact error and passes on this branch (run withMORPHIK_BACKUP_TEST_S3=0).🤖 Generated with Claude Code
Need help on this PR? Tag
@codesmith-botwith what you need. Autofix is disabled.