Skip to content

repoobj: add OBJ_MAGIC blob header for pack file format#9684

Open
mr-raj12 wants to merge 1 commit into
borgbackup:masterfrom
mr-raj12:pack-files-step2-obj-magic
Open

repoobj: add OBJ_MAGIC blob header for pack file format#9684
mr-raj12 wants to merge 1 commit into
borgbackup:masterfrom
mr-raj12:pack-files-step2-obj-magic

Conversation

@mr-raj12
Copy link
Copy Markdown
Contributor

Description

RepoObj blob header grows from 8 bytes (<II) to 17 bytes (<8sBII): 8-byte magic (BORG_OBJ), 1-byte version, then the existing meta/data sizes.

The magic is a re-sync point. If a pack is corrupted, borg scans forward for the next BORG_OBJ to find the next valid blob. The version byte lets borg reject unknown formats rather than misreading them.

This is the N=1 pack format (one chunk per pack file). No chunk id in the header yet -- for N=1 the pack filename is the chunk id (SHA-256 hex), so it is implicit. Chunk id in the header comes later when chunk id != pack id.

Changes:

  • repoobj.py: add OBJ_MAGIC/OBJ_VERSION constants, extend struct to <8sBII, validate magic and version in format(), parse(), parse_meta(), extract_crypted_data().
  • repository.py: import OBJ_MAGIC/OBJ_VERSION, update check_object() to check magic/version before sizes, fix get() partial-read path for 17-byte header.
  • repository_test.py, legacyrepository_test.py: import OBJ_MAGIC/OBJ_VERSION, update fchunk() to include magic and version, fix pchunk() slice to [2:4], update test_read_data().

refs #8572

Checklist

  • PR is against master (or maintenance branch if only applicable there)
  • New code has tests and docs where appropriate
  • Tests pass (run tox or the relevant test subset)
  • Commit messages are clean and reference related issues

@codecov
Copy link
Copy Markdown

codecov Bot commented May 30, 2026

Codecov Report

❌ Patch coverage is 31.03448% with 20 lines in your changes missing coverage. Please review.
✅ Project coverage is 83.78%. Comparing base (7f43779) to head (66a054b).
⚠️ Report is 5 commits behind head on master.
✅ All tests successful. No failed tests found.

Files with missing lines Patch % Lines
src/borg/repoobj.py 29.41% 6 Missing and 6 partials ⚠️
src/borg/repository.py 33.33% 4 Missing and 4 partials ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##           master    #9684      +/-   ##
==========================================
- Coverage   83.87%   83.78%   -0.10%     
==========================================
  Files          93       93              
  Lines       15586    15604      +18     
  Branches     2335     2343       +8     
==========================================
+ Hits        13073    13074       +1     
- Misses       1785     1794       +9     
- Partials      728      736       +8     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

Copy link
Copy Markdown
Member

@ThomasWaldmann ThomasWaldmann left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good.

Next after this, I guess you need to decouple the legacyrepository_test yourself (can't do that if you keep changing it although that test should only test legacy format repos).

The legacy repo (borg 1.x) shall use RepoObj1 format, not RepoObj. It can still use fchunk ("format a chunk") and p(d?)chunk ("parse a chunk"), but needs to be adapted so it works with the old format.

@mr-raj12
Copy link
Copy Markdown
Contributor Author

Thanks! Understood on legacyrepository_test, I'll fix fchunk/pchunk there to use RepoObj1.obj_header in the next PR.
Can this one be merged as-is, or should the fix come first?

@ThomasWaldmann
Copy link
Copy Markdown
Member

If you decouple the legacy tests first, you can remove the changes to the legacy test file from this one.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants