V2 serialization format for wide columns with blob references#14314
V2 serialization format for wide columns with blob references#14314xingbowang wants to merge 9 commits intofacebook:mainfrom
Conversation
xingbowang
force-pushed
the
2026_02_06_blob_wide_column
branch
from
aebf39b to
09fdfab
Compare
xingbowang
force-pushed
the
2026_02_06_blob_wide_column
branch
5 times, most recently
from
003bfee to
578db34
Compare
Introduce a new V2 serialization format for wide column entities that supports storing individual column values in blob files. The V2 format adds a column type section that marks each column as either inline or blob-index, enabling per-column blob storage for large values. Key additions: - SerializeWithBlobIndices(): serialize entities with blob column references - DeserializeColumns(): deserialize V2 entities extracting blob column metadata - HasBlobColumns(): lightweight check for V2 blob references - GetVersion(): return the format version of a serialized entity - ResolveEntityBlobColumns(): resolve all blob references in an entity - Updated Deserialize() and GetValueOfDefaultColumn() for V2 compatibility
xingbowang
force-pushed
the
2026_02_06_blob_wide_column
branch
from
578db34 to
154c72a
Compare
|
@xingbowang has imported this pull request. If you are a Meta employee, you can view this in D92832066. |
pdillinger
left a comment
pdillinger
left a comment
There was a problem hiding this comment.
My primary concern about this approach is that the natural evolution of the data model is to have columns function as "subkeys" and pick up more and more features per-column that you get with keys and no wide columns, like merge, single column overwrite, single column delete, TimedPut, and likely more (kTypeValuePreferredSeqno). And we eventually end up re-implementing the cross product of all those features rather than making the data model leverage the features (almost) without re-implementation, by having a wide column ValueType entry itself contain ValueType entries for each column.
|
Updated the code to use ValueType in wide column type. |
|
@xingbowang has imported this pull request. If you are a Meta employee, you can view this in D92832066. |
pdillinger
left a comment
pdillinger
left a comment
There was a problem hiding this comment.
Thanks for considering my feedback. LGTM overall
Summary: Test Plan: Reviewers: Subscribers: Tasks: Tags:
✅ clang-tidy: No findings on changed linesCompleted in 28.0s. |
…y current PR" This reverts commit 80e59f1.
|
@xingbowang has imported this pull request. If you are a Meta employee, you can view this in D92832066. |
|
@xingbowang has imported this pull request. If you are a Meta employee, you can view this in D92832066. |
|
@xingbowang has imported this pull request. If you are a Meta employee, you can view this in D92832066. |
|
@xingbowang merged this pull request in 4e11dd7. |
3 participants
Introduce a new V2 serialization format for wide column entities that supports storing individual column values in blob files. The V2 format adds a column type section that marks each column as either inline or blob-index, enabling per-column blob storage for large values.