Format Versioning¶
Feature Flags¶
As the table format evolves, new feature flags are added to the format.
There are two separate fields for checking for feature flags,
depending on whether you are trying to read or write the table.
Readers should check the reader_feature_flags to see if there are any flag it is not aware of.
Writers should check writer_feature_flags. If either sees a flag they don't know,
they should return an "unsupported" error on any read or write operation.
Current Feature Flags¶
| Flag Bit | Flag Name | Reader Required | Writer Required | Description |
|---|---|---|---|---|
| 1 | FLAG_DELETION_FILES |
Yes | Yes | Fragments may contain deletion files, which record the tombstones of soft-deleted rows. |
| 2 | FLAG_STABLE_ROW_IDS |
Yes | Yes | Row IDs are stable for both moves and updates. Fragments contain an index mapping row IDs to row addresses. |
| 4 | FLAG_USE_V2_FORMAT_DEPRECATED |
No | No | Files are written with the new v2 format. This flag is deprecated and no longer used. |
| 8 | FLAG_TABLE_CONFIG |
No | Yes | Table config is present in the manifest. |
| 16 | FLAG_BASE_PATHS |
Yes | Yes | Dataset uses multiple base paths (for shallow clones or multi-base datasets). |
| 32 | FLAG_DISABLE_TRANSACTION_FILE |
No | Yes | Transactions are recorded in the manifest rather than in a separate transaction file. |
| 64 | FLAG_UNSTABLE_DATA_OVERLAY_FILES |
Yes | Yes | Fragments may carry data overlay files. Unstable: release builds reject it unless explicitly opted in. |
| 128 | FLAG_COVERED_INDEX_METADATA |
Yes | Yes | Some index declares covering columns (IndexMetadata.covering_fields). Without FLAG_INDEPENDENT_COVERING_FIELDS, the carried columns form the trailing suffix of fields. An implementation without this flag may select or maintain an index using the wrong fields. |
| 256 | FLAG_MIXED_DATA_FILE_VERSIONS |
Yes | Yes | The snapshot may reference recognized V2 data files with different exact versions. Both bits must be set and remain set on later versions. |
| 512 | FLAG_FRAG_REUSE_WITH_STABLE_ROW_IDS |
Yes | Yes | The table uses stable row IDs and carries a Fragment Reuse Index. |
| 1024 | FLAG_FRAGMENT_REUSE_INDEX |
Yes | Yes | The fragment reuse index records tagged transitions (IndexMetadata.index_version >= 1). Readers must translate row addresses through them; writers must preserve them. An implementation without this flag would decode the details as the legacy format and silently drop the transitions when it next rewrites the fragment reuse index. See FRI index versions. |
| 2048 | FLAG_UNSTABLE_SPILLED_ROW_LINEAGE |
Yes | Yes | Some fragment stores its row ids or row version sequences as hidden columns of a data file rather than inline. A reader without this flag would see the fragment as having no row ids. Unstable: release builds reject it unless explicitly opted in. |
| 4096 | FLAG_FRAGMENT_TREE |
Yes | Yes | Fragment records live in a fragment tree. Manifest.fragments is empty. |
| 8192 | FLAG_INDEPENDENT_COVERING_FIELDS |
Yes | Yes | Requires FLAG_COVERED_INDEX_METADATA and is retained together with it. IndexMetadata.fields contains only key fields, while covering_fields independently declares carried fields and may overlap fields. Implementations that only support the legacy suffix contract must reject the dataset. |
Flags with bit values 16384 and above are unknown; unknown flags cause implementations to reject the dataset with an "unsupported" error. The paired mixed-version reader and writer bits must either both be set or both be clear; a half-set manifest is invalid.