Blob Columns¶
Lance can store large binary objects (images, videos, audio, model artifacts) in blob columns, where they are treated like any other column payload in the dataset. Blob columns support both planned full-payload reads and lazy file-like access.
For the logical input contract, stored descriptors, and payload ownership rules, see the Blob v2 Specification.
Choosing between read_blobs and take_blobs
- For data loaders and batch processing that need complete byte payloads, use
read_blobs. - Use
take_blobswhen you need aBlobFilehandle for streaming, seeking, or partial reads.

If you're unsure about whether you need a blob column in the first place (and why it's useful), read the "when to use blob column vs. inline binary" section below.
Quick Start: Blob v2¶
The core Blob v2 write and file-like read workflows below are available in the Python, Rust, and Java SDKs (the planned batch-read helpers later on this page use the Python API).
data_storage_versionmeans the Lance file format version of a dataset.- A dataset has a default file format version. Append and compaction can select
another exact
2.xversion without rewriting existing files or changing that default. - File format
0.1cannot be mixed with the2.xfamily.
import lance
import pyarrow as pa
from lance import blob_array, blob_field
schema = pa.schema([
pa.field("id", pa.int64()),
blob_field("blob"),
])
table = pa.table(
{
"id": [1],
"blob": blob_array([b"hello blob v2"]),
},
schema=schema,
)
ds = lance.write_dataset(table, "./blobs_v22.lance", data_storage_version="2.2")
_row_address, payload = ds.read_blobs("blob", indices=[0])[0]
assert payload == b"hello blob v2"
use std::sync::Arc;
use arrow_array::{Int64Array, RecordBatch, RecordBatchIterator};
use arrow_schema::{DataType, Field, Schema};
use lance::blob::{blob_field, BlobArrayBuilder};
use lance::dataset::{WriteMode, WriteParams};
use lance::Dataset;
use lance_file::version::LanceFileVersion;
let schema = Arc::new(Schema::new(vec![
Field::new("id", DataType::Int64, false),
blob_field("blob", true),
]));
let mut blob_builder = BlobArrayBuilder::new(1);
blob_builder.push_bytes(b"hello blob v2")?;
let batch = RecordBatch::try_new(
schema.clone(),
vec![
Arc::new(Int64Array::from(vec![1])),
blob_builder.finish()?,
],
)?;
let params = WriteParams {
mode: WriteMode::Overwrite,
data_storage_version: Some(LanceFileVersion::V2_2),
..Default::default()
};
let ds = Arc::new(
Dataset::write(
RecordBatchIterator::new(vec![Ok(batch)], schema),
"./blobs_v22.lance",
Some(params),
)
.await?,
);
let blobs = ds.take_blobs_by_indices(&[0], "blob").await?;
if let Some(blob) = &blobs[0] {
let payload = blob.read().await?;
assert_eq!(payload.as_ref(), b"hello blob v2");
}
import org.apache.arrow.vector.types.pojo.ArrowType;
import org.apache.arrow.vector.types.pojo.Field;
import org.apache.arrow.vector.types.pojo.FieldType;
import org.apache.arrow.vector.types.pojo.Schema;
import org.lance.BlobFile;
import org.lance.Dataset;
import org.lance.WriteParams;
import java.nio.charset.StandardCharsets;
import java.util.Arrays;
import java.util.List;
import java.util.Map;
Field blobField =
new Field(
"blob",
new FieldType(
true,
ArrowType.Struct.INSTANCE,
null,
Map.of("ARROW:extension:name", "lance.blob.v2")),
Arrays.asList(
Field.nullable("data", ArrowType.LargeBinary.INSTANCE),
Field.nullable("uri", ArrowType.Utf8.INSTANCE)));
Schema schema =
new Schema(Arrays.asList(Field.notNullable("id", new ArrowType.Int(64, true)), blobField));
try (Dataset ds =
Dataset.write()
.allocator(allocator)
.reader(reader) // ArrowReader matching `schema`
.uri("./blobs_v22.lance")
.mode(WriteParams.WriteMode.OVERWRITE)
.dataStorageVersion("2.2")
.execute()) {
List<BlobFile> blobs = ds.takeBlobsByIndices(List.of(0L), "blob");
try (BlobFile blob = blobs.get(0)) {
byte[] payload = blob.read();
assert Arrays.equals(payload, "hello blob v2".getBytes(StandardCharsets.UTF_8));
}
}
Version Compatibility¶
Blob support is tied to the dataset's file format version. Earlier file format versions
(< 2.2) stored blobs using the lance-encoding:blob metadata field, while Blob
v2 introduces a new storage layout that requires file format >= 2.2.
The target file version selects the physical Blob encoding. For >= 2.2,
dataset writers also accept legacy blob-marked binary inputs and convert them
to Blob v2 before encoding. Older files keep their original encoding.
Dataset data_storage_version |
Legacy blob metadata (lance-encoding:blob) |
Blob v2 (lance.blob.v2) |
|---|---|---|
0.1, 2.0, 2.1 |
Supported for write/read | Not supported |
2.2+ |
Accepted as input; written as Blob v2 | Supported for write/read (recommended) |
Mixing legacy Blob and Blob v2 files¶
Append with data_storage_version="2.2" to start writing Blob v2 into an
existing 2.0 or 2.1 dataset. Both legacy blob-marked binary inputs and
blob_field / blob_array inputs are accepted. No separate upgrade operation
or historical-payload rewrite is required.
When the first Blob v2 file for a legacy column is committed, the snapshot uses
the Blob v2 logical schema for that column. Descriptor scans return the v2
kind, position, size, blob_id, blob_uri struct across all fragments, including
selections containing only old files. Old payloads appear as Inline descriptors
pointing into their original data files. take_blobs, read_blobs, and
blob_handling="all_binary" continue to expose the same payloads, nulls, and
empty values. Historical snapshots retain their previous schema.
Descriptor consumers that access fields by position or compare exact Arrow
schemas must account for the extended structure. Interpret position together
with kind and the object reference.
The dataset's default file version remains unchanged. Select 2.2 or newer
explicitly when writing Blob v2 structs or compacting the mixed column into
Blob v2 files. Compaction rewrites the selected fragments; it is not required
to enable new Blob v2 writes.
Managed objects and client compatibility¶
Set schema metadata lance.blob.managed to "true" to store out-of-line Blob v2
payloads in independently named _blobs/<uuid>.blob objects. Without this
opt-in, writes and compaction keep Inline/Packed/Dedicated placement, and
prepared Managed references are rejected. This works with file formats 2.2
and 2.3; it does not require choosing 2.3.
Committing the opt-in immediately publishes the Managed Blob reader and writer capability, even without new blob files. Clients that do not understand the capability must refuse to open a flagged snapshot. The capability remains set across later writes and restores, including after removing the opt-in.
With the opt-in, compaction preserves Managed payload objects and can adopt
existing Packed or Dedicated sidecars in place. For example, it can replace
data/A.lance with data/B.lance while B's Managed descriptor records the base and path of
the existing data/A/0001.blob. Cleanup can then delete data/A.lance without
copying or deleting the blob that B still references. The blob's path may retain
A's name, but resolving and retaining the blob no longer requires A's data file.
Cleanup retains objects referenced by protected snapshots; a partially live
packed object is retained as a whole. Blob reads continue to return the same
bytes. Raw descriptor scans report kind = 4 for a Managed object relative to
the data file's table base, or kind = 5 with an explicit registered base ID.
The default table base is implicit: moving the complete dataset moves its local
Managed objects without retaining a URI to the old location. Explicit base IDs
must already be registered before writing references to them. A shallow clone
keeps the source table base through its data-file metadata; compaction preserves
that base when it moves descriptors into new files. A deep clone copies local
objects at the same relative paths without rewriting their descriptors.
Before activating Managed Blobs on an existing table, upgrade all clients that run table maintenance to a version that supports Managed Blobs, or stop their maintenance tasks. After activation, all maintenance must use a supporting client.
The table flag prevents older clients from opening a flagged snapshot. It does not revoke a handle opened before activation or prevent opening an unflagged historical snapshot. Older clients, including v11.0.0, can run cleanup through these handles without checking the feature flags of the other manifests they inspect. Such cleanup can delete adopted sidecars that newer snapshots still reference, because it treats the original data file as their owner. The table flag therefore does not protect against these old maintenance paths.
Explicit sharing within a table¶
Advanced writers can upload a payload once and reuse the returned descriptor
across rows and fragments. Open a LanceFileSession at the table root and call
open_managed_blob_writer(). This writes a uniquely named _blobs/<uuid>.blob
object through a ManagedBlobWriter, independently of data file names.
finish() completes the upload and
returns descriptors in write order; BlobDescriptorArrayBuilder.append can
reuse each descriptor any number of times. No content comparison is performed.
import lance
import pyarrow as pa
from lance.file import LanceFileSession
uri = "./shared_blobs.lance"
writer = LanceFileSession(uri).open_managed_blob_writer()
writer.write_blob(b"shared")
(shared,) = writer.finish()
builder = lance.BlobDescriptorArrayBuilder("blob")
for _ in range(4):
builder.append(shared)
references = pa.Table.from_arrays(
[builder.finish()],
schema=pa.schema(
[builder.field], metadata={"lance.blob.managed": "true"}
),
)
ds = lance.write_dataset(
references, uri, data_storage_version="2.2", max_rows_per_file=2
)
assert len(ds.get_fragments()) == 2
assert [data for _, data in ds.read_blobs("blob", indices=range(4))] == [b"shared"] * 4
ds.optimize.compact_files()
assert len(ds.get_fragments()) == 1
assert [data for _, data in ds.read_blobs("blob", indices=range(4))] == [b"shared"] * 4
The four rows share one immutable payload object, including after compaction. Cleanup retains the whole object while any retained snapshot references it. When reusing an object from an existing snapshot, keep that snapshot protected, for example with a tag, until the append commits if concurrent updates and cleanup can otherwise retire it. An open dataset handle alone does not protect a snapshot from cleanup.
Use references from the same table and preserve their base identity. Kind 5
descriptors require their registered blob_id as base_id; for kind 4, the base
comes from the source data file. The example uses newly written data in the
table's primary base. Do not copy an implicit reference from a shallow clone's
source file into a destination with a different base. Cross-table sharing and
automatic deduplication are outside this API's contract. All maintenance clients
must support Managed Blobs as described above.
For a registered base, open the session at that base's root and pass
open_managed_blob_writer(base_id=...). Register the same root under that ID
through the existing dataset base APIs before committing the descriptors.
This file-level API cannot check that a caller-supplied root matches a dataset's
base registry; the caller must preserve that binding. The commit rejects unknown
base IDs. The returned descriptors carry the object's relative path and actual
base identity; Managed objects do not use sidecar blob IDs.
An implicit reference (base_id=None) is interpreted relative to the destination
data file's base. Keep these data files in primary storage as in the example.
When writing data files to other bases, give the payload root an explicit
registered base ID and supply that ID to the Managed writer. The same reference
can then be used in data files written to different bases.
To reuse an existing immutable object or a subrange, use
append_managed(uri, offset, size, base_id=None). Managed zero-length ranges
represent empty values; they do not discover the object's size. Null descriptors
remain null. field, values = writer.finish_column(field_name) returns the
matching extension field and prepared array together when each uploaded payload
corresponds to one output row. For explicit sharing, reuse the descriptors from
finish() instead of uploading the payload again.
The upload alone does not publish a table version. Finish it before committing
references, and do not run cleanup with delete_unverified=True concurrently
with preparation: that option may delete in-flight, uncommitted objects. Failed
or abandoned uploads follow the existing unverified-object retention policy.
Cleanup retains the entire packed object while any protected reference remains,
including unused byte ranges within the same object.
Rust callers use ManagedBlobWriter::try_new(store, base_path, base_id),
then write_blob, finish, and BlobDescriptorArrayBuilder::push / extend.
Write the resulting record batches through Dataset::write with the same
schema opt-in. The method's Rustdoc includes a complete write, commit, compaction
and read example. These APIs preserve the existing Packed/Dedicated sidecar
preparation APIs for callers preparing a specific data file.
Blob v2: Write Patterns¶
Use blob_field and blob_array to build blob v2 columns.
Logical Arrow schema¶
A blob v2 field is tagged with ARROW:extension:name = "lance.blob.v2". Writers
accept these logical struct shapes:
| Shape | Children | Use |
|---|---|---|
| Minimal | data: LargeBinary?, uri: Utf8? |
Inline bytes or a complete external object |
| Complete | Minimal fields plus position: UInt64?, size: UInt64? |
An optional byte range within an external object |
Every non-null row must set exactly one of data and uri. For the complete
shape, position and size must either both be set or both be null, a range
requires uri, and an explicit range must have size > 0. Use inline b"" for
an empty blob; a URI without range fields still represents the complete external
object, including an empty object. Python's blob_field and BlobType use the
complete shape. Lance preserves an accepted logical shape, including child
fields, nullability, and metadata, across create, append, and merge-insert
writes; descriptor scans still return the compact stored descriptor shape.
import lance
import pyarrow as pa
from lance import Blob, blob_array, blob_field
schema = pa.schema([
pa.field("id", pa.int64()),
blob_field("blob", nullable=True),
])
# A single column can mix:
# - inline bytes
# - external URI
# - external URI slice (position + size)
# - null
rows = pa.table(
{
"id": [1, 2, 3, 4],
"blob": blob_array([
b"inline-bytes",
"s3://bucket/path/video.mp4",
Blob.from_uri("s3://bucket/archive.tar", position=4096, size=8192),
None,
]),
},
schema=schema,
)
ds = lance.write_dataset(
rows,
"./blobs_v22.lance",
data_storage_version="2.2",
)
Note:
- By default, external blob URIs must map to a registered non-dataset-root base path.
- If you need to reference external objects outside those bases, set
allow_external_blob_outside_bases=Truewhen writing in embedded mode. A controlled service can reject these references regardless of this option. - Blob v2 storage layout thresholds can be configured per column with
blob_field(..., inline_size_threshold=..., dedicated_size_threshold=...). The inline threshold controls when values move from the data file to packed.blobsidecar storage. The dedicated threshold controls when values move from packed sidecar storage to a dedicated.blobfile. The dedicated threshold is checked first. For existing columns, these thresholds are stored in the dataset schema; appends that explicitly provide different threshold metadata for the same column are rejected. blob_pack_file_size_thresholdis a write option for rolling packed.blobsidecar files. It does not control inline-vs-packed placement.- Blob v2 fields can be nested inside structs and variable-length lists. Blob-aware
scans preserve the surrounding nested layout; use
blob_handling="all_binary"to materialize nested blob payloads as bytes.
Service access policy and external object caches¶
Rust services can restrict external blobs to registered storage bases when constructing their shared session:
use std::sync::Arc;
use lance::session::{Session, UnregisteredExternalBlobPolicy};
use lance_io::object_store::ObjectStoreRegistry;
let registry = ObjectStoreRegistry::default()
.without_providers(["file", "file-object-store", "file+uring"])?;
let session = Session::new(0, 0, Arc::new(registry))
.with_unregistered_external_blob_policy(UnregisteredExternalBlobPolicy::Deny);
Pass this session through ReadParams.session, DatasetBuilder::with_session,
or WriteParams.session. Reads reject unregistered external descriptors
(base_id == 0) before parsing the URI, opening a store, or probing its size.
Logical URI writes under this policy must resolve to a registered storage base;
the local allow_external_blob_outside_bases option cannot override the policy.
The default session allows unregistered references for embedded use.
without_providers consumes an unshared registry and discards its cached store
bindings. Apply it before handing out sessions or store handles; it does not
revoke handles that already exist. Registered bases still need appropriate
credentials and deployment restrictions.
Externally owned objects can change independently of table versions. Rust store
wrappers prepare their external read view through
WrappingObjectStore::wrap_external_blob, reusing the same physical store,
credentials, endpoint, and connection pool. This hook retains the regular
wrapper by default. A service's immutable page-cache wrapper should return the
provided original view to bypass that cache, while retaining authorization and
observation wrappers in the chain. Inline, Packed, Dedicated, and Managed
payloads continue through the regular wrapping path.
Example: packed external blobs (single container file)¶
import io
import tarfile
from pathlib import Path
import lance
import pyarrow as pa
from lance import Blob, blob_array, blob_field
# Build a tar file with three payloads
payloads = {
"a.bin": b"alpha",
"b.bin": b"bravo",
"c.bin": b"charlie",
}
with tarfile.open("container.tar", "w") as tf:
for name, data in payloads.items():
info = tarfile.TarInfo(name)
info.size = len(data)
tf.addfile(info, io.BytesIO(data))
# Capture offset/size for each member
blob_values = []
with tarfile.open("container.tar", "r") as tf:
container_uri = Path("container.tar").resolve().as_uri()
for name in payloads:
m = tf.getmember(name)
blob_values.append(Blob.from_uri(container_uri, position=m.offset_data, size=m.size))
schema = pa.schema([
pa.field("name", pa.utf8()),
blob_field("blob"),
])
rows = pa.table(
{
"name": list(payloads.keys()),
"blob": blob_array(blob_values),
},
schema=schema,
)
ds = lance.write_dataset(
rows,
"./packed_blobs_v22.lance",
data_storage_version="2.2",
allow_external_blob_outside_bases=True,
)
Blob v2: Read Patterns¶
Choose the read API based on the payload shape you want:
| API | Returns | Use When |
|---|---|---|
read_blobs |
List[Tuple[int, Optional[bytes]]] |
You need complete blob payloads in memory, such as training loaders or batch preprocessing. |
read_blob_ranges |
List[Tuple[int, int, Optional[bytes]]] |
You need selected byte ranges from multiple rows without materializing complete blobs. |
take_blobs |
List[Optional[BlobFile]] |
You need file-like objects for streaming, seeking, or partial reads. |
lance.read_blob_files / lance.read_blob_file_ranges |
List[Tuple[int, Optional[bytes]]] |
You already hold BlobFile handles and read the same blobs repeatedly, such as a shard index followed by chunk ranges. |
scanner(..., blob_handling="all_binary") |
Arrow binary columns | You want blob columns in a scan result or pyarrow.Table. |
Do not wrap take_blobs in your own thread pool just to call read() or
readall() on every blob. Use read_blobs instead; it plans and executes
batched blob reads through Lance's scheduler.
Exactly one selector must be provided to read_blobs or take_blobs: ids,
indices, or addresses. read_blob_ranges accepts the same selector kinds
through its required selector argument.
| Selector | Typical Use | Stability |
|---|---|---|
indices |
Positional reads within one dataset snapshot | Stable within that snapshot |
ids |
Logical row-id based reads | Stable logical identity (when row ids are available) |
addresses |
Low-level physical reads and debugging | Unstable physical location |
Read complete payloads by row indices¶
import lance
ds = lance.dataset("./blobs_v22.lance")
rows = ds.read_blobs("blob", indices=[0, 1])
payloads = [payload for _row_address, payload in rows]
Read complete payloads by row ids¶
import lance
ds = lance.dataset("./blobs_v22.lance")
row_ids = ds.to_table(columns=[], with_row_id=True).column("_rowid").to_pylist()
rows = ds.read_blobs("blob", ids=row_ids[:2])
Read complete payloads by row addresses¶
import lance
ds = lance.dataset("./blobs_v22.lance")
row_addrs = ds.to_table(columns=[], with_row_address=True).column("_rowaddr").to_pylist()
rows = ds.read_blobs("blob", addresses=row_addrs[:2])
Blob selection APIs preserve logical result cardinality. read_blobs() and
take_blobs() return one element per selected row, and read_blob_ranges()
returns one element per request. A null blob is returned as None; a valid
empty blob remains a non-null empty payload or zero-length BlobFile.
Read row-specific byte ranges¶
Use read_blob_ranges to read multiple blob-local ranges with one planned API
call. Each request is a (row, offset, length) tuple, and selector determines
whether every row is interpreted as a row ID, row address, or dataset index.
import lance
ds = lance.dataset("./blobs_v22.lance")
results = ds.read_blob_ranges(
"blob",
requests=[
(7, 0, 1024),
(7, 4096, 1024),
(12, 0, 0),
],
selector="indices",
)
for request_index, row_address, data in results:
if data is None:
# The selected blob is null.
continue
print(request_index, row_address, len(data))
Each result contains the zero-based request_index, the resolved physical row
address, and the requested bytes. request_index identifies the original
request when the same row appears more than once.
A request on a null blob returns None, including when its range is empty. An
empty range on a non-null blob returns b"" without payload I/O. For every
request, offset + length must fit in an unsigned 64-bit integer. A range on a
non-null blob must not extend beyond its logical size; blob-local bounds are not
evaluated for null blobs because they have no logical payload length.
Read through handles you already hold¶
read_blobs and read_blob_ranges look up the selected rows' blob descriptors
on every call before they read any payload bytes. On object storage that lookup
is a separate request the payload reads wait for. When you read the same blobs
repeatedly, take handles once with take_blobs and pass them to
read_blob_files or read_blob_file_ranges: each call then issues only the
payload requests, planned and coalesced the same way.
import lance
ds = lance.dataset("./blobs_v22.lance")
handles = ds.take_blobs("blob", indices=[0, 1, 2])
payloads = [data for _request_index, data in lance.read_blob_files(handles)]
headers = lance.read_blob_file_ranges(
[(handle, 0, 16) for handle in handles]
)
Each result is a (request_index, data) pair. A None handle, which
take_blobs returns for a null blob, returns None; keep null handles in the
request list so request_index still matches the original handle positions.
Handles are not tied to a
dataset receiver. Handles retain the object locations and byte ranges resolved
when they were opened, including when passed alongside handles from other
datasets or snapshots. Later dataset commits do not re-resolve those descriptors;
cleanup must retain the referenced payloads. External objects follow their
storage provider's consistency semantics.
The first non-null handle supplies default I/O parallelism and buffer size.
Use the positive io_parallelism and io_buffer_size keyword arguments to
override them. Empty and all-None inputs issue no storage requests.
Read blob columns as Arrow binary¶
import lance
ds = lance.dataset("./blobs_v22.lance")
table = ds.scanner(columns=["blob"], blob_handling="all_binary").to_table()
payloads = table.column("blob").to_pylist()
Open file-like blob handles lazily¶
Sequential reads are buffered, 512 KiB by default. Pass buffer_size=0
for unbuffered reads. read_range and read_ranges do not use this
buffer.
import org.lance.BlobFile;
import org.lance.Dataset;
import java.util.List;
try (Dataset ds = Dataset.open("./blobs_v22.lance", allocator)) {
List<BlobFile> blobs = ds.takeBlobsByIndices(List.of(0L, 1L), "blob");
try {
if (blobs.get(0) != null) {
byte[] header = blobs.get(0).readUpTo(1024);
}
} finally {
for (BlobFile blob : blobs) {
if (blob != null) blob.close();
}
}
}
Example: decode video frames lazily¶
import av
import lance
ds = lance.dataset("./videos_v22.lance")
blob = ds.take_blobs("video", indices=[0])[0]
if blob is None:
raise ValueError("video blob is null")
start_ms, end_ms = 500, 1000
with av.open(blob) as container:
stream = container.streams.video[0]
stream.codec_context.skip_frame = "NONKEY"
start = (start_ms / 1000) / stream.time_base
end = (end_ms / 1000) / stream.time_base
container.seek(int(start), stream=stream)
for frame in container.decode(stream):
if frame.time is not None and frame.time > end_ms / 1000:
break
# process frame
pass
Legacy Compatibility (data_storage_version <= 2.1)¶
If you need to keep writing legacy blob columns, use file format 0.1, 2.0, or 2.1
and mark LargeBinary fields with a metadata kwarg "lance-encoding:blob": true.
import lance
import pyarrow as pa
schema = pa.schema([
pa.field("id", pa.int64()),
pa.field(
"video",
pa.large_binary(),
metadata={"lance-encoding:blob": "true"},
),
])
table = pa.table(
{
"id": [1, 2],
"video": [b"foo", b"bar"],
},
schema=schema,
)
ds = lance.write_dataset(
table,
"./legacy_blob_dataset",
data_storage_version="2.1",
)
With data_storage_version >= 2.2, dataset writers convert this input to Blob v2.
For new datasets, prefer blob_field / blob_array with file format 2.2.
Rewrite to a New Blob v2 Dataset¶
To replace all historical legacy Blob encodings at once, you can rewrite them
into a new dataset with data_storage_version="2.2". This is optional for
2.0 / 2.1 tables, which can instead append Blob v2 files as described above.
File format 0.1 tables still require a rewrite to move into the 2.x family.
import lance
import pyarrow as pa
from lance import blob_array, blob_field
legacy = lance.dataset("./legacy_blob_dataset")
raw = legacy.scanner(columns=["id", "video"], blob_handling="all_binary").to_table()
new_schema = pa.schema([
pa.field("id", pa.int64()),
blob_field("video"),
])
rewritten = pa.table(
{
"id": raw.column("id"),
"video": blob_array(raw.column("video").to_pylist()),
},
schema=new_schema,
)
lance.write_dataset(
rewritten,
"./blob_v22_dataset",
data_storage_version="2.2",
)
Warning
- The example above materializes binary payloads in memory (
blob_handling="all_binary"andto_pylist()). - For large datasets, prefer chunked/batched rewrite pipelines.
When to Use a Blob Column vs. Inline Binary¶
Not every binary column needs to be a blob column. Plain Arrow binary/large_binary stores bytes inline, interleaved with your other columns, which is simplest and fastest for really small blobs (e.g., thumbnail images). Using a blob column to store the binary payload makes sense when either of these holds:
- You need partial or streaming reads. Inline binary is always read in full; there is no way to fetch a byte range without materializing the entire value. Blob columns expose
read_blob_rangesfor planned row-specific range reads andtake_blobs→BlobFilehandles for caller-driven seeks, so you pay only for the bytes you touch. - Your values are large (roughly 1 MB or more on average). Operations that rewrite entire rows, such as compaction or some updates, must copy the large inline payloads forward into the new version — even when those bytes never changed. The bigger the payload, the more bytes you rewrite per logical change (write amplification). A blob column keeps large payloads in separate
.blobfiles that are referenced rather than re-copied, so these operations don't rewrite the heavy bytes.
Tip
As a rule of thumb, if average payload size is below a few tens of KB and you only ever read whole values, plain inline binary is fine. Above ~1 MB, or any time you want file-like access, prefer a blob column. Blob v2 also tunes this automatically: by default it keeps payloads up to 64 KiB inline, packs payloads over 64 KiB and up to 4 MiB into shared .blob sidecars, and gives payloads over 4 MiB their own dedicated .blob file. These thresholds are configurable per column.
Troubleshooting¶
This section contains commonly noticed issues or errors, and explains how to address them.
Blob v2 requires file version >= 2.2¶
Cause: You are writing blob v2 values into a dataset/file format below 2.2.
Fix: Write to a dataset created with data_storage_version="2.2" (or newer).
Legacy blob columns ... are not supported for file version >= 2.2¶
Cause: A lower-level file writer received legacy blob metadata instead of prepared Blob v2 input. Fix: Use the dataset writer, which adapts legacy byte inputs, or prepare Blob v2 columns with the file-level Blob APIs.
Exactly one of ids, indices, or addresses must be specified¶
Cause: read_blobs or take_blobs received none or multiple selectors.
Fix: Provide exactly one of ids, indices, or addresses.