Lance
Docs /User Guide /Core Operations /Blob API

Blob Columns

Lance can store large binary objects (images, videos, audio, model artifacts) in blob columns, where they are treated like any other column payload in the dataset. Blob columns support both planned full-payload reads and lazy file-like access.

For the logical input contract, stored descriptors, and payload ownership rules, see the Blob v2 Specification.

Choosing between read_blobs and take_blobs

  • For data loaders and batch processing that need complete byte payloads, use read_blobs.
  • Use take_blobs when you need a BlobFile handle for streaming, seeking, or partial reads.

Blob v2 overview Blob v2 overview

If you're unsure about whether you need a blob column in the first place (and why it's useful), read the "when to use blob column vs. inline binary" section below.

Quick Start: Blob v2

The core Blob v2 write and file-like read workflows below are available in the Python, Rust, and Java SDKs (the planned batch-read helpers later on this page use the Python API).

  • data_storage_version means the Lance file format version of a dataset.
  • A dataset has a default file format version. Append and compaction can select another exact 2.x version without rewriting existing files or changing that default.
  • File format 0.1 cannot be mixed with the 2.x family.
import lance
import pyarrow as pa
from lance import blob_array, blob_field

schema = pa.schema([
    pa.field("id", pa.int64()),
    blob_field("blob"),
])

table = pa.table(
    {
        "id": [1],
        "blob": blob_array([b"hello blob v2"]),
    },
    schema=schema,
)

ds = lance.write_dataset(table, "./blobs_v22.lance", data_storage_version="2.2")

_row_address, payload = ds.read_blobs("blob", indices=[0])[0]
assert payload == b"hello blob v2"
use std::sync::Arc;
use arrow_array::{Int64Array, RecordBatch, RecordBatchIterator};
use arrow_schema::{DataType, Field, Schema};
use lance::blob::{blob_field, BlobArrayBuilder};
use lance::dataset::{WriteMode, WriteParams};
use lance::Dataset;
use lance_file::version::LanceFileVersion;

let schema = Arc::new(Schema::new(vec![
    Field::new("id", DataType::Int64, false),
    blob_field("blob", true),
]));
let mut blob_builder = BlobArrayBuilder::new(1);
blob_builder.push_bytes(b"hello blob v2")?;
let batch = RecordBatch::try_new(
    schema.clone(),
    vec![
        Arc::new(Int64Array::from(vec![1])),
        blob_builder.finish()?,
    ],
)?;

let params = WriteParams {
    mode: WriteMode::Overwrite,
    data_storage_version: Some(LanceFileVersion::V2_2),
    ..Default::default()
};
let ds = Arc::new(
    Dataset::write(
        RecordBatchIterator::new(vec![Ok(batch)], schema),
        "./blobs_v22.lance",
        Some(params),
    )
    .await?,
);

let blobs = ds.take_blobs_by_indices(&[0], "blob").await?;
if let Some(blob) = &blobs[0] {
    let payload = blob.read().await?;
    assert_eq!(payload.as_ref(), b"hello blob v2");
}
import org.apache.arrow.vector.types.pojo.ArrowType;
import org.apache.arrow.vector.types.pojo.Field;
import org.apache.arrow.vector.types.pojo.FieldType;
import org.apache.arrow.vector.types.pojo.Schema;
import org.lance.BlobFile;
import org.lance.Dataset;
import org.lance.WriteParams;
import java.nio.charset.StandardCharsets;
import java.util.Arrays;
import java.util.List;
import java.util.Map;

Field blobField =
    new Field(
        "blob",
        new FieldType(
            true,
            ArrowType.Struct.INSTANCE,
            null,
            Map.of("ARROW:extension:name", "lance.blob.v2")),
        Arrays.asList(
            Field.nullable("data", ArrowType.LargeBinary.INSTANCE),
            Field.nullable("uri", ArrowType.Utf8.INSTANCE)));
Schema schema =
    new Schema(Arrays.asList(Field.notNullable("id", new ArrowType.Int(64, true)), blobField));

try (Dataset ds =
    Dataset.write()
        .allocator(allocator)
        .reader(reader) // ArrowReader matching `schema`
        .uri("./blobs_v22.lance")
        .mode(WriteParams.WriteMode.OVERWRITE)
        .dataStorageVersion("2.2")
        .execute()) {
  List<BlobFile> blobs = ds.takeBlobsByIndices(List.of(0L), "blob");
  try (BlobFile blob = blobs.get(0)) {
    byte[] payload = blob.read();
    assert Arrays.equals(payload, "hello blob v2".getBytes(StandardCharsets.UTF_8));
  }
}

Version Compatibility

Blob support is tied to the dataset's file format version. Earlier file format versions (< 2.2) stored blobs using the lance-encoding:blob metadata field, while Blob v2 introduces a new storage layout that requires file format >= 2.2.

The target file version selects the physical Blob encoding. For >= 2.2, dataset writers also accept legacy blob-marked binary inputs and convert them to Blob v2 before encoding. Older files keep their original encoding.

Dataset data_storage_version Legacy blob metadata (lance-encoding:blob) Blob v2 (lance.blob.v2)
0.1, 2.0, 2.1 Supported for write/read Not supported
2.2+ Accepted as input; written as Blob v2 Supported for write/read (recommended)

Mixing legacy Blob and Blob v2 files

Append with data_storage_version="2.2" to start writing Blob v2 into an existing 2.0 or 2.1 dataset. Both legacy blob-marked binary inputs and blob_field / blob_array inputs are accepted. No separate upgrade operation or historical-payload rewrite is required.

When the first Blob v2 file for a legacy column is committed, the snapshot uses the Blob v2 logical schema for that column. Descriptor scans return the v2 kind, position, size, blob_id, blob_uri struct across all fragments, including selections containing only old files. Old payloads appear as Inline descriptors pointing into their original data files. take_blobs, read_blobs, and blob_handling="all_binary" continue to expose the same payloads, nulls, and empty values. Historical snapshots retain their previous schema.

Descriptor consumers that access fields by position or compare exact Arrow schemas must account for the extended structure. Interpret position together with kind and the object reference.

The dataset's default file version remains unchanged. Select 2.2 or newer explicitly when writing Blob v2 structs or compacting the mixed column into Blob v2 files. Compaction rewrites the selected fragments; it is not required to enable new Blob v2 writes.

Managed objects and client compatibility

Set schema metadata lance.blob.managed to "true" to store out-of-line Blob v2 payloads in independently named _blobs/<uuid>.blob objects. Without this opt-in, writes and compaction keep Inline/Packed/Dedicated placement, and prepared Managed references are rejected. This works with file formats 2.2 and 2.3; it does not require choosing 2.3.

Committing the opt-in immediately publishes the Managed Blob reader and writer capability, even without new blob files. Clients that do not understand the capability must refuse to open a flagged snapshot. The capability remains set across later writes and restores, including after removing the opt-in.

With the opt-in, compaction preserves Managed payload objects and can adopt existing Packed or Dedicated sidecars in place. For example, it can replace data/A.lance with data/B.lance while B's Managed descriptor records the base and path of the existing data/A/0001.blob. Cleanup can then delete data/A.lance without copying or deleting the blob that B still references. The blob's path may retain A's name, but resolving and retaining the blob no longer requires A's data file. Cleanup retains objects referenced by protected snapshots; a partially live packed object is retained as a whole. Blob reads continue to return the same bytes. Raw descriptor scans report kind = 4 for a Managed object relative to the data file's table base, or kind = 5 with an explicit registered base ID. The default table base is implicit: moving the complete dataset moves its local Managed objects without retaining a URI to the old location. Explicit base IDs must already be registered before writing references to them. A shallow clone keeps the source table base through its data-file metadata; compaction preserves that base when it moves descriptors into new files. A deep clone copies local objects at the same relative paths without rewriting their descriptors.

Before activating Managed Blobs on an existing table, upgrade all clients that run table maintenance to a version that supports Managed Blobs, or stop their maintenance tasks. After activation, all maintenance must use a supporting client.

The table flag prevents older clients from opening a flagged snapshot. It does not revoke a handle opened before activation or prevent opening an unflagged historical snapshot. Older clients, including v11.0.0, can run cleanup through these handles without checking the feature flags of the other manifests they inspect. Such cleanup can delete adopted sidecars that newer snapshots still reference, because it treats the original data file as their owner. The table flag therefore does not protect against these old maintenance paths.

Explicit sharing within a table

Advanced writers can upload a payload once and reuse the returned descriptor across rows and fragments. Open a LanceFileSession at the table root and call open_managed_blob_writer(). This writes a uniquely named _blobs/<uuid>.blob object through a ManagedBlobWriter, independently of data file names. finish() completes the upload and returns descriptors in write order; BlobDescriptorArrayBuilder.append can reuse each descriptor any number of times. No content comparison is performed.

import lance
import pyarrow as pa
from lance.file import LanceFileSession

uri = "./shared_blobs.lance"
writer = LanceFileSession(uri).open_managed_blob_writer()
writer.write_blob(b"shared")
(shared,) = writer.finish()

builder = lance.BlobDescriptorArrayBuilder("blob")
for _ in range(4):
    builder.append(shared)
references = pa.Table.from_arrays(
    [builder.finish()],
    schema=pa.schema(
        [builder.field], metadata={"lance.blob.managed": "true"}
    ),
)
ds = lance.write_dataset(
    references, uri, data_storage_version="2.2", max_rows_per_file=2
)
assert len(ds.get_fragments()) == 2
assert [data for _, data in ds.read_blobs("blob", indices=range(4))] == [b"shared"] * 4

ds.optimize.compact_files()
assert len(ds.get_fragments()) == 1
assert [data for _, data in ds.read_blobs("blob", indices=range(4))] == [b"shared"] * 4

The four rows share one immutable payload object, including after compaction. Cleanup retains the whole object while any retained snapshot references it. When reusing an object from an existing snapshot, keep that snapshot protected, for example with a tag, until the append commits if concurrent updates and cleanup can otherwise retire it. An open dataset handle alone does not protect a snapshot from cleanup.

Use references from the same table and preserve their base identity. Kind 5 descriptors require their registered blob_id as base_id; for kind 4, the base comes from the source data file. The example uses newly written data in the table's primary base. Do not copy an implicit reference from a shallow clone's source file into a destination with a different base. Cross-table sharing and automatic deduplication are outside this API's contract. All maintenance clients must support Managed Blobs as described above.

For a registered base, open the session at that base's root and pass open_managed_blob_writer(base_id=...). Register the same root under that ID through the existing dataset base APIs before committing the descriptors. This file-level API cannot check that a caller-supplied root matches a dataset's base registry; the caller must preserve that binding. The commit rejects unknown base IDs. The returned descriptors carry the object's relative path and actual base identity; Managed objects do not use sidecar blob IDs.

An implicit reference (base_id=None) is interpreted relative to the destination data file's base. Keep these data files in primary storage as in the example. When writing data files to other bases, give the payload root an explicit registered base ID and supply that ID to the Managed writer. The same reference can then be used in data files written to different bases.

To reuse an existing immutable object or a subrange, use append_managed(uri, offset, size, base_id=None). Managed zero-length ranges represent empty values; they do not discover the object's size. Null descriptors remain null. field, values = writer.finish_column(field_name) returns the matching extension field and prepared array together when each uploaded payload corresponds to one output row. For explicit sharing, reuse the descriptors from finish() instead of uploading the payload again.

The upload alone does not publish a table version. Finish it before committing references, and do not run cleanup with delete_unverified=True concurrently with preparation: that option may delete in-flight, uncommitted objects. Failed or abandoned uploads follow the existing unverified-object retention policy. Cleanup retains the entire packed object while any protected reference remains, including unused byte ranges within the same object.

Rust callers use ManagedBlobWriter::try_new(store, base_path, base_id), then write_blob, finish, and BlobDescriptorArrayBuilder::push / extend. Write the resulting record batches through Dataset::write with the same schema opt-in. The method's Rustdoc includes a complete write, commit, compaction and read example. These APIs preserve the existing Packed/Dedicated sidecar preparation APIs for callers preparing a specific data file.

Blob v2: Write Patterns

Use blob_field and blob_array to build blob v2 columns.

Logical Arrow schema

A blob v2 field is tagged with ARROW:extension:name = "lance.blob.v2". Writers accept these logical struct shapes:

Shape Children Use
Minimal data: LargeBinary?, uri: Utf8? Inline bytes or a complete external object
Complete Minimal fields plus position: UInt64?, size: UInt64? An optional byte range within an external object

Every non-null row must set exactly one of data and uri. For the complete shape, position and size must either both be set or both be null, a range requires uri, and an explicit range must have size > 0. Use inline b"" for an empty blob; a URI without range fields still represents the complete external object, including an empty object. Python's blob_field and BlobType use the complete shape. Lance preserves an accepted logical shape, including child fields, nullability, and metadata, across create, append, and merge-insert writes; descriptor scans still return the compact stored descriptor shape.

import lance
import pyarrow as pa
from lance import Blob, blob_array, blob_field

schema = pa.schema([
    pa.field("id", pa.int64()),
    blob_field("blob", nullable=True),
])

# A single column can mix:
# - inline bytes
# - external URI
# - external URI slice (position + size)
# - null
rows = pa.table(
    {
        "id": [1, 2, 3, 4],
        "blob": blob_array([
            b"inline-bytes",
            "s3://bucket/path/video.mp4",
            Blob.from_uri("s3://bucket/archive.tar", position=4096, size=8192),
            None,
        ]),
    },
    schema=schema,
)

ds = lance.write_dataset(
    rows,
    "./blobs_v22.lance",
    data_storage_version="2.2",
)

Note:

  • By default, external blob URIs must map to a registered non-dataset-root base path.
  • If you need to reference external objects outside those bases, set allow_external_blob_outside_bases=True when writing in embedded mode. A controlled service can reject these references regardless of this option.
  • Blob v2 storage layout thresholds can be configured per column with blob_field(..., inline_size_threshold=..., dedicated_size_threshold=...). The inline threshold controls when values move from the data file to packed .blob sidecar storage. The dedicated threshold controls when values move from packed sidecar storage to a dedicated .blob file. The dedicated threshold is checked first. For existing columns, these thresholds are stored in the dataset schema; appends that explicitly provide different threshold metadata for the same column are rejected.
  • blob_pack_file_size_threshold is a write option for rolling packed .blob sidecar files. It does not control inline-vs-packed placement.
  • Blob v2 fields can be nested inside structs and variable-length lists. Blob-aware scans preserve the surrounding nested layout; use blob_handling="all_binary" to materialize nested blob payloads as bytes.

Service access policy and external object caches

Rust services can restrict external blobs to registered storage bases when constructing their shared session:

use std::sync::Arc;
use lance::session::{Session, UnregisteredExternalBlobPolicy};
use lance_io::object_store::ObjectStoreRegistry;

let registry = ObjectStoreRegistry::default()
    .without_providers(["file", "file-object-store", "file+uring"])?;
let session = Session::new(0, 0, Arc::new(registry))
    .with_unregistered_external_blob_policy(UnregisteredExternalBlobPolicy::Deny);

Pass this session through ReadParams.session, DatasetBuilder::with_session, or WriteParams.session. Reads reject unregistered external descriptors (base_id == 0) before parsing the URI, opening a store, or probing its size. Logical URI writes under this policy must resolve to a registered storage base; the local allow_external_blob_outside_bases option cannot override the policy. The default session allows unregistered references for embedded use.

without_providers consumes an unshared registry and discards its cached store bindings. Apply it before handing out sessions or store handles; it does not revoke handles that already exist. Registered bases still need appropriate credentials and deployment restrictions.

Externally owned objects can change independently of table versions. Rust store wrappers prepare their external read view through WrappingObjectStore::wrap_external_blob, reusing the same physical store, credentials, endpoint, and connection pool. This hook retains the regular wrapper by default. A service's immutable page-cache wrapper should return the provided original view to bypass that cache, while retaining authorization and observation wrappers in the chain. Inline, Packed, Dedicated, and Managed payloads continue through the regular wrapping path.

Example: packed external blobs (single container file)

import io
import tarfile
from pathlib import Path
import lance
import pyarrow as pa
from lance import Blob, blob_array, blob_field

# Build a tar file with three payloads
payloads = {
    "a.bin": b"alpha",
    "b.bin": b"bravo",
    "c.bin": b"charlie",
}

with tarfile.open("container.tar", "w") as tf:
    for name, data in payloads.items():
        info = tarfile.TarInfo(name)
        info.size = len(data)
        tf.addfile(info, io.BytesIO(data))

# Capture offset/size for each member
blob_values = []
with tarfile.open("container.tar", "r") as tf:
    container_uri = Path("container.tar").resolve().as_uri()
    for name in payloads:
        m = tf.getmember(name)
        blob_values.append(Blob.from_uri(container_uri, position=m.offset_data, size=m.size))

schema = pa.schema([
    pa.field("name", pa.utf8()),
    blob_field("blob"),
])

rows = pa.table(
    {
        "name": list(payloads.keys()),
        "blob": blob_array(blob_values),
    },
    schema=schema,
)

ds = lance.write_dataset(
    rows,
    "./packed_blobs_v22.lance",
    data_storage_version="2.2",
    allow_external_blob_outside_bases=True,
)

Blob v2: Read Patterns

Choose the read API based on the payload shape you want:

API Returns Use When
read_blobs List[Tuple[int, Optional[bytes]]] You need complete blob payloads in memory, such as training loaders or batch preprocessing.
read_blob_ranges List[Tuple[int, int, Optional[bytes]]] You need selected byte ranges from multiple rows without materializing complete blobs.
take_blobs List[Optional[BlobFile]] You need file-like objects for streaming, seeking, or partial reads.
lance.read_blob_files / lance.read_blob_file_ranges List[Tuple[int, Optional[bytes]]] You already hold BlobFile handles and read the same blobs repeatedly, such as a shard index followed by chunk ranges.
scanner(..., blob_handling="all_binary") Arrow binary columns You want blob columns in a scan result or pyarrow.Table.

Do not wrap take_blobs in your own thread pool just to call read() or readall() on every blob. Use read_blobs instead; it plans and executes batched blob reads through Lance's scheduler.

Exactly one selector must be provided to read_blobs or take_blobs: ids, indices, or addresses. read_blob_ranges accepts the same selector kinds through its required selector argument.

Selector Typical Use Stability
indices Positional reads within one dataset snapshot Stable within that snapshot
ids Logical row-id based reads Stable logical identity (when row ids are available)
addresses Low-level physical reads and debugging Unstable physical location

Read complete payloads by row indices

import lance

ds = lance.dataset("./blobs_v22.lance")
rows = ds.read_blobs("blob", indices=[0, 1])
payloads = [payload for _row_address, payload in rows]

Read complete payloads by row ids

import lance

ds = lance.dataset("./blobs_v22.lance")
row_ids = ds.to_table(columns=[], with_row_id=True).column("_rowid").to_pylist()

rows = ds.read_blobs("blob", ids=row_ids[:2])

Read complete payloads by row addresses

import lance

ds = lance.dataset("./blobs_v22.lance")
row_addrs = ds.to_table(columns=[], with_row_address=True).column("_rowaddr").to_pylist()

rows = ds.read_blobs("blob", addresses=row_addrs[:2])

Blob selection APIs preserve logical result cardinality. read_blobs() and take_blobs() return one element per selected row, and read_blob_ranges() returns one element per request. A null blob is returned as None; a valid empty blob remains a non-null empty payload or zero-length BlobFile.

Read row-specific byte ranges

Use read_blob_ranges to read multiple blob-local ranges with one planned API call. Each request is a (row, offset, length) tuple, and selector determines whether every row is interpreted as a row ID, row address, or dataset index.

import lance

ds = lance.dataset("./blobs_v22.lance")
results = ds.read_blob_ranges(
    "blob",
    requests=[
        (7, 0, 1024),
        (7, 4096, 1024),
        (12, 0, 0),
    ],
    selector="indices",
)

for request_index, row_address, data in results:
    if data is None:
        # The selected blob is null.
        continue
    print(request_index, row_address, len(data))

Each result contains the zero-based request_index, the resolved physical row address, and the requested bytes. request_index identifies the original request when the same row appears more than once.

A request on a null blob returns None, including when its range is empty. An empty range on a non-null blob returns b"" without payload I/O. For every request, offset + length must fit in an unsigned 64-bit integer. A range on a non-null blob must not extend beyond its logical size; blob-local bounds are not evaluated for null blobs because they have no logical payload length.

Read through handles you already hold

read_blobs and read_blob_ranges look up the selected rows' blob descriptors on every call before they read any payload bytes. On object storage that lookup is a separate request the payload reads wait for. When you read the same blobs repeatedly, take handles once with take_blobs and pass them to read_blob_files or read_blob_file_ranges: each call then issues only the payload requests, planned and coalesced the same way.

import lance

ds = lance.dataset("./blobs_v22.lance")
handles = ds.take_blobs("blob", indices=[0, 1, 2])

payloads = [data for _request_index, data in lance.read_blob_files(handles)]
headers = lance.read_blob_file_ranges(
    [(handle, 0, 16) for handle in handles]
)

Each result is a (request_index, data) pair. A None handle, which take_blobs returns for a null blob, returns None; keep null handles in the request list so request_index still matches the original handle positions. Handles are not tied to a dataset receiver. Handles retain the object locations and byte ranges resolved when they were opened, including when passed alongside handles from other datasets or snapshots. Later dataset commits do not re-resolve those descriptors; cleanup must retain the referenced payloads. External objects follow their storage provider's consistency semantics.

The first non-null handle supplies default I/O parallelism and buffer size. Use the positive io_parallelism and io_buffer_size keyword arguments to override them. Empty and all-None inputs issue no storage requests.

Read blob columns as Arrow binary

import lance

ds = lance.dataset("./blobs_v22.lance")
table = ds.scanner(columns=["blob"], blob_handling="all_binary").to_table()
payloads = table.column("blob").to_pylist()

Open file-like blob handles lazily

Sequential reads are buffered, 512 KiB by default. Pass buffer_size=0 for unbuffered reads. read_range and read_ranges do not use this buffer.

import lance

ds = lance.dataset("./blobs_v22.lance")
blobs = ds.take_blobs("blob", indices=[0, 1])

blob = blobs[0]
if blob is not None:
    with blob as f:
        header = f.read(1024)
use std::sync::Arc;
use lance::Dataset;

let ds = Arc::new(Dataset::open("./blobs_v22.lance").await?);
let blobs = ds.take_blobs_by_indices(&[0, 1], "blob").await?;

if let Some(blob) = &blobs[0] {
    let header = blob.read_up_to(1024).await?;
}
import org.lance.BlobFile;
import org.lance.Dataset;
import java.util.List;

try (Dataset ds = Dataset.open("./blobs_v22.lance", allocator)) {
  List<BlobFile> blobs = ds.takeBlobsByIndices(List.of(0L, 1L), "blob");
  try {
    if (blobs.get(0) != null) {
      byte[] header = blobs.get(0).readUpTo(1024);
    }
  } finally {
    for (BlobFile blob : blobs) {
      if (blob != null) blob.close();
    }
  }
}

Example: decode video frames lazily

import av
import lance

ds = lance.dataset("./videos_v22.lance")
blob = ds.take_blobs("video", indices=[0])[0]
if blob is None:
    raise ValueError("video blob is null")

start_ms, end_ms = 500, 1000

with av.open(blob) as container:
    stream = container.streams.video[0]
    stream.codec_context.skip_frame = "NONKEY"

    start = (start_ms / 1000) / stream.time_base
    end = (end_ms / 1000) / stream.time_base
    container.seek(int(start), stream=stream)

    for frame in container.decode(stream):
        if frame.time is not None and frame.time > end_ms / 1000:
            break
        # process frame
        pass

Legacy Compatibility (data_storage_version <= 2.1)

If you need to keep writing legacy blob columns, use file format 0.1, 2.0, or 2.1 and mark LargeBinary fields with a metadata kwarg "lance-encoding:blob": true.

import lance
import pyarrow as pa

schema = pa.schema([
    pa.field("id", pa.int64()),
    pa.field(
        "video",
        pa.large_binary(),
        metadata={"lance-encoding:blob": "true"},
    ),
])

table = pa.table(
    {
        "id": [1, 2],
        "video": [b"foo", b"bar"],
    },
    schema=schema,
)

ds = lance.write_dataset(
    table,
    "./legacy_blob_dataset",
    data_storage_version="2.1",
)

With data_storage_version >= 2.2, dataset writers convert this input to Blob v2. For new datasets, prefer blob_field / blob_array with file format 2.2.

Rewrite to a New Blob v2 Dataset

To replace all historical legacy Blob encodings at once, you can rewrite them into a new dataset with data_storage_version="2.2". This is optional for 2.0 / 2.1 tables, which can instead append Blob v2 files as described above. File format 0.1 tables still require a rewrite to move into the 2.x family.

import lance
import pyarrow as pa
from lance import blob_array, blob_field

legacy = lance.dataset("./legacy_blob_dataset")
raw = legacy.scanner(columns=["id", "video"], blob_handling="all_binary").to_table()

new_schema = pa.schema([
    pa.field("id", pa.int64()),
    blob_field("video"),
])

rewritten = pa.table(
    {
        "id": raw.column("id"),
        "video": blob_array(raw.column("video").to_pylist()),
    },
    schema=new_schema,
)

lance.write_dataset(
    rewritten,
    "./blob_v22_dataset",
    data_storage_version="2.2",
)

Warning

  • The example above materializes binary payloads in memory (blob_handling="all_binary" and to_pylist()).
  • For large datasets, prefer chunked/batched rewrite pipelines.

When to Use a Blob Column vs. Inline Binary

Not every binary column needs to be a blob column. Plain Arrow binary/large_binary stores bytes inline, interleaved with your other columns, which is simplest and fastest for really small blobs (e.g., thumbnail images). Using a blob column to store the binary payload makes sense when either of these holds:

  • You need partial or streaming reads. Inline binary is always read in full; there is no way to fetch a byte range without materializing the entire value. Blob columns expose read_blob_ranges for planned row-specific range reads and take_blobs → BlobFile handles for caller-driven seeks, so you pay only for the bytes you touch.
  • Your values are large (roughly 1 MB or more on average). Operations that rewrite entire rows, such as compaction or some updates, must copy the large inline payloads forward into the new version — even when those bytes never changed. The bigger the payload, the more bytes you rewrite per logical change (write amplification). A blob column keeps large payloads in separate .blob files that are referenced rather than re-copied, so these operations don't rewrite the heavy bytes.

Tip

As a rule of thumb, if average payload size is below a few tens of KB and you only ever read whole values, plain inline binary is fine. Above ~1 MB, or any time you want file-like access, prefer a blob column. Blob v2 also tunes this automatically: by default it keeps payloads up to 64 KiB inline, packs payloads over 64 KiB and up to 4 MiB into shared .blob sidecars, and gives payloads over 4 MiB their own dedicated .blob file. These thresholds are configurable per column.

Troubleshooting

This section contains commonly noticed issues or errors, and explains how to address them.

Blob v2 requires file version >= 2.2

Cause: You are writing blob v2 values into a dataset/file format below 2.2.
Fix: Write to a dataset created with data_storage_version="2.2" (or newer).

Legacy blob columns ... are not supported for file version >= 2.2

Cause: A lower-level file writer received legacy blob metadata instead of prepared Blob v2 input. Fix: Use the dataset writer, which adapts legacy byte inputs, or prepare Blob v2 columns with the file-level Blob APIs.

Exactly one of ids, indices, or addresses must be specified

Cause: read_blobs or take_blobs received none or multiple selectors.
Fix: Provide exactly one of ids, indices, or addresses.