Read and Write Data¶

Lance datasets exchange Apache Arrow batches across the Python (pylance), Rust (lance), and Java (org.lance:lance-core) SDKs. The core read, write, mutation, and maintenance workflows below include examples for all three languages.

Writing Lance Dataset¶

Create a new Lance dataset from an Arrow table or record batch reader:

import lance
import pyarrow as pa

table = pa.Table.from_pylist([{"name": "Alice", "age": 20},
                              {"name": "Bob", "age": 30}])
ds = lance.write_dataset(table, "./alice_and_bob.lance")

use std::sync::Arc;
use arrow_array::{Int32Array, RecordBatch, RecordBatchIterator, StringArray};
use arrow_schema::{DataType, Field, Schema};
use lance::Dataset;

let schema = Arc::new(Schema::new(vec![
    Field::new("name", DataType::Utf8, false),
    Field::new("age", DataType::Int32, false),
]));
let batch = RecordBatch::try_new(
    schema.clone(),
    vec![
        Arc::new(StringArray::from(vec!["Alice", "Bob"])),
        Arc::new(Int32Array::from(vec![20, 30])),
    ],
)?;
let reader = RecordBatchIterator::new(vec![Ok(batch)], schema);
let ds = Dataset::write(reader, "./alice_and_bob.lance", None).await?;

import org.apache.arrow.memory.BufferAllocator;
import org.apache.arrow.memory.RootAllocator;
import org.apache.arrow.vector.ipc.ArrowReader;
import org.lance.Dataset;
import org.lance.WriteParams;

try (BufferAllocator allocator = new RootAllocator();
    Dataset ds =
        Dataset.write()
            .allocator(allocator)
            .reader(reader) // ArrowReader supplying (name: Utf8, age: Int32) batches
            .uri("./alice_and_bob.lance")
            .mode(WriteParams.WriteMode.CREATE)
            .execute()) {
  System.out.println("Rows written: " + ds.countRows());
}

If the dataset is too large to fully load into memory, lance.write_dataset also supports streaming an Iterator of pyarrow.RecordBatches. You will need to provide a pyarrow.Schema for the dataset in this case.

from typing import Iterator

def producer() -> Iterator[pa.RecordBatch]:
    """An iterator of RecordBatches."""
    yield pa.RecordBatch.from_pylist([{"name": "Alice", "age": 20}])
    yield pa.RecordBatch.from_pylist([{"name": "Bob", "age": 30}])

schema = pa.schema([
    ("name", pa.string()),
    ("age", pa.int32()),
])

ds = lance.write_dataset(producer(),
                         "./alice_and_bob.lance",
                         schema=schema, mode="overwrite")
print(ds.count_rows())  # Output: 2

lance.write_dataset supports writing pyarrow.Table, pandas.DataFrame, pyarrow.dataset.Dataset, and Iterator[pyarrow.RecordBatch].

Adding Rows¶

Append new batches to an existing dataset without rewriting existing fragments:

import lance
import pyarrow as pa

table = pa.Table.from_pylist([{"name": "Alice", "age": 20},
                              {"name": "Bob", "age": 30}])
ds = lance.write_dataset(table, "./insert_example.lance")

new_table = pa.Table.from_pylist([{"name": "Carla", "age": 37}])
ds.insert(new_table)

new_table2 = pa.Table.from_pylist([{"name": "David", "age": 42}])
ds = lance.write_dataset(new_table2, ds, mode="append")

use std::sync::Arc;
use arrow_array::{Int32Array, RecordBatch, RecordBatchIterator, StringArray};
use lance::Dataset;

let mut ds = Dataset::open("./alice_and_bob.lance").await?;
let arrow_schema = Arc::new(arrow_schema::Schema::from(ds.schema()));
let new_batch = RecordBatch::try_new(
    arrow_schema.clone(),
    vec![
        Arc::new(StringArray::from(vec!["Carla"])),
        Arc::new(Int32Array::from(vec![37])),
    ],
)?;
ds.append(RecordBatchIterator::new(vec![Ok(new_batch)], arrow_schema), None)
    .await?;

import org.lance.Dataset;
import org.lance.WriteParams;

try (Dataset ds =
    Dataset.write()
        .allocator(allocator)
        .reader(newBatchReader)
        .uri("./alice_and_bob.lance")
        .mode(WriteParams.WriteMode.APPEND)
        .execute()) {
  System.out.println("Total rows: " + ds.countRows());
}

Deleting rows¶

Lance supports deleting rows from a dataset using a SQL filter, as described in Filter push-down. For example, to delete Bob's row from the dataset above, one could use:

import lance

dataset = lance.dataset("./alice_and_bob.lance")
dataset.delete("name = 'Bob'")
print(dataset.to_table().to_pandas())
#     name  age
# 0  Alice   20

use lance::Dataset;

let mut dataset = Dataset::open("./alice_and_bob.lance").await?;
dataset.delete("name = 'Bob'").await?;

import org.lance.Dataset;

try (Dataset dataset = Dataset.open("./alice_and_bob.lance", allocator)) {
  dataset.delete("name = 'Bob'");
}

Updating rows¶

Lance supports updating rows based on SQL expressions. For example, if we notice that Bob's name in our dataset has been sometimes written as Blob, we can fix that with:

import lance

dataset = lance.dataset("./alice_and_bob.lance")
dataset.update({"name": "'Bob'"}, where="name = 'Blob'")

use std::sync::Arc;
use lance::dataset::UpdateBuilder;
use lance::Dataset;

let dataset = Arc::new(Dataset::open("./alice_and_bob.lance").await?);
let update_result = UpdateBuilder::new(dataset)
    .update_where("name = 'Blob'")?
    .set("name", "'Bob'")?
    .build()?
    .execute()
    .await?;
let dataset = update_result.new_dataset;

import org.lance.Dataset;
import org.lance.update.UpdateParams;
import org.lance.update.UpdateResult;
import java.util.Map;

UpdateResult result =
    dataset.update(new UpdateParams(Map.of("name", "'Bob'")).withWhere("name = 'Blob'"));
try (Dataset updated = result.getDataset()) {
  System.out.println("Updated rows: " + result.getNumRowsUpdated());
}

The update values are SQL expressions, which is why 'Bob' is wrapped in single quotes. This means we can use complex expressions that reference existing columns if we wish. For example, if two years have passed and we wish to update the ages of Alice and Bob in the same example, we could write:

import lance

dataset = lance.dataset("./alice_and_bob.lance")
dataset.update({"age": "age + 2"})

let update_result = UpdateBuilder::new(dataset)
    .set("age", "age + 2")?
    .build()?
    .execute()
    .await?;

UpdateResult result = dataset.update(new UpdateParams(Map.of("age", "age + 2")));
try (Dataset updated = result.getDataset()) {
  // updated reflects the new dataset version
}

If you are trying to update a set of individual rows with new values then it is often more efficient to use the merge insert operation described below.

import lance

# Change the ages of both Alice and Bob
new_table = pa.Table.from_pylist([{"name": "Alice", "age": 30},
                                  {"name": "Bob", "age": 20}])

# This works, but is inefficient, see below for a better approach
dataset = lance.dataset("./alice_and_bob.lance")
for idx in range(new_table.num_rows):
  name = new_table[0][idx].as_py()
  new_age = new_table[1][idx].as_py()
  dataset.update({"age": new_age}, where=f"name='{name}'")

Merge Insert¶

Lance supports a merge insert operation. This can be used to add new data in bulk while also (potentially) matching against existing data. This operation can be used for a number of different use cases.

Bulk Update¶

The lance.LanceDataset.update method is useful for updating rows based on a filter. However, if we want to replace existing rows with new rows then a lance.LanceDataset.merge_insert operation would be more efficient:

import lance

dataset = lance.dataset("./alice_and_bob.lance")
print(dataset.to_table().to_pandas())
#     name  age
# 0  Alice   20
# 1    Bob   30

# Change the ages of both Alice and Bob
new_table = pa.Table.from_pylist([{"name": "Alice", "age": 2},
                                  {"name": "Bob", "age": 3}])
# This will use `name` as the key for matching rows.  Merge insert
# uses a JOIN internally and so you typically want this column to
# be a unique key or id of some kind.
rst = dataset.merge_insert("name") \
       .when_matched_update_all() \
       .execute(new_table)
print(dataset.to_table().to_pandas())
#     name  age
# 0  Alice    2
# 1    Bob    3

Note that, similar to the update operation, rows that are modified will be removed and inserted back into the table, changing their position to the end. Also, the relative order of these rows could change because we are using a hash-join operation internally.

Insert if not Exists¶

Sometimes we only want to insert data if we haven't already inserted it before. This can happen, for example, when we have a batch of data but we don't know which rows we've added previously and we don't want to create duplicate rows. We can use the merge insert operation to achieve this:

# Bob is already in the table, but Carla is new
new_table = pa.Table.from_pylist([{"name": "Bob", "age": 30},
                                  {"name": "Carla", "age": 37}])

dataset = lance.dataset("./alice_and_bob.lance")

# This will insert Carla but leave Bob unchanged
_ = dataset.merge_insert("name") \
       .when_not_matched_insert_all() \
       .execute(new_table)
# Verify that Carla was added but Bob remains unchanged
print(dataset.to_table().to_pandas())
#     name  age
# 0  Alice   20
# 1    Bob   30
# 2  Carla   37

Update or Insert (Upsert)¶

Sometimes we want to combine both of the above behaviors. If a row already exists we want to update it. If the row does not exist we want to add it. This operation is sometimes called "upsert". We can use the merge insert operation to do this as well:

import lance
import pyarrow as pa

# Change Carla's age and insert David
new_table = pa.Table.from_pylist([{"name": "Carla", "age": 27},
                                  {"name": "David", "age": 42}])

dataset = lance.dataset("./alice_and_bob.lance")

# This will update Carla and insert David
_ = dataset.merge_insert("name") \
       .when_matched_update_all() \
       .when_not_matched_insert_all() \
       .execute(new_table)
# Verify the results
print(dataset.to_table().to_pandas())
#     name  age
# 0  Alice   20
# 1    Bob   30
# 2  Carla   27
# 3  David   42

Replace a Portion of Data¶

A less common, but still useful, behavior can be to replace some region of existing rows (defined by a filter) with new data. This is similar to performing both a delete and an insert in a single transaction. For example:

import lance
import pyarrow as pa

new_table = pa.Table.from_pylist([{"name": "Edgar", "age": 46},
                                  {"name": "Francene", "age": 44}])

dataset = lance.dataset("./alice_and_bob.lance")
print(dataset.to_table().to_pandas())
#       name  age
# 0    Alice   20
# 1      Bob   30
# 2  Charlie   45
# 3    Donna   50

# This will remove anyone above 40 and insert our new data
_ = dataset.merge_insert("name") \
       .when_not_matched_insert_all() \
       .when_not_matched_by_source_delete("age >= 40") \
       .execute(new_table)
# Verify the results - people over 40 replaced with new data
print(dataset.to_table().to_pandas())
#        name  age
# 0     Alice   20
# 1       Bob   30
# 2     Edgar   46
# 3  Francene   44

Reading Lance Dataset¶

Open an existing Lance dataset from a local path or cloud URI:

import lance

ds = lance.dataset("s3://bucket/path/imagenet.lance")
# Or local path
ds = lance.dataset("./imagenet.lance")

use lance::Dataset;

let ds = Dataset::open("s3://bucket/path/imagenet.lance").await?;
// Or local path
let ds = Dataset::open("./imagenet.lance").await?;

import org.lance.Dataset;

try (Dataset ds = Dataset.open("./imagenet.lance", allocator)) {
  // read from ds
}

The most straightforward approach for reading a Lance dataset is to utilize the lance.LanceDataset.to_table method in order to load the entire dataset into memory.

table = ds.to_table()

Due to Lance being a high-performance columnar format, it enables efficient reading of subsets of the dataset by utilizing Column (projection) push-down and filter (predicates) push-downs.

table = ds.to_table(
    columns=["image", "label"],
    filter="label = 2 AND text IS NOT NULL",
    limit=1000,
    offset=3000,
)

use arrow_array::RecordBatch;
use futures::TryStreamExt;

let batches: Vec<RecordBatch> = ds
    .scan()
    .project(&["image", "label"])?
    .filter("label = 2 AND text IS NOT NULL")?
    .limit(Some(1000), Some(3000))?
    .try_into_stream()
    .await?
    .try_collect()
    .await?;

import org.apache.arrow.vector.ipc.ArrowReader;
import org.lance.ipc.LanceScanner;
import org.lance.ipc.ScanOptions;
import java.util.Arrays;

ScanOptions options =
    new ScanOptions.Builder()
        .columns(Arrays.asList("image", "label"))
        .filter("label = 2 AND text IS NOT NULL")
        .limit(1000)
        .offset(3000)
        .build();
try (LanceScanner scanner = ds.newScan(options);
    ArrowReader reader = scanner.scanBatches()) {
  while (reader.loadNextBatch()) {
    // process reader.getVectorSchemaRoot()
  }
}

Lance understands the cost of reading heavy columns such as image. Consequently, it employs an optimized query plan to execute the operation efficiently.

Iterative Read¶

If the dataset is too large to fit in memory, you can read it in batches using the lance.LanceDataset.to_batches method:

for batch in ds.to_batches(columns=["image"], filter="label = 10"):
    # do something with batch
    compute_on_batch(batch)

Unsurprisingly, lance.LanceDataset.to_batches takes the same parameters as lance.LanceDataset.to_table function.

Filter push-down¶

Lance embraces the utilization of standard SQL expressions as predicates for dataset filtering. By pushing down the SQL predicates directly to the storage system, the overall I/O load during a scan is significantly reduced.

Currently, Lance supports a growing list of expressions.

• >, >=, <, <=, =
• AND, OR, NOT
• IS NULL, IS NOT NULL
• IS TRUE, IS NOT TRUE, IS FALSE, IS NOT FALSE
• IN
• LIKE, NOT LIKE
• regexp_match(column, pattern)
• any_element(list_column, 'predicate') (experimental), see below
• CAST

For example, the following filter string is acceptable:

((label IN [10, 20]) AND (note['email'] IS NOT NULL))
    OR NOT note['created']

Nested fields can be accessed using the subscripts. Struct fields can be subscripted using field names, while list fields can be subscripted using indices.

If your column name contains special characters or is a SQL Keyword, you can use backtick (`) to escape it. For nested fields, each segment of the path must be wrapped in backticks.

`CUBE` = 10 AND `column name with space` IS NOT NULL
  AND `nested with space`.`inner with space` < 2

Literals for dates, timestamps, and decimals can be written by writing the string value after the type name. For example

date_col = date '2021-01-01'
and timestamp_col = timestamp '2021-01-01 00:00:00'
and decimal_col = decimal(8,3) '1.000'

A plain number in a comparison, BETWEEN or IN with a decimal column, such as decimal_col < 0.055, is compared exactly with the digits as written; it is not rounded to the column's scale or converted to a float. A number whose digits as written do not fit a 256-bit decimal, such as 1e-80 or 1e200, is not supported.

For timestamp columns, the precision can be specified as a number in the type parameter. Microsecond precision (6) is the default.

Lance internally stores data in Arrow format. The mapping from SQL types to Arrow is:

(1) See precision mapping in previous table.

Filtering on elements of a list of structs¶

any_element(list_column, 'predicate') keeps a row when at least one element of a list<struct> column satisfies the predicate, with every condition evaluated on that same element. For a column detections: list<struct<label: string, confidence: float>>:

any_element(detections, 'label = ''car'' AND confidence > 0.9')

matches rows with a car detected above 0.9 confidence, and not rows that have a car below 0.9 next to another object above it.

The predicate is a string literal that refers to the element's fields by name, and accepts the same expressions as any other filter. Because it is a string, single quotes inside it are doubled. It cannot refer to columns outside the element; combine those with AND or OR outside the call. It may only call immutable functions, so functions such as now() and random() are rejected.

The result follows SQL three-valued logic: NULL for a null list, TRUE if the predicate is TRUE on some element, otherwise NULL if it is NULL on some element, otherwise FALSE. An empty list is FALSE.

The same filter can be written as a lambda, which needs no quoting:

array_any_match(detections, d -> d.label = 'car' AND d.confidence > 0.9)

Lance rewrites it to the any_element call above, so the two behave the same. any_match is an alias, and fields can also be written d['label']. The lambda has one parameter, and its body may only refer to that parameter's fields. Other lambdas, such as in array_filter, are not supported in filters.

A listelement scalar index on the list column answers any_element filters from the index. Its keys name the element fields it indexes, in order:

from lance.indices import IndexConfig

dataset.create_scalar_index(
    "detections",
    IndexConfig(index_type="listelement", parameters={"keys": ["label", "confidence"]}),
)

The index is used when the predicate fixes the first key with =, IN or IS NULL. Conditions on further keys narrow the search: equalities on a prefix of the keys, then one range on the next key. When the predicate has conditions the index cannot answer, such as one on a field that is not a key, the rows the index finds are rechecked with any_element itself, so the index never changes the result. Keys may be booleans, integers, floats, dates, timestamps, strings or binaries. The index follows its key fields through renames. If a key field is dropped or changes type, queries stop using the index and scan instead; drop and recreate it.

Random read¶

One distinct feature of Lance, as columnar format, is that it allows you to read random samples quickly.

# Access the 2nd, 101st, and 501st rows
data = ds.take([1, 100, 500], columns=["image", "label"])

use lance::dataset::ProjectionRequest;

// Access the 2nd, 101st, and 501st rows
let batch = ds
    .take(
        &[1, 100, 500],
        ProjectionRequest::from_columns(["image", "label"], ds.schema()),
    )
    .await?;

import org.apache.arrow.vector.ipc.ArrowReader;
import java.util.Arrays;

// Access the 2nd, 101st, and 501st rows
try (ArrowReader reader =
    ds.take(Arrays.asList(1L, 100L, 500L), Arrays.asList("image", "label"))) {
  while (reader.loadNextBatch()) {
    // process reader.getVectorSchemaRoot()
  }
}

The ability to achieve fast random access to individual rows plays a crucial role in facilitating various workflows such as random sampling and shuffling in ML training. Additionally, it empowers users to construct secondary indices, enabling swift execution of queries for enhanced performance.

Table Maintenance¶

Some operations over time will cause a Lance dataset to have a poor layout. For example, many small appends will lead to a large number of small fragments. Or deleting many rows will lead to slower queries due to the need to filter out deleted rows.

To address this, Lance provides methods for optimizing dataset layout.

Compact data files¶

Data files can be rewritten so there are fewer files. When passing a target_rows_per_fragment to dataset.optimize.compact_files(...), Lance will skip any fragments that are already above that row count, and rewrite others. Fragments will be merged according to their fragment ids, so the inherent ordering of the data will be preserved.

import lance

dataset = lance.dataset("./alice_and_bob.lance")
dataset.optimize.compact_files(target_rows_per_fragment=1024 * 1024)

use lance::dataset::optimize::{compact_files, CompactionOptions};
use lance::Dataset;

let mut dataset = Dataset::open("./alice_and_bob.lance").await?;
compact_files(
    &mut dataset,
    CompactionOptions {
        target_rows_per_fragment: 1024 * 1024,
        ..Default::default()
    },
    None,
)
.await?;

import org.lance.Dataset;
import org.lance.compaction.CompactionOptions;

try (Dataset dataset = Dataset.open("./alice_and_bob.lance", allocator)) {
  dataset.compact(
      CompactionOptions.builder()
          .withTargetRowsPerFragment(1024 * 1024L)
          .build());
}

During compaction, Lance can also remove deleted rows. Rewritten fragments will not have deletion files. This can improve scan performance since the soft deleted rows don't have to be skipped during the scan.

When files are rewritten, the physical row addresses (_rowaddr) of the compacted rows change. Run compact_files() before optimize_indices() or index rebuilds so Lance can remap or update secondary indices against the compacted fragments (see Compaction and Rebuilding). If your application needs logical row identifiers that remain constant across updates and compaction, enable Stable Row IDs.

Cleanup old versions¶

Lance is an immutable format — every write creates a new version. The new version only writes the data that changed, so an insert writes the new rows and an update rewrites the affected columns for the affected rows. Even a delete creates a small deletion file. However, old versions still reference the previous data files, so those files are kept on disk until explicitly removed. Over time this means storage grows with each operation — inserts, updates, and deletes alike.

Keeping old versions has important benefits: readers that opened an older version can continue reading it without interference from concurrent writers, providing snapshot isolation. Old versions also enable time travel queries, letting you read snapshots from the retained version history.

cleanup_old_versions deletes old version metadata and any data files that are no longer referenced by any version, reclaiming the accumulated storage.

import lance

dataset = lance.dataset("./my_dataset.lance")
dataset.cleanup_old_versions()

By default, versions older than two weeks are removed. You can override this with the older_than parameter (a timedelta):

from datetime import timedelta

dataset.cleanup_old_versions(older_than=timedelta(days=1))

For multiple retention windows, pass a CleanupPolicy containing ordered CleanupRule entries. This example keeps full recent history, hourly samples between two and seven days, daily samples between seven and thirty days, and expires everything older than thirty days:

from lance import CleanupPolicy, KeepOne, RemoveAll

dataset.cleanup_old_versions(
    policy=CleanupPolicy(rules=[
        KeepOne(per_interval=timedelta(hours=1), older_than=timedelta(days=2)),
        KeepOne(per_interval=timedelta(days=1), older_than=timedelta(days=7)),
        RemoveAll(older_than=timedelta(days=30)),
    ])
)

KeepOne and RemoveAll implement CleanupRule. A CleanupPolicy contains the ordered rules; CleanupPolicy() defaults to a single RemoveAll(older_than=timedelta(days=14)), matching existing cleanup behavior. Ages must strictly increase. Each version uses the oldest matching KeepOne rule, with separate buckets for each band. RemoveAll is optional and must be last; without it, sampled history has no expiration limit. Empty policies, duplicate thresholds and unsupported rules are rejected. Comparisons are strictly older-than: a version exactly at a threshold stays in the younger band.

KeepOne.older_than must be at least two days. Sampling ages are relative to the hinted commit timestamp, falling back to this dataset's commit when no hint exists. The hinted version and every later version are protected from thinning, even with backwards clocks or LANCE_USE_VERSION_HINT=0 in the cleaner. RemoveAll.older_than is a wall-clock expiration age and expires sampled versions too. Each sampling interval aligns to the UTC Unix epoch and selects the newest eligible commit timestamp, breaking ties by version number.

Tags, current versions and branch dependencies remain protected. versions and retain_versions limit which versions either operation can remove. Cleanup also preserves the data, indices and blobs needed by retained snapshots. The same policy is accepted by explain_cleanup_old_versions for a read-only preview.

Retention rules work with both values of delete_unverified. By default, cleanup preserves recent files that no manifest references. With delete_unverified=True, it can remove those files immediately; files needed by retained snapshots remain protected in either mode.

The existing top-level older_than keeps its wall-clock expiration meaning. Existing cleanup calls and defaults continue to work. The single-window shorthand older_than=timedelta(days=7), keep_one=KeepOne(timedelta(days=1), timedelta(days=2)) is also supported. Do not combine policy with older_than or keep_one.

Rust uses cleanup::retention::CleanupPolicy::new(Vec<CleanupRule>) through .with_retention_policy(...); the legacy cleanup::CleanupPolicy remains available for version and file-cleanup filters. Java accepts dataset.cleanup(new CleanupPolicy(rules)), using CleanupRule.keepOne(perInterval, olderThan) and CleanupRule.removeAll(olderThan). Existing native policy filters still apply.

For automatic cleanup, use the same rules with duration strings:

import json

dataset.update_config({
    "lance.auto_cleanup.interval": None,
    "lance.auto_cleanup.older_than": None,
    "lance.auto_cleanup.policy_interval": "100",
    "lance.auto_cleanup.policy": json.dumps({"rules": [
        {"type": "keep_one", "per_interval": "1h", "older_than": "2days"},
        {"type": "keep_one", "per_interval": "1day", "older_than": "7days"},
        {"type": "remove_all", "older_than": "30days"},
    ]}),
})

The policy interval is a version count. Older clients ignore this activation key and leave automatic retention cleanup disabled. The legacy interval must be removed atomically when activating either policy form; unsafe combinations are rejected before publication. Do not combine this policy with lance.auto_cleanup.older_than or the single-window lance.auto_cleanup.keep_one.* settings.

Thinning also supports local filesystems and S3 Express. The hint-relative two-day minimum provides a retention buffer for delayed hint writes; it does not make hint updates monotonic. On stores using hints, unreadable or malformed hints fail cleanup instead of using the wall clock. If explicit expiration has removed an older hinted manifest, cleanup falls back to the dataset snapshot, as latest version discovery does when a hint target is missing. A missing current manifest still fails cleanup.

By default, Lance only removes files that it can verify are no longer needed. A file is verified when Lance can see that it was referenced by an older version and is no longer referenced by any newer version. However, some orphaned files cannot be verified this way — for example, files left behind by aborted or failed commits that were never recorded in any version. These files are indistinguishable from files being written by an in-progress operation, so their age is the only evidence that no operation still owns them: by default Lance removes an unverifiable file only if it is at least 7 days old.

Cleanup will never delete the current (active) version. This means passing older_than=timedelta(0) is safe and will delete all versions except the current one.

The delete_unverified flag enables a more aggressive strategy that will also delete unverified files regardless of their age:

dataset.cleanup_old_versions(
    older_than=timedelta(hours=2),
    delete_unverified=True,
)

Control file deletion throughput with two independent options:

• delete_concurrency: maximum concurrent file deletes. If omitted, uses the object store's I/O parallelism, configurable via the LANCE_IO_THREADS environment variable.
• delete_rate_limit: maximum delete requests per second. Unlimited when omitted.

Both values must be positive integers.

dataset.cleanup_old_versions(delete_concurrency=32, delete_rate_limit=100)

Automatic cleanup¶

Instead of calling cleanup_old_versions manually, you can configure Lance to clean up old versions automatically during writes. When auto cleanup is enabled, Lance will run cleanup every N commits (the interval), removing versions older than a specified duration.

Auto cleanup can be enabled when creating a new dataset:

import lance
import pyarrow as pa
from lance import AutoCleanupConfig

table = pa.table({"id": range(100)})
ds = lance.write_dataset(
    table,
    "./my_dataset.lance",
    auto_cleanup_options=AutoCleanupConfig(
        interval=20,             # run cleanup every 20 commits
        older_than_seconds=3600, # remove versions older than 1 hour
    ),
)

Or enabled on an existing dataset:

ds = lance.dataset("./my_dataset.lance")
ds.optimize.enable_auto_cleanup(
    AutoCleanupConfig(
        interval=20,
        older_than_seconds=3600,
    )
)

And disabled again:

ds.optimize.disable_auto_cleanup()

Auto cleanup parameters can also be set directly via dataset config keys. See the Table Config Specification for the full list of lance.auto_cleanup.* keys:

ds.update_config({
    "lance.auto_cleanup.interval": "20",
    "lance.auto_cleanup.older_than": "3600s",
    "lance.auto_cleanup.delete_concurrency": "32",
    "lance.auto_cleanup.delete_rate_limit": "100",
})

Other cleanup strategies¶

It is common to run cleanup as a periodic background task on a dedicated server (for example, via a cron job or scheduled workflow). This keeps cleanup off the write path entirely, avoiding any impact to write latency, but requires setting up and maintaining additional infrastructure.

Choosing a data file version¶

data_storage_version selects the format of newly written data files. For an existing V2 dataset, an operation can select "2.0", "2.1", "2.2", or "2.3" without rewriting the other files. The dataset's data_storage_version property is the default for writes that omit a target, not a summary of its existing files. Create and overwrite establish this default; append, update, merge-insert, and compaction do not change it. V1 and V2 cannot be mixed.

import lance
import pyarrow as pa

data = pa.table({"id": [1, 2], "value": [10, 20]})
ds = lance.write_dataset(data, "./versions.lance", data_storage_version="2.1")
ds.update({"value": "value + 1"}, data_storage_version="2.2")
assert ds.data_storage_version == "2.1"

ds.merge_insert("id").when_not_matched_insert_all().data_storage_version("2.2").execute(
    pa.table({"id": [3], "value": [30]})
)
ds.optimize.compact_files(data_storage_version="2.2")

The "stable" and "next" selectors resolve according to the engine release. Use exact versions when the output identity must be independent of that release. V2.3 is currently unstable: files written by one unstable revision may not be readable by a later revision. Use it only for experimentation. Compaction plans fix the target before distributing tasks, including when the target comes from the dataset default. The target survives Python pickle and Java serialization; workers do not reinterpret it using their own release defaults.

Java update, merge-insert, and compaction options use withDataStorageVersion(DataStorageVersion.V2_2). The shared DataStorageVersion enum also provides STABLE and NEXT selectors.

Compaction can convert selected fragments to another supported V2 version. Binary copy requires every selected file to match the output version, as well as the usual eligibility checks. In particular, overlays need reencoding to preserve updated values. try_binary_copy falls back to reencoding when inputs are ineligible; force_binary_copy rejects them. A version mismatch error includes the target version, actual version, and file path. A persistent compaction target can be set through lance.compaction.data_storage_version in the table config; an explicit operation target takes precedence.

Upgrading clients before mixed-version writes¶

Before writing files that differ from the dataset default, upgrade every reader and writer to a mixed-version-aware release. Drain, restart, or fence writers that opened the dataset using an older release. The commit automatically sets the paired mixed-version reader/writer feature flags when needed; there is no separate activation API. These flags remain set even if compaction later makes the files homogeneous again.

Older clients that do not recognize the flags reject the resulting snapshots. The flags cannot retroactively fence an older writer that already read a prior manifest. Historical homogeneous datasets remain readable by clients that support their file versions; see table versioning for the feature flag contract.