Versioning Your Datasets with Lance¶

Lance supports versioning natively, allowing you to track changes over time.

In this tutorial, you'll learn how to append new data to existing datasets while preserving historical versions and access specific versions using version numbers or meaningful tags. You'll also understand how to implement proper data governance practices with Lance's native versioning capabilities.

Install the Python SDK¶

pip install pylance pandas pyarrow numpy tqdm

Set Up Your Environment¶

First, you should import the necessary libraries:

import shutil

import lance
import pandas as pd
import pyarrow as pa

Create a Dataset¶

Start with a fresh dataset, which becomes version 1:

shutil.rmtree("/tmp/versioning.lance", ignore_errors=True)

df = pd.DataFrame({"a": [5]})
dataset = lance.write_dataset(df, "/tmp/versioning.lance")

Append New Data to Your Dataset¶

You can add new rows to your existing dataset, creating a new version while preserving the original data. Here is how to append rows:

df = pd.DataFrame({"a": [10]})
tbl = pa.Table.from_pandas(df)
dataset = lance.write_dataset(tbl, "/tmp/versioning.lance", mode="append")

dataset.to_table().to_pandas()

    a
0   5
1  10

Overwrite Your Dataset¶

You can completely replace your dataset with new data, creating a new version while keeping the old version accessible.

Here is how to overwrite the data and create a new version:

df = pd.DataFrame({"a": [50, 100]})
tbl = pa.Table.from_pandas(df)
dataset = lance.write_dataset(tbl, "/tmp/versioning.lance", mode="overwrite")

dataset.to_table().to_pandas()

     a
0   50
1  100

Access Previous Dataset Versions¶

You can also check what versions are available and then access specific versions of your dataset.

List all versions of a dataset with versions() (each entry includes version, timestamp, and metadata):

[v["version"] for v in dataset.versions()]

[1, 2, 3]

If you only need version numbers, use the lightweight reference API. It lists manifest locations without reading and deserializing every manifest:

dataset.version_refs()

[{'version': 1}, {'version': 2}, {'version': 3}]

Use dataset.latest_version instead when only the latest version of the current branch is needed.

You can also access any available version:

# Version 1
print(lance.dataset("/tmp/versioning.lance", version=1).to_table().to_pandas())

# Version 2
print(lance.dataset("/tmp/versioning.lance", version=2).to_table().to_pandas())

   a
0  5
    a
0   5
1  10

Tag Your Important Versions¶

Create named tags for important versions, making it easier to reference them by meaningful names.

dataset.tags.create("stable", 2)
dataset.tags.create("nightly", 3)
dict(sorted((name, info["version"]) for name, info in dataset.tags.list().items()))

{'nightly': 3, 'stable': 2}

Tags can be checked out like versions:

lance.dataset("/tmp/versioning.lance", version="stable").to_table().to_pandas()

    a
0   5
1  10

For advanced tag operations (e.g., tagging versions on specific branches), see Tags and Branches.

Work with Branches¶

Branches manage parallel lines of dataset evolution. You can create branches from existing versions or tags, read and write to them independently, and checkout different branches.

main refers to the default branch in checkout and reference APIs, but it is reserved and cannot be used as the name of a new branch.

# Create a branch from the current latest version
experiment_branch = dataset.create_branch("experiment")

# Write to the branch (affects only that branch's history)
tbl = pa.Table.from_pandas(pd.DataFrame({"a": [42]}))
experiment_branch = lance.write_dataset(tbl, experiment_branch, mode="append")
experiment_branch.to_table().to_pandas()

     a
0   50
1  100
2   42

For more details, see Tags and Branches. To preserve logical row identity across updates and compaction, see Stable Row IDs and Lineage.

Next Steps¶

Now that you've mastered dataset versioning with Lance, check out Vector Indexing and Vector Search With Lance. You can learn how to build high-performance vector search capabilities on top of your Lance tables.

This will teach you how to build fast, scalable search capabilities for your versioned datasets.