Object Store Configuration¶

Lance supports object stores such as AWS S3 (and compatible stores), Azure Blob Store, and Google Cloud Storage. Which object store to use is determined by the URI scheme of the dataset path. For example, s3://bucket/path will use S3, az://bucket/path will use Azure, and gs://bucket/path will use GCS.

These object stores take additional configuration objects. There are two ways to specify these configurations: by setting environment variables or by passing them to the storage_options parameter of lance.dataset and lance.write_dataset. So for example, to globally set a higher timeout, you would run in your shell:

export TIMEOUT=60s

If you only want to set the timeout for a single dataset, you can pass it as a storage option:

import lance
ds = lance.dataset("s3://path", storage_options={"timeout": "60s"})

General Configuration¶

These options apply to all object stores.

Per-Base Configuration¶

A dataset can register additional base paths that store part of its data, and each base may live in a different bucket, account, or storage provider. A storage option key of the form base_<id>.<key> applies <key> only to the base path with that manifest id. Every base inherits the unscoped options; base-scoped entries add to or override them for that base only.

import lance
ds = lance.dataset(
    "az://account-a/path",
    storage_options={
        # Shared defaults, used by the primary dataset and inherited by bases
        "account_name": "account-a",
        "account_key": "key-a",
        # Overrides for the base path with id 1
        "base_1.account_name": "account-b",
        "base_1.account_key": "key-b",
    },
)

Base ids are assigned when bases are registered (initial_bases ids are assigned sequentially starting at 1, in order) and can be inspected through the manifest base paths. Keys that do not match base_<id>.<key> exactly (e.g. base_url) are treated as regular storage options. Exact per-base parameter maps (base_store_params, keyed by base path URI) take precedence over base-scoped keys for that base.

S3 Configuration¶

S3 (and S3-compatible stores) have additional configuration options that configure authorization and S3-specific features (such as server-side encryption).

AWS credentials can be set in the environment variables AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, and AWS_SESSION_TOKEN. Alternatively, they can be passed as parameters to the storage_options parameter:

import lance
ds = lance.dataset(
    "s3://bucket/path",
    storage_options={
        "access_key_id": "my-access-key",
        "secret_access_key": "my-secret-key",
        "session_token": "my-session-token",
    }
)

If you are using AWS SSO, you can specify the AWS_PROFILE environment variable. It cannot be specified in the storage_options parameter.

The following keys can be used as both environment variables or keys in the storage_options parameter:

Credential provider selection¶

By default, Lance uses the standard AWS credential provider chain (environment variables, shared config file, web identity tokens, ECS, EC2 instance metadata).

The aws_provider_scheme storage option pins a dataset to a specific credential provider, which is useful when two datasets in the same process need different AWS auth (for example, one bucket using IRSA and another using ECS container credentials).

import lance

# Bucket A — use IRSA (web identity token from the environment)
ds_a = lance.dataset(
    "s3://bucket-a/path",
    storage_options={"aws_provider_scheme": "irsa"},
)

# Bucket B — use ECS container credentials
ds_b = lance.dataset(
    "s3://bucket-b/path",
    storage_options={"aws_provider_scheme": "ecs"},
)

S3-compatible stores¶

Lance can also connect to S3-compatible stores, such as MinIO. To do so, you must specify both region and endpoint:

import lance
ds = lance.dataset(
    "s3://bucket/path",
    storage_options={
        "region": "us-east-1",
        "endpoint": "http://minio:9000",
    }
)

This can also be done with the AWS_ENDPOINT and AWS_DEFAULT_REGION environment variables.

S3 Express (Directory Bucket)¶

Lance supports S3 Express One Zone buckets, a.k.a. S3 directory buckets. S3 Express buckets only support connecting from an EC2 instance within the same region. By default, Lance automatically recognize the --x-s3 suffix of an express bucket, there is no special configuration needed.

In case of an access point or private link that hides the bucket name, you can configure express bucket access explicitly through storage option s3_express.

import lance
ds = lance.dataset(
    "s3://my-bucket--use1-az4--x-s3/path/imagenet.lance",
    storage_options={
        "region": "us-east-1",
        "s3_express": "true",
    }
)

Google Cloud Storage Configuration¶

GCS credentials are configured by setting the GOOGLE_SERVICE_ACCOUNT environment variable to the path of a JSON file containing the service account credentials. Alternatively, you can pass the path to the JSON file in the storage_options

import lance
ds = lance.dataset(
    "gs://my-bucket/my-dataset",
    storage_options={
        "service_account": "path/to/service-account.json",
    }
)

The following keys can be used as both environment variables or keys in the storage_options parameter:

Azure Blob Storage Configuration¶

Azure Blob Storage credentials can be configured by setting the AZURE_STORAGE_ACCOUNT_NAME and AZURE_STORAGE_ACCOUNT_KEY environment variables. Alternatively, you can pass the account name and key in the storage_options parameter:

import lance
ds = lance.dataset(
    "az://my-container/my-dataset",
    storage_options={
        "account_name": "some-account",
        "account_key": "some-key",
    }
)

These keys can be used as both environment variables or keys in the storage_options parameter:

AliCloud Object Storage Service Configuration¶

OSS credentials can be set in the environment variables OSS_ACCESS_KEY_ID, OSS_ACCESS_KEY_SECRET, OSS_REGION, and OSS_SECURITY_TOKEN. Alternatively, they can be passed as parameters to the storage_options parameter:

import lance
ds = lance.dataset(
    "oss://bucket/path",
    storage_options={
        "oss_region": "oss-region",
        "oss_endpoint": "oss-endpoint",
        "oss_access_key_id": "my-access-key",
        "oss_secret_access_key": "my-secret-key",
        "oss_security_token": "my-session-token",
    }
)

Volcengine TOS Configuration¶

TOS credentials can be set in the environment variables TOS_ACCESS_KEY_ID, TOS_SECRET_ACCESS_KEY, TOS_ENDPOINT, TOS_REGION, and TOS_SECURITY_TOKEN. Lance also accepts the corresponding VOLCENGINE_ environment variable prefix. Alternatively, credentials can be passed as parameters to the storage_options parameter; explicit storage_options override environment variables:

import lance
ds = lance.dataset(
    "tos://bucket/path",
    storage_options={
        "tos_endpoint": "https://tos-cn-beijing.volces.com",
        "tos_region": "cn-beijing",
        "tos_access_key_id": "my-access-key",
        "tos_secret_access_key": "my-secret-key",
        "tos_security_token": "my-session-token",
    }
)

Tencent Cloud COS Configuration¶

COS (Cloud Object Storage) credentials can be set in environment variables prefixed with COS_ or TENCENTCLOUD_ (for example, COS_ENDPOINT, COS_SECRET_ID, COS_SECRET_KEY, TENCENTCLOUD_REGION, TENCENTCLOUD_SECURITY_TOKEN). Alternatively, credentials can be passed as parameters to the storage_options parameter; explicit storage_options override environment variables:

import lance
ds = lance.dataset(
    "cos://bucket/path",
    storage_options={
        "cos_endpoint": "https://cos.ap-guangzhou.myqcloud.com",
        "cos_secret_id": "my-secret-id",
        "cos_secret_key": "my-secret-key",
    }
)

[dependencies]
lance = "*"

GooseFS Configuration¶

GooseFS is a distributed caching filesystem. Lance accesses GooseFS through its Master gRPC service. The URL format is goosefs://host:port/path, where host:port is the GooseFS Master address (default port: 9200, may be omitted, e.g. goosefs://10.0.0.1/path) and /path is the filesystem path within GooseFS.

Manifest commits on goosefs:// use ConditionalPutCommitHandler (PutMode::Create / if-not-exists), backed by GooseFS master's atomic no-replace rename so concurrent writers cannot clobber each other's versioned manifests.

import lance

ds = lance.dataset(
    "goosefs://10.0.0.1:9200/data/my-dataset.lance",
    storage_options={
        "goosefs_auth_type": "simple",
        "goosefs_auth_username": "lance",
    },
)

[dependencies]
lance = "*"

use lance_io::object_store::ObjectStore;

let uri = "goosefs://10.0.0.1:9200/lance-test/lance-io";
let (store, path) = ObjectStore::from_uri(uri).await?;

// Read / write through the underlying `object_store::ObjectStore` API
store.inner.put(&path, (&b"hello"[..]).into()).await?;
let result = store.inner.get(&path).await?;
let bytes = result.bytes().await?;

use std::collections::HashMap;
use lance::dataset::DatasetBuilder;

let mut storage_options = HashMap::new();
storage_options.insert("goosefs_master_addr".to_string(), "10.0.0.1:9200".to_string());
storage_options.insert("goosefs_auth_type".to_string(), "simple".to_string());
storage_options.insert("goosefs_auth_username".to_string(), "lance".to_string());

let dataset = DatasetBuilder::from_uri("goosefs://10.0.0.1:9200/data/my-dataset.lance")
    .with_storage_options(storage_options)
    .load()
    .await?;

import org.lance.Dataset;
import org.lance.ReadOptions;

import java.util.HashMap;
import java.util.Map;

Map<String, String> storageOptions = new HashMap<>();
storageOptions.put("goosefs_master_addr", "10.0.0.1:9200");
storageOptions.put("goosefs_auth_type", "simple");
storageOptions.put("goosefs_auth_username", "lance");

ReadOptions options = new ReadOptions.Builder()
    .setStorageOptions(storageOptions)
    .build();

try (Dataset dataset = Dataset.open()
        .uri("goosefs://10.0.0.1:9200/data/my-dataset.lance")
        .readOptions(options)
        .build()) {
    // ... use the dataset
}

The Master address can be resolved from (in priority order):

1. The goosefs_master_addr storage option (supports HA: "addr1:port,addr2:port").
2. The GOOSEFS_MASTER_ADDR environment variable.
3. The host and port from the URL authority.

The following keys can be used as both environment variables or keys in the storage_options parameter:

cargo test -p lance-io --features "goosefs goosefs-test" \
    --test goosefs_integration -- --ignored --nocapture --test-threads=1