Full-Text Search in Lance¶

Lance provides powerful full-text search (FTS) capabilities using an inverted index. This tutorial guides you through building and using FTS indexes to dramatically speed up text search operations while maintaining high accuracy.

By the end of this tutorial, you'll be able to build and use an FTS index, understand performance differences between indexed and non-indexed searches, and learn how to tune search parameters for optimal performance.

Install the Python SDK¶

First, install the required dependencies:

pip install pylance pandas pyarrow numpy tqdm

Set Up Your Environment¶

Import the necessary libraries for working with Lance datasets:

import lance
import pyarrow as pa

Prepare Your Text Data¶

In this quickstart, we'll create a simple dataset with text documents:

table = pa.table(
    {
        "id": [1, 2, 3],
        "text": [
            "I left my umbrella on the evening train to Boston",
            "This ramen recipe simmers the broth for three hours with dried mushrooms.",
            "This train is scheduled to leave for Edinburgh at 9:30 in the morning",
        ],
    }
)

ds = lance.write_dataset(table, "/tmp/fts.lance", mode="overwrite")

This creates a Lance dataset with three text documents containing overlapping keywords that we'll use to demonstrate different search scenarios.

Explore Your Dataset Schema¶

Let's examine the structure of our dataset:

ds = lance.dataset("/tmp/fts.lance")
print(ds.schema)

This prints the PyArrow schema of the dataset:

id: int64
text: string

Build the Full-Text Search Index¶

Full-text search is created with an inverted scalar index on your text column. Choose the INVERTED index type when calling create_scalar_index on your Lance dataset. Lance uses the BM25 ranking algorithm for relevance scoring. Results are automatically ranked by relevance, with higher scores indicating better matches.

ds.create_scalar_index(
    column="text",
    index_type="INVERTED",
)

The index creation process builds an efficient lookup structure that maps words to the documents containing them. This enables high-performance keyword-based search, even on large datasets.

Advanced Index Configuration¶

You can customize the index creation with various parameters to optimize for your specific use case:

ds.create_scalar_index(
    column="text",
    index_type="INVERTED",
    name="text_idx",  # Optional index name (defaults to "text_idx")
    replace=True,  # Replace existing index with the same name
    with_position=False,  # Set True to enable phrase queries
    base_tokenizer="simple",  # "simple", "whitespace", "raw", "icu", or "ngram"
    language="English",  # Language for stemming and stop words
    max_token_length=40,  # Drop tokens of 40 UTF-8 bytes or more
    lower_case=True,  # Lowercase text before tokenization
    stem=True,  # Stem tokens (language-dependent)
    remove_stop_words=True,  # Remove stop words (language-dependent)
    custom_stop_words=None,  # Optional additional stop words
    ascii_folding=True,  # Fold accents to ASCII (e.g., "é" -> "e")
)

Tokenizer Options¶

• simple: Splits tokens on whitespace and punctuation
• whitespace: Splits tokens only on whitespace
• raw: No tokenization (useful for exact matching)
• code: Splits code-like text into identifiers and operators for source-code search. Use analyzer="code", which changes the tokenizer defaults. See the code analyzer section for its flags and its FTS format v3 requirement.

Lance also supports multilingual tokenization:

• icu: Unicode word segmentation with built-in ICU dictionaries
• jieba/default: Chinese text tokenization using Jieba
• lindera/ipadic: Japanese text tokenization using Lindera with IPAdic dictionary
• lindera/ko-dic: Korean text tokenization using Lindera with Ko-dic dictionary
• lindera/unidic: Japanese text tokenization using Lindera with UniDic dictionary

Language Processing Features¶

• stemming: Reduces words to their root form (e.g., "running" → "run")
• stop words: Removes common words like "the", "and", "is"
• ascii folding: Converts accented characters to ASCII (e.g., "é" → "e")

Search With FTS Queries¶

Now you can run FTS queries using your inverted index:

ds = lance.dataset("/tmp/fts.lance")
ds.to_table(full_text_query="umbrella train").to_pandas()

This query returns documents that contain either "umbrella" or "train" (or both). The search is case-insensitive and uses the inverted index for fast retrieval.

   id                                                                   text  _score
0   1                      I left my umbrella on the evening train to Boston  1.6161
1   3  This train is scheduled to leave for Edinburgh at 9:30 in the morning  0.4606

Combining Full-Text Search with Metadata¶

It can be useful to combine FTS with metadata filtering in a single query to find more relevant results. You can do this by passing a filter expression to the filter parameter.

meta_table = pa.table(
    {
        "id": [1, 2, 3],
        "text": [
            "I left my umbrella on the morning train to Boston",
            "This ramen recipe simmers the broth for three hours with dried mushrooms.",
            "This train is scheduled to leave for Edinburgh at 9:30 AM",
        ],
        "category": ["travel", "food", "travel"],
    }
)

meta_ds = lance.write_dataset(meta_table, "/tmp/fts_metadata.lance", mode="overwrite")
meta_ds.create_scalar_index(column="text", index_type="INVERTED")
meta_ds = lance.dataset("/tmp/fts_metadata.lance")

meta_ds.to_table(
    full_text_query="three",
    filter="category = 'food'",
).to_pandas()

   id                                                                       text category  _score
0   2  This ramen recipe simmers the broth for three hours with dried mushrooms.     food  0.8855

Advanced Search Features¶

Boolean Search Operators¶

You can use boolean search operators by constructing a structured query object.

All terms: AND¶

from lance.query import FullTextOperator, MatchQuery

and_query = MatchQuery("umbrella train boston", "text", operator=FullTextOperator.AND)
ds.to_table(full_text_query=and_query).to_pandas()

   id                                               text  _score
0   1  I left my umbrella on the evening train to Boston  2.7087

Any terms: OR¶

or_query = MatchQuery("morning evening", "text", operator=FullTextOperator.OR)
ds.to_table(full_text_query=or_query).to_pandas()

   id                                                                   text  _score
0   1                      I left my umbrella on the evening train to Boston  1.0926
1   3  This train is scheduled to leave for Edinburgh at 9:30 in the morning  0.9612

Mix AND/OR queries via operators¶

You can mix AND/OR queries using operators in Python:

q1 = MatchQuery("morning evening", "text", operator=FullTextOperator.OR)
q2 = MatchQuery("train", "text")
ds.to_table(full_text_query=(q1 & q2)).to_pandas()

   id                                                                   text  _score
0   1                      I left my umbrella on the evening train to Boston  1.6161
1   3  This train is scheduled to leave for Edinburgh at 9:30 in the morning  1.4218

To combine OR queries via operators, use the pattern q1 | q2.

Every query combined with AND becomes a scoring MUST clause: all clauses must match, and every matching clause contributes to the final _score.

Exclude terms: NOT¶

Queries that exclude specific keywords are explicitly written using BooleanQuery/Occur as shown below.

from lance.query import BooleanQuery, Occur

q = BooleanQuery(
    [
        (Occur.MUST, MatchQuery("umbrella", "text")),
        (Occur.MUST_NOT, MatchQuery("train", "text")),
    ]
)
ds.to_table(full_text_query=q).to_pandas()

Empty DataFrame
Columns: [id, text, _score]
Index: []

Phrase Search¶

For exact phrase matching, ensure you enable with_position=True during index creation (disabled by default) and wrap the phrase in double quotes (or pass a PhraseQuery):

ds.create_scalar_index(
    "text",
    "INVERTED",
    with_position=True,
    remove_stop_words=False,
    replace=True,
)
ds = lance.dataset("/tmp/fts.lance")
ds.to_table(full_text_query='"train to boston"').to_pandas()

   id                                               text  _score
0   1  I left my umbrella on the evening train to Boston  2.0614

Substring matches with N-gram indexing¶

NGRAM is a type of scalar index for substring / pattern-style searches over text. It is a good alternative to wildcard-style queries like term* / *term (which are not parsed by full_text_query in Lance).

The N-gram index creates a bitmap for each N-gram in the string. By default, Lance uses trigrams. This index can be used to speed up queries using the contains function in filters.

ds.create_scalar_index(column="text", index_type="NGRAM", name="text_ngram")
ds = lance.dataset("/tmp/fts.lance")
ds.to_table(filter="contains(text, 'ramen')").to_pandas()

   id                                                                       text
0   2  This ramen recipe simmers the broth for three hours with dried mushrooms.

You can explain the query plan to confirm the N-gram index's usage as shown below:

print(ds.scanner(filter="contains(text, 'train')").explain_plan())

LanceRead: uri=tmp/fts.lance/data, projection=[id, text], num_fragments=1, range_before=None, range_after=None, row_id=false, row_addr=false, full_filter=contains(text, Utf8("train")), refine_filter=--
  ScalarIndexQuery: query=[contains(text, Utf8("train"))]@text_ngram(NGram)

Fuzzy Search¶

Fuzzy search is supported for FTS MatchQuery on INVERTED indexes. Pass a positive integer fuzziness (typically 1 or 2) to specify the maximum Levenshtein edit distance used to match terms with typos or slight variations:

ds.to_table(
    full_text_query=MatchQuery(
        "rammen",  # Misspelled 'ramen'
        "text",
        fuzziness=1,
        max_expansions=50,
    )
).to_pandas()

   id                                                                       text  _score
0   2  This ramen recipe simmers the broth for three hours with dried mushrooms.  0.9808

To enforce exact prefixes during fuzzy matching, set prefix_length. This means the first N characters must match exactly before fuzzy edits are allowed on the rest of the term. For example, with prefix_length=2, "rammen" can match terms starting with "ra" (like "ramen"), but not terms starting with other prefixes.

ds.to_table(
    full_text_query=MatchQuery(
        "rammen",
        "text",
        fuzziness=1,
        prefix_length=2,  # "ra" must match exactly
    )
).to_pandas()

   id                                                                       text  _score
0   2  This ramen recipe simmers the broth for three hours with dried mushrooms.  0.9808

Searching Multiple Columns¶

When a query should span several text columns (for example title and body), Lance offers two strategies. They differ in how they combine the matches from each column.

best_fields with MultiMatchQuery¶

MultiMatchQuery runs the query against each column independently and keeps the score of the best matching column for each row. Each column is scored against its own corpus statistics, and an optional per-column boosts multiplier is applied before taking the maximum.

from lance.query import MultiMatchQuery

multi_table = pa.table(
    {
        "id": [1, 2],
        "title": ["Albino elephant sighting", "Safari packing guide"],
        "body": [
            "Rangers photographed a rare calf near the river.",
            "Visitors hoping to spot an albino elephant should pack binoculars.",
        ],
        "first_name": ["john", "jane"],
        "last_name": ["smith", "doe"],
    }
)
multi_ds = lance.write_dataset(multi_table, "/tmp/fts_multicol.lance", mode="overwrite")
for col in ("title", "body", "first_name", "last_name"):
    multi_ds.create_scalar_index(col, "INVERTED")
multi_ds = lance.dataset("/tmp/fts_multicol.lance")

multi_ds.to_table(
    columns=["id", "title", "_score"],
    full_text_query=MultiMatchQuery(
        "albino elephant",
        ["title", "body"],
        boosts=[2.0, 1.0],  # weight title matches twice as much
    ),
).to_pandas()

   id                     title  _score
0   1  Albino elephant sighting  2.7726
1   2      Safari packing guide  1.3440

This works well when the columns are interchangeable and a strong match in one column is enough to make a row relevant.

combined_fields (BM25F) with CombinedFieldsQuery¶

CombinedFieldsQuery instead treats the target columns as one virtual field and combines their statistics into a single BM25F score. This makes scores comparable across columns, and a query can match with terms spread over different columns.

from lance.query import CombinedFieldsQuery

multi_ds.to_table(
    columns=["id", "first_name", "last_name", "_score"],
    full_text_query=CombinedFieldsQuery(
        "john smith",
        ["first_name", "last_name"],
        boosts=[1.0, 1.0],
        operator=FullTextOperator.AND,
    ),
).to_pandas()

   id first_name last_name  _score
0   1       john     smith  1.3863

Prefer combined_fields in these cases:

• A term is rare in one column but common in another. Because document frequencies are combined, a chance hit in a short column does not outrank a better match elsewhere. best_fields cannot do this.
• The query terms are spread over several columns. With operator=AND, "john smith" matches a row whose first_name is "john" and whose last_name is "smith", even though neither column contains both terms. best_fields would need a single column to contain the whole query.

Per-column boosts set how much each column contributes to the combined term frequency. Each weight must be at least 1, and fractional weights are allowed. For example, a title boost of 2 counts every title term twice.

Performance Tips¶

Index Maintenance¶

When you append new rows after creating an INVERTED index, Lance still returns those rows in full_text_query results. It searches indexed fragments using the FTS index, scans unindexed fragments with flat search, and then merges the results.

To keep FTS latency low as new data arrives, periodically add unindexed fragments into the existing FTS index by calling ds.optimize.optimize_indices():

new_rows = pa.table(
    {
        "id": [4],
        "text": ["The next train leaves at noon"],
    }
)
ds.insert(new_rows)

ds.optimize.optimize_indices(index_names=["text_idx"])
ds = lance.dataset("/tmp/fts.lance")

stats = ds.stats.index_stats("text_idx")
print(stats["num_unindexed_rows"], stats["num_indexed_rows"])

0 4

If you changed tokenizer settings (such as with_position, base_tokenizer, stop words, or stemming), rebuild the index with create_scalar_index(..., replace=True) so the full dataset is indexed with the new configuration.

Index Configuration Best Practices¶

• Enable with_position when you need phrase queries, because it stores word positions within documents. For simple term searches, disabling this option can save considerable storage space without impacting performance.

• Keep lower_case=True enabled for most applications to ensure case-insensitive search behavior. This provides a better user experience and matches common search expectations, though you can disable it if case sensitivity is important for your use case.

• Enable stemming (stem=True) when you want better recall by matching word variations (e.g., "running" matches "run"). Disable stemming if you need exact term matching or if your domain requires precise terminology.

• Consider enabling remove_stop_words=True for cleaner search results, especially in content-heavy applications. This removes common words like "the", "and", and "is" from the index, reducing noise and improving relevance. Keep stop words if they carry important meaning in your domain.

Query Optimization¶

Using specific, targeted search terms often yields better performance than broad, generic queries. More specific terms reduce the number of potential matches and allow the index to work more efficiently. Consider analyzing your most common search patterns and optimizing your index configuration accordingly.

Combining full-text search with metadata filters can significantly reduce the search space and improve performance. Use structured data filters to narrow down results before applying text search, or vice versa. This approach is particularly effective for large datasets where you can eliminate many irrelevant documents early in the query process.

Further Reading¶

For advanced usage instructions with different tokenizers and more technical details on the index training process, including information about the expected memory and disk usage, visit the full-text index specification.

Next Steps¶

Continue with the Full-Text Search User Guide for index configuration, query types, and index maintenance.