Loading Overview¶

supernova's loading pipeline streams pre-embedded parquet files from S3 or HuggingFace into vector stores like Qdrant. An embedding run typically produces many parquet files (one per chunk/slice) under a shared S3 prefix -- the loader reads all of them. It uses DuckDB for efficient remote parquet reads and async concurrency for parallel upserts.

Configuration¶

Loader configs live in configs/loader/. The same file is consumed by both nova load (single machine) and nova load-dist (distributed via SkyPilot); the distributed dispatcher reads the optional dispatch: and resources: blocks and the single-machine loader ignores them.

vectors:
  dense:
    type: dense
    column: dense_embedding
    distance: cosine
  sparse:
    type: sparse
    column: sparse_embedding
  colbert:
    type: multivector
    column: multivector_embedding
    distance: cosine
    comparator: max_sim

datasource:
  type: s3                          # s3 or huggingface
  bucket: my-bucket
  prefix: dataset/model
  id_expression: "vf_point_id(filename, file_row_number)"   # see below
  payload_fields:                   # what ends up in the vector store payload
    text: text                      # payload key: parquet column name
    source: source

vectorstore:
  type: qdrant
  collection_name: my-collection
  url: ${QDRANT_URL}                # env var substitution with ${VAR}
  api_key: ${QDRANT_API_KEY}

loader:
  batch_size: 1000                  # points per upsert call
  prefetch_size: 100000             # rows per DuckDB fetch
  concurrency: 8                    # parallel upsert tasks

Running¶

nova load configs/loader/my_dataset.yaml

Datasources¶

S3¶

Streams parquet files via DuckDB's httpfs extension. No local download.

datasource:
  type: s3
  bucket: my-bucket
  prefix: stanford-oval--ccnews/baai_bge_large_en_v1.5

Reads all parquet files matching s3://bucket/prefix/**/*.parquet.

HuggingFace¶

Streams directly from HuggingFace Hub via DuckDB's hf:// protocol.

datasource:
  type: huggingface
  repo_id: CohereLabs/wikipedia-2023-11-embed-multilingual-v3
  subdir: en

Point IDs (id_expression)¶

id_expression is a DuckDB SQL expression the loader evaluates per row to produce the Qdrant point ID. The default (row_id) is just a bare column name and works if your parquets carry a pre-baked row_id column. The recommended form for supernova-produced corpora is the built-in macro:

datasource:
  id_expression: "vf_point_id(filename, file_row_number)"

The macro hashes (parquet path, physical row index) into a deterministic UUID — the same form used by nova brute-force and nova generate-queries, so recall ground truth lines up across the eval pipeline.

file_row_number is critical here: it's a DuckDB virtual column that always reflects the physical row index, regardless of parallel scan order. Do not use ROW_NUMBER() OVER (PARTITION BY filename) — that reflects DuckDB's scan ordering and produces different IDs from the brute-force side under concurrency. There's a regression test for this in tests/test_loader_id_expression.py.

The base reader auto-enables read_parquet(..., filename=true, file_row_number=true) whenever your id_expression mentions either column, so you don't have to wire that yourself. See Loader Architecture for the full ID-space discussion.

Vectors¶

The top-level vectors: block declares one or more named vectors. Each key becomes the vector name in Qdrant; each entry needs type (dense, sparse, or multivector) and column (the parquet column).

A collection with multiple named vectors lets you do hybrid retrieval (e.g. dense + sparse + late-interaction multivector).

Payload composition¶

payload_fields controls what data gets stored alongside each vector:

payload_fields:
  text: text              # store parquet "text" column as "text" in payload
  abstract: text          # ...or rename it to "abstract"
  source: source
  url: url

JSON-string columns that parse to a dict are automatically unpacked into the payload.

How it works¶

1. DuckDB streams parquet data in large chunks (prefetch_size rows per fetch)
2. Chunks are sliced into batch_size upsert batches locally
3. Async upserts run concurrently, controlled by a semaphore (concurrency)
4. Deferred indexing -- HNSW construction is disabled during load, then built in one pass
5. Retry with backoff -- failed upserts are retried up to 3 times

Tuning¶