Load¶

nova load reads pre-embedded Parquet files from S3 or local disk and loads them into a vector store. Qdrant is built in; Elasticsearch, OpenSearch, and Milvus are optional.

Config¶

datasource:
  type: s3
  path: s3://my-bucket/dataset/model
  id_expression: "vf_point_id(filename, file_row_number)"
  payload_fields:
    text: text
    source: source

vectors:
  dense:
    type: dense
    column: dense_embedding
    distance: cosine
  sparse:
    type: sparse
    column: sparse_embedding

vectorstore:
  type: qdrant
  url: ${QDRANT_URL}
  api_key: ${QDRANT_API_KEY}
  collection_name: my-collection

loader:
  batch_size: 256
  concurrency: 8

Environment variables support ${VAR} and ${VAR:-default} expansion.

Running¶

Single machine:

nova load run my.yaml

Distributed:

nova load prepare my.yaml
nova load load my.yaml --num-jobs 32 --job-rank $RANK
nova load finalize my.yaml

Workers independently process subsets of the input files. nova dist load can provision and launch the workers automatically.

Other commands:

nova load inspect my.yaml   # dry run
nova load reindex my.yaml   # rebuild with new index settings
nova load delete my.yaml    # delete the collection

Resuming¶

Use --continue to resume an interrupted load:

nova load load my.yaml --num-jobs 32 --job-rank $RANK --continue

Resuming requires the same files, worker count, and a deterministic id_expression.

Datasources¶

S3:

datasource:
  type: s3
  path: s3://my-bucket/dataset/model

Local:

datasource:
  type: local
  path: /data/dataset/model

file_list may be used to restrict either source to specific files.

Point IDs¶

id_expression is a DuckDB expression evaluated for each row. The default uuid() generates new IDs on every load.

For stable IDs compatible with nova bf ground truth:

datasource:
  id_expression: "vf_point_id(filename, file_row_number)"

An existing ID column or other DuckDB expression may also be used.

Payload¶

payload_fields maps stored payload names to DuckDB expressions:

payload_fields:
  text: text
  title_upper: upper(title)

String, integer, float, and boolean values are supported directly. Cast other types when needed.

Vectors¶

Each entry under vectors defines a named vector.

Qdrant Settings¶

Connection settings are configured under vectorstore:

vectorstore:
  type: qdrant
  url: http://localhost:6334
  collection_name: my-collection

Collection and index settings go under vectorstore.params:

vectorstore:
  params:
    shard_number: 6
    replication_factor: 2
    on_disk_payload: true

    hnsw:
      m: 16
      ef_construct: 100
      on_disk: true

    quantization:
      type: scalar
      quantile: 0.99

    optimizers:
      indexing_threshold: 20000

Unset values use Qdrant defaults.

Supported quantization types are scalar, product, binary, turbo, and none.

Reindexing¶

nova load reindex updates HNSW, quantization, and optimizer settings on an existing collection without reloading the data.

nova load reindex my.yaml

Custom Sharding¶

custom_sharding routes points using a DuckDB expression:

vectorstore:
  custom_sharding:
    shard_key: "org_id"
    shards_number: 2
    replication_factor: 2

Shard keys must be strings or non-negative integers, and the expression should be deterministic.

Other Backends¶

Optional backends support dense vectors:

• opensearch
• elastic
• milvus

Build support with:

make load LOAD_FEATURES=elastic,opensearch,milvus

Backend-specific connection and index settings are configured under vectorstore.

Tuning and Failures¶

Common loader settings:

If the vector store is overloaded, increase vectorstore.timeout_s, then reduce concurrency or set max_points_per_sec.

Loader limits are per worker, so fleet-wide throughput scales with the number of workers.