Class FtsIndex

SQLite FTS5 inverted index over chunked note content. Opt-in via --persistent-index. The production evidence linked in this module reports 50–100ms BM25 top-10 at 1,771 chunks / 368 files; other corpora and hardware must be measured independently. Falls back transparently to the in-memory parallel-scan path when better-sqlite3 isn't installed.

Construct, then call open(), then drive incremental sync via diff + reindexFile / reindexPdfFile / dropFile. Query with search; deep-link to individual chunks with getChunk.

Example

const idx = new FtsIndex({ file, vaultRoot, tokenize: "unicode61" });
await idx.open();
idx.reindexFile(relPath, mtimeMs, content, wikilinkTargets, tags);
const hits = idx.search("vector retrieval", { limit: 25 });
await idx.closeAndRelease();

Constructors

constructor

new FtsIndex(
opts: { file: string; tokenize?: TokenizeMode; vaultRoot: string },
): FtsIndex
Parameters
- opts: { file: string; tokenize?: TokenizeMode; vaultRoot: string }
  Exact .fts5.db file, owning vault root, and optional tokenizer.
Returns FtsIndex

Throws
If opts.file is outside the exact FTS namespace.

Throws
If the tokenizer is not exactly unicode61 or trigram.
- Defined in src/fts5.ts:1129

Methods

auditKind

auditKind(kind: ChunkKind): FtsKindAudit
Audit the physical FTS rows for one content kind without mutating them.

A declared file is complete only when it has a positive integer n_chunks, the same number of physical rows, and one valid row at every integer index in 0..n_chunks - 1. The audit also rejects invalid line ranges or raw-content storage, chunk-only paths, a different kind on a declared path, missing/invalid source revisions, quarantine markers for the requested kind, and invalid kinds anywhere in the index. Those global checks deliberately fail closed so a scoped markdown or PDF audit cannot certify an index whose other rows have unknown provenance.
Parameters
- kind: ChunkKind
  Content-source kind to audit.
Returns FtsKindAudit
Declared and physical file/chunk counts plus the number of unique mismatched paths.
Example
```
const audit = idx.auditKind("md");
if (audit.mismatched_files > 0) {
  throw new Error("FTS index is physically incomplete");
}
```
- Defined in src/fts5.ts:2200

clearOnDisk

clearOnDisk(): Promise<boolean>
Remove the index file + WAL/SHM/rollback-journal sidecars after validating every present leaf. Missing files are idempotent; directories, special objects, and non-ENOENT inspection/deletion failures refuse the operation.

Returns Promise<boolean>
true when at least one artifact was removed.

Throws
If any main/WAL/SHM/rollback-journal leaf is unsafe or a non-ENOENT operation fails.
- Defined in src/fts5.ts:1291

close

close(): void
Close the underlying SQLite handle synchronously and begin best-effort asynchronous lifetime release. This preserves the historical synchronous API; callers that own process shutdown must await closeAndRelease to prove both lease markers are gone. A release rejection is observed here (never unhandled), retained, and retried by a later awaited close.

Returns void
- Defined in src/fts5.ts:1372

closeAndRelease

closeAndRelease(): Promise<void>
Close SQLite without reopening it and await exact family-then-namespace lifetime release. If a previous best-effort close failed, this call makes one new attempt through the same core handle; terminal integrity failures remain fail-closed while retryable failures can complete.

Returns Promise<void>
Only after the SQLite handle is closed and both markers are gone.
- Defined in src/fts5.ts:1393

currentSourceReceiptMask

currentSourceReceiptMask(receipts: readonly FtsSourceReceipt[]): boolean[]
Verify a bounded receipt batch in one synchronous SQLite read snapshot. Empty input returns immediately. Oversized input is rejected before any allocation or SQLite work; malformed entries receive false verdicts while preserving positional association for the accepted batch.
Parameters
- receipts: readonly FtsSourceReceipt[]
  Persisted source receipts to verify, in caller order.
Returns boolean[]
One current/not-current verdict per accepted input receipt.
Throws
If more than 512 receipts are supplied.
Example
```
const current = index.currentSourceReceiptMask(index.searchWithReceipts("retrieval"));
```
- Defined in src/fts5.ts:2087

diff

diff(
    liveEntries: { mtimeMs: number; relPath: string }[],
    kind?: ChunkKind,
): {
    added: string[];
    deleted: string[];
    unchanged: string[];
    updated: string[];
}
Diff the on-disk source_state against the live vault snapshot. Returns categorized lists; caller is expected to feed added + updated paths back into reindexFile() and pass deleted to dropFile().

v2.8.0: optional kind filter — when set, the diff only considers source_state rows of that kind. Lets the markdown-sync and PDF-sync paths run independently against the same DB without one's "missing files" being mistakenly deleted by the other. Default undefined means "all kinds" (used by older callers + diff queries that want a global view).
Parameters
- liveEntries: { mtimeMs: number; relPath: string }[]
- Optionalkind: ChunkKind
Returns { added: string[]; deleted: string[]; unchanged: string[]; updated: string[] }
- Defined in src/fts5.ts:2014

dropFile

dropFile(relPath: string): void
Drop a file's chunks, state row, and quarantine marker. Idempotent.

v3.7.18 R-8 — wrapped in db.transaction() for atomicity. Pre-3.7.18 the two DELETE statements ran independently; a crash / SIGKILL / DB lock contention between them could leave source_state saying "this file is indexed at mtime X" while chunks had no rows — causing the next watcher event to skip re-indexing (state matches) but search to miss the file (no chunks). Sibling of v3.7.10 audit #10 fix that wrapped reindexFile / reindexPdfFile / source_state in a txn for the same reason. Caught by round-20 external audit.
Parameters
- relPath: string
Returns void
- Defined in src/fts5.ts:2500

fingerprintKind

fingerprintKind(kind: ChunkKind): string
Hash the exact physical source declarations and FTS chunk payload for one content kind without materializing all rows in memory.

The manifest is intended for before/after integrity checks in strict evidence runs. It includes source mtimes, monotonic revision tombstones, timestamps, every stored searchable/metadata column of BOTH searchable tables — chunks and the v7 sibling chunk_parts, whose rows decide what a search finds — and durable quarantine markers, so an in-place mutation that keeps aggregate counts unchanged still changes the digest.

Generation v3 added the chunk_parts rows; a digest taken before v7 is not comparable with one taken after it.
Parameters
- kind: ChunkKind
  Content-source kind to fingerprint.
Returns string
Lowercase SHA-256 digest of the ordered physical rows.
- Defined in src/fts5.ts:2369

getChunk

getChunk(
relPath: string,
chunkIndex: number,
): { content: string; line_end: number; line_start: number } | null
Fetch a single raw chunk with the stable receipt-free public shape. This compatibility wrapper delegates to getChunkWithReceipt and strips the internal authority fields.
Parameters
- relPath: string
  Exact vault-relative source path from a prior FTS hit.
- chunkIndex: number
  Zero-based chunk index within that source.
Returns { content: string; line_end: number; line_start: number } | null
Verbatim chunk content and line bounds, or null when unavailable.
Example
```
const chunk = index.getChunk("Projects/plan.md", 0);
```
- Defined in src/fts5.ts:2914

getChunkWithReceipt

getChunkWithReceipt(relPath: string, chunkIndex: number): FtsReceiptChunk | null
Fetch a receipt-bound chunk by (rel_path, chunk_index). Backs the obsidian://chunk/{chunkIndex}/{+notePath} resource so MCP clients can deep-link into specific chunks returned by a prior search. Returns the RAW chunk text (the unenriched original); the FTS5 content column additionally carries a synthetic wikilink-targets meta-line for recall, which would otherwise pollute resource responses (audit v0.10.4 P1). The returned source mtime and monotonic revision are selected in the same SQL snapshot as the bytes; callers that own a live Vault must compare both before exposing persisted content.
Parameters
- relPath: string
  Exact vault-relative source path from a prior FTS hit.
- chunkIndex: number
  Zero-based chunk index within that source.
Returns FtsReceiptChunk | null
Receipt-bound raw content, or null when the row is absent, orphaned, invalid-kind, or quarantined.
Example
```
const chunk = index.getChunkWithReceipt("Projects/plan.md", 0);
```
- Defined in src/fts5.ts:2945

isCurrentSourceReceipt

isCurrentSourceReceipt(
    relPath: string,
    kind: ChunkKind,
    indexedMtimeMs: number,
    indexedRevision: number,
): boolean
Verify that one internal FTS receipt still names the source generation currently eligible for egress. The monotonic revision closes same-mtime replacement and delete/re-add ABA gaps that an mtime-only comparison cannot distinguish. This convenience wrapper uses the same atomic batch verifier as multi-hit callers.
Parameters
- relPath: string
  Vault-relative source path carried by the hit.
- kind: ChunkKind
  Content-source kind carried by the hit.
- indexedMtimeMs: number
  Source mtime committed with the indexed bytes.
- indexedRevision: number
  Monotonic revision committed with the indexed bytes.
Returns boolean
True only for a finite, safe, non-quarantined current receipt.
Example
```
const hit = index.searchWithReceipts("retrieval", { limit: 1 })[0];
const current = hit
  ? index.isCurrentSourceReceipt(hit.rel_path, hit.kind, hit.indexed_mtime_ms, hit.indexed_revision)
  : false;
```
- Defined in src/fts5.ts:2164

open

open(expectedDiscovery?: FtsIndexDiscovery): Promise<void>
Open the SQLite database, admit only a fresh or same-vault FTS schema, bootstrap the FTS5 virtual table + helpers, then enable WAL and best-effort tighten file perms to 0o600 on the db + sidecars. A populated foreign, malformed, or newer-schema database is refused before Enquire issues persistent PRAGMA, DDL, DML, or chmod operations. SQLite itself may still take locks or perform recovery while the live handle reads ownership metadata. Before dependency loading and again immediately before native open, the main, WAL, SHM, and rollback-journal leaves must be wholly absent or every present leaf must be a singly linked regular file; orphan sidecars refuse. Idempotent — a second open() call is a no-op.
Parameters
- OptionalexpectedDiscovery: FtsIndexDiscovery
  Optional readonly preflight result to bind this mutating open to. No argument preserves the low-level intentional-rebuild contract; a supplied stale result is refused before bootstrap.
Returns Promise<void>

Throws
If better-sqlite3 fails to load, the native binding is unavailable, or a populated database cannot prove same-vault FTS ownership with a supported non-future schema.
- Defined in src/fts5.ts:1158

quarantineFile

quarantineFile(relPath: string, kind?: ChunkKind): void
Hide a source's retained rows from every public FTS read until the same source is successfully reindexed or dropped. The marker is durable so a failed refresh cannot become visible again after process restart.
Parameters
- relPath: string
  Vault-relative source path whose indexed bytes are stale.
- kind: ChunkKind = "md"
  Content-source kind; defaults to Markdown for watcher/sync callers.
Returns void
Nothing.
Example
```
index.quarantineFile("Private/rotated.md");
```
- Defined in src/fts5.ts:2067

reindexFile

reindexFile(
    relPath: string,
    mtimeMs: number,
    content: string,
    wikilinkTargets?: string[],
    tags?: string[],
    title?: string,
    aliases?: string[],
): number
Re-chunk a single markdown file, replacing its existing chunks atomically.

v3.7.10 (external audit #10) — wrapped DELETE + N×INSERT + source_state UPDATE in a single SQLite transaction. Pre-fix a crash/error between statements could leave partially-updated chunks (some new, some stale) with a stale source_state row pointing at the wrong chunk count. The transaction guarantees all-or-nothing atomicity. better-sqlite3 db.transaction() wraps + auto-rolls back on throw.
Parameters
- relPath: string
- mtimeMs: number
- content: string
- wikilinkTargets: string[] = []
- tags: string[] = []
- title: string = ""
- aliases: string[] = []
Returns number
- Defined in src/fts5.ts:2526

reindexPdfFile

reindexPdfFile(
    relPath: string,
    mtimeMs: number,
    pages: readonly { pageNumber: number; text: string }[],
): number
v2.8.0 — re-chunk a single PDF, replacing its existing chunks atomically. Caller pre-extracts page text via extractPdfText (src/pdf.ts) so this method stays decoupled from pdfjs-dist (which is an optionalDependency).

Page boundaries are preserved as [page: N] markers in the joined text before chunking — the chunker may split a page across chunks or merge short pages, but the markers travel with the text so search snippets carry page citations. Same chunkContent pipeline as markdown so chunk IDs match across the BM25 / TF-IDF / embeddings rankers (RRF requires stable IDs).
Parameters
- relPath: string
- mtimeMs: number
- pages: readonly { pageNumber: number; text: string }[]
Returns number
- Defined in src/fts5.ts:2655

search

search(
    rawQuery: string,
    opts?: {
        folder?: string;
        limit?: number;
        sinceMtimeMs?: number;
        tag?: string;
    },
): FtsSearchHit[]
BM25-ranked search with the stable, receipt-free public result shape. This compatibility wrapper applies the same provenance and quarantine joins as searchWithReceipts, then strips internal receipt fields.
Parameters
- rawQuery: string
  User query string. Whitespace-only returns [].
- opts: { folder?: string; limit?: number; sinceMtimeMs?: number; tag?: string } = {}
  - Optionalfolder?: string
    
    Vault-relative prefix filter.
  - Optionallimit?: number
    
    Max results. Default 25.
  - OptionalsinceMtimeMs?: number
    
    Recency filter in source mtime milliseconds.
  - Optionaltag?: string
    
    Exact-tag membership filter.
Returns FtsSearchHit[]
Receipt-free hits sorted by descending score.
Example
```
const hits = index.search("vector retrieval", { limit: 25 });
```
- Defined in src/fts5.ts:2725

searchWithReceipts

searchWithReceipts(
    rawQuery: string,
    opts?: {
        folder?: string;
        limit?: number;
        sinceMtimeMs?: number;
        tag?: string;
    },
): FtsReceiptSearchHit[]
BM25-ranked search over chunk content with persisted source receipts. Folder + tag + recency filters are pushed down to the SQL layer. Hyphenated identifiers (e.g. "claude-telegram") are quote-escaped via safeFts5Query so FTS5 doesn't interpret - as the NOT operator.
Parameters
- rawQuery: string
  User query string. Whitespace-only returns [].
- opts: { folder?: string; limit?: number; sinceMtimeMs?: number; tag?: string } = {}
  - Optionalfolder?: string
    
    Vault-relative prefix filter.
  - Optionallimit?: number
    
    Max results. Default 25.
  - OptionalsinceMtimeMs?: number
    
    Recency filter — only return chunks from files modified at or after this mtime.
  - Optionaltag?: string
    
    Exact-tag membership filter (only matches the full tag, not core-team for core).
Returns FtsReceiptSearchHit[]
Provenance-bound, non-quarantined hits sorted by descending score. The ranked chunks pass runs first and is scored exactly as it was before v7; only if it under-fills limit is a second pass over the sibling chunk_parts table appended, with score 0 — those hits are FOUND by the words an identifier is spelled from, never RANKED by them. Each hit carries the source_state mtime and monotonic revision committed with its indexed bytes; callers that own a live Vault must compare both before exposing persisted text. Empty array if no usable query tokens or no matches.
Example
```
const hits = index.searchWithReceipts("vector retrieval", { limit: 25 });
const current = index.currentSourceReceiptMask(hits);
```
- Defined in src/fts5.ts:2769

totalChunks

totalChunks(): number
Total chunks across the index. Used by stats / banner / UI.

Returns number
- Defined in src/fts5.ts:2986

totalFiles

totalFiles(): number
Total source files (notes + PDFs) tracked in source_state. Used by the ready banner so users can verify the index actually built.

Returns number
- Defined in src/fts5.ts:2994

Class FtsIndex

Example

Index

Constructors

Methods

Constructors

constructor

Parameters

Returns FtsIndex

Throws

Throws

Methods

auditKind

Parameters

Returns FtsKindAudit

Example

clearOnDisk

Returns Promise<boolean>

Throws

close

Returns void

closeAndRelease

Returns Promise<void>

currentSourceReceiptMask

Parameters

Returns boolean[]

Throws

Example

diff

Parameters

Returns { added: string[]; deleted: string[]; unchanged: string[]; updated: string[] }

dropFile

Parameters

Returns void

fingerprintKind

Parameters

Returns string

getChunk

Parameters

Returns { content: string; line_end: number; line_start: number } | null

Example

getChunkWithReceipt

Parameters

Returns FtsReceiptChunk | null

Example

isCurrentSourceReceipt

Parameters

Returns boolean

Example

open

Parameters

Returns Promise<void>

Throws

quarantineFile

Parameters

Returns void

Example

reindexFile

Parameters

Returns number

reindexPdfFile

Parameters

Returns number

search

Parameters

Optionalfolder?: string

Optionallimit?: number

OptionalsinceMtimeMs?: number

Optionaltag?: string

Returns FtsSearchHit[]

Example

searchWithReceipts

Parameters

Optionalfolder?: string

Optionallimit?: number

OptionalsinceMtimeMs?: number

Optionaltag?: string

Returns FtsReceiptSearchHit[]

Example

totalChunks

`Optional`folder?: string

`Optional`limit?: number

`Optional`sinceMtimeMs?: number

`Optional`tag?: string

`Optional`folder?: string

`Optional`limit?: number

`Optional`sinceMtimeMs?: number

`Optional`tag?: string