The words a compound identifier is spelled from, for the v7 chunk_parts
column. Case boundaries only: the tokenizer already splits _, - and .,
so pool_day_data is reachable by its parts without help; poolDayData was
one opaque token no reader could reach by the words it is made of.
Boundaries: lower→Upper (poolDay → pool, day), an upper-case RUN followed by
a capitalised word (parseHTTPResponse → parse, http, response), and letter↔
digit (sha256Hash → sha, 256, hash). Input is NFC-normalised first so a
decomposed accent cannot split a letter from its mark, and lengths count code
points, not UTF-16 units. Only identifiers that split into two or more
segments emit parts; single words are already tokens.
Parameters
text: string
Chunk text to scan.
Returns string[]
Lower-cased, de-duplicated parts; at most 256 per chunk, each at
most 64 code points — truncated, never refused.
The words a compound identifier is spelled from, for the v7
chunk_partscolumn. Case boundaries only: the tokenizer already splits_,-and., sopool_day_datais reachable by its parts without help;poolDayDatawas one opaque token no reader could reach by the words it is made of.Boundaries: lower→Upper (
poolDay→ pool, day), an upper-case RUN followed by a capitalised word (parseHTTPResponse→ parse, http, response), and letter↔ digit (sha256Hash→ sha, 256, hash). Input is NFC-normalised first so a decomposed accent cannot split a letter from its mark, and lengths count code points, not UTF-16 units. Only identifiers that split into two or more segments emit parts; single words are already tokens.