Chunk boundaries: preserve source identity while budgeting overlap

For a teaching fixed-window rule, tokenize one nonempty normalized document into L tokens. Use window length w and overlap o, so successive starts differ by w−o. Start at zero and stop after the first window reaches the document end, keeping a possibly shorter final window. This rule differs from always creating every start below L, which can add a redundant trailing chunk. Empty documents, special tokens, headers, and parser failures need explicit policies. A character count is not a token count, especially across languages or structured content.

Tokenization itself includes normalization, segmentation, vocabulary rules, and possible post-processing. Changing any of these can change token positions without changing the displayed document meaning. Store the tokenizer artifact and normalization version with the ingestion manifest. If source citations use byte or character offsets, name the coordinate system and retain a tested mapping from parsed text to the original version. A token interval in one tokenizer is not automatically a valid byte span or a stable locator after document editing.

Overlap can keep a sentence or local relation visible in more than one chunk, but it does not guarantee that every relevant argument fits or that a table remains understandable. Under the exact window rule, repeated token occurrences increase index payload even though source coverage is unchanged. Neighboring retrieved chunks can therefore repeat the same evidence. Count unique source spans separately from raw hit count, and avoid interpreting several overlapping hits as independent confirmation of a claim.

Structure-aware cuts can keep headings, paragraphs, code blocks, or table headers with their content. Those choices replace the simple fixed-window arithmetic with measured chunk lengths and explicit maximum-size handling. A giant table may still need a specialized representation, while a short heading alone may be a poor embedding unit. Preserve parent structure and parser lineage so downstream expansion can recover needed context. Evaluate boundary-sensitive questions, cross-section references, and malformed extraction instead of optimizing only the average chunk length.

A chunk record should distinguish logical document identity, immutable source version, parser and tokenizer configuration, local interval, and content digest. Two documents with identical text can have different ownership or permissions, so a digest alone is not a safe authorization identity. Rechunking creates a new index generation or mapping rather than silently reusing identifiers for different spans. Retain deletions and replacements in the update manifest so stale evidence and citations can be detected. Measure indexing cost and retrieval quality at a fixed document snapshot.

The modular lab isolates stride arithmetic only. Reset, set m=12 and a=3, predict period four, then Play or Step. Separately Reset, set m=12 and a=4, predict period three, then Play or Step. The orbit repeatedly adds a modulo twelve; it is not the linear document-window rule and cannot establish text coverage or overlap quality. Use the contrast to practice tracing starts under a precisely stated rule, then compute the actual half-open document intervals by hand without importing the cyclic model's wrapping behavior.