Preserve nulls, empty lists, and repetition without row objects

A null list means the list value is absent; an empty list is present but contains no elements. A present list can itself contain null elements if the element type is optional. These states can produce different results for length, null tests, and unnesting, so replacing all of them with an empty value stream is incorrect. Define the schema's optional and repeated nodes before constructing an example. An encoded leaf stream must provide enough structural evidence to reassemble the intended parent rows and containers.

In the Dremel-style approach used by Parquet, a definition level describes how far the optional/repeated path is defined, while a repetition level identifies the repeated ancestor at which a subsequent occurrence continues. A repetition level of zero begins a new top-level record. Their maximum values come from the schema, not from the largest list observed in one batch. Physical nonnull leaf values are stored only when the required path is fully defined; structural entries can still exist for null or empty states that contribute no physical value.

Use a required root containing optional group tags annotated as LIST; tags contains repeated group list; list contains optional int32 element. The optional list contributes the first definition level, the repeated entry the second, and the optional element the third. Thus a null tags list has definition zero; a present empty list has definition one; a null element in a present entry has definition two; and an actual integer has definition three. Only the last state contributes an integer to the physical value stream.

For this single repeated ancestor, repetition zero starts a row and repetition one continues the same row's list. More deeply nested repeated paths need more levels, so do not apply this small table universally. Decode levels and leaf values together, using the schema and declared counts, then reconstruct or directly evaluate the required operation. Test lists spanning page boundaries and empty/null parents as well as simple nonempty arrays. A value-count mismatch can shift all later rows even if every integer decoder output is individually correct.

Arrow's ordinary variable-size List layout uses parent validity, offsets into a child array, and the child's own validity/value representation. Equal neighboring offsets can describe an empty present list; a parent validity bit distinguishes a null list, whose child segment is not semantically interpreted as an ordinary valid list. This layout supports different access patterns from an on-disk level stream. Converting Parquet to Arrow therefore reconstructs structure, and cannot be assumed to be a zero-copy reinterpretation of the same buffers. ORC also has its own presence and length-oriented streams.

For a narrow optional-scalar analogy, Reset the probability lab, set both p=0.6 and n=10, predict six present physical values on average, then Play or Step. The exact binomial model assumes independent scalar presence at one optional path; playback highlights bars rather than sampling records. Reset and re-enter both controls for another case. Repeated lists violate any simple equation between row count and trial count, and the instrument emits no definition levels, repetition levels, offsets, or nested rows. Use the worked trace, rather than this probability analogy, to verify reconstruction.