Parameter-efficient adaptation

Use a column-vector convention: a frozen linear map W has output width d_out and input width d_in. A low-rank update is BA, where A maps the input to rank r and B maps that bottleneck to the output. The pretrained weight can remain full rank. It is the learned difference that is constrained. The rank of BA is at most r, even if the two factors together contain more than r numbers.

An update scale controls the contribution of the adapter relative to the base branch. A common initialization makes one factor zero and the other nonzero so the initial update is zero but useful gradients can reach one factor. Setting both factors to zero makes the bilinear update's initial gradients zero. Implementations may use different scaling or initialization rules, so preserve those details when comparing rank choices rather than treating rank as the entire method.

Low-rank factors need r times the sum of input and output widths parameters rather than their product. Gradients and optimizer moments are then needed only for trainable parameters if the implementation freezes the base correctly. However, the base weights still occupy storage, and backpropagation through frozen layers may still require activations to propagate gradients to adapters. Activation checkpointing and reduced-precision storage address different memory terms.

For a static linear adapter, its update can be merged into the base weight before inference, removing the extra low-rank branch in exact arithmetic. Finite-precision rounding can cause small numerical differences, and keeping multiple task adapters available may favor unmerged execution. Nonlinear bottleneck adapters generally cannot be absorbed into a single fixed linear weight. Quantizing a base model while training adapters adds precision and kernel constraints; it is an additional technique rather than an automatic property of low rank.

Choose target modules according to the hypothesis. Adapting only attention projections constrains a different part of the model from adapting feed-forward projections or embeddings. A fixed rank in every layer is convenient but does not prove every layer needs the same update capacity. Compare validation quality, trainable parameter count, peak memory, training time, and serving behavior. Include a frozen-model baseline to determine whether adaptation helps at all.

Keep task data, evaluation splits, prompt formats, and tuning budgets aligned when comparing full fine-tuning with adapters. Test forgetting and out-of-domain behavior, because a small parameter update can still cause a large behavioral change. Track the exact base checkpoint and adapter compatibility; matching tensor dimensions does not prove that an adapter belongs to a different base revision. Store adapter metadata as part of the model artifact so loading a compact file does not silently select the wrong computation.