LoRA rank is a parameter budget, not a quality guarantee

Use column-vector convention: a base weight W₀ maps din inputs to dout outputs. Choose A with shape r×din and B with shape dout×r, so BA has the same shape as W₀. Standard LoRA adds a scaled BA update. Libraries may store some matrices transposed or provide different module wrappers, so verify actual target names and conventions. The formula here excludes biases, embedding-specific adapters, DoRA, and other variants unless explicitly added to the budget.

The rank bound limits the update matrix, not the rank of the complete adapted model or a universal set of behaviors it can learn. Increasing r adds parameters but can change optimization and overfitting as well as capacity. Targeting attention projections, MLP matrices, or another module changes which computations adapt. A configuration with a valid rank and no matching target modules can train nothing useful. Inspect matched modules and actual trainable tensors before starting a long job.

For one ordinary dense target, the two factors contain r(din+dout) parameters instead of din×dout fully trained entries. Sum that count across actual matched targets, including any separately trained biases or saved modules. Trainable parameter count alone omits frozen base storage and forward activations. Depending on optimizer and precision, gradients, moment estimates, master copies, temporary buffers, and distributed communication can add substantial memory. Measure peak live use at representative sequence lengths and batch settings.

QLoRA combines low-bit frozen-base storage with trainable adapters under a defined computation and quantization scheme. Four-bit payload arithmetic is not a complete GPU-memory estimate: scales, metadata, compute dtype, activations, and allocator/workspace behavior remain. Quantization and adaptation introduce implementation-dependent numerical changes that require evaluation. The original paper reports experiments under its own conditions;do not turn them into a promise that every dataset, rank, target module, or quantized merge preserves quality.

Compare rank and target selections with the same base, data, masks, token budget, and evaluation protocol. A larger adapter trained for more supervised tokens is not a clean rank-only comparison. Include retention tasks and task-specific failure cases, not just training loss. Record optimizer steps, learning-rate schedule, and effective target tokens because those can explain differences that otherwise look like representation capacity. Keep a safe baseline and stop configurations that violate memory or correctness checks.

The matrix lab shows a fixed two-dimensional transform rather than a trainable adapter. Reset and set both a=1, b=1, predict determinant 1 and a shear, then Step. Its matrix [[a, b], [0,1]] is not a general BA factorization, and playback changes geometry rather than learning weights. Use the diagram to track row/column orientation and input/output roles only. Calculate actual adapter factors separately;the visual cannot choose rank, target modules, or predict instruction quality.