Convolution and inductive bias

For an input shaped B by C_in by H by W, a dense two-dimensional convolution has weights shaped C_out by C_in by K_h by K_w. At each output location it multiplies a local patch by the same filter bank and sums over input channels and kernel positions. Most deep learning APIs implement cross-correlation without reversing the kernel, although the layer is called convolution. The learned parameter count is unaffected by the image width and height; the amount of computation is not.

Locality says each output initially sees a small neighborhood. Weight sharing says the detector is reused spatially. On an infinite grid, stride-one shared filtering commutes with integer translation. Finite boundaries, padding choices, striding, and subsequent pooling qualify that statement. A classifier can become approximately invariant to position through aggregation and training, but a convolutional feature map is generally equivariant rather than invariant: translating an input translates its internal response instead of leaving every response unchanged.

Stacked stride-one three-by-three layers enlarge the theoretical receptive field by two pixels per axis per layer. Nonlinearities between layers add representational capacity. The effective influence learned from data can be concentrated in a smaller region, so the theoretical receptive field is only a connectivity bound. Dilation expands spacing between sampled positions without increasing the number of filter coefficients. It can also create sampling artifacts that dense connectivity would not produce.

A standard convolution performs one dot product of length C_in times K_h times K_w for each output channel and spatial position. Count multiply-accumulate operations explicitly because FLOP conventions differ: one multiply plus one add is often counted as two floating-point operations. Depthwise convolution filters each channel separately; a following pointwise layer mixes channels. This changes both capacity and arithmetic. Lower arithmetic does not ensure lower latency when memory traffic, small kernels, or hardware utilization dominate.

Local shared patterns are useful for many images and time signals, but they are assumptions. A task depending on distant interactions may need depth, dilation, global pooling, or another mixing mechanism. A coordinate-dependent sensor calibration may require position information because complete weight sharing alone cannot express arbitrary location-specific rules. On irregular graphs, neighboring array indices may not correspond to meaningful physical neighbors at all. Start by stating what a one-step neighborhood means.

When comparing a convolutional model with a token mixer, align input resolution, augmentation, training examples, and compute budgets. Report memory at the intended batch size and measure the deployed kernel implementation. Strided downsampling can alias high-frequency content; anti-aliasing or different sampling choices alter behavior, rather than simply improving every model. Use controlled translations and boundary cases to test the invariance you actually need, including whether a one-pixel shift changes a safety-relevant prediction.