research29 min studyLesson 648 in Physics

Response matrices and unfolding regularization

Define migration and efficiency in a forward response, diagnose amplified inverse fluctuations, and quantify a simple regularization bias.

Before you begin

On this page

The question for today

What you’ll understand

Specify response-matrix indices and column normalization.
Invert a two-bin response and calculate fluctuation amplification.
Explain regularization’s bias–variance tradeoff and the role of forward folding.

Finite detector resolution moves events between reconstructed bins, and inefficiency removes some events entirely. An observed histogram is therefore a distorted view of the particle-level distribution. Unfolding estimates that distribution by solving an inverse problem, whose unstable directions can magnify small statistical fluctuations. Regularization suppresses those directions by adding structure or limiting iteration, at the cost of bias. We will make the response convention explicit, solve a two-bin synthetic example, and show why a smooth unfolded curve needs a covariance and a bias assessment rather than visual trust alone.

Write the forward response before inverting it

Let truth-bin yields be t j and reconstructed-bin expected counts be nu i. Define R i j as the probability that an event in truth bin j is selected and reconstructed in bin i. Columns sum to the truth-bin selection efficiencies and can therefore sum to less than one. A different convention conditions on selection and normalizes each column to one; it must then include efficiency separately. Stating both the index order and normalization prevents accidental transposition or double efficiency correction.

Backgrounds add to reconstructed expectations, while selected signal originating outside the reported truth range needs its own feed-in convention. The response is typically estimated from finite simulation and depends on within-bin spectra and detector assumptions. Fine binning does not remove that dependence automatically. Large migrations, small efficiencies, or nearly indistinguishable response columns make some truth combinations poorly identifiable from the observed data, even when the forward prediction itself is numerically stable.

\nu_i=\sum_jR_{ij}t_j+b_i,\qquad \sum_iR_{ij}=\epsilon_j\le1,\qquad \widehat{\mathbf t}=R^{-1}(\mathbf n-\mathbf b),\qquad V_t=R^{-1}V_n(R^{-1})^{\mathsf T}

The inverse and covariance formulas assume a square invertible known R and known background, without positivity or regularization constraints. General response matrices can be rectangular or rank deficient. R is dimensionless and t, nu, n, and b are event yields or counts.↔ Scroll wide equations horizontally.

Small singular values amplify uncertain directions

A singular-value decomposition separates truth combinations by how strongly they change reconstructed expectations. Dividing a small reconstructed fluctuation by a small singular value creates a large truth fluctuation. The inverse can thus return oscillatory or negative estimates even when all observed counts are nonnegative. A negative estimate need not be a software error; it can result from an unconstrained linear inverse with insufficient information. Physical constraints change the estimator and its sampling distribution.

Regularization can penalize roughness, shrink selected singular directions, or stop an iterative procedure before it fully fits fluctuations. A quadratic penalty is useful for illustration, but it embodies a preference about the truth shape. Choosing its strength after seeing the desired features can make uncertainty claims misleading. Smoothing reduces variance by accepting bias for some possible truths; it does not recover information that was never present in the response.

\widehat{\mathbf t}_\lambda=\underset{\mathbf t}{\operatorname{argmin}}\left[(R\mathbf t-\mathbf y)^{\mathsf T}V^{-1}(R\mathbf t-\mathbf y)+\lambda\|D\mathbf t\|^2\right],\qquad \operatorname{MSE}=\operatorname{Var}+\operatorname{Bias}^2

Y is the background-corrected data vector in a Gaussian approximation, D selects the penalized differences, and lambda has whatever units make the objective dimensionless. The scalar MSE identity applies to each estimator component and emphasizes that a smaller variance alone does not prove a better reconstruction.↔ Scroll wide equations horizontally.

Publish enough information to test another model

Closure with the same generated truth used to construct the response can miss dependence on an incorrect prior spectrum or within-bin composition. Useful stress tests change the injected truth, detector response, and background assumptions, then examine bias and interval coverage across those alternatives. The uncertainty of a regularized estimator should reflect the declared procedure, including tuning choices and nuisance parameters. Ignoring correlations between unfolded bins can turn a statistically acceptable feature into an apparent disagreement.

For many physics tests, forward folding a proposed truth prediction through the response and comparing at detector level avoids introducing an unfolded intermediate estimator. It still needs a trustworthy response, backgrounds, and uncertainty model. Unfolded results can remain valuable for reuse when the target definition, regularization prescription, covariance, and relevant response information are available. Neither approach grants model independence simply by being expressed in particle-level units.

Work it through

Invert a two-bin migration

Use the synthetic response $R=\begin{pmatrix}3/4&1/4\\1/4&3/4\end{pmatrix}$ with no losses or background. Observed counts are 90 and 70. Recover the unconstrained truth estimate, then find its change if reconstructed counts change by +2 and −2.

Each column sums to one, so total yield is preserved. The determinant is $9/16-1/16=1/2$ and $R^{-1}=\begin{pmatrix}3/2&-1/2\\-1/2&3/2\end{pmatrix}$ .
Multiplication gives truth estimates $135-35=100$ and $-45+105=60$ . Folding them back gives 90 and 70, verifying the arithmetic.
The perturbation maps to $((3/2)2+(-1/2)(-2),(-1/2)2+(3/2)(-2))=(4,-4)$ . The difference direction is amplified by two while the sum remains unchanged.

The result

The truth estimate is (100, 60). A reconstructed change (2, −2) becomes (4, −4), illustrating amplification along the migration-sensitive direction.

The living notebook

Explore the model

Interactive lab

Interactive lab · matrix

A small singular value weakens inversion

Reset, set Diagonal scale a = 0.5 and Shear b = 0, and predict selected-final singular values 1 and 0.5 with condition number 2. For a fresh run Reset, set a = 0.25 and b = 0, and predict condition number 4. Use Play or Step to inspect geometry: it interpolates from identity, while the readouts always describe the selected final matrix.

Interpolated geometryOriginal unit squareSelected final transform

Area scale · det A: 1.4
Singular values · σmax, σmin: 1.597, 0.876
Condition number · κ₂(A): 1.823

Diagonal scale a1.4Shear b0.6

t = 0 s

The selected matrix is A = [[a,b],[0,1]]. Playback interpolates geometry between the identity and A; readouts always describe A. Its determinant is a and its singular values come from AᵀA. Positive a makes every supported matrix invertible. This two-dimensional family is not a general eigensolver or a learned network. Parameter changes pause playback and recompute the model at the current elapsed time. Reset restores both parameters and returns time to zero. Playback starts paused. Adjust either parameter, then choose Play or Step.

Look forThis positive triangular matrix family illustrates conditioning, not a probability response or unfolding algorithm. It has no measured histogram, background, regularization, or detector-efficiency model.

Pause & reason

Check your understanding

Interactive checks need JavaScript. You can still read the explanation.

Read the checkpoint explanation

Regularization favors selected shapes or directions. This can improve mean-squared error, but variance reduction alone does not establish unbiasedness or correct interval coverage.

Your turn

Try it yourself

For the same two-bin response, the truth sum is fixed at 160 and truth difference is delta. The reconstructed difference is 20. Minimize the illustrative dimensionless objective $(0.5\delta-20)^2+0.25\delta^2$ . Find the regularized difference and the two truth estimates, and compare with the unregularized result.

Take a moment to make a prediction or work it out on paper. Open a hint when you need one.

Give me a hint

Differentiate the quadratic in delta, then use the fixed sum to recover each bin.

Compare with the solution

The derivative is $2(0.5)(0.5\delta-20)+0.5\delta=\delta-20$ , giving delta = 20. Thus truth estimates are $(160+20)/2=90$ and $(160-20)/2=70$ . Without the penalty, the reconstructed difference implies delta = 40 and truth (100, 60). Shrinking the difference reduces sensitivity to fluctuations but biases a truth whose actual difference is 40; the penalized solution deliberately does not fit the observed difference exactly.

Keep the idea

Take it with you

Response indices, efficiency normalization, and outside-range feed-in must be declared.
Small singular values amplify some reconstructed fluctuations during inversion.
Regularization, covariance, stress tests, and forward folding determine the interpretation of an unfolded result.

Read further

Original sources and deeper explanations to follow at your own pace.

PDG 2025 Statistics, section 40.2.7, pp. 13–15 (response, folding, unfolding, and regularization)

Optional progress, saved in this browser when available. Revisit any lesson at any time.

Lesson not marked complete.