Lepski adaptation and model selection

Consider scalar estimates $T_0,\ldots,T_{J-1}$ of the same target θ, ordered from coarse to fine. Let deterministic positive thresholds $s_j$ be nondecreasing with resolution. Select the smallest j such that $|T_j-T_k|\le4s_k$ for every finer k. The finest candidate always passes because it has no finer comparisons. This simplified scalar rule teaches the comparison mechanism; function-valued methods replace absolute differences by a chosen norm and require appropriate stochastic majorants.

A coarse candidate usually has low noise but may retain substantial smoothing bias. A fine candidate may be less biased but too noisy to serve as an oracle truth. The comparison therefore uses the fine candidate's uncertainty rather than demanding exact agreement. Looking only at neighboring candidates can miss accumulated drift, so this rule checks every finer candidate. Other Lepski and Goldenshluger–Lepski procedures use different estimators, comparison operators and penalties; the numerical constant here belongs to the elementary argument below, not to all published variants.

Assume deterministic bias bounds $|\mathbb ET_j-\theta|\le B_j$ decrease with j. Work on the event where $|T_j-\mathbb ET_j|\le s_j$ for every j. If a benchmark j* satisfies $B_{j*}\le s_{j*}$, then every finer k has bias at most $s_{j*}\le s_k$. Triangle inequalities bound $|T_{j*}-T_k|$ by $2s_{j*}+2s_k\le4s_k$. Thus j* passes and the selected index is no finer than j*.

If the selected index is strictly smaller, its defining comparison with j* bounds their difference by four times the benchmark threshold. The benchmark itself is within twice its threshold of the target, giving an error bound of six times that threshold for the selected estimate. If the selected index equals the benchmark, the smaller two-threshold bound applies. This deterministic reasoning explains the oracle mechanism but does not automatically identify the best benchmark or integrate risk over the complementary event.

For finitely many Gaussian candidates with known marginal standard deviations, a union bound supplies simultaneous thresholds by multiplying each standard deviation by the square root of twice the logarithm of twice the number of candidates divided by delta. Independence between the candidates is unnecessary for that bound. It is the marginal Gaussian tail bound that matters. In nonparametric function problems, supremum-norm concentration and unknown variances can require substantially more work than this scalar illustration.

A full adaptive estimator aims to approach the best bias–noise balance over several smoothness classes without knowing which one contains the truth. Some settings pay logarithmic costs, and some permit rate adaptation without such a cost under their particular loss. Model selection by penalized risk or cross-validation follows a related complexity tradeoff but is not identical to this comparison rule. Most importantly, selecting a good center for estimation does not make the same selected noise threshold an honest confidence radius: the unknown bias and selection event still have to be addressed.