Distribution shift and impossibility results

Let P be the labeled source distribution and Q the labeled target distribution. Under covariate shift, their conditional label distributions agree wherever target covariates occur, while their input marginals may differ. This is an assumption about the data-generating relationship; unlabeled target observations cannot generally verify it. If the label conditional itself changes, covariate reweighting alone is not a correction. A low source risk is therefore meaningful for transfer only after the relevant form of shift has been specified.

Assume the target input marginal is absolutely continuous with respect to the source marginal, with known density ratio w. Combining the shared conditional label law with change of measure gives target risk as source expectation of w times the loss, for integrable losses. The overlap condition prevents positive target mass on source-null regions. Estimating w from data introduces another source of error; using the exact identity with unknown or poorly estimated ratios is not a complete finite-sample guarantee.

For a predictor fixed independently of a source sample, known weights and iid observations give an unbiased weighted empirical estimate of target risk. If the original loss lies in[0,1] and weights lie in[0,W], the weighted loss lies in[0,W]. Hoeffding therefore pays a W-squared factor in its exponent. An unbounded density ratio does not permit this bounded-loss certificate. Selecting a predictor from the weighted sample again needs a class-uniform or algorithm-specific analysis, beyond the fixed-predictor estimate.

Large weights reflect rare source regions that matter more under the target distribution. They can increase variance even when the target-risk identity is exactly correct. Self-normalizing weights changes the estimator and usually introduces finite-sample bias, so it should not be substituted into the unbiasedness claim without analysis. A finite two-region calculation makes the two issues distinct: first compute the correct target expectation, then assess whether the source sample has enough information to estimate the heavily weighted part accurately.

Take inputs a and b. The source puts all mass at a and always labels it zero, while the target puts all mass at b. Consider two worlds that agree on this source distribution and the target input marginal, but assign target label zero in one world and one in the other. Every labeled source sample and every unlabeled target sample have exactly the same law in both worlds. Thus no learner receiving only those observations can distinguish the worlds, regardless of sample size.

If the learner predicts one at b with probability q, including its internal randomness, its target errors in the two worlds are q and one minus q. At least one is at least one half. A deterministic learner even has error one in one of the two worlds. This is an identifiability obstruction, not an optimization failure. The example violates overlap; without other restrictions it can be made consistent with a common deterministic labeling rule within each world by defining that rule on both points. Additional target labels, overlap or structural assumptions can exclude the indistinguishable pair, but must be stated explicitly.