research30 min studyLesson 944 in Physics

Histogram reweighting and overlap limits

Transport ensemble averages between nearby couplings while diagnosing weight concentration, unsampled support, and phase cancellation.

On this page

What you’ll understand

  • Derive the sign of the Ising coupling reweighting factor.
  • Compute a self-normalized estimate and a positive-weight concentration diagnostic.
  • Separate overlap failure from a sign or complex-phase problem.

One equilibrium trajectory can contain information about a neighborhood of its simulated coupling. Histogram reweighting changes the relative statistical weight of configurations already represented in that trajectory. It does not create new configurations in a phase the simulation never visited. This distinction matters most near coexistence, where small coupling changes can transfer most probability between separated energy ranges. We first assume a real positive target measure and a common finite configuration space, then examine why a fluctuating sign or phase introduces an additional cancellation problem.

A ratio of partition sums gives the estimator

At zero external field, write B for the sum of neighboring spin products in the once-counted graph convention. The target action is minus K times B. Increasing K therefore favors larger B, so the reweighting factor carries a positive coupling difference in its exponent. Dividing the reweighted observable average by the average weight cancels the unknown ratio of partition functions. An additive action constant also cancels from this normalized estimator if treated consistently in numerator and denominator.

A histogram of B is sufficient for observables that depend only on B; more general observables need their joint measurements or conditional averages in each bin. Reweighting a nonlinear statistic requires reweighting its primary moments and then reconstructing the statistic. For example, averaging a configuration-level Binder-like ratio is not the same operation as forming a ratio of reweighted ensemble moments. The self-normalized finite-sample estimator generally has ratio bias, even though the infinite-ensemble identity is exact.

B(s)=∑⟨ij⟩sisj,w(s)=e(K−K0)B(s),⟨O⟩K=⟨Ow⟩K0⟨w⟩K0,O^K=∑tOtwt∑twtB(s)=\sum_{\langle ij\rangle}s_is_j,\qquad w(s)=e^{(K-K_0)B(s)},\qquad \langle O\rangle_K=\frac{\langle Ow\rangle_{K_0}}{\langle w\rangle_{K_0}},\quad\widehat O_K=\frac{\sum_tO_tw_t}{\sum_tw_t}
K, K₀, B, and w are dimensionless. Both ensembles use the same graph, counting measure, and external field; changing another coupling adds its corresponding action difference.↔ Scroll wide equations horizontally.

Weight concentration is a warning, not an overlap certificate

For positive weights, a commonly quoted effective size compares the square of their sum with the sum of their squares. Equal weights give the number of entries, while a single dominant entry gives approximately one. This measures empirical concentration among observed configurations. It is not the autocorrelation-based effective size for an arbitrary reweighted observable, and multiplying two unrelated effective-size diagnostics does not generally yield a justified error estimate. Joint blocked analysis of numerator and denominator remains necessary.

A large empirical effective size can still be misleading if every recorded configuration misses the region that dominates the new target. No manipulation of existing weights can reveal that absent support. In a well-behaved single energy peak, a rough local warning scale is coupling change times the standard deviation of B becoming appreciable, but this is not a universal sufficiency bound. Inspect overlapping histograms from additional couplings and independent runs. Multi-histogram combinations extend coverage only when their sampled regions connect adequately.

nw=(∑twt)2∑twt2,1≤nw≤n(wt>0)n_w=\frac{(\sum_tw_t)^2}{\sum_tw_t^2},\qquad 1\le n_w\le n\quad(w_t>0)
n_w is a positive-weight concentration diagnostic. The bounds are algebraic; they do not establish thermalization, autocorrelation correction, or overlap with target-important states absent from the data.↔ Scroll wide equations horizontally.

Sign cancellation adds a different difficulty

When a formal target weight has a sign or complex phase, sample its absolute magnitude only if that defines a normalizable accessible positive measure. The observable then becomes a ratio containing the fluctuating phase in both numerator and denominator. A small average phase makes the ratio sensitive to cancellation and correlated noise. The positive-weight concentration formula cannot simply be applied to signed weights as if they were independent probabilities. Complex-Langevin or other alternative methods require separate correctness arguments and are not implied by this identity.

Even independent phase measurements can require a large sample to resolve a small denominator. For a sign taking values plus or minus one, its variance is one minus the square of its mean. Keeping a fixed relative uncertainty as that mean shrinks costs roughly its inverse square in independent measurements. In some systems the mean phase also falls exponentially with volume, making the expense severe. This is a statement about a specified representation and regime, not a proof that every physical model has an identical unavoidable numerical cost.

⟨O⟩=⟨Oeiθ⟩∣w∣⟨eiθ⟩∣w∣,SE⁡(s‾)=1−⟨s⟩2n(s=±1 independent)\langle O\rangle=\frac{\langle Oe^{i\theta}\rangle_{|w|}}{\langle e^{i\theta}\rangle_{|w|}},\qquad \operatorname{SE}(\overline s)=\sqrt{\frac{1-\langle s\rangle^2}{n}}\quad(s=\pm1\text{ independent})
The phase identity assumes a nonzero partition-function ratio and well-defined absolute-weight averages. The standard error formula concerns independent signs only; an actual chain requires correlation corrections.↔ Scroll wide equations horizontally.

Reweight an exactly enumerated two-spin ensemble

Enumerate the four states of a two-spin, one-bond graph at K0=0K_0=0. Reweight to K=(ln⁡2)/2K=(\ln2)/2. Find the spin-product expectation and the usual positive-weight concentration diagnostic using one copy of each state.

  1. The two aligned states have B=1B=1 and weight 2\sqrt2 each; the two antialigned states have B=−1B=-1 and weight 1/21/\sqrt2 each. Their total weight is 323\sqrt2.

  2. The weighted spin-product numerator is 22−2/2=22\sqrt2-2/\sqrt2=\sqrt2. Dividing by the total yields a correlation of one third, agreeing with direct enumeration of the target.

  3. The squared-weight sum is 2(2)+2(1/2)=52(2)+2(1/2)=5, while the squared total is 18. Thus nw=18/5=3.6n_w=18/5=3.6. Here the state enumeration is exact; this diagnostic is an algebraic weight-concentration comparison, not a measured statistical error for four random samples.

The result

The target correlation is one third and the positive-weight concentration diagnostic is 3.6 for the four exactly listed configurations.

Explore the model

Interactive lab

Interactive lab · probability

A finite sample can miss a rare region

Reset and set success probability p=0.1p=0.1, trials n=10n=10; predict zero successes with probability 0.9100.9^{10}. Repeat from Reset with both controls p=0.1p=0.1, n=20n=20; the zero-count bar becomes 0.9200.9^{20}. These exact independent binomial probabilities illustrate missing a rare outcome. They are not sampled energy histograms, field configurations, or reweighted Monte Carlo estimates.

A finite sample can miss a rare regionThese are exact binomial probabilities for independent trials with one fixed success probability. Playback tours the distribution; it does not resample outcomes. Current readings: Mean · np: 5; Variance · np(1−p): 2.5; Total probability: 1. All parameters are adjustable with the labeled controls below.03581000.070.150.220.3Successes kP(X = k)k = 0 · P = 9.77e-4
Exact probabilityHighlighted outcomeMean np
Mean · np
5
Variance · np(1−p)
2.5
Total probability
1
t = 0 s

These are exact binomial probabilities for independent trials with one fixed success probability. Playback tours the distribution; it does not resample outcomes. Parameter changes pause playback and recompute the model at the current elapsed time. Reset restores both parameters and returns time to zero. Playback starts paused. Adjust either parameter, then choose Play or Step.

Look forA region with nonzero target probability can be absent from a finite record; reweighting cannot manufacture its missing observations.

Check your understanding

What does a large positive-weight diagnostic n_w establish by itself?

Try it yourself

Assume independent signs with true mean 0.05. Estimate the standard error of their sample mean for n = 400 and n = 40000, and compare the relative errors of the reweighting denominator.

Take a moment to make a prediction or work it out on paper. Open a hint when you need one.

Give me a hint

Use sign variance 1 − 0.05², then divide the standard error by the mean magnitude.

Compare with the solution

The variance is 0.9975. With 400 signs, standard error is 0.9975/400≃0.04994\sqrt{0.9975/400}\simeq0.04994, nearly the entire mean, giving relative error about 0.999. With 40000 it is about 0.004994, giving relative error about 0.0999. A hundredfold increase in independent measurements improves the uncertainty tenfold. Autocorrelation or a correlated numerator would add further analysis; this denominator calculation alone does not supply a trustworthy error bar for the full observable ratio.

Take it with you

  • The reweighting exponent follows the declared action convention.
  • Observed weight concentration, time correlation, and missing overlap are different diagnostics.
  • A small average sign or phase creates cancellation noise that positive-weight bookkeeping does not remove.

Read further

Original sources and deeper explanations to follow at your own pace.

Optional progress, saved in this browser when available. Revisit any lesson at any time.

Lesson not marked complete.