Margins and surrogate calibration

Let R01(f) denote actual zero-one classification risk under a fixed tie rule, such as predicting positive at score zero. For labels plus or minus one and score f(x), the signed margin is y times f(x). Positive margin means the sign is correct; negative margin means it is wrong. For a conservative margin certificate, count zero margins as errors; this is an upper bound on R01, not its definition, because a tie-rule prediction can be correct. The ramp at scale rho is the value one minus margin divided by rho, clipped to[0,1]. It is one at nonpositive margins, zero at margins at least rho, and changes linearly between them.

For a predetermined positive rho and a measurable score class with finite Rademacher complexity, the ramp is 1/rho-Lipschitz and bounded. A uniform Rademacher deviation bound therefore controls its population expectation by its empirical expectation, a complexity penalty and a confidence term. The empirical ramp is at most the fraction of margins strictly below rho, giving the displayed certificate. The scale must be fixed independently of this theorem’s data or selected with an additional uniform argument; choosing the best rho after seeing a bound is another selection step.

Multiplying every score by a positive number leaves its decision boundary and signs unchanged. It increases the numerical margins, but also scales the score class’s Rademacher complexity. A meaningful margin comparison therefore needs a norm budget or another control on the score scale. Selecting a giant scaling constant can make a fixed dataset look confidently classified without reducing classification error or establishing a better population bound. In a norm-controlled linear class, the relevant relationship involves margin relative to the input and weight budgets.

The hinge loss is the positive part of one minus margin. Unlike the ramp it is convex, but it is not bounded over unrestricted scores. Using a bounded-loss generalization theorem for hinge loss requires a score range or a different tail argument. Its pointwise upper bound on classification error does not by itself make a small excess hinge risk imply a small excess classification risk: excess compares each risk to its own optimum. That additional comparison is the purpose of calibration.

For hinge loss, the classification excess risk of a measurable score is at most its excess hinge risk relative to the infimum over all measurable scores, assuming the relevant risks are finite. This follows from checking each conditional label probability: choosing the wrong sign incurs hinge excess at least the conditional classification penalty. It is a population relationship and does not say that every empirical hinge minimizer generalizes. Sampling control and optimization control still have to be supplied.

A restricted score family may not approximate the pointwise hinge-optimal score, even when it can represent a good classifier’s signs. Thus a theorem comparing excess hinge risk only to the best member of that family cannot silently replace the unrestricted hinge optimum in the calibration statement. Its surrogate approximation error must be included. Other convex surrogates have different calibration transformations and rate conversions, so convexity alone is an insufficient substitute for analyzing the actual loss and the classification target.