Scaling and numerical stability
Avoid misleading results from incompatible units and unstable arithmetic.
- Standardize with training statistics
- Handle constant features
- Compute a stable softmax
Units change the geometry
If one feature is measured in millions and another in fractions, distance calculations and gradient updates can be dominated by the larger scale. Standardization subtracts a training mean and divides by a training standard deviation.
A constant feature has zero standard deviation. Handle it explicitly rather than dividing by zero. Save the training statistics and apply the same transformation during validation and inference. Recomputing them on each request changes the meaning of the input.
Equivalent formulas can behave differently
Softmax converts a vector of logits into positive weights that sum to one. Directly exponentiating very large logits can overflow. Subtracting the largest logit from every logit preserves the mathematical result while keeping exponentials bounded above by one.
Numerical stability is separate from calibration. A correctly computed softmax can still produce overconfident predictions. Inspect finite values and reasonable ranges, then evaluate whether the resulting scores match observed outcomes.
A small experiment you can run.
Subtracting the maximum avoids math.exp(1001), which would overflow. It leaves the ratios between exponentials unchanged.
import math
def softmax(logits):
if not logits or not all(math.isfinite(x) for x in logits):
raise ValueError("Use nonempty finite logits")
largest = max(logits)
exps = [math.exp(x - largest) for x in logits]
total = sum(exps)
return [x / total for x in exps]
weights = softmax([1000., 1001., 999.])
print([round(x, 4) for x in weights])
assert abs(sum(weights) - 1) < 1e-12
Save the file, open your terminal in that folder, and run python scaling-and-numerical-stability.py. Use python3 or py if required by your installation. Setup guide
The weights are approximately 0.2447, 0.6652, and 0.0900.
Test shift invariance.
- Run softmax on [0, 1, -1].
- Compare with [1000, 1001, 999].
- Explain why adding the same constant should not change the probabilities.
Compare with a suggested solution
All exponentials gain the same multiplicative factor, which cancels in the denominator. A stable implementation produces matching results within floating-point tolerance.
One idea to take with you.
Make it part of your progress.
Finish the practice and answer the knowledge check to mark this lesson complete.
Go deeper with primary documentation
Optional references for further study. This lesson and its examples were written for Artificials.
Python math module