The Only Loss Function Cheat Sheet You Need

The loss is the specification

Your model does exactly what the loss function asks. Every complaint about a model — it ignores rare cases, it is over-confident, it is wrecked by outliers — is usually a complaint about the loss. This is a working reference: what each loss optimises, when it is the right call, and what it does badly.

Regression losses

Mean squared error

Optimises the conditional mean. Smooth gradients everywhere, which optimisers love. Squaring means one outlier can dominate the entire batch — use it when large errors genuinely are disproportionately costly.

Mean absolute error

Optimises the conditional median, so it is robust to outliers. The gradient is constant in magnitude, which slows fine convergence near the optimum — decay the learning rate.

Huber

Quadratic near zero, linear in the tails: MSE's smooth convergence with MAE's robustness. δ is the outlier threshold, and it is worth tuning.

Quantile (pinball)

Asymmetric by design: train separate heads at τ = 0.1, 0.5, 0.9 and you get a prediction interval instead of a point estimate. The right default when under- and over-prediction have different costs.

def mse(e):   return e * e
def mae(e):   return abs(e)
def huber(e, d=1.0):
    return 0.5 * e * e if abs(e) <= d else d * (abs(e) - 0.5 * d)
def pinball(e, tau=0.9):
    return max(tau * e, (tau - 1) * e)

print("%8s %8s %8s %8s %10s" % ("error", "MSE", "MAE", "Huber", "Pinball.9"))
for e in [-4, -1, -0.2, 0.2, 1, 4]:
    print("%8.1f %8.2f %8.2f %8.2f %10.2f" % (e, mse(e), mae(e), huber(e), pinball(e)))
print()
print("At error 4, MSE charges 16 and Huber charges 3.5 — that ratio is the robustness.")
print("Pinball charges 3.6 for over-prediction and 0.4 for under — that asymmetry is the point.")

Classification losses

Binary cross-entropy

The default for two-class problems. It is a proper scoring rule, so it produces calibrated probabilities — and it punishes confident mistakes brutally, which is exactly what you want. Always compute it from logits for numerical stability.

Categorical cross-entropy

The multi-class generalisation, paired with a softmax. Add label smoothing (ε ≈ 0.1) when the model becomes over-confident or labels are noisy; it trades a little accuracy for much better calibration.

Focal loss

Down-weights examples the model already gets right, so a flood of easy negatives cannot drown the rare positives. Built for dense object detection; excellent for extreme class imbalance generally. γ = 2 is the usual starting point.

from math import log

def bce(p, y):
    p = min(max(p, 1e-12), 1 - 1e-12)
    return -(y * log(p) + (1 - y) * log(1 - p))

def focal(p, y, gamma=2.0, alpha=0.25):
    p = min(max(p, 1e-12), 1 - 1e-12)
    pt = p if y == 1 else 1 - p
    a = alpha if y == 1 else 1 - alpha
    return -a * ((1 - pt) ** gamma) * log(pt)

print("%-28s %8s %8s" % ("case", "BCE", "Focal"))
cases = [("easy negative  p=0.02 y=0", 0.02, 0),
         ("easy positive  p=0.98 y=1", 0.98, 1),
         ("hard positive  p=0.20 y=1", 0.20, 1),
         ("confident wrong p=0.99 y=0", 0.99, 0)]
for name, p, y in cases:
    print("%-28s %8.4f %8.4f" % (name, bce(p, y), focal(p, y)))
print()
print("Focal shrinks the easy cases towards zero so the hard positive dominates the gradient.")

Representation losses

• Contrastive: pull matched pairs together, push mismatched pairs apart beyond a margin. Simple, but sensitive to how you mine negatives.
• Triplet: anchor, positive, negative with a margin. Works well with semi-hard negative mining; collapses without it.
• InfoNCE / NT-Xent: treat the batch as a classification over candidates. The workhorse behind modern embedding and multimodal models — larger batches directly mean more negatives and better representations.
• Temperature is the hidden hyperparameter in all of them. Too low and training is unstable; too high and everything looks similar.

Choosing under pressure

• Need calibrated probabilities? Cross-entropy, and do not replace it with a metric surrogate.
• Severe imbalance? Focal loss or class weights — and change the metric too, not just the loss.
• Outliers in the target? Huber, or MAE if they are common.
• Need intervals? Quantile loss with multiple heads.
• Business cost asymmetry? Encode it directly in the loss weights; do not fix it with a threshold afterwards and hope.
• Optimising a non-differentiable metric (F1, AUC)? Train on a smooth surrogate and select the threshold on validation data.