Algorithms From Scratch

How To Implement Machine Learning Metrics From Scratch in Python

Suresh Madhra·Feb 17, 2026·9 min read

A prediction without a metric is an opinion

Once your model produces predictions you need one number that says whether they are any good — and the choice of number quietly defines what your model will optimise for. This tutorial implements the four metrics you will use most, in plain Python, so their behaviour holds no surprises.

1. Classification accuracy

accuracy = \frac{correct predictions}{total predictions} \times 100

Simple, intuitive, and dangerous on imbalanced problems. If 98% of transactions are legitimate, predicting "legitimate" every time scores 98%.

accuracy.py

def accuracy_metric(actual, predicted):
    correct = sum(1 for a, p in zip(actual, predicted) if a == p)
    return correct / float(len(actual)) * 100.0

actual    = [0, 0, 0, 0, 0, 1, 1, 1, 1, 1]
predicted = [0, 1, 0, 0, 0, 1, 0, 1, 1, 1]
print("Accuracy: %.1f%%" % accuracy_metric(actual, predicted))

imbalanced_actual    = [0] * 98 + [1, 1]
lazy_predictions     = [0] * 100
print("Lazy model on imbalanced data: %.1f%%" % accuracy_metric(imbalanced_actual, lazy_predictions))
print("...and it never once found the thing you care about.")

2. Confusion matrix

The confusion matrix is accuracy's honest cousin: it shows you which classes get confused with which. Everything else — precision, recall, F1 — is arithmetic on its four cells.

confusion_matrix.py

def confusion_matrix(actual, predicted):
    unique = sorted(set(actual))
    index = {value: i for i, value in enumerate(unique)}
    matrix = [[0 for _ in unique] for _ in unique]
    for a, p in zip(actual, predicted):
        matrix[index[a]][index[p]] += 1
    return unique, matrix

def print_confusion_matrix(unique, matrix):
    print("      " + " ".join("p=%-3s" % u for u in unique))
    for i, u in enumerate(unique):
        print("a=%-3s " % u + " ".join("%-5d" % v for v in matrix[i]))

actual    = [0, 0, 0, 0, 0, 1, 1, 1, 1, 1]
predicted = [0, 1, 0, 0, 0, 1, 0, 1, 1, 1]
unique, matrix = confusion_matrix(actual, predicted)
print_confusion_matrix(unique, matrix)

tn, fp = matrix[0]
fn, tp = matrix[1]
precision = tp / float(tp + fp)
recall = tp / float(tp + fn)
f1 = 2 * precision * recall / (precision + recall)
print()
print("Precision %.3f  Recall %.3f  F1 %.3f" % (precision, recall, f1))

3. Mean absolute error

MAE = \frac{1}{n} i = 1 \sum n ∣ \overset{y}{^}_{i} - y_{i} ∣

MAE is in the same units as your target and treats every unit of error equally. Report it when a stakeholder asks "how far off are we, typically?"

4. Root mean squared error

RMSE = \frac{1}{n} i = 1 \sum n (\overset{y}{^}_{i} - y_{i})^{2}

RMSE squares errors before averaging, so large misses hurt disproportionately. Use it when a single big error is much worse than several small ones. RMSE is always greater than or equal to MAE; the gap between them tells you how spread out your errors are.

regression_metrics.py

from math import sqrt

def mae_metric(actual, predicted):
    return sum(abs(p - a) for a, p in zip(actual, predicted)) / float(len(actual))

def rmse_metric(actual, predicted):
    total = sum((p - a) ** 2 for a, p in zip(actual, predicted))
    return sqrt(total / float(len(actual)))

def r_squared(actual, predicted):
    mean = sum(actual) / float(len(actual))
    ss_res = sum((a - p) ** 2 for a, p in zip(actual, predicted))
    ss_tot = sum((a - mean) ** 2 for a in actual)
    return 1 - ss_res / ss_tot

actual    = [0.1, 0.2, 0.3, 0.4, 0.5]
steady    = [0.11, 0.19, 0.29, 0.41, 0.5]
one_blowup= [0.1, 0.2, 0.3, 0.4, 1.5]

for name, pred in [("steady errors", steady), ("one blow-up", one_blowup)]:
    print("%-14s MAE=%.3f RMSE=%.3f R2=%.3f" %
          (name, mae_metric(actual, pred), rmse_metric(actual, pred), r_squared(actual, pred)))
print()
print("Same idea, very different penalties. Choose the one that matches the cost of being wrong.")

Choosing a metric

Balanced classification: accuracy is fine, confusion matrix is better.
Imbalanced classification: precision, recall, F1 or ROC AUC — never accuracy alone.
Cost-sensitive problems: weight the confusion matrix by the real cost of each error type.
Regression with outliers you care about: RMSE. Regression with outliers you do not: MAE.
Explaining to a business audience: MAE plus a percentage error, not R².

← Back to blog