Algorithms From Scratch

How to Scale Machine Learning Data From Scratch With Python

Suresh Madhra·Mar 03, 2026·8 min read

Why scaling matters

Many algorithms treat every input as if it lived on the same ruler. Gradient descent takes a step proportional to the input value; k-nearest neighbours measures distance; neural networks initialise weights assuming inputs are small. Feed them a column measured in thousands next to a column measured in tenths and the big column dominates the model for reasons that have nothing to do with predictive power.

Two rescalings cover almost everything: normalization to the 0–1 range, and standardization to zero mean and unit variance.

1. Normalization (min–max)

x^{'} = \frac{x - x _{m i n}}{x _{m a x} - x _{m i n}}

Use it when you know sensible bounds, when the distribution is not Gaussian, or when the algorithm expects bounded inputs (sigmoid outputs, image pixels). The cost: it is sensitive to outliers, since a single extreme value squashes everything else.

normalize.py

def dataset_minmax(dataset):
    return [[min(r[i] for r in dataset), max(r[i] for r in dataset)]
            for i in range(len(dataset[0]))]

def normalize_dataset(dataset, minmax):
    for row in dataset:
        for i in range(len(row)):
            lo, hi = minmax[i]
            row[i] = (row[i] - lo) / (hi - lo) if hi != lo else 0.0

dataset = [[50, 30], [20, 90], [30, 50]]
minmax = dataset_minmax(dataset)
print("Min/Max per column:", minmax)
normalize_dataset(dataset, minmax)
for row in dataset:
    print([round(v, 3) for v in row])

2. Standardization (z-score)

x^{'} = \frac{x - μ}{σ}

Use it when the column is roughly Gaussian, when you need unbounded values, or when the algorithm assumes centred data (linear and logistic regression, PCA, SVMs). Outliers still move the mean, but far less violently than they move a min or a max.

standardize.py

from math import sqrt

def column_means(dataset):
    return [sum(r[i] for r in dataset) / float(len(dataset))
            for i in range(len(dataset[0]))]

def column_stdevs(dataset, means):
    stdevs = []
    n = len(dataset)
    for i in range(len(dataset[0])):
        variance = sum((r[i] - means[i]) ** 2 for r in dataset) / float(n - 1)
        stdevs.append(sqrt(variance))
    return stdevs

def standardize_dataset(dataset, means, stdevs):
    for row in dataset:
        for i in range(len(row)):
            row[i] = (row[i] - means[i]) / stdevs[i] if stdevs[i] else 0.0

dataset = [[50, 30], [20, 90], [30, 50]]
means = column_means(dataset)
stdevs = column_stdevs(dataset, means)
print("Means: ", [round(m, 3) for m in means])
print("Stdevs:", [round(s, 3) for s in stdevs])
standardize_dataset(dataset, means, stdevs)
for row in dataset:
    print([round(v, 3) for v in row])

3. The mistake everyone makes once

Compute the statistics on the training set only, then apply them to the test set. Scaling the whole dataset before splitting leaks information about the test distribution into training and inflates your score — sometimes dramatically, always misleadingly.

no_leakage.py

def fit_minmax(train):
    return [[min(r[i] for r in train), max(r[i] for r in train)]
            for i in range(len(train[0]))]

def apply_minmax(dataset, minmax):
    out = []
    for row in dataset:
        scaled = []
        for i, v in enumerate(row):
            lo, hi = minmax[i]
            scaled.append((v - lo) / (hi - lo) if hi != lo else 0.0)
        out.append(scaled)
    return out

train = [[10.0, 1.0], [20.0, 2.0], [30.0, 3.0]]
test  = [[40.0, 4.0]]           # unseen, and outside the training range

stats = fit_minmax(train)        # fit on TRAIN only
print("Train scaled:", apply_minmax(train, stats))
print("Test scaled: ", apply_minmax(test, stats))
print()
print("The test value scales above 1.0 — that is correct and expected.")
print("Clipping it, or refitting on test, is how leakage sneaks in.")

Which one should you use?

Neural networks and gradient descent: standardize, or normalize if inputs are naturally bounded.
k-NN, k-means, SVM with RBF: always scale — distance metrics are meaningless otherwise.
Decision trees, random forests, gradient boosting: scaling changes nothing; splits are order-based.
Regularized linear models: standardize, otherwise the penalty punishes large-scale features unfairly.

← Back to blog