Algorithms From Scratch

How to Implement Resampling Methods From Scratch In Python

Suresh Madhra·Feb 24, 2026·11 min read

Estimating performance on data you do not have

A model's score on its training data is not a performance estimate — it is a memory test. Resampling methods carve your dataset into pieces so that every score is computed on rows the model has never seen. Which method you choose is a trade-off between bias, variance and compute.

1. Train/test split

Split once, typically 60/40 or 80/20. Cheap and unbiased, but high variance: with a small dataset, a different random split can move your score by several points.

train_test_split.py

from random import seed, randrange

def train_test_split(dataset, split=0.60):
    train, copy = [], list(dataset)
    n_train = round(len(dataset) * split)
    while len(train) < n_train:
        train.append(copy.pop(randrange(len(copy))))
    return train, copy

seed(1)
dataset = [[i] for i in range(10)]
train, test = train_test_split(dataset)
print("Train:", [r[0] for r in train])
print("Test: ", [r[0] for r in test])

2. k-fold cross-validation

Split into k folds; each fold takes a turn as the test set. You get k scores, so you can report a mean and a spread. k = 5 or k = 10 are the standard choices — large enough to keep the training sets representative, small enough to run.

kfold.py

from random import seed, randrange

def cross_validation_split(dataset, n_folds=3):
    folds, copy = [], list(dataset)
    fold_size = int(len(dataset) / n_folds)
    for _ in range(n_folds):
        fold = []
        while len(fold) < fold_size:
            fold.append(copy.pop(randrange(len(copy))))
        folds.append(fold)
    return folds

seed(1)
dataset = [[i] for i in range(10)]
for i, fold in enumerate(cross_validation_split(dataset, 4)):
    print("Fold %d: %s" % (i + 1, [r[0] for r in fold]))
print()
print("Note: 10 rows / 4 folds leaves 2 rows unused. Real implementations distribute the remainder.")

3. Stratified folds

On classification problems, keep the class ratio the same in every fold. Without stratification a rare class can vanish entirely from a training fold, and that fold's score becomes meaningless.

stratified.py

from random import seed, shuffle

def stratified_folds(dataset, n_folds=5):
    by_class = {}
    for row in dataset:
        by_class.setdefault(row[-1], []).append(row)
    folds = [[] for _ in range(n_folds)]
    for label, rows in by_class.items():
        shuffle(rows)
        for i, row in enumerate(rows):
            folds[i % n_folds].append(row)
    return folds

seed(1)
dataset = [[i, 1 if i < 10 else 0] for i in range(100)]   # 10% positive
for i, fold in enumerate(stratified_folds(dataset)):
    pos = sum(1 for r in fold if r[-1] == 1)
    print("Fold %d: %d rows, %d positive (%.0f%%)" % (i + 1, len(fold), pos, 100.0 * pos / len(fold)))

4. Repeated k-fold and the bootstrap

Repeated k-fold runs the whole procedure several times with different shuffles and averages the results — the cheapest way to reduce the variance of your estimate. The bootstrap samples with replacement, leaving roughly 36.8% of rows out of each sample to serve as the test set, and gives you a genuine confidence interval.

bootstrap.py

from random import seed, randrange

def subsample(dataset, ratio=1.0):
    n = round(len(dataset) * ratio)
    return [dataset[randrange(len(dataset))] for _ in range(n)]

seed(1)
dataset = [[i] for i in range(20)]
sample = subsample(dataset)
in_sample = set(r[0] for r in sample)
out_of_bag = [r[0] for r in dataset if r[0] not in in_sample]
print("Sample size:", len(sample), "unique:", len(in_sample))
print("Out-of-bag rows:", out_of_bag)
print("Out-of-bag fraction: %.1f%% (theory says ~36.8%%)" % (100.0 * len(out_of_bag) / len(dataset)))

Which method, when

Thousands of rows or more, expensive model: single train/test split, plus a held-out validation set.
Hundreds to low thousands: 10-fold cross-validation, stratified for classification.
Very small datasets: repeated stratified k-fold, or leave-one-out if you can afford it.
Time series: never shuffle. Use forward-chaining splits where the test window is always after the training window.
Grouped data (multiple rows per customer): split by group, or the same customer appears on both sides.

← Back to blog