Feature Engineering Still Beats Deep Learning on Tabular Data

Why the boring answer keeps winning

On structured business data — transactions, customers, sensors, claims — a gradient boosted tree fed thoughtfully constructed features remains extraordinarily hard to beat. Deep learning on tabular data has improved, but the gap it closes is usually smaller than the gap a good aggregation feature opens. Trees handle mixed types, missing values, monotone transformations and irrelevant columns natively; neural networks need you to solve all of that first.

This is not an argument against neural networks. It is an argument for spending your next four hours on features rather than architecture.

1. Aggregations are the highest-yield features

Most tabular problems are secretly relational: one row per event, one prediction per entity. Group-level summaries — count, mean, max, standard deviation, recency, and the ratio of the current row to its group mean — routinely add more than any model change.

transactions = [
    # customer, day, amount
    ("c1", 1, 20.0), ("c1", 3, 25.0), ("c1", 9, 400.0),
    ("c2", 2, 55.0), ("c2", 4, 60.0),
    ("c3", 1, 10.0),
]
TODAY = 10

groups = {}
for cust, day, amount in transactions:
    groups.setdefault(cust, []).append((day, amount))

print("%-8s %6s %9s %9s %9s %9s" % ("cust", "n", "sum", "mean", "recency", "amt/mean"))
for cust, rows in groups.items():
    amounts = [a for _, a in rows]
    n = len(amounts)
    total = sum(amounts)
    mean = total / n
    recency = TODAY - max(d for d, _ in rows)
    last_ratio = rows[-1][1] / mean
    print("%-8s %6d %9.2f %9.2f %9d %9.2f" % (cust, n, total, mean, recency, last_ratio))
print()
print("The ratio column is where the signal usually hides: c1's last spend is 8x their norm.")

2. Categorical encoding, ranked

• Low cardinality (under ~15): one-hot, or native categorical support in your booster.
• Medium cardinality: ordinal/label encoding is fine for trees — they split on order, not magnitude.
• High cardinality (postcodes, product ids): target encoding with out-of-fold means and smoothing, or count/frequency encoding, which leaks nothing.
• Never target-encode on the full dataset. Compute the encoding inside each cross-validation fold or you will inflate your score and be disappointed in production.

# Out-of-fold, smoothed target encoding — the safe way.
rows = [("north", 1), ("north", 1), ("north", 0), ("south", 0),
        ("south", 0), ("south", 1), ("east", 1), ("east", 1)]
n_folds, smoothing = 4, 10.0
prior = sum(y for _, y in rows) / len(rows)

def encode_out_of_fold(rows, n_folds, smoothing, prior):
    encoded = []
    for i, (cat, _) in enumerate(rows):
        fold = i % n_folds
        others = [(c, y) for j, (c, y) in enumerate(rows) if j % n_folds != fold]
        matching = [y for c, y in others if c == cat]
        n = len(matching)
        mean = sum(matching) / n if n else prior
        value = (n * mean + smoothing * prior) / (n + smoothing)
        encoded.append(value)
    return encoded

print("prior = %.3f" % prior)
for (cat, y), enc in zip(rows, encode_out_of_fold(rows, n_folds, smoothing, prior)):
    print("%-6s y=%d -> %.4f" % (cat, y, enc))
print()
print("Every value is computed without the row's own fold, and shrunk towards the prior.")

3. Dates are a feature factory

• Decompose: hour, day of week, day of month, month, quarter, is_weekend, is_holiday.
• Encode cyclical fields as sine and cosine pairs so December sits next to January.
• Compute durations, not timestamps: days since signup, days since last purchase, time to next scheduled event.
• Never feed a raw timestamp to a tree. It will split on it and learn a date range that will never occur again.

4. Ratios, differences and domain terms

Trees can only split on axis-aligned thresholds, so a relationship like debt divided by income takes many splits to approximate — but a single column to express. Give the model the ratios a domain expert would compute by hand: utilisation rates, price per unit, error rate per request, BMI, current value versus rolling average.

5. Leakage: the failure mode that looks like success

• Any feature computed after the prediction moment — a status field updated at resolution time is the classic offender.
• Statistics computed across the full dataset before splitting: scalers, target encodings, imputation values.
• Duplicate or near-duplicate rows straddling the split, common when one entity appears many times.
• Ids that correlate with the target because of how the data was collected or sorted.
• The tell: a suspiciously excellent validation score and one feature with overwhelming importance. Investigate before you celebrate.

6. When to actually reach for deep learning

• Very large datasets — millions of rows — where a network has room to learn interactions you did not encode.
• Free text, images or audio alongside your columns. Embed those and feed them in; this is a genuine tree weakness.
• High-cardinality categoricals with meaningful similarity structure that learned embeddings can capture.
• Multi-task or transfer settings where you want a shared representation across related targets.
• Even then: benchmark against a tuned gradient boosting baseline, and ensemble rather than replace. The two families make different errors, and averaging them is often the actual winner.

A practical order of work

Baseline with a gradient booster on raw columns. Add aggregations. Add ratios and date decompositions. Fix categorical encoding. Only then tune hyperparameters, and only after that consider a neural model. Teams that follow that order tend to stop at step three, because the score is already good enough.