Two ways a backtest lies to you (and how to avoid them)

You test a strategy, or a model, on historical data. The backtest gives a pretty number. And then, live, it doesn't show up. It's almost always one of these two illusions — and both are ruled out with very little code.

I packaged the two corrections as a library: honest-eval, pure Python, no dependencies. They came out of a trading bot, but the rigor has nothing specific to trading.

Illusion 1: the model saw the future

Splitting the data with the classic random train_test_split is correct for independent data. On a time series it's a silent disaster: it puts tomorrow's samples into the training set, and the model "predicts" in the test things that in production wouldn't have happened yet. The metric comes out inflated, and you trust an edge that doesn't exist.

The honest test is always the future: the most recent stretch in time.

from honest_eval import temporal_split

train_idx, test_idx = temporal_split(timestamps, test_frac=0.20, embargo=24)
X_tr, X_te = X[train_idx], X[test_idx]

It returns indices, so you apply it to numpy, pandas or lists alike.

The embargo closes a subtler leak: if your label looks h steps ahead, a training sample within h of the cut already knows part of the test's outcome. embargo=h discards that edge. The metric drops — but it's finally the real out-of-sample one.

Illusion 2: the variant won by luck

You have several variants and you want the best. You pick the one with the highest mean. Mistake: with few samples, that rewards variance, not edge. The noisiest variant usually ends up on top by chance.

Two corrections, both inside select_best_variant:

Pair them. Measure variant and baseline on the same trial and work with δ = variant − baseline. The trial's common variance cancels in the subtraction, and you're left with the signal.

Require a lower confidence bound > 0. Promote a variant only if mean − z·SE > 0: "even being pessimistic within the confidence margin, it stays above the baseline".

from honest_eval import select_best_variant

variants = {
    "chandelier": [0.8, 1.1, -0.2, 0.9, 1.0, 0.7],
    "momentum":   [2.0, -1.5, 3.0, -0.5, 1.2, -1.0],   # high mean, very dispersed
}

elegido = select_best_variant(variants, z=1.6449, min_effective_n=5)
politica = elegido.name if elegido else "baseline"

momentum may have the highest mean and still fail to promote: its dispersion sinks the lower bound. That's exactly what you want to happen — it's the mechanism that rejects the lucky winner.

When recent data weighs more

If the process changes over time, weight by recency. Kish's effective sample size (Σw)² / Σw² keeps a few heavily-weighted samples from passing themselves off as many:

from honest_eval import halflife_weight

weights = [halflife_weight(now - t, halflife=7*86400) for t in exit_ts]
variants = {"chandelier": (deltas, weights)}

They don't create signal — they stop manufacturing it

Neither of them invents edge: that's data and good features. What they do is stop manufacturing it where there is none. They're the minimum for not lying to yourself before risking something real.

It's free software. Since August 2026 the code is not published to registries or on GitHub — we work in security, and having our source extracted would be an argument against the product —: it's delivered on request, signed and with a SHA-256, at contacto@xiliux.com. The honest-eval repo has the description and how to request it.