Same Model, Ten Seeds, a 3.5-Point Swing: Why One Split Is Not a Result

Every model comparison you read has a hidden parameter: which rows landed in the test set. Most write-ups never mention it. They give one accuracy number, as if it were a property of the model. It is not. It is a property of the model plus one particular split.

I ran the test on purpose, on a dataset small enough that the effect is easy to see and honest enough that it cannot be waved away. The dataset is the Wisconsin diagnostic breast cancer set that ships with scikit-learn: 569 rows, 30 numeric features, 62.7% benign in the dataset encoding. Two models, both standard: logistic regression on standardized features, and a random forest with 300 trees. Ten seeds, 0 through 9. Each seed gives a stratified 75/25 split, so every test set has 143 rows. Everything else stays fixed.

The numbers

Split seed	Logistic regression	Random forest	Forest minus logistic
0	95.8%	94.4%	-1.4 points
1	96.5%	94.4%	-2.1 points
2	97.2%	96.5%	-0.7 points
3	98.6%	97.2%	-1.4 points
4	97.2%	97.9%	+0.7 points
5	97.9%	97.2%	-0.7 points
6	99.3%	96.5%	-2.8 points
7	97.9%	96.5%	-1.4 points
8	97.2%	98.6%	+1.4 points
9	97.9%	96.5%	-1.4 points

Summary across the ten splits: logistic regression averaged 97.6% with a standard deviation of 1.0 points, ranging from 95.8% to 99.3%. The random forest averaged 96.6% with a standard deviation of 1.3 points, ranging from 94.4% to 98.6%. The gap between the two models averaged -1.0 points in the forest's disfavour, but it ranged from -2.8 to +1.4 points, a 4.2-point swing in the comparison itself.

Read that last line again, because it is the whole article. The model comparison, the thing a reader actually cares about, moved 4.2 points without any model changing. On seeds 4 and 8, the forest wins. On the other eight, logistic regression wins. A writer who ran only seed 8 could publish "random forest beats logistic regression" with a straight face and a true number. A writer who ran only seed 6 could publish "logistic regression hits 99.3%" with a true number. Both would be misleading, and both numbers would be real.

Why the swing is this big

Three reasons, in order of importance.

First, the test set is small. With 143 test rows, one row changing sides moves accuracy by 0.7 points. The 3.5-point logistic range is five rows. The 2.8-point worst gap for the forest is four rows. When a leaderboard lead is two points on a test set this size, it is three rows, and three rows is not a finding.

Second, the split decides which hard rows get tested. Stratification keeps the class balance fixed, but it does not keep difficulty fixed. Some seeds put more borderline cases in the test set; some put them in training, where they help the model instead of hurting its score. That is luck, and it is baked into every single-split number.

Third, and this is the surprise in the run, the model seed barely mattered at all. I fixed the split at seed 42 and varied only the random forest's own random seed across ten values. All ten forests produced identical predictions on the test set: 137 of 143 correct, 95.8% every time, zero differing predictions. With 300 trees, the vote is stable enough that reshuffling the trees changes nothing here. So the noise people usually worry about, the model's internal randomness, was zero in this run, while the noise people usually ignore, the split, moved results by up to 4.2 points. If you set a model seed for reproducibility and then report one split, you have controlled the small source of variance and left the large one running free.

What to do instead

This is not a counsel of despair. Four habits fix most of it, and none of them is hard.

1. Report the distribution, not the draw. Run at least ten splits, or repeated cross-validation, and report the mean and standard deviation. "97.6% plus or minus 1.0 across ten splits" tells a reader something. "99.3%" tells them which seed you liked.
2. Compare models on the same splits. Paired comparisons, where both models see identical train/test partitions, cancel out the easy-split and hard-split luck. The paired gap here still ranged from -2.8 to +1.4 points, which is itself information: it says the two models are close, and the honest conclusion is "close", not a winner.
3. Fix the split before you tune. Pick your evaluation protocol, including the seed or seeds, before you start comparing models. Changing the protocol after seeing results is the same selection problem in a different coat.
4. Scale the test set to the claim. If your claimed improvement is 1 point, a 143-row test set cannot support it: one point is about one and a half rows. Either get more test data, or say the difference is within noise. A small honest claim beats a large fragile one.

Against and in favour

In favour

Single splits are cheap, standard, and easy to reproduce exactly. A fixed seed and a fixed split let anyone rerun your code and get your number, which is a real virtue. For large test sets, split noise shrinks to the point where one split is a reasonable summary. And when one model beats another by 10 points, no plausible split rescues the loser. This article is not an argument that all single-split results are wrong. It is an argument that small single-split gaps are unproven.

Against

One counterargument deserves a fair hearing: the models here are close in skill, so of course the winner flips. On a task where the models differ by a wide margin, the split would not change the ranking. That is true, and it cuts both ways. If your models are far apart, you do not need careful statistics to see it. If they are close enough that a paper or a blog post is worth writing about the difference, they are close enough for the split to decide it. The closer the contest, the more the protocol matters, and close contests are exactly where people report winners.

Another objection: real projects use cross-validation already, so this is a solved problem. Some do. Many notebook-level comparisons do not, and published model cards routinely report a single holdout number with no spread. The habit this article attacks is live and common, not a straw man.

What would make me wrong

Three findings would change my conclusion. First, if repeated cross-validation on this dataset gave a much smaller spread than the ten splits did, the split story would weaken; the ten-split spread is the evidence, and a tighter protocol result would supersede it. Second, if a larger test set, say the full 569 rows under cross-validation, kept a stable 2-plus-point forest lead, the "models are close" reading would be wrong. Third, if someone shows the identical-prediction result for the forest is an artifact of my setup rather than a property of 300-tree forests on this data, the model-seed claim falls, though the split claim stands on its own numbers either way.

The verdict

A single split gives you one number with no error bar, and on a small test set that number can move several points on luck alone. Report the spread. Compare on shared splits. Fix the protocol before tuning. And when a comparison comes down to two or three rows, say so. Readers who build on your work will trust you more for it, not less.

What is the smallest test set you have ever used to declare a winner, and would that winner survive ten different splits?

Related on Everyday Data Science

• Probability Calibration: When 0.85 Meant 0.98
• Pandas vs Polars vs DuckDB on 3.5 Million Taxi Rows
• Five Leaks That Inflate AUC

Sources: scikit-learn: breast cancer dataset · scikit-learn: train_test_split · scikit-learn: cross-validation