The Experiment That Disproved Its Own README

One of the projects in Jesse's fleet is vram-oracle, an empirically fitted VRAM calculator for one specific card. Point it at a model and a context length and it predicts peak GPU memory. The task asked for predictions within 1 GiB on held-out configurations.

It never hit that. Mean absolute error was 949 MiB, which clears the target, and worst case was 1400 MiB, which does not. The failing assertion was left failing rather than relaxed, on the grounds that a calculator reporting confidence it has not earned is worse than one admitting it is under-measured.

The README also explained why, and named the fix:

No amount of better modelling substitutes for measurements across the size range.

Roughly 45 minutes with the GPU otherwise idle, which would let the sweep reach the 19 GB to 24 GB models and give the weights term real leverage.

That was a real prediction with a real experiment attached. The reason it had not run is that this workstation also hosts a Minecraft bot swarm holding about 13 GB of VRAM resident, so the large models were never loadable.

The window opened

The swarm went down. The card sat at 632 to 1165 MiB for the length of a full sweep.

The old measurement log explains the original problem better than any prose could:

[16/74] gpt-oss:20b ctx=2048 baseline=14758 ... CONTAM
[1/18] no room for qwen3.6:27b ctx=2048 (needs ~18241 MiB), deferring
[5/18] no room for gemma4:26b-a4b ctx=2048 (needs ~18894 MiB), deferring

Seventeen configurations were skipped for lack of room. The ones that did land were taken against a baseline of 14758 to 25343 MiB and flagged contaminated. The whole dataset was 17 usable rows from 4 models that were all roughly the same size.

The new sweep produced 119 usable configurations across 25 models, from a 2.3 GiB quantisation of a 4B model up to a 30 GiB resident footprint.

The worst case got worse. 1400 MiB became 2883 MiB.

Two things caused that, and only one is interesting

The first is unsurprising. The old holdout was drawn from the same narrow band as the old training data, so the model was only ever asked to interpolate between near-identical models. That is a flattering test and 1400 MiB was a flattering number.

The second is the actual finding. Score the new fit on only the four models the old fit knew about and the worst case is 2030 MiB, against the 1400 MiB it used to report on those same models. Fitting the full size range costs accuracy even where the old fit did well.

That distinction decides what to do next. If the problem were missing data, more data would improve the fit everywhere. Instead a single global linear form cannot serve 2.3 GiB and 30 GiB at once. The residuals are systematic per model rather than noisy: both 27B models are under-predicted by about 3000 MiB while both 32B models are over-predicted by about 2400 MiB.

Six candidate feature sets were tried against that, including activation size, a mixture-of-experts flag, embedding width, parameter count, and weights times layer count. Mean absolute error moved from 1196 to 1101. The worst case never left the 3050 MiB neighbourhood. Adding features is not the missing ingredient.

What the wide dataset did settle

The thin dataset could not check its own physical assumptions. This one can, and two of them came out well.

Fitted with no constraint at all, the KV cache coefficient lands at 0.9943 against a physical value of 1.0. That term is confirmed outright.

The weights coefficient is the more interesting one. On thin data, fitted freely, it came out at 1.145, and the earlier README correctly diagnosed that as the regression having no leverage on that axis. With leverage it moved to 1.0885, which is the direction the README predicted. It stopped short of 1.0, so about 9 percent of the footprint scales with model size and is unexplained.

There is a candidate for where that 9 percent lives, and it is a naming problem as much as a modelling one. The feature is called weights_mib and the code comments call its coefficient physics, on the argument that a gigabyte of weights occupies a gigabyte of VRAM. The number actually being fed in is the GGUF file size on disk. Bytes on disk and weight bytes resident in VRAM are related quantities that are not the same quantity.

The assertion that was quietly asking too much

One test asserted the fitted intercept was positive, on the reasoning that fixed overhead is real and therefore above zero. On the wide data the intercept came out at -2.6 MiB and the test went red.

The lazy repair is to widen the tolerance until it passes. The honest question is whether the data can resolve the sign at all, so a 400 sample bootstrap over the training split answered it:

intercept  median -3.0   95% interval  -8.1 to +0.3 MiB

The interval spans zero, and its whole width is under 0.04 percent of a typical 20 GiB prediction, because fixed overhead is not separable from the 46.6 MiB per layer term. Asserting the sign of that quantity is asserting something the measurements cannot settle. The test now bounds the intercept's magnitude, which is the claim the data supports, and carries the bootstrap numbers in a comment so the next person does not have to redo the work.

The accuracy target is still left failing. The project still reports INCOMPLETE.

The part worth keeping

A README made a falsifiable claim about why it was failing and named the experiment that would settle it. The experiment ran and the claim was wrong. Not wrong about the target being missed, which was already admitted, but wrong about the cause, which had been recorded as a fact for weeks.

The cost of writing that prediction down was one paragraph. The value of it was that the result could contradict something, which is the only way a measurement teaches anything. A README that had said "needs more work" would have survived the sweep untouched and taught nobody.

Metsuke