Bare-metal benchmark report by Anthropic Claude · IBM Cloud, dual-socket Sapphire Rapids
Exact model editing beats gradient correction on both correctness and energy
A functor-style, one-shot direct-sum edit mechanism is compared against conventional per-example online SGD for fixing a trained classifier's mistakes: measured for correction success, collateral damage to unrelated predictions, and energy cost via Intel RAPL, cross-validated against the server's BMC power telemetry.
Hardware CPU CORES / THREADS Memory
Lenovo SR650 V3 2× Xeon Gold 6434H 16C / 32T 512 GB DDR5 ECC
Accelerator. OS. Model Dataset
AMX bf16 (no GPU) Ubuntu 24.04.4 LTS. ResNet‑18, from scratch Imagenette2 (10‑class)
01. Headline result
Both mechanisms received the identical task, repeated across five independent trials of 150 randomly-sampled misclassified images each, drawn from the same frozen 89.40%-accurate checkpoint: make the model get this specific example right, right now, without retraining from scratch.
Correction success rate Collateral accuracy loss Energy / successful fix
100.0% vs 18.9%. 0.0% vs 12.9% 4.56 J vs 15.80 J
Functor (direct-sum commit) figures are exact, zero-variance results, not statistical estimates that happened to land favorably — they follow from the mechanism's construction (below), not from luck across these five trials. Baseline (online SGD) figures are the best the conventional approach could do: its learning rate was swept over three orders of magnitude and selected specifically to maximize its own success rate before this comparison was run.
02. Why functor edits are exact
The functor path wraps a frozen base model f in a hash-keyed override table. A correction commits as an exact direct sum:
fnew(x) = fold(x) + Δf(x)
applied only where the gate fires — an exact-match hash of the input, not a distance-decayed kernel. An earlier design used a soft RBF gate that blended corrections into nearby-but-different inputs; that was deliberately removed, because interpolation breaks the guarantee this benchmark depends on. With a binary exact-match gate, every commit is provably local: it can only ever change the output for the one input it was anchored to, never for anything else. That is why the collateral-drift column reads exactly 0.0% rather than merely low — it is not measured, it is structural.
The baseline path instead applies one online SGD step (forward + backward + optimizer update) directly to the model's weights for each correction — the conventional way to "teach" a network a new fact. Because weights are shared across every input, a step that nudges the model toward one correct answer also perturbs its behavior everywhere else, which is exactly the collateral effect measured below.
03. The training bug that had to be found first
Before any correction could be benchmarked, the base ResNet-18 needed a real, honestly-trained checkpoint. The first several training attempts diverged after epoch 0 — loss exploding 30–50×, validation accuracy pinned at exactly chance regardless of what changed. Six controlled A/B diagnostics isolated the cause:
-
LR warmup schedule - removed, divergence persisted
-
bf16 autocast precision itself - isolated via fp32 control, ruled out
-
Gradient clipping - removed, divergence persisted identically
-
Eval/train mode switching (BatchNorm) - isolated, output stayed healthy
-
DataLoader worker respawn per epoch - disabled, divergence persisted
-
Corrupt training images - full 9,469-image scan, none found
-
Root cause: torch.autocast() wrapped around an entire epoch's training loop let CPU autocast serve a stale, cached bf16 cast of the model's weights after optimizer.step() had already updated them - silently decoupling what was being trained from what was being computed, for every step after the first.
Fix & validation
Entering the autocast context fresh on every step (not once per epoch) resolved it completely: 82% validation accuracy in 8 epochs post-fix, versus chance-level accuracy in every pre-fix variant regardless of hyperparameters. The full 60-epoch run then reached 89.40% validation accuracy cleanly, with no further intervention.
04. Protocol
-
Base model. ResNet-18, random initialization, trained 60 epochs on Imagenette2 (9,469 train / 3,925 val images, 224×224), SGD (momentum 0.9, nesterov, cosine LR decay from 0.025, no warmup — unnecessary at batch 64), bf16 autocast via AMX. Best checkpoint: 89.40% validation accuracy, 416 of 3,925 validation images misclassified.
-
Correction pool. All 416 misclassified images, each paired with its true label, form the pool both arms draw from.
-
Baseline LR selection. Swept 1×10⁻⁵ – 3×10⁻² (8 points, log-spaced) on a 200-example pilot set. Selection rule fixed in advance: highest fix rate first, then lowest energy among near-ties. lr=3×10⁻⁵ won outright (35% fix rate on the pilot set — notably higher than its 18.9% mean across the resampled trials below, itself evidence for why repeated trials matter over a single point estimate).
-
Repeated trials. 5 independent trials; each draws 150 examples at random from the 416-image pool and runs both arms on the identical subset, from the identical frozen checkpoint, back to back.
-
Energy measurement. Intel RAPL, both CPU package domains, read in batches of 10 corrections (individual commits are sub-millisecond — below RAPL's useful measurement granularity). Idle baseline (135.18 W, averaged over 3 independent measurements) is subtracted to isolate marginal energy.
05 Results
Mean ± standard deviation across N=5 trials; 95% CI in parentheses (normal approximation indicative at this N, not exact).
The raw per-operation energy is not where functor wins — at 2.94J, a single baseline SGD step is actually cheaper than a single functor commit's 4.56J, because a commit's validation pass evaluates the base model at several nearby points before accepting the edit. The gap opens entirely on outcome: baseline spends its energy on corrections that mostly don't stick, so its cost per correction that actually lands is 3.5× higher — before even accounting for the damage those steps do elsewhere.
On the baseline's collateral damage
Every baseline trial started from a control set of 200 validation images the model already classified correctly. After 150 single-example SGD corrections elsewhere, an average of 12.9% of that unrelated, previously-correct control set flipped to wrong — at the LR chosen specifically to maximize the baseline's own success rate. Higher learning rates tested in the sweep were worse on both axes simultaneously: from lr ≥ 1×10⁻³ the model collapsed to an identical degenerate state regardless of exact LR (20/200 fixed, 89% collateral loss, unchanged across a 30× LR range) — a distinct representation-collapse failure mode, not the autocast bug from Section 3.
06. Independent power cross-check
RAPL measures CPU package energy only. To check it against an independent instrument, the server's Lenovo XClarity BMC was queried in-band via IPMI (no separate management-network credentials required).
IDLE-STATE CROSS-VALIDATION
SOURCE READING
RAPL, CPU package (this benchmark's instrument) 139.5 W
IPMI, "CPU Power" sensor (independent instrument) 140 W
IPMI, "Sys Power" (whole chassis: PSU, fans, NICs, drives) 264 W
The two independent CPU-power instruments agree to within 1 W at idle, the RAPL numbers this report is built on are not just internally consistent, they match a second, independently-calibrated sensor. The ~124 W gap between CPU package power and whole-chassis power is real, constant non-CPU overhead on this dual-PSU enterprise server; it does not affect the functor-vs-baseline comparison (identical hardware, identical constant overhead, in both arms), but it does mean RAPL alone understates total facility-level draw for anyone costing this at the wall.
A limitation, reported rather than hidden
Under sustained synthetic CPU load, RAPL correctly tracked the increase (139 W idle → 293–301 W loaded, confirmed over two independent windows up to 45 seconds). The BMC's IPMI DCMI power-reading command did not: across 14 samples spanning 42 seconds of continuous load, it stayed flat at 269–271 W, indistinguishable from idle. This is a genuine limitation of that specific telemetry path on this BMC firmware under CPU-only load, not a transient lag; further sampling would not have resolved it. The idle cross-validation stands as confirmation of RAPL's accuracy; the load-condition figures in Section 5 rely on RAPL alone.
07. Scope & limitations
-
Dataset scale. Imagenette (10-class, 13k images), not full ImageNet-1k — chosen deliberately to prioritize repeated-trial statistical rigor over one long training run inside a fixed 3-week compute window. Architecture (ResNet-18) is unmodified and standard.
-
Baseline design. One online SGD step per correction, matching the functor path's one-commit-per-correction granularity for a direct comparison. A periodic-batch retraining baseline would trade a different set of costs and is not covered here.
-
Energy granularity. RAPL readings are batched over groups of 10 corrections, not individual commits, to stay above counter measurement noise — appropriate for the means reported here, not for single-correction-level claims.
-
Hardware generality. All figures are specific to this Sapphire Rapids / AMX configuration; both arms were measured on identical hardware, so the relative comparison should generalize better than either arm's absolute numbers.
Base checkpoint resnet18_base.pt · 89.40% val. accuracy, epoch 58/60 · artifacts frozen post-XCC-validation, no further methodology or model changes.
