Introduction

The seductive pitch of GPU-accelerated backtesting is “one big matrix”: stack every parameter combination into a tensor and let the hardware eat it. The awkward fact underneath the pitch is that the same hardware, handed one combination at a time, is often slower than the CPU it was meant to replace. The speedup of a batched processor over a per-call one is not a property of the chip; it is a function of how much work you hand it per call.

The reason is a fixed cost. Every dispatch pays a roughly constant price — launch the kernel, move the input across the (unified, but not free) memory boundary, move the result back — that does not shrink when the call does less work. Run a sweep of B combinations one per call and you pay that overhead B times; batch the same B into a single call and you pay it once. A small sweep cannot amortize the overhead, so it sits in the regime where fixed cost dominates and the accelerator barely earns its keep; a large sweep amortizes the overhead across many combinations and approaches the true marginal-cost ratio.

This paper isolates that amortization into its simplest exact form. We model the cost of running B combinations two ways — batched (fixed overhead O paid once, plus marginal work b per combination) and per-call (the same overhead re-paid on every combination) — and take the ratio. The result is a one-line closed-form law, S(B)=B(O+b)/(O+bB), whose entire behavior is set by the single dimensionless number O/b: it rises monotonically from S(1)=1, saturates toward 1+O/b, has its half-way “ridge” at B=O/b, and reaches any target fraction f of its ceiling at a batch B^\star(f)=fO/[b(1-f)] that is independent of the marginal work.

We then make the law concrete and, crucially, machine-invariant. We run a real, seeded NumPy [2] weighted-moving-average (WMA) sweep two ways over the same synthetic series — batched and one-per-call — but measure work as a deterministic operation count rather than wall-clock seconds. The per-call path re-does the fixed launch-plus-transfer charge on every combination; the batched path does it once. Because the reported quantity is an integer op count, the whole curve is bit-reproducible run-to-run and machine-to-machine, and it realizes S(B) exactly. Both paths compute the identical WMAs and return bit-identical numbers; only the overhead accounting differs. Fitting the law back out of the op-count curve recovers its parameters with zero residual: the analog does not merely resemble the law, it is the law.

Scope and honesty constraints.

Two constraints bound the claims, and they are load-bearing. First, we measure no GPU and no wall-clock time. Every number in this paper is either a closed-form value of the model or a deterministic operation count from the seeded NumPy sweep; there are no timings and no hardware measurements. Second, the device speedups quoted in the accompanying blog post — a GPU-over-CPU curve rising from 54.5\times at one combination per call to 359.6\times at 61 on an Apple M2 Max, the 3.2\times/6.2\times hardware ratios, and the 167\times=27\times\cdot6.2\times algorithm/hardware decomposition — are prior engineering measurements that this paper does not reproduce and does not claim. What is reproduced here is the underlying law those numbers obey (a speedup that rises with batch size and saturates as fixed overhead is amortized), together with a machine-invariant op-counted realization of it. All results derive from one seeded run of one public harness (scripts/run_all.py), with the model and the analog in scripts/roofline.py.

The amortization law

Two cost models.

Consider a sweep of B parameter combinations over one series. Each combination requires the same marginal work b. Each call additionally pays a fixed overhead O — the launch charge plus the transfer of the series across the memory boundary — that is independent of how many combinations the call computes. Batching all B combinations into one call pays O once; dispatching them one per call re-pays O on every combination: \begin{equation} \label{eq:costs} T_{\mathrm{batched}}(B) = O + bB, \qquad T_{\mathrm{percall}}(B) = B\,(O + b). \end{equation} Both paths do the identical useful work bB in total; they differ only in how often the fixed overhead is paid. (In a slightly more general form T_{\mathrm{percall}}=B(O+a) one writes the per-call marginal as a; here each per-call combination re-pays the full fixed cost O on top of the shared marginal, so a=b, and we use that throughout.)

The speedup.

The speedup of batched over per-call execution is the ratio of the two costs: \begin{equation} \label{eq:speedup} S(B) = \frac{T_{\mathrm{percall}}(B)}{T_{\mathrm{batched}}(B)} = \frac{B\,(O + b)}{O + bB}. \end{equation} For O>0 and b>0 this is monotone increasing in B, with S(1)=1 (a single combination cannot be amortized against anything) and a decelerating rise. It is the parameter-sweep instance of Amdahl's law [1]: the fixed overhead O is a serial cost that a batch dilutes but never deletes.

The saturation ceiling.

As B\to\infty the fixed overhead is spread over infinitely many combinations and the speedup approaches \begin{equation} \label{eq:asymptote} \lim_{B\to\infty} S(B) = \frac{O+b}{b} = 1 + \frac{O}{b}, \end{equation} a ceiling the curve rises toward but — for any finite B — never reaches. The fraction of this ceiling attained at batch B is S(B)/(1+O/b).

The batch ridge.

The roofline model [3] locates a ridge point where a workload stops being overhead-bound and becomes compute-bound. In batch space that ridge is the batch at which the batched path's fixed overhead equals its marginal compute for the whole batch, O = b\,B: \begin{equation} \label{eq:ridge} B_{\mathrm{ridge}} = \frac{O}{b}. \end{equation} Past the ridge, the batched cost O+bB is dominated by the bB marginal rather than by the fixed O: the call is compute-bound. Note that the ridge and the asymptote differ by exactly one, 1+O/b = 1 + B_{\mathrm{ridge}}.

The compute-bound crossover.

How wide must the sweep be to reach a chosen fraction f of the ceiling? Setting S(B)=f\,(1+O/b) in Eq. (2) and solving, \begin{equation} \label{eq:crossover} \frac{B(O+b)}{O+bB} = f\,\frac{O+b}{b} \;\Longrightarrow\; bB = f\,(O + bB) \;\Longrightarrow\; B^\star(f) = \frac{f\,O}{b\,(1-f)}. \end{equation} The marginal b appears only through O/b, and a (had we kept it) cancels entirely: the crossover is a pure overhead-amortization threshold, determined by the overhead-to-marginal ratio and the target fraction alone. We report it for f=0.90 and f=0.95.

The two-term decomposition.

The batched cost O+bB splits cleanly into a fixed-overhead term and a useful-compute term, and their fractions are \begin{equation} \label{eq:decomp} \phi_{\mathrm{ovh}}(B) = \frac{O}{O+bB}, \qquad \phi_{\mathrm{use}}(B) = \frac{bB}{O+bB} = \frac{S(B)}{1+O/b}, \end{equation} with \phi_{\mathrm{ovh}}+\phi_{\mathrm{use}}=1. As B grows the overhead fraction falls and the useful fraction rises toward one: this is the two-ceiling roofline picture — overhead-bound on the left, compute-bound on the right — drawn along the batch axis. The per-call path, by contrast, re-pays O on every combination, so its overhead fraction O/(O+b) is constant in B.

The CPU analog: operation-count work

The law of Section 2 is an identity; to show it governs real work we instantiate it in a genuine computation — but one whose “cost” is a deterministic operation count, not a timing. This buys two things: the result is bit-reproducible on any machine (no wall-clock noise, no hardware dependence), and the accounting of fixed versus marginal work is exact rather than fitted.

The workload.

We sweep window lengths of a linearly weighted moving average over a single seeded synthetic price series. The series is a geometric random walk of n=150{,}000 bars from numpy.random.default_rng(0) (initial price 30{,}000, per-bar log-return standard deviation 5\times10^{-4}); each combination convolves the series with a normalized linear kernel. A batch of size B uses B distinct window lengths.

The op-count model of overhead and marginal.

Each call pays a fixed overhead: a launch charge plus a transfer of the whole series across an (analog) memory boundary, \begin{equation} \label{eq:opsO} O_{\mathrm{ops}}(n) = L_{\mathrm{launch}} + \tau\, n = 5000 + 24 \cdot 150{,}000 = 3{,}605{,}000, \end{equation} with L_{\mathrm{launch}}=5000 the dispatch charge and \tau=24 the transfer cost per series element. This is independent of how many combinations the call computes — exactly the quantity a batch amortizes. Each combination's marginal useful work is one windowed pass over the series, \begin{equation} \label{eq:opsb} b_{\mathrm{ops}}(n) = n = 150{,}000, \end{equation} a constant n ops per combination. (Keeping the marginal constant is a modelling choice, not an approximation: it makes the batched marginal exact, so the op-count curve equals the closed-form law rather than approximating it.)

Batched versus per-call, and their equivalence.

The batched sweep pays O_{\mathrm{ops}} once and accumulates b_{\mathrm{ops}} per combination; the per-call sweep re-pays O_{\mathrm{ops}} on every combination: \begin{equation} \label{eq:opscosts} \mathrm{total}_{\mathrm{batched}}(B) = O_{\mathrm{ops}} + b_{\mathrm{ops}}B, \qquad \mathrm{total}_{\mathrm{percall}}(B) = B\,(O_{\mathrm{ops}} + b_{\mathrm{ops}}), \end{equation} and the analog speedup is their ratio, which by construction equals Eq. (2) with O=O_{\mathrm{ops}}, b=b_{\mathrm{ops}}. Critically, both paths run the same NumPy convolutions over the same data and accumulate the same deterministic checksum of the outputs: they produce bit-identical numerical results. Batching changes only the overhead accounting, never the arithmetic — so the speedup is a pure amortization effect, not a numerical artifact. Across the whole batch grid the equivalence check holds (all_paths_equivalent= true).

Fitting the law back out.

Finally we recover the law's parameters from the analog's op-count speedup curve by least squares (normalizing b=1, since S is invariant to a common scale of O,a,b) and check that the fitted law reproduces the curve. The fit is exact: it returns O/b=24.03, asymptote 1+O/b=25.03, and a maximum relative residual of exactly 0. Because the marginal is a genuine constant, there is no residual to leave — the analog is a faithful realization of the closed form, not a noisy sample of it.

Results

All numbers below come from one seeded run (seed 0, Python 3.12.3, NumPy 2.5.1) of scripts/run_all.py, calibrated to O=3{,}605{,}000, b=150{,}000 (so O/b=24.03), over the batch grid B\in\{1,2,4,8,16,32,61\} that mirrors the shape of the blog's sweep.

The speedup curve and its landmarks.

Table 1 reports the closed-form speedup S(B) and the fraction of the asymptote it attains at each batch. The curve climbs from S(1)=1.00 (a single combination amortizes nothing) to S(61)=17.96, which is 71.7\% of the way to the ceiling 1+O/b=25.03. The batch ridge sits at B_{\mathrm{ridge}}=O/b=24.03; the compute-bound crossover is B^\star(0.90)=216.3 and B^\star(0.95)=456.63 — i.e. reaching 90\% of the ceiling needs a batch of 216, and 95\% needs 457, both far to the right of the 61-wide grid. The grid lives on the rising overhead-amortized part of the curve, well left of saturation, which is exactly the regime a modest sweep occupies.

The CPU analog reproduces the law exactly.

Table 2 reports the op-counted analog. Each row shows the total batched op count O+bB, the total per-call op count B(O+b), and their ratio — the analog speedup — alongside the useful fraction of the batched path. The analog speedup equals the model speedup of Table 1 to every printed digit (S(1)=1.00, S(61)=17.96), the two paths are numerically equivalent at every batch, and the batched useful fraction rises from 3.99\% at B=1 to 71.7\% at B=61 (identical, as Eq. (6) requires, to the fraction-of-asymptote column of Table 1). The per-call path's overhead never amortizes: its total is B copies of the single-call cost.

The overhead/compute decomposition.

Table 3 splits the batched cost into its two roofline terms per batch. The fixed overhead, which is 96.0\% of the batched cost at B=1, is amortized down to 28.3\% at B=61; the useful-compute fraction rises correspondingly from 3.99\% to 71.7\%. The per-call path's overhead fraction stays pinned at 96.0\% for every batch, because it re-pays O on every combination and so never crosses out of the overhead-bound regime. This is the mechanism of the whole paper in one table: batching moves the workload rightward along the roofline; per-call keeps it stuck on the left.

Fit-back verification.

Table 4 closes the loop: fitting the closed-form law to the analog's op-count speedup curve recovers the ridge O/b=24.03 and the asymptote 1+O/b=25.03 (fitted with b normalized to 1, so O=24.03, a=1.00), with a maximum relative residual of exactly 0. The analog does not approximate the law; it is the law, realized to machine precision. The zero residual is a direct consequence of the constant marginal in Eq. (8): there is no curvature for the fit to miss.

Discussion

The speedup is a curve, not a number.

The single most useful thing the law says is that “how much faster is batched than per-call” has no scalar answer: it is S(B), a function of batch size. Quoting one point on that curve — the small-batch number or the large-batch number — and hiding the rest is not measuring the amortization, it is choosing a headline. The blog's device curve rising from 54.5\times to 359.6\times (prior work, not reproduced here) is the empirical shadow of exactly this S(B); the law explains why any such measurement must rise with batch size, whatever the hardware.

Amdahl and the roofline, in batch space.

The law is two classical results wearing a batch-size coordinate. The fixed overhead O is Amdahl's serial fraction [1]: it can be amortized across a batch but never removed, which is why S(B) saturates at 1+O/b rather than growing without bound. And the ridge B_{\mathrm{ridge}}=O/b is the roofline's ridge point [3] cast into batch space: left of it the batched call is overhead-bound (the flat O dominates), right of it compute-bound (the bB marginal dominates). The two-term decomposition of Table 3 is the roofline's memory-bound/compute-bound split drawn along B.

A decision rule.

The crossover B^\star(f)=fO/[b(1-f)] turns the law into an actionable threshold. It says: to reach 90\% of the achievable amortization on this calibration you need a batch of 216; a sweep of a few dozen combinations lives far to the left of that, deep in the overhead-bound regime, where most of the cost is still fixed launch-and-transfer. The lever is either a wider batch (move rightward on the curve) or richer per-combination work (raise b, which lowers the ridge O/b and pulls the crossover left).

Why an operation count, not a timing.

Reporting work as a deterministic op count, rather than seconds, is what makes the analog machine-invariant: the same seed produces the same curve on any CPU, with no warm-up, no thermal drift, no scheduler noise, and no dependence on a particular device. The cost is that the analog demonstrates the law, not a speed: it shows that batched-versus-per-call amortization obeys S(B) exactly, without asserting how fast any real processor runs. That trade — give up the timing to gain reproducibility and portability — is the deliberate design of this study.

Limitations

• A model and an analog, not a GPU measurement. We measure no GPU and no wall-clock time. Every number is a closed-form value of S(B) or a deterministic operation count from the seeded NumPy sweep. The study establishes the amortization law and a machine-invariant realization of it; it does not measure any device's speed.

• Prior GPU numbers are not reproduced. The accompanying blog's device speedups — 54.5\times to 359.6\times on an Apple M2 Max, the 3.2\times and 6.2\times hardware ratios, the 167\times=27\times\cdot6.2\times algorithm/hardware decomposition — are prior engineering measurements on specific hardware. They are cited only to motivate the law and are explicitly not reproduced or claimed here.

• Idealized cost structure. The law assumes a constant per-call overhead O and a constant marginal b. Real dispatch overhead can vary with batch, transfers can be non-linear, and marginal work can depend on the parameter (window length here is held marginal-constant by construction). Those effects add curvature and quantization (e.g. chunk-boundary wobble) that the idealized law smooths over; the qualitative shape — monotone rise to a ceiling with a ridge and a crossover — is robust to them.

• One calibration and one workload. The reported landmarks (ridge 24.03, asymptote 25.03, crossover 216) are specific to the seeded calibration O=3{,}605{,}000, b=150{,}000. Only the ratio O/b matters to the shape; a different overhead-to-marginal ratio moves every landmark, but the functional form of S(B) does not change.

Conclusion

A parameter sweep dispatched one combination at a time re-pays a fixed per-call overhead on every combination; batched into one call it pays that overhead once. The speedup of batched over per-call execution is therefore an elementary closed-form law, S(B)=B(O+b)/(O+bB): monotone rising from S(1)=1, saturating toward a ceiling 1+O/b it never reaches, with a batch-space ridge at O/b and a compute-bound crossover at fO/[b(1-f)]. On a seeded calibration with O/b=24.03 the law climbs from S(1)=1.00 to S(61)=17.96 toward an asymptote of 25.03, with the ridge at 24.03 and the 90\% crossover at 216; the fixed overhead falls from 96.0\% of the batched cost at B=1 to 28.3\% at B=61. A real, seeded NumPy weighted-moving-average sweep — with work measured as a deterministic operation count rather than wall-clock time — reproduces the law exactly, produces bit-identical output on both paths, and, fitted back, recovers O/b=24.03 and the asymptote 25.03 with a maximum relative residual of 0. The whole demonstration is machine-invariant because the reported work is an operation count, not a timing. We measure no GPU and reproduce none of the prior work's device speedups; the contribution is the law and its machine-invariant analog. If a batched sweep gets dramatically faster as you feed it more combinations, that is not the hardware waking up — it is the fixed overhead being amortized, exactly as far as S(B) says and no further.

Reproducibility.

All numbers derive from one seeded run. scripts/run_all.py regenerates results/results.json from seed 0 (Python 3.12.3, NumPy 2.5.1) deterministically; the closed-form model and the op-counted CPU analog are in scripts/roofline.py. The synthetic series is a geometric random walk from numpy.random.default_rng(0) (initial price 30{,}000, per-bar log-return standard deviation 5\times10^{-4}, n=150{,}000). tests/ contains deterministic invariant tests for the model (monotonicity, saturation, ridge, crossover), the analog (path equivalence, overhead amortization), their agreement, and byte-level reproducibility, and a guard asserting that only operation counts and model values — never timings — are reported.

References