eicalc: LLM Inference Capacity Calculator

eicalc estimates how many concurrent users a given GPU system can serve with a given LLM at given service levels. It is one self-contained index.html: pick a system, pick a model, describe your user mix, and read off the capacity, the bottleneck, and the memory breakdown. The math is a pinned analytic model (engine spec v1.2 + errata v1.3 + errata v1.4 + errata v1.5 + errata v1.6); every catalog number carries its source, and known vendor data conflicts are surfaced instead of hidden.

All research sources were fetched on 2026-09-11; the Qwen3.8-27B model and RTX PRO 6000 system sources were fetched on 2026-10-13.

1. What it is and how to run it

Run the app. Open index.html in any modern browser. It works from file:// with no server. If you prefer a URL, serve the folder:

python3 -m http.server 8000
# then open http://localhost:8000/

The UI persists a slim form state ({systemId, modelId, custom, wl, adv}) into the URL hash. The "Copy share link" button produces a URL that reproduces the exact scenario on another machine; nothing is stored on a server.

Run the tests.

node tests/run-tests.mjs

Node 18 or newer, zero npm dependencies. The suite extracts the engine from index.html between the // ENGINE-BEGIN and // ENGINE-END markers, evaluates it in a fresh VM context, and checks hand-computed golden fixtures plus property tests (curve monotonicity, cache-hit scaling, URL-state round-trip). Exit code 0 means all pass.

Conventions. SI decimal units everywhere: 1 GB = 1e9 bytes, 1 TB/s = 1e12 B/s. Vendor figures published in GiB or KiB were converted to raw bytes before storage. The UI is English.

Quick usage. Pick one of 6 system presets and one of 6 model presets (or Custom), enter user counts for 5 context buckets (<50k, <100k, <250k, <500k, >=500k; buckets 1 to 4 are evaluated at their cap, which is conservative for memory), then turns per user, tokens per turn, cache-hit rate, and your per-user tok/s target. Advanced inputs (eta, MFU, prefill budget, reserve, layout, weight quantization, KV precision, KV transfer rate) are editable. Section 2.9 documents every default with its source and uncertainty band.

2. The model of a limit

For each of five resources the engine asks: by what factor can the entered user mix grow before that resource saturates? That factor is the scale s. The reported capacity is:

maxUsers   = floor(min(scales) * totalUsers)
bottleneck = argmin over scales

A limit with zero demand reads Infinity and drops out. Everything below is the pinned engine spec; tests/run-tests.mjs enforces it, so this section doubles as the spec's prose form.

2.1 Inputs

computeCapacity({sys, model, wl, adv}) takes four objects:

sys   = { gpus, vramPerGPU, hbmBW, flopsDense{bf16, fp8, fp4|null},
          nvlinkPerGPU, pciePerGPU, islands, mode:'agg'|'pd',
          pdGPUs{prefill, decode}, layouts[{id, tp, replicas, dp?, penalty, arUs}],
          kvTransferGBps }
model = { totalParams, activePrefillParams, activeDecodeBytes, layers, growingLayers,
          kvMode:'computed'|'official', kvFp16PerToken, officialKvPerToken,
          constStateBytes, kvSharding:'heads'|'replicated', experts, expertsPerTok,
          attnHeads, attnCoef, ctxMax, weightBytes{bf16, fp8, nvfp4}, kvHeads }
wl    = { counts[5], ctx[5], turns, newInputTok, outputTok, cacheHit,
          targetTok, activity, intervalS, ttftTargetS }
adv   = { eta, mfu, prefillBudget, reserve, kvPrec, quant, layout, kvTransferGBps,
          hostRAM, hostReserve, hostStreamGBps }

quantEff selects the weight-byte column for the chosen weight quantization. quantFlops selects the FLOP column: nvfp4 uses the fp4 slot, and on H200 that slot is null, so the engine falls back to FP8 FLOPs and notes it. wl.turns and wl.outputTok are live inputs, not just narrative: both grow the per-session context peak ctxPeak that feeds the memory limit (section 2.3), so raising them lowers memory capacity directly.

2.2 Layout selection (first step)

A layout is feasible if the sharded weights fit under the reserve on one GPU:

model.weightBytes[quantEff] / layout.tp  <=  adv.reserve * sys.vramPerGPU

If no layout is feasible the result is {ok:false, error:'NO_LAYOUT_FITS'}. Selection order: adv.layout if given and feasible, otherwise an argmax — every feasible layout runs through the full limit computation and the one with the highest final maxUsers wins (errata v1.4, replacing the old max-replicas heuristic). At equal maxUsers the layout with the larger decode scale wins (it keeps scaling once the binding limit is lifted); a true tie falls back to catalog order. The argmax is host-aware — host RAM can change the winning layout (errata v1.6). Disaggregated systems always run tp8 per node. replicas = layout.replicas feeds every limit below.

DP-attention layouts (dp field, errata v1.4). A layout with dp > 1 runs DP-attention: the dp ranks shard the users (each rank holds only its own sessions' KV) and serve one model copy by definition, so replicas is clamped to 1. Production serving engines (SGLang, NVIDIA Dynamo, vLLM) standardize on this shape because attention-TP collapses concurrency when the latent KV cannot be head-sharded — TensorRT-LLM's DeepSeek write-up reports DP-attention reaching about +400% max throughput versus KV-replicating TP on Blackwell (TRT-LLM blog). That is why the argmax lands on dp8-attn at high concurrency.

2.3 KV cache per user

kvTok    = model.kvMode=='official' ? model.officialKvPerToken
                                    : model.kvFp16PerToken * KV_PREC_FACTOR[adv.kvPrec]
// KV_PREC_FACTOR: fp16 1, fp8 0.5, fp4 0.25 (NVFP4 KV, errata v1.4; Blackwell only — the UI
// gates the option off on H200). 'official' models ignore the control entirely.
ctxPeak_i = wl.ctx[i] + (wl.turns-1) * (wl.newInputTok + wl.outputTok)
kvUser_i = ctxPeak_i * kvTok + model.constStateBytes
total    = sum(wl.counts[i])
kvNeed   = sum(wl.counts[i] * kvUser_i)      // resident KV bytes, all users
kvAvg    = kvNeed / total
kvPerGpu = kvAvg * (model.kvSharding=='heads' ? 1/layout.tp : 1)   // non-dp layouts
kvPerGpu = kvAvg / layout.dp                                        // dp > 1 (user-sharded)

turns and output tokens grow the session context (KV holds all tokens; prefix cache saves prefill compute, not memory); buckets describe the per-turn starting context.

• official KV mode: the model card publishes a fixed deployed KV size per token (DeepSeek-V4.1-Flash: 890 B, see section 4). The KV-precision control does not apply; the engine pins kvTok.
• constStateBytes: linear-attention layers (KDA and similar) keep a constant-size recurrent state per sequence instead of per-token KV. Kimi K3 carries about 0.2 GB per sequence (derived value); its 69 KDA layers add zero per-token KV.
• kvSharding='heads': the KV is head-shardable (GQA). Per-GPU KV = kvUser/tp, valid when kvHeads >= tp. Below that the engine treats the layout as replicated and emits the TP_GT_KV_HEADS warning.
• kvSharding='replicated': MLA/MQA latent KV cannot be head-sharded, so every TP rank holds the full kvUser. All four presets are MLA/MQA-latent and use replicated. Production engines behave the same way: TensorRT-LLM's DeepSeek write-up reports that attention-TP collapses concurrency because KV is replicated per rank, which is why attention-DP exists (TRT-LLM blog). The dp8-attn layouts (errata v1.4) are the modeled escape: each DP rank holds only its own sessions, so kvPerGpu = kvAvg/dp.
• wl.activity does not scale KV. Inactive sessions keep their context resident; activity only scales the decode batch and prefill demand.

2.4 Memory limit

decodeGPUs = (sys.mode=='pd') ? sys.pdGPUs.decode : sys.gpus
pool  = adv.reserve * decodeGPUs * sys.vramPerGPU - replicas * model.weightBytes[quantEff]
s_mem = pool / kvNeed       // guard s_mem >= 0; a negative pool is caught by layout feasibility

Host KV tier (errata v1.6). The optional host tier (HiCache) adds a host DRAM pool poolHost = adv.hostRAM * adv.hostReserve bytes (0 when the tier is off; the reserve holds back the host OS, kernel, and transfer buffers). With the tier on, the v1.5 s_mem = pool / kvNeed line becomes a split:

s_mem = min( pool / (wl.activity * kvNeed),      // HBM must hold the active sessions
             (pool + poolHost) / kvNeed )        // HBM + host hold every session
             // guard s_mem >= 0, the same max(0, ·) clamp as v1.5

Two identities pin it: with hostRAM = 0 the split collapses to the v1.5 pool / kvNeed bit-exactly, and at activity = 1 it returns pool / kvNeed even with the tier on — the HBM-active term only binds when the active working set alone outgrows the pool. Assumption (uniform-mix steady state): HBM keeps active sessions resident first and its spares fill with idle sessions, so total capacity across both tiers is bounded by (pool + poolHost) / kvNeed. The host tier holds hostHeld = max(0, kvNeed - pool) bytes of KV.

The pool subtracts one full weight copy per replica: a tp2x4 layout stores 4 copies. The result exposes:

memoryBreakdown = { weightsGB, kvGB, reserveGB = (1-adv.reserve)*decodeGPUs*vramPerGPU/1e9,
                    freeGB = max(0, pool - kvNeed)/1e9 }   // reserveGB = 0.08*decodeGPUs*vramPerGPU at the default reserve 0.92

// with the host tier on (errata v1.6), the result adds (kvGB stays kvNeed for compatibility):
memoryBreakdown += { hostKvGB = max(0, kvNeed - pool)/1e9, hostPoolGB = poolHost/1e9,
                     hostFreeGB = max(0, poolHost - hostHeld)/1e9,
                     kvHbmGB = min(kvNeed, pool)/1e9 }      // GPU stackbar segment, so the two bars do not double-count offloaded KV

Default reserve = 0.92: a planning heuristic covering activations, CUDA context, and allocator fragmentation. It is not a measured value; no public benchmark pins a universal free-VRAM fraction. It is editable, and capacity scales roughly linearly with it.

2.5 Decode limit (memory bandwidth)

One weight read per step is shared across the replica's batch; each user's KV is read every step:

wTot = model.weightBytes[quantEff] / layout.tp     // sharded weights per GPU
wAct = model.activeDecodeBytes / layout.tp         // active experts + shared experts per GPU

// dense models:
wStep(B) = wTot

// MoE models, TP layouts (replicas > 1 or dp absent/1):
Bsat     = max(1, floor(model.experts / (layout.tp * model.expertsPerTok)))
wStep(B) = min(wTot, wAct * (1 + B/Bsat))

// MoE models, DP-attention layouts (dp > 1, errata v1.4): the rank holds experts/tp
// FULL routed experts, and B tokens hit the fraction 1-(1-expertsPerTok/experts)^B
// of them at least once — the step read grows from wAct to wTot along that curve:
wStep(B) = min(wTot, wAct + (wTot - wAct) * (1 - (1 - expertsPerTok/experts)^B))

Both wStep forms are monotone non-decreasing in B, which the binary search relies on.

tBase(B) = (wStep(B) + B*kvPerGpu) / (adv.eta * sys.hbmBW) * layout.penalty
           + (layout.tp > 1 ? 2 * model.growingLayers * layout.arUs * 1e-6 : 0)

// cache-aware duty (errata v1.4):
pfCapFull = adv.mfu * TF * sys.gpus / flopsTok     // full-node prefill capacity (see 2.6)
util      = demand / pfCapFull
share     = min(adv.prefillBudget, util)           // never MORE reserved than the budget
duty      = (sys.mode=='pd') ? 1 : 1 - share       // demand==0 -> duty 1
tEff(B)   = tBase(B) / duty

• The MoE term is a two-regime amortization heuristic. At small batch a step reads mostly active-expert bytes (wAct includes the always-on shared experts); at B >= Bsat every routed expert on the rank is hit once per step and the read converges to the full sharded weights wTot. Bsat is the batch at which all local experts go active: Kimi K3 at tp8 gives floor(896/(8*16)) = 7.
• layout.penalty scales the roofline for interconnect losses (1.00 on a full NVSwitch domain, up to 1.80 for cross-island TP8 over PCIe, see section 3).
• The trailing term adds two allreduces per growing layer per step at arUs microseconds each, only when tp > 1.
• duty: in aggregated mode prefill and decode share the same GPUs. Errata v1.4 makes the split cache-aware: prefill claims only the GPU-time share its actual uncached demand needs (demand/pfCapFull, computed against the FULL node, not the budgeted share), capped at adv.prefillBudget so the worst case stays the old hard split. A high cache hit cuts demand to a sliver and hands the time back to decode; demand = 0 gives decode the whole node. In pd mode decode owns its node, so duty = 1.

B_max is the largest integer batch in [1, 1e6] with 1/tEff(B) >= wl.targetTok, found by binary search (tEff is monotone non-decreasing because wStep is a min of non-decreasing functions). Then:

s_dec = B_max * replicas / wl.activity / total

Reported fields: batchPerReplica = B_max, stepMs = 1000 * tEff(B_max), perUserTokS = 1/tEff(B_max), perUserTokS_at_B1 = 1/tEff(1), scale = s_dec.

PCIe reload limit (reload, errata v1.6). With the host KV tier on (section 2.4), a session evicted to host DRAM must stream its KV back over PCIe when it wakes. The engine prices that stream as a fifth scale (serverGPUs = sys.pdGPUs.decode in pd mode, else sys.gpus):

reloadX  = total * wl.activity * kvAvg / wl.intervalS   // turn-start KV resume demand per second (full mix-average kvAvg)
reloadY  = wl.activity * pool / wl.intervalS            // HBM-resident credit served without crossing PCIe
pcieAgg  = hostStreamBps * serverGPUs                   // aggregate PCIe reload bandwidth
s_reload = (pcieAgg + reloadY) / reloadX                // Infinity when the tier is off or reloadX == 0

Derivation: at growth factor s the resume stream is s * reloadX, of which reloadY is already HBM-resident, so the PCIe load is resumeBps(s) = s * reloadX - reloadY once s > pool/(total*kvAvg) and 0 below; the crossing (pcieAgg + reloadY)/reloadX is exact in that linear region. Structural invariant: reloadY / reloadX = pool / kvNeed exactly, so s_reload = pool/kvNeed + pcieAgg/reloadX strictly exceeds the v1.5 memory scale — the reload limit can only bind inside the capacity region the host tier unlocked, and memory can never regress vs v1.5.

2.6 Prefill limit (compute)

pfGPUs   = (sys.mode=='pd') ? sys.pdGPUs.prefill : floor(sys.gpus * adv.prefillBudget)  // >= 1
ctxAvg   = sum(wl.counts[i] * wl.ctx[i]) / total
T        = wl.newInputTok * (1 - wl.cacheHit)      // uncached input tokens per turn
flopsTok = 2*model.activePrefillParams + model.attnCoef * (ctxAvg + T/2)
TF       = sys.flopsDense[quantFlops]
pfTok    = adv.mfu * TF * pfGPUs / flopsTok        // prefill tokens per second
demand   = total * wl.activity * T / wl.intervalS  // uncached tokens arriving per second
s_pf     = demand==0 ? Infinity : pfTok / demand

The attention term: a token at position p in a quadratic-attention layer costs about 4 * attnHeads * dKv * p FLOPs across the QK^T and AV matmuls, so a chunk of T tokens landing on top of context S costs on average 4 * H * dKv * (S + T/2). Summed over the quadratic layers this gives the catalog rule attnCoef = 4 * growingLayers * attnHeads * dKv. Only growing layers contribute: linear-attention layers are O(1) per token, and their cost is folded into activePrefillParams. The (ctxAvg + T/2) midpoint is why long-context buckets inflate prefill cost superlinearly. The preset attnCoef values follow this rule exactly, with per-model dKv: K3 192, GLM-5.3 256, GLM-5.3-Flash 256, DeepSeek 512 (see section 4); they remain editable catalog fields.

Default mfu = 0.40: kernel-level FP8 GEMMs reach 66 to 76% of peak (DeepGEMM), but end-to-end prefill lands far lower, about 39% measured on a GB200 vLLM R1 run (vLLM blog).

2.7 TTFT limit (queueing)

base = T/pfTok + (sys.mode=='pd' ? kvAvg * (1 - wl.cacheHit) / kvTransferGBps : 0)
              + reloadLatencyS      // agg mode + host tier on (errata v1.6); 0 otherwise

// reloadLatencyS (errata v1.6, aggregated mode + host tier on):
evictShare1    = max(0, 1 - pool/kvNeed)     // fraction of the average session pushed to host at s=1
reloadLatencyS = evictShare1 * kvAvg / hostStreamBps

At growth factor s the prefill fleet runs at utilization u = s * demand / pfTok. The engine charges queueing delay with an M/M/1-style factor:

ttft(s) = base * (1 + u/(1-u))      for u < 1    (Infinity at u >= 1)

The SLO scale solves ttft(s) = wl.ttftTargetS in closed form:

u*     = max(0, wl.ttftTargetS/base - 1) / (wl.ttftTargetS/base)   // base >= target -> s_ttft = 0
s_ttft = u* * pfTok / demand         (Infinity when demand == 0)

The u/(1-u) factor is deliberately conservative: it is the single-queue M/M/1 waiting term, a worst case for continuous batching, which behaves better at moderate load. The disaggregated base adds the transfer time for the uncached fraction of a turn's KV over kvTransferGBps (default 400 GB/s).

The reload term is charged on the per-GPU link: for the replicated-KV/DP-attention presets a session's FULL KV crosses one rank's link, which is the binding path; head-sharded GQA could stripe across ranks (tp× faster), so the per-link choice is conservative — a documented choice. In pd mode there is no term (documented assumption: the tier is decode-node-local and reload overlaps with prefill). The hostRAM > 0 and hostStreamBps > 0 conjuncts also guard 0/0 on systems without pciePerGPU when the tier is off. TTFT is the only limit that can regress with the tier on: baseS grows by reloadLatencyS, so on a TTFT-tight aggregated mix enabling the tier can return FEWER users — the ttft card names the reload share explicitly.

Reported: limits.ttft = {baseS, ttftS_at_s1, scale}. baseS = per-request prefill+transfer time before queueing; ttftS_at_s1 includes the M/M/1-style queue factor at utilization 1; when the host tier is on, baseS includes the KV-reload share (limits.reload.reloadLatencyS).

2.8 Errors, warnings, exports

• Errors: {ok:false, error:'ZERO_USERS'} when all counts are 0; 'NO_LAYOUT_FITS' per 2.2; 'BAD_INPUT' on missing or NaN required fields, or hostRAM > 0 with no positive reload bandwidth (a system without pciePerGPU and no adv.hostStreamGBps override).
• Warnings: FP4_NO_TC (nvfp4 weights on H200, memory-only, see section 3) and TP_GT_KV_HEADS (tp exceeds kvHeads on a computed-KV model).
• decodeCurve(input, points=160) samples per-user tok/s from 0 to 2 * B_max * replicas users in the requested number of intervals (161 points): tokS = 1/tEff(B) with B = ceil(users * activity / replicas), and B = 1 at 0 users. The curve is non-increasing by construction.
• serializeState / deserializeState round-trip input objects through the URL hash (base64-encoded JSON; non-finite numbers travel as the explicit string sentinels __Infinity__ / __-Infinity__ and are reconstructed on load — plain nulls pass through unchanged).

2.9 Defaults: sources and uncertainty bands

Input	Default	Engine field	Basis	Uncertainty
HBM efficiency	0.65	adv.eta	Batched serving with CUDA graphs reaches 50 to 85% of peak HBM bandwidth; batch-1 eager only 25 to 35% (arXiv 2605.30571, TRT-LLM perf tables)	0.5 to 0.85; halve it for single-stream eager decode
Prefill MFU	0.40	adv.mfu	Kernel-level DeepGEMM FP8 hits 66 to 76% of peak; end-to-end vLLM GB200 R1 measures about 39%	0.3 to 0.5 end-to-end
Prefill budget	0.40	adv.prefillBudget	Modeling decision, not a measurement: an upper bound on prefill's GPU-time share; the cache-aware duty (errata v1.4) only reserves what demand needs, so decode's duty auto-shrinks upward when cache hit cuts prefill demand	Shifts decode and prefill scales inversely; a higher cache hit moves capacity toward decode
VRAM reserve	0.92	adv.reserve	Planning heuristic (activations + CUDA context + fragmentation), not measured	0.03 either way moves capacity by about 3%
KV transfer rate	400 GB/s	kvTransferGBps	Dynamo transport table: NVLink 450 to 900 GB/s, InfiniBand 20 to 50, RoCE 10 to 25, TCP 1 to 3	20 to 900 GB/s, a 45x span; InfiniBand drops the default 10 to 20x
Host RAM presets	Off / 2 / 3 / 4 TB	adv.hostRAM	DGX B300 user guide "2 TB default (up to 4 TB)"; DGX H200 user guide 2 TB; HGX H200 OEM 2 TB typical / 4 TB max (Dell XE9680)	vendor range covered exactly
Host RAM reserve	0.92	adv.hostReserve	Same planning heuristic as the GPU reserve (OS, kernel, transfer buffers), not measured	editable; ±0.03 moves host capacity ~3%
HiCache reload BW	0.55 × PCIe per GPU (35 GB/s H200 / 70 GB/s B300)	adv.hostStreamGBps	Measured PCIe-5.0 KV streaming 15 GB/s (arXiv:2601.19910) to ~50–56 GB/s (vLLM KV offloading connector, arXiv:2512.16056, LMCache tech report Tab. 5); no Gen6 measurement exists — same efficiency band assumed	15–56 GB/s on Gen5, heuristic ×0.55 band
Cache hit	0.90	wl.cacheHit	Agent traces: 94.2% of Codex tokens cacheable, 92.2% achieved with a shared store (vLLM Mooncake blog); plain chat about 40% (FAST'25 Mooncake paper)	0.85 to 0.95 agents, 0.3 to 0.5 chat
Allreduce latency	23 us	layout.arUs	Measured NVSwitch allreduce floor 22.7 us eager, 12 to 15 us with CUDA graphs (nccl-collectives-bench); cross-island PCIe hops several times worse (90 us on the h200-nvl8 tp8 layout)	plus/minus 20%
Layout penalties	1.00 to 1.80	layout.penalty	Island measurements: intra-pair about 336 GB/s vs cross-pair 64 to 76 GB/s (nccl issue #1161); TP8 over PCIe at 30 to 43% efficiency (comm-cost-modeling)	Analytical, plus/minus 20%
Target tok/s per user	20	wl.targetTok	Planning SLO, not physics	User choice
TTFT target	30 s	wl.ttftTargetS	Planning SLO; large-context agent workloads tolerate multi-second TTFT	User choice
Turn interval	30 s	wl.intervalS	Inter-turn gaps measured 5 to 81 s (vLLM Mooncake blog)	Workload
Turns / new input / output per turn	10 / 2200 / 800	wl.turns, wl.newInputTok, wl.outputTok	Codex agent traces: median about 2.2K new tokens per turn, input:output about 131:1 (vLLM Mooncake blog). Live inputs: turns and output tokens grow the session KV via ctxPeak (section 2.3)	Workload

The default workload mix is counts [20, 15, 10, 5, 2] across the five buckets; bucket 5 is evaluated at an editable mean of 750k context.

2.10 Ideal P/D split (errata v1.5)

The result carries a pd block answering: at what prefill:decode GPU ratio is this workload balanced? It is pure arithmetic over numbers the result already carries; no second solve.

prefillCapPerGPUTokS      = pfCapFull / sys.gpus          // FULL node, not the budgeted share; per-GPU capacity is pool-independent (mfu*TF/flopsTok) in both modes
demandPerUserTokS         = wl.activity * T / wl.intervalS    // T = uncached input per turn (2.6)
usersPerDecodeGPU         = decodeUsers / decodeGPUs      // decodeUsers = B_max * replicas / activity; decodeGPUs = decode-side GPUs (2.4)
idealPrefillPerDecodeGPU  = demandPerUserTokS * usersPerDecodeGPU / prefillCapPerGPUTokS
hitRateFor1to4            = clamp(1 - 0.25 * prefillCapPerGPUTokS * intervalS / (usersPerDecodeGPU * newInputTok * activity), 0, 1)

The ratio is a linear-decode-scaling estimate: with DP-attention the KV term (which dominates at long context) shards with the users while the weight floor amortizes, so decode capacity grows near-linearly with decode GPUs. hitRateFor1to4 is the cache-hit rate below which disaggregation beats aggregation at a 1:4 split. It is a property of the mix, not of the current wl.cacheHit: the full wl.newInputTok enters, and the hit rate is the unknown being solved for. B_max = 0 (SLO unreachable at B=1) drives usersPerDecodeGPU to 0 and the threshold to null. The UI renders this as the "Ideal P/D split" block under Bottleneck & limits.

Worked example (GLM-5.3, 150 users @ 200k, 80% cache hit, NVFP4, FP8 KV, b300-hgx; the ex-coders200k example chip): demandPerUserTokS 14.67, usersPerDecodeGPU 19.88, prefillCapPerGPUTokS 4,894 → idealPrefillPerDecodeGPU 0.0596, one prefill GPU per ~17 decode GPUs; P/D pays off below ~16% cache hit at this mix. Caveat: the linear decode extrapolation is exact for the KV term and optimistic for the weight floor at very large decode pools.

3. Hardware presets

Preset	System	GPUs	VRAM/GPU	HBM BW	Dense BF16/FP8/FP4 per GPU	NVLink/GPU	PCIe	Topology	Mode
h200-hgx	8x H200 SXM	8	141 GB	4.8 TB/s	989 / 1979 / n/a TF	900 GB/s	Gen5, 64 GB/s per direction	1 island, NVSwitch domain	agg
h200-nvl8	8x H200 NVL PCIe	8	141 GB	4.8 TB/s	835 / 1670 / n/a TF	900 GB/s (bridges)	Gen5, 64 GB/s per direction	2 islands x 4 GPUs	agg
h200-nvl4	4x H200 NVL PCIe	4	141 GB	4.8 TB/s	835 / 1670 / n/a TF	900 GB/s (bridges)	Gen5, 64 GB/s per direction	1 island (4-way bridge)	agg
b300-hgx	8x B300 (HGX)	8	288 GB	8 TB/s	2250 / 4500 / 13500 TF	1.8 TB/s	Gen6, 128 GB/s per direction	1 island, NVSwitch domain	agg
b300-pd2x8	2 nodes x 8x B300, prefill/decode split	16	288 GB	8 TB/s	2250 / 4500 / 13500 TF	1.8 TB/s	Gen6, 128 GB/s per direction	2 nodes	pd, 8+8 editable
rtxpro6000	1x RTX PRO 6000 Blackwell (workstation)	1	96 GB	1.79 TB/s	500 / 1000 / 2000 TF	none	Gen5, 64 GB/s per direction	single GPU	agg

Engine layouts (penalty multiplies the decode roofline; arUs is the per-allreduce latency; dp8-attn = DP-attention, user-sharded KV, errata v1.4):

Preset	Layouts (tp x replicas: penalty, arUs)
h200-hgx	tp8 x 1: 1.00, 23 us (default); tp4 x 2: 1.05; tp2 x 4: 1.15; dp8-attn: 1.00, 23 us
h200-nvl8	tp2 x 4: 1.00, 23 us (default); tp4 x 2: 1.05, intra-island; tp8 x 1: 1.80, 90 us, crosses islands over PCIe; dp8-attn: 1.80, 90 us — the EP all-to-all crosses the islands too
h200-nvl4	tp4 x 1: 1.00, 23 us; tp2 x 2: 1.00, 23 us (both intra-island); dp4-attn: 1.00, 23 us
b300-hgx	tp8 x 1: 1.00, 23 us (default); tp4 x 2: 1.05; tp2 x 4: 1.12; dp8-attn: 1.00, 23 us
b300-pd2x8	tp8 per node: 1.00, 23 us; dp8-attn on the decode node: 1.00, 23 us
rtxpro6000	tp1 x 1: 1.00, no allreduce (single GPU, tp=1)

Per-system sources:

• h200-hgx / h200-nvl8. nvidia.com H200: 141 GB HBM3e, 4.8 TB/s, NVLink 4 at 900 GB/s, "2- or 4-way NVLink bridge: 900 GB/s per GPU" (the 4-way bridge is what makes 2 islands of 4), PCIe Gen5 128 GB/s bidirectional (about 64 GB/s per direction), TDP up to 700 W SXM / 600 W NVL. NVIDIA quotes TFLOPS with sparsity (1979/3958 SXM, 1671/3341 NVL); the catalog stores dense = sparse / 2. The HGX H100/H200 datasheet adds 4x NVSwitch Gen3 and a 7.2 TB/s aggregate NVLink domain.

• Island penalties (h200-nvl8). Measured on the analogous H100 NVL/PCIe topology: intra-pair P2P about 336 GB/s, cross-pair 64 to 76 GB/s over PCIe (nccl issue #1161); end-to-end NVL step time 2.89x SXM (bluenotebook H100 study); TP8 over PCIe-class links at 30 to 43% efficiency (comm-cost-modeling). These drive the 1.80 penalty and 90 us allreduce latency on the cross-island tp8 layout.

• h200-nvl4. Same card as h200-nvl8; with 4 cards the 4-way bridge spans all GPUs: one island, no cross-island PCIe penalty. Only 564 GB total HBM: the earlier 550 GB DeepSeek estimate did not fit (usable 518.9 GB); the verified 510.3 GB checkpoint fits by 1.6% (section 4).

• b300-hgx / b300-pd2x8 compute and fabric. nvidia.com HGX: FP4 108 PF dense per node = 13.5 PF per GPU, FP8 72 PF sparse per node = 4.5 PF dense per GPU, BF16 36 PF sparse per node = 2.25 PF dense per GPU. BF16 is not cut vs B200; the real Blackwell Ultra cuts are FP64 (10 TF vs 296 TF per node) and INT8 (3 POPS vs 72 POPS per node). NVLink 5 at 1.8 TB/s per GPU, 14.4 TB/s aggregate; PCIe Gen6 at 256 GB/s bidirectional.

• b300 memory. DGX B300 user guide: 288 GB per GPU, 2.3 TB node total. NVIDIA HGX reference architecture: 288 GB and 8 TB/s per GPU. Blackwell Ultra datasheet (NVIDIA-authored partner mirror): 270 GB at 7.7 TB/s in the HGX B300 column, 279 GB in the GB300 column. Blackwell technical brief v2.1 (bot-gated mirror): 288 GB at 8 TB/s. NVIDIA dev blog: "up to 288 GB by SKU".

• b300-pd2x8 KV transfer. The preset note quotes the HGX page's 1.6 TB/s per node networking figure; the engine default is a deliberately conservative effective 400 GB/s, editable. Transport reality per the Dynamo communication guide: NVLink 450 to 900 GB/s (intra-pod only), InfiniBand RDMA 20 to 50 GB/s (adds 200 to 500 ms TTFT), RoCE 10 to 25 GB/s (300 to 800 ms), TCP 1 to 3 GB/s (about 98 s TTFT measured). The P:D ratio is workload-derived, not universal: 1:1 at 2K/2K contexts, up to 1:8 to 1:10 for fast-chat shapes (lmsys large-scale EP, vLLM GB200 disagg).

• rtxpro6000. nvidia.com RTX PRO 6000 Blackwell Workstation Edition datasheet and the RTX PRO Blackwell GPU architecture whitepaper v1.1: 96 GB GDDR7 ECC, 512-bit at 28 Gbps = 1792 GB/s, 188 SMs at 2617 MHz boost, PCIe 5.0 x16, 600 W. "4000 AI TOPS" is FP4 with sparsity; the catalog stores dense = sparse / 2 (2000/1000/500 TF), matching the Server Edition page's peak FP4 4 PF / FP8 2 PF / BF16 1 PF. No NVLink: TP1 only. The vendor 96 GB figure is kept as a decimal (the GDDR7 frame is physically 96 GiB, about 103.1e9 B) — conservative, same convention as the H200's 141 GB. FP4 tensor cores are present, so nvfp4 weights and FP4 KV both run at full rate.

Documented NVIDIA conflicts (B300)

Quantity	Value A	Value B	Catalog uses
Per-GPU memory	288 GB: DGX B300 user guide, HGX reference architecture, tech brief, dev blog ("up to 288 GB by SKU")	270 GB at 7.7 TB/s (Blackwell Ultra datasheet, HGX B300 column); 279 GB (GB300 column)	288 GB, surfaced as a UI tooltip
Node total	2.3 TB (8 x 288 GB, DGX user guide)	2.1 TB (HGX/DGX marketing pages)	follows 8 x 288 GB
HBM bandwidth	8 TB/s (reference architecture, tech brief, dev blog)	7.7 TB/s (datasheet HGX column)	8 TB/s
FP4 dense per GPU	13.5 PF (HGX page: 108 PF / 8)	13.1 PF (tech brief: 105 PF)	13.5 PF

Reading: 288 GB is the physical chip maximum by SKU; 270 and 279 GB look like derated or CUDA-available figures per SKU column. The catalog picks the larger documented values so the memory pool is not understated, and the UI shows the conflict rather than hiding it.

No FP4 tensor cores on Hopper. NVFP4 compute is Blackwell-only (NVIDIA NVFP4 blog). On H200 the fp4 FLOP slot is null: selecting nvfp4 weights halves weight bytes but keeps FP8 compute rates, and the engine emits an FP4_NO_TC warning. NVFP4 itself costs at most 1% accuracy vs FP8 with 1.8x memory savings.

4. Model presets

Architecture data from fetched HuggingFace configs and model cards:

Preset	Total / active params	Layers (KV-growing)	Attention	Experts routed/top (+shared)	KV B/token BF16 / FP8	Const state/seq	Native precision	Context
Kimi K3	2.8T / 104B	93 (24)	24 Gated MLA + 69 KDA linear	896 / 16 (+2)	27,648 / 13,824	~0.2 GB (derived)	MXFP4 (QAT)	1M
GLM-5.3	744B / 40B	78 (78)	MLA + DSA indexer (top-k 2048), all layers	256 / 8 (+1)	89,856 / 44,928	0	FP8 (BF16 repo exists)	1M
GLM-5.3-Flash	320B / 18B	45 (11)	11 DSA/MLA + 34 linear	288 / 8 (+1)	11,264 / 5,632	~20 MB	FP8	1M
DeepSeek-V4.1-Flash	552B backbone + 196B Engram / 8B prefill, 16B decode	40 (40)	MQA latent (1 kv head, 512 dim) + CSA2 KV sharing + SWA 128	384 / 6 (+1)	official: 890 fixed (naive BF16 formula: 81,920)	folded into official KV	FP8 + FP4 experts, FP4 KV	1M (YaRN)
Qwen3.8-Flash-Next	180B / 6B	48 (12)	12 QSA full-attention (2 KV heads × 256 dim) + 36 Gated-DeltaNet linear	512 / 10 (+1)	24,576 / 12,288	~113 MB (fp32 state, derived)	BF16	256k native (YaRN → 1M)
Qwen3.8-27B	27.8B / 25.6B	64 (16)	16 QSA full-attention (4 KV heads × 256 dim) + 48 Gated-DeltaNet linear	dense — none	65,536 / 32,768	~151 MB (fp32 state, derived)	BF16	262k native (YaRN → 1M)

Engine compute fields: activePrefillParams 104B / 40B / 18B / 8B / 6B / 25.6B. activeDecodeBytes 52 GB (104B at about 0.5 B/param MXFP4), 40 GB (FP8), 18 GB (FP8), 10 GB (16B decode at mixed FP4/FP8), 6 GB (FP8), 25.6 GB (FP8; informational — the dense engine path reads wTot per step). attnCoef (editable catalog fields, formula-exact per section 2.6): K3 1,769,472 (= 4 x 24 x 96 x 192), GLM-5.3 5,111,808 (= 4 x 78 x 64 x 256), Flash 720,896 (= 4 x 11 x 64 x 256), DeepSeek 5,242,880 (= 4 x 40 x 64 x 512), Qwen 294,912 (= 4 x 12 x 24 x 256, QSA layers only), Qwen3.8-27B 393,216 (= 4 x 16 x 24 x 256, QSA layers only). The four MLA/latent presets use kvSharding='replicated' because their latent KV cannot be head-sharded; Qwen3.8-Flash-Next (kvHeads 2) and Qwen3.8-27B (kvHeads 4) use kvSharding='heads' (GQA — head-shards while tp ≤ kvHeads, replicates per rank with the TP_GT_KV_HEADS warning above that).

Weight bytes per quantization (catalog values; total params count for memory because all routed experts are resident):

Preset	BF16	FP8	NVFP4/MXFP4	Note
Kimi K3	not published	1.6 TB (estimate)	1.6 TB (native MXFP4)	vLLM day-0 post lists 8xB300 (or 16xB200) as the minimum deployment; K3 mixes MXFP4 routed experts with BF16 attention, dense, shared experts, and lm_head
GLM-5.3	1.52 TB	0.76 TB	0.38 TB	753.3B checkpoint tensors vs 744B vendor headline
GLM-5.3-Flash	0.656 TB	0.328 TB	0.164 TB	FP8 = 306 GiB per the deployment guide
DeepSeek-V4.1-Flash	1.42 TB	510.3 GB (verified)	510.3 GB	verified HF checkpoint: FP4 experts 278.6 GB + FP8 backbone/engram 204 GB; nvfp4 slot = fp8 (experts already FP4); engram ≈203 GB host-offloadable; fits 4× H200 by 1.6%
Qwen3.8-Flash-Next	360 GB (verified blob sum)	185.5 GB (official mirror)	92.8 GB (ESTIMATE)	180B = 125B backbone + 51B n-gram embedding + 4B MTP; no official NVFP4 checkpoint exists — the nvfp4 slot is a plain halve, not a shipped quant
Qwen3.8-27B	55.56 GB (verified blob sum)	30.87 GB (official FP8 mirror)	21.92 GB (official NVIDIA NVFP4 build)	all three slots verified, no ESTIMATEs: HF API blob sums 55,563,006,776 / 30,866,866,928 / 21,921,697,280 B; 27.78B checkpoint = 24.35B text layers + 2.54B embed/lm_head + ~0.51B vision + ~0.37B MTP (derived); the NVFP4 build is NOT a plain halve — BF16 embed/lm_head/vision/MTP ride along

KV math per model:

Kimi K3 (config, card)

• Only the 24 Gated-MLA layers grow KV: 24 x (kv_lora_rank 512 + rope 64) elements x 2 B = 27,648 B/token BF16, 13,824 at FP8 KV. The other 69 KDA linear-attention layers hold a constant ~0.2 GB recurrent state per sequence regardless of context and add zero per-token KV.
• Cross-check: a community 16xH200 deployment measured a KV pool of about 25.8 KB/token at BF16 (HF discussion #136), within about 7% of the computed 27.0 KiB/token. The same MLA formula, layers x (kv_lora_rank + rope) x bytes, was verified against 11 model families' HF configs during research.

GLM-5.3 (config, card)

• All 78 layers grow KV: 78 x (512 + 64) x 2 B = 89,856 B/token BF16, 44,928 FP8. The densest KV of the four.
• The DSA indexer (top-k 2048) sparsifies attention compute only; the KV cache size is unchanged. The indexer's own K pool adds roughly 2.7 KB/token over the 21 shared-indexer layers at FP8 (derived, not modeled).

GLM-5.3-Flash (config)

• 11 DSA/MLA layers with rope 0: 11 x 512 x 2 B = 11,264 B/token BF16, 5,632 FP8. The 34 linear-attention layers carry ~20 MB of constant state per sequence.

DeepSeek-V4.1-Flash (config, card, tech report)

• Weight size VERIFIED at 510.3 GB (the earlier 550 GB estimate was ~8% high): the HF API blob sum is 510,296,708,312 B, matching Tensorfuse ≈510.30 GB and the vLLM recipe vram_minimum_gb 614 (weights × 1.2). Breakdown: FP4 routed experts 557.17B params = 278.6 GB, FP8 dense/attention/engram 204.0 GB — the ~203 GB engram shards store the 196.6B-parameter lookup tables at FP8 (tech report §2.4.2, "Both the embedding table and the key/value projections use FP8 precision"). The engram tables are host-offloadable (SGLANG_ENABLE_DSV41_ENGRAM_HOST_TABLE=1 → ~307 GB GPU-resident) — outside the GPU-only model scope, a documented footnote, no host tier modeled. At 510.3 GB the checkpoint fits 4× H200 NVL (usable 518.9 GB) by 1.6%, and the nvfp4 slot equals fp8 because the experts already ship FP4: the catalog models the shipped mixed checkpoint, not a re-quantized backbone.
• The naive GQA formula gives 2 x 40 layers x 1 kv head x 512 dim x 2 B = 81,920 B/token BF16. That is wrong by roughly 90x.
• The official deployed KV is 890 B/token, fixed: FP4 E2M1 entries with one E4M3 scale per 16 channels, plus CSA2 full/reindex/reuse KV sharing across layers (component sum ~768 B plus metadata gives 890). The model card's 1M-context figure, about 0.89 GB, confirms it: 890 B x 1e6 = 0.89 GB.
• Engine behavior: kvMode='official' pins kvTok to 890 regardless of the KV-precision control; the UI disables that control and shows a note. A 128k-context user then costs 128,000 x 890 = 113.92 MB of KV.

Qwen3.8-Flash-Next (config, card, vLLM recipe)

• Hybrid Gated-DeltaNet + QSA: only the 12 QSA full-attention layers (every 4th of 48) grow KV: 12 x 2 kv_heads x 256 head_dim x 2 (K+V) x 2 B = 24,576 B/token BF16, 12,288 at FP8 KV, 6,144 at FP4 KV. The 36 Gated-DeltaNet linear-attention layers hold an O(1) recurrent state of ~113 MB fp32 per sequence (constStateBytes 1.18e8) regardless of context. A QSA indexer adds roughly 768 B/token (derived, not modeled).
• Weight size VERIFIED at 360.00 GB BF16 (HF API blob sum). The official FP8 mirror is 185.5 GB (vLLM recipe: 172.78 GiB; GB300 TP2/TP4; 4x H100 FP8 about 1,430 tok/s at concurrency 64). The NVFP4 slot 92.8 GB is an ESTIMATE — no official NVFP4 checkpoint exists.
• GQA with only 2 KV heads: the heads kvSharding guard head-shards KV on tp2 but treats tp4/tp8 as replicated per rank (with the TP_GT_KV_HEADS warning), matching serving-engine behavior for 2-head GQA. DP-attention layouts sidestep it by sharding users: kvPerGpu = kvAvg/dp.
• Context 262,144 native, 1M with YaRN; Artificial Analysis measures 52.7 tok/s on the hosted API.

Qwen3.8-27B (config, card, vLLM recipe)

• The catalog's first DENSE model: no routed experts, so the engine's decode step reads the full sharded weights every step (wStep = wTot); the MoE amortization curves never apply, and activeDecodeBytes is informational for dense models.
• Same hybrid skeleton as Flash-Next, one size up: only the 16 QSA full-attention layers (every 4th of 64) grow KV: 16 x 4 kv_heads x 256 head_dim x 2 (K+V) x 2 B = 65,536 B/token BF16, 32,768 at FP8 KV, 16,384 at FP4 KV. The 48 Gated-DeltaNet layers hold an O(1) recurrent state of ~151 MB fp32 per sequence (constStateBytes 1.51e8 = 48 layers x 48 value-heads x 128 x 128 x 4 B — the same formula that reproduces Flash-Next's ~113 MB). A QSA indexer exists but is not modeled (same [derived] omission as Flash-Next).
• GQA with 4 KV heads: head-shards on tp2/tp4, replicates per rank on tp8 with the TP_GT_KV_HEADS warning — h200-nvl4's tp4x1/tp2x2 layouts shard cleanly, h200-hgx tp8 pays the replicated-KV tax (FIX-RTX2 pins it: 100 users @ 128k decode-bound at 30).
• Checkpoint decomposition (config shapes calibrated against the safetensors total 27,781,427,952 BF16 params): 24.35B text layers + 2.54B embed/lm_head = ~26.90B text, plus ~0.51B vision tower and ~0.37B built-in MTP head (derived). Active prefill = text layers + lm_head = 25.62B; the embedding is a lookup (not a matmul), and vision + MTP are excluded — text serving is what the vLLM recipe verifies. All three weight slots are verified official checkpoints: BF16 blob sum 55.56 GB, the official Qwen3.8-27B-FP8 mirror 30.87 GB (block-scaled, cross-checked against the recipe's 14.28 GiB/GPU at TP2 = 30.67 GB), the official nvidia/Qwen3.8-27B-NVFP4 build 21.92 GB — no ESTIMATE slots. The NVFP4 build is not a plain halve: BF16 embed/lm_head/vision/MTP ride along.
• Multimodal (image-text-to-text, SigLIP-class vision tower). Context 262,144 native, 1M with YaRN. The recipe's headline — "fits one Blackwell GPU in every precision" — is what the rtxpro6000 preset models: NVFP4 weights 21.92 GB on a 96 GB card leaves ~66 GB for KV.

Quantization options: FP8 (E4M3) is standard on Hopper and Blackwell with about 0.5% accuracy loss; NVFP4 is Blackwell-only with at most 1% loss vs FP8 and 1.8x memory savings (NVIDIA NVFP4 blog). FP8 KV halves KV bytes; FP4 KV (NVFP4, errata v1.4) quarters them — Blackwell only, the UI disables the option with a tooltip on H200, and official-KV models ignore it (DeepSeek stays at its published 890 B/token).

5. Limitations

• Analytic estimates, not benchmarks. Every number is a roofline or queueing approximation with an editable efficiency factor. Validate against your own serving runs before committing capacity. Published serving results served only as sanity anchors during research, never as engine inputs.
• MoE decode amortization is a heuristic. TP layouts use wStep(B) = min(wTot, wAct*(1 + B/Bsat)), interpolating between the active-bytes regime and the all-experts regime; DP-attention layouts use the expert-hit-fraction curve wAct + (wTot-wAct)*(1-(1-expertsPerTok/experts)^B) (errata v1.4). Shared experts ride inside activeDecodeBytes (always on); both terms amortize routed experts only. Real kernels add cache-resident expert scheduling effects between the regimes.
• attnCoef is a first-order model of attention FLOPs (4 * growingLayers * attnHeads * dKv; preset values match this rule exactly and stay editable). Exact attention FLOPs depend on kernel details (FlashAttention tiling, MLA absorb, chunked prefill).
• DSA and CSA sparsify compute, not cache size. GLM's DSA indexer and DeepSeek's CSA sharing cut attention FLOPs and, for CSA, the deployed KV, which is already folded into DeepSeek's official 890 B/token. GLM's indexer K pool (~2.7 KB/token, derived) is not modeled.
• FP4 on H200 is memory-only. Hopper has no FP4 tensor cores; nvfp4 weights halve bytes but FLOPs stay at the FP8 rate (FP4_NO_TC warning). Kimi K3 on Hopper additionally needs W4A16 repacking at roughly +33% expert bytes, which the catalog does not model.
• Disaggregation assumes lossless KV transfer at the configured kvTransferGBps: no congestion, retransmit, or scheduling overhead. Real fabrics add 200 to 800 ms TTFT (Dynamo guide), and transfer-bound configurations exist where disaggregation gains nothing.
• The duty cycle is a coarse time share. Errata v1.4 makes it cache-aware (share = min(prefillBudget, demand/pfCapFull)), but it still models the GPU-time split as a static fraction of aggregate demand. Real chunked-prefill interleaving beats a hard split at low load and can be worse at tail.
• Reserve 0.92 is a planning heuristic, not a measurement.
• Buckets 1 to 4 are evaluated at their cap (conservative for memory); bucket 5 uses an editable mean (750k default). Memory holds all users' KV even at activity < 1 because inactive sessions stay resident; decode and prefill scale with activity.
• The host KV tier assumes a uniform-mix steady state (errata v1.6): no per-user idle timers are modeled — the engine never decides which session is offloaded, it assumes HBM keeps active sessions resident first and its spares fill with idle. Real tiers demote pages under pressure (SGLang HiCache design).
• HiCache L2 is private per serving instance: host KV is not pooled across nodes or instances, and the model does not pool it either.
• Decode KV must stay HBM-resident: the host tier serves waiting sessions only; waking a session streams its KV over PCIe at 15 to 56 GB/s measured, roughly 75 to 240x below HBM bandwidth.
• pd-mode reload is assumed to overlap prefill (decode-node-local tier, no TTFT term). In aggregated mode TTFT is the one path that can regress when the tier is on: on a TTFT-tight mix, enabling the tier can LOWER the seat count (the ttft card names the reload share).

6. Non-goals

• No backend, server code, accounts, database, analytics, or any network call at runtime. State lives in the URL hash only.
• No npm, node_modules, bundler, TypeScript, JSX, build step, or CDN resources (no web fonts, no chart libraries). The app ships as editable plain files.
• Exactly 6 system presets and 6 model presets plus the custom model form (h200-nvl4 / rtxpro6000 and qwen3.8-flash-next / qwen3.8-27b were added additively). No Rubin, B200, MI300X, or GB200 NVL72 in v1; the data catalog is data-only, so new presets are additive entries.
• No live fetching of GPU or model data, no service worker, no i18n framework (English strings inline).
• Charts are hand-built SVG strings; no framework libraries (no React, Vue, Chart.js, D3).
• The engine implements the pinned analytic spec exactly. No ML-accuracy emulation of real kernels; deviations from the spec are bugs.