HiSparse concurrency calculator

Estimate ordinary GPU KV capacity, HiSparse’s fully offloaded steady state, and native indexer offloading for sparse-attention models. Results are planning estimates, not guaranteed serving limits.

Model

GPU

H200 · 141 GB HBM3e per GPU

FP8 KV cache

Workload and topology

Sequence length

Data parallel size

Tensor parallel size

Expert parallelism across DP×TP

With EP off, every DP replica holds the same TP-sharded experts, so DP no longer changes available KV HBM per GPU.

HiSparse

Minimum HiSparse host pool / DP rank

–

Avoids host capacity becoming the concurrency limit.

Host pool / DP rank

Indexer offload total

Hot-buffer rows

System RAM budget

Native indexer offloading extends the prefix cache in CPU memory. Active indexer history must still fit in HBM, so this pool does not directly raise the running-request limit.

Share the host cache across TP ranks

Sharing changes physical CPU RAM usage, not logical request capacity.

Advanced layout values

Available KV HBM / GPU override

Automatically estimated from the model’s TP/EP-sharded weights at 92% GPU utilization. Override only to match vLLM’s reported startup value.

Baseline GPU block bytes

Baseline block tokens

Baseline fixed blocks / request

Baseline admission reserve blocks

HiSparse GPU block bytes

HiSparse indexer block tokens

HiSparse fixed blocks / request

HiSparse admission reserve blocks

Host block bytes

Host block tokens

Retained pages / group

Estimated non-HiSparse total

–

Estimated HiSparse total

–

Estimated capacity multiplier

–

Topology: – Model weights / GPU: – Physical host allocation: – Indexer offload coverage: –

Non-HiSparse

HiSparse

HiSparse memory composition

Concurrency / DP group –

Standard KV remains resident until pressure activates hot buffers and releases older pages.

GPU HBM per DP group –

Model weights – vLLM runtime + HBM headroom – Hot buffers – Indexer KV – Fixed per-request state – Admission reserve – Standard KV – Unused KV –

CPU host pools per DP group –

HiSparse latent KV – Native indexer offload – Unused –

Concurrency vs. sequence length

–

HiSparse Non-HiSparse

Shows total concurrency across the deployment and assumes the minimum non-limiting HiSparse host pool at each sequence length. Native indexer offloading does not change active-request HBM.

KV placement by layer

–

Offloaded to host Full history on GPU Fixed-size state on GPU Shared from another layer Static table in host memory

With HiSparse. Without it, every cache stays on the GPU. Offloaded layers keep a top-K hot buffer on the GPU.

Hover or tap a layer for details.

Per-DP-replica accounting

Resource	Non-HiSparse	HiSparse
GPU pool capacity
Unique GPU / request
↳ Indexer history	included
↳ Hot buffers	–
↳ Retained pages	–
Unique host / request	–
Native indexer offload / request	–
Native indexer offload coverage	–
GPU concurrency limit
Host concurrency limit	–
Final concurrency limit

Model and assumptions

HiSparse GPU usage is the unique indexer history plus per-request hot buffers, two retained resident pages per hot group, and fixed per-request state that stays on the GPU (sliding-window KV, compressor or linear-attention state). Each block costs the full packed stripe, so fixed state is counted in blocks. Models with small-block sliding or resident groups also hold an admission reserve for the next request’s prompt and in-flight prefill chunks (8,192 batched tokens, async scheduling). HiSparse host usage includes the full offloaded KV history. The native OffloadingConnector pool caches completed indexer blocks for prefix reuse, but does not remove active indexer history from HBM. The sparse-MLA latent and indexer KV are replicated across TP ranks; therefore TP does not directly divide KV bytes per request. It can still leave more HBM for KV by reducing the per-GPU weight shard. With TP host sharing enabled, one host pool serves the whole TP group. This is a steady-state estimate; spill transitions, request-length skew, routing imbalance, and temporary workspaces can lower the reachable concurrency. The automatic HBM estimate shards routed-expert weights across DP×TP when EP is enabled; otherwise each DP replica duplicates them. Shared weights always shard across TP.