What It Does

Two questions decide whether an LLM feels fast: how long until the first token appears, and how quickly tokens stream after that. They have different bottlenecks, and this tool estimates both.

Pick a model and a GPU (or enter your own bandwidth / FLOPS / memory), set your prompt and output lengths and batch size, and get:

Time to first token (TTFT) - the prefill pass
Decode speed - per-request tokens/second while streaming
Total latency - end to end for one request
Aggregate throughput - tokens/second across the batch

The Model

It’s a first-order roofline estimate, which is the right mental model for LLM serving:

Prefill is compute-bound. Processing the prompt is a big matrix multiply: roughly 2 × active_params × prompt_tokens FLOPs, divided by the GPU’s achievable throughput (peak FLOPS × MFU). This sets TTFT.
Decode is memory-bandwidth-bound. Generating each token re-reads the model’s weights from memory, so speed is roughly memory_bandwidth ÷ active_weight_bytes - not compute. This is why a bigger, faster GPU with the same bandwidth barely moves decode speed, and why quantization (fewer bytes per weight) speeds decoding up.

Crucially, it separates total params (which set the memory footprint) from active params (which drive compute and decode traffic), so a Mixture-of-Experts model like Mixtral or GPT-OSS is modelled correctly - huge in memory, cheap per token.

GPU specifications are representative published figures and every field is editable, so you can plug in your exact accelerator.

For the memory-fit side, use the GPU VRAM Calculator; for API rate limits, the Rate-Limit Planner; for spend, the Reasoning Cost Calculator.

Limitations

First-order, not a benchmark. It ignores KV-cache read bandwidth (which grows with context and eventually dominates long-generation decode), attention overhead, kernel-launch latency, speculative decoding, and framework differences. Real numbers will differ; treat it as the right order of magnitude and the right intuition.
Ideal sharding assumed when a model doesn’t fit one GPU - it flags the shard count but doesn’t model inter-GPU communication cost.
MFU is your input. Achievable utilization varies widely by kernel and sequence length; 30-50% is a reasonable default.