Explainer · Articles

Why VRAM matters more than GPU speed for LLMs

A faster GPU that can’t hold the model loses to a slower one that can. Here’s the one spec to read first.

Published Oct 27, 2026 Updated Oct 27, 2026 6 min read

We earn from qualifying purchases via affiliate links. Disclosure.

Thumbnail: “VRAM Beats Speed” beside an ASUS RTX 5060 Ti maker photo with a 16 GB VRAM tag — Illustration by FitMyModel with maker product photos (local review copy).

Key takeaways

First check that the model file fits in VRAM, with room left for context.
When it doesn’t fit, part of the model moves to system RAM and runs more slowly.
Once it fits, memory bandwidth, not raw compute, sets how fast words appear.

Gamers shop by frame rates. Local AI shoppers should shop by a different number: how many gigabytes of memory sit on the card. Get that wrong and the fastest GPU in the store turns into a bottleneck.

Fit comes first

Ollama’s FAQ describes three outcomes after a model loads: 100% GPU, 100% CPU, or a split such as 48%/52% CPU/GPU when the model is “loaded partially onto both the GPU and into system memory”. The split works, but every layer left in system RAM runs at system-RAM speed.

So the first question for any card is simple. Does the download fit? gpt-oss 20B is 14 GB in Ollama. On a 12 GB card it can’t fit fully. On a 16 GB card it fits, with little room to spare (our estimate).

Illustration: a 14 GB model fits fully on a 16 GB card but splits between GPU and system RAM on a 12 GB card — Illustration: gpt-oss 20B (14 GB) on a 16 GB card versus a 12 GB card.

Then bandwidth sets the pace

“On decode, the model weights are loaded for every token, which is generated via much less-expensive vector-matrix multiplication.”

From Baseten Inference Engineering: Calculating inference bottlenecks

That is why AI people talk about memory bandwidth. NVIDIA lists the RTX 5060 Ti at 448 GB/s and 16 GB. A rough rule that follows from the quote: the more gigabytes the model reads per word, the fewer words per second. We won’t turn that into a tokens-per-second number without measuring it.

ASUS Dual GeForce RTX 5060 Ti 16GB OC graphics card — ASUS Dual RTX 5060 Ti 16GB: 16 GB at 448 GB/s per NVIDIA. Maker product photo (local review copy).

Context eats VRAM too

The model file is not the whole bill. The context cache grows with the conversation. Ollama defaults to a 4,096-token context and lets you raise it, and it can quantize the cache: q8_0 uses “approximately 1/2 the memory of f16”, per its FAQ. That’s why our tier pages budget 1–4 GB of headroom above the download (our estimate).

What to do with this

Pick the model first in the model finder.
Buy the card whose VRAM holds it with headroom. A 16 GB card covers 20B-class models; 24–32 GB covers 27B–35B models.
Only then compare speed specs between cards that fit.