What actually fits in 8 GB of VRAM

Every "best local LLM" list quotes the download size. The download is the small half: what decides whether a model runs on your card is the weights plus its context, and what decides whether it is any use is whether it can call a tool at all. Here are both, measured.

Last checked 2026-09-21

Free to cite. Quote any of this with a link back to this page.

The number that matters is not the file size

A GGUF's file size is the weights. Loading it costs the weights plus the KV cache for the context you intend to run, plus the runtime's own overhead. The cache is the part the lists leave out, and it scales with context: the same model at 4,096 tokens and at 32,768 tokens are two different machines' worth of memory.

These are the figures Eleos ships in its signed catalogue, each one the memory a build needs at its full 32,768-token context, with the KV cache at q8_0.

In GiB, because that is the unit a card is sold in. An "8 GB" card is 8 GiB, and quoting a model in decimal gigabytes against it overstates what fits by about seven per cent — which is the difference between a fit and a refusal at exactly the sizes that matter here:

"Card needed" is the loaded size plus 25% headroom, which is what Eleos actually requires before it will start one: a card with nothing spare stutters, and something else on your desktop always wants some of it.

So on an 8 GiB card, at full context: the 2B and the 4B fit, and the 9B does not — 8.40 GiB of model on an 8 GiB card, before headroom. A 5.29 GiB file looks like a comfortable fit and is not one: the KV cache is 2.81 GiB of it.

The floor an agent needs, which is higher than you think

Chatting with a model and giving one a job are different memory problems. An agent harness sends its tool definitions with every turn, before you have typed anything:

Measured by pointing the harness at a logging proxy and reading what it actually sent, not estimated from schema sizes.

The consequence: a model published with a 4,096 or 8,192-token context cannot be an agent, on any machine, however much memory you have. The prompt does not fit. Eleos refuses to start one below 28,672 tokens for that reason, and that single rule disqualifies most of the small models people recommend for 8 GB cards.

Whether it can call a tool at all

Context is necessary and not sufficient. Measured on the processor, five runs each, on the same prompt with one tool available:

Phi-4 Mini does not fail to format a call. It denies having tools, in prose, having just been handed one. It is a good small model and it is not an agent, and no amount of VRAM changes that.

CPU medians, deliberately: the badge is a fact about the model, so it must not depend on which card ran the test. A GPU is several times faster.

So, for an 8 GB card

• Qwen3.5 2B Q4_K_M — 3.20 GiB loaded, calls tools reliably, leaves the card free for everything else you are running. The one to start with.
• Qwen3.5 4B Q4_K_M — 5.03 GiB loaded, the same tool reliability, slower. Worth it when the 2B's answers are not good enough.
• Qwen3.5 9B — no, and shortening the context does not rescue it. At the 28,672-token agent floor it still wants 8.04 GiB, which is more than the card has before any headroom. Below that floor it is not an agent at all. This one needs a bigger card.
• Anything published at 4,096 or 8,192 tokens — fine to chat with, useless as an agent, whatever the benchmarks say.

Eleos offers the 2B today, in four quality levels, and starts it for you: see local AI on Windows.

Questions

Does quantisation save as much as it looks like it does?

Less than the file sizes suggest. Between Q4_K_M and Q6_K the 4B's file grows by 0.73 GiB and its loaded size by exactly the same — the KV cache does not change with the weights' quantisation, so the proportional saving shrinks as your context grows. On the 4B at full context the cache is 2.19 GiB whichever build you pick.

Why is the KV cache at q8_0 rather than full width?

It halves what a long context costs in memory for a quality difference nobody has been able to show us in an agent's output. On a 4B at full context that is about two gigabytes, which is the difference between comfortable on an 8 GB card and scraping it.

Can I run a bigger model by shortening the context?

Yes, down to a point, and the point is 28,672 tokens if you want the model to use tools. Below that the harness's own prompt no longer fits and the first message is refused. It is a smaller lever than it sounds: the 9B Q4 saves only

0.35 GiB going from 32,768 tokens to the floor, because most of its footprint is weights.

Is more VRAM the answer?

Only after the two rules above are satisfied. A 24 GB card running a model published at 8,192 tokens still cannot run an agent on it.

Where to go next

• Local AI on Windows, offline and on your own machine
• Ten dead ends between download and first answer
• LM Studio alternatives on Windows, compared