The GPU is only half the answer

There is an open tool called llmfit that answers a question everyone asks and almost nobody measures: will this model actually run on my hardware, and how fast. It keeps a community table of real results, the kind you only get by running the thing, not by reading a spec sheet. I ran it on two of my graphics cards and added their numbers to that table. Forty models, two cards, and two of the results stopped me.

Two very different cards

The cards could not be more different in age or size. An old RTX 2080 with 8 GB, and a newer RTX 5070 with 12 GB. I measured every model the same careful way: warm up first, three runs, average the tokens per second, same quantization on both. Twenty nine models on the small card, forty on the larger one, all local, all reproducible.

I expected a clean story: newer card, bigger numbers, end of article. That is not what the data said.

Lesson one: VRAM headroom is a cliff, not a slope

Take a 14B model. On the 8 GB card it does not fit, so part of it spills into system memory and the CPU has to help carry it. It crawls at about 17 tokens per second. Give the exact same model, same weights, same quantization, four more gigabytes of VRAM so it fits entirely on the GPU, and it jumps to about 58.

More than three times faster, for one reason only: it stopped talking to system RAM. There is no gentle slope here. There is a cliff between “just does not fit” and “just fits”, and almost all of the speed lives on one side of it. That is why the four gigabytes that let a model fit matter far more than the raw teraflops printed on the box. You are not buying speed, you are buying the right to stop offloading.

Lesson two: a better GPU can lose

Now the result I did not see coming. My 30B mixture-of-experts model ran at roughly 35 tokens per second on the old 2080, and only about 26 on the newer, stronger 5070. The worse card, on paper, was faster in practice.

The reason is the machine around the card, not the card itself. That model is too big to fit in either GPU, so its experts stream from system memory on every token. The moment that happens, the ceiling stops being the GPU and becomes the memory bandwidth of the host. The 2080 lives in a workstation with fast quad channel memory. The 5070 lives in a compact mini PC with slower dual channel memory. Feed the same hungry model from a slower memory bus and it runs slower, however good the GPU is.

So the honest rule of thumb is two lines, not one. Buy VRAM to fit the model. Buy memory bandwidth to feed it. The GPU is only half the answer.

Why measure in the open

None of this needed permission or a budget. A benchmark submission is a few kilobytes of JSON, but it is verifiable, it makes a shared tool a little more useful for the next person, and it puts two cards on the map that were not there before. Reading someone’s launch post and moving on is easy. Running the tool on your own hardware and sending the numbers back is where you actually learn something.

The most surprising result was also the most honest one, so I published it: sometimes the newer, more expensive card is the slower one. The community table is where those numbers belong.

The tech moves fast. The fundamentals don’t.

Akciali