Eighty models on one card, and the protocol behind them

There is an open tool called llmfit that answers a question everyone asks and almost nobody measures: will this model actually run on my hardware, and how fast. It keeps a community table of real results, the kind you only get by running the thing rather than reading a spec sheet.

Until this campaign, every entry in that table came from an NVIDIA card through CUDA. I measured 80 models on a single AMD card through Vulkan and contributed the results, which makes this the first AMD hardware and the first Vulkan backend in the table.

The numbers are the easy part. What makes them worth anything is the protocol, so that is most of what follows.

The platform

One machine, one card under test.

Two build dependencies are worth writing down because they cost me an evening. glslc comes from the glslc package, not from glslang-tools where you would look for it, and the CMake configuration also requires spirv-headers. Neither failure message points at the right package.

One thing I deliberately do not claim: the memory channel configuration of this host. An earlier note of mine recorded it as quad channel at roughly 102 GB/s, and that turned out to be wrong. Since host memory bandwidth is exactly what decides throughput for a model that does not fit in VRAM, an unverified figure there is worse than no figure. I measured 62 GB of RAM and stopped at what I could observe.

There is a second, older GPU in the same host. It matters for the protocol, as you will see in a moment, and for nothing else.

The tool and the baseline

llmfit 1.1.3, not the 1.1.6 that was available.

That is a deliberate choice. Two earlier campaigns of mine ran on 1.1.3, and I wanted these 80 results to sit next to those without an asterisk. Before freezing the version I checked what I would be giving up: releases 1.1.4 through 1.1.6 touch the API surface, the MLX provider and the terminal UI. Nothing in the benchmark engine, nothing in the submission schema. So the baseline holds and the comparison stays clean.

Every model ran at the same settings: Q4_K_M quantization, context 4096, three runs, single sequence, and the average tokens per second as the reported figure alongside the minimum and the maximum.

The harness

llmfit measures. It does not manage the server, and it does not warm anything up. Four rules sit in my wrapper, and every one of them was paid for by a run I had to throw away.

Pin the device, then prove it

Both GPUs in this host are exposed to the same Vulkan backend, as Vulkan0 and Vulkan1. Without explicit pinning, a model can quietly land on the wrong card and the measurement means nothing.

Pinning is the easy half: --device Vulkan0. Verifying it is where the trap is. My first instinct was to grep the server log for the adapter name, which produced a false negative and cost me a valid run, because llama.cpp does not print the selected GPU at default verbosity. There is nothing to grep.

So the check is a measurement, not a string match. Read /sys/class/drm/card0/device/mem_info_vram_used before launching and again after the model loads, then require a delta of at least min(gguf_size * 0.5, 1 GiB). Had the weights gone to the other card, the AMD counter would have stayed flat. Two file reads, and the result cannot be faked.

Warm up, then check health again

A cold first run wrecks the average. Before benching, the harness sends one completion of 200 tokens at temperature zero and waits.

Then it re-checks /health, and this is the part that is easy to skip. A model can load into VRAM perfectly, report healthy, and still die during generation once the KV cache grows. Loading successfully is not the same as running successfully, and only a post-warmup health check catches the difference.

Fit by walking down, not by guessing

For a dense model the harness starts at full offload and walks down: -ngl 99, 60, 48, 40, 32, 24, 16, stopping at the first configuration that loads, passes the device check and survives the warmup. For a mixture of experts model, full offload stays and --n-cpu-moe rises instead: 8, 16, 24, 32.

On this card the ladder turned out to be decoration. All 80 models ran at -ngl 99, including every 32B and the 34B and 35B entries. Not one needed partial offload. That single fact killed a plan I had going in, which was to split the results into two pull requests by VRAM tier: on 20 GB, the offload-bound tier does not exist.

The largest entry located the actual ceiling. ornith-1.0-35B committed 19.95 GiB on a card that exposes 19.98, held the warmup, and completed the benchmark. That is the real upper limit at Q4_K_M and context 4096, measured rather than estimated.

One model at a time

The runner is strictly sequential. One GPU means two concurrent runs would fight over VRAM and poison both measurements, so the loop benches one model, tears the server down, sleeps, and moves on.

It also holds a lock file. That guard exists because my automation environment retries a call that times out, which without a lock would start a second campaign on top of the first.

Choosing the models

Downloading 80 large files to discover that six of them do not exist is a bad way to spend an afternoon. A probe script asks the HuggingFace API for each candidate repository, finds the quantized file at the root, reads its size, and reports whether it exists and whether it fits the budget.

Two rules in that probe:

• Q4_K_M wins whenever it is published. It is the quantization used by all my previous campaigns, so results stay comparable. MXFP4 is accepted only where the publisher ships nothing else.
• The size budget is 19.2 GiB, not 19.98. The gap is the KV cache and the compute buffers.

Selection went in two batches of 40, with no overlap. The first covered the ground the table was missing on this hardware. The second deliberately went after 2026 releases, long-standing models that had never been measured here, less common architectures including a hybrid and several mixture of experts designs, and five entries chosen specifically to probe the 20 GB ceiling.

Each batch has its own manifest file, and the manifest is a required argument to the download and run scripts, not a default. An earlier version silently fell back to the first manifest, which re-downloaded 40 models I already had.

What the numbers say

Two findings, one per batch, and they are complementary rather than redundant.

Capacity is a cliff, not a slope

Compared against the same models on a 12 GB card, the average gain is 2.17x. That average is useless, because there are three regimes hiding inside it.

The large models explode. gemma-2-27b goes from 3.50 to 34.19 tokens per second, a 9.77x jump. Qwen2.5-Coder-32B from 4.09 to 31.13. Qwen3-32B from 4.42 to 31.58. These are not speedups in any interesting sense. They are the difference between unusable and comfortable, and the whole of it comes from weights no longer crossing the PCIe bus on every token.

The middle barely moves. Models from 7B to 14B gain between 1.02x and 1.16x. They already fitted on both cards, so there was nothing to win.

The small models regress. Llama-3.2-1B drops from 465.98 to 375.97, a 0.81x. Below 4B nothing was ever offloaded anywhere, so what is being compared is clock speed and backend maturity, and on that ground RADV and Vulkan do not match CUDA. I published the regressions with the rest.

The useful signal is therefore VRAM capacity, not the vendor and not the generation. The moment a model stops fitting, capacity dominates everything else. As long as it fits, capacity buys you nothing.

Once everything fits, active parameters decide

The second batch answers the question the first one leaves open: if capacity is no longer the constraint, what sets the speed?

Not file size. At near-identical file size and identical VRAM:

A 35B mixture of experts runs roughly four times faster than a 34B dense model occupying the same 20 GB. The ordering holds further down the table: gemma-4-26B-A4B reaches 111.84 while gemma-4-12B, dense and less than half the size, reaches 63.35.

The fastest of all 80 is LFM2-8B-A1B at 290.45 tokens per second, an 8B model with roughly 1B parameters active, ahead of every dense 3B measured.

So for a fixed VRAM budget, total parameter count is a poor predictor of throughput and active parameter count is a good one.

This does not contradict what an earlier campaign of mine found, namely that host memory bandwidth is the ceiling for a model that streams from system RAM. It confirms it from the other end. You do not win by making offload faster. You win by removing it.

What this protocol does not measure

Every number above is a throughput number under one specific configuration. Six things it says nothing about, listed because they are the honest limits and because someone will ask.

1. Output quality. Nowhere in this campaign is a model’s answer evaluated. A mixture of experts model at 110 tokens per second is not thereby as good as a dense model at 29. Speed at a fixed memory budget is the only axis measured.
2. Vulkan against ROCm. Everything ran on RADV. I did not build a HIP comparison, so I cannot tell you what these models would do on the same card through ROCm. What I can say is that all 80 ran on the same backend, so the relative ordering holds even if the absolute values would shift.
3. Anything other than Q4_K_M at context 4096. One quantization, one context length, one sequence at a time. No concurrent batching, which is precisely the regime that matters for serving rather than for a single user.
4. Prompt processing. Time to first token is not populated in these submissions. The figure reported is generation throughput only.
5. Long generations. Generation is capped at 300 tokens. That cap produced a visible artifact worth reporting: Qwen3-32B came back with a suspiciously tight spread of 0.04 tokens per second across runs, which a reviewer flagged. The explanation is that it is a reasoning model which fills the token budget every time, so runs two and three both terminated at the cap within a third of a millisecond of each other. An artifact of the cap, not a measurement problem, and better explained than hidden.
6. Sustained thermal behaviour. Each model runs for minutes, not hours. Nothing here tells you what this card does under a sustained production load.

Submitting the results

Three traps stand between a correct measurement and a correct submission, and all three fail silently.

The tool sums the GPUs of the host. On a two-card machine llmfit writes gpuCount: 2, a vramGb of 27.98 which is the sum of both cards, a memory tier of 24, and a generic PCI name covering four different products. Submitting that declares hardware that does not exist. The hardware block has to be rewritten by hand to describe the single pinned card, and the rewrite has to be declared in the pull request. Silently correcting it would be worse than the bug.

The model field carries the full path passed to the server, not the file name the community schema expects. A basename is mandatory. Useful side effect: it also deduplicates a model benched twice, once from the production directory and once through a symlink.

The pending store lives in the home directory, not on the scratch volume that gets wiped between campaigns. Ten result files from a campaign a week earlier were still sitting there, ready to be folded into this submission. The build step filters on modification time, on the hardware name, and on the batch manifest.

The assembled file is then validated with the project’s own validation script before anything is pushed. Both files passed on the first attempt.

The first 40 models are in the community table. The second 40 are measured, validated and queued behind them, because I want the two batches to land as two visible contributions rather than one large diff.

Why bother

None of this needed permission or a budget. A submission is a few kilobytes of JSON. But it is verifiable, it puts a card and a backend on the map that were not there before, and the next person choosing between a 30B mixture of experts and a 32B dense model for a 20 GB card now has measured numbers instead of a forum thread.

The protocol is the contribution as much as the results are. A number without a method is an opinion with decimals.

The tech moves fast. The fundamentals don’t.

Akciali