A new GPU is a bad deal for a reader whose real question is which card gets the most VRAM per dollar for running open-weight models at home. The used market is where that question has a sensible answer. The cards on it are two generations old; the constraint is no longer price. It is whether current software still runs on the card.
Start with the Software Floor, Not the Card
The first filter is the runtime, not the model. Ollama’s GPU documentation is explicit: “Ollama supports Nvidia GPUs with compute capability 5.0+ and driver version 550 and newer.” It then narrows the lower end: “Nvidia GPUs with compute capability 5.0 through 6.2 require driver version 570 or newer.” That second line catches readers out: a card between 5.0 and 6.2 clears Ollama’s floor but pulls in a stricter driver requirement the host OS has to deliver. Compute capability is in NVIDIA’s CUDA GPUs table; every GeForce RTX 30, 40 and 50 card is well above the floor. The check is whether the driver branch on the host machine sits at 570 or higher, which the used-GPU checklist covers.
llama.cpp’s build documentation does not publish a fixed compute capability minimum. It documents the backends - CUDA, HIP (AMD), SYCL (Intel), Vulkan, Metal - and the build flags to enable each. What falls out is which card is fastest, not which card works. For NVIDIA the path is cmake -B build -DGGML_CUDA=ON; the CUDA backend supports every GeForce RTX card on the used market.
VRAM Is the Real Decision
Two RTX cards from the same generation with very different used prices often differ by one figure: how much VRAM they carry. That number, not the model name and not the clock speed, decides which open-weight models will load. NVIDIA’s GeForce compare page resolves a card name to its VRAM. Model sizes in the table below are from the Ollama library tag pages, which publish each tag’s default quantization.
| VRAM tier | What fits comfortably | Representative model and download size |
|---|---|---|
| 6 GB | Small chat, code completion on tiny contexts | qwen3:1.7b at 1.4 GB |
| 8 GB | A usable 8B model, short contexts | qwen3:8b at 5.2 GB |
| 12 GB | A 13B-class model, or 8B with long context | qwen3:8b at 5.2 GB with headroom |
| 16 GB | A 14B-30B dense model, or 30B-A3B at default | qwen3:30b-a3b at 19 GB partial offload |
| 24 GB | A 30B-32B dense model in full | qwen3:32b at 20 GB |
The 12 GB tier is the workhorse of the used market: the smallest tier that holds a useful model with context to spare, and the largest tier several mainstream cards reach. The 24 GB tier is where a used card stops being a budget and starts being a serious workstation card, which is why the RTX 3090, 3090 Ti and 4090 command the premiums they do.
Cards Worth Targeting by Tier
VRAM and compute capability below are from NVIDIA’s GeForce compare and CUDA GPUs pages. Where a card ships in multiple variants, both are listed; the used-GPU checklist covers reading the model number.
| Card | VRAM | Compute capability | Note |
|---|---|---|---|
| RTX 3050 (6 GB) | 6 GB GDDR6 | 8.6 | Entry tier; long contexts queue |
| RTX 3060 (12 GB) | 12 GB GDDR6 | 8.6 | The used sweet spot at 12 GB |
| RTX 3060 (8 GB) | 8 GB GDDR6 | 8.6 | Same name, half the VRAM - verify which |
| RTX 3060 Ti | 8 GB GDDR6 / GDDR6X | 8.6 | Faster than the 3060 at the same VRAM |
| RTX 3070 | 8 GB GDDR6 | 8.6 | Speed upgrade, not a VRAM upgrade, over the 3060 Ti |
| RTX 3070 Ti | 8 GB GDDR6X | 8.6 | Same VRAM as the 3070, slightly faster |
| RTX 3080 (10 GB) | 10 GB GDDR6X | 8.6 | The original 3080; verify not the 12 GB variant below |
| RTX 3080 (12 GB) | 12 GB GDDR6X | 8.6 | The 12 GB variant to seek out |
| RTX 3080 Ti | 12 GB GDDR6X | 8.6 | Slightly faster than the 3080 12 GB, same VRAM |
| RTX 3090 | 24 GB GDDR6X | 8.6 | The cheapest path to 24 GB used |
| RTX 3090 Ti | 24 GB GDDR6X | 8.6 | Same VRAM as the 3090, premium price for marginal speed |
| RTX 4060 | 8 GB GDDR6 | 8.9 | Ada; newer drivers, lower power |
| RTX 4060 Ti (16 GB) | 16 GB GDDR6 | 8.9 | 16 GB variant opens the 16 GB tier |
| RTX 4060 Ti (8 GB) | 8 GB GDDR6 | 8.9 | The 8 GB is the common listing; verify |
| RTX 4070 | 12 GB GDDR6X | 8.9 | Ada 12 GB, balanced |
| RTX 4070 SUPER | 12 GB GDDR6X | 8.9 | Faster than the 4070 at the same VRAM |
| RTX 4070 Ti | 12 GB GDDR6X | 8.9 | Same VRAM as the 4070 SUPER, marginally faster |
| RTX 4070 Ti SUPER | 16 GB GDDR6X | 8.9 | The first 16 GB Ada card |
| RTX 4080 | 16 GB GDDR6X | 8.9 | Faster than the 4070 Ti SUPER at the same VRAM |
| RTX 4080 SUPER | 16 GB GDDR6X | 8.9 | Marginal speed gain over the 4080 |
| RTX 4090 | 24 GB GDDR6X | 8.9 | The fastest used Ada card; 24 GB |
| RTX 5050 | 8 GB GDDR6 | 12.0 | Blackwell entry; driver support newer |
| RTX 5060 | 8 GB GDDR7 | 12.0 | Blackwell 8 GB; same VRAM caveat as the 4060 |
| RTX 5060 Ti (16 GB) | 16 GB GDDR7 | 12.0 | The Blackwell 16 GB card |
| RTX 5060 Ti (8 GB) | 8 GB GDDR7 | 12.0 | Same name, half the VRAM - verify which |
| RTX 5070 | 12 GB GDDR7 | 12.0 | Blackwell 12 GB; used supply thin |
| RTX 5070 Ti | 16 GB GDDR7 | 12.0 | Blackwell 16 GB midrange |
| RTX 5080 | 16 GB GDDR7 | 12.0 | Blackwell high-end 16 GB |
| RTX 5090 | 32 GB GDDR7 | 12.0 | The only 32 GB consumer card |
The Three Traps on the Used Market
Two are variant confusion, one is generation confusion.
The RTX 3060 ships in 12 GB and 8 GB variants under the same model name. The 12 GB opens the 12 GB VRAM tier; the 8 GB sits at the same tier as a 3060 Ti. Listings sometimes fail to mention which is being sold, and the price gap is large.
The RTX 4060 Ti and 5060 Ti repeat the trick, both in 8 GB and 16 GB variants. The 8 GB is the more common listing because most retailers stocked it, and it leaves the card at the same VRAM tier as the cheaper 4060.
The third trap is a card older than the floor. Anything below compute capability 5.0 - the GTX 780, 690, 580 era - sits below Ollama’s floor and below every current CUDA toolkit. Those cards still appear on the used market because they were excellent in their day, but they are not local-AI cards by any current definition. The names look similar; the architecture does not.
When the Used Card Is the Wrong Answer
The cost crossover page covers when running locally beats paying per token; for low-volume use, an API is cheaper than any hardware purchase. A used GPU is the same trade at a smaller scale: right for a reader who will run a model often enough to amortise the card, wrong for one who will use it a handful of times. For a reader with no PCIe slot (laptop, thin client), the local AI without a GPU guide covers what runs on system RAM alone.
Bottom Line
VRAM, then generation, then variant. The 12 GB tier has the deepest used supply (RTX 3060 12 GB, RTX 3080 12 GB, RTX 4070 and 4070 SUPER) and is the smallest tier that runs the models people actually want. The 24 GB tier is the RTX 3090, 3090 Ti and 4090, the cheapest path to running dense 30B models at full context; the price premium reflects that. The 16 GB tier is the Ada and Blackwell midrange (4070 Ti SUPER, 4080, 5060 Ti 16 GB, 5070 Ti), where new and used prices meet.
Verify before paying: which variant of a same-name card is being sold, whether the generation sits below the runtime floor, and whether the host driver branch clears Ollama’s 570 requirement. None are on the box; all are on the linked sources above.