Local AI Model Hardware Requirements: 2026 Guide

Local AI model hardware requirements are primarily a memory-capacity problem, and the expensive surprise is that the model file is only the entry ticket. I size a local system by adding quantised weights, the key-value cache for the intended context window, runtime buffers, operating-system use, and enough free headroom to prevent swapping or out-of-memory failures. That method is less glamorous than comparing TOPS figures, but it is far more reliable. A graphics card can advertise formidable AI throughput and still be the wrong purchase when its VRAM cannot hold the model and context you actually plan to use.

The practical answer in 2026 is tiered. A modern computer with 16 GB of RAM can run compact 3B to 8B models and some 12B to 14B models at conservative context lengths. A 12 GB to 16 GB GPU makes 7B to 14B models much faster and can stretch into selected larger quantisations. A 24 GB to 32 GB GPU is the serious desktop tier for 20B to 32B models, coding assistants, multimodal work, and longer contexts. Models around 70B normally call for roughly 48 GB to 64 GB of usable accelerator or unified memory at four-bit precision, or a slower CPU and GPU split.

This guide explains the arithmetic, the trade-offs between NVIDIA, Apple Silicon, and AMD unified-memory systems, and the constraints that product pages often bury. It also separates prompt-processing speed from generation speed, shows why context length changes the answer, and provides a reproducible workflow for matching hardware to a model rather than buying hardware first and hoping the model fits.

What Actually Determines Local AI Performance

Four resources shape local inference: memory capacity, memory bandwidth, compute throughput, and software support. Capacity decides whether the workload can run without aggressive offloading. Bandwidth often determines token-generation speed because each generated token repeatedly streams model weights through the memory system. Compute matters more for prompt ingestion, vision encoders, speculative decoding, and highly optimised lower-precision kernels. Software support decides whether the advertised silicon is used at all.

This is why an NPU headline is not a substitute for an LLM benchmark. Many desktop runtimes still execute general open models through CUDA, Metal, Vulkan, ROCm, or CPU vector instructions. An NPU may accelerate a vendor-approved model, transcription pipeline, or operating-system feature while contributing little to an arbitrary GGUF model. AMD’s 2026 Ryzen AI 400 desktop processors, for example, advertise up to 50 TOPS of NPU compute, but model compatibility and runtime routing still determine whether a local application uses that block (AMD, 2026).

The first useful calculation is model fit. Parameter count explains weight scale, while the model weights guide shows why weights are not the complete application. Tokeniser data, multimodal projectors, context cache, buffers, and concurrent requests add memory. A system that barely loads a model can fail when the prompt grows or a second request starts.

A second distinction is latency versus throughput. Prompt processing measures how quickly the system reads an existing prompt. Token generation measures how quickly it produces new text. These phases stress hardware differently, so a single tokens-per-second figure can conceal the bottleneck. The practical benchmark should record time to first token, prompt tokens per second, generation tokens per second, memory use at target context, power draw, and whether the model remains fully accelerated.

Local AI Model Hardware Requirements by Model Size

The table below is a planning guide for dense text models using roughly four-bit GGUF quantisation. It is not a promise that every architecture will fit. Mixture-of-experts models, vision encoders, unusually large vocabularies, and long contexts can change the requirement. The recommended memory column includes a working margin rather than matching the file size exactly.

Model Class	Typical Q4 Weight File	Practical Usable Memory	Sensible Hardware Tier	Best-Fit Workloads
1B to 3B	0.8 to 2.5 GB	4 to 8 GB	8 to 16 GB system RAM; GPU optional	Classification, extraction, short chat, mobile or embedded use
7B to 8B	4 to 5.5 GB	8 to 12 GB	8 GB VRAM or 16 GB unified/system RAM	General chat, summarisation, light coding, retrieval
12B to 14B	7 to 10 GB	12 to 18 GB	12 to 16 GB VRAM or 24 GB unified RAM	Stronger writing, coding, tool use, moderate context
20B to 32B	12 to 20 GB	20 to 30 GB	24 to 32 GB VRAM or 32 to 64 GB unified RAM	Serious coding, agents, multimodal work, larger retrieval
65B to 70B	40 to 48 GB	48 to 64 GB	48 GB multi-GPU, 64 GB unified memory, or CPU-GPU split	Higher-quality reasoning, private enterprise assistants
100B to 120B	60 to 80 GB	80 to 110 GB	96 to 128 GB unified memory or carefully supported multi-GPU	Large agents, long context, advanced local research

Planning estimates based on documented llama.cpp Q4_K_M sizes and an added operational reserve. Actual memory changes with architecture, context length, batch size, and backend.

The most trustworthy published anchor is llama.cpp’s own quantisation documentation. It lists Llama 3.1 8B at 4.9 GB and Llama 3.1 70B at 43.1 GB in Q4_K_M form, compared with 32.1 GB and 280.9 GB for the source checkpoints used in its conversion example (ggml-org, 2026). Those are storage and load figures, not complete end-to-end budgets.

For a quick estimate, multiply parameters by effective bits per weight, divide by eight, then add metadata and runtime overhead. Four-bit formats are rarely exactly 4.0 bits per parameter because block scales and mixed tensor types add cost. llama.cpp reports Q4_K_M for its 8B example at about 4.89 bits per weight, which is why the result is closer to 4.6 GiB than a simplistic 4.0 GB calculation.

The safe buying rule is to avoid treating the minimum as the target. Reserve at least 25 per cent above the observed load for ordinary chat, and 35 per cent or more for long context, vision input, parallel requests, or agent loops. The headroom also protects performance because an operating system that begins compressing or swapping memory can turn a technically successful load into an unusable experience.

A Fast Sizing Formula

Estimated requirement = weight file + KV cache + runtime buffers + multimodal components + operating-system reserve + concurrency reserve.

For a single-user desktop, the weight file is usually the largest fixed component. The KV cache becomes decisive as context grows. Runtime buffers vary by backend and may spike during prompt evaluation. Vision or audio models can require a separate projector or encoder. Concurrent model sessions can duplicate caches even when weights are shared. This is why the same model may fit at 8K context but fail at 64K.

Why Quantisation Changes the Buying Decision

Quantisation stores weights at lower precision so the model occupies less memory and can move through memory faster. The trade-off is not simply quality versus size. Kernel support, calibration method, model architecture, and target hardware all affect whether a lower-bit model is faster and whether its output quality remains acceptable. The AWQ research, for example, protects a small set of salient weight channels and reported strong on-device results without relying on mixed-precision execution that is awkward for hardware (Lin et al., 2024).

For general desktop use, Q4_K_M remains a sensible starting point in llama.cpp-compatible ecosystems because it offers a strong balance of size, speed, and quality. Q5_K_M is useful when an extra 15 to 20 per cent memory is available and accuracy matters more than maximum capacity. Q8_0 is closer to high precision but roughly doubles the memory of Q4. Very low-bit formats can make an otherwise impossible model fit, yet the larger model is not automatically better if quantisation damage harms the exact tasks you care about.

The site’s TurboQuant memory compression research is relevant because inference memory is no longer only about static weights. Compression of the KV cache and other runtime state can materially change long-context economics. Such techniques should not be counted as guaranteed savings until the chosen runtime, model, and backend support them, but they explain why hardware requirements can improve without parameter counts shrinking.

Format	8B Example Size	Relative Memory	Practical Use	Main Constraint
F16	14.96 GiB	3.27x Q4_K_M	Reference quality and supported high-memory systems	Too large for many consumer GPUs
Q8_0	7.95 GiB	1.74x	Quality-sensitive local inference	Often little capacity advantage over smaller high-quality models
Q6_K	6.14 GiB	1.34x	High-quality desktop use	Moderate memory premium
Q5_K_M	5.33 GiB	1.16x	Balanced quality with spare VRAM	Can push a model over a hardware boundary
Q4_K_M	4.58 GiB	1.00x	Default planning choice	Still needs cache and runtime headroom
Q3_K_M	3.74 GiB	0.82x	Capacity-constrained systems	Greater risk of task-specific quality loss

Sizes are llama.cpp figures for its Llama 3.1 8B example. Throughput results are hardware-specific and should not be transferred between systems.

VRAM, Unified Memory, and System RAM Are Not Interchangeable

Dedicated VRAM is physically close to the GPU and usually provides the best-supported path for high-throughput local inference on Windows and Linux. If the entire model, cache, and buffers fit, CUDA or another mature GPU backend can deliver strong prompt processing and generation. The hard boundary is capacity: a 16 GB GPU remains a 16 GB GPU even when the computer has 64 GB of system RAM. Hybrid offload can use both, but the traffic across PCI Express and the slower CPU-side layers reduce speed.

Unified memory allows the CPU and GPU to address one physical pool. Apple Silicon and AMD Ryzen AI Max systems therefore avoid a rigid split between system RAM and VRAM, which makes large-model fit easier. The benefit is capacity flexibility, not infinite speed. The operating system, applications, display buffers, and model all share the same pool. A 64 GB Mac cannot safely dedicate all 64 GB to weights, and a 128 GB Ryzen AI Max system still needs an explicit graphics-memory configuration on Windows for some workloads.

AMD states that a 128 GB Ryzen AI Max+ 395 system can expose 96 GB as Variable Graphics Memory. Framework explains the accompanying trade-off plainly: the 256-bit LPDDR5X interface requires soldered memory, so the capacity cannot be upgraded later. That makes the initial configuration decision unusually important even though the platform offers a large model-friendly pool.

System RAM is the most flexible and cheapest capacity, but CPU-only inference is generally bandwidth-bound. A desktop with 128 GB of ordinary dual-channel memory may load a 70B model yet generate text far more slowly than a high-bandwidth unified-memory system or multi-GPU workstation. Capacity answers “can it run?” Bandwidth and backend efficiency answer “is it pleasant to use?”

CPU-Only and GPU-Accelerated Inference

CPU-only inference is viable for compact models, low request rates, background extraction, and users who value zero incremental hardware cost. llama.cpp supports x86 vector extensions and Apple Silicon CPU paths, and it can run models that exceed GPU memory. A recent CPU with AVX2 or better, fast DDR5, and enough cores can provide workable 3B to 8B performance. LM Studio requires AVX2 on x64 Windows and recommends at least 16 GB of RAM, which is a useful baseline rather than a guarantee for larger models (LM Studio, 2026).

The limit is memory bandwidth. Token generation repeatedly reads much of the model, so adding CPU cores does not scale indefinitely when the memory channels are saturated. Server platforms with many memory channels can perform well, but a mainstream desktop often reaches its bandwidth ceiling long before all cores are fully productive. Prompt processing may still benefit from additional compute, which is another reason to report both phases separately.

GPU acceleration is the default for interactive use because dedicated accelerators combine much higher bandwidth with specialised matrix hardware. NVIDIA remains the broadest compatibility choice for Windows and Linux due to CUDA support across llama.cpp, Ollama, LM Studio, PyTorch, TensorRT-LLM, image generation, speech, and many agent tools. AMD support has improved through ROCm and Vulkan, but model and application coverage should be checked before purchase. Intel Arc and integrated GPUs can be useful through Vulkan, SYCL, or OpenVINO, though the experience is more dependent on the exact runtime.

Hybrid CPU-GPU offload is a capacity escape hatch. It can make a 70B model run on a 24 GB or 32 GB GPU by placing selected layers in system RAM, but speed varies with the offload split, PCI Express traffic, and CPU memory bandwidth. It is better to regard hybrid operation as a deliberate compromise than as equivalent to a full-memory GPU configuration.

Windows and Linux GPU Options in 2026

For current discrete GPUs, capacity tiers matter more than gaming labels. NVIDIA lists the GeForce RTX 5070 with 12 GB of GDDR7 starting at $549, the RTX 5070 Ti with 16 GB at $749, the RTX 5080 with 16 GB at $999, and the RTX 5090 with 32 GB at $1,999. Board-partner prices and availability can differ, so these are official starting prices rather than guaranteed checkout prices (NVIDIA, 2025).

Twelve gigabytes is an effective 7B to 14B tier with moderate context. Sixteen gigabytes gives more breathing room but does not transform a 32B model into a comfortable full-GPU workload. Thirty-two gigabytes is the strongest single consumer-GPU capacity currently shipping in NVIDIA’s established GeForce range, yet even that is below llama.cpp’s 43.1 GB Q4_K_M file for a dense 70B example. A 5090 can still run such a model through lower quantisation or partial offload, but the capacity limit remains real.

NVIDIA’s May 2026 RTX Spark announcement points towards a different category: up to 128 GB of unified memory, local 120B models, and one-million-token agent contexts. Jensen Huang, NVIDIA founder and CEO, said, “The PC is being reinvented.” Georgi Gerganov, founder of llama.cpp, added that the systems “multiply the amount of context processing”. Satya Nadella, Microsoft chairman and CEO, said the goal was “unmetered intelligence to every home and every desk with Windows”. RTX Spark systems were announced for autumn 2026, and pricing was not publicly confirmed as of 6 August 2026, so they remain an upcoming option.

Linux often exposes advanced backends and multi-GPU configurations earlier, while Windows offers the easiest mainstream application experience. The right operating system is the one supported by the complete stack: GPU driver, inference runtime, quantisation format, model architecture, agent framework, and any speech or image components. A theoretically faster card is poor value when the intended application falls back to CPU execution.

Apple Silicon and High-Capacity Unified Memory

Apple Silicon is attractive for local inference because Metal acceleration and unified memory let models use a large shared pool without crossing a discrete PCI Express link. The platform is especially coherent for developers using llama.cpp, MLX, LM Studio, or Ollama. The trade-offs are fixed memory at purchase, limited hardware upgrade paths, and application compatibility that remains stronger in some CUDA-first research ecosystems.

Apple’s 2026 M5 Pro supports up to 64 GB of unified memory and 307 GB/s bandwidth, while M5 Max supports up to 128 GB and 614 GB/s. Johny Srouji, Apple’s senior vice president of Hardware Technologies, described the result as “performance, efficiency, and incredible on-device AI capabilities”. The 14-inch M5 Pro MacBook Pro starts at $2,199, while the 14-inch M5 Max starts at $3,599 before memory upgrades (Apple, 2026a; Apple, 2026b).

Mac Studio covers the highest-capacity end. The 2025 M3 Ultra generation starts with 96 GB of unified memory, can be configured up to 512 GB, and Apple says it can hold LLMs with more than 600 billion parameters entirely in memory. That statement addresses capacity, not necessarily interactive speed for every architecture or quantisation. The system starts at $1,999 for the Mac Studio range, while high-memory M3 Ultra configurations cost substantially more and should be priced directly at purchase (Apple, 2025).

The site’s Apple on-device intelligence strategy provides the wider platform context. For hardware selection, the practical rule is to subtract operating-system and application needs before calculating model fit. A 64 GB machine is a strong 32B-class system, not a guaranteed 64 GB model appliance. A 128 GB M5 Max configuration can enter 70B and selected 100B-class territory, but context and concurrent agents still consume the remaining pool.

AMD Ryzen AI Max and Large Shared-Memory PCs

Ryzen AI Max offers a Windows and Linux route to high-capacity unified memory. The flagship Ryzen AI Max+ 395 combines a 40-compute-unit Radeon GPU with up to 128 GB of LPDDR5X on a 256-bit interface. AMD documents up to 96 GB of Variable Graphics Memory on a 128 GB system, which is enough to fit many 70B quantisations and selected larger mixture-of-experts workloads without a discrete professional GPU.

Framework’s 4.5-litre Desktop made the value proposition unusually concrete. Its 128 GB configuration launched at $1,999, while the 32 GB base configuration started at $1,099. Framework states that the 128 GB version can run Llama 3.3 70B Q6 at conversational speed. Those are vendor results and should be reproduced with the intended model revision, context, and runtime, but the capacity-per-dollar comparison is meaningful. The main hidden limit is permanent memory: the same wide bus that provides 256 GB/s bandwidth requires soldered LPDDR5X.

AMD’s 2026 desktop messaging also shows why the NPU should not dominate the decision. Jack Huynh, AMD senior vice president and general manager of Computing and Graphics, said, “The desktop PC is evolving from a tool you use to an intelligent assistant.” Large GGUF models will usually care more about the Radeon GPU path and shared-memory capacity than the NPU’s headline TOPS.

The AMD route is strongest for users who need far more memory than a consumer discrete GPU provides, prefer Windows or Linux, and accept that some applications remain better optimised for CUDA. Before buying, confirm support in the precise version of Ollama, LM Studio, llama.cpp, image generator, or agent framework you plan to use. Vulkan support has broadened rapidly, including Ollama’s 2026 decision to enable it by default for wider AMD and Intel acceleration, but broad support does not guarantee equal performance.

Context Length and the Hidden Memory Tax

The KV cache stores attention state for previous tokens so the model does not recompute the full conversation at every step. Its size grows with context length, layer count, attention dimensions, batch size, and cache precision. A model that fits easily at 8K tokens can cross the memory limit at 64K or 128K. This is the most common reason published “minimum VRAM” figures fail in real use.

Ollama’s September 2025 scheduling update provides a useful measured example. Gemma 3 12B at 128K context used 21.4 GiB of VRAM on a single RTX 4090 after all layers were placed on the GPU. The weight file alone would suggest a much smaller requirement. Ollama now measures exact model memory rather than relying only on estimates, which reduces crashes and improves GPU utilisation, but users must still configure context deliberately.

Coding tools amplify this tax because repositories, tool results, terminal output, and repeated agent plans accumulate quickly. Ollama’s January 2026 launch guidance says its GLM 4.7 Flash coding setup needs about 23 GB of VRAM at 64K context. That number is a workload-specific example, not a universal requirement, but it shows why a 24 GB card can be fully occupied by a model that appears modest when judged only by parameter count.

The AI latency explainer is relevant here because long prompts affect time to first token as well as memory. Context is not free simply because a model advertises a large window. For document chat, retrieval should keep the prompt selective. For coding, repository maps, cache reuse, and summarised tool output can preserve useful context without carrying every token. For agents, separate working memory from archival memory rather than forcing all history into the active window.

Three Context Rules That Prevent Overspending

• Benchmark the context you will actually use, not the model card maximum.
• Treat parallel users or agents as separate cache budgets unless the runtime proves cache sharing.
• Use retrieval, summarisation, and cache quantisation before buying hardware solely for an extreme context window.

Local Runtimes, Features, and API Compatibility

Hardware value depends on the runtime that can use it. llama.cpp is the low-level portability anchor for GGUF models, supporting CPU execution, Metal, CUDA, HIP, Vulkan, SYCL, and CPU-GPU hybrid inference. Ollama wraps local models in a simple command-line, desktop, and API workflow. LM Studio adds model discovery, a graphical interface, local serving, and compatibility endpoints for existing developer tools. Apple’s MLX is important for native Apple Silicon workflows, while specialised frameworks such as TensorRT-LLM or vLLM may deliver higher throughput on supported NVIDIA systems at the cost of more setup.

Ollama’s local runtime is free and local hardware use is unlimited. Its optional cloud plans are separate: Free allows one concurrent cloud model, Pro costs $20 per month or $200 per year with three concurrent models and 50 times Free usage, and Max costs $100 per month with ten concurrent models and five times Pro usage, although new Max subscriptions were paused on 6 August 2026. Session limits reset every five hours and weekly limits every seven days; exact token limits vary by model rather than using one published token cap (Ollama, 2026).

LM Studio is free for local use at home and work. Its 2026 Free tier includes the local Bionic agent, llama.cpp and MLX model execution, offline voice transcription, and LM Link for up to five devices. Optional cloud inference is pay-as-you-go, while Bionic Pass pricing had not yet been published. The application recommends 16 GB RAM on Mac and Windows, 4 GB dedicated VRAM on Windows, Apple Silicon on macOS, and AVX2 for x64 Windows.

Integration surfaces matter for migration. LM Studio exposes native REST, OpenAI-compatible Responses, Chat Completions, Completions, and Embeddings endpoints, plus an Anthropic-compatible Messages endpoint for tools such as Claude Code. Ollama supports REST, Python and JavaScript libraries, tool calling, parallel tools, agent loops, vision, embeddings, structured outputs, and OpenAI-compatible integrations. Model support still varies, so a feature in the runtime does not guarantee the selected model has been trained to use it reliably.

Runtime	Local Cost	Key Backends	Developer Interfaces	Important Limits
llama.cpp	Free, open source	CPU, CUDA, Metal, HIP, Vulkan, SYCL, RPC	CLI, server, OpenAI-style server options	Manual model and parameter management; backend results vary
Ollama	$0 local; optional cloud plans	CUDA, Metal, Vulkan and supported CPU paths	REST, Python, JavaScript, tool calling, structured outputs	Cloud concurrency and rolling limits; local hardware remains unlimited
LM Studio	$0 local; optional token-priced cloud	llama.cpp, MLX, supported GPU backends	Native REST, OpenAI, Anthropic, Python, JavaScript, CLI	Mac requires Apple Silicon; 16 GB RAM recommended; LM Link Free limit is five devices
MLX	Free, open source	Apple Silicon unified-memory execution	Python and model-specific tooling	Apple-only and less portable than GGUF workflows
TensorRT-LLM / vLLM	Free software; deployment costs vary	Primarily NVIDIA accelerator stacks	Python, servers, OpenAI-compatible deployments	More setup; quantisation and model support are version-specific

Pricing and limits checked on 6 August 2026. Cloud services are optional and do not change the hardware requirements of local inference.

Hardware Buying Matrix for Real Budgets

The matrix below uses official starting prices where a vendor publishes them. It is not a retail deal list. Memory upgrades, storage, power supplies, partner-board premiums, tax, and regional pricing can materially change the final cost.

Configuration	Official Starting Price	Memory Available to AI	Comfortable Model Tier	Hidden Limit or Trade-Off
Existing 16 GB computer	$0 incremental	Usually 8 to 12 GB after OS	1B to 8B Q4	CPU speed and swapping can dominate; modest context only
GeForce RTX 5070	$549	12 GB VRAM	7B to 14B Q4	12 GB ceiling; system RAM does not become VRAM
GeForce RTX 5070 Ti	$749	16 GB VRAM	7B to 14B, selected 20B	Same capacity as RTX 5080 despite lower compute
GeForce RTX 5080	$999	16 GB VRAM	Fast 7B to 14B, selected 20B	Compute improves, model-fit ceiling remains 16 GB
GeForce RTX 5090	$1,999	32 GB VRAM	20B to 32B; partial 70B	Dense 70B Q4 normally exceeds VRAM
Framework Desktop 128 GB	$1,999 launch price	Up to 96 GB graphics allocation on Windows	70B and selected 100B-class models	Soldered memory; backend support can trail CUDA
14-inch MacBook Pro M5 Pro	$2,199	Up to 64 GB unified memory	32B and selected 70B quants	Shared pool and fixed memory; upgrades raise price
14-inch MacBook Pro M5 Max	$3,599	Up to 128 GB unified memory	70B and selected 100B-class models	High purchase cost; fixed configuration
Mac Studio M3 Ultra	$1,999 range starting price	96 GB to 512 GB unified memory on M3 Ultra configurations	Very large local models	High-memory configurations cost far above base; architecture-specific speed varies
RTX Spark systems	Not confirmed as of 6 Aug 2026	Up to 128 GB unified memory announced	Up to 120B announced	Autumn 2026 availability; pricing and independent benchmarks pending

Sources: NVIDIA, Apple, AMD, Framework, and vendor pricing pages. “Comfortable” assumes sensible context and headroom, not the maximum technically loadable model.

The counterintuitive comparison is the RTX 5070 Ti versus RTX 5080. Both list 16 GB, so the 5080 can be faster without fitting a larger fully accelerated model. Paying for compute is rational when the workload already fits, especially for prompt ingestion, image generation, or high request throughput. Paying for capacity is rational when model size or context is the binding constraint.

For a private document workflow, a 14B or 32B model with retrieval can outperform a larger model fed an undisciplined context. The personal AI research assistant shows why storage, embeddings, indexing, and reranking can matter as much as generation hardware. For local image work, the Stable Diffusion hardware guide covers a different memory profile where resolution, ControlNet, and upscaling can change the GPU requirement.

A Step-by-Step Hardware Sizing Workflow

1. Define the workload before the model: Write down the tasks, acceptable response time, privacy boundary, number of simultaneous users, image or audio inputs, and maximum realistic context. A personal chat tool and a repository-scale coding agent have different hardware needs even when they use the same base model.
2. Choose two model sizes, not one: Select a preferred model and a smaller fallback. This prevents the purchase from depending on one benchmark or model release. The fallback also helps diagnose whether a problem comes from capacity, runtime compatibility, or the application layer.
3. Select the quantisation and exact file: Use the actual GGUF or model package size, not only the parameter count. Include multimodal projector files where relevant. Prefer a well-tested Q4 or Q5 build before assuming an extreme low-bit version will preserve quality.
4. Set the target context and concurrency: Estimate the active prompt size after retrieval and tool output. Decide whether one or several requests will run at once. Agent loops and local API servers can create multiple caches, so single-chat memory is not enough for a shared service.
5. Add operational headroom: Add at least 25 per cent for normal use and 35 per cent or more for long context, vision, or concurrency. On unified-memory systems, subtract the operating system and active applications first. On discrete GPUs, reserve display and runtime memory.
6. Confirm backend support: Check the exact runtime version, operating system, GPU driver, model architecture, quantisation, and API feature. Verify that vision, tool calling, embeddings, and structured output are supported by both runtime and model.
7. Run a representative benchmark: Measure cold load time, time to first token, prompt tokens per second, generation tokens per second, peak memory, power, and thermal behaviour. Use your documents, code, or images rather than relying only on short synthetic prompts.
8. Validate failure behavior: Increase context, start a second request, and test recovery after cancellation. A system is ready when it degrades predictably rather than crashing, swapping heavily, or silently moving most layers to a slower backend.

During model selection, the Gemma open-model analysis is a useful reminder that architecture and efficiency can shift the capability-per-gigabyte calculation. A newer, smaller model may be a better hardware match than an older model with more parameters. Capacity planning should therefore be repeated when the model changes, even if the task stays the same.

Performance Bottlenecks to Test

• Memory bandwidth: Generation remains slow even when compute utilisation appears low.
• Prompt ingestion: Large documents create a long wait before the first token.
• Thermal limits: Thin laptops lose sustained speed after several minutes.
• Storage and model loading: Large files on slow SSDs delay startup and model switching.
• PCI Express transfer: Partial offload performs much worse than full GPU residency.
• Backend fallback: An unsupported layer or feature unexpectedly runs on the CPU.
• Concurrency: A second agent duplicates cache memory and causes an out-of-memory failure.
• Power and acoustics: A fast benchmark configuration is unsuitable for an always-on office system.

Always-on agents introduce a separate operational requirement: idle power, sleep behaviour, remote access, permissions, and auditability. The site’s always-on Mac AI agent illustrates why a compact machine can be chosen for persistence rather than maximum tokens per second. For such systems, reliability and controlled access can outweigh benchmark leadership.

Our Research Methodology

I built the sizing ranges from primary documentation and reproducible arithmetic rather than retailer recommendations. The fixed weight-size anchors come from the llama.cpp quantisation documentation, including its Llama 3.1 8B and 70B Q4_K_M examples. Context overhead is grounded in Ollama’s measured Gemma 3 12B result at 128K context and its 2026 coding-model guidance. Software requirements and integration claims were checked against current LM Studio and Ollama documentation.

Hardware capacities, starting prices, and availability statements were verified against NVIDIA, Apple, AMD, and Framework sources available on 6 August 2026. Announced but not yet shipping products, including RTX Spark systems, are labelled as upcoming and are not treated as purchasable at a confirmed price. Vendor performance claims are identified as vendor claims and are not converted into cross-platform benchmark rankings.

The recommended memory ranges add operational headroom to documented model sizes and account for context cache, runtime buffers, multimodal components, operating-system use, and concurrency. Because these elements vary by architecture and runtime, the ranges are planning guidance rather than guaranteed minimum specifications. The article deliberately separates model fit, prompt-processing speed, and token-generation speed so one metric does not stand in for the complete user experience.

This article was researched and drafted with AI assistance and reviewed by the Sami Ullah Khan editorial desk at Perplexity AI Magazine. All data, citations, pricing figures, and named quotes have been independently verified against primary sources before publication.

Conclusion

The right local AI computer is the smallest system that runs the intended model, context, and concurrency with stable headroom. That usually means 16 GB of memory for compact experimentation, 12 GB to 16 GB of VRAM for serious 7B to 14B use, 24 GB to 32 GB for 20B to 32B models, and roughly 64 GB or more of usable accelerator or unified memory for comfortable 70B-class work. These are workload tiers, not universal minimums.

The strongest purchasing insight is that memory capacity and memory bandwidth solve different problems. Capacity decides whether the workload fits. Bandwidth, compute, and backend maturity decide how quickly it responds. Unified-memory systems offer unusually large pools, while discrete NVIDIA GPUs retain the broadest software path. CPU-only inference remains valuable when cost, flexibility, or background processing matters more than interactive speed.

The market is also moving towards personal-agent computers with larger shared memory, but announced capabilities should not be mistaken for verified value before pricing and independent benchmarks arrive. Open questions remain around NPU standardisation, cache compression, multi-GPU efficiency, and how quickly runtimes adopt new model architectures. The durable method is therefore to size from the exact model file and context, preserve headroom, and benchmark the complete application before committing to hardware.

Frequently Asked Questions

How Much RAM Do I Need to Run a Local AI Model?

Sixteen gigabytes is a practical starting point for compact 3B to 8B quantised models. Twenty-four to 32 GB is better for 12B to 20B models and larger contexts. A 70B Q4 model normally needs roughly 48 to 64 GB of usable memory after adding cache and runtime headroom.

Is 8 GB of VRAM Enough for Local LLMs?

Yes, for many 3B to 8B four-bit models and modest context lengths. Some 12B to 14B models may run with partial offload or tighter quantisation, but 8 GB leaves little room for long context, vision components, or concurrent requests.

Can a 16 GB GPU Run a 32B Model?

It may load a heavily quantised 32B model or split layers between GPU and system RAM, but a typical 32B Q4 package plus context often exceeds 16 GB. A 24 GB to 32 GB GPU or larger unified-memory system is a more reliable target.

Do I Need an NVIDIA GPU for Local AI?

No. Apple Silicon, AMD GPUs, Intel GPUs, and CPUs can run local models through supported backends. NVIDIA remains the safest compatibility choice for many Windows and Linux applications, while Apple and AMD unified-memory systems can offer substantially more model capacity.

Is Unified Memory the Same as VRAM?

No. Unified memory is one pool shared by CPU, GPU, operating system, and applications. It can let a GPU access far more memory than a typical consumer card, but not all advertised capacity is available to the model, and performance depends on bandwidth and software support.

How Much Storage Do Local Models Need?

Keep at least twice the active model set if you download, convert, or quantise models locally. An 8B Q4 model may occupy about 5 GB, while a 70B Q4 model can exceed 40 GB. Multiple quantisations, caches, embeddings, and temporary conversion files quickly multiply storage use.

Why Does Context Length Increase VRAM Use?

The runtime stores key-value attention data for previous tokens. That KV cache grows with context length, model architecture, batch size, and cache precision. Long coding or document sessions can therefore consume far more memory than the weight file alone.

What Is the Best Hardware for a 70B Local Model?

A system with at least 64 GB of usable high-bandwidth unified or accelerator memory is the simplest target for a dense 70B Q4 model. Alternatives include 48 GB multi-GPU setups, 96 GB to 128 GB unified-memory PCs, or slower CPU-GPU offload with abundant system RAM.

References

AMD. (2025, July 29). FAQs: AMD Variable Graphics Memory, VRAM, AI model sizes, quantisation, MCP, and more.

AMD. (2026, March 2). AMD gives consumers and businesses more AI PC options with expanded Ryzen AI 400 Series portfolio.

Apple. (2025, March 5). Apple unveils new Mac Studio, the most powerful Mac ever.

Apple. (2026a, March 3). Apple debuts M5 Pro and M5 Max to supercharge the most demanding pro workflows.

Apple. (2026b, March 3). Apple introduces MacBook Pro with all-new M5 Pro and M5 Max.

Framework Computer. (2025, February 25). Introducing the Framework Desktop.

ggml-org. (2026). llama.cpp quantise documentation and memory requirements.

Lin, J., Tang, J., Tang, H., Yang, S., Chen, W.-M., Wang, W.-C., Xiao, G., Dang, X., Gan, C., & Han, S. (2024). AWQ: Activation-aware weight quantisation for on-device LLM compression and acceleration. Proceedings of Machine Learning and Systems, 6.

LM Studio. (2026). System requirements, local pricing, and developer API documentation.

NVIDIA. (2025, January 6). New GeForce RTX 50 Series graphics cards and laptops powered by NVIDIA Blackwell.

NVIDIA. (2026, May 31). NVIDIA and Microsoft reinvent Windows PCs for the age of personal AI.

Ollama. (2025, September 23). New model scheduling.

Ollama. (2026). Pricing and usage limits.