For 7 billion parameters at 4 bits, the theoretical weights-only size is 3.5 decimal GB. Real files also contain scales, metadata and possibly higher-precision tensors. Runtime and cache need additional memory, so file size is not the complete RAM or VRAM requirement.
The three items that occupy memory
Weights. The model parameters in the chosen format. Quantization can reduce their size, but its effect on answers depends on the model, method and task.
Runtime. Libraries, buffers, support structures and other memory used by the host application.
KV cache. In autoregressive models with attention, the cache keeps intermediate states of already-processed tokens so they are not recomputed at every step ( Hugging Face ). Growth depends on length, concurrency and strategy; bounded windows, static allocation, offloading and quantized caches are also available.
Why no single number exists
The KV cache depends on layer count, attention-head configuration, sequence length, batch size and precision. Two implementations of the same model can use different cache strategies with different trade-offs between memory and speed ( Hugging Face ).
So the right question is not “how much RAM does this model need”, but “how much RAM does this model need, with this context, in this application”.
Measure your workload
| Item | What to record |
|---|---|
| Identity | model, revision, format, quantization |
| Context | maximum expected input and output, not the theoretical maximum |
| Runtime | version, backend, offload settings |
| Memory | observed peak and free memory while the model remains loaded |
| Experience | time to first token (TTFT) and completion time |
Measure at three moments: idle, during loading, and in the heaviest use case you actually expect. The peak is what counts.
Offload is not a free solution
Splitting weights between RAM and GPU memory can make a load possible that would otherwise fail. Measure the resulting speed: data transfers, memory access and CPU computation may introduce overhead, with effects that depend on the runtime and workload. Offload does not intentionally change the weights, but different backends and numerical precision can produce different outputs.
When memory is almost full, allow room for longer inputs and other applications. Check stability and timing under the expected workload before relying on that configuration in daily work.
Keep the measured values alongside the model, context length and request count: they will help when the workload changes. Use the weights calculation as a starting estimate, then measure memory use in the application you intend to run. For the whole service, see comparing local and cloud AI .
Sources
- Cache strategies ↗ — Hugging Face · accessed 2026-11-01
- Loading models ↗ — Hugging Face · accessed 2026-11-01


