Q4 and Q8 describe families of quantized representations, not guaranteed quality levels. Reducing weight precision can help a model fit in memory, but its effect on answers must be checked for the model, method and task ( Hugging Face ).

How much space can you save?

A theoretical weights-only calculation for 7 billion parameters gives 14 GB at 16 bits, 7 GB at 8 bits and 3.5 GB at 4 bits, using decimal GB. These are calculated sizes, not measured files or runtime memory. Scales, metadata, higher-precision tensors, cache and runtime add overhead.

In a uniform representation, 4 bits distinguish 16 codes and 8 bits distinguish 256. Real formats also depend on blocks, scales and method; the label does not describe the entire model.

AspectWhat to compare
Memoryfile, RAM/VRAM and peak under load separately
Qualityerrors on the tasks and languages you will use
Contextrealistic lengths and demanding cases
Speedtime to first token and generation speed on the chosen backend
Formatexact variant and runtime support

Why the method matters

Methods may require calibration, treat layers differently or retain parts of a model at higher precision. Two files labelled Q4 are not necessarily equivalent. The GGUF documentation helps identify the format; it does not certify performance on your work.

Check how the runtime loads the format too: it may keep the weights packed or dequantize them to a higher precision. A small file therefore does not guarantee the same reduction in inference memory.

How to make a useful comparison

Use the same model revision, input, generation settings and context limits. Record format and backend. Prepare cases with explicit criteria and repeat runs when sampling introduces variability. No universal minimum number of examples makes a result reliable.

Assess number extraction, instruction following, code generation and language performance separately. Include long contexts without assuming the gap must always grow. If Q4 meets your criteria and saves enough memory, it may be useful; greater compression does not guarantee better answers.

Speed does not follow a fixed ratio

Reading less data may help when memory bandwidth is the bottleneck. Kernels, hardware and dequantization overhead may change or cancel the benefit. Moving from Q8 to Q4 does not guarantee twice the speed.

Choose a variant that preserves the quality your work needs, then check its memory budget . If you also change model or service, use the comparison protocol to separate the variables.

Sources

Report a correction ↗