A parameter is a learned numerical value that determines part of a machine-learning model’s computation. Parameter count provides a practical signal of model size: LLaMA variants include 7B, 13B, 33B, and 65B parameters. section 3.4.7
Parameter count matters operationally because more parameters require more computation, memory, and usually more inference time. Streaming output can make a slow model feel responsive even when latency remains unsuitable for autocomplete or large text-processing workflows. section 3.2.3
Use parameter count as a first-pass planning heuristic:
- Estimate inference memory by multiplying parameter count in billions by two for FP16 or BF16 storage; a 7B model therefore needs about 14 GB before additional overhead. section 3.4.7
- Add memory for embeddings, tokens, and batch size; 512 tokens require about 512 MB, so 16 such batches require another 8 GB. section 3.4.7
- Plan for roughly 16 bytes per parameter during full-precision training, including optimizer tensors and gradients; training a 7B model therefore needs about 112 GB. section 3.4.7
Choosing hardware for a 7B model
A 7B-parameter model needs roughly 14 GB for inference weights at half precision, plus memory for embeddings, tokens, and batching. Training requires a much larger budget—about 112 GB before other operational considerations—so inference and training should not be sized the same way. section 3.4.7
