Precision
Numerical precision is the fineness with which a numerical representation distinguishes values, often determined by the number of significant digits or bits available.
Learn Go with Pocket-Sized Projects
Numeric precision is the degree of detail or significant accuracy with which a numerical value is represented or computed.
- How it is used
- Uses precision to prevent representation errors and enforce currency scales in a money-conversion CLI.
- How deep it goes
- Explains floating-point limitations and implements fixed-precision parsing, conversion, validation, and formatting; the converter remains a tutorial.
- Examples
- Illustrates lost subtraction, approximate sine, decimal subunits, currency precision checks, and exchange-rate test cases.
- What you can do after
- Builds a bounded fixed-precision decimal representation and applies it to a tutorial currency converter.
Read how Learn Go with Pocket-Sized Projects explains precision →
Modern C
Precision is the C-standard property that constrains the number of distinguishable or representable digits and thereby the valid range of a type.
- How it is used
- Uses precision to explain representable values and consequences for integer and floating-point operations.
- How deep it goes
- Defines precision in bits and works through ranges, shifts, and a simplified floating-point representation.
- Examples
- Works through a 16-bit representation of 240, left shifts, and double-precision limit macros.
- What you can do after
- Evaluate/understand how type precision affects ranges, bit operations, and floating-point computations.
Modern C, Third Edition
Precision is the number of significant binary digits available to represent a floating-point value in the C standard's restricted sense.
- How it is used
- Uses precision to explain representable values and the effects of limited digits in floating-point and integer types.
- How deep it goes
- Works through a binary floating-point representation and an unsigned integer example, using a simplified floating-point model.
- Examples
- Calculates −5/32 from sign, exponent, and mantissa bits, and identifies the set bits of 240 in a 16-bit unsigned type.
- What you can do after
- Evaluate representable values and distinguish type precision from computational accuracy.
Clojure: The Essential Reference
Numeric precision is the amount of numerical detail a representation or calculation can retain.
- How it is used
- Guides numerical design when rounding, decimal arithmetic, money, or repeated calculations make representation errors consequential.
- How deep it goes
- Explains floating-point artifacts, BigDecimal contexts, rounding, and contagion rules, while noting exceptions to rough promotion behavior.
- Examples
- Demonstrates 22/7, decimal subtraction artifacts, account-share calculations, and a hash function using unchecked integer arithmetic.
- What you can do after
- Apply it via with-precision, BigDecimal promotion, explicit rounding, and deliberate numeric-type choices.
Read how Clojure: The Essential Reference explains precision →
Deep Learning with R, Third Edition
Floating-point precision is the degree of numerical detail representable by a floating-point data format.
- How it is used
- Uses precision to choose numerical formats for machine-learning training, inference, memory use, and computation speed.
- How deep it goes
- Explains finite numerical resolution, precision levels, floating-point encoding, uneven spacing, and value-dependent representation error.
- Examples
- Uses uint8 ranges, the real-number line, and float32 encoding of π to make finite numerical resolution concrete.
- What you can do after
- Apply float16 or bfloat16 for memory-efficient inference, while recognizing why float32 can support stable training gradients.
Read how Deep Learning with R, Third Edition explains precision →
Python How-To
Numeric precision formatting controls the number of digits displayed for a numeric value.
- How it is used
- Formats decimal, scientific-notation, and percentage values for readable f-string output.
- How deep it goes
- Explains precision syntax through worked outputs and challenges, stopping at displayed formatting rather than numeric representation.
- Examples
- Uses decimal values, scientific notation, percentages, temperature input, and a formatted product price.
- What you can do after
- Apply it via Python f-strings to present numeric values with a chosen number of displayed digits.
The Art of Code
Numerical precision is the degree of exactness with which a numeric value is represented or calculated.
- How it is used
- Guides numeric type selection when calculations must preserve exactness, especially for monetary values.
- How deep it goes
- Defines precision through a type-choice example but does not explain floating-point representation or numerical error mechanics.
- Examples
- Contrasts representing a price with BigDecimal rather than float.
- What you can do after
- Evaluate numeric type choices against the precision requirements of a value’s domain.
Domain-Specific Small Language Models
A numeric representation setting that determines how many bits are used to store or compute model values during serving.
- How it is used
- Frames floating-point precision as a factor in language model memory use and computational speed.
- How deep it goes
- Defines 32-bit float bit allocation and contrasts 16-bit, mixed-precision, and BF16 behavior.
- Examples
- Uses 32-bit float bit allocation and FP16/BF16 comparisons as format examples.
- What you can do after
- Evaluate memory and speed tradeoffs among floating-point formats and distinguish them from low-bit quantization.
Read how Domain-Specific Small Language Models explains precision →
CUDA for LLMs
Numerical precision is the level of detail with which a numerical representation can distinguish values.
- How it is used
- Frames precision as a choice when assembling and tuning CUTLASS GEMM kernels.
- How deep it goes
- Names precision choices as an implementation dimension but does not specify formats or tradeoffs.
Related concepts
- bit shiftingconfigures · Bit shifting is a bitwise operation that moves the positions of an integer’s binary digits left or right by a specified count.
- quantizationused by · Model quantization represents model parameters or computations with reduced-precision numerical values, often to lower storage or computational requirements.
- recallcontrasts with · Recall is the proportion of all relevant or actual positive items that a system successfully retrieves or correctly identifies.
- format stringpart of · A format string is a character sequence containing literal text and directives or placeholders that govern how supplied values are represented in formatted text.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.









