Precision
Numerical precision is the fineness with which a numerical representation or computation distinguishes and retains values.
Learn Go with Pocket-Sized Projects
Numeric precision is the degree of detail or significant accuracy with which a numerical value is represented or computed.
- How it is used
- Guides safe representation, conversion, validation, and display of amounts in a Go money-converter CLI.
- How deep it goes
- Works through floating-point failures and a fixed-decimal implementation, including parsing and precision adjustment, but does not establish a rounding rule.
- Examples
- Uses a lost one-euro subtraction, a near-zero sine result, decimal parsing tests, and exchange-rate conversion cases.
- What you can do after
- Build it from scratch: represent decimal amounts with integer subunits, check currency precision, and test conversion limits.
Read how Learn Go with Pocket-Sized Projects explains precision →
Modern C
Precision is the C-standard property that constrains the number of distinguishable or representable digits and thereby the valid range of a type.
- How it is used
- Uses precision to reason about representable ranges, unsigned bit operations, and floating-point values.
- How deep it goes
- Defines integer precision as a bit count and floating-point precision as a mantissa-bit count, then works through their practical consequences.
- Examples
- Works through a 16-bit unsigned value and its shifts, and gives mantissa and exponent values for a floating-point example.
- What you can do after
- Evaluate/understand type ranges and floating-point limitations, and check whether an unsigned shift count is valid.
Modern C, Third Edition
Precision is the number of significant binary digits available to represent a floating-point value in the C standard's restricted sense.
- How it is used
- Defines representable binary digits for C numeric types and supports reasoning about ranges, shifts, and floating-point approximation.
- How deep it goes
- Derives unsigned ranges from bit positions and works through a simplified floating-point formula, without specifying every implementation’s type precision.
- Examples
- Uses a three-bit mantissa to calculate −5/32 and unsigned bit positions to establish the maximum 2ᵖ − 1.
- What you can do after
- Evaluate/understand representable ranges and precision-dependent operations without confusing precision with computational accuracy.
Deep Learning with PyTorch, Second Edition
- How it is used
- Evaluates how often predicted nodules are genuine when overall accuracy hides failures on imbalanced data.
- How deep it goes
- Defines the true-positive ratio, implements it from classification counts, and examines undefined results when no positives are predicted.
- Examples
- Uses a cautious guard dog and a nodule classifier that reports high accuracy while missing positive samples.
- What you can do after
- Diagnose/debug with it by logging precision alongside recall and checking for false positives or undefined values.
Read how Deep Learning with PyTorch, Second Edition explains precision →
Clojure: The Essential Reference
Numeric precision is the amount of numerical detail a representation or calculation can retain.
- How it is used
- Guides numerical design when rounding, decimal arithmetic, money, or repeated calculations make representation errors consequential.
- How deep it goes
- Explains floating-point artifacts, BigDecimal contexts, rounding, and contagion rules, while noting exceptions to rough promotion behavior.
- Examples
- Demonstrates 22/7, decimal subtraction artifacts, account-share calculations, and a hash function using unchecked integer arithmetic.
- What you can do after
- Apply it via with-precision, BigDecimal promotion, explicit rounding, and deliberate numeric-type choices.
Read how Clojure: The Essential Reference explains precision →
Deep Learning with R, Third Edition
Floating-point precision is the degree of numerical detail representable by a floating-point data format.
- How it is used
- Uses precision to choose numerical formats for machine-learning training, inference, memory use, and computation speed.
- How deep it goes
- Explains finite numerical resolution, precision levels, floating-point encoding, uneven spacing, and value-dependent representation error.
- Examples
- Uses uint8 ranges, the real-number line, and float32 encoding of π to make finite numerical resolution concrete.
- What you can do after
- Apply float16 or bfloat16 for memory-efficient inference, while recognizing why float32 can support stable training gradients.
Read how Deep Learning with R, Third Edition explains precision →
Python How-To
Numeric precision formatting controls the number of digits displayed for a numeric value.
- How it is used
- Formats decimal, scientific-notation, and percentage values for readable f-string output.
- How deep it goes
- Explains precision syntax through worked outputs and challenges, stopping at displayed formatting rather than numeric representation.
- Examples
- Uses decimal values, scientific notation, percentages, temperature input, and a formatted product price.
- What you can do after
- Apply it via Python f-strings to present numeric values with a chosen number of displayed digits.
The Art of Code
Numerical precision is the degree of exactness with which a numeric value is represented or calculated.
- How it is used
- Guides numeric type selection when calculations must preserve exactness, especially for monetary values.
- How deep it goes
- Defines precision through a type-choice example but does not explain floating-point representation or numerical error mechanics.
- Examples
- Contrasts representing a price with BigDecimal rather than float.
- What you can do after
- Evaluate numeric type choices against the precision requirements of a value’s domain.
Domain-Specific Small Language Models
A numeric representation setting that determines how many bits are used to store or compute model values during serving.
- How it is used
- Frames precision formats as a production-serving concern for LLM inference.
- How deep it goes
- Distinguishes precision formats from smaller-precision quantization and names multiple techniques and libraries without implementation detail.
- Examples
- Uses precision formats, smaller-precision formats, and quantization techniques and libraries as concrete options.
- What you can do after
- Evaluate quantization as a way to address LLM inference performance.
Read how Domain-Specific Small Language Models explains precision →
Related concepts
- recallcontrasts with · Recall is the fraction of relevant, positive, or target outcomes that a system successfully identifies, retrieves, or captures.
- quantizationused by · Quantization maps numerical values to a finite set of discrete representations, often using reduced precision to lower model storage or computation requirements.
- bit shiftingconfigures · Bit shifting is a bitwise operation that moves the positions of an integer’s binary digits left or right by a specified count.
- format stringpart of · A format string is a character sequence containing literal text and directives that specify how supplied values are represented in formatted input or output.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.