Floating-point arithmetic stores numbers with finite precision, so pandas calculations can produce tiny discrepancies even when the results are effectively equivalent. section 6.2
Nearly identical fare averages
Averages of the 20 longest and shortest taxi-ride fares can appear as 290.01000000000005, 290.00999999999993, or 290.0100000000001. Rounding produces 290.01, making the practical result clearer without pretending the underlying representations are identical. section 6.2
Choosing greater precision involves trade-offs:
- Using 128-bit floats can make forward and backward calculations agree more closely. section 6.2
- Higher precision doubles storage relative to 64-bit floats and can reduce portability because support varies by platform and pandas method. section 6.2
df = pd.read_csv(filename,
usecols=['passenger_count',
'trip_distance',
'total_amount'],
dtype={'total_amount':np.float128})
Try loading the fare column with NumPy's 128-bit floating-point dtype and compare the resulting calculations. section 6.2
