Enter Calculation Inputs
Use the form below to compare a reference value with a computed value and inspect floating point sensitivity, tolerance, ULP distance, and rounding growth.
Formula Used
Signed error: computed value − exact value
Absolute error: |computed value − exact value|
Relative error: absolute error ÷ |exact value|
Percent error: relative error × 100
Machine epsilon: base1 − p
Unit roundoff: 0.5 × base1 − p
Accumulated rounding bound: γn = (n × u) ÷ (1 − n × u), when n × u < 1
Forward error estimate: κ × γn
ULP spacing estimate: basee − p + 1, where e is the local exponent near the value
absolute error = |x̂ − x| relative error = |x̂ − x| / |x| percent error = relative error × 100How to Use This Calculator
- Enter the reference value you trust most.
- Enter the computed or rounded value you want to test.
- Select a preset or manually set the mantissa and exponent sizes.
- Choose the base, usually 2 for binary arithmetic.
- Add an estimated operation count to model cumulative rounding growth.
- Enter a condition number if your problem is known to be sensitive.
- Set an absolute tolerance for pass or fail checking.
- Submit the form to view the result, graph, and export buttons.
Example Data Table
These examples show how error can grow from truncation, limited precision, and underflow-like behavior.
| Example | Reference value | Computed value | Absolute error | Relative error | ULP error |
|---|---|---|---|---|---|
| 1/3 estimate | 0.333333333333 | 0.3333333 | 3.333333e-8 | 1.000000e-7 | 1.118481066078 |
| Square root of 2 | 1.414213562373 | 1.4142 | 1.356237e-5 | 9.590046e-6 | 113.769431445748 |
| Pi truncation | 3.14159265359 | 3.14 | 0.00159265359 | 0.000506957383 | 6,680.073322283104 |
| Large-scale subtraction | 1.000000e+6 | 1.000000e+6 | 0.099999999977 | 9.999999e-8 | 1.599999999627 |
| Tiny value to zero | 1.000000e-8 | 0 | 1.000000e-8 | 1 | 1.125900e+7 |
Frequently Asked Questions
1. What does floating point error mean?
Floating point error is the gap between a mathematically ideal value and its stored or computed machine representation. It appears because many real numbers cannot be represented exactly in finite binary or decimal precision.
2. Why can two close decimal inputs still produce error?
A decimal number may look exact to people but still map to a nearby binary value inside hardware. Arithmetic then works on that stored approximation, not the original mathematical quantity.
3. What is the difference between absolute and relative error?
Absolute error measures raw distance between values. Relative error scales that distance by the true value size. Relative error is often better when comparing results with very different magnitudes.
4. What is ULP error?
ULP means unit in the last place. ULP error estimates how many representable steps separate the computed number from a nearby ideal value at the same local scale.
5. Why is machine epsilon important?
Machine epsilon estimates the spacing near 1 for a chosen precision. It helps you judge the smallest meaningful change that arithmetic can reliably distinguish in that representation.
6. What does the condition number tell me?
The condition number describes how much output error can grow from small input or rounding disturbances. Large values usually mean the mathematical problem is sensitive, even before implementation details are considered.
7. Why can repeated operations worsen results?
Each operation may introduce a small rounding change. Many operations can accumulate those changes, especially in unstable formulas, long sums, iterative methods, or subtractive cancellation scenarios.
8. When should I worry about floating point error?
You should pay attention when tolerances are tight, magnitudes vary widely, subtraction cancels leading digits, or algorithms run many steps. Scientific, financial, and optimization tasks often need explicit error checks.