Skip to main content

1.3 Double Precision, Extended Formats & Arithmetic Error Analysis

📚Module 01: Data Representation & Floating-Point StandardsTopic 1.3⏱️16 min read
🎯High-Yield For:University Semester Exams • Hardware Engineering Rounds • Numerical Computing & Scientific Simulation

💡 Core Intuition: The High-Definition Camera vs. The Quantum Microscope​

Imagine filming an everyday family picnic using a 1080p standard HD camera versus filming subatomic crystal growth using an 8K ultra-magnification electron microscope.

🍳 The Everyday Analogy: 1080p Standard vs. 8K High-Dynamic Imaging​

  • The 1080p Camera (Single Precision FP32): For everyday video playback, 1080p resolution provides roughly 2 million pixels. The image looks crisp and clean on a mobile phone or living room television. But if an astrophysicist attempts to digitally zoom in 50,000×50,000\times onto a microscopic star cluster at the outer edge of the frame, the image instantly dissolves into blurry, blocky compression artifacts.
  • The 8K Quantum Microscope (Double Precision FP64): By quadrupling the pixel density and dynamic bit depth, the sensor captures subtle tonal gradations and micro-structures that survive millions of recursive digital zoom operations. In scientific supercomputing, you cannot afford pixel blurring: an aerospace simulation calculating atmospheric drag across millions of iterations will veer into catastrophic instability if microscopic rounding errors compound at every time step.

💻 Bridging to Computer Science​

Single Precision (binary32) allocates 23 fractional bits, providing ≈7.22\approx 7.22 decimal digits of precision. For simple game graphics or mobile user interfaces, 7 decimal digits are more than sufficient.

However, in numerical linear algebra, celestial mechanics, weather modeling, and structural engineering, systems execute billions of chained arithmetic operations:

xn+1=f(xn)x_{n+1} = f(x_n)

If each individual multiplication introduces a tiny truncation error of 10−710^{-7}, those rounding errors rapidly cascade into total numerical garbage after a few thousand iterations.

To guarantee mathematical stability, modern computer architectures implement 64-Bit Double Precision (binary64), expanding effective significand precision to 53 bits (≈15.95\approx 15.95 decimal digits) and expanding the dynamic range to over 600 orders of magnitude (10±30810^{\pm 308}).



🔬 IEEE-754 64-Bit Double Precision Architecture (binary64)​

In the official IEEE-754 standard, Double Precision allocates an 8-byte (64-bit) word across three standardized fields:

  1. Sign Bit (SS, 1 Bit — Bit 63):
    • S=0  ⟹  Positive NumberS = 0 \implies \text{Positive Number}
    • S=1  ⟹  Negative NumberS = 1 \implies \text{Negative Number}
  2. Biased Exponent Field (EE, 11 Bits — Bits 62..52):
    • Total combinations: 211=20482^{11} = 2048 distinct states (00 to 20472047).
    • Standard Bias: Bias=2k−1−1=211−1−1=210−1=1024−1=1023\text{Bias} = 2^{k-1} - 1 = 2^{11-1} - 1 = 2^{10} - 1 = 1024 - 1 = \mathbf{1023}
    • True Exponent calculation: e=E−1023e = E - 1023
    • Normal range: 1≤E≤20461 \le E \le 2046 (corresponding to true exponents e∈[−1022,+1023]e \in [-1022, +1023]).
    • Reserved states: E=0E = 0 (Zeros and Subnormals) and E=2047E = 2047 (Infinities and NaNs).
  3. Mantissa / Fraction Field (MM, 52 Bits — Bits 51..0):
    • Employs implicit hidden-bit normalization: Significand=(1.M)2=1+∑i=152b52−i⋅2−i\text{Significand} = (1.M)_2 = 1 + \sum_{i=1}^{52} b_{52-i} \cdot 2^{-i}
    • Effective Precision: 52 physical bits+1 hidden bit=53 bits of precision52 \text{ physical bits} + 1 \text{ hidden bit} = \mathbf{53 \text{ bits of precision}} (equivalent to ≈15.95\approx 15.95 decimal digits).
IEEE-754 / ISO/IEC 60559Total Frame: 8 Bytes (64 Bits)

IEEE-754 Double Precision (64-Bit binary64) Word Format

Standard Hardware Word Layout: 1 Sign Bit, 11 Biased Exponent Bits, 52 Mantissa Fraction Bits

Sign (S)
1 Bit Bit 63 (MSB)
Exponent (E)
11 Bits Bits 62..52
Fraction (M)
52 Bits Bits 51..0
1 Bit Bit 63 (MSB)Sign (S)

0 for positive values, 1 for negative values. Follows sign-magnitude symmetry.


⚖️ The Precision Spectrum: From Half-Precision to Extended Precision​

Modern computer architectures—spanning cloud servers, mobile SoCs, and AI accelerators—support a spectrum of floating-point formats tailored to specific computational trade-offs:

Floating-Point Architectural Spectrum

Balancing memory bandwidth, silicon area, dynamic range, and numerical precision

AI / Tensor Cores

bfloat16 (Brain Float)

🧠
Dominant Architecture / DomainLarge Language Models & Deep Learning Training
  • •16 bits total: 1 Sign, 8 Exponent, 7 Mantissa
  • •Identical dynamic range to FP32 (spanning 10^±38)
  • •Dramatically slashes GPU memory bandwidth and SRAM footprint
  • •Prevents gradient underflow without requiring loss-scaling algorithms
"Sacrifices precision to preserve dynamic range, perfect for deep neural network matrix multiplications."
Scientific Gold Standard

FP64 (Double Precision)

🔬
Dominant Architecture / DomainSupercomputing, Molecular Dynamics & Astrophysics
  • •64 bits total: 1 Sign, 11 Exponent, 52 Mantissa
  • •Vast dynamic range spanning 10^±308
  • •Delivers 16 full decimal digits of mathematical precision
  • •Prevents catastrophic cancellation across billions of recursive iterations
"The bedrock of modern engineering simulations where cumulative rounding error cannot be tolerated."

Comprehensive Floating-Point Format Comparison​

Standard FormatTotal Bits (NN)Exponent Bits (kk)Bias (2k−1−12^{k-1}-1)Mantissa Bits (mm)Effective PrecisionSmallest Positive Normalized (Nmin⁡N_{\min})Largest Positive Normalized (Nmax⁡N_{\max})Decimal Digits
Half (binary16)165151011 bits2−14≈6.10×10−52^{-14} \approx 6.10 \times 10^{-5}(2−2−10)×215≈65,504(2 - 2^{-10}) \times 2^{15} \approx 65,504≈3.3\approx 3.3
bfloat1616812778 bits2−126≈1.18×10−382^{-126} \approx 1.18 \times 10^{-38}(2−2−7)×2127≈3.39×1038(2 - 2^{-7}) \times 2^{127} \approx 3.39 \times 10^{38}≈2.4\approx 2.4
Single (binary32)3281272324 bits2−126≈1.18×10−382^{-126} \approx 1.18 \times 10^{-38}(2−2−23)×2127≈3.40×1038(2 - 2^{-23}) \times 2^{127} \approx 3.40 \times 10^{38}≈7.2\approx 7.2
Double (binary64)641110235253 bits2−1022≈2.23×10−3082^{-1022} \approx 2.23 \times 10^{-308}(2−2−52)×21023≈1.80×10308(2 - 2^{-52}) \times 2^{1023} \approx 1.80 \times 10^{308}≈16.0\approx 16.0
Quadruple (binary128)1281516383112113 bits2−16382≈3.36×10−49322^{-16382} \approx 3.36 \times 10^{-4932}(2−2−112)×216383≈1.19×104932(2 - 2^{-112}) \times 2^{16383} \approx 1.19 \times 10^{4932}≈34.0\approx 34.0

⚙️ The 5-Stage Floating-Point Addition & Subtraction Pipeline​

Unlike integer addition (which requires a single adder circuit), floating-point addition and subtraction require a complex, multi-stage hardware execution pipeline:

Architecture Flow

5-Stage Hardware Floating-Point Addition Pipeline

Sequential execution stages executed inside high-performance CPU/GPU FPUs

🔍Stage 1

Zero Check & Exp Compare

Inspect zero operands; calculate exponent delta ΔE = E_A - E_B.

→
Delta E
📐Stage 2

Significand Alignment

Shift smaller significand right by |ΔE| bit positions to align scales.

→
Aligned
⚡Stage 3

Mantissa Add / Subtract

Carry-lookahead adder computes sum or difference of aligned mantissas.

→
Raw Sum
🎯Stage 4

Normalization & Exp Adjust

Leading Zero Detector (LZD) shifts mantissa and adjusts exponent.

→
Normalized
🛡️Stage 5

IEEE Rounding & Flags

Apply Guard, Round, Sticky (GRS) bits and set IEEE exception flags.

1. Stage 1: Zero Check & Exponent Comparison​

  • The FPU checks if either operand is zero. If A=0A = 0, output BB; if B=0B = 0, output AA.
  • The exponent difference is computed: ΔE=EA−EB\Delta E = E_A - E_B
  • If ΔE>0\Delta E > 0, operand AA is larger; if ΔE<0\Delta E < 0, operand BB is larger.

2. Stage 2: Significand Alignment (Denormalization Shift)​

  • Addition cannot occur until both numbers share the exact same exponent scale.
  • The smaller operand's significand (including its implicit leading 1) is shifted to the right by ∣ΔE∣|\Delta E| bit positions.
  • The larger exponent is adopted as the provisional exponent of the result: Eresult=max⁡(EA,EB)E_{\text{result}} = \max(E_A, E_B)

3. Stage 3: Significand Addition or Subtraction​

  • If the operand signs are identical (SA=SBS_A = S_B), the aligned significands are added: Msum=MA+MBM_{\text{sum}} = M_A + M_B
  • If the operand signs differ (SA≠SBS_A \ne S_B), the smaller significand is subtracted from the larger significand: Mdiff=Mlarger−MsmallerM_{\text{diff}} = M_{\text{larger}} - M_{\text{smaller}}
  • The sign of the larger magnitude operand determines the provisional sign (SresultS_{\text{result}}).

4. Stage 4: Result Normalization​

  • Case A (Carry Out on Addition): If significand addition generated a carry bit (M≥2.0M \ge 2.0), the significand is shifted right by 1 bit, and the exponent is incremented by 1 (Eresult←Eresult+1E_{\text{result}} \leftarrow E_{\text{result}} + 1).
  • Case B (Cancellation on Subtraction): If subtraction resulted in leading zeros (M<1.0M < 1.0), a hardware Leading Zero Detector (LZD) determines the position of the first 1. The significand is shifted left by kk positions, and the exponent is decremented by kk (Eresult←Eresult−kE_{\text{result}} \leftarrow E_{\text{result}} - k).

5. Stage 5: Rounding & Exception Flagging​

  • Shifts in Stages 2 and 4 push low-order bits out of the physical mantissa register. The FPU retains three extra hardware bits:
    • Guard Bit (GG): The first bit shifted out beyond the mantissa.
    • Round Bit (RR): The second bit shifted out.
    • Sticky Bit (SstickyS_{\text{sticky}}): The logical OR of all subsequent bits shifted out (Ssticky=b3∨b4∨b5…S_{\text{sticky}} = b_3 \lor b_4 \lor b_5 \dots).
  • The FPU applies the selected IEEE-754 rounding mode (default: Round to Nearest, Ties to Even).
  • Status flags are updated: Overflow, Underflow, Inexact, or Invalid Operation.

⚠️ Arithmetic Error Analysis: Machine Epsilon, Cancellation & Non-Associativity​

1. Machine Epsilon (ϵmach\epsilon_{\text{mach}})​

Machine Epsilon defines the upper bound on the relative approximation error due to rounding:

📐Machine Epsilon Definitionformula
ϵmach=2−m\epsilon_{\text{mach}} = 2^{-m}

Where mm is the number of physical mantissa bits:

  • Single Precision (m=23m = 23): ϵsingle=2−23≈1.19209×10−7\epsilon_{\text{single}} = 2^{-23} \approx 1.19209 \times 10^{-7}
  • Double Precision (m=52m = 52): ϵdouble=2−52≈2.22045×10−16\epsilon_{\text{double}} = 2^{-52} \approx 2.22045 \times 10^{-16}

In practical programming, machine epsilon represents the smallest positive number ϵ\epsilon such that: 1.0+ϵ>1.01.0 + \epsilon > 1.0 Any number smaller than ϵ/2\epsilon / 2 added to 1.01.0 is completely absorbed and discarded!

2. Catastrophic Cancellation​

When subtracting two nearly identical floating-point numbers (x≈yx \approx y), the leading significant bits cancel out, leaving only the least significant bits:

x=1.2345678andy=1.2345671  ⟹  x−y=0.0000007x = 1.2345678 \qquad\text{and}\qquad y = 1.2345671 \implies x - y = 0.0000007

The result shifts left to normalize (7.0×10−77.0 \times 10^{-7}), filling the low-order mantissa bits with meaningless zeros or accumulated rounding noise. A value that previously possessed 8 digits of precision is suddenly reduced to a single reliable digit!

3. Non-Associativity and Non-Distributivity​

In standard algebra, real numbers obey associative and distributive laws. In computer hardware, floating-point arithmetic obeys neither:

(a+b)+c≠a+(b+c)(a + b) + c \ne a + (b + c) a×(b+c)≠(a×b)+(a×c)a \times (b + c) \ne (a \times b) + (a \times c)

Why Compilers Care: Modern optimizing compilers (GCC, Clang, MSVC) cannot legally reorder floating-point operations or vectorize math loops across multiple cores unless the developer explicitly provides the -ffast-math or -Ofast compiler flag, granting permission to violate strict IEEE-754 associativity rules.


🏭 Real-World Production Context: The 1991 Patriot Missile Failure & Fast-Inverse Square Root​

1. The Patriot Missile Software Tragedy (Dhahran, 1991)​

On February 25, 1991, an Iraqi Scud missile struck an American military barracks in Dhahran, Saudi Arabia, killing 28 soldiers. An incoming Patriot missile defense battery failed to track and intercept the Scud due to a software timing truncation bug:

  • The Root Bug: The Patriot battery's internal clock measured time in tenths of a second (0.1 s0.1\text{ s}) using an integer counter. To calculate time in seconds, the software multiplied the counter by 0.10.1.
  • In binary, 0.1100.1_{10} is an infinite non-terminating fraction: 0.110=0.0001100110011001100110011…20.1_{10} = 0.0001100110011001100110011\dots_2
  • The computer register truncated this value after 24 bits, introducing an error of: Truncation Error≈0.000000095 seconds per tick\text{Truncation Error} \approx 0.000000095\text{ seconds per tick}
  • The Compounding Failure: The battery had been left running continuously for over 100 hours (360,000360,000 seconds). Multiplying the microscopic per-tick error across 100 hours accumulated a total system clock drift of: Δt=360,000×0.000000095≈0.3433 seconds\Delta t = 360,000 \times 0.000000095 \approx \mathbf{0.3433\text{ seconds}}
  • A Scud missile travels at Mach 5 (≈1676 m/s\approx 1676\text{ m/s}). In 0.34330.3433 seconds, the incoming missile traveled over 575 meters575\text{ meters}. When the Patriot radar looked for the missile in its predicted tracking gate, the sky was empty, and the tracking system aborted!

2. Quake III Arena's Legendary Fast Inverse Square Root​

In 1999, id Software released Quake III Arena, featuring one of the most celebrated algorithmic hacks in computing history: Fast Inverse Square Root (Q_rsqrt).

To compute 3D vector lighting and surface reflections, the GPU must normalize vectors by computing 1/x1 / \sqrt{x}. In 1999, hardware division and square root instructions took up to 40 clock cycles. John Carmack's engine solved it in 4 clock cycles by directly manipulating IEEE-754 single precision bit fields:

float Q_rsqrt(float number) {
long i;
float x2, y;
const float threehalfs = 1.5F;

x2 = number * 0.5F;
y = number;
i = * ( long * ) &y; // Evil floating point bit-level hacking
i = 0x5f3759df - ( i >> 1 ); // What the...? (Magic Constant)
y = * ( float * ) &i;
y = y * ( threehalfs - ( x2 * y * y ) ); // 1st iteration of Newton-Raphson

return y;
}

The Architectural Insight: By treating the IEEE-754 32-bit floating-point bit pattern as a standard 32-bit integer, shifting it right by 1 bit (i >> 1) halves the exponent (emulating a square root x=x1/2\sqrt{x} = x^{1/2}). Subtracting from the magic constant 0x5f3759df negates the exponent (emulating the inverse x−1/2x^{-1/2}) and provides a brilliant mathematical seed approximation, refined to 99.8%99.8\% accuracy in a single Newton-Raphson step!


📐 Numerical Applications & Worked Examples​

The following worked engineering problems demonstrate step-by-step IEEE-754 64-bit double precision encoding, FPU addition pipeline simulation, and machine epsilon error bounds.

Example 1: 64-Bit Double Precision Hexadecimal Encoding​

Problem Statement:
Encode the decimal value −0.0625-0.0625 into standard IEEE-754 64-bit double precision format. Express the result as a 64-bit binary bit stream and its corresponding 16-digit hexadecimal machine word.

Step-by-Step Solution:

  1. Determine Sign Bit (SS):
    The number is negative   ⟹  S=1\implies S = 1.

  2. Convert Magnitude to Binary:
    0.062510=116=2−4=(0.0001)20.0625_{10} = \frac{1}{16} = 2^{-4} = (0.0001)_2

  3. Normalize to 1.M×2e1.M \times 2^e Form:
    (0.0001)2=1.0×2−4(0.0001)_2 = 1.0 \times 2^{-4}

    • True exponent: e=−4e = -4.
    • Fractional mantissa string: all zeros.
  4. Calculate Biased Exponent (EE):
    Apply double-precision bias (Bias=1023\text{Bias} = 1023): E=e+1023=−4+1023=101910E = e + 1023 = -4 + 1023 = 1019_{10} Convert 1019101019_{10} into an 11-bit binary integer: 1019=512+256+128+64+32+16+8+2+1=(01111111011)21019 = 512 + 256 + 128 + 64 + 32 + 16 + 8 + 2 + 1 = (01111111011)_2

  5. Construct 52-Bit Mantissa Field (MM):
    Since the significand is (1.0)2(1.0)_2, the 52 fractional bits are all zeros: M=0000…00002 (52 zeros)M = 0000\dots 0000_2 \text{ (52 zeros)}

  6. Assemble and Translate to Hexadecimal:
    [S]1 [E]11 [M]52=1 01111111011 0000…00002[S]_1 \ [E]_{11} \ [M]_{52} = 1 \ 01111111011 \ 0000\dots 0000_2 Group into 4-bit nibbles: 1011 1111 1011 0000 0000…000021011 \ 1111 \ 1011 \ 0000 \ 0000 \dots 0000_2 Translate to 16 hex digits: 0xBFB0000000000000\mathbf{0xBFB0000000000000}


Example 2: Step-by-Step Floating-Point Addition Pipeline​

Problem Statement:
Execute the floating-point addition of two single precision normalized numbers AA and BB through the hardware execution pipeline: A=1.5×22andB=1.25×2−1A = 1.5 \times 2^2 \qquad\text{and}\qquad B = 1.25 \times 2^{-1}

Show the exact significand alignment, mantissa addition, normalization, and resulting output in scientific notation.

Step-by-Step Solution:

  1. Express Operands with Implicit Leading Bits:

    • A=(1.100)2×22  ⟹  EA=2,MA=1.1002A = (1.100)_2 \times 2^2 \implies E_A = 2, M_A = 1.100_2
    • B=(1.010)2×2−1  ⟹  EB=−1,MB=1.0102B = (1.010)_2 \times 2^{-1} \implies E_B = -1, M_B = 1.010_2
  2. Stage 1 & 2: Exponent Comparison & Significand Alignment:

    • Exponent difference: ΔE=EA−EB=2−(−1)=3\Delta E = E_A - E_B = 2 - (-1) = 3
    • Shift the smaller operand's significand (BB) to the right by 33 bit positions: MB′=1.0102≫3=0.0010102M_B' = 1.010_2 \gg 3 = 0.001010_2
    • Both operands now share the common exponent E=2E = 2: A=1.1000002×22A = 1.100000_2 \times 2^2 B=0.0010102×22B = 0.001010_2 \times 2^2
  3. Stage 3: Significand Addition:
    Add the aligned binary significands:

📐Stage 3: Aligned Significand Binary Additionformula
+1.1000002+ 0.0010102+1.1010102  ⟹  (1.100000)2+(0.001010)2=(1.101010)2\begin{array}{r} \phantom{+} 1.100000_2 \\ +\ 0.001010_2 \\ \hline \phantom{+} 1.101010_2 \end{array} \qquad\implies\qquad (1.100000)_2 + (0.001010)_2 = (1.101010)_2
  1. Stage 4: Normalization:
    The resulting significand 1.10101021.101010_2 already has exactly one 1 before the binary point (1.0≤M<2.01.0 \le M < 2.0). No shift or exponent adjustment is required!

  2. Convert to Decimal Verification:
    M=1+2−1+2−3+2−5=1+0.5+0.125+0.03125=1.65625M = 1 + 2^{-1} + 2^{-3} + 2^{-5} = 1 + 0.5 + 0.125 + 0.03125 = 1.65625 Result=1.65625×22=1.65625×4=6.62510\text{Result} = 1.65625 \times 2^2 = 1.65625 \times 4 = \mathbf{6.625_{10}} Verification: A=1.5×4=6.0A = 1.5 \times 4 = 6.0; B=1.25×0.5=0.625B = 1.25 \times 0.5 = 0.625; A+B=6.0+0.625=6.625A + B = 6.0 + 0.625 = 6.625 (Exact match!).


Example 3: Machine Epsilon & Catastrophic Cancellation Error Bounds​

Problem Statement:
A scientific computation evaluates the mathematical expression f(x)=x+1−xf(x) = \sqrt{x + 1} - \sqrt{x} for x=108x = 10^8 using IEEE-754 single precision (ϵsingle≈1.19×10−7\epsilon_{\text{single}} \approx 1.19 \times 10^{-7}).

  1. Explain why direct evaluation suffers catastrophic cancellation.
  2. Reformulate the expression algebraically to eliminate cancellation.
  3. Calculate the percentage relative error avoided by the reformulation.

Step-by-Step Solution:

  1. Identify the Cancellation Hazard:
    For x=108x = 10^8: x=108=10,000.0\sqrt{x} = \sqrt{10^8} = 10,000.0 x+1=100,000,001≈10,000.00005\sqrt{x + 1} = \sqrt{100,000,001} \approx 10,000.00005 In single precision (7 decimal digits), both 108+1\sqrt{10^8 + 1} and 108\sqrt{10^8} round to 10,000.010,000.0! Direct subtraction yields: f(x)=10,000.0−10,000.0=0.0f(x) = 10,000.0 - 10,000.0 = \mathbf{0.0} This results in a total 100%100\% loss of significant digits.

  2. Algebraic Reformulation (Rationalization):
    Multiply and divide by the algebraic conjugate (x+1+x)(\sqrt{x + 1} + \sqrt{x}): f(x)=(x+1−x)(x+1+x)x+1+x=(x+1)−xx+1+x=1x+1+xf(x) = \frac{(\sqrt{x + 1} - \sqrt{x})(\sqrt{x + 1} + \sqrt{x})}{\sqrt{x + 1} + \sqrt{x}} = \frac{(x + 1) - x}{\sqrt{x + 1} + \sqrt{x}} = \frac{1}{\sqrt{x + 1} + \sqrt{x}}

  3. Evaluate the Stable Reformulation:
    x+1+x≈10,000.0+10,000.0=20,000.0\sqrt{x + 1} + \sqrt{x} \approx 10,000.0 + 10,000.0 = 20,000.0 f(x)=120,000.0=0.00005=5.0×10−5f(x) = \frac{1}{20,000.0} = \mathbf{0.00005} = 5.0 \times 10^{-5} By replacing subtraction with addition in the denominator, catastrophic cancellation is completely eliminated, restoring 7 digits of genuine precision!


🎯 Exam & Interview Pitfall Check​

Core Conceptual Questions
  • Why did Google create bfloat16 instead of using IEEE standard FP16 for TPU deep learning?
    IEEE FP16 allocates 5 exponent bits, providing a dynamic range of only 10±510^{\pm 5}. In deep learning backpropagation, neural network weight gradients frequently shrink below 10−510^{-5}, causing catastrophic gradient underflow and halting training unless developers implement complex loss-scaling algorithms. By allocating 8 exponent bits (the same as FP32), bfloat16 spans 10±3810^{\pm 38}, completely eliminating underflow while halving memory footprint and doubling matrix multiply throughput.

  • What are the Guard, Round, and Sticky bits (G,R,SG, R, S) used for in hardware FPUs?
    During significand alignment, bits shifted to the right out of the mantissa register would be permanently lost. The FPU maintains three temporary bits:

    • Guard (GG): The first shifted-out bit.
    • Round (RR): The second shifted-out bit.
    • Sticky (SS): The logical OR of all subsequent discarded bits. These three bits allow the FPU to implement IEEE-754 "Round to Nearest, Ties to Even" with complete mathematical rigor as if infinite precision registers were available.
Common Interview Traps
  • Trap: Confusing Double Precision Bit Allocations with Single Precision.
    Never assume double precision merely doubles all fields. Single precision has an 8-bit exponent and 23-bit mantissa. Double precision has an 11-bit exponent (not 16) and a 52-bit mantissa (not 46). The bias is 10231023 (not 255255).

  • Trap: Believing Machine Epsilon is the Smallest Representable Number.
    Candidates frequently confuse machine epsilon ϵ\epsilon with the smallest positive subnormal number Nmin⁡N_{\min}. Machine epsilon ϵ=2−m\epsilon = 2^{-m} is the smallest difference such that 1.0+ϵ>1.01.0 + \epsilon > 1.0. The smallest representable positive number is 2−1492^{-149} (single precision) or 2−10742^{-1074} (double precision), which is over 30 orders of magnitude smaller than ϵ\epsilon!

  • Trap: Assuming 64-Bit Integers and 64-Bit Floats Have the Same Precision.
    A 64-bit integer (uint64_t) has 64 bits of integer precision (exact up to 264−1≈1.84×10192^{64} - 1 \approx 1.84 \times 10^{19}). A 64-bit float (double) has only 53 bits of precision (exact only up to 253=9,007,199,254,740,9922^{53} = 9,007,199,254,740,992). Storing a 64-bit integer larger than 2532^{53} in a double permanently discards the lowest integer digits!


💬

Discussion & Doubts