Senior 3 min · March 24, 2026

GCD/LCM Overflow — RSA Signatures Fail 0.1% of Keys

A 0.

N
Naren · Founder
Plain-English first. Then code. Then the interview question.
About
 ● Production Incident 🔎 Debug Guide
Quick Answer
  • Euclidean algorithm finds GCD by repeatedly replacing (a,b) with (b, a mod b) until b=0
  • Extended GCD also finds coefficients x,y such that ax + by = GCD(a,b)
  • LCM(a,b) = ab // GCD(a,b) — compute as (a//GCD)b to avoid overflow
  • Time complexity: O(log min(a,b)) — at most 93 steps for 64-bit integers
  • Production nightmare: overflow in LCM when using a*b before division in languages with fixed-width integers
  • Biggest mistake: assuming GCD gives positive result for negative inputs — behavior varies across languages
Plain-English First

The Euclidean algorithm for GCD is one of the oldest algorithms known — it appears in Euclid's Elements (300 BC). The insight: if you want to find the largest tile that fits perfectly in a 252×105 room, try 105×105 tiles first. You have a 42×105 leftover strip. Now find the largest tile for 105×42. Remainder is 21×42. Then 42×21 — perfect fit! GCD(252,105) = 21. The key insight: GCD(a,b) = GCD(b, a mod b).

The Euclidean algorithm is 2300 years old, appears in Euclid's Elements (Book VII, Proposition 2), and is still one of the fastest algorithms relative to input size that exists. For 64-bit numbers, it terminates in at most 93 steps. The RSA cryptographic system — which secures most HTTPS traffic — depends on extended Euclidean to compute private keys. Every time your browser establishes a TLS connection, a variant of this algorithm runs.

The extended Euclidean algorithm (Bézout's identity: ax + by = GCD(a,b)) is the computational backbone of modular inverses. Modular inverses are the backbone of modular arithmetic. Modular arithmetic is the backbone of RSA, Diffie-Hellman, elliptic curve cryptography, and the Chinese Remainder Theorem. The entire edifice of public-key cryptography rests on an algorithm Euclid described in 300 BC.

Euclidean Algorithm for GCD

Based on the identity: GCD(a, b) = GCD(b, a mod b). Repeat until b = 0 — then GCD = a.

gcd.pyPYTHON
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
def gcd(a: int, b: int) -> int:
    while b:
        a, b = b, a % b
    return a

def gcd_recursive(a: int, b: int) -> int:
    return a if b == 0 else gcd_recursive(b, a % b)

def lcm(a: int, b: int) -> int:
    return a * b // gcd(a, b)  # avoid overflow: a // gcd * b

print(gcd(252, 105))    # 21
print(gcd(48, 18))      # 6
print(lcm(12, 18))      # 36
print(lcm(4, 6))        # 12

# Python 3.9+ built-in
import math
print(math.gcd(252, 105))  # 21
print(math.lcm(12, 18))    # 36
Output
21
6
36
12
21
36
Implementation Note
The iterative while loop is preferred over recursion in production because it avoids stack depth limits. Python's default recursion limit is 1000, which could be exceeded with large Fibonacci inputs.
Production Insight
Using modulo on negative numbers in Python produces non-negative remainders (consistent with floor division).
But in C++, the remainder can be negative, leading to incorrect GCD if you don't use absolute values.
Always ensure both operands are non-negative or use a wrapper that calls abs().
Key Takeaway
GCD(a,b) = GCD(b, a mod b) reduces to GCD(a,0)=a in O(log min(a,b)) steps.
Fibonacci pairs are worst case — Lamé's theorem bounds steps at ≤5×decimal digits of smaller number.
Use iterative version in production; handle negative numbers with abs().
Choose Your GCD Implementation
IfNeed GCD of fixed-width integers (C++, Java long)
UseUse iterative version with abs() on inputs; beware of overflow in a % b if b == 0.
IfNeed GCD of arbitrary precision (Python, Java BigInteger)
UseUse iterative version; Python's math.gcd is implemented in C and is faster than pure Python.
IfPerformance-critical for many GCD calls
UseConsider binary GCD (Stein's algorithm) which replaces modulo with bit shifts — faster on hardware with slow division.

GCD Implementations Across Languages

The Euclidean algorithm is nearly identical in every language. The key differences are handling of negative numbers and integer overflow. Below we show the iterative implementation in six widely used languages: C++, Java, C#, JavaScript, C, and Python (for completeness). All follow the same logic: while b != 0, set (a, b) = (b, a % b).

gcd_multilang.cppCPP
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
// C++ (modulo may return negative for negative inputs)
#include <cstdlib>
int gcd(int a, int b) {
    a = std::abs(a); b = std::abs(b);
    while (b) {
        int t = b;
        b = a % b;
        a = t;
    }
    return a;
}

// Java (similar to C++, use Math.abs)
public static int gcd(int a, int b) {
    a = Math.abs(a); b = Math.abs(b);
    while (b != 0) {
        int t = b;
        b = a % b;
        a = t;
    }
    return a;
}
// BigInteger version: a.gcd(b)

// C# (use Math.Abs, modulo is remainder, can be negative)
static int Gcd(int a, int b) {
    a = Math.Abs(a); b = Math.Abs(b);
    while (b != 0) {
        int t = b;
        b = a % b;
        a = t;
    }
    return a;
}

// JavaScript (Numbers are 64-bit floats, integer-safe up to 2^53)
function gcd(a, b) {
    a = Math.abs(a); b = Math.abs(b);
    while (b) {
        let t = b;
        b = a % b;
        a = t;
    }
    return a;
}

// C (unsigned avoids sign issues, but handle negative externally)
unsigned int gcd(unsigned int a, unsigned int b) {
    while (b) {
        unsigned int t = b;
        b = a % b;
        a = t;
    }
    return a;
}

// Python (already shown above, included for completeness)
# def gcd(a,b): while b: a,b=b,a%b; return a
Built-in Alternatives
C++17 has std::gcd in <numeric>. Java has BigInteger.gcd(). C# has BigInteger.GreatestCommonDivisor. JavaScript does not have a built-in GCD, but it's easy to write. Python's math.gcd is the fastest option.
Production Insight
In C++ and Java with fixed-width integers, overflow in a % b is not a concern (the modulo operation always produces a result with magnitude less than |b|). However, in LCM calculations, multiplication after division is critical. In JavaScript, all numbers are 64-bit floats, so integers up to 2^53 are exact — beyond that, bitwise operations fail. Use BigInt for large numbers in JS.
Key Takeaway
The iterative Euclidean algorithm works identically in every language when you handle negative inputs with abs(). Always use abs() to ensure a correct, non-negative GCD.

Visualizing the Euclidean Algorithm: Step by Step

Let's trace GCD(252, 105) step by step. The algorithm repeatedly replaces (a, b) with (b, a mod b) until b = 0. The final a is the GCD. We can see how quickly the numbers shrink — Lamé's theorem predicts at most 5×3 = 15 steps for 3-digit numbers, but we finish in just 4 steps.

Production Insight
Visualizing the algorithm makes it obvious that each step reduces the numbers quickly. In debugging, printing intermediate (a,b) pairs can help identify if the algorithm is stuck in a loop (e.g., due to overflow causing incorrect modulo). Logging the steps can be invaluable in production when GCD behaves unexpectedly.
Key Takeaway
The Euclidean algorithm reaches GCD in O(log min(a,b)) steps. Every two iterations at least halve the larger number, making it extremely efficient even for 64-bit inputs.

Euclidean vs Binary GCD: Advantages and Disadvantages

Binary GCD (Stein's algorithm) is an alternative that replaces modulo with bit shifts and subtraction, which can be faster on processors with slow division. However, it is more complex and offers no advantage on modern hardware with fast modulo. Below is a comparison.

AspectEuclidean (modulo)Binary GCD (Stein)
Algorithmic stepsO(log min(a,b))O(log min(a,b))
Worst-case stepsFibonacci numbers (~93 for 64-bit)Slightly higher constant
Core operationa % b>>, &, subtraction
Speed on modern CPUsVery fast (modulo pipeline)Comparable, no significant edge
Speed on slow division CPUsSlowerFaster (no division)
Implementation complexityTrivialMore complex (handle even cases, loops)
Handling negativesRequires abs() on inputsRequires abs() on inputs
Use casesGeneral purposeEmbedded systems, microcontrollers

In most production systems, the standard Euclidean algorithm is preferred due to its simplicity and excellent performance. Binary GCD is reserved for environments where division is especially expensive.

When Binary GCD Wins
Some 8-bit microcontrollers have no hardware divider and software modulo can be 10× slower than bit shifts. In those cases, binary GCD can be a meaningful optimization. For server-side applications, stick with Euclidean.
Production Insight
In cloud services, always benchmark before switching to binary GCD. The modulo operation on modern x86_64 takes about 1 clock cycle on recent architectures — there is no real benefit. However, if your code runs on diverse ARM or RISC-V cores, consider profiling.
Key Takeaway
For 99% of production use cases, the standard Euclidean algorithm with modulo is the best choice. Binary GCD is a niche optimisation for hardware without fast division.

Practice Problems on GCD and LCM

To solidify your understanding, work through these problems from competitive programming platforms. They cover GCD of arrays, coprime pairs, and Euler's totient, which directly build on the Euclidean algorithm.

  1. [GCD of an Array](https://cses.fi/problemset/task/1080) (CSES) – Compute GCD of a list of numbers. A straightforward application.
  2. [Coprime Pairs](https://www.codechef.com/problems/COPRIME) (CodeChef) – Count pairs in an array that are coprime. You'll need GCD(given pair) == 1.
  3. [Euler's Totient Function](https://cses.fi/problemset/task/1081) (CSES) – Compute φ(n) for each n in a list. Formula: φ(n) = n * ∏(1 - 1/p). Needs prime factors and GCD concepts.
  4. [LCM Sum](https://www.spoj.com/problems/LCMSUM/) (SPOJ) – Sum of LCMs from 1 to n. Requires efficient GCD and divisor enumeration.
  5. [GCD Extreme](https://www.spoj.com/problems/GCDEX/) (SPOJ) – Sum of GCD of all pairs for numbers up to n. Uses Euler's totient and prefix sums.
  6. [Modular Inverse](https://www.hackerrank.com/challenges/modular-inverse/problem) (HackerRank) – Compute x such that a*x ≡ 1 mod m. Direct extended GCD.
  7. [RSA Key Generation](https://www.cryptool.org/en/cto/highlights/rsa-step-by-step) (CrypTool) – Practice key generation involving GCD and modular inverse.
  8. [Binomial Coefficients and GCD](https://atcoder.jp/contests/abc156/tasks/abc156_d) (AtCoder) – Uses GCD for fraction simplification.
Key Takeaway
Practice problems transform theoretical knowledge into debugging skill. Focus on GCD of arrays, coprime checks, and Euler's totient — these patterns recur in production cryptography and competitive programming alike.
● Production incidentPOST-MORTEMseverity: high

Incorrect GCD in RSA Key Generation Caused Signature Failures

Symptom
RSA signatures generated by the library would sometimes fail verification. The failure rate was ~0.1% and not reproducible on every key generation.
Assumption
The RSA implementation was believed to be correct. Developers assumed the standard formula e * d ≡ 1 mod φ(n) was always satisfied because they used the extended Euclidean algorithm.
Root cause
In the key generation script, LCM was computed as lcm = (a b) // gcd using 64-bit integers. For some large primes used in RSA, the intermediate product a b overflowed before the division, producing a truncated LCM. This led to an incorrect modular inverse for the private exponent d.
Fix
Changed the LCM calculation to lcm = (a // gcd) * b, ensuring the division happens first. Applied the same fix to all LCM uses in the codebase. Added unit tests with large prime test vectors.
Key lesson
  • Always compute LCM as (a // gcd) * b to avoid overflow — a single multiplication after division is safe.
  • Test cryptographic functions with known test vectors (NIST, RFC 8017).
  • Never assume fixed-width integer math is safe for large intermediate values.
  • In production, use language-provided arbitrary precision integers (Python's int, Java's BigInteger) or verify overflow limits.
Production debug guideSymptom → Action for common failures4 entries
Symptom · 01
Fraction simplification produces unexpected results (e.g., 6/4 reduces to 3/2 but shows as 2/1)
Fix
Check the sign of the GCD — negative GCD flips fraction signs. Use abs(gcd) or ensure both numerator and denominator are non-negative.
Symptom · 02
Modular inverse function throws 'no inverse' error for valid input
Fix
Verify that a and m are coprime: gcd(a, m) == 1. If not, no inverse exists. Also check that a is reduced modulo m before calling extended GCD.
Symptom · 03
Cryptographic signature verification fails sporadically
Fix
Check the key generation code for integer overflow in LCM. Reproduce with large primes (e.g., 1024-bit). Run the same steps using Python's math.lcm as a reference.
Symptom · 04
Pairwise LCM of an array produces extremely large numbers causing slowdown or memory issues
Fix
Use functools.reduce(lambda a,b: a//gcd(a,b)*b, numbers) and monitor intermediate size. If numbers are large, consider using arbitrary-precision integers or stopping early if LCM exceeds a threshold.
★ Quick Fixes for Common GCD/LCM BugsOne-liner commands and checks for debugging Euclidean algorithm problems in production
GCD returns negative value
Immediate action
Check input signs: pass absolute values if non-negative GCD needed.
Commands
python -c "import math; print(math.gcd(-15, 25))"
python -c "def ngcd(a,b): return math.gcd(abs(a),abs(b)); print(ngcd(-15,25))"
Fix now
Wrap calls with abs() on both arguments.
Modular inverse fails for valid coprime numbers+
Immediate action
Verify `gcd(a, m) == 1` and reduce `a % m`.
Commands
python -c "from math import gcd; print(gcd(15, 26))"
python -c "def modinv(a,m): g,x,_ = extended_gcd(a%m,m); return x%m if g==1 else None; print(modinv(3,7))"
Fix now
Use built-in pow(a, -1, m) in Python 3.8+ or implement extended GCD with proper modulo reduction.
LCM overflow in C++/Java+
Immediate action
Change `a*b/gcd` to `a/gcd*b`.
Commands
C++: `long long lcm = a / gcd(a,b) * b;`
Java: `BigInteger lcm = a.divide(gcd).multiply(b);`
Fix now
Always divide first, then multiply. Use 128-bit or BigInteger if numbers exceed 2^63.
GCD Algorithm Comparison
AlgorithmSteps (64-bit worst-case)OperationsProsCons
Euclidean (iterative)~93Modulo, subtractionSimple, correct, widely understoodModulo is slower than bit shifts on some hardware
Binary GCD (Stein)~140Bit shifts, subtractionFaster on processors with slow division (e.g., some ARM)More complex to implement correctly
Python math.gcdSame as EuclideanC implementationFast, handles negatives, built-inNot available in all Python versions (3.5+)

Key takeaways

1
GCD(a,b) = GCD(b, a mod b)
reduces to GCD(a,0)=a in O(log min(a,b)) steps.
2
LCM(a,b) = a*b / GCD(a,b)
always compute via GCD to avoid large intermediate values.
3
Extended GCD finds Bézout coefficients x,y such that ax + by = GCD(a,b).
4
Modular inverse of a mod m exists iff GCD(a,m) = 1
compute via extended GCD.
5
Fibonacci pairs are worst case for Euclidean algorithm
Lamé's theorem bounds at 5×digits steps.

Common mistakes to avoid

4 patterns
×

Assuming GCD handles negative numbers the same way across languages

Symptom
Fraction simplification flips sign unexpectedly; modular inverse returns negative.
Fix
Always apply abs() to both arguments before passing to GCD. For modular inverse, reduce a modulo m first, then apply x % m to the result.
×

Overflow in LCM by multiplying before dividing

Symptom
LCM produces incorrect (truncated) results for large numbers, causing downstream failures in scheduling or cryptography.
Fix
Use the formula lcm = (a // gcd(a,b)) * b. In languages without arbitrary precision, use BigInteger or similar.
×

Forgetting to check GCD == 1 before computing modular inverse

Symptom
mod_inverse function throws an error or returns incorrect value for non-coprime numbers.
Fix
Check gcd(a, m) == 1 before proceeding. If not, the inverse does not exist. For RSA key generation, verify that the chosen e and φ(n) are coprime.
×

Using recursive Euclidean algorithm without depth limit awareness

Symptom
Recursion depth exceeded for large Fibonacci numbers, causing stack overflow crash.
Fix
Use iterative version (while b: a,b = b, a%b) instead of recursive. Python's recursion limit can be increased, but iterative is safer in production.
INTERVIEW PREP · PRACTICE MODE

Interview Questions on This Topic

Q01JUNIOR
What is the time complexity of the Euclidean algorithm and why?
Q02SENIOR
How does the extended Euclidean algorithm compute modular inverses?
Q03JUNIOR
When does the modular inverse of a (mod m) not exist?
Q04SENIOR
How would you compute LCM of an array of numbers without overflow?
Q05SENIOR
Explain the worst-case input for the Euclidean algorithm and why it occu...
Q01 of 05JUNIOR

What is the time complexity of the Euclidean algorithm and why?

ANSWER
O(log min(a,b)) because each two iterations reduce a by at least half. Lamé's theorem proves the number of iterations is at most 5 times the number of decimal digits in the smaller number. For 64-bit integers, the limit is ~93 steps.
FAQ · 4 QUESTIONS

Frequently Asked Questions

01
Why use a*b//gcd instead of a//gcd*b for LCM?
02
What is Bézout's identity?
03
How does the Euclidean algorithm perform on negative numbers?
04
Can I use the Euclidean algorithm for polynomials?
🔥

That's Number Theory. Mark it forged?

3 min read · try the examples if you haven't

Previous
Sieve of Eratosthenes — Prime Number Generation
2 / 6 · Number Theory
Next
Prime Factorization and Divisors