AI Math Stress Test
The criterion: Ask where the math runs. A language model doesn't calculate — it generates what a right answer looks like. For arithmetic, "looks right" and "is right" are different numbers, and it can't tell you which one it handed you.
Run it against the AI you already use. Copy a question, paste it in, write down its answer — then reveal the worked key here.
How to run it: paste each question into ChatGPT, Copilot, Gemini — whatever you use. Write down its answer. Then reveal the key below: the exact figure, the worked steps, and the source. Where they don't match, you've found math you can't trust on your own screen.
A single filer has $100,000 of taxable income. Using these federal brackets — 10% to $11,600, 12% to $47,150, 22% to $100,525, 24% above — what is their total federal income tax?
Reveal the answer
Where AI slips: the tempting shortcut is the top rate on the whole income — 22% × $100,000 = $22,000. Wrong by ~$5,000. Each bracket only taxes its own slice.
tax = Σ (income in bracket) × rate
= 11,600 × 10% + 35,550 × 12% + 52,850 × 22%
= 1,160 + 4,266 + 11,627
= $17,053 (effective rate 17.05%, marginal rate 22%)
Progressive marginal-bracket method, IRS Publication 17.
What is the monthly payment on a $25,000 loan at 6% APR over 5 years (60 payments)?
Reveal the answer
Where AI slips: it reverts to principal ÷ term ($416.67) or fumbles the amortization factor, landing a few dollars off — small per line, real over a portfolio of loans.
pmt = pv · r / (1 − (1 + r)^−n), r = 6%/12 = 0.005, n = 60
= 25,000 × 0.005 / (1 − 1.005^−60)
= $483.32 / month (total repaid ≈ $28,999)
Standard amortizing-loan / annuity payment formula (Excel PMT convention).
A municipal bond yields 3.5%, tax-free. For an investor in the 37% bracket, what taxable yield would be equivalent?
Reveal the answer
Where AI slips: the intuitive move is to add the rate — 3.5% × 1.37 = 4.80%. The correct inverse is to divide by (1 − rate).
TEY = tax-free yield / (1 − tax rate)
= 0.035 / (1 − 0.37)
= 0.035 / 0.63
= 5.56%
Investopedia — Tax-Equivalent Yield.
A fund's 12 monthly returns are below. With a 0% risk-free rate, what is its annualized Sharpe ratio?
Reveal the answer
Where AI slips: a convention trap — using population stdev (÷ n) instead of sample (÷ n−1), or forgetting to annualize by √12. Miss either and the number drifts.
Sharpe = mean / stdev(sample) × √12
mean = 0.101 / 12 = 0.008417 (0.84%/mo)
stdev = 0.026190 (sample, ÷ n−1)
= 0.008417 / 0.026190 × 3.4641
= 1.11
Sharpe ratio (annualized), Sharpe 1966 / standard definition.
Same 12 monthly returns, 0% risk-free — what is the annualized Sortino ratio?
Reveal the answer
Where AI slips: it reuses the full standard deviation instead of downside deviation, which counts only losing months and divides by n. Get that wrong and Sortino collapses toward Sharpe.
Sortino = mean / downside-deviation × √12
downside-dev = √( Σ min(0, r)² / n )
= √( (0.015² + 0.027² + 0.009² + 0.041²) / 12 )
= √( 0.002716 / 12 ) = 0.015044
= 0.008417 / 0.015044 × 3.4641
= 1.94
Sortino ratio (downside-deviation), standard definition.
An account's month-end values (indexed to 100) were below. What was the maximum drawdown?
Reveal the answer
Where AI slips: it reports the biggest single-month drop (130 → 116 = −10.8%). Max drawdown is the largest peak-to-trough fall — the peak of 130 all the way down to the later trough of 109.
max drawdown = (trough − prior peak) / prior peak
peak = 130 (month 6), subsequent trough = 109 (month 12)
= (109 − 130) / 130
= −16.15%
Drawdown (peak-to-trough), standard definition.
Using those same 12 monthly returns, what is the 95% one-month historical Value at Risk?
Reveal the answer
Where AI slips: it switches to parametric VaR (mean − 1.645 × σ) and presents that as the historical answer — a different method, a different number — or it just grabs the single worst month (−4.1%).
Historical VaR(95%) = −(5th-percentile return)
sorted returns, rank = 0.05 × (12 − 1) = 0.55
interpolate between −4.1% and −2.7%: −4.1% + 0.55 × 1.4% = −3.33%
VaR = 3.33% (a one-month loss exceeded ~5% of the time)
Historical Value at Risk (return quantile), standard definition.
An investment costs $100,000 today and pays back $20,000, $30,000, $40,000, $50,000 over the next four years. What is the IRR?
Reveal the answer
Where AI slips: IRR has no closed form — it requires iterative root-finding. A model can't iterate, so it guesses a plausible rate (often "~10%") or averages the cash back. Confident, and wrong.
IRR solves NPV = 0:
−100,000 + 20,000/(1+r) + 30,000/(1+r)² + 40,000/(1+r)³ + 50,000/(1+r)⁴ = 0
solved numerically → r = 12.83%
Internal rate of return (NPV = 0), standard definition.
In a 50/50 liquidity pool, one asset's price doubles relative to the other. What is the impermanent loss versus simply holding the two?
Reveal the answer
Where AI slips: an exotic formula it almost always misremembers — and the answer is counterintuitively small. Guesses run wild (some say ~25%, confusing it with the price move).
IL = 2√r / (1 + r) − 1, r = price ratio = 2 = 2√2 / 3 − 1 = 2.8284 / 3 − 1 = −5.72% (vs. simply holding the two assets)
Uniswap v2 documentation — impermanent loss.
A yield pool advertises 12% APR, compounded monthly. What is the actual APY?
Reveal the answer
Where AI slips: it ignores the compounding and answers 12% flat. The monthly compounding adds 68 basis points — small here, decisive across a yield comparison.
APY = (1 + APR/n)^n − 1, n = 12
= (1 + 0.12/12)^12 − 1
= 1.01^12 − 1
= 12.68%
Investopedia — Annual Percentage Yield.
This is how FactoryOS answers the same questions. It doesn't let the model do the math: the AI chooses the data and the calculation, and a deterministic engine computes every figure exactly — the same way every time — and shows the work with the source attached. Every answer above was reproduced by that engine, not transcribed.
What you're really testing isn't whether AI is useful — it is — but whether it should be the thing doing the arithmetic. The criterion for anything you'd put near your numbers: ask where the math runs. Inside the model, it's a guess in a suit.
If your AI got even one of these wrong, the question isn't whether it's smart — it's whether it should be the thing doing your arithmetic. FactoryOS keeps the math out of the model.
See math done deterministically