TL;DR: For a year the standard critique of LLM-designed strategies was “the model can’t actually do the math — the backtest is hallucinated.” That critique is now dead. ChatGPT, Claude and Gemini all execute real, sandboxed Python with pandas and Matplotlib (OpenAI, Anthropic, Google). The code runs. The Sharpe is computed, not written. And that is the trap: execution didn’t fix the problem — it laundered it. A memorized price path, a best-of-fifty silent search, a snooped published factor — each now emerges as a real number from real code, which is far harder to dismiss than an obvious hallucination. I call this laundered alpha. Below: why the failure moved up the stack from arithmetic to epistemology, the April 2026 benchmark that proves runnable code is still the wrong code, what Man Group and Two Sigma actually built instead, the one place LLM-designed code genuinely works (+0.62 to +1.83 out-of-sample Sharpe, verified), and a due-diligence checklist you can run before any LLM strategy touches capital.
Why did the old “LLMs can’t compute a backtest” argument just collapse?
Three months ago I would have told you the viral “which LLM writes the best strategy” posts were fiction — that the model was describing a backtest the way it describes a sunset, never running one. I would have been wrong, and I want to be candid about that, because the correction is the whole point of this post.
I sat down to re-run the experiment properly. I opened a frontier model, handed it five years of daily prices, and typed “design a momentum strategy and backtest it in Python.” It wrote Backtrader code, executed it in a sandbox, hit a KeyError on a column name, read its own traceback, patched the code, and re-ran — all before printing an equity curve and a Sharpe of 1.9. No arithmetic was faked. Every one of those P&L steps was computed by a CPU.
This is now the documented baseline, not a party trick. OpenAI’s data-analysis mode gives the model “a secure code execution environment” pre-loaded with “hundreds of Python libraries” that can “interpret errors and resolve issues with the generated code automatically” (OpenAI Help Center). Anthropic’s Claude “can analyze data, create visualizations, perform complex calculations, run system commands, create and edit files” via its code-execution tool (Anthropic docs). Gemini’s API “enables the model to generate and run Python code” and “learn iteratively from the code execution results” (Google AI).
Code interpreter — a sandboxed environment where the model writes Python, executes it on a real CPU, reads the output (including errors), and revises. Example: the model writes df['SMA50'] = df.Close.rolling(50).mean(), the backtest throws because the column is named close not Close, the model sees the traceback, rewrites with the correct case, and re-runs — no human touched it.
So the easy critique is gone. And its death is dangerous, because everyone who spent a year saying “the numbers aren’t real” is about to be shown a number that is real — and will have no argument left. The real problem was never the arithmetic. It was always the experiment the arithmetic was pointed at.
What is the bigger context — why is everyone rushing to let the model run its own backtests?
This isn’t a tooling fad; three forces converged.
Inference collapsed in price. In my Medallion Test post I clocked the same strategy-generation query falling from roughly $1 to $0.02 in cost. When a full “design and backtest a strategy” round-trip costs two cents, you run it fifty times before your coffee cools — and, crucially, you stop reading the code.
Institutional cover arrived. The AIMA industry survey found 95% of fund managers now use generative AI (up from 86% in 2023) and 58% expect to increase its use inside the investment process, versus just 20% saying so in 2023 (AIMA, corroboration). When the boss has blessed the tool, scrutiny drops.
Cheap alpha got scarce. With beta everywhere, desks now tolerate model risk they’d have laughed out of the room in 2021.
Free tool + boss’s blessing + scarce edge = nobody asks whether the experiment was valid. They only check whether the code ran. It did. That’s the trap.
What actually goes wrong when the code runs perfectly?
Here is the thesis in one line: execution didn’t remove the failure — it moved it up the stack, from arithmetic to epistemology, and gave it a clean alibi.
When a hallucinated backtest was obviously fake, you could dismiss it. When a real backtest computes a beautiful Sharpe on a contaminated experiment, you get laundered alpha: a number that is arithmetically true and scientifically worthless, dressed in the credibility of executed code.
Laundered alpha — a performance figure produced by genuinely executed code run on a contaminated or mis-specified experiment, such that the number is real but meaningless, and the realness makes it harder to reject. Example: a model “backtests” a 2019–2023 mean-reversion rule on a stock whose 2019–2023 path it memorized during training; the code is flawless, the Sharpe is computed, and the edge is pure hindsight — laundered into a chart.
There are four laundering channels, and code execution makes every one of them more convincing, not less.
Channel 1: The code runs, but it’s the wrong code
The freshest, sharpest evidence is QuantCode-Bench (arXiv, submitted 16 April 2026), the first benchmark built specifically to test whether LLM-generated executable trading-strategy code is actually correct. It runs 400 tasks from Reddit, TradingView, StackExchange and GitHub through a multi-stage pipeline: (1) does it compile, (2) does the backtest execute, (3) does it place trades, (4) does it semantically match the requested strategy — judged by an LLM. The headline finding is a scalpel through the entire “code interpreter solved it” narrative:
“the main limitations of current models are not related to syntax, but rather to the correct operationalization of trading logic, proper API usage, and adherence to task semantics” (QuantCode-Bench, arXiv:2604.15151).
Read that again. The model doesn’t fail because it can’t write Python. It fails because the Python it writes — which compiles, runs, and prints a Sharpe — implements a different strategy than the one you asked for. This is code hallucination: the AAAI-2025 CodeHalu paper defines it as code that is “syntactically correct and even semantically plausible, but may not execute as expected or fulfill specified requirements” (CodeHalu, arXiv:2405.00253).
Semantic misalignment — code that runs and returns a plausible number while implementing logic that differs from the specification. Example: you ask for “enter when RSI crosses below 30,” the model writes rsi < 30 evaluated on every bar instead of on the cross, so it enters and holds through the entire oversold region — a different strategy, a different (often flattering) equity curve, zero syntax errors.
Channel 2: The model memorized the future
This is the deepest channel and it connects directly to my July 1 prequel. In Your LLM Doesn’t Pick Stocks — It Remembers Them I named the Anti-Alpha Paradox: for an LLM, capability and contamination are the same axis — the smarter the model, the more of the historical price path it has memorized, so the better it “predicts” the past.
Code execution does nothing to fix this. If anything it weaponizes it. The Memorization Problem paper (arXiv, revised December 2025) is blunt: LLMs “cannot be trusted for economic forecasts during periods covered by their training data,” they have “memorized economic and financial data, recalling exact values before their knowledge cutoff,” and “instructions to respect historical boundaries fail to prevent recall-level accuracy” (arXiv:2504.14765). The corollary is devastating: “post-cutoff, we observe no recall.” The Profit Mirage paper (October 2025) shows exactly this in agents — “dazzling back-tested returns evaporate once the model’s knowledge window ends” (arXiv:2510.07920).
So the code runs a perfect backtest — on a price path the model already knew the ending of. Laundered alpha.
Channel 3: The silent best-of-N search (p-hacking at machine speed)
When a query costs two cents, the model — or you — can try fifty configurations and show you the winner. That is textbook backtest overfitting, and the math on it is settled. Bailey, Borwein, López de Prado and Zhu proved that “high simulated performance is easily achievable after backtesting a relatively small number of alternative strategy configurations,” and — the line every LLM strategy-shopper should tattoo on their wrist — “backtest overfitting leads to negative expected returns out-of-sample, rather than zero performance” (Pseudo-Mathematics, SSRN 2308659). The Deflated Sharpe Ratio was built precisely to correct for “selection bias under multiple testing” (SSRN 2460551). Code execution industrializes the trial count while hiding it — you see trial 50, never trials 1–49.
Channel 4: Crowding — everyone’s model read the same textbooks
Because every frontier model is trained on roughly the same corpus of published factors, independent users prompting “give me an alpha” get correlated strategies. AlphaAgent (arXiv, 2025) states it directly: LLM-driven approaches “often rely too heavily on existing knowledge, creating homogeneous factors that worsen crowding and accelerate decay” (arXiv:2502.16789). This is what I called prompt crowding in the Medallion post: N traders, one prompt, one crowded trade. Executed code makes the crowded strategy look artisanal — it’s still crowded.
Let me demolish the strawman with the actual specimen
The viral piece everyone shared is InsightBig’s “Comparing 3 LLMs for Generating Profitable Trading Strategies”: feed AAPL data to ChatGPT, Gemini and Perplexity, have each design and “backtest” a strategy. Perplexity “wins” — ~90% cumulative return, Sharpe 1.87, three trades, zero losses, ~−13.1% drawdown.
A year ago I’d have said “those numbers were never computed.” Having checked that specific article, the backtest there does read as narrated rather than executed. But that is the weak critique, and I want to retire it publicly. The strong critique is this: it does not matter whether that article ran the code. Assume it did. Assume every number is arithmetically perfect. You still can’t invest, because the experiment is contaminated on all four channels above — AAPL 2020–2025 is inside every model’s training window (Channel 2), the “three trades, zero losses” reeks of an unreported configuration search (Channel 3), and a plain-vanilla momentum rule on the most-analyzed stock on earth is maximally crowded (Channel 4). The realness of the code is not evidence of alpha. It’s the laundering.
First principles vs. inversion: two models, one conclusion
Mental model 1 — First principles. Strip the question to physics. Where can genuine, tradeable alpha come from? Only from information or structure the rest of the market hasn’t priced. An LLM’s raw material is its training corpus — the most public information that exists. By construction, a strategy derived purely from what the model already knows cannot contain private edge. Executing that strategy in flawless Python changes the presentation of the output, not its information content. First principles says: you cannot compute your way to alpha out of exclusively public priors, no matter how clean the code.
Mental model 2 — Inversion. Instead of asking “how do I get a good backtest from an LLM,” ask “how would I fake a great one without lying?” Answer: memorize the price path (Channel 2), silently search configs and show the winner (Channel 3), pick a crowded published factor everyone will validate (Channel 4), and — critically — run it in a code interpreter so the number is real. That last step is the innovation. Every plausible way to produce a spectacular, defensible, executed backtest is a contamination pathway. Inversion says the clean equity curve is the tell, not the trophy.
Both models converge: the danger scales with the realism of the execution. That’s the second-order effect nobody priced — better tools made the failure more persuasive.
Where does LLM-designed code genuinely work? (The honest counter-evidence)
I promised candor, so here’s the part that cuts against my own headline. There is a domain where LLM-generated, executed code produces real, verifiable, out-of-sample gains — and understanding its boundary is the whole game.
MadEvolve (arXiv, May 2026) applies an AlphaEvolve-style evolutionary loop to Bitcoin trading and splits the problem cleanly: forecasting (predict the price) vs. algorithm optimization (execution strategies, portfolio construction, risk models). The result is the most important nuance in this entire debate:
- On the algorithmic subproblems, “the evolved solutions improve substantially over hand-engineered baselines while maintaining out-of-sample validity.” Out-of-sample Sharpe improves in all four runs — +0.62, +1.29, +1.29, and +1.83 — with Run 1 lifting test-set Sharpe from 3.82 to 4.45 and impact-adjusted PnL from $47K to $271K (train 2022-01-01→2023-12-31, validation 2024, test 2025-01-01→2025-10-10) (MadEvolve PDF, arXiv:2605.23007).
- On forecasting — “the noisiest domain — the picture is more nuanced, with some signal persisting OOS, but with the optimized prediction model also showing some degree of overfitting” (MadEvolve).
Why the split? Because DeepMind’s FunSearch — the ancestor of all this — works “best for problems having… availability of an efficient evaluator” and “rich scoring feedback” (FunSearch, Nature 2023). Execution optimization has a crisp, cheap, near-deterministic evaluator: fill quality, slippage, turnover. Price forecasting has a noisy, non-stationary, adversarial one. LLM-guided code search wins where the evaluator is honest and loses where it’s noisy — and price is the noisiest evaluator in finance.
The rule I now operate by: point LLM code generation at the mechanical parts of trading (execution, portfolio construction, risk) where the scorekeeper is trustworthy — and never at the forecast, where the scorekeeper lies.
Evidence: what happens when LLMs trade real money forward?
Backtests are theater; forward tests are truth. Alpha Arena (Nof1.ai) gave six frontier models $10,000 each to trade crypto perpetuals live, same prompts, no human intervention. Season 1 results:
| Rank | Model | Final return | From $10,000 |
|---|---|---|---|
| 1 | Qwen3 Max (Alibaba) | +22.32% | ~$12,232 |
| 2 | DeepSeek Chat V3.1 | +4.89% | ~$10,489 |
| 3 | Claude Sonnet 4.5 | −30.81% | ~$6,919 |
| 4 | Grok 4 | −45.3% | ~$5,470 |
| 5 | Gemini 2.5 Pro | −56.71% | ~$4,329 |
| 6 | GPT-5 | −62.66% | ~$3,734 |
Source: iWeaver, SCMP, Longbridge. Four of six lost money; GPT-5 lost 62.66% of real capital — every trade genuinely executed, none of it a hallucination. Mid-competition, DeepSeek briefly spiked to ~+125% before giving it back (iWeaver) — extreme path-dependence, exactly what you’d expect from noise dressed as skill.
And the contamination-free academic benchmarks agree. StockBench (v2, March 2026): “most models struggle to outperform the simple buy-and-hold baseline” (arXiv:2510.02209). LiveTradeBench (November 2025, 21 LLMs, 50-day live): “high LMArena scores do not imply superior trading outcomes” (arXiv:2511.03628). FINSABER (KDD 2026, two decades, 100+ symbols): “previously reported LLM advantages deteriorate significantly,” with strategies “overly conservative in bull markets… and overly aggressive in bear markets, incurring heavy losses” (arXiv:2505.07078).
The pattern is total: the moment you remove the memorized window, the executed backtest’s edge evaporates.
What do the firms that actually deploy this build instead?
Not a strategy-writing chatbot. A funnel that inverts the bottleneck. Two Sigma names it precisely: “the research funnel is inverting: LLMs are widening the top, shifting the bottleneck from ‘we need more ideas’ to ‘we need to evaluate ideas faster‘” (Two Sigma 2026 Outlook).
Man Group’s AlphaGPT is the architecture made concrete — a three-agent pipeline: an Idea Person (“dozens of viable concepts within minutes”), an Implementer (“production-grade Python code” against proprietary data), and an Evaluator applying statistical-significance tests where “every AI-produced signal must demonstrate clear economic rationale and pass identical evaluation thresholds” as human research (Man Group). Crucially, Man admits hallucination “remains a big issue” and ships anyway, because “the architecture is designed to absorb hallucination rather than prevent it” (Electronic Trading Hub). Balyasny reports ~95% of its investment teams use its AI platform, built by a 20-person Applied AI team (OpenAI). AQR now runs ML in ~20% of its flagship signals (Bloomberg).
The one firm chasing full autonomy is the cautionary tale: Bridgewater’s AIA fund returned ~+11.9% in 2025 — trailing its own human-run Pure Alpha (+33%) and the S&P 500 (Reuters, Longterm Wiki). The machines generate; the humans still hold the veto. As I put it in the Medallion post: the model proposes; the desk disposes.
Historical precedent: the spreadsheet error that moved nations
In 2010, Reinhart and Rogoff’s paper arguing that debt above 90% of GDP crushes growth became gospel for austerity policy across Europe. In 2013 a graduate student, Thomas Herndon, requested their spreadsheet and found an Excel error — a formula range that averaged rows 30–44 instead of 30–49, silently excluding five countries (Australia, Austria, Belgium, Canada, Denmark) and flipping the headline result from −0.1% growth to +2.2% (BBC, Salon). The arithmetic in every cell was perfect. Excel computed every sum correctly. The experiment was wrong, and because the output was a real computed number in an authoritative format, it survived unquestioned and steered austerity across a continent.
That is laundered alpha, thirteen years early. The tool computed flawlessly; the flaw was upstream, in what was fed to it. An executed LLM backtest on a memorized, over-searched, crowded strategy is the Reinhart-Rogoff spreadsheet with a nicer font. Analogy: trusting a strategy because “the code ran” is like trusting a weighing scale that’s perfectly calibrated but sitting on a slope — every reading is precise, and every reading is wrong.
The Laundered-Alpha Litmus Test
My July 1 post gave you the Look-Ahead Litmus Test for stock-picking. Here is the strategy-design companion — five questions, and if you can’t answer all five, the executed backtest is theater:
1. WINDOW Is the entire backtest period AFTER the model's training cutoff?
(If no → Channel 2 memorization. The pass looks real; it's recall.)
2. TRIALS How many configurations were generated before this one was shown?
(Unknown or >1 unreported → Channel 3. Apply Deflated Sharpe.)
3. NOVELTY Would 1,000 other users get a materially different strategy from
the same prompt? (No → Channel 4 crowding.)
4. SEMANTICS Does the executed code implement the EXACT strategy specified?
(Re-read the code, not the chart → QuantCode-Bench failure mode.)
5. EVALUATOR Is the target a MECHANICAL objective (execution/portfolio, crisp
evaluator) or a FORECAST (noisy evaluator)? Trust the former only.Here’s a minimal check I actually run for question 4 — verifying the code does what the prose claimed:
# Did the model implement a CROSS or a LEVEL? These are different strategies.
import pandas as pd
def rsi_cross_below(rsi: pd.Series, level: float = 30) -> pd.Series:
"""Enter only on the bar RSI crosses DOWN through `level`."""
return (rsi < level) & (rsi.shift(1) >= level)
def rsi_level_below(rsi: pd.Series, level: float = 30) -> pd.Series:
"""Enter on EVERY bar RSI is below `level` (a different, often flattering rule)."""
return rsi < level
# If the LLM's 'cross' strategy actually used rsi_level_below, its trade count,
# exposure, and Sharpe are all measuring a strategy you never asked for.So what? Three dated predictions
Applying second-order thinking to where this goes:
- By Q2 2027, “How many strategy configurations were generated and tested?” becomes a standard institutional due-diligence question — the Deflated-Sharpe trial count moves from academic footnote to allocator checklist, because laundered alpha is invisible without it. (I made the DDQ prediction in the Medallion post; I’m now dating it and naming the specific question.)
- By end-2027, the first peer-reviewed paper quantifying “prompt crowding” — a measured correlation coefficient across strategies independent users derive from the same frontier model — will be published. AlphaAgent supplies the mechanism (arXiv:2502.16789); the number is white space today.
- The winning architecture will not be “LLM designs strategy.” It will be “LLM optimizes execution; human owns the thesis.” MadEvolve already draws the boundary empirically (arXiv:2605.23007); the firms that respect it will compound, and the ones that let the model forecast will donate their capital to those that don’t.
Alternative perspectives (steelmanning the optimist)
“You’re describing 2026 models. Retrieval-augmented, tool-using agents with live post-cutoff data close Channel 2.” Partly true — LiveTradeBench and StockBench are built exactly to be leakage-free, and still most models don’t beat buy-and-hold (arXiv:2511.03628, arXiv:2510.02209). Removing memorization exposes the absence of edge; it doesn’t create one.
“MadEvolve proves LLMs can genuinely design profitable systems.” Yes — on the mechanical subproblems with crisp evaluators, and the authors themselves caveat that results “do not directly transfer to live trading” (arXiv:2605.23007). That’s my point, not a counter to it.
“Frontier models will keep improving.” They will — and the Anti-Alpha Paradox says a better model has memorized more history, making laundered alpha more convincing, not less. Capability and contamination remain the same axis.
FAQ
Can LLMs actually run a backtest now, or is the number fake?
They genuinely run it. ChatGPT, Claude and Gemini execute real sandboxed Python (OpenAI, Anthropic, Google). The Sharpe is computed. The danger is that a real number can still describe a contaminated experiment — laundered alpha.
What’s the single fastest tell that an LLM backtest is worthless?
Check the backtest window against the model’s training cutoff. If any of the test period predates the cutoff, you’re likely measuring memorization, not skill (arXiv:2504.14765).
Where do LLM-designed systems actually add value?
The mechanical layer — execution, portfolio construction, risk — where the evaluator is crisp. MadEvolve shows +0.62 to +1.83 out-of-sample Sharpe there (arXiv:2605.23007), while forecasting overfits.
Did any LLM make money trading live?
In Alpha Arena Season 1, only 2 of 6 finished positive (Qwen3 Max +22.32%, DeepSeek +4.89%); GPT-5 lost 62.66% (iWeaver). Real execution, mostly real losses.
Is this regulated?
The SEC withdrew its 2023 predictive-analytics proposal in June 2025 (SEC) but named “AI technologies and trading algorithms” in its 2026 exam priorities. Keep an auditable prompt/output trail.
I’m Jiri Pik — founder of RocketEdge, AI-cloud architect and quant trader in Singapore. This is Part 3 of my LLM-and-markets trilogy, after look-ahead bias in stock-picking and the Medallion Test. I write about what I build, usually before it hits mainstream.
Disclaimer: This reflects my personal views and experience, not financial advice. Past performance doesn’t guarantee future results. Nothing here promises any return.