Archive / Check an AI stock-picking backtest before you believe it

Check an AI stock-picking backtest before you believe it

A prompt that reads a trading system's results like a skeptic. It asks whether the rules were set before the data, whether the benchmark is fair and whether the forward results tell the same story.

Nothing here is financial advice or a recommendation to buy, sell or trade anything. Do your own research.

For: Solo founders, New to AI

Topics: Investment

Why I'm sharing it

I'm studying how people build AI tools for investing, and I read a lot of results posts. Most of them show a chart that only goes up.

One write-up stood out because the author was honest about it. A retired auto dealer built an AI stock-picking system and shared everything: the replay on past months, the forward paper trading and the flaws they found in their own work. The replay looked far better than the forward results. Their own list of flaws explained most of the gap.

That honesty is the lesson. Before anyone believes a backtest, theirs or mine, it should go through the same questions.

What makes it work

It checks when the rules were written. Rules tuned after seeing the data will always look good on that data.

It asks for a fair benchmark. A result only means something next to buying an S&P 500 index fund over the same dates.

It separates replay from forward results. Only trades made after the rules were locked show what the system can do.

It looks past the win rate. Winning six trades out of ten means little until you know how big the wins and the losses were.

It asks for the bugs. Data errors and trades placed before a signal existed quietly inflate a replay.

The prompt

Prompt

Act as a skeptical reviewer of a trading system's results. I'll paste a results write-up or my own report: [paste it]. Go through it and answer each question plainly, saying "not shown" when the write-up doesn't say. 1. When were the rules and signals fixed, before or after the period they were tested on? 2. Which results are replay on past data and which are forward results made after the rules were locked? Give each separately. 3. What benchmark is used, and is it measured over exactly the same dates? If none, compare to an S&P 500 index fund over the same window. 4. Besides the win rate, what are the average win, the average loss and the worst losing streak? 5. Are costs, fees and slippage included? 6. Are the test windows overlapping, so fewer results are truly independent than it looks? 7. List any data errors, bugs or trades placed before a signal existed, and how each could change the result. Finish with three things that would have to be true before I'd believe the forward results, and what I should track next to find out.

How to use it

Paste the whole write-up, not the summary. The flaws are usually near the bottom.

Run it on your own results too, every time. It's easy to be skeptical of other people's charts and gentle with your own.

If the answer to the first question is "after", treat every replay number as a story, not a result.

This post is about reading results, not about what to buy. It isn't financial advice, and nothing here is a recommendation to buy, sell or trade anything.

Inspired by a retired auto dealer's public write-up of their own AI stock-picking experiment, including the flaws they found in it. The author wasn't named in the copy I saved, so if it's yours, tell me and I'll credit you. The questions here are my own.

Use it for: Reading any AI trading or stock-picking result, yours or someone else's, before you trust it with money.

Back to the archive