Why I'm sharing it
I'm studying how people build AI tools for investing, and I read a lot of results posts. Most of them show a chart that only goes up.
One write-up stood out because the author was honest about it. A retired auto dealer built an AI stock-picking system and shared everything: the replay on past months, the forward paper trading and the flaws they found in their own work. The replay looked far better than the forward results. Their own list of flaws explained most of the gap.
That honesty is the lesson. Before anyone believes a backtest, theirs or mine, it should go through the same questions.
What makes it work
It checks when the rules were written. Rules tuned after seeing the data will always look good on that data.
It asks for a fair benchmark. A result only means something next to buying an S&P 500 index fund over the same dates.
It separates replay from forward results. Only trades made after the rules were locked show what the system can do.
It looks past the win rate. Winning six trades out of ten means little until you know how big the wins and the losses were.
It asks for the bugs. Data errors and trades placed before a signal existed quietly inflate a replay.
The prompt
Act as a skeptical reviewer of a trading system's results. I'll paste a results write-up or my own report: [paste it]. Go through it and answer each question plainly, saying "not shown" when the write-up doesn't say. 1. When were the rules and signals fixed, before or after the period they were tested on? 2. Which results are replay on past data and which are forward results made after the rules were locked? Give each separately. 3. What benchmark is used, and is it measured over exactly the same dates? If none, compare to an S&P 500 index fund over the same window. 4. Besides the win rate, what are the average win, the average loss and the worst losing streak? 5. Are costs, fees and slippage included? 6. Are the test windows overlapping, so fewer results are truly independent than it looks? 7. List any data errors, bugs or trades placed before a signal existed, and how each could change the result. Finish with three things that would have to be true before I'd believe the forward results, and what I should track next to find out.
How to use it
Paste the whole write-up, not the summary. The flaws are usually near the bottom.
Run it on your own results too, every time. It's easy to be skeptical of other people's charts and gentle with your own.
If the answer to the first question is "after", treat every replay number as a story, not a result.
This post is about reading results, not about what to buy. It isn't financial advice, and nothing here is a recommendation to buy, sell or trade anything.
Inspired by a retired auto dealer's public write-up of their own AI stock-picking experiment, including the flaws they found in it. The author wasn't named in the copy I saved, so if it's yours, tell me and I'll credit you. The questions here are my own.