Person one: ‘Listen to this — 40% of people support the Prime Minister and 50% support the Opposition.’
Person two: ‘That’s only 90%.’
Person one: ‘Yeah, there was one guy who said the samples weren’t big enough to be statistically significant.’
Good old statistics. They can be incredibly valuable, or utterly useless.
The opening quote is from an ABC comedy series called The Dingo Report. The show briefly appeared on our screens in 1987. It offers a classic example of statistical misuse.
Statistics have long been a source of controversy. While numbers never lie, the people compiling them sometimes do. This means you need to think carefully about the stats you read.
The truth is statistics can make almost anything sound good. You just need to pick a suitable sample group and ask the right questions.
Take toothpaste for instance. In 2007, the British regulator banned a manufacturer from saying more than 80% of dentists recommend its brand.
Yes, the statistic was technically correct. The maker just didn’t mention the dentists surveyed could endorse more than one product. It turns out a competing brand was almost as popular.
Statistics can be tricky indeed. As Mark Twain famously wrote ‘There are three kinds of lies: lies, damned lies, and statistics.’
Trading is another area that uses statistics. These are vital to understanding a strategy. But you need to think critically about the figures you read. Things aren’t always as they seem.
Smoke and mirrors
Let me put you to the test.
I’m going to show you some performance stats for a trading system. This strategy only trades a portfolio of 10 blue chip stocks. Take a look at its performance…
This is an impressive chart. Look how it steadily racks up the profits over the three years from 1 July 2012. Better still, you can trade every signal with just $10,000 in capital.
The system’s total gain was $6,000 — that’s about 20% per annum. Then there are dividends to consider. And there are lots of them from these big name companies.
And the best bit….
The system had a 100% strike rate during the test period. Not a single trade lost money. This appears to be a high profit/low risk strategy. The statistics say this system is a serious winner.
So what do you think…would you be tempted to have a go?
If you’re like me, your scam detector will be going off.
Take the test period for instance. It runs from 1 July 2012 to 1 July 2015. Many people think three years is a long time. But in reality, it’s just a snapshot.
I always wonder what was happening before the start date. If the system is as good as it looks, then it should work at other times. There shouldn’t be anything special about a test period.
Let’s see what happens if we start the test five years earlier…
The chart doesn’t look so flash now…nor do the statistics.
Forget the 20% per annum return — this back-test barely breaks even. The 100% win rate is also gone. Only 40% of trades make money over the longer period.
Sure, the three years from July 2012 were good. But the five years before that were lousy. Selecting a favourable start date gave the statistics a positive bias.
But that’s not the only trick. You should also be wary of statistics that use a small sample group — in this case 10 stocks.
You see, it’s easy to tailor a system to suit the past data of a small group. A select group of blue chips may sound good. But it’s open to manipulation — especially over a short timeframe.
It’s a bit like asking 10 dentists, that you know use brand A, what toothpaste they recommend. The deck is stacked. Yes, the stats will look good. But they probably won’t stack up in real life.
The true test of a system is to open it up to many stocks. You see, a robust strategy will work across hundreds of different companies — not just a portfolio of handpicked blue chips.
Let’s see how this system handles the entire ASX population…
It’s an epic fail. Not only has the system lost money, its win rate is just 27%.
This is exactly the same system that returned 20% per annum with 100% accuracy. The only difference is I’ve included many more stocks and increased the test period.
Back-testing can give us some valuable statistics. But you need to have your eyes open. Some stats aren’t worth the time of day.
Back-testing meets reality
System traders do a lot of back-testing. It’s our key tool for assessing a strategy before committing real money. Robust back-testing can help avoid mediocre systems.
But you have to be aware of a few traps.
I’ll only consider a system that’s been back-tested over many years. It needs to show consistent performance over a long period. That tells me I’m not just looking at a lucky year or two.
The other factor I look for is the number of trades. You see, a good strategy will work many times over hundreds of different stocks. Its success doesn’t hinge on a handful of trades.
Go back and look at today’s first example. You’ll see it was over a relatively short period and only included 10 stocks. This is exactly the sort of back-test to avoid.
Quant Trader’s back-testing goes a lot further. It stretches back to 1993 (the extent of my data) and includes practically every ASX company. This is the way back-testing should be done.
Have a look at the next chart.
This is Quant Trader’s performance chart going back to 1993. It assumes $1,000 on each signal and doesn’t allow for costs and dividends. Both long and short trades are included.
The chart is largely due to back-testing. It uses historic data to see what would have happened if we were trading in the past. The signals are mostly hypothetical. No one was actually trading them.
Now have a look at the red circle at the top of the chart. This is the period of live signals. It’s where back-testing meets reality. These signals did occur — they’re the real deal.
The last 12 months show a familiar pattern. Performance is consistent with back-testing. That’s what we want to see — theory and real life becoming one.
Good back-testing can be a guide to the future. It can help determine if a strategy sits on solid footing or sand. This sort of information is invaluable.
I’ve seen plenty of systems that look good in testing, but fall apart in live trading. Often the problem comes down to dubious stats — like the ones I showed you earlier.
Quant Trader hasn’t had this issue. Real life is treading a similar path to the back-testing. That’s the power of good statistical analysis.
Until next week,
Editor’s note: If you are concerned about the recent market volatility, here’s one thing you should do…
Check out Jason McIntosh’s Quant Trader advisory service. It’s a fully algorithmic trading system for ASX stocks. Quant Trader scans practically every company. I can just about guarantee you’ve never heard of many of these unique ideas.
If you’re not familiar with Jason, he was a trader at one of the world’s most powerful investment banks for nearly a decade. He’s seen dozens of market corrections and crashes. This experience is hard coded into Quant Trader’s algorithms. And the results speak for themselves.
So if you are at all worried about the markets…if you’re not sure when to buy or sell…I strongly suggest you look into Quant Trader now.
Try it. See if it makes sense to you. It could change the way you trade forever.