Six ways investors can guard against misleading data

Something we very much pride ourselves on at rockwealth is that our investment philosophy is strictly evidence-based. But what do we mean by evidence-based investing? After all, “evidence” can be used to claim almost anything.

So what sort of evidence specifically are we talking about? In other words, what characteristics does a piece of evidence, new or otherwise, need to have to be considered worthy of consideration?

There are four main things we look for. First, the evidence has to be genuinely independent. Unfortunately, much of the evidence cited in support of different investment strategies is either produced, or at least commissioned and paid for, by companies with a commercial interest in publicising it.

Secondly, it needs to have been peer-reviewed. The peer review process that serious academic journals insist on isn’t fool-proof, but it does make the evidence in question more reliable.

Thirdly, the results of any academic research need to have been time-tested. So, for instance, a strategy may have worked over a particular time period, but has it been tested across multiple environments and timeframes?

For us, however, there is a fourth characteristic that evidence absolutely needs to have to merit being taken seriously: it must be based on robust data analysis.

Data can be highly misleading

You’ve probably heard the expression, “lies, damned lies and statistics”. Well, it certainly applies to investing.

Investors are bombarded with statistics. Never before has so much data been so readily accessible. But if you don’t have a basic grasp of statistics, or at least a financial adviser who does, there is a grave danger of being misled and, worse still, of making costly mistakes.

Note, we’re not talking here about mathematics. Yes, having an understanding of key mathematical concepts, or at least not being frightened of numbers, is advantageous for investors. But statistics is a subject in its own right — a branch of applied science that focuses on the collection, sorting, and representation of data. We ignore the lessons it teaches us at our peril.

Here are six things you can do to help ensure your investment strategy is built on statistical evidence that’s sound and reliable, and not misleading.

1. Understand statistical significance

A vital concept for investors to understand is statistical significance — in other words, how much importance should you attach to a particular set of data? Does the evidence provide useful information, or is it just random noise?

Short-term movements in the financial markets are totally unpredictable; they tell us little or nothing about future returns. Yes, there is value in looking at returns over very long periods of time, but history doesn’t repeat itself exactly; what happened in the past is only one of many possible future outcomes.

One of the reasons why most investors, including professional ones, achieve suboptimal returns is that they are far too focussed on specific time periods that are far too short. The danger is that they assume that just because a particular strategy has worked over, say, the last ten years, it will inevitably work over the next ten years as well.

Understanding statistical significance will help you to avoid falling into the same trap.

2. Insist on rolling time periods

Because we at rockwealth believe so strongly in the importance of statistical significance, we insist on using so-called rolling time periods — in other words, overlapping intervals of a fixed length within a larger dataset. For example, in a dataset spanning 50 years, a rolling ten-year period analysis would involve examining ten-year periods that start at each year in the dataset (e.g., 1970-1980, 1971-1981, 1972-1982 and so on).

Rolling time periods have many benefits. They reduce the impact of start- and end-point biases. They help to smooth out short-term volatility and provide a clearer picture of long-term returns. Crucially, they allow for a better assessment of risk over time and for better comparison between different asset classes.

So, avoid drawing conclusions from a single snapshot. Always insist on viewing historical performance through the lens of rolling time periods.

3. Be sceptical of back tests

It’s very common for investment professionals to use what are called back tests. Back-testing refers to the process of testing a strategy using historical data. The goal is to see how the strategy would have performed in the past and to gain insights into its potential effectiveness in the future.

But back tests are fraught with danger. For example, they often capture noise rather than underlying trends. Many of them fail to account for the transaction costs and taxes the strategy in question entails.

Those who rely on back tests also tend to forget that financial markets are influenced by factors — economic conditions, regulations and investor behaviour, for example — that are constantly changing. A strategy that worked well in the past may not be effective in the future, because of these evolving factors.

For all of these reasons, then, investors should view back testa with a healthy dose of scepticism, and steer clear of advisers who rely heavily on them.

4. Beware of survivorship bias

One of the biggest mistakes investors make when analysing different funds is that they fail to consider survivorship bias.

This refers to the distortion that arises when only the performance of currently existing funds is considered, while those that have been closed or merged due to poor performance are excluded from the analysis. This can lead to an overestimation of the success and returns of active funds because the less successful funds are not accounted for.

Using data that accounts for survivorship bias, like the ongoing SPIVA analysis from S&P Dow Jones Indices, for example, helps investors to make more informed assessments of fund performance and ultimately better decisions.

5. Don’t confuse luck with skill

Another common error investors make is that they confuse luck with investment skill. So, for instance, they’ll make money on a stock or cryptocurrency, and then jump to the conclusion that they’re savvy investors and are likely to enjoy similar success in the future. In reality, they were probably just lucky.

In much the same way, investors assume that an active fund manager who has outperformed in the past will continue outperfoming in the future. But, at any one time, there are so many funds to choose from that there will always be funds that have recently outperformed, and the proportion of funds that do so is entirely consistent with chance alone.

It takes many years of performance data to be confident that a fund manager is genuinely skilful. A 2002 study found that it takes eight years of outperformance for a test of a manager’s skill to have 50% power and 22 years of data to have 90% power. In other words, even if manager has outperformed over a 22-year period, there is still a 10% chance that it was simply down to luck.

Remember: assuming that a positive outcome is down to skill is an all-too-human trait. Never underestimate the role of random chance.

6. Find an adviser who understands statistics

Perhaps you’ve read the previous five suggestions and they’ve all seemed pretty obvious. If that’s the case, you may well have the statistical sophistication required to handle your investments on your own.

In our experience, however, most people don’t have it, and even those who do are still tempted to act on their emotions and make irrational decisions from time to time.

That’s the reason why, in our view, virtually everyone will benefit from having a professional adviser to help them focus on their long-term goals and remind them of what the academic evidence tells us whenever they’re tempted to change course.

So, if you don’t already have one, it’s time to look for one. Just ensure it’s someone you really feel you can trust and that their investment philosophy is based on independent, peer-reviewed and time-tested evidence. Crucially, do they know enough about statistics to guard against misleading data?

Image: The illustration shows a one-sample t-test equation. T-tests are used to measure statistical significance. In the context of active fund management, a t-test can provide insights into whether outperformance by a particular manager is likely due to chance or manager skill.