It is often said that "beating the market" is incredibly difficult and that even most professional investors are unable to do it consistently.
At the same time, we regularly hear stories of legendary investors who were able to beat the market successfully over many decades. This includes Warren Buffett, Peter Lynch, and many others.
But is it really true that professional investors usually fail to beat the market?
And does it mean that regular investors shouldn't even try, and just put their money in an index fund?
Before we get to the research, let's define what it really means to "beat the market."
What does it mean to beat the market?
"Beating the market" means getting higher investment returns than the S&P500 stock index.
The S&P500 index is an index of 500 large-cap stocks in the US and is the most commonly used benchmark of overall stock market performance.
Historically, it has had average returns of 8–10% per year, which is very high.
Investors with higher percentage returns than the S&P500 index are said to beat the market. Those who have lower returns are said to underperform the market.
Investing in the has become very popular in recent years. This is a popular approach for regular investors to earn similar returns as "the market."
You won't beat the market with this approach, but at least you won't perform much worse than the market as a whole.
Investing in indexes is also referred to as passive investing, as opposed to active investing via stock picking or market timing.
Research: 89% of fund managers fail to beat the market
S&P Dow Jones Indices regularly researches how actively managed mutual funds perform compared to the S&P500 index.
These are funds that actively buy and sell assets and are managed by professionals, often with very high salaries from the management fees.
Their last report was published in April 2020 and included data for the full year 2019.
According to this report, 88.99% of large-cap US funds have underperformed the S&P500 index over ten years.
Source: S&P Dow Jones Indices
As a whole, 78–97% of actively managed stock funds failed to beat the indexes they were benchmarked against over ten years.
In addition, all professional fund investing styles underperformed the market — large caps, mid-caps, small-caps, all-caps, value, growth, etc.
The longer the funds are measured for, the greater the likelihood of them underperforming their benchmark indices.
It is relatively common to beat the market for 1–3 years at a time. That can largely be explained by luck.
But the data clearly shows that even professional fund managers are unable to beat the market consistently over a longer period of time, like 10–15 years.
Most hedge funds also underperform the market
Hedge funds are investment funds that often use complicated strategies to achieve better returns than the market.
Contrary to popular belief, most hedge funds actually perform worse than the market, on average — far worse.
In 2008, Warren Buffett made a $1 million bet that hedge funds would fail to beat the market over a multi-year period.
In 2016, the hedge funds had returned 22.04% on average while the S&P500 had returned 85.4%, almost four times as much.
Source: Berkshire Hathaway Shareholder Letter
Part of the reason for this is that hedge funds have very high fees. It's common for them to charge a 2% annual management fee, plus 20% of profits.
Because of these high fees, hedge funds are mostly useful for making their owners and managers rich. Most of them drastically underperform the market.
Regular investors have some advantages over professionals
It is clear from the statistics that beating the market is incredibly hard. Even most professional investors are unable to do it.
Because of this, it seems logical that most regular investors would also be unable to beat the market over the long-term.
Although this is true, regular investors also have some surprising advantages that can possibly give them a slight edge over the professionals.
1. No management fees
A big part of why professionally managed funds and hedge funds underperform is the high fees they charge.
Even if they were able to beat the market slightly, they end up underperforming the S&P500 when the fees have been subtracted from the returns.
A regular investor does not need to pay management fees, only trading commissions and taxes.
But many brokers offer commission-free trading these days, and it is possible to reduce the taxes by investing in a tax-advantaged account like a 401(k).
2. No career risk
Most professional funds take a cut of the total amount of money that they manage, often in the range of 1–2%.
Therefore, these funds are incentivized to maximize their total assets under management.
Maximizing returns is not as important, especially not if it means taking risks that could cause clients to withdraw their money from the funds.
If they try to beat the market by taking risks, the chances are high that they will end up drastically underperforming the market for some quarterly or annual periods.
When that happens, investors are highly likely to pull their money out of the fund, causing the fund manager to lose money or even get fired.
This is called "career risk." Fund managers need to worry about the safety of their careers when deciding what to invest in.
One consequence of this is that many professionally managed funds end up becoming "closet indexers" — they invest in a lot of the same companies that are in their benchmarks, so they end up mostly tracking their benchmarks.
Regular investors don't have to worry about this. They can stick with high-conviction bets without having to worry about getting fired.
This is important because good investments often underperform before they end up outperforming.
3. Smaller size
The more money you have, the harder it will be to beat the market.
As a small investor, no one is keeping track of what you are buying or selling. And the amounts you are trading are way too small to move the prices of the stocks.
On the other hand, a big fund that starts buying stock will often cause the price to move up because the demand for the stock then outweighs the supply. When a big fund starts selling, it can cause the price to go down.
Because of this, the sizes of big funds cause them to have reduced performance.
They get worse prices when buying because it causes the price to go up, and they get worse prices when selling because it causes the price to go down.
Because most regular investors have very small portfolios compared to big funds, this gives them a size advantage. They can buy and sell at better prices.
4. More varied investment options
Having a lot of money under management limits the investment options.
Someone with tens of billions of dollars won't be investing in small-cap stocks, for example.
That's because the returns are unlikely to move the needle for the fund as a whole, not to mention the effects of the fund's buying or selling on the price of a small-cap stock.
In addition, many funds have strict mandates about what they can and can not invest in. They are also forced to remain mostly invested at all times.
If a fund sells everything and goes to cash for an extended period, then the clients are likely to pull their money out.
Regular investors don't have these restraints. They can invest in small- and mid-cap stocks, and even buy different types of investments if they can't find any stocks that look attractive.
Regular investors can even go part- or all-cash if they think the risk of staying invested outweighs the potential rewards (although trying to time the market in this way usually fails).
Why stock picking can still be a good idea
Passive investing in index funds may be the best approach for regular people who aren't that interested in the stock market but simply want to build enough wealth to retire comfortably one day.
But for people who have a passionate interest in stocks and investing, there is absolutely nothing wrong with picking stocks.
Stock picking is fun, and it can lead to immense rewards if you pick a few stocks that end up performing really well.
For example, if you had invested even a small percentage of your portfolio in stocks like Apple (AAPL) or Amazon (AMZN) a decade ago, you would have made a lot of money.
Thinking about a stock as representing part ownership of a company is a good idea. Don't just buy it because you think it will go up, buy it because you believe that the company is going to do well in the future.
If you buy solid companies with good future prospects at fair prices, then you are likely to make money from your stock picks over time.
If you want to hedge your bets, then you could put 50% or even 90% of your stock portfolio in an S&P500 index fund, but then use the rest to pick individual stocks.
Then, if you end up underperforming the market, at least it won't be by as large of a margin because you had a big chunk of your money in an index fund.
