Backtest on today’s biggest stocks and you double your return

What we found

Holding today’s 50 largest US companies since September 2016 returned 31.0% a year. The 50 that were actually the largest in September 2016 returned 14.3% a year, and SPY returned 15.4%. Over five years the gap was 28.4% against 14.2%.

Picking today’s winners builds their success into the test. In 2016 NVIDIA was only the 110th-largest US company and AMD the 505th, so no one choosing the 50 largest then would have held either.

The full results

Why it happens

A list of today’s largest companies is a list of the companies that did best. Starting a backtest from it quietly assumes you knew in 2016 which companies would win the next ten years. NVIDIA, Broadcom, Tesla and Palantir are in today’s 50 because of the returns the backtest then credits them with.

The 21 companies that fell out of the 50 since 2016, such as AT&T, Verizon, Pfizer and Altria, did the opposite: an honest test would have held them, and a test built on today’s list never does.

How to avoid it

Choose the universe as it stood at the start of the test, not as it stands now. When that isn’t possible, compare a strategy with simply holding the same universe rather than with SPY: holding today’s 50 largest beats SPY by itself, so a strategy on them that “beats the market” may have added nothing.

The QuantLabs backtester is built to avoid it. A screener universe is the largest companies as of the start date, ranked by the market cap known then, never today’s; the screener’s own filters are read as of each rebalance too. Every result shows simply buying and holding that same list beside the strategy and the benchmark, so a strategy can’t look good just because its stocks did. The one bias left is that companies which have since delisted are missing from the data, and each result says so.

How it was measured

• Equal weight at the start, then held, on total returns (dividends reinvested).
• The 2016 list uses point-in-time market caps: share counts from SEC filings as of that date, times that day’s price.
• One share class per company; US 10-K filers only.

Caveats

• Four of today’s 50 weren’t listed in 2016 (PLTR, SNDK, GEV, CRWD), so today’s line holds 46.
• The 2016 list leaves out companies acquired or delisted since (such as Celgene, Allergan, Walgreens and Time Warner), and Disney, whose SEC identifier changed in 2019. Most of those did worse than average, so the 2016 return is if anything flattered.

Questions

What is survivorship bias in backtesting?

Testing a strategy only on companies that exist or lead today. Those companies are known winners, so the test credits the strategy with returns no one could have picked at the time.

How big is survivorship bias?

Large. Holding today’s 50 largest US companies since September 2016 returned 31.0% a year; the 50 that were actually largest in 2016 returned 14.3%, against 15.4% for SPY.

How do you avoid it?

Pick the universe as it stood at the start of the test, using point-in-time data, and compare a strategy with simply holding the same universe rather than with an index.