The data you don’t see is just as important as the data you do.
Survivorship bias is when you aren’t working with all of the information needed to make a complete analysis. We tend to focus on the information we have and mistakenly forget to consider the information we don’t have.
Missing Data
During WWII Hungarian mathematician Abraham Wald helped the US military determine where to add reinforcing armor on bomber planes. If you reinforce the whole plane it’s too heavy so you want to only add weight where absolutely needed. The military collected data from returning planes on where they had taken damage (from bullets, shrapnel, etc). From this they created scatter plots on plane diagrams showing where the damage tended to be. The initial military analysis was to reinforce the heaviest hit areas but Wald realized this was survivorship bias.
The military was only accounting for the planes that made it back and weren’t accounting for the planes that were shot down and never returned. The areas a plane could get shot, but still return, must not need additional armor to fly. Therefor the areas on returning planes with no damage (the cockpit, the engines, etc) must be the places needing reinforcement since the planes that never returned were probably hit in those places. The military had worked with the data they had but they forgot to account for the planes that never made it back, the data that was missing.
The Value of Failure
We tend to over-appreciate success stories and under-appreciate failures. Success stories are easy to find while failures are usually ignored or lost to time. Survivorship bias comes up frequently in think-pieces about successful people, businesses, investments, etc. The focus tends to be on the “winners” but rarely on the “losers.”
While successful people can give advice on what to do, people who failed can give advice on what not to do (which is just as valuable). Successful people giving advice is only one part of the data, it’s survivorship bias because we’re only hearing from the ones who “made it” and not the ones who didn’t. You hear how Steve Jobs, Mark Zuckerberg, and Bill Gates successfully went from college dropouts to billionaires, but there aren’t many stories on the majority of college dropouts who don’t become billionaires.
In the Red
In the world of business we mostly hear from the businesses that are successfully still around, and not from the ones who closed. Most new businesses, around 90%, will fail but we rarely get advice from them after they do. Instead we hear inspirational stories about the very small percentage of scrappy startups that were incredibly lucky who went from operating in garages to being juggernauts, such as Amazon.
Investments are similar. Funds that are losers are only allowed to lose for so long. When an investment company closes a fund the fund ceases to exist and no longer drags down the company’s overall performance. By removing/hiding the failures you can get a false overall sense of positive performance. This also means that the funds available to invest in are either proven winners or brand new funds, never any long-time losers.
Here Today, Gone Tomorrow
We can see survivorship bias in architecture and anthropology. The best built and/or most appreciated buildings of the past continue to stand, while the weak or unwanted buildings are brought down. This can lead to a false sense that all buildings of the past were stronger or more beautiful than today’s, but there were plenty of weak and/or ugly buildings in the past just like today. Ancient cultures had lots of buildings that were torn down or fell down over time, but we talk about the pyramids of the world because they’re a great shape to arrange stone that won’t fall over and are still standing today (no aliens needed).
When studying ancient cultures, it’s easy to account for cultures who built permanent structures with durable materials that have survived to be studied. But cultures that utilized temporary structures, moveable structures, or buildings made from biodegradable materials are harder to document. We have to rely on other clues to understand these groups and account for them in history.
When I Was Younger
Survivorship bias can also apply to things that are more subjective. It’s easier to remember the good art than all the bad art that got thrown away. People make statements that music / TV / movies were better in some previous time period than today, but they frequently forget all of the bad music / TV / movies of that previous time period. It’s a survivorship biased rose-tinted view of the past.
Today’s music is made up of new songs, both good and bad. When playing songs from the past however radio stations / channels tend to only play the successful hit songs and skip the bad songs, adding to the survivorship bias. This means it’s easier to remember the hits songs of the past that are still played than it is to remember the songs you never wanted to listen to. Even when you’re choosing the music to listen to you tend to pick the songs you want to hear and skip all of the songs you don’t. Sure Nirvana’s generational anthem Smells Like Teen Spirit came out in 1991, but so did Tom Cochrane’s Life Is A Highway (a song which is straight-up trash).
In analyzing a situation, thinking about the secrets of success, or flashing-back to a past that never really existed, remember to factor in all of the data you are forgetting.
