[00:00:00.490] – David Maples

Are you sure you’re looking at your data the right way?

 

[00:00:03.330] – Inigo Montoya

You keep using that word. I do not think it means what you think it means.

 

[00:00:08.630] – David Maples

Survivorship bias, bombers, and why they don’t make music like they used to. All this and more on this episode of The Buck Stops Here.

 

[00:00:25.350] – David Maples

Hello, and welcome back to another episode of The Buck Stops Here business podcast. I’m your host, David Maples. I’m excited to welcome you to season two of the podcast. A whole bunch of new stuff is going to be coming this season.

 

[00:00:37.920] – David Maples

We’re going to have a lot more people we’re interviewing for different activities. My production team has upped the ante this season, and they’re going to be adding sound effects and other things to make your listening enjoyment more enjoyable.

 

[00:00:52.530] – Producer

Level up.

 

[00:00:53.940] – David Maples

Just a word from our sponsor; this episode has been brought to you by Catapult Creative Media. Catapult is a full-service digital marketing agency dedicated to providing unmatched marketing and advertising results coupled with data. They are an agency that has some proprietary software and systems that can provide you data reporting that’s not available from any other agency in the United States. So, they chose to sponsor this episode, in particular, today because we’re going to be talking about data. That’s right. The thing everybody likes to talk about on a Friday night is data.

 

[00:01:27.100] – David Maples

There’s no better way to pick up somebody at a bar than to say, “Hey, I couldn’t help but notice your data.” Well, the truth about it is data, for most people, is kind of boring. So, we’re going to try to make this kind of interesting.

 

[00:01:41.010] – Data

Yes, I hate this!

 

[00:01:43.200] – David Maples

Today we’re going to talk about one of the major problems in data collection and how people do things, which is called survivorship bias. So, you may say to yourself, what is that? Well, here’s an example. In World War II, the United States of America had bomber aircraft that were flying out over enemy territory, and they would return with all these holes shot up in the aircraft. And the army obviously knew- The military obviously knew that they could reduce the number of bombers lost in combat if they were actually able to add armor to the aircraft.

 

[00:02:24.070] – David Maples

The problem with adding a whole bunch of metal, I mean, if everything was a metal flying bathtub, no one would ever get shot down. The problem with that is that it turns out bathtubs don’t have really good flying characteristics. And if you put too much armor on an aircraft, it will cease to fly. You can’t get off the ground, right? So, what they did was they looked at what was coming back, the aircraft is coming back, and they said, man, there’s a whole bunch of holes on these aircraft.

 

[00:02:49.210] – David Maples

I mean, they look like Swiss cheese sometimes. And they found a whole bunch of holes under the wings and the tail section, et cetera. And the military came to the logical conclusion, we need to put armor on those bombers in order to increase their survivability in combat.

 

[00:03:09.150] – David Maples

And this information was passed along to a Hungarian statistician named Abraham Wald, and he said that is completely wrong. And the thing is, this is not designed to be- it’s not designed to be a trick question. It’s like, obviously there are holes in the aircraft. Let’s put armor on these places we got all these holes. He’s like, no. What we have is a survivorship problem; is that we’re only looking on the aircraft that came back.

 

[00:03:36.830] – David Maples

The aircraft that came back had holes on the wings and the tail section, but they didn’t have a lot of damage over the engines. He said, that’s where we need to put the armor. And he’s 100% correct. And in businesses every day, podcasts, business advisors, all kinds of things, we fall prey to this all the time. We kind of worship at this altar of its kind of hero worship.

 

[00:04:05.710] – David Maples

We only look at the survivors. We look at the winners and say, “Wow, this winner did this. Therefore we should do that to be successful. Right? Nobody wants to look at the losers like they didn’t win anything.”

 

[00:04:17.400] – David Maples

Why do we look at them? It happens a lot of times on kind of like local news stations will find the person who lives to be 110 years old. Yes, Ms. Jane, what was the thing you did to be 110?

 

[00:04:29.540] – David Maples

And she says, I’ve smoked cigarettes every day of my life. You look at people who are 110 years old, and you don’t see a lot of smoking cigarettes. Maybe she knows something we don’t. The reality of it is that’s an example, a prime example, of survivorship bias. We are only looking at the data set of the people who have survived, and it leads us to these wild, erroneous conclusions. I did certain things every day.

 

[00:04:57.680] – David Maples

I took cold showers every day of my life. And this is the classic thing. Correlation does not equal causation. Now, it’s not a bad example to look at the things that people did that maybe did help them… But the idea is like something like if you smoke cigarettes every day, the odds are you won’t live to be 110.

 

[00:05:23.410] – David Maples

There’s a whole bunch of data on this that suggest otherwise. And going into this stuff, you’ll hear some things that sound very similar to you if you’ve fallen victim to survivorship bias, and we all have. It’s a very human thing. Nobody really wants to look at why things fail, and this happens to all of us.

 

[00:05:39.570] – David Maples

So, they don’t make music like they used to. Sure. You are comparing the hits of today, of which there are thousands put out every week and released, and you’re comparing it to the 60 or so songs that were classic hits from the 1960s. And the problem is that you’re looking at an entire decade of which only 60 songs made it into that classic hits list, right? You’re looking at the very best examples of a decade’s worth of music, and you’re comparing it to what came out in a month in the year you’re in.

 

[00:06:14.650] – David Maples

Example of survivorship bias. They don’t build buildings like they used to. No, because the buildings that weren’t built really, really well got torn down. The Taj Mahal is a perfect example or the Great Pyramid of Giza. We should build everything like they built the Great Pyramid of Giza.

 

[00:06:29.170] – David Maples

There’s only one or two or three examples of something from that age that have held up, you know, over thousands and thousands of years. And the problem with that, again, survivorship bias. World War I, another example. They knew that people were dying from head injuries, so they gave everybody metal helmets.

 

[00:06:45.460] – David Maples

The number of head injuries went up 100-fold. It could lead you to the conclusion that metal helmets were causing more head injuries. No, it’s just people were surviving. And this is true. Survivorship biases; you’re looking at the ones who lived, and there’s a lot of examples of those things. We tend to do that.

 

[00:07:03.560] – David Maples

We tend to look at the companies that were successful, the ten companies that were successful, people who dropped out of college and set up – Bill Gates is a perfect example – dropped out of college, made- Zuckerberg dropped out of college, made this multi-billion dollar, one of the richest men on the planet, et cetera. But you’re not looking at the tens of thousands or hundreds of thousands or millions of people who failed following the same pathway, and that’s kind of a thing. So, one of the reasons Catapult was excited to sponsor this episode today is because they built a piece of analytics reporting software that allows you to track your successes and see kind of how that works. I mean, it’s great!

 

[00:07:40.550] – David Maples

A thousand people come to your website in a month, and 100 of them buy from you or ten buy from you, or whatever it is. Right? And that’s great to be able to track that and get more of those people. But their software also lets you see kind of what those user journeys were and the ones that were not successful. You know, you start looking at these characteristics, and you say, okay, our successful ones spent this much time on my site and did these things. And it results in X amount of company- x amount of money to my company.

 

[00:08:05.340] – David Maples

That’s incredibly valuable data. And they built a piece of really cool analytics software that lets you do that. But at the same time, you also need to look at that data set, and you need to say, how do we do that for the ones that weren’t successful? If they had these other characteristics, why do these other 200 that have the successful hall markers of our really successful things that make us all running, how do we increase those? And these you need to kind of push on both ends.

 

[00:08:31.570] – David Maples

You need to kind of look at all those things and see what can happen. So. To that end, today, what we’re going to give you is what we call the Quad-A method. At the Buck Stops Here business podcast. We have some different tools we like to let people know about. So, the Quad-A Method is to ask, acquire, analyze, and act.

 

[00:08:50.350] – David Maples

Ask, acquire, analyze, act. And if you follow this procedure, when you’re looking at your data, it can make you much, much, much more successful. And it sounds very simple because it really is. But every time you look at your data sets, you want to see those things and see what it looks like. So, the first thing you want to do whenever you’re looking at anything you want to know about, anything, because you don’t have any data, you start from zero.

 

[00:09:14.600] – David Maples

You don’t have any data. If you have data, we can do that. The other thing you can look at your data a little bit. But it’s assuming you don’t have any data. And then I’ll talk about what happens if you already do; you need to ask the right questions. World War II example; why are bombers falling out of the sky?

 

[00:09:33.370] – David Maples

Well, they have holes in them. Holes happen when people shoot flat cannons at aircraft. Why are people dying in World War I? Well, they’re getting shot in the head. You want to ask questions, and then you want to go and acquire that data.

 

[00:09:50.740] – David Maples

You want to ask yourself questions like, why are companies failing? And the question is, do you just want to look at the positive companies? You want to look at both. It’s kind of the thing. And then you want to challenge your assumptions, but you’re going to ask the questions first.

 

[00:10:02.870] – David Maples

So, let’s use the bomber example in World War II. First question you’re going to say is like, hey, why are they falling out of the sky? They have holes in them. We need to find out where the holes are so we can patch them and eliminate them. We know that if we put more metal over those hole areas, or whatever it is, they won’t get as many holes.

 

[00:10:21.690] – David Maples

So, you need to acquire and collect the data. So, you need to have a method that you put in place, and it must be a process that you follow. If you’re not collecting the data in the same method, it’s very hard to get predictable and reliable results. You need to acquire and collect the data. Okay? The next piece is to analyze and review the data in light of the asking questions you did at the beginning.

 

[00:10:45.990] – David Maples

So, you want to analyze the data, and you want to review it. And this is where you need to be brutally honest with yourself. So, for a second, this is your no BS segment. Okay? I got a whole bunch of data. Doesn’t tell me the things I wanted to do.

 

[00:11:03.060] – David Maples

I had to prove my boss right or prove him wrong or her wrong. I needed to say something about what we’re doing here, right? This is an exact recipe to get mired in your own BS. What is the question they’re asking? They don’t make music like they used to.

 

[00:11:17.800] – David Maples

So, if we go back and make music today that sounds like chart-topping hits from The Beatles in the 60s, it’s going to be more successful. If it turns out my band already makes stuff that sounds like The Beatles from the 1960s am I just producing stuff to help make them happy about what they already believe? This is where you need to be brutally honest with yourself and say, what do you need? The military in World War II could have said, yeah, that makes perfect sense. Let’s go do that stuff. And instead, they gave it to this outside statistician, Abraham Wald, and said, hey, look at our data.

 

[00:11:51.770] – David Maples

And he says, this is wrong. This is totally wrong. You’re falling victim due to the classic blunders, you know, which is survivorship bias. That’s why I need to look at that, and your person you’re asking to give you this unvarnished feedback, it helps if they’re not too close to the problem, if they don’t have a vested interest, okay? If they’re not the company who’s already got the contract to armor up these other pieces of your airplanes, right?

 

[00:12:14.020] – David Maples

Those are problems. And if you want to be honest with yourself as a business and you want to not fall into that BS trap, you need to look at the numbers, and you need to really ask yourself hard questions about that. Analyze it. Get a third-party consultant or somebody else who has no skin in the game, not based on the outcome, not based on them doing extra work for you. Say, hey, what does this mean to you as a third-party observer?

 

[00:12:35.530] – David Maples

If you’re not going to hire a third-party observer, then ask some other people who are smart in your company who aren’t tied. If it’s a new marketing venture for your company, you can ask the marketing people, you can find the right people in that department, but maybe you go ask somebody in your accounting department and say, I need another set of eyes on this, just to give me another angle on this to see if we’re falling victim according to our own blunders. Are we only looking at our successes? And one of the challenges for a lot of business owners is that when you’ve been successful, it’s really, really common to attribute that to something magical or mystical that only you have. Well, it’s my ability to talk on stage.

 

[00:13:12.790] – David Maples

It’s obviously led my company to become a titan of industry. It’s because I run really wicked smart radio campaigns. When it turns out the secret to your success is not with that at all, it turns out none of your money’s coming in that way. Maybe you do have wicked smart radio campaigns. But it turns out you’ve built an amazing sales force nationwide, and they’re knocking on doors, and that’s where it’s generating your stuff.

 

[00:13:40.350] – David Maples

And it’s really easy to ask those people in the marketing department to say, oh, it’s obviously because of this. Challenge your own preconceived notions. Challenge your own things. And when you have data, you know, ask hard questions about it, right? But at the same time, you need to not make it adversarial, and that’s what you need to do. No BS means looking at the stuff from different angles, seeing if it makes sense to you, and then going from them.

 

[00:14:03.990] – David Maples

So, analyze and review your data, make sure you’re not falling victim to these other fallacies, such as survivorship bias, and then you have to take action. And the thing about this is it’s kind of a repeating scenario. You want to go back to step one. You want to ask questions and refine your data.

 

[00:14:19.530] – David Maples

You want to acquire and collect more data. It’s the same thing. You want to analyze and review data, and now you want to take action. And it turns out the Quad-A, or 4A method if you choose to call it that, is what we should always be doing with the data. We should always be going through these things and taking these four steps.

 

[00:14:37.850] – David Maples

Not incredibly hard to do, but 95 plus percent of companies don’t do that. And by the way, it’s sometimes hard to not fall into your own BS trap. It’s very difficult as a business owner when you have been successful.

 

[00:14:57.230] – David Maples

You know, I provided some consulting work for a client a long time ago, and it reminded me of an interview I did last season with Abby Foster, and she said, you know, I was a terror. Sometimes as a consultant, you have to be careful with delivering the message really bluntly, you know? They say they want blunt method, but sometimes you need to kind of wrap it up and give it to them in swaddling clothes because they’re not going to accept it very well. But I literally was talking to a client, and it was obvious as the nose on my face that the client was successful in spite of their marketing efforts and not because of them—for example, dot com implosion in 2002.

 

[00:15:35.850] – David Maples

Case in point, your company survived the dot com implosion. And if you’re of a certain age, you remember this, right? Pets.com and all these companies, hundreds of millions of dollars were spent on them, and they all failed. It was the wrong time.

 

[00:15:51.830] – David Maples

But you didn’t get your product to market, you didn’t get your website launched during the failure process, and say you launched six months after dot com implosion, and you were the only company making, I don’t know, household furniture, grills, something like, I don’t know, something in the marketplace, right? But you’re the only company now selling them online. It’s really easy to go back and write a revisionist version of history. It’s like, oh, we totally did this, and we launched, and we were successful because of our marketing efforts. No, it’s because there’s no one else to buy from, right? And there’s one thing that the Pandemic taught us is that there are a lot of winners and losers in that process, and some people just got lucky.

 

[00:16:31.640] – David Maples

Look, there’s nothing wrong with being lucky, you know? If you are lucky, I mean, some people- There’s one of the stories historically, and I am a student of history, is that there are generals, like, you know, who are famously successful, and some of them just were lucky. Right place, right time. They weren’t bad, but they weren’t masterful battlefield tacticians. They weren’t master strategists, they weren’t kings of logistics, but they were in the right place at the right time.

 

[00:16:58.930] – David Maples

And there’s a lot to be said for that, right? But it’s really easy to kind of fall into your own faults and irritate what you’re doing. So, don’t do that. So, kind of what we’re saying today is think about survivorship bias when you’re looking at your data. Look at your successes, which are great to look at, and they’re fun to look at, but analyze your failures.

 

[00:17:19.010] – David Maples

And by the way, don’t kick yourself while you’re down. Looking at your failures is one of the hardest things you’ll ever have to do as a business owner. You can often learn more from a failure than from a success because when you have a success, you can’t always point to the particular thing that made you successful. But when you look at your failures, sometimes it’s really, really obvious that, oh, that’s how I lost this client.

 

[00:17:38.360] – David Maples

That’s how I did these things. That’s how I made these bad hires. Usually, though, I find that these are usually what we call multi-factorial problems. Usually, there’s a level of things you can look at. So, one of the reasons you want to go back to this – ask, acquire, analyze, act – you want to keep repeating that.

 

[00:17:54.820] – David Maples

Now, if you already have a data set, the only thing you change in this process is because you probably had a question in mind, is were you asking the right questions? So, ask the right questions. Before you get data, you can get really bogged down with asking the right questions. Have a problem? Ask yourself what you need to do to solve it.

 

[00:18:15.640] – David Maples

Holes in planes making them fall out of the sky. Okay, there’s a lot of different ways to tackle that, right? But we want to acquire and collect data, right? There’s a lot of different ways to make that happen. And then kind of, you know, follow that old Pareto principle.

 

[00:18:28.410] – David Maples

You know, the easiest solutions sometimes you get a lot of success from those things. So, if you had been a war planner in World War II, you could’ve said, “Ah! There’s a couple of different ways we could do this. We could put armor on the areas of the plane that need to be to increase survivability. Or we can introduce special forces teams to go throughout Europe and shut down all the anti-aircraft cannons, so no anti-aircraft fire would fire on airplanes.”

 

[00:18:51.650] – David Maples

Undoubtedly that would be successful, but it’s also not very realistic. So, what you want to do is you want to kind of think about where you get your best bang for the buck early on. Okay, so three takeaways from this episode today. First of all, look at- First of all, what data is available to you in the marketplace right now? What data is available to you in your company, and what are you currently acting on?

 

[00:19:15.750] – David Maples

Okay? Look for things that would probably provide the best benefits. Get a small team of people in your office and say, hey, where can we get data? And what questions can be asked about the data to become more successful in our efforts? And the production company? It could be supply chain problems right now.

 

[00:19:31.240] – David Maples

Those are paramount across the planet. Always an issue. Is there a way we can do those things? And then follow the 4 A’s. Figure out what the plan is for that. You want to ask the questions, you want to acquire the data, you want to analyze and review the data, and you want to take action on it. So, those are the things you’re going to do. Make sure as step two, as you’re looking at your data, make sure that you do not fall victim to the survivorship blunders, okay? Make sure you’re not just looking at them.

 

[00:19:58.070] – David Maples

All right? And the last thing that we’re going to advise you to do on this is to look at not only the successes, but also analyze your failures, because it’s going to give you a lot more questions. Now, this could turn into a big exercise where you’re doing a whole bunch of different things. Look for those 80-20 kind of wins. Look for the quick wins that you can address immediately.

 

[00:20:19.110] – David Maples

We have an earlier episode where we do talk about kind of breaking yourself into the Eisenhower quadrants over maximum impact, lower effort. You want to look at those and see what kind of works in that way. So, I’d like to thank our sponsor, Catapult, today about data. They’re a great marketing agency and data analytics collection company. If you’d like to talk to them, you can see more about them at catapultcreativemedmedia.com.

 

[00:20:44.610] – David Maples

I’d like to thank everyone for listening to this episode. If you like what you heard here, please give us a good review on Spotify, Audible, or Apple Podcasts, or any place you have to be listening to this podcast. If you’re watching us online, we would appreciate a thumbs up on YouTube and share it with your friends. With that, thank you for listening. Be well, and don’t fall victim of survivorship bias.

 

[00:21:09.010] – David Maples

Remember, that may not mean what you think it means. Thank you. Have a great week. Go out there and be awesome.