There’s a lot of social influence around trusting intuition and making decisions based on emotional response. The story around trusting a hunch is so strong that even people in powerful business positions can get caught up in the “magic” of it.
“Trust your gut.”
“I’ll know it when I see it.”
“I’ve got to feel it.”
The truth, though, is that people are not very good at making decisions based on their feelings. Which might be ok except when a decision could make or break a company. We see more and more that data-driven decision making is in the DNA of
Reliable and accessible data, robust and responsive analytics and the capability to make decisions based on them are absolute necessities for any business looking to be competitive in today’s economic climate. Why is it that data beats out intuition? Let’s take a closer look.
Humans have evolved over millennia to be keen observers and to make decisions that are literally life or death. We have eons of neurobiology designed to help us make the decisions that will keep us alive, but not all of this evolution is relevant in a modern society. In fact, much of it is downright incompatible with the types of decisions that we have to make in an interconnected, global, and technological society.
The Past Does Not Always Predict the Future
Much of our intuition is merely a judgment based on experience. If we have a “bad feeling” about something, there’s a good chance that we’re remembering (consciously or subconsciously) a similar experience in the past and projecting the way we felt about it onto this future event.
People are Terrible at Estimating
In the absence of data collected to show us the actual numbers we’re dealing with, we do the next best thing: we guess. The thing is, we’re horrible at estimating. As Berkeley researchers report, we’ve evolved with the skills necessary for survival with our ancestral wilderness in mind, but we’re not prepared to estimate things in the scope or purpose of the business world.
Our Memories are Not Stable
Our memories change to fit the stories we tell ourselves. This is useful when we want to tell a funny tale about our mix-up at the grocery store last week and need to fill in some gaps to make the retelling better, but it can be problematic. In fact, many eyewitness accounts in criminal cases
We Are Terrible at Calculating Risks
Our brains use shortcuts to help us operate in a fast-paced world filled with too many data points for a human
The Future is Data-Driven
If you’ve ever read the book or seen the film Moneyball, you know how data-driven decision making can take an industry by storm. It’s the story of how Billy Beane used playing statistics to choose baseball players instead of going with those who made the biggest splash. By trusting the data—instead of his gut or the media hype—he assembled a winning team and changed the nature of sports.
Savvy, data-driven businesses across the globe are doing something similar. Not only do they know that data-driven decisions are more reliable and predictive of future results, but they have the right tools, skill-sets and processes to gather the insights they need quickly, accurately, and affordably.
Creating a data-driven culture places analytics at the very core of the business model. It’s what’s allowed companies like Netflix, Amazon, and LinkedIn to thrive. Netflix has woven its user data into every decision it makes, turning its wealth of information into a flywheel of original content. Amazon’s data set is so vast and so thorough that the company can predict and recommend products before a shopper even knows they want it. In fact, if decisions were based on hunches at Amazon, this feature (commonly known as the recommendation engine) may never have seen the light of day.
Data engineer, Greg Linden, pioneered the recommendation engine. He’s a dogmatic innovator who measures everything – a trait that is now commonplace in Amazon but, at the time, Amazon’s culture was still finding its feet and Greg’s prototype had been challenged by a HiPPO (the Highest Paid Person with an Opinion).
I heard the SVP was angry when he discovered I was pushing out a test. But, even for top executives, it was hard to block a test. Measurement is good.
Reading through Greg’s blog post (which pre-dates the iPhone), I can almost sense the cultural transformations that were taking hold. The internal struggles, pitting experienced and influential decision makers against math and terabytes of data. Fortunately for Greg (and Amazon), testing prevailed.
The results were clear. Not only did it win, but the feature won by such a wide margin that not having it live was costing Amazon a noticeable chunk of change. With new urgency, shopping cart recommendations launched.
So what does a data-driven culture look like?
- Complete buy-in and commitment – it begins with the highest levels of upper management making a pact — a pervasive agreement that, henceforth, no decision is to be made on intuition alone. It must be championed incessantly and become standard practice at every level of the business.
- Access to data and tools – a mandate to use data in decision making would be fruitless if relevant, trust-worthy data and intuitive, self-service tools (to prepare, blend and visualize) were not made available to everyone.
- Data-literacy – there’s a data-literacy gap in most businesses. Simply put, data practitioners know what to do with the data that’s shared with them while others don’t. For most businesses, this gap is large, and ever-widening. The answer is not to hire more data practitioners. The answer is training. Every employee (from the C-Suite down) should be empowered to combine their experience (intuition) with the knowledge to extract insights (data). That’s powerful.
As Greg put it – “Everyone must be able to experiment, learn, and iterate…For innovation to flourish, measurement must rule.” Aside from the “power in numbers” rationale, there’s a strong argument to be made for cutting out the middleman by equipping subject matter experts with the analytics know-how and capabilities to produce their own insights.
The lesson from data-driven businesses is that they do not rely on intuition alone. Intuition would likely have told Netflix that House of Cards would be a flop. It would have kept Amazon from inventing the recommendation engine that helped catapult the company to gargantuan proportions. Instead, these businesses innovated with and relied on hard evidence.
Not only that, but these companies employ a (some might say) obsessive drive to innovate with data. A true data culture. They invest in their people, they invest in data collection, and they continually iterate their tools and processes. They’ve created data alchemy – a way to turn mountains of raw data into valuable insights that put them head and shoulders above the competition . . . even when it goes against their gut.
Get our Updates as They Happen
Subscribe to our blog to get new posts delivered straight to your inbox.
As Director of Enterprise Analytics, James helped Thomson Reuters establish data management capabilities and an enterprise-wide analytics competency.
Latest posts by James Nanscawen (see all)
- The Case for Data Democratization: Data Scientists Can’t Keep Up . . . Nor Should They - July 14, 2019
- What is a Data Catalog and Why Do You Need One? - June 26, 2019
- How Satya Nadella used Data to Reignite Microsoft - March 18, 2019