Bill Petti, Global Data Strategist at Gallup, outlined the strategy for getting past company barriers to assure adoption of and success in advanced analytics.
Petti began his thought leadership presentation at the 2017 Leadership in Big Data and Analytics Forum, held on December 5 in Chicago, by noting that his company often feels frustrated when it provides clients with predictive business models that the clients don’t adopt. “We’d do an awesome analysis of how a client could better engage their customers and drive higher wallet share and retention, and, three years later, they’re still stuck in the same place. Often, their response would be to fire us and move on to someone else. The more we dug into this, we noticed the business and decision makers hadn’t done anything different even though the data analysis told them they should,” he explained.
“We’d do an awesome analysis of how a client could better engage their customers and drive higher wallet share and retention, and, three years later, they’re still stuck in the same place. Often, their response would be to fire us and move on to someone else.”
“We’ve discovered a number of organizational barriers to both the adoption and success of advanced analytical programs. The four prerequisites for getting the most out of data and advanced analytics are: right metrics, right people, right data systems, and right culture/processes. Getting value from analytics takes more than excellence in any one area. It’s more than a platform, hiring data scientists, or subscribing to proprietary data sets,” Petti said.
“I’m going to focus on the people piece, especially the people in the organization who aren’t data scientists. There’s a certain skill level within the rest of the organization that you have to have. If there isn’t a baseline statistical literacy, this will create a barrier to getting more out of this work.”
“There’s a certain skill level within the rest of the organization that you have to have. If there isn’t a baseline statistical literacy, this will create a barrier to getting more out of this work.”
Statistical literacy is the ability to read and interpret data, to use statistics as evidence, and to think critically about statistics. “Statistical literacy doesn’t require deep, technical understanding of statistics, mathematics, or machine learning, but it does require enough understanding of basic concepts to be able to examine models and analysis with a critical eye, to ask the right questions in order to check the work of modelers, and to be able to make the right requests,” said Petti.
Petti provided a sample of key requirements for statistical literacy:
“Accuracy” is usually a bad way to judge a model’s performance. “Models that try to predict a behavior are often judged by accuracy—how often they make an accurate prediction. But in cases where the data is ‘unbalanced’—where the behavior doesn’t occur half of the time—accuracy can be misleading. When evaluating classification models, be skeptical of any that only report overall accuracy or any single evaluation metric,” he advised.
If you didn’t test it out of a sample, it doesn’t count. “Models will always perform better when applied to the data that was used to build them,” explained Petti. “The risk is that the models are ‘over fit’ to the data available to them and won’t perform as well on data they haven’t seen. When considering the value and performance of a model, never use metrics based on the training data—rely on metrics based on data that weren’t used to build the model.”
“Models will always perform better when applied to the data that was used to build them. The risk is that the models are ‘over fit’ to the data available to them and won’t perform as well on data they haven’t seen.”
Ponies hide in the aggregate. “We often study something at the aggregate level, but the aggregate can hide important patterns—or, in some cases, show the opposite relationship. This is known as Simpson’s Paradox: a phenomenon in statistics in which a trend appears in different groups of data but disappears or reverses when these groups are combined.” Petti provided an example in the answer to the question, What’s the relationship between tenure and performance in sales? “At the aggregate level, it appears that the longer someone is in a role, the less they sell. However, when we model the relationship within each type of sales role, we see the opposite—the longer a person is in the role, the greater the sales. Most people in business aren’t trained to think about this.”
In summary, Petti stated, “We also suggest having a strategic translator—someone who can speak both to the highly technical folks who are creating the models and doing the analysis and to the business leaders. There’s often a big communication barrier between those two groups.”
ABOUT BILL PETTI:
Bill Petti currently serves as Gallup’s Global Data Strategist. He directs advanced analytical consulting globally for the company. He helps clients use data and analytics more productively and build “data cultures” that embrace analytics in decision making. Bill is an expert at applying various methodologies—both quantitative and qualitative—to understand and influence human behavior.
Prior to joining Gallup, Bill worked for Gerson Lehrman Group (GLG). While at GLG, Bill worked with senior pharmaceutical and technology executives to execute primary research. This research supported market analysis, licensing and partner evaluation, new product development, and human capital management.
Bill received his bachelor’s degree in political science from The College of New Jersey. He also earned master’s degrees in political science from both Temple University and the University of Pennsylvania.
Bill is a nationally recognized expert in talent evaluation, player analysis, and performance analytics for professional baseball. He has served as a consultant and advertiser to several major league teams.