Pilot tests of new merchandising concepts or new product designs are standard fare for most successful retailers and consumer goods companies. Still, interpreting the rich data surfaced by those initiatives remains a highly subjective affair.
“It matters if Bill or Betty does your analysis. You don’t have an analysis, you have an opinion,” said Jim Manzi, founder and chairman of Applied Predictive Technologies, a company focused on making the process of business experimentation far regimented. (A mathematician by training, Manzi bears no relation to the former CEO of personal computer software pioneer Lotus Development who shares his name.)
“Every company that we visit is experimenting in terms of trying new things,” he said. “What they lack is the capacity to design and read these experiments reliably.”
APT’s software and testing methodology helps companies set more rigorous parameters around business experiments. It use existing transaction data to pick better control groups, pinpoint the variables that should be examined, and (eventually) interpret the results. Its clients include 100 of the world’s biggest consumer brands including Anheuser-Busch InBev, Coca-Cola, McDonald’s, Procter & Gamble, Starbucks, Target, and Walmart. APT’s access to 20% of all retail transaction in the United States (aggregated anonymously) helps inform decisions.
Manzi’s philosophy is detailed in a December article published by the Harvard Business Review (“The Discipline of Business Experimentation”), which he co-authored with Harvard professor Stefan Thomke.
They cite the successful example of Kohl’s, which used the ideas behind APT’s approach to test the impact of adding furniture to its product mix through a pilot in 70 locations. The reasonable thesis: these big-ticket goods would significantly increase sales. To the retailer’s surprise, the results actually showed a net decrease for the stores—mainly because lots of other items were displaced to make room.
“The Kohl’s example highlights the fact that experiments are often needed to perform objective assessments of initiatives backed by people with organization cloud,” Thomke and Manzi write. “Of course, there might be good reasons for rolling out an initiative even when the anticipated benefits are not supported by the data—for example, a program that experiments have show will not substantially boost sales might still be necessary to build customer loyalty.”
Manzi declined to discuss pricing for APT’s software and services. APT’s lead investor is Accel-KKR, but the 15-year-old company’s approach last year also attracted a $100 million minority investment from the merchant banking division of Goldman Sachs.
Noted managing director Joe DiSabato at the time of the investment: “APT is a rare innovator, with multiple patented and commercially proven approaches for truly leveraging big data to generate shareholder value. The collection of talent at APT representing deep data science, advanced math and computer science is delivering that opportunity to customers every day, driving tremendous value.”
This item first appeared in the Nov. 24 edition of Data Sheet, Fortune’s daily newsletter on the business of technology. Sign up here.
