- About Us
Invited Speakers Session 2012: Bandit's Paradise
Bandit’s Paradise: The Next Generation of Test-and-Learn Marketing
Eric M. Schwartz, Eric T. Bradlow, Peter S. Fader
Department of Marketing | The Wharton School | University of Pennsylvania
Invited Speaker: Dr. Peter S. Fader, Frances and Pei-Yuan Chia Professor of Marketing, The Wharton School, University of Pennsylvania
Testing the effectiveness of marketing activities via experimentation has long been a hallmark of direct marketers, but now almost every kind of marketer has “joined the party.” Changes in the digital environment have led to the proliferation of marketing experiments, such as A/B and multivariate tests, but they are now run on a much more frequent basis and for an ever-widening array of marketing decisions. Virtually every outbound marketing activity (e.g., sending emails, serving display ads, customizing websites, recommending products) can be framed as adaptive experiments to “test and learn” on a continuous basis.
But as these tests evolve from standalone projects to being deeply integrated into a firm’s day-to-day operations (and culture), it is more important than ever to ensure that testing is done in a profitable manner. It is no longer acceptable to merely “test and learn” with the hope of making greater profits in the future; today’s digital marketers must “earn while they learn.” In this session we demonstrate that:
- running tests on a relatively naïve basis (e.g., manipulating only one feature at a time or picking the “winner” too early) can leave money on the table or might be unnecessarily risky.
- managers need to be more cognizant of the “exploration-exploitation” dilemma between the desire to continue testing to benefit in the future (explore) versus the desire to quickly leverage initial results and select actions that seem to be performing well (exploit).
In order to resolve this tension, we focus on the application and further development of the so-called “multi-armed bandit” paradigm as an optimal framework to perform adaptive testing in a profitable manner. We have
- carefully reviewed and synthesized the relevant literature from a variety of domains (e.g., computer science, operations research, clinical trials, and of course, marketing) and
- developed new algorithms and operational procedures that represent the best of all worlds.
We discuss how various aspects of the managerial problem (e.g., decision context, data sparseness) can affect the relative performance of different methods. For instance, three of the relevant issues are:
(1) How rare is the event of interest?
(2) How small are the effect sizes differentiating the actions? and
(3) How big is the batch of simultaneous allocations that are made for each decision period?
The results from this integrative research were obtained from a variety of rich, realistic simulations, as well as a number of large-scale live field experiments (with a multi-channel retailer and a major retail bank) aimed at increasing advertising effectiveness. While we continue testing and learning in our research program, we discuss our observations to date, as well as specific tactics and strategies to help managers perform adaptive testing activities more profitably than ever.
PETER S. FADER
Dr. Fader's expertise centers around the analysis of behavioral data to understand and forecast customer shopping/purchasing activities. He works with firms from a wide range of industries, such as consumer packaged goods, interactive media, financial services, retailing, and pharmaceuticals. Managerial applications focus on topics such as customer relationship management, lifetime value of the customer, and sales forecasting for new products. Much of his research highlights the consistent (but often surprising) behavioral patterns that exist across these industries and other seemingly different domains.
Many of these cross-industry experiences have led to the development of the Wharton Customer Analytics Initiative, a new research center that serves as a “matchmaker” between leading-edge academic researchers and top companies that depend on granular, customer-level data for key strategic decisions.
Professor Fader believes that marketing should not be viewed as a “soft” discipline, and he frequently works with different companies and industry associations to improve managerial perspectives in this regard. His work has been published in (and he serves on the editorial boards of) a number of leading journals in marketing, statistics, and the management sciences. He has won many awards for his teaching and research accomplishments.