IS MANAGEMENT WITHOUT EVIDENCE LIKE BUILDING ON SAND? DOES A CULTURE AND TOOL KIT OF EVIDENCE PROVIDE US WITH A BETTER FOUNDATION?
/
I’m getting ready to update my syllabus for my Executive MBA course on Managing Innovation and Change. We cover the same material as in the regular MBA program, but with a greater focus on issues for the experienced manager. In my course we focus our attention on the creation and management of innovation, motivation/compensation, negotiation/change management, and the integration of technology tools with management practice. For each major course concept there is a written assignment where students explain the basics of the concept and provide citations, apply the concept to the student’s work or other related organization, and design an evaluation method to assess the ROI of the application. This last section is a major roadblock for most students, and I’m trying to understand why. I’m planning on changing the final project such that students must select, implement, and evaluate one of these concepts – not just write about how they might do these things.
Given the trouble that students have with even just writing about this process, how successful will they be at full implementation? That said, if they can’t implement what they are learning, and show value, then what’s the benefit to the organization? Even if you have good ideas about intertwining technology and organizational practice -- if you can’t implement and evaluate, how will you be able to manage the situation as it unfolds?
(Pure speculation begins here.) Is this difficulty because the earlier portions of the Executive MBA program – accounting, operations management, finance, marketing, international management – imply that only certain kinds of objective outcomes have value? Do they imply that their measurement techniques are only valid in their particular disciplines? These students have been in the program for over 14 months when I meet them. They have gained the majority of the skills available in the program. They (and MBA students world-wide) have measurement and evaluation skills based on accounting, operations, finance, marketing – but why aren’t we teaching them measurement and evaluation skills for general management and organizational change, or at least showing them how these other, more traditionally empirical, disciplines may help them evaluate their management/technology practice?
I’ve mentioned evidence based management before/, but Tracy Allison Altman’s recent post on the loneliness of the evidence based manager made me look at my industry to see if we were doing our students a disservice by not formally teaching research methods appropriate to management. I have added a couple of class sections on the topic, but the underlying skills (survey design, experimental design, etc.) are generally given a full course in industrial psychology degree programs. Measuring more subjective outcomes is difficult and requires a specific skill set (bad surveys, for example, being far worse than no survey at all). An extensive search turned up very few appropriate readings (see below for links). Harvard Business School Press lists 997 cases and articles on negotiation. Eighteen cases and articles come up when you search on evidence-based management – and eleven of these are by the same two authors (Jeffrey Pfeffer & Robert Sutton) largely focused on their excellent “Hard Facts,” ideas, but you’d hope that this critical topic were addressed by more management scholars (again, see my list below for what is available). We don’t have room to add whole new courses in most MBA programs, so we better find a way to teach this material more efficiently.
The students have good intuition. They realize that their decisions may impact only a few and so fancy statistical assessments may not work with such small sample sizes. They also understand that a control group would be of value for being able to discern differences – but they don’t see how they can manage both an “experimental” and a control group in the real world. I am surprised by how rarely before and after measures are considered. I believe this has something to do with the limited culture of measurement, let alone evidence, in many of their organizations (i.e., they don’t have the “before” data). This harkens back to my earlier visualization posts. How can we know that what we implement is working if we didn’t know from where we started?
I’d appreciate thoughts on how you or your firms are working to build a culture of evidence. Both technology focused and general management practice can be measured, and as a result, adapted as need be. Evidence can also provide the background to make changes your intuition alone cannot support. On the other hand...
Background material:
Given the trouble that students have with even just writing about this process, how successful will they be at full implementation? That said, if they can’t implement what they are learning, and show value, then what’s the benefit to the organization? Even if you have good ideas about intertwining technology and organizational practice -- if you can’t implement and evaluate, how will you be able to manage the situation as it unfolds?
(Pure speculation begins here.) Is this difficulty because the earlier portions of the Executive MBA program – accounting, operations management, finance, marketing, international management – imply that only certain kinds of objective outcomes have value? Do they imply that their measurement techniques are only valid in their particular disciplines? These students have been in the program for over 14 months when I meet them. They have gained the majority of the skills available in the program. They (and MBA students world-wide) have measurement and evaluation skills based on accounting, operations, finance, marketing – but why aren’t we teaching them measurement and evaluation skills for general management and organizational change, or at least showing them how these other, more traditionally empirical, disciplines may help them evaluate their management/technology practice?
I’ve mentioned evidence based management before/, but Tracy Allison Altman’s recent post on the loneliness of the evidence based manager made me look at my industry to see if we were doing our students a disservice by not formally teaching research methods appropriate to management. I have added a couple of class sections on the topic, but the underlying skills (survey design, experimental design, etc.) are generally given a full course in industrial psychology degree programs. Measuring more subjective outcomes is difficult and requires a specific skill set (bad surveys, for example, being far worse than no survey at all). An extensive search turned up very few appropriate readings (see below for links). Harvard Business School Press lists 997 cases and articles on negotiation. Eighteen cases and articles come up when you search on evidence-based management – and eleven of these are by the same two authors (Jeffrey Pfeffer & Robert Sutton) largely focused on their excellent “Hard Facts,” ideas, but you’d hope that this critical topic were addressed by more management scholars (again, see my list below for what is available). We don’t have room to add whole new courses in most MBA programs, so we better find a way to teach this material more efficiently.
The students have good intuition. They realize that their decisions may impact only a few and so fancy statistical assessments may not work with such small sample sizes. They also understand that a control group would be of value for being able to discern differences – but they don’t see how they can manage both an “experimental” and a control group in the real world. I am surprised by how rarely before and after measures are considered. I believe this has something to do with the limited culture of measurement, let alone evidence, in many of their organizations (i.e., they don’t have the “before” data). This harkens back to my earlier visualization posts. How can we know that what we implement is working if we didn’t know from where we started?
I’d appreciate thoughts on how you or your firms are working to build a culture of evidence. Both technology focused and general management practice can be measured, and as a result, adapted as need be. Evidence can also provide the background to make changes your intuition alone cannot support. On the other hand...
Background material:
- Pfeffer & Sutton’s website/blog. Includes teaching materials and case examples.
- Thomke’s Enlightened Experimentation: The New Imperative for Innovation
- Perla’s Do your Metrics Measure Up
- Lawler’s Why HR Practices are Not Evidence-Based
- Roussea & McCarthy’s Educating Managers from an Evidence-Based Perspective
- World Bank Selecting an Evaluation Design
