A Short History of Scientific Evidence for Managers
In my first post on evidence-based management, I explored the gap between what researchers know managers should do to improve organizational performance, and what managers do. Before telling you how to close that gap for yourself, in this post I want to make clear you aren’t responsible for the gap! It turns out there are many barriers to finding good, objective evidence as opposed to “expert” opinions—which are often wrong, per that earlier post.
At first glance, it seems like good-quality evidence is readily available. After all, “scientific management” was pioneered by Frederick Taylor at the turn of the last century. Psychologist William McDougall wrote a book called The Group Mind in 1920 whose five bases for improving group work remain accurate as far as they go.[1] My mother observed time-and-motion studies in a battery plant during World War II, and my earliest source for a paper on Agile organizational change[2] dated to 1948.[3] A number of now-common group training techniques have roots in the founding of the National Training Laboratory in the late 1940s. Flat hierarchies with leaderless teams were successfully implemented as early as the late 1950s. Many of the concepts modern American managers think of as Japanese were invented in the U.S. and taught in Japan by American consultants as part of the postwar reconstruction effort. The foundational thinking of bestsellers from different decades like The Peter Principle, Theory Z, In Search of Excellence, and Management 3.0 bear striking resemblance, though each had details contradicted by scientific findings.
The issue is too much information, and separating out the valid from the myth.
The Good, the Bad, and the Mixed
For a possible journal paper, I reviewed all business-related articles trending on Pocket from December 2017 to March 2018. Pocket is an aggregator from Mozilla that appears on the Firefox default landing page.
Some articles were well-researched, using scientific sources, which were cited such that you could double-check them if you were skeptical:
- A professor who researched the famous tulip mania of the 1600s using archives in the Netherlands wrote that contrary to common misperceptions, there’s no evidence anyone was bankrupted; not one suicide was recorded; and the Dutch economy was unaffected by the price collapse.[4]
- An article showing how much uncontrollable factors—luck—impact success cited multiple studies.[5]
- A blog post titled, “Work Smarter Not Harder,” named the academic source and described the study methodology.[6]
Unfortunately, a larger number of articles provided little or no scientifically valid proof for the claims made, and in many cases went against research-based evidence:
- An article on remote teams in Harvard Business Review (HBR)—a source many readers assume to be scientific—made no reference to scientific evidence and missed many evidence-based concepts, most critically the problem of communication delays when teams are in disparate time zones.[7]
- A training and development professional asked to create a leadership development program turned to her “favorite thought leader” at Forbes.com, who cited no evidence in her list of “Ten Skills Every Manager Needs—But 90% Of Managers Lack” (or for the “90%” claim), and missed many skills the research literature would recommend more highly.[8]
- Despite its title, “Why It’s so Hard to Actually Work in Shared Offices,” a blog post merely talked about the many distractions of the WeWork environment without explicitly addressing the likely productivity impacts, based on 100 years of workspace research.[9]
Some fell into mixed territory:
- An infographic on unconscious biases that impact our decision-making was accurate, but listed no sources.[10]
- Though written by researchers and reporting on a cited study, the headline of an HBR article, “When Empowering Employees Works, and When It Doesn’t,” was misleading because no situation was reported where empowerment didn’t work at all; it just worked better in some than in others.[11]
Even sources one might expect to provide good evidence often don’t. A research team first surveyed 85 editorial board members of top academic journals relevant to HR. The team asked which research findings “all practicing managers should know.” The researchers compared that list to a content analysis of HR Magazine, HBR, and Human Resource Management. The best-covered of the editors’ top three topics appeared in only 1.2% of the articles. Only one of these magazines used evidence-based information on two of the three topics. The third topic received barely any coverage at all. “Finally, treatment of personality in HBR seems to be completely divorced from academic research…” the scientists wrote, “with no mention of the Big Five, continued discussion of narrow rather than broad personality traits, and no research-based summary of generalizable personality-performance relationships.”[12] (Big Five tests are the only scientifically valid personality assessments.)
Popular books from researchers or science writers exist, but are nearly drowned out by first-hand accounts from executives and by consultants reporting on their own exploits. The bestsellers are usually based on “anecdotal evidence,” stories told from the narrow perspectives of the authors with no filter against their biases. And scientists themselves sometimes blur the lines. Organizational change consultant John Kotter was a professor. Yet his most influential book was “based on his personal business and research experience, and did not reference any outside sources,” according to a study comparing his model favorably to the evidence.[13] My reading of that book, Leading Change, confirmed there is little reference to journal publications on the subject, including his own![14] He knew he was practicing EBM (before it got that name), but his readers had no way of knowing that.
Education Fails the Test
Formal education doesn’t necessarily help. Many textbooks don’t present evidence, many instructors don’t have Ph.Ds (even in MBA programs), and many managers don’t get management degrees.[15] For example, the current undergraduate textbook Principles of Management mostly references newspaper and magazine sources, and includes some information debunked by researchers decades ago.[16]
A content analysis of syllabi from more than half of the required MBA classes at accredited business schools in the U.S. found that only two used the phrase “evidence-based management.” The research team, apparently surprised, had to expand its search to related terms like “research” or “evidence.” Even then, only 26% of courses “included EBM-related statements.”[17]
A former librarian turned researcher concludes, “Small wonder that multiple studies have shown that managers read little academic research.”[20]
Another research team says the fact management is not a “profession” contributes to the research-practice gap. “As such, there is no requirement that managers be exposed to scientific knowledge about management, that they pass examinations in order to become licensed to practice, or that they pursue continuing education in order to be allowed to maintain their practice.” Job ads want someone like me to have certifications in project management or Agile regardless of what degrees they have. Yet I’ve never been interviewed by a supervisor who was certified as a supervisor!
For now, managers wanting to practice EBM by finding scientific studies related to the issues they are facing are mostly on their own. A Web search finds little in-person or instructor-led training on EBM, and the only comprehensive series costs more than $1,000 US. At least two detailed guides on doing EBM are freely available, fortunately. “Evidence-based Management: The Basic Principles” from the Center for Evidence Based Management is the more specific.[18] “Using Research Evidence for Success: A Practice Guide,” is directed at government and nonprofit agencies, but most of its advice is applicable to managers as well.[19]
Read with a Skeptical Eye
I will detail how to find and read studies in my next two EBM posts. Meanwhile, you can help yourself by being a critical reader. Per the “Good, Bad” listing above, when reading about management or attending a presentation, notice whether the writer/presenter provides a list of sources. If so, how many are there? Are they mostly articles in scientific journals (like “Psychological Review”) or just magazine articles or other presentations? The more the content seems based on the writer’s personal experiences, magazine articles, and anecdotal evidence from other people, the less sure you should be it will apply to your world.
Please share this post at the bottom of the page.
[1] See “The Familiar History of Teambuilding” from my Web book, The Truth about Teambuilding.
[2] Available for free from the Social Science Research Network: https://papers.ssrn.com/sol3/papers.cfm?abstract_id=3306206.
[3] Coch, L., & French, J. R. P. (1948). Overcoming Resistance to Change. Human Relations, 1(4), 512–532. https://doi.org/10.1177/001872674800100408
[4] Anne Goldgar, “Tulip Mania: The Classic Story of a Dutch Financial Bubble Is Mostly Wrong,” The Conversation, 2018, http://theconversation.com/tulip-mania-the-classic-story-of-a-dutch-financial-bubble-is-mostly-wrong-91413.
[5] Scott Barry Kaufman, “The Role of Luck in Life Success Is Far Greater Than We Realized,” Scientific American Blog Network, March 1, 2018, https://blogs.scientificamerican.com/beautiful-minds/the-role-of-luck-in-life-success-is-far-greater-than-we-realized/.
[6] Eric Barker, “This Is How To ‘Work Smarter Not Harder’: 3 Secrets From Research,” Barking Up The Wrong Tree (blog), February 28, 2018, https://www.bakadesuyo.com/2018/02/work-smarter-not-harder-2/.
[7] Erica Dhawan and Tomas Chamorro-Premuzic, “How to Collaborate Effectively If Your Team Is Remote,” Harvard Business Review, February 27, 2018, https://hbr.org/2018/02/how-to-collaborate-effectively-if-your-team-is-remote.
[8] Liz Ryan, “Ten Skills Every Manager Needs — But 90% Of Managers Lack,” Forbes, 2018, https://www.forbes.com/sites/lizryan/2018/02/11/ten-skills-every-manager-needs-but-90-of-them-lack/.
[9] Nicholas Hune-Brown, “Why It’s so Hard to Actually Work in Shared Offices,” The Walrus (blog), February 15, 2018, https://thewalrus.ca/why-its-so-hard-to-actually-work-in-shared-offices/.
[10] Chris Jager, “24 Cognitive Biases You Need To Stop Making [Infographic],” Lifehacker Australia, March 21, 2018, https://www.lifehacker.com.au/2018/03/find-out-which-cognitive-biases-alter-your-perspective/.
[11] Allan Lee, Sara Willis, and Amy Wei Tian, “When Empowering Employees Works, and When It Doesn’t,” Harvard Business Review, March 2, 2018, https://hbr.org/2018/03/when-empowering-employees-works-and-when-it-doesnt.
[12] Sara L. Rynes, Tamara L. Giluk, and Kenneth G. Brown, “The Very Separate Worlds of Academic and Practitioner Periodicals in Human Resource Management: Implications for Evidence-Based Management,” Academy of Management Journal 50, no. 5 (2007): 987–1008.
[13] Steven H. Appelbaum et al., “Back to the Future: Revisiting Kotter’s 1996 Change Model,” Journal of Management Development 31, no. 8 (August 10, 2012): 764–82, https://doi.org/10.1108/02621711211253231.
[14] John Kotter, Leading Change (Boston, Mass.: Harvard Business School Press, 1996).
[15] Rynes, et al. 2007, op cit.
[16] Chuck Williams, MGMT9: Principles of Management, Ninth edition (Boston, MA: Cengage Learning, 2017).
[17] Steven D. Charlier, Kenneth G. Brown, and Sara L. Rynes, “Teaching Evidence-Based Management in MBA Programs: What Evidence Is There?,” Academy of Management Learning & Education 10, no. 2 (2011): 222–236.
[18] Eric Barends, Denise M. Rousseau, and Rob B. Briner, “Evidence-Based Management: The Basic Principles” (Amsterdam, The Netherlands: Center for Evidence Based Management, 2014).
[19] Nesta and Alliance for Useful Evidence, “Using Research Evidence for Success: A Practice Guide” (Nesta/Alliance for Useful Evidence), accessed March 21, 2018, https://www.alliance4usefulevidence.org/assets/Using-Research-Evidence-for-Success-A-Practice-Guide-v6-web.pdf.
[20] Roye Werner, “Buried Treasure: Finding the Evidence Today,” in Handbook of Evidence-Based Management: Companies, Classrooms, and Research. New York: Oxford, 2012, https://www.cebma.org/wp-content/uploads/Werner-Buried-Treasure.pdf.