Specialists vs Generalists -the road ahead?
David J. Epstein:
Range: Why generalists Triumph in a Specialized World.
Riverhead Books, New York, 2019
This book – Range – is one that has created a lot of discussion lately. Locally here in Finland, both the country’s biggest newspaper, Helsingin Sanomat, and the local equivalent of Times, Suomen Kuvalehti, have published articles about this book recently. Internationally, the same applies, here are just a few examples from NY Times and the Guardian.
In my social media feed, most people that have been commenting on this book and the recensions have been very positive, almost enthousiastic. Most of these comments seem to focus around two different approaches.
Some people seem personally touched by the book. The book tells you that you do not have to specialise early, you can succeed perfectly well even if you are a generalist or have tried many things in your life. For many (who probably didn’t practice just one thing as intensively as Tiger Woods playing golf and devoting his life to it since he was two) this acts almost as a restoration of their self-esteem. Suddenly someone proves that generalists, who often find their path late in life, are the ones who have the better chances at succeeding. This is the case especially if your field is complex and unpredictable. Epstein also shows that generalists are more creative, more agile, and more able to make connections than their specialised peers. I saw many comments of the type “finally someone gets me!” or “maybe my first degree in X wasn’t a waste of time after all…”.
The second approach, which many people in my social media feed have been reacting to, is the fact that the book tells us that we should all have more art in our lives. Epstein says that scientists and members of the general public are about equally likely to have artistic hobbies, but scientists inducted into the highest national academies are much more likely to have this kind of interests. And those who have won the Nobel Prize are more likely still. Compared to other scientists, Nobel laureates are at least twenty-two times more likely to partake in art; in many forms. Nationally recognized scientists are much more likely than other scientists to be musicians, sculptors, painters, printmakers, woodworkers, mechanics, electronics tinkerers, glassblowers, poets, or writers, of both fiction and nonfiction. And, again, Nobel laureates are far more likely still.
Both of these points were emphasised in the book. I agree with them, but I still found a couple of other points the book makes, more important.
I find two other main points particularly interesting. Firstly, the discussion around how the whole question setting around us is changing, to kind and wicked questions, is one I have heard of before but I think Epsteins book is good at capturing what kind of abilities and challenges are involved in both of these settings.
Secondly, I also thought the discussion of what is needed if we want to be more creative very enlightening. We need more foresighting and practice in seeing beyond the existing perspectives. But we also need to allow for more abuiguity, incongruence and paradoxicality, which is not easy in todays structured world and within the processes we love so dearly.
I’ll try to open both of these viewpoints a little more.
Epstein is critical to the current trend of (only) deep specialisation. He claims that increasing specialization has created a “system of parallel trenches” in the quest for innovation. Everyone is digging deeper into their own trench and rarely standing up to look in the next trench, even though the solution to their problem happens to reside there. The challenge we all face is how to maintain the benefits of breadth, diverse experience, interdisciplinary thinking, and delayed concentration in a world that increasingly incentivizes, even demands, hyperspecialization.
Epstein, following many other scholars, divides our environments into kind and wicked. In kind environments, where the goal is to re-create prior performance with as little deviation as possible, teams of specialists work superbly. When the path is unclear— Epstein compares this to a game of Martian tennis—those same routines no longer suffice. If we use our old tools, we might end up with incremental improvements rather than true innovation.
The division of kind and wicked environments, also relates to the future usage of AI (see for example previous discussion by Lee or Tegmark ). In a kind environment, where the settings for the task is stable, it is a lot easier to define an algorithm that could perform the task. Epstein refers to the world famous chess-player, Garry Kasparov, who has said that “Anything we can do, and we know how to do it, machines will do it better,” and “If we can codify it, and pass it to computers, they will do it better.”
For chess, this has already been the case. We now have algorithms that play chess better than humans. What was really interesting to know, and new to me, was that since then, chess tournaments have been played with teams where all users are allowed to use any AI support they want. In chess, this changed the pecking order instantly. “Human creativity was even more paramount under these conditions, not less,” according to Kasparov. The new winners were those who could utilise these abilities in the most creative way, when the old benefit of knowing many game strategies by heart was lost.
This to me gives hope for the future – when we do not need to memorize long strings of data (which computers clearly do better than us), the winners are those who can utilize the computing power in the most creative way. The more a task shifts to an open world of big-picture strategy, the more humans have to add. The game’s strategic complexity provides a lesson: the bigger the picture, the more unique the potential human contribution. Our greatest strength is the exact opposite of narrow specialization. It is the ability to integrate broadly, says Epstein.
This leads us back to Moravec’s paradox: machines and humans frequently have opposite strengths and weaknesses. (And if you don’t remember what Moravec’s paradox was: “it is comparatively easy to make computers exhibit adult level performance on intelligence tests or playing checkers, and difficult or impossible to give them the skills of a one-year-old when it comes to perception and mobility”)
Not just games, in open ended real-world problems we’re still crushing the machines. Maybe our current inclination to specialisation is a product of industrialisation and we are heading back towards more of a generalist approach? Or maybe increased utilisation of AI is actually allowing us to be more creative?
Epstein claims that when narrow specialization is combined with an unkind domain, the human tendency to rely on experience of familiar patterns can backfire horribly—he gives examples of expert firefighters who suddenly make poor choices when faced with a fire in an unfamiliar structure, or the people in the Challenger project at NASA. The very tool that had helped make NASA so consistently successful, what Diane Vaughan called “the original technical culture” in the agency’s DNA, didn’t work in an unfamiliar situation. Hints about the weaknesses that caused the Challenger to crash had been noticed, but because they couldn’t be specified enough, they weren’t really considered. Reason without numbers was not accepted. In the face of an unfamiliar challenge, NASA managers failed to drop their familiar tools.
Rather than adapting to unfamiliar situations, whether airline accidents or fire tragedies, organizational behaviour expert Karl Weick saw that experienced groups became rigid under pressure and “regress to what they know best.”
Epstein also claims that universities currently aren’t grasping this change. He refers to James Flynn, a professor of Political Science, saying that“Even the best universities aren’t developing critical intelligence,” and “They aren’t giving students the tools to analyse the modern world, except in their area of specialization. Their education is too narrow.” He does not mean this in the simple sense that every computer science major needs an art history class, but rather that everyone needs habits of mind that allow them to stretch across disciplines. Or as Doug Altman put it, “Everyone is so busy doing research they don’t have time to stop and think about the way they’re doing it.” Specific academic departments are generally not big fans of generalisation. They want students to take more specialized classes in a single department, and not be broad. This might be a challenge we need to overcome in the future, in one way or the other.
According to education economist Greg Duncan, “jobs that pay well increasingly require employees to be able to solve unexpected problems, often while working in groups. These shifts in labour force demands have in turn put new and increasingly stringent demands on schools.”
Making connections and interleaving makes knowledge flexible, and useful for problems that never appeared in training. Another important component is experimentation. Epstein shows many examples of people, who find pure joy in this experimentation. If they cannot experiment enough in their own job, they do it on a Saturday, just out of the pure pleasure to be able to try new things.
So specialists work very, very well on well-defined and well-understood problems, but as ambiguity and uncertainty increases, which is the norm with systems problems, breadth and the ability to try new things becomes increasingly important. The successful adapters are excellent at taking knowledge from one pursuit and applying it creatively to another, and at avoiding cognitive entrenchment. They employed what psychologist Robin Hogarth called a “circuit breaker.” They drew on outside experiences and analogies to interrupt their inclination toward a previous solution that may no longer work. Their skill was in avoiding the same old patterns. In the wicked world, with ill-defined challenges and few rigid rules, range can, according to Epstein, be a life hack.
In addition to experimentation, the usage of analogies, and the ability to create many of them, is important. Our natural inclination to take the inside view can be defeated by following analogies to the “outside view.” The outside view probes for deep structural similarities to the current problem in different ones. The outside view is deeply counterintuitive because it requires a decision maker to ignore unique surface features of the current project, on which they are the expert, and instead look outside for structurally similar analogies. It requires a mindset switch from narrow to broad.
Sometimes this approach doesn’t need much. Epstein refers to a case where just being reminded to analogize widely made business students more creative.
In addition to making individuals more broad-minded, it also helps to put people from different backgrounds to work together. In the research project of psychologist Kevin Dunbar, the labs in which scientists had more diverse professional backgrounds were the ones where more and more varied analogies were offered, and where breakthroughs were more reliably produced when the unexpected arose.
In the lone lab that did not make any new findings during Dunbar’s project, everyone had similar and highly specialized backgrounds, and analogies were almost never used. “When all the members of the laboratory have the same knowledge at their disposal, then when a problem arises, a group of similar minded individuals will not provide more information to make analogies than a single individual,” Dunbar concluded.
So exploration is not just a whimsical luxury; it is a central benefit. Some unanticipated experience leads to an unexpected new goal or the discovery of an unexplored talent.
This is something we do naturally when we are young, but have to fight for to remember later on in life. Adults tend to become more agreeable, more conscientious, more emotionally stable, and less neurotic with age, but less open to experience.
Epstein claims that there is, to be sure, no comprehensive theory of creativity. But there is a well-documented tendency people have to consider only familiar uses for objects, an instinct known as functional fixedness. He exemplifies this with the “candle problem,” in which participants are given a candle, a box of tacks, and a book of matches and told to attach the candle to the wall such that wax doesn’t drip on the table below. Solvers try to melt the candle to the wall or tack it up somehow, neither of which work. When the problem is presented with the tacks outside of their box, solvers are more likely to view the empty box as a potential candle holder, and to solve the problem by tacking it to the wall and placing the candle inside. For some people, the tacks were always outside the box.
But we need different kinds of people – not for everyone to switch from specialisation to generalisation. Lateral and vertical thinkers are best together, even in highly technical fields.
The eminent physicist and mathematician Freeman Dyson styled it this way: we need both focused frogs and visionary birds. This approach is supported by many studies in the book.
Andy Ouderkirk and his small team studied inventors at 3M. They found both very specialized inventors who focused on a single technology, and generalist inventors who were not leading experts in anything, but had worked across numerous domains. Their data suggest that narrowly focused specialists in technical fields are still absolutely critical, it’s just that their work is widely accessible, so fewer might suffice. Oudekirk says that when information became more widely disseminated, it became a lot easier to be broader than a specialist, and to start combining things in new ways.
A similar view is presented by University of Utah professor Abbie Griffin in her work on modern Thomas Edisons—“ serial innovators,” as she and two colleagues have termed them. Their findings about who these people are similar: “high tolerance for ambiguity”; “systems thinkers”; “additional technical knowledge from peripheral domains”; “repurposing what is already available”; “adept at using analogous domains for finding inputs to the invention process”; “ability to connect disparate pieces of information in new ways”; “synthesizing information from many different sources”; “they appear to flit among ideas”; “broad range of interests”; “they read more (and more broadly) than other technologists and have a wider range of outside interests”; “need to learn significantly across multiple domains”; “Serial innovators also need to communicate with various individuals with technical expertise outside of their own domain.” Abbie Griffin and her coauthors also lift the need for “π-shaped people” who dive in and out of multiple specialties.
In addition to experimentation, the ability to make analogies, and a broad skillset, Epstein also lifts the need for forecasting. This I find really interesting. In my opinion, this is something that hasn’t been lifted so much in other books. There are of course plenty of books out there on how to do forecasting as part of company strategy building. But I believe that Epstein is talking about something different. He positions foresighting not just as an activity done by a specific unit in an organisation, but more as a general trait, that anyone can have and train.
Scott Eastman described the core trait of the best forecasters to be as: “genuinely curious about, well, really everything.” Agreement is not what they are after; they are after aggregating perspectives, lots of them. Similarily, a hallmark of interactions on the best teams is what psychologist Jonathan Baron termed “active open-mindedness.” Law and psychology professor Dan Kahan also documented a similar personality feature: science curiosity. Not science knowledge, science curiosity.
Just as Philip Tetlock says of the best forecasters, it is not what they think, but how they think. The best forecasters are high in active open-mindedness. They are also extremely curious, and don’t merely consider contrary ideas, they proactively cross disciplines looking for them. “Depth can be inadequate without breadth,” wrote Jonathan Baron, the psychologist who developed measurements of active open-mindedness.
Some people tend to see simple, deterministic rules of cause and effect framed by their area of expertise, like repeating patterns on a chessboard. Some see complexity in what others mistake for simple cause and effect. They understand that most cause-and-effect relationships are probabilistic, not deterministic. There are unknowns, and luck, and even when history apparently repeats, it does not do so precisely. They recognize that they are operating in the very definition of a wicked learning environment, where it can be very hard to learn, from either wins or losses.
Tetlock’s and Barbara Mellers’s research group showed that with just an hour of basic training in these habits we can become better at looking at the future.
The other interesting insight that Epstein makes, is how unconfortable it can often be to go beyond existing boundaries. He talks of “congruence”, a social science term for cultural “fit” among an institution’s components—values, goals, vision, self-concepts, and leadership styles. Since the 1980s, congruence has been a pillar of organizational theory. An effective culture is both consistent and strong. When all signals point clearly in the same direction, it promotes self-reinforcing consistency, and people like consistency.
Still, the most effective leaders and organizations have range; according to Epstein they were, in effect, paradoxical. They could be demanding and nurturing, orderly and entrepreneurial, even hierarchical and individualistic all at once. A level of ambiguity, it seemed, was not harmful. In decision making, it can broaden an organization’s toolbox in a way that is uniquely valuable.
Philip Tetlock and Barbara Mellers showed that thinkers who tolerate ambiguity make the best forecasts; one of Tetlock’s former graduate students, University of Texas professor Shefali Patil, spearheaded a project with them to show that cultures can build in a form of ambiguity that forces decision makers to use more than one tool, and to become more flexible and learn more readily. The managers were benefitting from incongruence. The formal, conformist company process rules were balanced out by an informal culture of individual autonomy in decision making and dissent from the typical way of doing things. Business school students are widely taught to believe the congruence model, that a good manager can always align every element of work into a culture where all influences are mutually reinforcing—whether toward cohesion or individualism. But cultures can actually be too internally consistent. With incongruence, “you’re building in cross-checks,” according to Tetlock.
If they were used to improvising, encouraging a sense of loyalty and cohesion did the job. The trick was expanding the organization’s range by identifying the dominant culture and then diversifying it by pushing in the opposite direction.
The challenge of course is, as Epstein puts it, that work that builds bridges between disparate pieces of knowledge is less likely to be funded, less likely to appear in famous journals, is more likely to be ignored upon publication, and then more likely in the long run to be a smash hit in the library of human knowledge.
This is naturally something we can all work with. How can we “push for the uncomfortable”, when it would be so much easier to just keep doing what we already do? Even if we know that areas outside our comfort zone could be more fruitful?
Maybe one way to tackle it is just to keep doing experiments. To find the time for them, even if it means doing them outside the normal working hours. To dare to do them in areas where we are not yet sure of their success. And to make sure we land in environments with a diverse set of people, with various backgrounds, traits, viewpoints and values.
Experiments do not need to succeed; the important thing is to do them. Even an experiment that didn’t immediately lead to something isn’t wasted, according to Epstein, it will add on our ability to make new connections later on.
And experiments do not necessarily need to be easy, either. Epstein shows that projects that are difficult and frustrating, can in fact ensure that we actually learn more from them, and more “deep learning” and connection making actually happens.
To me, this is a lot more inspiring than the easy “you don’t need to specialise” or “do more art” quotes so many people cherish from this book. I take it as a personal invitation, for all of us, to just keep doing stuff that might seem weird when we first propose it, and not to get dis-hearted by challenges. Trying and failing is way better than not doing anything at all…
Anva 