The Research-Practice Gap

Note: This was published as part of my bi-monthly column in the ACM CHI magazine, Interactions. I urge you to read the entire magazine -- subscribe. It's a very important source of design information. See their website at interactions.acm.org. (ACM is the professional society for computer science. CHI = Computer-Human Interaction, but better thought of as the magazine for Interaction Design). This article was printed in Interactions, Volume 17 Issue 4, July + August 2010.

"Oh, East is East, and West is West, and never the twain shall meet." (Rudyard Kipling, 1892. Barrack-room ballads)

There is an immense gap between research and practice. I'm tempted to paraphrase Kipling and say "Oh, research is research, and practice is practice, and never the twain shall meet," but I will resist. The gap between these two communities is real and frustrating. Sometimes the gap is deliberate. Some researchers proudly state that they are unconcerned with the dirty, messy, unsavory details of commercialization while also complaining that practitioners ignore them. And some practitioners deride the research results as coming from a pristine ivy tower, interesting perhaps, but irrelevant for anything practical. Sometimes the gap is accidental, caused by a misunderstanding by both sides of the requirements and goals of the other. I have heard researchers who would like their ideas to impact practice complain that when their ideas do get used, the practitioners do it wrong, leaving out (or messing up) the most critical aspects. Practitioners, in turn, complain that the research results, even if relevant, are not in any form that can readily be translated into practice.

The gap between researchers and practitioners extends to the professional societies. The major societal home for many researchers is the Association for Computing Machinery's Special Interest Group on Computer-Human Interaction: ACM SIGCHI, the same group that brings you this magazine. Although CHI pretends that it is home for both researchers and practitioners, it is mainly deluding itself. In the major conferences, most especially the flagship CHI conference, CHI proudly proclaims that the conference includes people from both universities and industry. Although this is true, the people from industry are seldom the developers and practitioners. Instead, they are primarily researchers who work for industrial research labs. Researchers in companies tend to be far more closely attuned to their academic brethren than to the people within the product divisions of their own companies. This close connection to research and separation from practice marks the research community and the CHI conferences. This magazine, Interactions, has made a valiant and reasonably successful attempt to bridge the gap, but the fundamental distinction remains strong. I know this problem well because I faced it when I headed a large research group in a product company (the Advanced Technology Group at Apple).

The gap between research and practice is fundamental. The knowledge and skill sets required of each group differ. Consider the research community within design: the area called design research. This community attempts to understand basic patterns of human and social behavior and how these are impacted by technology. Most of the studies focus upon the problems and difficulties, in part because these are far easier to study than the benefits and changes in work and life patterns, but also because new technologies are mostly accompanied by problems and most benefits do not show up for a long time, perhaps decades. Other researchers probe the technological boundaries, demonstrating new potential capabilities, and new experiences. Both these kinds of research are valuable. Both produce new insights. But both are far removed from the intense attention to detail, reliability and robustness that characterizes products or the concern with how and what people actually buy, with the cost structure of potential products and the resulting profitability. Practitioners do not have time to deal with debates about the problems and difficulties that people face with new technologies. They do want to hear about the benefits and the new product directions to pursue. Studies of technology-induced alienation or concerns (or lack of concern) with privacy might be of great importance for society, but not for driving the next product cycle.

Even when some research demonstration excites the product side of a company, it is seldom ready for release. Transforming a research demonstration into a practical product that can be sold profitably in the marketplace is a complex and demanding job, a job for which the research community has neither the skills, patience, nor interest in doing. The skill sets that make for a creative, insightful researcher are very different than the skills required of the development engineers to make something work reliably and inexpensively or of the marketing teams who must determine not what people actually need (which is where researchers tread), but what they will actually purchase. Product people have to worry about sales and profitability, reliability and cost. These issues are seldom of interest to researchers and, moreover, are not within their normal skill sets.

I emphasize the difference in skills required of the different activities to ensure that these comments are not taken as criticism. They are meant to reflect the reality that it is rare for a single individual to have the breadth and depth of knowledge and skills to understand business plans and marketing strategy, to know how to lead a team of perhaps hundreds of developers to produce a reliable, bug-free set of code with millions of lines of instructions, that can work across the many platforms and perturbations of equipment and applications found in the real world, while simultaneously making use of all the advanced research learnings of the multiple relevant disciplines: the social sciences, business, and technology. This is why I think the research-practice gap is so universal and so difficult to overcome.

Re-examining the basics of design

The research-practice gap is only one of many problems facing the design profession. One other problem is that many of our basic beliefs about how to develop and design are built upon a shallow, insecure foundation. In the many years I have been writing this column, I have reconsidered some most cherished beliefs in the practice of design and found many of the principles wanting. We know surprisingly little about how to do design. There is no science of the practice in the same sense that there is a science to the structural analysis of buildings and bridges, or to the building of circuits. Design is still an art, taught by apprenticeship, with many myths and strong beliefs, but incredibly little evidence. We do not know the best way to design something. The real problem is that we believe we do. Beliefs are based more on faith than on data.

This is a problem that confronts all professional disciplines: law, art, music, business, medicine, and design. Each of these disciplines often has some scientific field behind it (e.g., art and music has perceptual psychology, interaction design has well-established psychological roots, many parts of business have a basis in decision theory, economics, and finance, and medicine has biology and chemistry. But even in the fields with a substantive scientific basis, the practical applications to the daily practice are very limited. Thus, although biology is important as a foundation for medicine, it gives no guidance to patient-doctor interaction, to the taking of patient histories, or to diagnosis, nothing to say about patient empathy or best hospital practices. In business, finance and economics provide a rationale for some kinds of investment decisions, but where do best management principles come from? In law, what science underlies jury selection or presentations? Music has lots of theory, but very little is directly relevant to music performance. In the end, practical disciplines are all taught by apprenticeship, internships, residencies, and long periods of training.

In science, there are clear links among hypotheses, conclusions, and evidence. But in the practices of most professions, the links are tenuous at best. Instead, there is much reliance upon "best practice," where "best" is often defined by short-term measurements, usually of variables that are easy to measure as opposed to those that are of most significance. Long-term measures are seldom taken. Methods are seldom compared. Note that it is not easy to figure out how to do these studies or comparisons: once again, this is not meant to be a criticism. It is meant to describe the current state of affairs. Scientists usually operate in what has been called "white room" conditions, carefully forming abstract characterizations of the phenomena under consideration and studying them in a controlled research environment or the clean precision of the laboratory. Similarly, the theories are of necessity simplified and abstracted to a pristine form of mathematical or simulation models. Science works best when all the variables are understood and controlled. The real world is complex and messy, with uncontrolled variables, sometimes behaving in ways that contradict the neat, tidy, logical assumptions of the scientist. No wonder there is a gap.

The lack of scientific studies of practice is due to two things: First, practitioners are not trained in scientific research. They do not understand the need for experimental controls nor do they understand statistical variability and experimental biases. Moreover, they don't wish to: they want to get on with their work. Second, even when researchers well versed in experimental methods attempt to study practices, they discover that the very nature of a practical discipline throws in so many idiosyncratic variables that rigor is simply not possible.

In the field of design, many researchers end up studying the designers themselves. "How do designers think?" is a standard research question. I have seen many studies comparing individuals with groups, or people in one culture with those of another. All of these studies make for interesting reading, but I find them of little value in helping us know how designers ought to work or how they ought to think. As a result, we have many myths about the power of design research, brainstorming, rapid prototyping, iterative test and design, but zero evidence.

When researchers try to collect evidence, they often will take a bunch of students, do some simple manipulations and then try to state a general conclusion. The entire study lasts a few hours, or at most an academic quarter or semester. I am continually amazed that the research community believes that the study of naïve, unskilled students tells us anything at all about the practical problems of design in a large company, with multiple constraints and requirements, working in teams, with highly practiced and accomplished skills. Moreover, real design projects take months and sometimes years. The difference between the researchers notion of the design setting and reality is immense.

Misunderstandings: The Case of "Technology first, needs last"

In my column "Technology first, needs last" (Interactions, March+April, 2010), I stated that design research was quite effective at improving existing products both for their intended uses and also to move them into new unexpected application areas, but that it played little or no role in original invention. New technology occurred first, I argued, with inventors doing their thing without the benefit of any design research, with few people besides the inventors believing there was a need. This blind, research-free invention often fails, but it is also where our major innovations have come from. This approach has led to such major innovations as the telephone, phonograph, radio, automobile, internet, CD, portable music player, and camera. Design researchers had nothing to do with the initial developments. Instead, the researchers have come along afterwards where they have sometimes made valuable contributions, demonstrating how the product could be improved for its intended usage and, more importantly, by noticing needs not satisfied by the existing products, how it might serve a vastly different audience than was originally intended.

The traditional folklore of the research and development is that there is a smooth, steady chain from pure, basic research, to more applied research, to advanced products, to commodity products. This nice logical progression is false, as a large number of studies of research and development have shown. The myth still persists.

Donald Stokes's book, Pasteur's Quadrant, provides a nice antidote. Here, Stokes argues that the most effective research is what Pasteur did when he developed the smallpox vaccine. He started with a real, practical problem, realized that it needed some fundamental scientific advances before it could be solved, did the science, and then applied it back to the problem. In other words, the research is done in search of solutions to real problems, or what Stokes calls "use-inspired basic research." Stokes argued that research can often be characterized along two dimensions. The first dimension is about the kind of knowledge that is sought: fundamental or practical. The second dimension concerns consideration of use: whether it is a search for pure knowledge without consideration of use or whether it is aimed at some fundamental, practical problem.

The two dimensions give rise to four quadrants. Those who seek fundamental knowledge without consideration of how it might be used fit the general view of the longhaired, impractical mathematician or scientist. Inventors such as Thomas Edison fit the quadrant of searching for relevant knowledge to solve an applied problem, but without any attempt to expand our general understanding of phenomena. Hence, Edison's classic search for the material that would improve the already existing light bulb, allowing it to function more efficiently for greater duration, is a classic example. Although he succeeded in his quest, he did not advance our understanding of science or engineering. Edison provides an example of someone who did not attempt to add to our fundamental understanding, but was consumed with making sure his inventions were practical and useful. He did read the scientific literature, but he did not try to add to it. A third quadrant is filled with tinkerers who produce inventions that neither add to fundamental understanding nor have any use.

For Stokes, the most powerful quadrant is not that of the pure scientist. Rather it is the quadrant occupied by Pasteur, the quest for fundamental knowledge within a specific use context. This is where the biggest payoffs lie, at least so argued Stokes.

In the four quadrants formed by the axes of pure versus applied science, usages versus no use, researchers most often play in the fun quadrant, finding lovely problems to work on without regard for whether anyone cares outside of their fellow research in-group. This is one reason for the research-practice gap. I recommend aiming at Pasteur's quadrant, fundamental research aimed at solving important applied problems.

But even if researchers aim at the solution to practical problems, they still face the fundamental differences in the knowledge and skill sets required by those who conduct the research and those who attempt to translate those results into practical, reliable, and affordable form.

Translational Development

Between research and practice a new, third discipline must be inserted, one that can translate between the abstractions of research and the practicalities of practice. We need a discipline of translational development. Medicine, biology, and the health sciences have been the first to recognize the need for this intermediary step through the funding and development of centers for translational science. This intermediate field is needed in all arenas of research. It is of special importance to our community. We need translational developers who can act as the intermediary, translating research findings into the language of practical development and business while also translating the needs of business into issues that researchers can address. Notice that the need for translation goes in both directions: from research to practice and from practice to research.

Translational developers are needed who can mine the insights of researchers and hone them into practical, reliable and useful results. Similarly translational developers must help translate the problems and concerns of practice into the clear, need-based statements that can drive researchers to develop new insights. Neither direction of translation is easy.

Great innovations can come from anywhere, any place. Usually they come about when a new technology is unleashed upon the world and inventors and technologists scurry to find something they can do with it. Most of these attempts fail, but a few stick. The researchers come aboard after the technology has been unleashed. But this is precisely when they can be most effective, because it is now that they can play Pasteur's game, starting with a real need, figuring out what the scientific needs are, doing the science, and then feeding the results back to a practitioner community that is desperately awaiting the findings.

There is a huge gap between research and practice. To bridge the gap we need a new kind of practitioner: the translational developer. The gap is real, but it can be bridged.

Reference:

Stokes, D. E. (1997). Pasteur's quadrant: Basic science and technological innovation. Washington D C: Brookings Institution Press.

Don Norman wears many hats, including co-founder of the Nielsen Norman group, Professor at Northwestern University, Visiting Professor at KAIST (South Korea), and author, his latest book being Living with Complexity (To be published by MIT Press in September, 2010). He lives at jnd.org.

There is a huge gap between research and practice. To bridge the gap we need a new kind of practitioner: the translational developer. The gap is real, but it can be bridged.

Column written for Interactions. © ACM. This is the author's version of the work. It is posted here by permission of ACM for your personal use. It may be redistributed for non-commercial use only, provided this paragraph is included. The definitive version will be published in Interactions.