Why Human Systems Integration Fails (And Why the University Is the Problem)

Invited talk for the 30th anniversary of the Human-Systems Integration Board of the National Research Council, the National Academies. Washington, DC. December 2, 2010.

In preparation for this talk I read Nancy Cooke and Frank Durso's book "Stories of Modern Technology Failure and Cognitive Engineering Successes." It is an excellent book, but after I finished it, I was feeling even more discouraged than when I had started. Why am I so discouraged by a book that glorifies the successes? Because the successes should have been unnecessary. They were rescues.

Imagine this newspaper headline: "Heroic Firefighters Rescue 25 People trapped on Roof!"

Celebrate the firefighters? Sure, but why was there a fire in the first place? And why were the people trapped?

I consider the book to be a tragic reminder that we are still in fire-fighting mode when we should be in fire-prevention mode. This problem was not lost on the authors, Cooke and Durso. The book consists of an introduction followed by seven chapters of successful rescues, one per chapter. Then in Chapter 9, the authors interview the protagonists of each of the seven rescues, letting them reflect upon their accomplishments and their attempts to establish their procedures into the normal design or operational cycle. In a few cases, their procedures were formalized and adopted. In most, however, this it was a one-shot rescue. So, most of the HSI researchers went back to their laboratories, the better to await the next fire. Finally, in Chapter 10, Bill Howell explores the issues that led to the problems in the first place, attempting to understand why the rescue had been necessary.

What was the cause of the problem? Why was it necessary to call in the rescuers? Howell puts the blame on several causes:

**The commonsense illusion. **

The belief that what human/factors and Cognitive Engineering does is simply "common sense," so it is not necessary to call in experts.

**The labeling and identity problem. **

We have so many different names for what we do that people just give up in confusion: Human Factors, Ergonomics, HCI, CSCW, Cognitive Engineering, and Human-Centered Design.

This group has too many names. So I will simply call it HSI, Human-Systems Integration. You may wish to substitute Human Factors, Ergonomics, HCI, Cognitive Engineering, Cognitive Science, Applied Psychology - Whatever name you wish to use, my arguments remain the same.

**The tracing and attribution problem. **

The applied problem fixer gets more credit than the fundamental researcher who provided the groundwork for the fix, so funding for the research programs languishes.

**The abstractness, lack of pizzazz, and credibility problem. **

Finding fault with designs or fixing things that are broken never gets the same respect as fancy new gimmicks with high-stated capabilities (a fully automated, whiz at traveling at Mach 6.3, and capable of delivering its payload with precision, low cost, and negative energy consumption, using the new hyper-fusion, regenerative quantum tunneling OLPSN-MOD engine with hyper-nano planishing.)

**The marketing problem. **

Nobody knows we are here, nor what we do. If quantum computing and nuclear fission can gain the respect of the world, why can't we? We can actually accomplish things whereas they can only promise.

I believe that all these causal factors do play a role, but I do not believe any of them are the fundamental issue. I believe we have far more fundamental issues. If anyone is to blame for our lack of success it is we who are to blame. We, the field of Human-Systems Integration.

WHY HUMAN-SYSTEMS INTEGRATION FAILS

There are many reasons for our failure. One problem is that we are not an engineering discipline. We all try hard to be scientists. A second problem is the lack of good, solid practical, handbook-like knowledge. The third is the lack of design skills - we are analyzers, not synthesizers in a world that needs synthesis. And while I am at it, let me blame the educational profession for championing the abstract over the particular, research over application, and narrowness over breadth.

People who apply findings are practitioners, yet almost everyone in this field is a researcher. There is a huge research -practice gap, which I have written about elsewhere. We may all know of Pasteur's Quadrant, but we forget that even this quadrant is for research, not for practice.

I have even more factors on my blame list, but these will give you an idea of what you are about to hear.

The Research-Practice Gap

The talents and skills required of one differ from those required of the other. We need a translational engineering group to translate from one to the other. There are numerous publications on this topic. Mine is one my jnd.org website.

We see this problem in NRC's reports. The Human-Systems Integration community (HSI) produces many studies that are correct, important, and useless. Why useless? Because they get read by others in the HSI community, people who already agree, and who have similar views and complaints. The critical people who need to know never read them, but even if they did, they would be unlikely to make a difference. Why? Because the papers are research papers, pointing out problems. They are analytical, whereas what system implementers need are synthesizing papers -- design papers.

System developers do not want to know what the problems are. They want to know what the solutions are. it is time the HSI community moved from being complainers and analyzers to being proactive designers and solution-givers.

I realize that this is not a message the community wants to hear. And I also realize that there are a few exceptions (mostly in consumer software and commercial and military aviation, and conceivably in some military programs, e.g., MANPRINT), but the exceptions are lost among the huge litany of problem areas. Doesn't the HSI community - that you -- wonder why it has have been ignored for so long? The HSI community produces paper after paper, book after book, wondering why they are only called upon after the fact -- as heroic fireman rescuing people from fires. But we need people to make it so that the fires never happen. HSI doesn't do that. HSI explains why the fire happened. Too late, guys.

The Narrow Specialties of the University

Disciplines are deep and narrow. Real problems are broad. So the practitioners must have broad, general knowledge, while employing narrow deep specialists.

The university rewards expertise in a narrow topic, not broad general knowledge. Thus, HSI requires knowledge from the social sciences, design, and engineering, but in the university they are separated by departments and by schools and in the National Academies, by divisions and Institutes. The university reward system: Theory, not practice, even in engineering. The National Academies mostly follows this trend.

The Organization of Companies, the National Academies, and NRC

The Organization of the National Research Council (NRC) is part of the problem. The National Academies and NRC perpetuate the problem by having divisions organized by discipline rather than by problems. Yes, studies are about problems, but they have limited interdisciplinary composition, usually one member from whatever groups members think are most relevant, but with the home discipline being dominant. NRC should be organized by problems, not disciplines. So HSI should have computer scientists, civil, mechanical, industrial and electrical and computer engineers (and scientists), as well as the existing expertise in the behavioral and social sciences. And medicine, and security and ...
The separation of the engineering and social sciences is especially distressing because engineering applications are for the benefit of people, so a human-centered approach is essential in such disciplines as chemical, civil, computer, industrial and mechanical engineering, but of these disciplines, only computer science has permitted the inclusion of human concerns, and even there it is barely tolerated as a minority representative in some departments.

Moreover, the academies, NAE, NAS, and NIM, are primarily research based, even as they staff the problem-based NRC. But even NRC is mostly comprised of researchers who continually lament the lack of theory in most practice. The two boards I know best are HSI and CSTB, and both are good examples of the research-practice gap in action. (I served for 6 years as a member of CSTB and have also been a committee member of many of their report panels. Many members of HSI, I discovered to my great surprise, had never heard of CSTB, even though the areas of study of these two groups have substantial overlap.)

(Note: NAE, the National Academy of Engineering, requires that 50% of its members must come from Industry. Many of those, however, are from the research side of industry. But still, the mandate is encouraging.)

Lack of Project Management Skills in HSI (it's all done by intuition)

Traditional PM focuses upon Time, Money, resources, and scope. No questions about the design effectiveness or appropriateness.

The Assumption that one must first specify requirements, then build: Get the requirements right and the rest is straightforward even if complex. In fact, the requirements are the hardest part of many projects, for they are dynamic, changing even while the project is underway. The times change, the technology changes, and people's needs - and understanding of their needs - changes. Moreover, if the project is successful, it will change people's behavior, therefore changing the requirements.

That programmers and engineers want to build first, analyze later (if ever). But without initial studies and thought, how can one ensure that the correct thing is being constructed, with the correct underlying architecture that will enable modification and expansion with time.

The Traditional Project Management focuses upon schedule, discipline, and cost. It doesn't really care if the product is correct. In order to manage large projects, it uses a regulated process: Stage Gate doesn't work well with projects involving people or organizations because the requirements change as the project continues. The designer's solution is iterative design, always refining the requirements, but this doesn't work well with large projects. But neither does the programmer's favorite, Agile. We do not have good methods of managing large projects especially ones involving people or, for that matter, rapidly changing technology.

The proper role - and the proper form - of testing during design

Designers have the notion of iterative design and test. Engineers and policy makers tend to design from first principles, relegating testing to the end, simply to refine the implementation (i.e., catch errors in implementation)

The design philosophy is to catch errors in formulation of the problem: Big difference.

In addition in large projects, it is often desirable to release it one small piece at a time.

Alas, in some things, it is not possible to do incremental releases (e.g., a large building)

But the real culprit is the combination of the engineering and policy maker's attitude that they can design it properly. This is sometimes specified as, give us the right specs, and we can build it. The problem is that getting the right specs is the hardest part of the job and is often not known until after the job is complete. If known at ail: the more successful the project, the more the specs will change, as a direct result of the project.

HSI needs to develop a special expertise in project management: call it HSI PM. There should be a recommended curriculum, a training program, and a new set of degree courses (e.g., a terminal Masters in Human Systems project management, MHSPM - clearly we need a better acronym).

HSI should mandate (as if it had that power!) that certified HSI PMs be involved in the development of all HSI projects. Of course, this will only work if HSI people are called in at the start of the project instead of to put out the fires.

But this also requires that HSI be up to the task, and as long as HSI is research based, it cannot deliver.

An illustration From the United Nations

Despite agreement about the value of local knowledge (of the cultural variety) among our UN interlocutors, generally speaking agencies still lack an institutional orientation to the generation of such knowledge, or its systematic use in the creation of designs for local action. Further, there appears to be no real framework or method employed to translate local research findings systematically into action on the ground in partnership with local communities, nor a design phase available in the project cycle in which to do this. Despite vast resources dedicated to 'monitoring and evaluation' of projects and programmes on peace and security, there is presently no comparable commitment to the design of projects and programmes. (From Derek B Miller, Lisa Rudnick, and Lucy Kimball (2010, October), Designing Programmes in Contexts of Peace and Security. SE Bulletin, 1 (4), 2-7. www.seeproject.org)

SOLUTIONS

I come not to complain but to explain. Human-systems integration is essential. But it is difficult, difficult for fundamental reasons having to do with human psychology, organizational complexity, and the culture of people, groups, organizations, and society. after all, if the problem were easy, it would already have been solved.

I also come to brainstorm, to think deeply about the fundamental issues that bedevil us. The problem doesn't exist for lack of will, understanding, or perspective (although those issues do exist). The problems exist because there is a fundamental mismatch between the skills of a scholar and a practitioner, between the knowledge base of the specialist and the generalist, and between the world of engineering, logic, and precision, and the messy world of people, societies, and politicians.

Human-Systems Integration is only one piece of this complex puzzle, for systems of all sorts fail, some of which barely involve people. Large, complex systems are essential to modern civilization, But they entail huge problems: integration across the multiple domains necessary to pull it off, the ability to forecast the unforecastable - the unexpected outcomes of large system. And the compromises necessary for budgetary, aesthetic, social, and political needs. The requirements of each specialized discipline often conflict with one another, leading to yet more compromises or, in the worst case, fights, arguments, and dramatically suboptimal results.

Although I do not come to criticize, if I had to pinpoint one root cause of these complex difficulties I would point my figure at the University. I spell University with a capital U, for I mean the modern system of education common throughout the world. The modern university is a few hundred years old. Although at first, it contained many practical disciplines, over time, the search for pure knowledge dominated. Knowledge, and especially depth of knowledge, gains the highest prestige. Quantitative measurement and theories have come to dominate: "If you can't measure it, you can't improve it," is the standard refrain. The result is the quantification of everything, quite often losing the essence of the phenomenon under study.

Moreover, depth of understanding leads to specialization, so much that a specialist in one domain often does not understand other specialists in what is supposedly the same domain, say mechanical engineering.

What has been missed is real world experience. The understanding of how political issues, society, culture, and people's behavior, desires, and wants impact systems. Business issues are critical, for cost is always a critical factor. Engineers tend not to like these issues for they are not rational, logical, or quantifiable. They measure what can be measured and define the rest to be unimportant.

The Area Rule of Knowledge

Over the years I have come to formulate an informal hypothesis about knowledge. The total knowledge a person can have is reasonably constant. Let the knowledge in any domain be represented by a vertical column. Specialists are usually experts in limited number of columns, were the modal number is probably close to one. A specialist is narrow and deep. Generalists in a domain are wider and shallower: the Area Rule states that the total area - breadth times width - is constant. Generalists cut horizontal swatches across the domains, so their knowledge is represented by horizontal rectangles, which by the area rule means that they know a little bit about a lot of topics.

Real problems require both specialists and generalists. We need people expert in their domain, but we also need people who know what domains are required , how to stitch the knowledge from each of them together, and how to translate the language and thought patterns of one domain into another. But generalists never know enough specific information to solve the problem.

Ideally, the university ought to have both forms of people. No. The promotion policies prevent this. Promotion is usually given by worldwide recognition as deep specialists and contributors to some topic domain. Generalists fail the test.

Note that the business schools (and designers) are fond of talking about T-shaped individuals who combine broad knowledge with one specialty. Their heart is in the wrong place, but the notion of alphabetically shaped knowledge and people is badly flawed and therefore, badly misleading. Nonetheless, the emphasis on Ts and other shapes does indicate growing awareness of the problem. Alas, it also seems to be yet another example of naming a solution substituting for making any change in practice. Saying we are T-shaped does nothing.

ENGINEERING IS IN TROUBLE

Engineering is in trouble. Engineering has played an important role in the development of the world whether in ancient Rome or China, brilliant innovators developed sophisticated technology for bringing water, food, and the necessities of life to their expanding nations. As this experience grew, the formal field of engineering emerged. Today, engineers are taught science, mathematics, and the increasingly specialized components of their chosen field.

But this specialization leads to three difficulties.

First: Real projects require broad, generalized thinking. Universities train deep specialists, and so in the world of practical accomplishments, engineers are often ill-trained to work with other disciplines, or even to know what knowledge they need from these disciplines.

Second, engineers are taught how to solve problems, but where do the problems come from? In the world of education problems are well-formed, with well-defined solutions. In the real world, problems are fuzzy, ill-defined, where it is often not even apparent that thee is a problem, or even when it is agreed that a problem exists, neither the approach nor the solution is known, nor recognizable at first. Designers and economists call these "wicked" problems. Wicked problems require creative, right brain thinking, not the logical, systematic left-brain thinking so prized by engineers.

And Third, much of what engineers develop goes into systems meant to be used by people. Yet the scientific study of people takes place in the social sciences, not the engineering sciences. To many engineers, people are a nuisance ("if it weren't for people," I have heard engineers complain, "our systems would work much better").

The solution is to rethink engineering education. As Julio Ottino, Dean of engineering at Northwestern likes to stress, we need whole-brain thinkers, not just left- or right-brain. Yes, we need logical, deep-thinking specialists, but we also need broad, creative thinking generalists. We need people who can solve the "wicked" problems of real life, who can work in broadly based, multidisciplinary teams with engineers, social scientists, economists, and business people.

Design, the discipline, is ideally suited to help solve these problems. Modern designers are problem solvers, relishing complex, wicked problems, developing creative, original solutions. They are broadly trained, experienced in a wide variety of disciplines. But these virtues come at a price: designers lack the necessary depth in each area. Combine designers and engineers, engineering training and design training, and we can create the people needed to solve the problems of the 21st century. Systems problems, integrating across disciplines, constrained by cost and existing solutions, psychology, sociology, and culture, political differences, and business needs. This is the real world: we need to rethink education to address it.

What are the techniques needed? Emphasize teamwork. Introduce design projects to all engineers in the Freshman year (as is done at Northwester and KAIST, to chose the two universities I am associated with). Have multiple project courses. Be less concerned with covering al the essential material and more concerned with teaching students how to think and how to learn on their own. Studies at MIT and elsewhere indicate that even the best students do not retain much of what they have been taught in their classes. The best students, however, are capable of learning the material when they need it: So emphasize just-in-time learning, the complexity of real problems, the need to integrate across all disciplines, including non-engineering ones, and teamwork.

WE NEED A PROFESSIONAL DISCIPLINE OF HSI

We need a professional discipline of HSI. Today we mostly have a research discipline.

HSI must become an applied discipline, not just a research activity, not just a science (it needs all three: science, research, practice).

The problem is this: suppose, magically, HSI was asked to take part in all new projects from the very start. No more fire-fighting. Would HSI be able to deliver? I don't think so.

HSI has to stop being an analytical field doing analysis after the fact and become a design field, synthesizing answers on the spot. Providing answers in hours or days, not in six months. Doing quick and dirty experiments and quick calculations to ensure that the designs are "good enough." Practical designs look for large effects. Traditional science looks for small differences. HSI has to change how it thinks. Today, HSI Is not ready for real time.

We need a special certification program (ideally offering a Masters Degree in HSI project management)  for the management of projects involving HSI.

Universities need to change. Not only are they too narrow, but they keep disciplinary walls that inhibit cross-fertilization. Professors lack practical experience and usually feel that such experience is inferior in value to theoretical and research skills. So practice is not rewarded. Only publications in refereed, research-based journals are recognized.

The separation of the social and behavioral sciences from engineering is most unfortunate: they need one another. But the social and behavioral sciences shun applications except to proclaim in incredible naiveté the fanciful applications of their work, but without actually trying to do the applications. And the engineering disciplines ignore the human factor, even though, in theory, engineering builds and designs for human use and benefit. (Of course, given the theoretical bent of modern engineering, it seldom actually builds anything.) Of all the engineering disciplines, only Computer Science (note how they don't like to call themselves engineers) is willing to take the step, allowing HCI into their midst. But HCI is only allowed in reluctantly, as minor appendages to some department's offerings. Although the American Society of Mechanical Engineers, ASME, does have a design section, it is difficult to find a Mechanical Engineering department that includes this, except perhaps as a tiny addition to their product design offering, which itself is barely tolerated in modern Mechanical Engineering.

The field of Human Factors and its many descendants -- Cognitive Engineering, Human-Computer Interaction, Cognitive Ergonomics, Human-Systems Integration, ... -- has made numerous, wonderful advances in the many decades since the enterprise began. But the discipline still serves many to rescue rather than to create.  It is time for a change.