Colby Magazine - Spring '98 The Soul of Technology
book cover A few years ago, a woman in her mid-60s was asked whether she thought she would own a computer someday. She said no. "If you plug one of those things into the wall, can't someone from anywhere find out all about you?" she asked. At the time, the question was funny. But in these days of "cookies" (the footprints you leave behind when you visit a Web site), usage maps on hard drives and companies that spy on employees through their desktop machines, it seems critically relevant.
    Computers, editor BatyaFriedman (associate professor of computer science) and the other contributors to this book say, are tools. And like other tools, they are designed to do a certain job in a particular way. In themselves, they have no human values, but the implications of computer use are fraught with human values--defined, for the purposes of this book, as values that deal with human welfare and justice.
    Consider the example of the designers who, charged with keeping production costs at a minimum, removed the on-off switch from a computer's built-in microphone. Not only would that save money when it came to building the machines, they said, but because users would no longer have to remember to turn their microphones off, companies would spend less replacing dead batteries. But for the people using the computers, the always active microphones presented dilemmas involving privacy. Somebody was plugging that machine into the wall, and no matter what management promised, it would always be possible for the company to make use of employee conversations in ways employees couldn't predict.
    Then there is the fast-growing technology that will help drivers navigate highways better. The goal is to someday create a fully automatic car that doesn't need a driver. Such a system, used widely enough, might prevent highway deaths. But in the meantime, as the movements of cars are increasingly tracked by computers and satellites, someone, somewhere, is finding out where we go and when we go there. The state of Kentucky's tourism office used that kind of information to mount a public relations assault on Canada, having discovered that almost none of the Canadians who passed through on theway to other Southern states wasstaying overnight in Kentucky. Police departments and federal agencies, the authors say, also might find information about the habits of individual drivers very useful.
    And what if the hospital software APACHE,designed to recommend whether doctors should withdraw life support from patients in intensive care--software theoretically backed up by the judgment of human professionals on the scene--were to become so apparently reliable that doctors trusted it too much? "Within this social context," write Friedman and co-author Peter Kahn (of Colby's Education and Human Development Department), "it may become the practice for critical-care staff to act on APACHE's recommendations somewhat automatically and increasingly difficult for even an experienced physician to challenge the `authority' of APACHE's recommendation, since to challenge APACHE would be to challenge the medical community."
    One section of the book deals with the ways people interact with computers. In the essay "Computers Are Social Actors," for instance, the authors detail the results of five experiments conducted to see whether people treat computers as if they were human. In one experiment, the computers were programmed to put the subjects through a training exercise. Afterwards, the subjects were asked to evaluate the computers' performance. Those who did the evaluation using the computer were less critical than those who used a pencil and paper. The first group was using human norms for politeness. They didn't want to hurt the computers' feelings (and a later experiment proved that they were not thinking of the programmers' feelings but those of the machines). The subjects also responded positively to random flattery from the computers and evaluated computers with "male" or "female" voices along the lines of common gender stereotyping.
    A great feature of the book is that the essays include speculation on the ways information gathered about computers could be applied to practical situations. "Computers Are Social Actors" suggests that because most of the feedback users now receive from computers is negative (indecipherable "error" messages and the like), and because people respond so well to flattery from the computer, flattery ought to be built into future software. "Software programs designed to aid users in performing unpleasant tasks would be ideal candidates for positive message systems," the authors write. "For example, many programs are designed to help users pay their taxes, balance a spreadsheet or draft legal documents. The results of this study suggest that incorporating a flattering feedback mechanism would boost user enjoyment of these tedious undertakings."
    Not so fast, say Friedman and Kahn in the essay "Human Agency and Responsible Computing." If we give computers a human face (literally or figuratively), we'll encourage ourselves to think of them as if they really were people. People who can act intentionally. People who can be blamed when their circuits fail.
    Friedman and Kahn recount an experiment that simulated a nuclear power plant failure. The subjects, power plant operators, had at their command an "expert" computer system to help them respond to the emergency. But, even though the subjects had instruction on the computer's capabilities, they assumed it "knew" without being told that the cooling system had stopped operating. The scientist who ran the experiment said the theoretical, potentially fatal accident happened because the plant employees assumed that the computer had certain information, "presumably the type of information that any responsible human expert would know or attempt to find out in the situation."
    "Because nonanthropomorphic design does not encourage people to attribute agency to the computational system," Friedman and Kahn write, "such designs can better support responsible computing."
    This book is a fascinating examination of computer design in its full-blooded, mature state. And it is comforting to know that the eminent scholars represented in the volume are asking increasingly more probing questions about the ways people and computersintersect. The ramifications, they prove amply, go well beyond the plug in the wall.
Books & Authors
Just Desserts
Fresh Prints