Curing cancer by committee

PAULINA STREET JOURNAL

http://chicago.straightdope.com/sdc2010crowdsourcing.php

July 29, 2010

My eldest decided the other day it was time I knew about something about Web 2.0 — Web 2.0 being broadly understood to mean whatever happens next in the world of the Internet. So he presented me with a couple of books plus an essay, which he felt would get me up to speed. I've now perused these learned works, which have given me insight into the shape of things to come. For example, is your idea of diversion an evening in front of the tube? I'm sorry — in the future, you'll have to cure a loathesome disease first.

Let's start with the essay. It's entitled "Gin, Television, and Social Surplus" by Clay Shirky, a teacher at New York University, a hotbed of advanced online thought. The essay appeared in 2008 and is well known among the cognoscenti, and my bringing it up now will strike some as on a par with getting excited about the Gettysburg Address. My apologies to these cutting-edge individuals. For the rest of you, here's the gist:

In the early nineteenth century, the chief way people dealt with the stresses of factory work and urbanization was the consumption of vast amounts of gin. Not till much later did institutions arise — Shirky cites museums, libraries and schools — that took advantage of urban aggregation to provide the citizenry with more constructive ways to occupy their time.

The equivalent of gin in the latter twentieth century was the TV sitcom, which soaked up the excess energy generated by increased postwar affluence and leisure: "Desperate Housewives essentially functioned as a kind of cognitive heat sink, dissipating thinking that might otherwise have built up and caused society to overheat," Shirky writes. In short, TV = opiate of the masses.

The Web has at last provided us with an outlet for the mental energy heretofore sucked up by sitcoms. Exhibit A is Wikipedia, and one can only guess what additional marvels await.
The general formulation, then, is as follows: (a) massive societal upheaval produces social surplus; (b) lacking alternatives, we then blow off said surplus in useless dissipation (gin, TV); (c) eventually institutions and/or technology evolve to put the surplus to more productive use; (d) world becomes better place. To illustrate, Shirky calculates that Wikipedia to date has consumed 100 million hours of effort, which seems impressive until you consider that, in the U.S. alone, we spend 200 billion hours every year watching television. If we harnessed those squandered resources, Shirky observes (the common expression is "unused cycles"), we could produce the equivalent of 2,000 Wikipedias per year — and if that doesn't fill you with guilt, I don't know what will.

Next on the reading list was The Wisdom of Crowds (2004). In it New Yorker business columnist James Surowiecki argues that groups of people, not all of whom are necessarily geniuses, often get better results than geniuses working alone. He cites some uncanny predictions, which I think it's worth my describing in detail:

In 1968 the U.S. submarine Scorpion vanished in the North Atlantic en route to its home port. The Navy knew little other than the vessel's last reported location. Rather than search randomly, naval officer John Craven drew up scenarios of the sub's fate — nature of failure, speed, rate of descent, and so on. Then he submitted these to experts who were asked to gauge the likelihood of each without consulting the others. Finally, Craven performed statistically wizardry on the result to come up with a collective estimate of the sub's location. The sub, which might have been anywhere within a 20-mile circle, was found 220 yards from the estimated spot. Whoa.

British scientist Francis Galton in 1906 analyzed the results of a contest at a regional agricultural fair, in which participants were asked to guess the dressed weight (after butchering) of a certain ox. The contest was open to anyone, including farmers and ordinary jamokes. Eight hundred people gave it a shot. After the contest was over, Galton collected the entries and averaged the guesses: 1,197 pounds. The actual weight of the dressed ox: 1,198 pounds. Whoa again.

On January 28, 1986, the space shuttle Challenger blew up shortly after launch. Investors promptly began dumping the stock of four contractors who'd help build it: Lockheed, Martin Marietta, Rockwell, and Morton Thiokol. (The last made the booster rockets, which featured something known as an O-ring.) All four stocks at first fell sharply. The first three partially recovered by the end of the day, but Morton Thiokol's kept falling and at the close of trading was down 12 percent, suggesting the market had determined the firm bore primary responsibility for the disaster. After painstaking investigation an official commission confirmed this conclusion six months later. The market had figured it out in one day.

By this time the reader is thoroughly freaked out. Now, the first two stories you can rationalize. Perusing the Wikipedia article about the Scorpion, we learn that underwater breakup noises helped the Navy get a fix on the vessel's location. Likewise, in the Galton story we may surmise that the collective wisdom of the knowledgeable farmers swamped the scattered efforts of everyone else.

Correctly assessing the blame for the Challenger disaster, on the other hand … there's a miracle for you. Surowiecki cites an analysis by two economists, who wind up conceding they can't explain it — there's no evidence anyone had inside information, and the traders had no scientific knowledge. So how did they do it — magic? ESP? Did they sense a disturbance in the force? No. Surowiecki explains:

At heart, the answer rests on a mathematical truism. If you ask a large enough group of diverse, independent people to make a prediction or estimate a probability, and then average those estimates, the errors each of them makes in coming up with an answer will cancel themselves out. Each person's guess, you might say, has two components: information and error. Subtract the error, and you're left with information.

Who can argue with that?

The other book my kid presented me with, which for me put matters over the top, was Crowdsourcing: Why the Power of the Crowd Is Driving the Future of Business (2008) by Wired magazine contributing editor Jeff Howe. Howe describes various successful online collaborations, the best known example of which other than Wikipedia is undoubtedly Linux, the free open-source operating system. He quotes a fellow named Ted Gulley, who runs contests in which programmers compete online to produce the best software:

We have truly brilliant people playing. One of them will make a breakthrough, and on its own it would have been the best solution in an old-school contest. Just because they're brilliant. But with … the contest, immediately people are able to come along and tweak it. No single person could do that. It's the swarm, this great big collective brain we have access to. What will really be amazing is if we can tap that brain to cure cancer.

Now we start to get to the heart of the thing. However, some subtleties remain to be grasped. You might think the great big collective brain consists only of the brilliant. Not so. In the Web 2.0 view of things, dopes have their place. Surowiecki writes:

Bringing new members into the organization, even if they're less experienced and less capable, actually makes the group smarter because what little the new members do know is not redundant with what everyone else knows.

He quotes organizational theorist James March:

The development of knowledge may depend on maintaining an influx of the naïve and the ignorant, and … competitive victory does not reliably go to the educated.

Anybody who follows politics is thinking: no shit. However, there are broader implications. Surowiecki goes on to say:

What is striking … is just how much information a group's collective verdict so often contains … the crowd is holding a nearly complete picture of the world in its collective brain.

So let's review:

• Online collaborations have accomplished amazing things.
• Crowds of mopes (and let's be frank, muchachos — this means you and me) are collectively smarter than isolated geniuses, since we all have random quanta of information that no single individual possesses.
• If you were to add up the thoughts of everybody in the world, the dimwitted parts would cancel out and what's left would be complete global knowledge.
• We all have computers (well, one billion of us do) plus lots of free time, provided we don't blow it on Desperate Housewives and Jersey Shore.

You see where this is headed. Once they get the work assignments figured out, they're going to be after us to cure cancer, probably for free, although maybe we'll get a T-shirt out of it. How and when are we supposed to go about this? I expect they'll be e-mailing us instructions. In the meantime, if you know what's good for you, don't watch sitcoms on TV.

— Ed Zotti