Human Herd

Fascinating article in the NYT today (I am again quoting NYT…It seems they have really gotten their act together of late in the high-tech/network world space). The article talks about the theory of “Cumulative Advantage” or the “rich get richer” effect. In summary the theory suggests that our preferences/decisions are very much effected by what other people are doing. So if a technology or a singer or a movie is liked by our peers we are more likely to try it and like it. We provided another example of this phenomenon (without naming the theory) in a prior post related to behavior of users at Digg where we observed that a fake article got a number of Diggs just because a user paid of a few Diggs to get initial momentum.

Conventional marketing wisdom holds that predicting success in cultural markets is mostly a matter of anticipating the preferences of the millions of individual people who participate in them. From this common-sense observation, it follows that if the experts could only figure out what it was about, say, the music, songwriting and packaging of Norah Jones that appealed to so many fans, they ought to be able to replicate it at will. And indeed that’s pretty much what they try to do. That they fail so frequently implies either that they aren’t studying their own successes carefully enough or that they are not paying sufficiently close attention to the changing preferences of their audience.

The common-sense view, however, makes a big assumption: that when people make decisions about what they like, they do so independently of one another. But people almost never make decisions independently — in part because the world abounds with so many choices that we have little hope of ever finding what we want on our own; in part because we are never really sure what we want anyway; and in part because what we often want is not so much to experience the “best” of everything as it is to experience the same things as other people and thereby also experience the benefits of sharing.

The authors set out to test out the theory with an interesting experiment:

Because it’s not possible in the real world to test theories about events that never happened, most of what we know about cumulative advantage has been worked out using mathematical models and computer simulations — an approach that is often criticized for glossing over the richness of real human behavior. Fortunately, the explosive growth of the Internet has made it possible to study human activity in a controlled manner for thousands or even millions of people at the same time. Recently, my collaborators, Matthew Salganik and Peter Dodds, and I conducted just such a Web-based experiment. In our study, published last year in Science, more than 14,000 participants registered at our Web site, Music Lab (www.musiclab.columbia.edu), and were asked to listen to, rate and, if they chose, download songs by bands they had never heard of. Some of the participants saw only the names of the songs and bands, while others also saw how many times the songs had been downloaded by previous participants. This second group — in what we called the “social influence” condition — was further split into eight parallel “worlds” such that participants could see the prior downloads of people only in their own world. We didn’t manipulate any of these rankings — all the artists in all the worlds started out identically, with zero downloads — but because the different worlds were kept separate, they subsequently evolved independently of one another.

This setup let us test the possibility of prediction in two very direct ways. First, if people know what they like regardless of what they think other people like, the most successful songs should draw about the same amount of the total market share in both the independent and social-influence conditions — that is, hits shouldn’t be any bigger just because the people downloading them know what other people downloaded. And second, the very same songs — the “best” ones — should become hits in all social-influence worlds.

What we found, however, was exactly the opposite. In all the social-influence worlds, the most popular songs were much more popular (and the least popular songs were less popular) than in the independent condition. At the same time, however, the particular songs that became hits were different in different worlds, just as cumulative-advantage theory would predict. Introducing social influence into human decision making, in other words, didn’t just make the hits bigger; it also made them more unpredictable.

Where does this leave us with the rational choice and perfect market theory? Do you think people are more rational when it comes to money? What about making investments? How should VCs or any investor for that matter, evaluate a new consumer technology or a mass market product? This is powerful stuff.