Prediction markets, Wikipedia, and dispersed knowledge

Hayek is generally credited with the insight that the problem with socialism and command economies in general is that knowledge is widely dispersed throughout the population. Everyone knows something, and as a whole, we know much more than any small group or individual. Capitalist economies tend to extract this knowledge through price signals whereas command economies take their best guess but end up not having enough information to make the proper decisions.

A fascinating (to me at least) result of this is the ability of large numbers of people to properly guess random things like the weight of the horse that won the Kentucky derby. Several studies have been done on this and they always end up being within a pound or two of the actual number. There are other examples as well, but how do they work? The idea is a simple one. If people have an idea of the right answer, the “average” of their answers will converge on the correct one given enough answers. Now most people don’t have a very good idea about the specific weight of horses, but we all know that they aren’t 5 pounds, and we know that they aren’t 5000 pounds either. So we have a good enough idea that we guess within a reasonable amount plus or minus around the actual value. This statistical “noise” cancels out with enough answers and you’re left with the proper weight.

Things get even more interesting when you start dealing with “experts.” It takes far fewer answers for the average to come to the correct number. This makes sense since we would expect that experts would have far less variance to their answers. How do you limit these questions to “experts?” The simplest way is to put some money on the line, make a bet. This keeps the people who really don’t have a good idea from contributing and limit your answers to expert opinions. This is curious when dealing with the weight of horses, but it gets downright amazing when you start talking about policy issues, business decisions, and even presidential candidates.

Google has been using these sorts of “Prediction markets” with great success when dealing with internal questions, and there is a thriving market for presidential nominations. This may sound kind of strange, but it is a way of extracting knowledge from a dispersed population. Imagine if before the recent invasion of Iraq, a decision market was formed in the CIA asking what they thought the likelihood of WMD being in Iraq. If you allow anyone with some knowledge to contribute instead of a select few that may be guided by politics moreso than facts, you are much more likely to come up with the right answer. If people from not only the Iraq and middle east bureaus were allowed to vote but also anti-terrorism, Africa, nuclear proliferation teams, in short anyone that would have some sort of expertise on the proliferation of these sorts of things were allowed to vote, you would gain much more information than just following a handful of “experts” advice. There’s a good chance that the opinion CIA-wide about the probability of WMD in Iraq would have been negative. Of course who knows if that would have changed anything….

There is a potential problem of course. If there is a systemic bias among the “experts”, the results can be way off. It has to be systemic though, if only a few suffer from it, the results will wash out with enough contributers. I learned about this from a recent podcast from econtalk. There are some amazing examples given where this has worked very well. Of course the host gave a chilling counter example. he regularly holds seminars for the press and people up on capital hill. One of the questions he usually asks everyone is what percentage of Americans are paid the minimum wage. He claims that the median answer given to him is usually between 15% and 20%. The actual percentage is just below 3%. Clearly, this approach won’t work with everything, but it has promise and is being used successfully in the business world.

In the same podcast they also discussed the Wikipedia phenomena. For those of you not familiar with it, Wikipedia is an online encyclopedia that is written by the people that read it. Anyone may contribute to it, and may edit whatever entry that they are reading. Most people never would have thought that this could work, but it does, and amazingly well. Errors are usually seen quite quickly and corrected. Once again, the idea is that everyone is knowledgeable about something, and our collective knowledge is far greater than any ad hoc group formed with the idea of writing an encyclopedia. The amount of knowledge out there is impressive, not just in depth, but in breadth. Wikipedia has articles on things that would never be in a regular encyclopedia, primarily because the publishers wouldn’t think of including some topics. Even if they did think about it, they may not have access to people that know much about them. Wikipedia keeps growing, both in the number of topics it covers and the amount of information included on any given topic.

The CIA is using a variation of this, their own intellipedia. The idea is that people can post articles on whatever topic that they feel is important. People then read the articles and they can add to it or edit it as they see fit. The article can be edited and reedited numerous times until some sort of consensus is formed, either in agreement or in opposing camps. By allowing the free flow of information among all of the people there, topics can be fleshed out quite a bit more completely and quickly than if it went through the traditional editing process. It also allows people from widely differing specialties to offer their expertise. Even if an expert that has tangential interest only qualified one sentence in the report, that is information that wouldn’t have been in there if done the traditional way. Once again, the idea is to take advantage of the dispersed knowledge that is out there. If enough people contribute, just about everything that is known about a topic can be gathered.