Is the crowd biased? Understanding binary value judgments on user-contributed content

Binary voting mechanisms are extensively used in open contribution systems (OCS), crowdsourcing the decision as to which contributions are most valued. Often, aggregated votes are used to rank a shared collection of artifacts (eg, the most helpful reviews of a medication shared by those who use it; the best-liked reader responses to a news story.) Previous work suggests that voting mechanisms pick up on salient dimensions of content quality and that users agree which contributions are “helpful” or “likable” in particular contexts. However, researchers have also documented systematic biases in the rankings that few systems reveal to users. After illustrating the need for a comprehensive study of social voting bias in OCS across domains, we propose a framework under which such research can be carried out. Finally, we discuss implications for OCS design as well as for end-users.