Social Cues, Social Biases: Stereotypes in Annotations on People Images

Human computation is often subject to systematic biases. We consider the case of linguistic biases and their consequences for the words that crowdworkers use to describe people images in an annotation task. Social psychologists explain that when describing oth- ers, the subconscious perpetuation of stereotypes is in- evitable, as we describe stereotype-congruent people and/or in-group members more abstractly than others. In an MTurk experiment we show evidence of these bi- ases, which are exacerbated when an image’s “popular tags” are displayed, a common feature used to provide social information to workers. Underscoring recent calls for a deeper examination of the role of training data quality in algorithmic biases, results suggest that it is rather easy to sway human judgment.