A Crowdsourcing Approach for Identifying Potential Stereotypes in the Collected Data

Data generation through crowdsourcing has become a common practice for building or augmenting an Artificial Intelligence (AI) system. These systems often reflect the stereotypical behaviors expressed by humans through the reported data, which can be problematic, especially when dealing with sensitive tasks. One such task is the interpretation of images depicting people. In this work, we evaluate a crowdsourcing approach aimed at identifying the stereotypes conveyed in the collected annotations on people images. By including closed-ended, categorical responses as well as open-ended tags during the data collection phase, we can detect potentially harmful crowd behaviors. Our results suggest a means to assess descriptive tags, as to their alignment with stereotypical beliefs related to gender, age, and body weight. This study concludes with a discussion on how our analytical approach can be applied to pre-existing datasets with similar characteristics or to future knowledge being crowdsourced such as to audit for stereotypes. � The Author(s), under exclusive license to Springer Nature Switzerland AG 2024.