Crowdwork as a Snapshot in Time: Image Annotation Tasks during a Pandemic

While crowdsourcing provides a convenient solution for tapping into human intelligence, a concern is the bias inherent in the data collected. Events related to the COVID-19 pandemic had an impact on people globally, and crowdworkers were no exception. Given the evidence concerning mood and stress on work, we explore how temporal events might affect crowdsourced data. We replicated an image annotation task conducted in 2018, in which workers describe people images. We expected 2020 annotations to contain more references to health, as compared to 2018 data. Overall, we find no evidence that health-related tags were used more often in 2020, but instead we find a significant increase in the use of tags related to weight (eg, fat, chubby, overweight). This result, coupled with the “stay at home” act in effect in 2020, illustrate how crowdwork is impacted by temporal events.
The COVID-19 pandemic is proving to be a public emergency without precedent. Worldwide, individuals are feeling the impact; eg, they may be suffering from confusion, isolation, and feelings of insecurity (Pfefferbaum and North 2020). Public health researchers are citing large-scale problems such as alcohol and drug abuse (Clay and Parker 2020) as well as increased levels of anxiety and sleep disturbances (eg, insomnia)(Sher 2020). Other concerns include increased issues with eating disorders (Fernández-Aranda et al. 2020) and the danger of weight gains in individuals over short periods of time (Bhutani and Cooper 2020). The “stay at home” movement and/or enforced lockdowns, in combination with the economic crisis during 2020, have created a fruitful