Crowdsourcing is defined as “a sourcing model in which individuals or organizations obtain goods and services, including ideas and finances, from a large, relatively open and often rapidly evolving group of internet users. Crowdsourcing spreads work among participants to achieve a cumulative result.” Some well-known examples of crowdsourcing include the online encyclopedia Wikipedia, the traffic and navigation mobile application Waze, and the crowdfunding site for creative projects Kickstarter.
As smartphones and internet access continue to become more pervasive, crowdsourcing is gaining popularity, especially in the field of research. To look at how prevalent crowdsourcing is in research, Benjamin Ranard and colleagues conducted a systematic review, published in the Journal of General Internal Medicine in 2014, to identify the scope and characterize the use of crowdsourcing in health-related research. The authors identified 21 studies that used crowdsourcing and collectively engaged >136,395 participants. A common crowdsourcing platform seen frequently in the articles is Amazon’s Mechanical Turk (MTurk).
MTurk links individuals or businesses with an anonymous online-based workforce to carry out tasks that require human intelligence. The platform has a wide variety of functions and has previously been used to conduct missing persons searches, tag objects in an image to improve advertising targeting, remove duplicate content from business listings, write content for websites, and provide human-powered translation services.
Advantages of this platform include an on-demand, cost-effective elastic workforce and mechanisms to assist with quality management. Further, the size of the MTurk marketplace is large. The 2015 World Bank report [PDF] estimated that MTurk had about 500,000 registered workers (although not all are active). The PEW Research Center also examined the MTurk marketplace between December 7, 2015 – December 11, 2015. During this time period, there were 294 different requesters (which averages out to 59 new requesters per day).
Since MTurk is increasing in popularity, how does it compare to other online survey platforms? An article by Karoline Mortensen and colleagues, just out in Medical Care, compared MTurk demographic, socioeconomic, and self-reported health status to similar measures obtained from two gold-standard data sources commonly used in health services research: the Medical Expenditure Panel Survey (MEPS) and the Behavioral Risk Factor Surveillance System (BRFSS).
The MTurk sample was more likely to be female, younger, non-Hispanic, single, college-educated, and employed when compared with respondents in the MEPS and BRFSS. MTurk respondents were also more likely to be in the lowest income bracket (<$10,000 per year), and were substantially more likely to report lower self-perceived health status. The authors conclude that convenience samples collected on crowdsourcing platforms are not equivalent to nationally representative samples and may not yield broadly generalizable findings.
Ranard and colleagues, in JGIM, suggest that, while crowdsourcing can be used to improve the quality, speed, and cost of research initiatives, standardized guidelines are needed for collecting and reporting metrics. Of the studies reviewed, the authors identified a wide variety of type and amount of data reported. Articles involving crowdsourcing typically didn’t discuss demographic data, size of the cohort, or participant information, such as age, gender, or geographic location. Guidelines could help ensure that these important characteristics are shared with the scientific communication and help improve comparability across crowdsourcing studies.
Similarly, Mortensen and colleagues, in Med Care, recognize that a challenge for health services researchers is to identify when crowdsourced data are appropriate. Improvements on the design and administration of crowdsourced surveys are needed to ensure their features match the necessary research value. Additional research is needed on the reliability and validity of crowdsourced data.
Catherine Gupta
Latest posts by Catherine Gupta (see all)
Alexa Ortiz