Do you want to avoid the hassle of traveling with your printed poster? IAPHS2026 is pleased to make poster printing available to you through our supplier PosterSessionOnline. Your poster will be professionally reviewed, printed and shipped directly to Portland and you will be able to pick it up from the Poster desk. Click here to learn more.
Primary Submission Category: Methodological approaches to studying public health
Analyzing Modern Data Quality Control Approaches on Crowdsourced Data from Prolific
Authors: Jon Agley, Yunyu Xiao, Mikyoung Jun,
Presenting Author: Jon Agley*
Introduction: Crowdsourced research studies conducted using tools like Amazon’s Mechanical Turk and Prolific represent potential opportunities to rapidly study issues of interest to the population health sciences. Unfortunately, there are many factors that can reduce the quality of data obtained from these platforms. Such risks include, but are not limited to, respondents’ inattention or dishonesty, bots, virtual private networks, and now, large language models (LLMs) and computer-using assistants (CUAs).
Objective: Given Prolific’s extensive identity verification procedures, including unscheduled, live video monitoring to prevent LLM use, we chose that platform to assess 13 study-level modifications to facilitate data quality control.
Method: We recruited a US-based nationally-representative sample by cross-sections of age, race/ethnicity, and sex from Prolific (n=450) to analyze raw quality check failure counts for each of 13 approaches, identify the total count of failed checks per participant, and obtain strength-of-association data for each bivariate pair of control checks (i.e., whether a given pair of failed checks is likely to co-occur, and the strength of that association).
Results: Of the 450 participants, 366 (81.3%) did not fail any checks, 66 (14.7%) failed one check, 12 (2.7%) failed two checks, and 6 (1.3%) failed three checks. Only one pair of quality checks were strongly associated: positive-valence and negative-valence LLM jailbreaking prompts (V=.717, p<.001).
Discussion: We will highlight implications for future crowdsourced studies, including the likely importance of using multiple different checks given a lack of strong associations between failed checks. We also discuss nuances for conducting checks using prompt leaking text and address ongoing concerns about the lack of “ground truth” prevalence data on LLM and CUA use by participants.
Disclosure: Portions of this abstract, including some verbatim text, are from a journal article under review.