Part 4: Thanks...but is it true?

Marcus Winter's picture

This is the fourth part in a series of blog posts discussing findings from our literature review about complex game-based crowdsourcing. Today we'll look at ways to ensure that user-generated content in crowdsourcing projects meets professional standards. (Check out Part 3: What motivates volunteers?)

Part 4: Thanks...but is it true?

One of the key advantages of crowdsourcing is that it combines audience engagement with the production of useful outcomes. In the context of 10 Most Wanted, this will hopefully translate into a sustainable model for maintaining collections by delegating aspects of curatorial work to members of the public. 

A potential downside is that the public usually lacks the expert knowledge and skills of professional curators. While it has been suggested that crowdsourcing can lead to solutions superior in quality and quantity to professional efforts [1], there are widespread concerns among professionals about data quality. Some of these concerns are highlighted in Alexandra Eveleigh's discussion of participatory archives [2]:           

"User participation initiatives in archives are haunted by a fear that a contributor might be wrong, or that descriptive data might be pulled out of archival context, and that researchers using collaboratively authored resources might somehow swallow all of this without question or substantiation." 

From a curator's perspective, data quality and verification are critical to avoid compromising quality standards for the collection as a whole. Introducing invalid data would not only impact on the collection's value as a research resource but also undermine the organisation's authority, which is a distinguishing quality specifically for heritage organisations [3]. Data quality is also important from the perspective of volunteers, who want to be reassured that the outputs of their efforts are useful and academically valid [4].  

Measures suggested in the literature to improve data quality in crowdsourcing projects can be broadly grouped into four approaches:

  1. Make the task easier. Holley [5] suggests that increasing the quality of the materials volunteers work with makes errors less likely. This is a specific form of the more general concept of making the task easier, which is a key idea at the root of crowdsourcing: breaking down complex problems into small, simple tasks that do not require any specialist knowledge.
  2. Train/inform volunteers. Cohn [6] suggests training volunteers in order to give them a better understanding of professional standards and practices. A more lightweight approach might just inform participants of the organisation's needs: Kidd [7] describes how citizen journalists during the Arab Spring met the requirements of broadcasters by using establishing shots to verify their positions and timings.
  3. Crowdsource quality control. User-generated classifications of galaxies in the GalaxyZoo project are "written into a database and compared with the findings of other volunteers" [8]. This approach can also be made explicit: Brooklyn Museum's Freeze Tag game [9] involves players in the clean-up of user-generated tags created in another crowdsourcing game.
  4. Professional quality control. Eveleigh [2] points out that curators play the role of gatekeepers when user-generated content is integrated into collections. While professional quality control has led in some cases to allegations of censorship, most users accept the organisation's decisions as guided by professional expertise.

10 Most Wanted combines several of these approaches to ensure findings meet professional standards. Firstly, we provide research tips for participants to help with their investigations and increase data quality. Secondly, we ask participants to work together in teams and critically assess each others' findings. Thirdly, we have a process in place where curators (case officers) screen contributions and piece together key information into an evidence trail (case notes) that proves a fact about an artefact. Check out one of our solved cases - do they convince you? 

Catch up on Part 3: What motivates volunteers? 

[1] Brabham, D. C. (2008). Crowdsourcing as a Model for Problem Solving: An Introduction and Cases. Convergence: The International Journal of Research into New Media Technologies, 14(1), 75–90. 
[2] Eveleigh, A. (2012). Welcoming the World: An Exploration of Participatory Archives. Presented at International Council on Archives (ICA) Conference, ICA 2012, Brisbane, Australia, 20-24 August 2012 (pp. 1–10).
[3] Oomen, J. and Aroyo, L. (2011). Crowdsourcing in the Cultural Heritage Domain: Opportunities and Challenges. Proceedings of the 5th International Conference on Communities and Technologies (pp. 138–149).
[4] Dunn, S. and Hedges, M. (2012). Engaging the Crowd with Humanities. A scoping study. Research Centre for e-Research , Department of Digital Humanities. King’s College London. Available
[5] Holley, R. (2009). Many Hands Make Light Work : Public Collaborative OCR Text Correction in Australian Historic Newspapers. National Library of Australia. Retrieved from:
[6] Cohn, J.P. (2008). Citizen Science: Can Volunteers Do Real Research? BioScience, 58(3), 192–197.
[7] Kidd, J. (2013). Visitor Generated Content (VGC) and Ethics - what we might learn from the media and journalism. Presentation at the iSay: State of Things conference, 31 Jan-2Feb, Leicester (pp. 1–8).
[8] Raddick, M. J., Bracey, G., Gay, P. L., Lintott, C. J., Murray, P., Schawinski, K., Szalay, A. S. and Vandenberg, J. (2010). Galaxy Zoo: Exploring the Motivations of Citizen Science Volunteers. Astronomy Education Review, 9(1), 1-18.
[9] Bernstein, S. (2009). Crowdsourcing the Clean-Up with Freeze Tag! Brooklyn Museum. Available