Chapter 7: Big Human Intelligence

Photo by davide ragusa on Unsplash

While AI is powerful, much of the time, human intuition and behavior is still more valuable for psychological research. This chapter focuses on crowdsourcing – a method for leveraging the intelligence of many people to complete a task. The chapter discusses the use of crowdsourcing and citizen science across several fields, and how to decide when to use crowdsourcing versus AI for analyzing complex psychological data. The chapter also provides practical advice on what platform to choose, and how to avoid low-quality data from bots or cheaters.

  • Crowd-sourcing platforms

    • Here are some resources for crowd-sourcing of psychology experiments

    • Amazon Mechanical Turk - a large global platform for crowdsourcing tasks online. They handle participant recruitment, experiment hosting, data saving, and participant payment. However, becauseit is so popular, there are concerns about data quality (with a large proportion of it becoming unusable; Marshall et al., 2023.)
    • Prolific - a global crowdsourcing platform that is gaining popularity. It pays better and reports better quality data. However, they only handle participant recruitment and payment -- not experiment hosting or data saving.
    • CloudResearch - a resource for crowd-sourcing studies that promises to have higher quality data, but involves more interaction with the company.
    • DataSpring - an Asia-specific crowdsourcing platform that promises high-quality data, but involves more interaction with the company.

  • Online psychology experiments

    • Check out these websites if you want to start participating in experiments yourself (and be part of the crowd being sourced!)

    • TestMyBrain - a psychology research initiative with experiments where you can learn about yourself.
    • The Music Lab - learn how you process sounds and music, while also creating valuable data for psychologists!
    • Children Helping Science - let your kids engage in research studies over the internet, and learn more about how they're thinking and growing.

  • How to run an online experiment super fast!

    • Here is simple code for running a 2-question experiment on Amazon Mechanical Turk. (Right click and choose "Save As" to save the code itself rather than just opening the link. Or, "View Page Source" when you open this link to get its code.) You will need to make a Requester account with Amazon Mechanical Turk, and upload some funds to the account. Then follow these steps:

    1. First, edit the question text to match what you are interested in. (You can use a simple text editor like Notepad.) Make sure to change the "name" (name for the question overall) and "value" (name for that option) in the radio button tags to reflect your question of interest. Also make sure to change the consent text to reflect your IRB / ethics approval.
    2. Next, upload the task to Mechanical Turk with these parameters: 1) We paid $0.02 (for our two quick questions) and recruited 100 workers, so the experiment cost $3 USD total including fees. It should be relatively quick to get 100 participants. 2) We allotted 5 minutes per worker, had the survey expire in 1 day, and auto-approved and paid workers in 1 day. 3) Workers did not have to be Masters to do the tasks. 4) We added the qualifications of: A) HIT Approval Rate % greater than 95; B) Number of HITS Approved greater than 50; C) Location is UNITED STATES.
    3. Copy and paste your code into the "Design Layout" section. Preview it to make sure it looks good, and then hit "Finish".
    4. When you are ready, choose "Publish Batch" for that task in Mechanical Turk's Create page. Go through the pages to make sure the task and cost look correct. Then hit "Publish".
    5. To view the results, go to Manage and then "Review Results" under the task. You can look at the data on that page together with the class, or "Download CSV".
    6. You can open the resulting CSV file in a program like Excel. For some quick analyses, you can count the number of "TRUE" values in a column with =COUNTIF(AB2:AB101, TRUE) where the first two values are the top and bottom cells of the range you want to calculate from. To calculate an average of values, you just use =AVERAGE(AB2:AB101). To calculate the number of values in a column matching a certain answer (like "PASTA"), you can use =COUNTIF(AB2:AB101, "PASTA"). This formula is case-insensitive.

    If you want an example, Prof. Tim Brady's Mechanical Turk Tutorial is a fantastic resource where he runs an experiment live in his talk (which inspired this demo).

  • The Original Mechanical Turk

    • The Mechanical Turk was an impressive feat that gave the appearance of a robot who could play chess, who was secretly operated by a hidden person. Spectators would see him like this:


    But this is in fact how he functioned:
    .

    You can learn more about the original Mechanical Turk on this Chess blog post.