Week 1

Experimentation in Psychology and Linguistics

Yaron McNabb

8/2/2018

 

Introduction

Plan for this course

    • Why run experiments?

 

  • General principles of experimental design

 

 

  • Methods used in experiments in psychology and linguistics

 

 

  • Data analysis

Why run experiments?

Where experimentation is useful

Obvious cases:

    • Hypothesis testing

 

  • Evaluating models

 

 

Where experimentation is useful

Less-obvious cases:

    • UX/interface design – not covered in class, but can be a (final) project!

 

  • Evaluating observational studies, e.g. from data mining
      • Pattern recognition – okcupid example later today

     

 

 

  • Human behaviour in social/emerging media
      • Social engineering – facebook example later today

     

 

 

  • Developing gold standards
      • Human capability as a baseline (at least) – aliexpress and microsoft text reading algorithms later today

     

 

Evaluating observational studies

(Thought) experiment

What should I say on a first message on okcupid?

https://theblog.okcupid.com/exactly-what-to-say-in-a-first-message-2bf680806c72

Avoid physical compliments!

    • Does adding awesome to an email increase the response rate?

 

  • Does adding beautiful decrease it?

 

 

Avoid physical compliments?

    • Is the addressee reacting to the words or the sender themselves?

 

  • Who is the sender? Are they always effusive?

 

 

  • Who is the addressee? What factors would change the addressee’s likelihood to reply to any message?

 

 

  • Is the addressee actually beautiful?
      • If so, this may affect two separate things (e.g., expectation to be lauded or compliment aversion)

     

 

(Liberally drawn upon O’Neil & Schutt 2014)

Designing an experiment to test okcupid’s observations

    • What are the independent and dependent factors?

 

  • How could we mitigate the confounds and random factors?

 

 

  • How do we make sure we have enough statistical power?

 

 

Take-home message

We can do better than correlation is not causation:

    • Evaluating dependent and independent factors

 

  • Anticipating confounds and extreneous factors

 

 

  • Better understanding the characteristic of a complex problem

 

 

Human behaviour in social networks

Emotional Contagion

    • Emotional status transferred via emotional contagion

 

  • Issues:
      • correlation?

     

  • misspecification of contextual variables?
  • failure to account for shared experiences?

 

 

  • Addressing these issues with a controlled experiment

 

 

Factors to consider

    • Contagion as a result of interaction with, or just exposure to, a (happy/sad) person

 

  • Passed on verbally or non-verbally?

 

 

  • Correlation between positive and negative moods:
      • The happiness of others might make us sad: alone together social comparison effect

     

 

Facebook’s experiment

Does exposure to positive/negative content lead to posting content consistent with exposure?

 

    • 689,003 (unsuspecting!) participants exposed to emotional expressions on their News Feed (Kramer et al. 2014)

 

  • Emotional content as a between-subject factor

 

 

  • Depending on condition, positive/negative emotional content was reduced from the News Feed

 

 

  • Emotional valence determined by Linguistic Inquiry and Word Count software (LIWC2007)

 

Verbally-stransmitted social contagion

 

Characteristics of social contagion

    • Transmitted also by mere exposure (not just direct interaction)

 

  • Verbally-transmitted (not just non-verbally)

 

 

  • No negative effect of positive posts

 

 

  • Small effect size

 

 

  • Dependent variable influenced by more than just facebook posts

 

 

Relevance of social contagions

 

 

 

Developing gold standards

Natural language comprehension

    • Developing AI-driven virtual assistants and chatbots

 

  • Emulating humans’ natural language understanding

 

 

  • Achieving human-like ability is already a challenge, not to mention surpassing it

 

 

Reading AI scoring better on SQuAD

 

Other examples of gold standard

    • Text classification

 

  • Perception and categorization (e.g. of objects)

 

 

  • Later this course: comparing human classification of tweets with text classifiers

 

 

Course structure and expectations

Structure

    • Lecture and discussion on Tuesdays

 

  • Lab on Thursdays

 

 

  • Attendance mandatory

 

 

Expectations

    • Hands-on course

 

  • Focus on discussions and collaboration rather than lectures

 

 

  • Developing a variety of skils
      • Experimental design: Follow-up experiment and final paper (weeks 2-3 and 6-9, respectively)

     

  • Abstract writing: homework assignment (weeks 2-3)
  • Research proposal writing (weeks 6-7)
  • Peer-review: review abstract (week 3), review experimental proposal (week 7)
  • Scripting and coding in various languages: R, JavaScript, Matlab, Python (throughout the course)
  • Data analysis (throughout the course)