Syllabus, Draft version
Yaron McNabb, firstname.lastname@example.org (lecturer)
Sjoerd Stuit, email@example.com (lecturer)
Frans Adriaans, firstname.lastname@example.org (lecturer)
Rick Nouwen, email@example.com (coordinator)
1 Goals of the course
This course is about the important role of experimentation in artificial intelligence. In this course, you will be introduced to standard experimentational methodologies in psychology and linguistics and learn to apply them within the broader context of AI. You will also learn to use standard statistical analyses to interpret experimental data and draw inferences and conclusions, which in turn will help build models that predict and simulate future human behaviour. The case studies and research questions used in this course will be taken from AI, psychology and linguistics. Thus, you’ll learn how to use experimental methodologies and statistical analyses for a broad array of applications, from arguing in favour of a certain theory to arguing for a new user experience that yields greater customer retension or increased revenue.
At the end of this course, you will be able to:
- design experiments on the basis of a given model and research question in the realm of linguistics and/or psychology with relevance to artificial intelligence
- implement experiments using various computational techniques, extract crucial data from experimental responses
- interpret experimental results and report on these in an appropriate way.
The students will acquire and develop the following skills:
- Developing research questions and hypotheses based on prior data and observations
- Using the appropriate experimental methodologies to address the research questions and test the hypotheses
- Data analysis:
- Descriptive statistics (mean, median, standard deviation, distribution, variance)
- Choosing the correct statistical analysis (t-test, ANOVA, regression)
- Data visualization that best communicates the insight the data provides.
- Deriving appropriate implications that can inform new algorithms or new models that predict future human behaviour.
Both science and industry are interested in creating precise formal models of human behaviour and cognition. To help build, test and optimise such models, one needs to create and run experiments. Students participating in this course will learn (I) how to design experiments given an existing model, (II) how to implement experiments using various tools and, finally, (III) how to extract data from the recorded responses for analysis purposes.
Most theoretical claims in linguistics and psychology are made by positing a formal model. The aim of such models is to make precise predictions. Moreover, the predictions of a model need to be tested with formal experiments. The results of the experiment may (or may not) lead to changes in the model and thus to a new set of testable predictions. Essential in the modelling-experimenting cycle is careful experimental design. The course covers the practical and theoretical considerations for experimental research, from posing the research question to interpreting and reporting experimental results.
Experiments are frequently used in industry, too. For example, to assess how people use interfaces – Where do they look or click? How does a particular text influence their subsequent choices? – to discover the best design of a product or the appropriateness of a user model. For example, do people learn what the model predicts them to learn? Do they have a more immersive experience when a model guides adaptation of the software?
In this course you will get an overview of various experimenting techniques that are used worldwide and some even by researchers in Utrecht, especially at the Department of Psychology and the Department of Linguistics). You will learn how to use such techniques for testing specific models, their uses as well as limitations. In the practicals you will also gain hands-on experience with the implementation, data manipulation and data analysis steps of experimenting.
3 Format of the course
Lectures: the main purpose of the lectures is to introduce the essentials of experimentation. Some lecture slots are partly used for presentations on the experiments prepared in the practicals.
Practicals: during the practicals, you work in small groups to implement an experiment and/or analyse data. The practicals are also used to work on your final project.
Attendance in both lectures and practicals is compulsory. There will be attendance sheets.
The learning goals will be examined in two ways:
- You will work in teams on practical assignments, which will involve implementing experiments and working with experimental data. You will also prepare a presentation and a conference abstract for your assignment. All of these will be graded.
- You will work in pairs to design and implement an experiment on a topic of your own choice (see below) and write a short paper reporting on the experiment. Implementation and paper will be graded.
There is no exam. The final grade is made up of:
- Practical assignments, including conference abstracts and presentation: 40%
- Final project, including paper: 60%
In order to pass the course, the following requirements need to be met:
- The weighted unrounded final grade is at least 5,5.
- An unrounded grade of at least 4,0 for each assignment (including presentation assignments)
- An unrounded grade of at least 4,0 for the final project
- Attendance of at least 80% of all the sessions that make up the course
If you fail a practical assignment, you will be allowed to hand in a revised version, which cannot received a grade higher than 5,5.
If you fail the final project (grade < 5,5), you will be allowed to hand in an improved version if and only if the unrounded grade for the project is at least 4,0.
Practicals and assignments
Bring your own device – Although the labs have computers in them, we strongly asvise that you bring your own laptop. This way you have full control over installing pragrams and packages and can easily continue working on the assignment after the practical session is over.
R – For data analysis we will use R, the standard programming language for statistical computing of experimental and non-experimental data. If you took the Cognitive Modeling course (INFOMCM), you will already have had some experience with R. If you did not take this course, there will be ways to catch up. A good start is the R crash course that was offered in week one of INFOMCM. You will find a copy of that document on blackboard.
In the final project you work in pairs (or, if need be, individually). Early in the course you get to choose a topic and formulate a research question. We’ll provide a list of topics on BlackBoard, but you could also discuss you own ideas with the lecturers to see if they are suitable. On the basis of your research question, you design, implement and run an experiment. You then analyse and interpret the results. Given the short time that is available it will often be impossible to get enough participants to allow you to draw conclusions that are based on statistical significant results. In that case you can still attempt an analysis and interpretation but will also be expected to discuss the limitations.
You’ll report on the experiment by producing a short paper. You will receive instructions on how to write this. While you may work on the experiment in collaboration with a fellow student, you write the report individually. That is: each student must hand in a unique report on the experimental research they have done, even if they’ve done so in collaboration with another student.
4 Tentative Course Overview
- Case-studies from real life and the industry in which results from a controlled study shed light and inform assumptions and predictions about relevant aspects of human behaviour
- Survey of experimental desgins (e.g., surveys, reaction/response times, truth value judgement task)
- Students divide into smaller groups which run pre-designed several different experiments, i.e. collect data, which the class will later use as a database to analyze and draw implications and ideas for follow-up studies.
Homework: run pre-designed experiments
- Basics of descriptive statistics, types of data and appropriate statistical analyses
- Groups analyze data from their respective experiments, present to the class
- Class discussion about the implications of each study and over implications from the individual studies, ideas about follow-up experiments.
Homework: write a conference abstract about the experiments
Statistical analyses: regression, ANOVA, ANCOVA, t-tests, correlations
Homework: design follow-up experiment
Dealing with confounds:
- What are confounds?
- Spurious correlations
- Underlying causes
Examples from own work: Image-statistics and Awareness – Modelling Confounding variables – Removing Confounding influences
- Use literature to create/replicate an experiment
- Connection to Lecture: Using the experimental design from the Lecture example
- Approach: Debugging
- Language: Matlab
Methodology and data visualization:
What’s the best experimental design (for the research questions)?
Task: analyze experimental design papers
- present various designs
- have students prepare a critique or pitch the design
- submit notes before class
- Cover a large array of topics – from low level (number of participants, within vs. between designs)
- Model comprison: regression probabilistic and others
- Corpus linguistics
- Artificial language modelling
Ethics in AI and experimentation
Guest lecture (graduate, industry)
Individual meetings about final projects
Final project papers due