Thank you for your interest in this study. Please use a laptop or computer to take part in this research - do not participate using a smartphone. Please consider this information carefully before deciding whether or not to proceed with this research.
Purpose of the research: To examine forecasting of effect sizes in psychological research.
What you will do in this research:
Read descriptions of research in psychology.
Express your beliefs about effect sizes through probability distributions.
Specify the plausibility of a hypothesis.
Answer knowledge-based questions on the effect size measure Cohen's d.
Time required: This study will take approximately 15 minutes to complete.
Risks: If you choose to participate, the effects should be comparable to those you would experience from viewing a computer monitor for 15 minutes while using a mouse or keyboard.
Confidentiality: Your participation in this study will remain confidential. The data will be shared publicly in anonymized form. No personally identifiable information will be associated with your data.
Participation and withdrawal: Your participation in this study is completely voluntary and you may withdraw without penalty at any time.
How to contact the researchers: If you have questions or concerns about your participation, or if you want to request a summary of the research findings, please contact: EJ.Wagenmakers@gmail.com, Department of Psychology, University of Amsterdam, Nieuwe Achtergracht 129-B, 1018 VZ Amsterdam, The Netherlands.
Whom to contact about your rights as a participant in this research: For questions, concerns, suggestions, or complaints that have not been or cannot be addressed by the researcher, or to report research-related harm, please contact Lourens Waldorp, L.J.Waldorp@uva.nl.
Compensation: One participant will be awarded a prize of €100, where your probability of winning depends on the quality of predictions you make during the study.
I understand the nature and purpose of this research and I agree to participate in this study. I understand that I am free to withdraw at any time without incurring any penalty. Please be sure to print a copy or to save a screenshot of this consent information for your records.
This study aims to investigate probabilistic forecasting of effect sizes in psychology.
You are about to read the description of 9 studies in psychology. All studies feature a comparison between two groups (for example, a treatment and a control group). The studies are recent replications of classic and contemporary findings in psychology conducted as part of a large scale replication project.
Cohen's d Knowledge Test
This study focuses on the effect size measure Cohen’s d for studies in psychology. To assess your prior familiarity with Cohen’s d we will now ask you two questions.
As a rule of thumb - what Cohen’s d values are typically regarded as small, medium, and large in the field of psychology?
Consider a comparison of two groups. The dependent variable in group A has a mean of 3, and that in group B has a mean of 2. The pooled standard deviation equals 2. What is Cohen’s d?
Cohen's 𝞭 Reminder
Thank you for answering the questions about Cohen's d. Importantly, our study is concerned with Cohen’s 𝞭, the population version of Cohen’s d. This means the focus is on the “true” effect size the way it would manifest itself in a hypothetical study with an infinitely-large sample size.
Cohen’s 𝞭 expresses the difference between two means in terms of the standard deviation.
As a rule of thumb: a Cohen's 𝞭 of 0.2 is regarded as small, a Cohen's 𝞭 of 0.5 as medium, and a Cohen's 𝞭 of 0.8 as large.
The figure below visualizes the overlap of control and treatment condition for different values of Cohen's 𝞭. You can operate the slider to get a feeling for what values correspond to which overlap of control and treatment group.
Your task will be to specify what effect size you would expect in several studies.
Instructions for Condition 1
For each study, you will be assigned to one of three conditions where you need to specify the population effect size in different ways. In this experiment you will go through all three conditions.
In condition 1 your task will be to specify a point prediction. You can specify your best guess for the effect size with the slider on the left. As you move the slider, the arrow in the graph will move along, representing the value of your best guess. In addition, you can tick the box ''I already know the outcome of this study''. In this case, we will exclude the study for our analysis; however, ticking this box does not affect your probability of winning.
Instructions for Condition 2
In this condition you specify a prior distribution.
A prior distribution represents your uncertainty about the true value of the effect size.
To keep things simple, we will use a normal distribution.
You can adjust the mean (=best guess) and standard deviation (=uncertainty) of your distribution using the sliders.
You should put more mass on values that you consider more plausible and less mass on values that you consider less plausible.
The more certain you are the more narrow your distribution should be.
The initial prior distribution that you should adjust is shown below on the right:
Below is an example that you could specify if your best guess is 0.5 and you are pretty sure that the effect is between 0 and 1.
Instructions for Condition 3
In this condition, you will first be asked to specify the probability of the null hypothesis.
If you choose a value of 1, you indicate that you are absolutely certain that the effect does not exist.
If you choose 0, this indicates that you are absolutely certain that the effect exists.
A value of 0.5 indicates that you believe the effect is as likely to exist as not.
Next you will be asked to specify your prior distribution for the effect size under the assumption that it exists.
Please provide more information below as you wish.