Audience
This page is intended for use by students and researchers in the University of Cambridge Schools of Technology and Physical Sciences whose research involves recruiting people from outside your own research team to take part in experiments. It is part of a larger set of research guidance pages on work with human participants.
Issues to note for ethical review
This page gives general guidance relating to conduct of experiments. The following issues are particularly relevant with regard to ethical review:
- Recruitment
- Treatment of Participants
- Informed Consent
- Debriefing
- Data Retention
- Incentives and Compensation
Definitions
A controlled experiment is an experimental setup designed to test hypotheses.
A controlled experiment has one or more conditions (independent variables) and measures (dependent variables).
A randomised controlled trial is an experiment in which participants are assigned at random to different conditions, in order to test in an objective way which of several alternatives is superior.
A pilot study is a trial run of an experimental procedure, not expected to produce valid research data.
Controlled experiments may or may not require human participants. This page is only about controlled experiments involving human participants.
Introduction - Controlled experiments
Controlled experiments are difficult to design and analyse. Students in experimental psychology take practical classes in experiment design before they attempt to conduct their own original research. However, all experiments with human participants conducted by students in Technology and Physical Sciences have the character of original research, from a psychology perspective. It is therefore a common experience for technology researchers to find that their first experiment produces meaningless or null results, often after a great deal of effort. This is wasteful of time and resources, both for the researchers and participants, so should be avoided. For this reason controlled experiments should only be carried out by researchers who are trained in experimental design and analysis, or under direct supervision of researchers with suitable training. If you have little experience you should consult senior researchers.
Some important considerations include design for:
- Reliability (Would you get the same measurement again?)
- Validity (Are you measuring what you claim to be measuring?)
- Internal validity is the relationship between your measurement and what you think it tells you about the experimental task.
- External validity is the relationship between what you measure in the lab, and the phenomenon in the outside world.
A well-known reference book is:
Krik, R.E. Experimental Design: Procedures for Behavioral Sciences.
Practicalities - Controlled experiments
Preparation
Experimental design
It is easy to make serious errors when you first attempt to design a controlled experiment. There are many textbooks and online guides - make use of them. Ask an expert to review your experimental design , and try it out in advance with several pilot studies.
There are a number of critical factors that could cause the experimental results to be invalid, and it is important to anticipate these and avoid them. One way to do so is to plan, in advance, how you propose to write up the results of the experiment. Think about the conclusions that you would draw if the result of the experiment is consistent with your hypothesis. How would you present your results in a way that convinces the reader that conclusion is justified? What would the results of data analysis have to be to support this kind of presentation? What experimental method will produce data that can be analysed in this way? What is the best way to express an hypothesis compatible with that method? If you can explain your reasoning in this way, before you start the experiment, you will have a much better chance of avoiding the invalid and/or inconclusive results that are so often obtained by inexperienced experiment designers.
Pilot studies
It is very hard to get an experimental procedure right the first time. Every experiment should therefore include at least one pilot session, with a participant whose results you expect to discard from the final data analysis. For this reason, it is common to use a pilot subject whose results you would not expect to be valuable - for example, because they are aware of the experimental hypothesis, have specialist expertise, or similar. Family members and (fellow) students can be useful.
Where an experimental paradigm is unconventional, or there is substantial uncertainty about either the measures or the hypotheses, you should consider a pilot study involving several participants, in which each of the experimental conditions is used, and a preliminary data analysis can be conducted.
Recruitment
In order for research to have good external validity, the recruited participants should be representative of the population about which you want to make research conclusions. However, in practice, undergraduate and graduate students are often recruited because this is easier. If you plan to do this, it is a good idea to think in advance how you will justify it to reviewers or assessors of your work.
Where children are involved in research, recruitment is likely to be via schools or parents. Some experiments with children, or with vulnerable adults, may also require that members of the research team undergo a Disclosure and Barring Service (DBS) check.
Where participants have been recruited on the basis of a medical condition, it is likely that your research will require approval via the NHS Research Ethics Service.
It is increasingly common to recruit experimental participants via platforms such as Amazon Mechanical Turk or Figure Eight (formerly CrowdFlower). There are many distinct ethical implications of experiments conducted using these tools that are rather different to those arising in the conduct of experiments in a laboratory. For further guidance, see the page on Crowd sourcing experiments
Conducting the experiment
Treatment of participants
In most experiments, participants are asked to carry out an experimental task while being observed, or while their responses are being measured. It is of paramount importance that participants are treated with dignity and respect. Remember that you are in a position of power from the participants' perspective. You need to inform yourself about participants' rights and then disclose these rights to the participants. Among those rights:
- The right to stop participating in the experiment, possibly without giving a reason.
- The right to obtain further information about the purpose and the outcomes of the experiment.
- The right to have their data anonymised.
This list is not exhaustive.
It is often the case that people being asked to use new technologies while under observation find the experience stressful. It is very important to reassure participants that your objective is to identify possible faults in the technology, and not to test the participants' own ability or intelligence. If they have trouble completing an experimental task, you should reassure them further, emphasising that they have had this experience because the technology is inadequate, and that it is not a reflection on their own ability. Experimenters should never offer any comment with regard to participants' intelligence, aptitude, or other factors that might give people the impression that a scientific judgment of their ability has been performed. This is especially the case if standard psychometric tests are being employed as one of the experimental measures. An experimental situation in technology or physical sciences is not a proper psychometric assessment, and psychometric test results should not be directly communicated to participants.
Informed consent
It is very important for participants to understand that their participation in the experiment is completely voluntary. In order to ensure that they understand this, experimenters should prepare a 'consent form', stating the nature of the experiment and the rights of the participant. Before the start of the experiment, participants should be asked to read this form, and sign it to indicate that they have read and understood their rights. An example consent form can be found on the University Research Ethics pages.
You may wish to assure participants that no personal data is collected, or if it is collected, that it will not be published, and will be destroyed. These things can be mentioned in a consent form.
If a participant appears to be experiencing any stress (for example due to task difficulty, or perhaps through factors unrelated to the experiment), it is important to remind them that they are free to withdraw at any time.
If a participant is experiencing physical pain (e.g. because of extensive use of the mouse for the task) then abort the experiment immediately and consult a senior colleague or the appropriate university ethics committee for advice on whether to proceed with the experimental procedure.
In the case of children (in the UK, under the age of 18), consent must be given by a parent. The experimenter may also be subject to a Disclosure and Barring Service (DBS) check.
Participant briefing
For the purposes of experimental control, every participant should be given the same instructions before they commence the experimental task. Briefing instructions are normally written out in full, in order to ensure that this is done. The instructions can either be read from a script by the experimenter, or given to the participant to read, after which they are asked if they have understood everything, and are ready to start.
If an experimenter script is used, it is a good idea for this to include all instructions and actions that the experimenter must carry out throughout the experimental session. This script should be tested during the experimental pilot, and helps gain maximum value from the pilot as a 'debugging' session for the main experimental procedure.
Debriefing
At the end of an experimental session, participants should normally be debriefed. Debriefing involves a short interview, often semi-structured, with some prepared questions that you ask every participant, and follow-up questions in the event that interesting points are raised.
This provides a valuable data collection opportunity, especially as participants' subjective experience of the experiment could be of value in interpreting either their individual performance, or behaviour observed more broadly across the sample group. It may be useful to discuss your experimental hypothesis with participants, because they might well be able to warn you of potential problems with task validity, from their perception of the task.
Whether or not you expect to gain useful information for research purposes, debriefing also provides an opportunity for the participant to reflect on the experience they have had. It is a good idea to complete the debriefing interview by asking whether there is anything else the participant would like to tell you.
Incentives and compensation
It is recommended to compensate participants for their time, although compensation need not be financial. People may be very willing to participate in experiments from which they gain interesting feedback, or experiments that are intrinsically enjoyable (for example games). A token gift (chocolates, a book or report, software, or a memento such as copies of a scan) may be sufficient reward. Nevertheless, many departments in Cambridge routinely recruit experimental participants, and payment may be expected after a formal experiment. If the participant has incurred direct costs such as travel these should be reimbursed.
If a participant chooses to withdraw, or not to complete the experiment, they should still be compensated. Experiments in which incentive payments are varied according to task performance are considered to be unethical. A standard procedure where incentive is a central hypothesis (for example experiments in economic judgment) is to offer participants variable payment at the outset, but then to pay all participants the same (usually maximum) amount at the close of the session.
The university has issued rules on procedure to be used, and how much compensation should be given to participants. Finance division policy on payments to research volunteers is described here.
Data Retention
If the data collected does not include any personal data, then the data may be retained. If they do contain personal data, then they fall within the terms of the Data Protection Act. Personal data should be kept secure. Data that would allow a participant to be identified should be kept in a separate place throughout the research project, with an anonymised code used during analysis work and at publication time. It is good practice to destroy any personal data after a stated period of time. In most cases, experimental data is used only by the person conducting the experiment. If this is not the case, see the page on academic research involving personal data.
Significant ethical issues
This page is intended to address relatively routine research in the schools of Technology and Physical Sciences. If your experiment involves any of the following activities, then more serious questions must be addressed, and you will need to consult the relevant university ethics committee:
- Experiments involving animals are subject to the animals scientific procedures act
- Medical and other invasive experiments on human participants must be reviewed by the NHS research ethics service.
- Psychological manipulation of human participants (deception, emotional manipulation, etc.).
This list is not exhaustive. When in doubt consult senior colleagues and relevant university ethics committees.
References
Some popular books are:
- Kirk, R.E. Experimental Design: Procedures for Behavioral Sciences.
- Robson, C. Experiment, Design and Statistics in Psychology
Future information: to include references to appropriate Cambridge courses on research and experimental design in Social Psychology, Experimental Psychology etc.
The initial version of this page was drafted by Per Ola Kristensson.
All comments and feedback are welcome. Please send any feedback to ethics@tech.cam.ac.uk