|Route map: HomePublicationsEvaluation CookbookDesigning Experiments|
1. Describe the Innovation
Exactly what will be different in the students' experience after the change you propose as compared to the current situation? The ideal experiment manipulates only one factor at a time, thus enabling very direct causal links to be explored. In practice, a number of changes may have to take place at once for reasons of expedience.
2. Decide the parameters of your experimental design
3. Define "success"
4. Decide how to measure successfulness
Be aware that what you measure, and what you are interested in, may be subtly or even profoundly different. Some things may be easily measured (like the scores in a multiple-choice examination) while others (like the depth of understanding of some concept) may be considerably more difficult to measure; and the temptation is always to take the simple course. On the other hand, good, simple proxy measures can often be found for the outcome of interest. It is not necessary that the measurement you collect be direct, but only that it is strongly correlated with what you need to know about.
5. Analyse your data.
Do not think about statistical significance as being an all or nothing thing but as an expression of your confidence in coming to a particular conclusion or making a particular claim.
Always begin the analysis with a general exploration or your data. Consider using confidence intervals first, as a good general comparison between datasets. If it appears that differences do exist, then proceed to some test of statistical significance.
Descriptive statistics (like an arithmetic mean) can be calculated, or some graphical technique (such as the plotting of a histogram) can be employed to display differences between your baseline (pre-intervention) and novel (post-intervention) measurements. Inferential procedures enable the exploration of the statistical significance of such differences. Basically, these latter procedures enable you to express the size of the differences between two (or more) groups in relation to the spread of the individual measurements within the groups.
Quote: a word of caution
© All rights reserved LTDI and content authors.
Last modified: 25 March 1999.