Evaluation Cookbook

1. Describe the Innovation
Construct a clear description of the innovation that you are seeking to monitor. Hint

Exactly what will be different in the students' experience after the change you propose as compared to the current situation? The ideal experiment manipulates only one factor at a time, thus enabling very direct causal links to be explored. In practice, a number of changes may have to take place at once for reasons of expedience.

2. Decide the parameters of your experimental design
What are you going to compare with what? Will it involve a comparison of what happened last year (before the initiative) with the experience of the current year (in which the initiative is in place)? Perhaps only part of the class will experience the new learning situation, and their performance (or their judgement of their enjoyment of the course, or whatever) will be compared with that of their immediate colleagues who have not experienced the change. Or perhaps you plan to continue with your normal practice and compare the learning outcomes of your students with those of an equivalent class taught by a colleague at another institution where some sort of innovation has been put in place. Hint

3. Define "success"
Decide what outcome would be needed for you to consider your experiment to be a success. Perhaps the objective is to address some issue of recruitment onto a subsequent level course. Are changes going to be reflected in the students' academic performance? Will they be expected to enjoy their learning experience more, or to express more confidence and satisfaction with their achievements? Hint

4. Decide how to measure successfulness
Decide how your predicted outcome can best be measured. Other sections in this guide have more to say about the different ways in which you can obtain qualitative or quantitative estimates of some dimension which tells you something about the outcomes of interest. Hint

Be aware that what you measure, and what you are interested in, may be subtly or even profoundly different. Some things may be easily measured (like the scores in a multiple-choice examination) while others (like the depth of understanding of some concept) may be considerably more difficult to measure; and the temptation is always to take the simple course. On the other hand, good, simple proxy measures can often be found for the outcome of interest. It is not necessary that the measurement you collect be direct, but only that it is strongly correlated with what you need to know about.

5. Analyse your data.
Analysis of data gathered through an experimental approach will most likely focus on deciding whether your innovation has had the predicted effect. Is there a difference to be seen in the outcome measure(s) gathered between your original (control) and post-intervention (experimental) situation? Is the difference in the direction which was predicted? And is the difference greater than a change which might be expected by random chance alone; that is, is it statistically significant? Hint

Do not think about statistical significance as being an all or nothing thing but as an expression of your confidence in coming to a particular conclusion or making a particular claim.

Always begin the analysis with a general exploration or your data. Consider using confidence intervals first, as a good general comparison between datasets. If it appears that differences do exist, then proceed to some test of statistical significance.

Descriptive statistics (like an arithmetic mean) can be calculated, or some graphical technique (such as the plotting of a histogram) can be employed to display differences between your baseline (pre-intervention) and novel (post-intervention) measurements. Inferential procedures enable the exploration of the statistical significance of such differences. Basically, these latter procedures enable you to express the size of the differences between two (or more) groups in relation to the spread of the individual measurements within the groups.

Note: differences in average value are not the only possible interesting outcome.

Quote: a word of caution