Evaluation Cookbook

1. Planning
Make sure a checklist is appropriate for the type of analysis you wish to carry out; on the one hand, checklists are good for quickly identifying issues, but they may not provide enough information to allow you to rectify any problems. Hint

Look at the material that you want to analyse with the checklist; did you have objectives which you can specifically test to see if they have been met? Can you do the same with previously unstated objectives?

2. Identifying you population sample
Decide who your population is and whether there is any information about their previous experience - for example, qualifications, previous courses, expectations etc. - which may help you interpret the information they provide. You can ask for this information in the checklist. Hint

3. Design
Carefully choose the best question type. Often, you may want a simple yes/no answer, e.g. did you find X useful, was it easy to carry out Y, etc. However, sometimes supplementary choices are appropriate. For instance, when assessing whether objectives have been met it might be useful to determine whether the respondents felt the objectives had been fully or partly met. Whether respondents had prior knowledge of the material might also modify the meaning of their answer. Hint

As closed questions are easy to answer you can ask many questions at once without risking overloading the user.

Keep the wording clear, trying not to introduce terminology. Rather, try to directly relate the question to specific parts of the materials, such as objectives. Hint

Try to group questions logically. Use subheadings and clear instructions to lead the users through the questions.

Pilot the checklist with someone who knows the material. As well as comments on clarity etc., they may be able to comment on the balance of the questions.

Example

4. Delivery
Paper or electronic delivery of the checklist evaluation is possible. WWW based forms can be used to collect responses efficiently, but should only be used where appropriate - for instance when the materials being evaluated have already been delivered electronically or possibly when face-to-face collecting of checklist feedback is not practical.

Low response-rate is a considerable problem with checklists as with other form-filling evaluations. You might want to consider how you can make completion of the checklist more relevant to the students - by making them feel that they will get something out of the exercise. For instance, you may make the questions relevant to revision by reiterating the objectives and asking the students to indicate whether or not they felt they had been met. You could also include ideas for further study/reading for each objective. Hint

Time your evaluation carefully - should your checklist be delivered directly after the implementation, whilst the class mind is focused (and the whole class is captive) or after a period of time (when concepts have been strengthened but individual details lost)?

5. Analysis
In addition to a collective analysis i.e. what proportion felt that a particular objective had not been met, you may want to relate different answers from the same respondent. Alternatively you could group students according to their responses to one particular question. Hint

A checklist can often be used very effectively as one component in an evaluation - possibly to identify specific issues that can be investigated further in a focus group or structured interview.

Try to feedback your results to your students and to follow-up any recommendations.