Evaluation Cookbook

Carrying out an evaluation involves time for both you and your students. Whether it is time taken out of a lecture or during a lunch hour, it is time away from a student's study or social time. Therefore, before you start, it is important to be clear in what you want to find out from your evaluation, which evaluation method is going to be the most appropriate for your needs and how many students it is going to be appropriate and practical to include in your study.

How you go about selecting your student sample will have an effect both on the information gathered and the impact that your findings might have. A number of sampling options are possible but it is worth bearing in mind what the benefits and limitations of each might be.

Letting your sample select itself:
Making materials available for students to look at or handing out questionnaires for students to complete in their own time can result in a disappointing if not unrepresentative sample for an evaluation study.

However, you can gain a snapshot impression of students' general opinions. Watch that you don't try to draw any major conclusions from the responses of a small section of a class sampled in this way. Individuals who just happened to turn up at the last lecture of the term can have very different reactions to the majority of the class with whom you are planning to use a computer package during the following year.

Asking for volunteers:
Again, this can produce an unrepresentative sample for your evaluation. Volunteers are likely to be the most conscientious of the class or the students who are just trying to please you.

However, when you are carrying out a fairly extensive and time consuming evaluation study, you are probably going to depend on the good will of people who are motivated enough to volunteer to help.

Selecting the sample yourself:
If you pick your own sample of students, you have the opportunity of being able to identify the students who are likely to be most co-operative or a group of students with the most appropriate skill levels. You can also select a random sample of students in order to try and get a more representative cross section from the class.

Watch, however, that by selecting one group from a class and involving them in an evaluation study you are not perceived as giving one group of students additional / better / preferential support or tutoring than the rest of the class. It can be easy for students complain that they feel disadvantaged from their peer group in some way.

Involving the whole class in the evaluation study:
This will provide a more representative sample than by taking just a small section, but you could have problems with managing and working with data from an evaluation study with large numbers. If you are planning to involve the whole class, you might also want to consider whether or not you are going to timetable this study as part of the normal syllabus or to include it as an additional extra. Ten minutes to complete a questionnaire at the beginning of the class doesn't cause too much of an upheaval, but asking all of a large class of students to work their way through a CAL package without evaluating the materials with a small group of students beforehand, could prove to be rather an unpredictable exercise.

Jen Harvey
Implementation Support Consultant,
LTDI, Heriot-Watt University.