Evaluation Cookbook

Where and when was the study carried out?
The evaluation study involved a class studying Manufacturing Systems during the year 1997/98. This was a one semester course which aimed to cover a basic introduction to manufacturing systems, including stock management techniques. Previously to this academic year, the class had been taught using a traditional combination of lectures, tutorials and labs. The course lecturer produced 2 computer based learning modules which cover the equivalent material to 4 hours of lecturing. These modules were integrated into the class and the related lectures removed from the class programme. An introductory lab session was added to introduce the system to the students. The students were also given a notes template to accompany the modules.

How many students were involved?
The class is composed of several distinct groups of students: Business, Engineering 1 and Engineering 2. Ninety-three students were involved in total.

What were the aims and objectives of the study?
The evaluation aimed to investigate several areas. These were:
 Usage of the CBL materials
 Level/depth of content covered
 Student attitudes to the use CBL
 Appropriateness of organisation of the class (e.g. group size)
 Notes template
 Learning development of the class.

Which evaluation techniques were used?
Two data collection techniques where used in this study:
 Confidence Logs and
 Questionnaires

Two questionnaires were designed for the study: The first was very short, concentrating on students' previous experience with computers and CBL materials, attitudes to CBL and their initial reactions to the system. The students were given this after their first use of the system during the supervised labs. This was intended to correspond with the steepest part of the learning curve, before the students had become familiarised with the interface and system design.

The second questionnaire was given to the students after they had finished using the system. This was more substantial than the first and covered a range of areas, not only those relating to the modules but those involving wider organisational and support issues for the class.

In addition to the questionnaires, the students were asked to fill in 2 confidence logs - the first before they had used the material, to provide a baseline of information. The second after several weeks use to see any development in their confidence during this period. In this case, the confidence logs consisted of 8 statements based on the knowledge and skills objectives of the course. Information from the confidence logs gives an indication of the change in confidence in these areas during the time investigated. Of course, any changes cannot be tied specifically to the use of the modules but it does show the effect of this part of the class as a whole.

What did we find out?
The response rate for the investigation as a whole was very high. The following percentage return rates were achieved:

The results of this evaluation indicated that the new resources combined effectively with the existing material, as expected. An interesting result was the difference between the different categories of students. The confidence logs revealed previous knowledge of particular parts of the course by the business students of which the lecturer was unaware. Further investigation revealed an overlap in material with a second year class for the business students.

What were my reflections on the study?
The evaluation highlighted parts of the material which the three different groups in the class found challenging, which were not the same areas! This information allowed the lecturer to further tailor the class to the specific mix of students.

Helyn Thornbury .
University of Strathclyde