|Route map: HomePublicationsEvaluation CookbookExemplars|
|Example of Questionnaires and Group Interviews|
Where and when was the study carried out?
Departments at different universities integrated the software into their teaching in different ways, and, indeed, integration methods varied between members of staff at the same university. Some got the students to work through the package in supervised lab sessions either after or, in some cases, before meeting the topic in lectures. Some used the package in a similar way but did not staff the labs. Some replaced conventional lectures on a topic with directed study of part of the package. Some used the software as a remedial aid only, or purely as an optional extra resource whose existence was pointed out to students.
The software itself had already been extensively evaluated, but this study was designed to evaluate the various ways of integrating it into courses. Both staff and student reactions were sought, but only the aspects involving students are reported here.
What methods did we use?
To gain a deeper insight into student views, two structured group interviews were held with students. To encourage comparison of integration methods and to spark discussion, each of these was planned to involve a group of about ten students from each of two universities who had studied the same topic or topics. The sessions were run by video-conference so that no one had to travel.
How many staff and students were involved?
We expected ten or so students from each university to attend the two discussion interviews. However, on one occasion only five turned up at each site and on the other occasion no students could be persuaded to attend at all from one site. A member of the staff involved in the study was present at each site to lead and record the discussion, as well as to run the short ice-breaking session held prior to the actual interview.
What were the aims and objectives of the evaluation?
What did we find out?
Factors found to influence how useful students found the package were:
Surprisingly, students who found the pace of the lectures 'too fast for me' were less likely than other students to find the CBL software useful.
What are our reflections on the evaluation methods used in this study?
The questionnaires yielded information in bulk and provided reassurance as to the reliability of the information. Despite being quite long (4 pages), there was little evidence of lack of care in their completion. Almost all students were still willing to write sensible and sometimes extensive comments in the open questions ('Main drawbacks', 'Main benefits') at the end. This was helped, as always in questionnaires, by making the early part of the questionnaire quick and easy to complete, and maintaining an interesting and logical progression of questions.
By getting students to write their matriculation number on the questionnaire, further analysis of how responses compare to, say, student ability or success in the course can be carried out later.
The group interviews gave fuller information and being a group session allowed comments to be taken up by other people and explored from several angles. Five students and one staff member at each site actually worked very well with all students feeling that they should say something. Having students from two different groups was also very successful with fresh ideas and differing experiences providing interest and allowing discussion of new points that had not occurred to one or other group. Using video-conferencing was a bit intimidating at first but people soon forgot about it. Staff involved felt that the benefits of the two-group sessions out-weighed any hassle in setting up the video-conferences.
Having the software at hand to refer to was helpful, and a demonstration at the start helped jog memories and break the ice. An initial ice-breaking exercise helped as well, as would have, in retrospect, provision of name badges. Forty-five minutes was found to be ample time for a session.
In this study the interviews were conducted after at least some of the questionnaire data was studied which allowed queries raised by the questionnaires to be investigated more fully in the interviews.
Judy Goldfinch and Kirsty Davidson
© All rights reserved LTDI and content authors.
Last modified: 26 March 1999.