Involving your participants:
Before the start of the evaluation, explain what your evaluation study is about and how the data you collect is going to be used. Also try to make it clear to students that you are evaluating the software and not them.
Allow some time for people to ask any questions before and after an evaluation session and make sure that they know what is to be expected of them if they become involved in your study. If you don't know the interviewees and they don't know each other, you might like to have coffee or tea first so that you aren't going into the evaluation phase of the session as complete strangers to each other. You might also feel that it is necessary to allow some time when the students can be in the room without you.
Before you start asking questions relating to the evaluation, try and relax the participants so that they feel comfortable both talking to you and the rest of the group.
Try and value people's opinions and thank them for taking the time to become involved in your study. If possible, feedback any data collected and act on the relevant comments made during the evaluation study.
The venue for your evaluations sessions:
Try and organise an appropriate venue for your evaluation study. If you are planning to have a discussion session, select somewhere where students are going to feel comfortable and able to discuss their feelings: a shared computer lab. might be the only place where you can run your practical session, but a seminar room with access to the software on a laptop, is going to be far more conducive to an evaluation discussion afterwards. When carrying out a pilot evaluation study, you might also want to select a similar environment to that which you are planning to use with the full class. This will enable you to evaluate the practicalities of your planned implementation: can all the students access and use the software? is there space for groups of students to work together?
Timing your evaluation study:
It is important to try and plan your evaluation study as far ahead as practical. This allows you to take course timetabling, reading weeks, examination weeks and holidays into account. It can be annoying to find that group of students have disappeared off on holiday when you just found time to be able to carry out the second part of your study. In addition, students are going to less inclined to turn up at the very beginning/end of terms or just before exams.
The number of times you involve students in an evaluative study of any sort can also influence your findings: students, particularly first year students can get questionnaire fatigue. Asking students to complete a questionnaire after a series of questionnaires can result in a lack of interest or quality in their responses, particularly if there wasn't any follow-up action to their previous recommendations.
The timing of your evaluative study relative to your teaching intervention can also affect your findings: too early and your students might not have the appropriate knowledge, too late and they might have forgotten how they felt while using a piece of software. Too often and you might miss any gradual changes, too few times and you could miss a sudden change.
Issues of personality:
Carrying out evaluations necessitates a level of trust e.g. between you and your students, you and the lecturer whose pet project you are evaluating or you and someone else's students. Some students seem quite happy to express their ideas and viewpoints whereas others can appear wary of criticising or pointing out problems. The way in which you relate to the individuals involved in the study can also influence the quality of the data obtained. In addition, the students' perception of the course in which the technology is going to be used or even how they feel about the lecturer involved in teaching this part of the course can colour the way in which they feel about the evaluation session.
Rewarding your students:
Some projects pay their students to take part in their evaluation studies. For many departments, this is likely to cause various administrative if not practical problems. If it is felt that some reward is necessary for your students, you might want to supply tea and coffee before and after the session or to even organise some cheese and wine for afterwards. Alternatively, you could consider approaching publishers to sponsor the purchase of course textbooks to give your students or you could ask the university to provide them with free car parking permits, if appropriate. Generally, you will find that students are willing to participate unrewarded particularly if they feel that there will be some follow up action taken.
Named or anonymous participants?
Various arguments can be put forward as to whether you should ask students to give their names when for example asking them to complete questionnaires. Some students might lack confidence or not feel comfortable voicing their opinions to an evaluator whereas others might only take the responsibility of taking part in an evaluation seriously if they have to put their name to their opinion. If you give your students the option of putting their name on the top of a questionnaire, then generally a large proportion will leave that part blank. This puts obvious limits on a follow-up study of individuals but it is still possible to monitor any class trends.
Sometimes students are more comfortable citing their matriculation number than giving their name. Another way of identifying students is to ask them to use a personal password, unfortunately, this can result in students either forgetting or using a different word in subsequent sessions. Alternatively you can ask students to create their own password for use in your study. This can be done by, for example, using their mothers initials followed by the number of brothers/sisters they have, then their house number etc. This method works as long as the combination used will provide a series of digits exclusive to each student. Students can be reminded about the construction of their personal password each time they are completing a questionnaire for you. This also means that you can follow-up students at different stages of a course and that the students can still maintain a feeling of anonymity.
Implementation Support Consultant,
LTDI, Heriot-Watt University.