Evaluation Cookbook

cookbook logo
Route map: Home=>Publications=>Evaluation Cookbook=>Exemplars
Example of Trials with Questionnaires and Focus Groups


Where and when was this study carried out?
This was part of a fairly large European project funded to develop both new network based technology and material that used innovative approaches to teaching. Trials were devised that involved students at one UK University to carry out tasks aimed at learning a second language (French). These students were directly recruited to take part in the trials although many were also undertaking optional modules in French. The trials took place over a period of six weeks and were scheduled in advance.

How many staff and students involved?
The trials involved twenty-four students, one evaluator, three members of teaching staff, and various support staff including some from European partners (this is not typical!).

Which evaluation techniques were used?
In addition to planning the trials, the evaluation used questionnaires that gathered the students' attitude to the technology, and confidence logs before and after the trial to provide for self-evaluation of progress. Many sessions were video taped to provide later study and some of the students were interviewed individually and in small focus groups. Students also carried out the same tasks without using computer support. Statistical analysis was used to present the data and search for significant results.

What were the aims and objectives of the study?
The main aim of the trial was to demonstrate a working and viable system. Further objectives were to see if we could measure any improvement in language ability and to determine the students attitude to the different components used in the system. In particular, high-quality desk top video conferencing was a major component of the system and we were interested in the value that the students placed on this.

What were the findings?
The trials supplied information about the relative value of aspects of the system but could not provide an absolute measure of success through comparisons. This was as expected and the value of such trials is perhaps first as project management, second as direct feedback, and last as good data for academic papers!

What were your reflections on this study?
The devising of tasks for the trials had a direct effect on the materials and technology development as anything that was to be trialled definitely had to be supported by the system. The main trials were across distributed sites and the installation process went right up to the start of the trials. This demonstrates the difficulty of completing anything before a real deadline and the power of a trial to provide that deadline. Contingency plans were in place to investigate only some elements of the complete system but in the end these were not necessary. Without these trials it is possible that this integration would never have happened.

Patrick McAndrew . Heriot-Watt university

versions for printing are available

[ Printable version ]
[ Exemplars | Contents ]

© All rights reserved LTDI and content authors.
Last modified: 26 March 1999.