LEARNING TECHNOLOGY DISSEMINATION INITIATIVE	Evaluation Studies
Route map: Home Publications Evaluation StudiesQuestion Styles

Evaluation of Computer-Assisted Learning Program Question Styles and Integration into a General Pathology Course. W Alexander Reid & Mark J Arends, University of Edinburgh

Also available as a pdf file.

Abstract

Background: Computer assisted learning programs (CAL) were integrated into an introductory course on general pathology for medical students in the University of Edinburgh Medical School. The study was undertaken to determine how the students reacted to the different question styles available in a package called CALScribe and how well the CAL was integrated into the course.

Method: Towards the end of the course, students were asked to complete written questionnaires.

Results: Students appreciated the different question styles to various degrees, preferring MCQs of either conventional or latent type over free response, modified essay or image-based questions. Students also felt that the CAL was well integrated with other parts of their teaching and helped them to understand more about the subject material.

Conclusion: Students found the CAL an effective learning tool which were well integrated with other course components and they most appreciated MCQ style questions.

Introduction

CALScribe is a set of templates developed at the Computers in Teaching Initiative (CTI) Centre for Medicine, Bristol (Whittlestone & Williams, 1995). These templates operate within the software package Toolbook 4.0 and allow entry of text and various styles of question. CAL written with these templates have been used since 1997 as part of the introductory teaching programme in general pathology for second year undergraduate medical students in the Pathology Department, University of Edinburgh. The CAL programs were integrated into the course so that the material interlocked with that covered by the other teaching methods, including lectures, tutorials for groups of 17-18 students, autopsy demonstrations, problem solving exercises and clinical cases with microscopy. The students were timetabled to work on the CAL in tutorial groups for one hour per week, although the computer cluster was freely available at other times. In the year under study (1998), 215 students participated and the course lasted eight weeks. As part of a larger study of the CAL packages used in the course, it was decided to investigate the use of different question styles available through CALScribe and how the students viewed the degree to which the CAL was integrated into the course. The study entailed using questionnaires at the end of the course.

Method

The CALScribe templates were downloaded from the CTI Centre, Bristol and were used to develop computer-assisted learning programs in pathology in the University of Edinburgh (Boyle, 1997). Thirty eight programs were relevant to the introductory course in the present study. The CAL packages run on the university computing network and are accessed in a cluster of 100 PCs in the Greenfield Suite, Medical School, University of Edinburgh. Each CAL program consists of a series of text and image screens interspersed with various question styles. In most cases the student is allowed two tries at a question and receives an appropriate response from the computer.

Question styles available:

• Free response questions, where the student types in a short answer. A correct first attempt gains two marks and a correct second attempt gains one mark.
• Multiple choice questions (MCQs) of conventional style of stem with five items, each true/false, with negative marking for incorrect responses. The scoring was followed by an explanation.
• "Latent MCQs": stem and five items, but differing from the conventional MCQs in that selection of each item is immediately followed by a score and immediate revelation of a hidden ("latent") explanation. Marks (positive only) are given only for a predefined number of attempts.
• Image-based "point to" questions, where the student is asked to select part of an image with the cursor. The computer responds by showing the correct area and awarding a mark if the area has been correctly identified.
• Modified essay questions: the student enters a more detailed answer, often in response to a conceptually more challenging question. A model answer is then revealed and the student is invited to award him/herself a mark out of 4.

Questionnaires

Towards the end of the course students were asked to complete questionnaires on their opinions of the different question styles and their perceptions of how well the CAL was integrated into the course.

Results

The students were asked how they rated each of the following question styles used in the CAL in terms of helping them learn (124 respondents)(SD =strongly disagree, D=disagree, N=neutral, A=agree, SA=strongly agree)

Table 1	SD	D	N	A	SA
Free response, where you type in the answer	5	14	28	50	26
MCQs, 5 items, true, false with explanation box	1	0	11	43	68
Latent MCQs, with answer appearing after choosing each item	1	5	24	42	49
Modified essay questions, where you award yourself a mark	14	27	42	31	7
Questions where you point to something on a picture	1	7	31	46	35

These results are also shown in Table.1.

Students were invited to write comments on the CAL and reported that the most useful aspects of the CAL were the questions and answers, summaries at the end, pictures, learning at one's own pace and practice in doing questions, especially MCQs: "Self-assessment enabled me to quantify my knowledge and note which areas required improvement". Some students commended the interaction generally: "One cannot go through the programs passively without learning anything". Other positive comments included "integration with problems in tutorial sessions", "going through topics in simple, easy, logical stages helps understanding" and "interesting to use, seemed relevant and concise". Among the least useful aspects of the CAL featured excessive detail, lack of time and uncertainty about the depth of knowledge necessary, while a few found the humour annoying.

The students were also asked how the CAL was integrated into the course: did you feel that the CAL packages:

Table 2	SD	D	N	A	SA
were linked into other parts of the pathology course	1	1	17	70	33
helped you understand more about the subject	1	1	16	65	41
helped you prepare for other parts of the course	2	12	48	44	18
explained some concepts that you had found difficulty with	2	4	24	66	28
made you feel like reading more about the subject	9	27	54	26	8

These results are also shown in Table. 2.

In a further questionnaire on the whole course, 131 students replied to the following question: If the pathology course were to be changed would you prefer to see CAL time (0=not at all, 5=certainly):

	0	1	2	3	4	5
decreased	25%	16%	18%	13%	20%	6%
increased	19%	16%	20%	18%	9%	13%

Discussion

The respondents tended to favour both types of multiple choice question (modal Likert score 5) over free response questions and "point to" questions (modal Likert score 4), but were less well disposed towards modified essay questions (modal Likert score 3, with a balanced spread of opinions). This could reflect the relative importance of MCQs in the end of course examinations, which comprise (marks in brackets): written paper (60%), MCQ paper (20%), practical examination (20%). During informal discussions with the course organiser (MJA), students often voiced concern about the MCQ component of the examination and many requested MCQ practice sessions. The effect of CAL or any other single learning technique on students' final performance in examinations is, of course, extremely difficult to measure, as this depends greatly not only on the nature of the examinations themselves, but also on other factors, such as self-esteem and achievement motivation (Abouserie, 1995).

The students felt that the packages were well integrated into other parts of the course, helped them understand more about the subject, helped them prepare for other parts of the course and explained some difficult concepts, but they appeared neutral about whether they were induced to read more about the subject. This could reflect the relatively intensive nature of this part of the medical course. The students also indicated that they would welcome an increase in CAL teaching time with various degrees of agreement (13% certainly, 19% not at all), but fewer favoured a decrease in CAL time (6% certainly, 25% not at all), with a similarly broad spread of opinion.

It is important for any CAL program to be incorporated fully into the course (Crook, 1994), not just made available as an optional extra. This promotes discussion of the material in relation to the other aspects of their work. In our study, the students felt that the packages were well linked into other parts of the course.

Conclusions

Students had a highly favourable view of CAL, appreciate that CAL are effective learning tools which were well linked to other course components. Although all question styles were appreciated in making the CAL interactive, a preference was expressed for MCQs of either conventional or latent type.

References

Abouserie R., (1995). Self-esteem and achievement motivation as determinants of students' approaches to studying. Studies in Higher Education 20 (1), 19-26.

Boyle T., (1997). Design for Multimedia Learning, 45-63. London; Prentice Hall Europe.

Crook C., (1994).Computers and the Collaborative Experience of Learning, 98; 105-137. London and New York; Routledge.

Reid W. A., Harvey J., Watson G. R., Luqmani R., Harkin P. J. R., Arends M. J. (1998).

Medical student and tutor appraisal of interactive computer-assisted learning programs embedded in a general pathology course. Submitted to Medical Education.

Whittlestone K., Williams J, (1995). CALScribe for Toolbook. University of Bristol: Educational Technology Service.

Acknowledgements

We thank all students who participated in the study and Professor A. H. Wyllie for continual support and encouragement.

To contact the maintainers - mail (ltdi@icbl.hw.ac.uk)
HTML by Phil Barker
© All rights reserved LTDI.
Last modified: 29 September 1998.