Implementing Learning Technology
Route map: Home Publications Imp. Learning Tech.Choosing courseware
Choosing courseware: Some guidelines to first step evaluation.Jen Harvey
With the growth in the use of Learning Technologies and the availability of more Computer Assisted Learning Packages, an increasing number of people have started looking at new pieces of software with the aim of perhaps using the material in their teaching.
Faced with a new piece of software which someone has recommended to you or hunting for a package which would mean that you won't have to teach that boring part of the syllabus for the Xth time; you probably have some idea as to exactly what your software requirements are. But what aspects of the software are important in the learning process? What features do you look for? Often the 'intuitive approach' is the process adopted and a first look at software reveals whether or not the developers have tackled a subject in the same way as you would or you feel it should have been tackled. But this type of approach does have some limitations: can we predict how students will respond to the material when used within one of our courses or how much the way in which the software is used will influence the quality of the learning outcome.
When considering whether or not to use a Learning Technology (LT) materials, it is recommended that you carry out a full evaluation study prior to the implementation of a new piece of software within a course (see chapters 11 &12 for more details). A full study would normally comprise:
This chapter deals with the first step evaluation: How do you decide whether or not a piece of LT could be used to support your teaching? The process is not dissimilar to that used when selecting appropriate textbooks for a course and in the same way, you might decide to use only some and not all parts from a software package. This section begins by reviewing a number of different ways of beginning to carry out a first step evaluative study of your own.
Evaluating Computer Assisted Learning Material produced by Durham University as part of one of the TLTP projects (IT in teaching and learning: a staff development pack) provides a useful structured way of helping you through the various steps involved in prioritising which aspects of a CAL package/LT materials are the most important. In summary, they recommend that you think about which aspects of the package are important for your particular needs and use these as a basis for a checklist which can act as a guide when you look through the package in a first step evaluative study. The four different aspects which Durham University suggests are:
Each of these four aspects can be subdivided into categories depending on the emphasis which can be placed on them within a given context, for example, text, graphics, colour and fonts would be considered when reviewing the layout.
Consideration will also have to be given to resource availability and necessary hardware specifications for the courseware being considered. These aspects are not covered in this chapter, see chapter 8 for more information on this area.
Learning effectiveness, categorised under pedagogy in the Durham study, is probably one of the main priorities when considering whether or not to use a piece of LT, for example: Does the material provide an effective way of teaching this part of the course? Does the package encourage the attainment of your course aims and objectives?
John Milne (1995) from the CTI CLUES has used the model first described by Phil Race (1993) as a basis for carrying out evaluative studies. This approach considers the effectiveness of the learning environment and whether or not how each of the stages of the learning process are encouraged within LT materials. Phil Race's model describes the process of learning as requiring an individual's desire to want, do, feedback and digest.
The learning effectiveness of a piece of software can then be measured by how well these features are supported within the CAL package/LT materials. However, whatever model is adopted it is the level at which the user becomes actively involved with subject material which will determine how much learning is undertaken. The more active the approach encouraged, the more likely it is that learning will take place.
Just picking out different aspects of LT materials, unfortunately, can encourage an approach whereby features are considered in isolation and not holistically within the context of a particular course. The way in which the software is integrated into a course is one of the most important factors determining learning effectiveness. An integrative rather than a 'bolt on' approach also encourages the students to feel that the LT material is relevant to the course, especially if they are to be assessed on the material. Providing additional support or embedding the LT materials within a series of classes are also useful methods of integration. For example, if adequate support is not provided from within the package you might decide to have pre or post classes in order to introduce the subject or to draw it all together retrospectively. Alternatively, you might use the material in a tutorial, set a group task and provide students with a handout which provides them with additional assistance.
I have produced a draft checklist (the "First Step Evaluation Checklist" in the appendix) which takes into account a range of aspects which you might like to consider when carrying out a first step evaluation. These include the learning effectiveness, usability, presentation and content of a piece of software. The checklist aims to give some pointers as to what you might consider as you work through a piece of software. It takes into account both of the evaluation models discussed earlier. What you select to use will be dependant on your own perceived requirements and in some cases you might feel that these categories are not applicable.
The following material is included to provide a background to each of the sections included in the questionnaire and to suggest some ways in which you might integrate the software into your course.
These are in the form of leading questions to bear in mind as you work through the package and to how the material might be integrated into specific courses: What background knowledge or additional support would be required by your students if they were to use the package? How do the learning strategies encouraged in the software fit into the rest of the course in which you plan to use the material i.e. if a problem based approach is used throughout the rest of a module then asking students to go through a sequence of factually based modules might be perceived as being inappropriate by students.
At first appearance, a new piece of software might explain a process through the use of an excellent simulation, but does not provide an appropriate context for the material. Used as part of a lecture where you provide relevant additional material, this could be a more effective way of explaining the subject than using a static graphic. If only parts of different packages are suitable why not set up a resource folder of material on a local fileserver which students can access for reference for coursework or to aid in their revision for assessments.
The introduction to a piece of software/LT material is important in that this can be the first time that a student might have been introduced to a subject. Providing an initial framework, acts as an advance organiser and means that any new material can be added to this cognitive framework in a structured way. In turn, hopefully this deeper or more effective processing of the information should allow a more efficient retrieval at a later date. If material remains unlinked to the student's existing background knowledge and the student is not encouraged to take an active involvement when using the LT materials this is likely to result in a surface type of processing where material remains unassociated and unrelated and ultimately less memorable.
Clear aims and objectives set out within the introductory section are helpful to the students as they clarify exactly what is expected of both them and the package. These are also helpful to you for ascertaining the level of the pre-requisite background knowledge as well as allowing you to rectify any shortfalls by providing any necessary additional supportive material.
You can always provide your students with your own module aims and objectives in the form of a handout, this is particularly useful if you want students to refer to only one section. Providing leading questions can also assist in the process of learning by encouraging students to adopt an active approach. These can then be used as the basis for a feedback session in a tutorial with a group of students.
In many packages, the introductory screens include a map or a menu of the subject areas to be covered. The way in which this is presented on screen can pre-determine the way in which the subject is perceived. For example, presenting material in unrelated sections in a linear type format could result in material being perceived as unrelated 'chunks' of information. Providing a fixed path through the material for students to follow limits a user's feeling of control and the level to which they might be able to use the package in a way which suits their individual learning style.
In addition, when you are looking at a piece of software for the first time, it is important to stop and review the LT materials at different stages and make sure that there are no gaps in the subject content or that relevant or important information is omitted. Sometimes if you are close to a subject area it is easy to skim through a package and not notice gaps in the subject content or whether there was a lack of appropriate student support.
Running pre and post LT sessions with students can assist in the quality of the learning experience by setting the context and encouraging the integration of the subject areas. For example, laboratory simulations might become more meaningful if preceded and followed up by practical exercises in the laboratory.
Some students can feel threatened by having to use computers if they are using them for the first time or do not have much computer experience. Providing information about any menu bars and explanations as to the function of icons is important prior to letting your students loose on the package, particularly, if the icons are less than self explanatory. Often packages will provide an introductory section about what icons mean but this is not always the case. Even if your students are reasonably computer literate and could make an educated guess as to how they might proceed, if the on-line LT support is negligible, then perhaps you might like to consider providing instruction about how to use the package either in a class or in the form of a handout. In addition, you could make sure that someone is at hand to help students who might have difficulty. You might even consider asking students from another year to provide assistance. However, it is always important to identify any perspective problem areas which your students might encounter and to draw their attention to these in advance of their using the LT materials. For example, the time taken to load a particular part of the package or if insufficient explanations or instructions are provided to some of the sections.
The usability of a package is determined in part, by the ease at which you can move about the software. The more flexibility the more the package can cater for an individual's learning style. On the negative side, the more a student might feel lost and not know how to proceed. Packages which enable students to go back and forwards, readily allow students to go over and to refer back to information and ultimately digest the subject matter. So it is useful to stop at different points of a package and to check how easy it is to navigate between pages and sections. Is it possible to leave the package and to go back to exactly that point at a later stage or do you have to wade through the material from the beginning again?
LT material with little flexibility built in, might require a bit more imagination as to its usage, however, it is possible to encourage interaction, by for example, introducing a group task which can stimulate interaction through discussion between students.
Simple electronic page turning packages with lots of factual material and references can result in students looking for hard copy print outs to take away for reference at a later stage. The provision of an on-line notepad within a package can be used to encourage an active engagement with the subject matter and to personalise the material, but sometimes you might have to encourage your students to make use of such a facility. Another way of encouraging students to take notes or to adopt a more active approach to a LT materials is to ask students to review or to report back on a package. This can also provide you with a useful method of obtaining feedback on the packages usability and also encourages students to feel more involved in their learning. However, more packages are now providing ways in which students might store their notes electronically, so this is worth checking out.
As mentioned earlier, the factual content and how material is presented are aspects of a package which perhaps would be considered almost intuitively by most people upon first inspection of a piece of software. However, it is important to pay attention to the legibility and how the material is presented on screen as this can also be influential in the way the student later prioritises the information. How are the key words identified and the main points prioritised? How do you know what is important?
Despite personal preferences for particular colour schemes, if the text which appears on screen is difficult to read or inappropriate then students are likely to lose interest and give up. In addition, consideration of the specifications of the machines on which your students are going to view the material is also worth thinking about at this stage. Although a piece of software might look excellent in a demonstration using a high resolution screen and a powerful machine, loading and using images and video clips might be less than meaningful on a slower machine.
The availability and the usefulness of student support and on line help facilities can, in many cases, determine when and how the package can be used. An excellent on line help facility can mean that students can be left to go through packages at a time which suits them. When reviewing a package for the first time, it is a useful exercise to stop at different stages and review what help facilities are available from different screens in the same way as a student might need to. How useful is this help? The most likely event if a student becomes lost and the package does not provide them with adequate help is that they will just give up working through the material, rather than trying to seek you out for assistance. It is also worth remembering to draw your student's attention to a good Help facility if it is provided from within the LT materials.
End of section summaries or quizzes can assist the learning process by encouraging students to review the material already covered and this again leads to a more effective storage of material. Cross referencing to other sections or to other pieces of software helps students to integrate and develop a deeper understanding of the information. When reviewing the package, think about how each section finishes and the next one begins. Is there any cross-linking provided?
An e-mail discussion group or setting up a database of frequently asked questions can provide additional support for your students. Summary sheets of the important points covered or running surgeries for students who perhaps are experiencing difficulties can be used to provide additional support.
Assessments can be used in a variety of ways which can help students to think more about a subject and often packages include short quizzes of multiple choice questions to test a students knowledge. If assessments do form part of the LT material's learning process then the level and the type of feedback is important in determining their usefulness. Providing constructive feedback is going to be more encouraging if a student gives the wrong answer than, for example, a monosyllabic 'Wrong'. Go through all the questions and check what happens when you obtain the right and the wrong answers. What happens if you repeatedly give the wrong answer? Is this constructive help?
Adding a multiple choice type question file, to be used in association with the LT materials and monitoring student responses can be a useful way of determining any grey areas (see chapter 10 on assessment). If the students' marks do not count towards their coursework, including such an assessment can also help students to rehearse for a summative assessment and this will also increase the student's feeling that the LT materials is relevant to their course . Giving students questions which require them to link ideas included in the LT materials can also be used to encourage them to integrate ideas and link information together. Some packages also provide the facility to track student's progress through the material and enable you to see which parts of a package the students have accessed and how much time was spent there.
When you have completed looking through the package, again it is a good time to consider how the material might fit into your course objectives and whether or not the subject benefited from the use of the technology. Could you have taught the subject better by either using another method of teaching and/or you had used the LT materials in a different way?
Most people are unlikely to find LT materials which exactly fits what they need or teaches the subject in exactly the same way as they would, but integration of the material into a course can mean that the software material becomes a useful method of teaching a particular part of a course, in the same way as a 'hands on' laboratory session might be used to teach a practical set of skills in another part of a course.
Some packages are written in a way which enables you to modify them, in part, so that they can be tailored more to your individual course needs. It is sometimes possible to amend examples cited in the package, assessments and/or references. This is worth considering in order to increase the relevance of the material for your students. If the possibilities for such modifications are not evident, sometimes the software developers are amenable to making the changes for you.
A number of papers have been written relating to the process of evaluation and the TILT and EMASHE groups of University of Glasgow have written a lot of material about the summative evaluation process. ELTHE (Evaluation of Learning Technology in Higher Education) is a useful self help group, for further information contact Philip Crompton (see Appendix 3: Contributors).
To contact the maintainers - mail (firstname.lastname@example.org)
HTML by Phil Barker
© All rights reserved.
Last modified: 25 July 2002 (fixed broken link), 30 December 1999. (formatting)
First web version: 03 October 1997.
First Published: July 1996.