The Role of Virtual Learning Environments in the Online Delivery of Staff Development
Report 1: Review of Experiences of Delivering TALiSMAN Online CoursesColin Milligan
Institute for Computer Based Learning, Heriot-Watt University, Riccarton, Edinburgh EH14-4AS
3 COURSE REVIEW
The original online course has now been delivered on 5 different occasions to a total of 284 participants. A slightly greater number of people have signed up for the TALiSMAN Online Study Centre (333 in October 1998). Formal feedback on the TALISMAN Online Course has been collected through electronic feedback forms. For the TALiSMAN Online Study Centre, only informal feedback has been collected - no formal evaluation has yet been carried out by TALiSMAN.
3.1 Online Course Feedback
Feedback from individual courses was used to revise subsequent courses. It was partly responsible for the introduction of an academic subject focus, input from external tutors and a change in use of the discussion forum. Feedback also led us to change the way we advertised the course and to clarify its aims.
Feedback was mainly positive, with most respondents indicating that the course met their expectations, that the level of tutor support was appropriate, and that they would be happy to recommend the course to others. The table below summarises the responses to these questions [Table 2].
Respondents were asked directly about a number of features of the course, the most and least useful aspects of the course, whether they would make any changes to the course, and any other general comments. The responses collected highlighted a number of issues as discussed below.
3.2 Expectations for the Online Course
Some participants expected practical training in the creation of web pages. Whilst we felt that this would have constituted a logical part of the course, it was thought impractical (in such a short course) to attempt to pass on a skill which people learn in very different ways. As other TALiSMAN courses covered web authoring it was felt acceptable to omit it.
Finally, a number of participants expected to be led to teaching materials tailored for their courses. As a generic course, we felt it more appropriate to pass on transferable skills for finding and evaluating learning materials and resources, leaving the academic to use their personal expertise in their own individual field to find specific resources. The subject specific courses delivered in the second year of the project were designed to address this issue. Course participants were directed to CTI centres and Information Gateways, many of which provide libraries of links to materials they have identified as useful and appropriate. Unfortunately, such resources are not available in all subject areas.
Many participants found that it was impossible to set aside 2-3 hours without being interrupted (e.g. by students or phone calls). Others reported that they had worked through the materials in their own time (staying late after work) to avoid interruption.
In planning the course, we were acutely aware that devoting sufficient time to the course by the participants would be a problem. We felt it appropriate to structure the delivery of the course (one lesson a week) rather than just to present the materials as self study. One advantage of this approach is that the regular delivery of chunks of materials can act as a guide through the course content. However, as our courses were not accredited, lecturers were never released from other tasks and there was no carrot which we could offer (or stick we could wield) to encourage or force participants to complete the course on time. Inevitably, a busy lecturer faced with spending 3 hours on the online course, or clearing a backlog of marking will choose the latter and our participants often emailed apologies for non-participation with reasons such as 'I've just been given extra teaching...' or 'It's exam time and I've got lots of marking...' A number of our participants asked whether the course materials would be available after the end of the course, or suggested that the courses themselves could, in future, be run over a longer period.
The main casualty of this inevitable constraint on participation time was the discussion forum. Although we tried to stress the benefits of discussion, we felt it was never used to its full potential, with a generally low level of use in all courses (though discussions in the subject specific courses were slightly more active). Some respondents felt that participants were shy and would have benefited from a face to face session at the start of the course (this was tried for some of the courses but seemed to have little effect on the usage of the discussion forum). A few users remarked that WebBoard was confusing as an interface, but this in itself did not seem to have an effect on the level of use. Realistically, six weeks is too short a time to expect students to become familiar and comfortable with any interface. The choice of discussion forum can significantly affect the way in which online discussion occurs. Although we felt that the Dialogue discussion forum was appropriate for the level of course and number of users, we were concerned that its rudimentary structuring was of little use for reading discussions retrospectively. As we knew that participants might only look in on the discussion area once a week, we felt that such structuring was an important feature. Before the subject specific courses were delivered, we decided to change to a more robust system. We chose to use WebBoard, which provided a simple and familiar interface. WebBoard was easy to integrate with the existing web pages and administration was simple. One criticism of WebBoard was that by being more rigidly structured and clearly presented, discussion may have been stifled, because participants felt that their submissions would come under greater scrutiny.
An alternative possibility might have been to use email as the main discussion medium. Email is familiar to virtually everyone in the community, so no familiarisation would have been required. As most participants check their email every day, they would be able to keep in touch easily with the discussion as it progressed. In practice, this model is unsatisfactory. Most participants tried to set aside time to work through the course in a single session per week, rather than several short sessions. Whilst single email messages might be passed over, and left unread, a single visit to the WebBoard would allow the participant to catch up on many discussions at once. Also, not every participant would be interested in every discussion thread and may come to resent the arrival of emails they considered to be irrelevant. Although we did use email to announce and pace each week of the course, but did not use it to deliver 'content'. If the level of discussion in the forum had been higher, it might have been appropriate to send an email digest of discussions to each participant, once or twice a week.
3.4 Technical Considerations
3.5 Online Study Centre Feedback
As the discussion forum is not used for tutor led discussions, the level of use tends to be low. Participants in the study centre realise that they are engaged in self-paced learning and tend to work independently. Although there are many messages in the system, users are given the option to display only new messages. There tend to be periods of activity as a topic is discussed, then a lull before the next point is raised. The main use of the bulletin board tends to be for technical queries, but it is also used to swap web addresses, allow users to air their own interests and to ask for general advice. As these are answered, they become part of the growing resource of the Study Centre. New users come across answers to commonly asked questions in the bulletin board. This seems to work quite well; if it weren't satisfactory, it would be an easy task to abstract the material to a FAQ (Frequently Asked Questions) page. The Online Study Centre has been running for almost one year now and will soon be internally evaluated as part of TALiSMAN's ongoing analysis of its' courses. The results of this evaluation will be made available when complete.
3.6 External Evaluation
From the questionnaire responses, many positive aspects of the Centre are highlighted. This user worked the whole way through courses, re-used the materials, and contributed to the discussion forums and resource database. This user felt the level of tutor support appropriate, felt that materials were of an appropriate level and was happy to (indeed already had) recommended the Centre to others. The user was critical of some of the course structuring and provided suggestions for more interactivity which was felt to be lacking.
The simple structure adopted for the TALiSMAN online course attracted no criticism. The Online Study Centre environment created within WebCT has been criticised by many users who dislike the navigation structure and the way some of the tools function. With the extra functionality being offered, the structure of the Online Study Centre materials is necessarily more complex and perhaps requires some familiarisation or induction period.
As a further part of the external evaluation, a 'typical' user was commissioned to work through all courses in the Online Study Centre and produce a case report. This user had already participated in one of the staged delivery TALiSMAN online courses (and hinted that devoting sufficient time had been difficult). The commissioned evaluator was also generally positive about the Online Study Centre (would recommend it to others) but outlined a number of criticisms. Navigation round the courses was felt to be poor - with 'Home' buttons on different pages having different destinations - confusing to the first time user. This failing is inherent in the structure of WebCT, but we had felt that as the materials were self-paced, familiarisation with the course structure could be gradual, rather than requiring a formal induction. A similar criticism was levelled against the homepages - although potentially useful, few participants create personal homepages and this is probably because the tools provided for this purpose by WebCT are not particularly good. Making a homepage for users of the study centre was entirely voluntary. Had we wanted to make this compulsory, we would have investigated and provided more appropriate tools. The final negative comment regarded a technical issue. The 'Web Images' course required software which was not available for the platform used by this evaluator, and there was some degree of frustration at the amount of time wasted before this was realised. This highlights the need for some formal technical support. Although we do give some guidance about the recommended specification of machines used for viewing the Online Study Centre and do our best to ensure that the learning materials can be fully utilised by as many participants as possible, there can be unanticipated difficulties. Again, with experience, this could be rectified.
In each of the cases cited above, interactivity (or at least activity) is cited as a key issue. Finding a balance between increasing interactive components and maintaining a focus for the course is vital. It is sobering to note how intolerant our users were of materials not specifically designed for online delivery.