1. Review of TALiSMAN Online

0 OVERVIEW
Computer networks provide the flexibility necessary for successful delivery of education and training in a learning society. Exploiting these networks with new communications technologies can enhance the quality of network based learning, if properly conceived and implemented. Learners can study in their own time, at their own pace and choose their own course, whilst feeling part of a virtual learning community rather than isolated learners.

For two years, the SHEFC funded TALiSMAN project has provided a programme of training and awareness to enable staff at Scottish Higher Education Institutions to benefit from the Metropolitan Area Networks which link them. A significant part of this programme (>600 training places) has been delivered online.

Here we describe two alternative approaches to delivering online staff development which we have followed. The report will discuss our experiences of these two approaches and highlight issues inherent in the design and delivery of staff development programmes online. We will recommend ways of maximising the learning experience whilst retaining a flexible and scalable delivery structure.

A second report of this project will extend these findings and discuss the suitability and potential role of commercial and freely available Virtual Learning Environments in the delivery of online staff development programmes. [TOP]

1.1 Introduction
The use of computers to support teaching in Higher Education has increased dramatically in recent years. Network infrastructure improvements and technology developments, (in particular the World Wide Web (WWW)) are providing exciting opportunities for the use of computers in all disciplines. These developments have coincided with an evolving role for education as more students wish to study part-time, at a distance, or wish to integrate their education with their professional career. In addition, the creation of a learning society, where individuals are encouraged to continue to learn throughout their life demands new models of teaching, which do not rely heavily on face to face teaching or strict progression schedules. Universities have recognised this changing need for educational provision and are investigating more flexible teaching methods to enter new markets such as distance learning, work-based learning and part-time study. Within Higher Education, there is a movement towards greater formalisation of staff skills and it is anticipated that new delivery mechanisms will be exploited to achieve this. The combination of new technologies and network delivery is ideally suited to provide flexible provision and also establish distributed learner communities to support the learning process. In the context of staff development, such a combination provides:

1.2 The Scottish MANs and UMI Projects
As an example of the improvement in IT infrastructure and its impact on staff development, consider the developments in Scottish HE. In 1995 and 1996 the Scottish Higher Education Funding Council (SHEFC) funded the creation and installation of a high speed, reliable network of Metropolitan Area Networks (MANs) linking the Higher Education Institutions within the sector. This network (the Scottish Metropolitan Area Network) provides a capacity of 155MBps (Mega Bits per second) split between Internet traffic (WWW, email, desktop videoconferencing etc.) and MAN (or studio quality) videoconferencing with extra capacity reserved for special uses such as parallel computing and music teaching.

To complement this investment in infrastructure, SHEFC funded a number of projects under the Use of MANs Initiative (UMI, [1]). These projects were intended to encourage the use of the newly created MANs. The projects funded under UMI phase one fell into three strands:

Whilst most of the training was to be delivered face to face it was anticipated that some of this training would be delivered online to:

1.3 Other Relevant Studies
Since the TALiSMAN training needs analysis was completed a number of other studies have identified similar needs within the UK HE sector. A few are discussed below.

In early 1998, SHEFC began a comprehensive review of their C and IT provision by carrying out a consultation exercise with representatives from the 21 Scottish HEI's and other relevant bodies. The results of this consultation were reported in a circular letter in April 1998 [4]. On staff development, the authors were clear about the potential role of C and IT: "The most widely recognised need, for the further development of the use of C and IT in higher education and especially in Teaching, Learning and Assessment (TL and A), is an extended range of staff development activities. These range from the provision of training in basic IT skills for all staff to the rethinking of the processes of TL and A in the light of the enhanced facilities that C and IT can provide. Many responses emphasised the need to embed the C and IT development in the wider issue of the reformulation of the processes of teaching and learning, moving from conventional delivery mechanisms to more independent, resource-based learning, facilitated and supported by academic staff. The programme should be pedagogy-led rather than being technology-led." The report goes on to highlight the role of new technologies in facilitating this staff development: "The important contributions of CTI, LTDI and TALiSMAN to increasing the awareness and abilities of academic staff in the Scottish HE sector were recognised. In the next stage of development, the need was expressed to exploit C and IT itself (through means such as videoconferencing, and the production and use of IT-based staff development materials) to bring the development activities closer to the individuals who need them, and to allow a wide participation from experts across the sector."

The report of the National Committee of Inquiry into Higher Education (the Dearing Report [5]) was concerned with all aspects of Higher Education, but some of its findings relate to the use of C and IT in Teaching and Learning. Discussing students and learning, the report recognises the central role of C and IT along with the attendant need for appropriate staff development (Recommendation 9). Furthermore it is thought important to formalise such training within a proposed "Institute for Learning and Teaching in Higher Education" (Recommendation 14). On staff development, the report stresses that whilst C& IT has much to offer, it is important to develop its potential carefully - with appropriate training, recognising the role already played by TALiSMAN: "Training and support in the use of C and IT is an issue in its own right. In such training we imagine that institutions will wish to draw on materials already developed in the sector such as the Netskills project at the University of Newcastle and the TALiSMAN activities being carried out by Heriot-Watt University on behalf of the Scottish Institutions" (para. 14.19).

The recent "Evaluation of the Computers in Teaching Initiative and Teaching and Learning Technology Support Network" report [6] scrutinises the current role of these initiatives in promoting the use of C and IT within the UK Higher Education community. This report recognised the importance of C and IT in enabling the development of new learning strategies, and satisfying the changing expectations of students and employers (para. 5).

Each of these studies recognises the future importance of C & IT to staff development and specifically the role of networks in making staff development flexible. [TOP]

1.4 TALiSMAN Online Activities
TALiSMAN has now investigated and used a number of approaches to deliver courses online - as staged delivery courses and as open learning. The aim of this study is to report our experiences and to draw general conclusions as to the future potential for networks in the delivery of staff development. TALiSMAN first used an approach based on simple web pages, supported with simple discussion forum software. Subsequently, we used the Virtual Learning Environment software WebCT to manage and administer whole courses within a single environment. The next section of this report will examine and contrast these two distinct approaches. Subsequent sections will then consider course feedback, experiences and lessons learned, before re-examining these conclusions in the light of findings and recommendations from other initiatives.

A second report from this project will examine a range of other Virtual Learning Environments (VLE's) and evaluate their potential and suitability for the delivery of staff development online within the UK HE community, with special attention paid to the issues identified here. [TOP]

2 Our Approaches
As previously described, TALiSMAN is a staff development project serving Scottish Higher Education Institutions. All participants on our courses are academic, research, administrative or support staff at these institutions. TALiSMAN has investigated two very different modes of online delivery of staff development. This section briefly describes the two approaches we followed: the TALiSMAN Online Course and the Online Study Centre (OSC).[TOP]

2.1 The TALiSMAN Online Course
In January 1997, TALiSMAN began planning for the delivery of the first online course, with the title 'Using the WWW in Teaching and Learning'. The course content was basic and aimed at staff with no previous experience of using the WWW for teaching, but who did have access to the WWW and had some familiarity with the Internet. (a course flyer is available at http://www.talisman.hw.ac.uk/online/www.html and included as Appendix A). In addition to the relatively modest aims of the course, it was anticipated that the course would provide a valuable opportunity for future practitioners to experience the benefits and frustrations of online learning.

In designing the online course materials we examined two existing courses: 'Using the Internet for Teaching and Learning', developed by Jackie Galbraith at MEDC (University of Paisley) for Scottish FE, and the short 'Finding Learning Opportunities on the Web' (FLOW) course run by the Open University [7]. We followed the structure of the MEDC course (6 weeks, staged delivery) and drew on the overall aims of the FLOW course, however we used our own content (revised from a face to face course) and delivery mechanism.

The initial implementation of the course (http://www.talisman.hw.ac.uk/online/100397/ - see Appendix B for an outline of the course) consisted of six 'lessons' made available at weekly intervals (usually on a Tuesday morning). Each lesson was announced by a brief email which summarised the week's activities. Usually, a reminder was sent out on the Friday urging those who had not yet participated to complete the coursework. Completion of the course required a commitment of 2-3hrs per week self-study, with activities mainly centred on the discussion forum. The courses were not accredited.

After requests from some participants, an extra break was added halfway through the course to provide a break for Easter. The course was advertised at TALiSMAN events, by mailshot to our Institutional contacts and from the TALiSMAN WWW site. 52 participants signed up and all registration and administration was carried out through email. Prior to commencement of the course, participants were given access to the course environment to enable them to familiarise themselves with its structure and interface. The course material was predominantly simple web pages, including links to example sites, and a course discussion area utilising the discussion software Dialogue [8]. After the course, feedback was collected by means of an electronic feedback form. This form is included as Appendix C.

A further 62 participants signed up for the second run of the course which started in May 1997 (http://www.talisman.hw.ac.uk/online/120597/). For this run of the course, very little content was altered, however significant changes were made to the environment and administration of the course. As before, we felt unable to 'force' participants to actively take part in the course, but we did ask them to give a commitment to the course by completing a 'declaration of intent' before starting. We also elicited participant information through a course 'expectation' questionnaire as well as detailed feedback questionnaires (all collected electronically) filled in during and after the course.

During the summer, a more significant revision of the course was undertaken. Changes were introduced to improve the structure, reflecting feedback from the initial two courses. The most significant change involved introducing an academic subject focus to the course material. [TOP]

2.2 Academic Subject Focus Courses
A recognised problem with the initial course had been the lack of "community" and it was hoped that by introducing an academic subject focus, we could bring together people with a similar background and aspirations. We also arranged a half-day face-to-face induction, but attendance at this was not compulsory. A table summarising the dates of these academic subject focus courses and participant numbers is shown below [Table 1].

Date	Course	Participants
November 1997	Statistics, http://www.talisman.hw.ac.uk/online/031197/ Medicine and Biosciences, http://www.talisman.hw.ac.uk/online/031197/ Engineering, http://www.talisman.hw.ac.uk/online/031197/ Social Science, http://www.talisman.hw.ac.uk/online/031197/	21 32 16 31
March 1998	Land Use and the Environment, http://www.talisman.hw.ac.uk/online/020398/ Art and Design, http://www.talisman.hw.ac.uk/online/020398/	22 27
April 1998	Business Education, http://www.talisman.hw.ac.uk/online/200498/	21

Table 1: Academic Subject-based Courses

These subject based courses each attracted approximately 20 to 25 participants. The Statistics and Art and Design courses, were advertised throughout UK HE in order to enrol enough participants from these rather small communities.

For the subject courses, help was sought from subject experts to deliver part of the course. Week 2 (Finding Resources) was replaced by a lesson delivered by an external tutor with a specific subject focus. The external tutors were from Information Gateway Projects (EEVL, SOSIG, OMNI, and Biz/Ed [9]) or Computers in Teaching Initiative (CTI) Centres (Statistics, Land Use and Environmental Sciences and Art and Design [10]) and they drew largely on material from their own site to provide alternative content for that week. In some cases we also provided alternative material for week 4 (Developing Web-based Resources). A significant advantage of using external tutors is that it gave courses 'authority' and a more diverse input.

The structure of the course environment was also changed from being essentially one web page per week to being a series of short pages; mostly viewable as single screens and with a simple navigation structure maintained across the whole course. This change reduced the amount of text in any one page and emphasised the common structure running through the course. An example screen is shown below [Figure 1]. This shows learning material and navigation links for week 5 of one of the academic subject-focus courses.

As different courses were being run in parallel, we decided to move to a more robust discussion system and chose WebBoard [11]. Whilst being a more sophisticated package than Dialogue, WebBoard was found to be somewhat daunting by those who had never used computer based conferencing systems before. An example screen from WebBoard is shown below [Figure 2]. For the standard view of WebBoard, the message thread is displayed on the left and individual messages on the right, buttons for submissions to the discussion are placed along the top of the browser window, below the standard toolbar and address panel.

Some constraints governed the format of the course and our expectations for it. As the participants were novices, it was important that the course would be accessible with even the most basic web browser and email client running on an older computer. We therefore used a very simple design with no use of HTML Frames, JavaScript, audio or video etc. As the course was not accredited, we had no way of forcing the participants to complete the course within a specified time and could only rely on them completing the course at the pace and in the manner that we recommended. In addition, we thought it would prove difficult (given the short timespan of the course) to try to build up any sense of community amongst the participants.

The course environment which we created, whilst simple, was robust and flexible enough to act as an effective delivery vehicle for the TALiSMAN Online Course. If the course was to be delivered on a larger scale, then it would help to have administrative tools that could be used to manage the course efficiently. Furthermore, if the course was accredited, then delivery and administration of assessment would require new tools. [TOP]

2.3 The Online Study Centre
Throughout the duration of the project, TALiSMAN has developed and delivered a variety of face-to-face training courses around the central theme of using networks for teaching. Courses delivered included 'Simple Web Authoring', 'Advanced Web Authoring' and 'Simple JavaScript'. With a potential audience in excess of 16,000 (academic, support, administrative and research staff at the 21 Scottish HEIs) and only 3 trainers, it was obvious that TALiSMAN could never satisfy the potential demand for our courses. Furthermore as new technologies appeared, we wished to spend time developing and delivering new courses whilst still providing access to the original courses.

The TALiSMAN Online Study Centre was created as a way of keeping our original courses accessible. We envisioned a single interface allowing access to adapted versions of our face-to-face training courses. Given its intended use, the requirements of the new environment were different to those for the original online course. This environment should provide tools for communication, student tracking and course administration. It must allow the management of a large number of students, and be flexible enough to incorporate different styles of course material. Finally, creation of online materials must be simple and quick. Rather than follow the 'home made' approach of our initial online course, we decided to use a commercial Virtual Learning Environment (VLE) package for the management and delivery of the courses within the online study centre.

One of several packages we considered was WebCT (Web Course Tools) developed at the University of British Columbia [12]. We chose WebCT because of the flexibility and customisability it offered. Learning materials can be created easily by experienced web developers and novices. Our second report will discuss WebCT and other VLEs in greater detail.

For the TALiSMAN Online Study Centre (OSC), we required a single environment, offering access to a range of courses, with shared communication facilities, a calendar, web based assessments and other features. Whereas some packages allowed no customisation, WebCT allowed us to include or discard individual tools and features as required. For instance we felt it inappropriate to force participants to use the WebCT email system for one-to-one communication, but felt the internal discussion forum was useful for one-to-many interactions. WebCT allowed us to separate these two modes of communication, unlike other systems, where one-to-one and one-to-many conferencing modes are integrated and cannot be offered separately.

The homepage of the Online Study Centre is shown below [Figure 3]. Generic materials reside in the left section whilst the right hand section holds the courses available, along with tools for the learner: student records, password administration, the course discussion area, an event calendar, student homepages and a keyword search.

As with the online course, the courses offered within the Study Centre were not accredited. Furthermore, no formal tutoring was provided (although there is tutor support). Users register to access a course or courses and are given a username and password. This initially provided users with access to individual courses and to the relevant course discussion forum. This was changed slightly so that registration provided access to all courses. Participants have to motivate themselves to complete the course - there is no tutor initiating discussions on specific topics at specific times. In this way, the materials are entirely self-paced and resemble open learning.

The Online Study Centre was expected to perform a dual role - that of providing access to our materials, and also to serve as a test bed for us to investigate how different course structures and different content could be fitted into a standard template. Of particular concern to us was whether the constraints imposed by WebCT on learning materials in any way compromised their quality or usability.

The OSC initially hosted two halves of a course 'New Opportunities in Teaching and Learning' which we had delivered on site at 19 of the 21 Institutions we serve. These courses ('Using Communications in Teaching and Learning' and 'Using the WWW in Teaching and Learning') represented approximately three hours face to face teaching each. The materials for the Communications course were already relatively interactive whilst those for the WWW course were more text based, having originally been designed as a stand alone booklet, to complement the face to face taught course. In fact the 'Using the WWW in Teaching and Learning' materials had already been utilised to create the original TALiSMAN Online Course. In addition to these two courses, the TALiSMAN Online Study Centre also contained a short 'Introduction to email' course which had been developed but never delivered and a web authoring skills course which represented approximately 2 days face-to-face teaching time. An example screen showing learning material is shown in Figure 4. The screen is divided into three frames, along the top are buttons giving access to general course tools such as the glossary and search facilities, along with navigation buttons. The main left hand frame provides an outline for the course and highlights the current page. The main right hand pane contains the learning material itself.

Over the eleven months since its launch, further courses have been added. 'Simple JavaScript' and 'Sound and Vision' (adapted from day long face to face courses) and 'Web Images', written especially for the Online Study Centre. A further course, 'Online Video Conferencing' (originally delivered with tutor support via WebCT) is in the process of conversion to self study material.

In many ways, the TALiSMAN Online Study Centre was developed as a complementary approach to that used by the Online Course. The course materials were made available as open learning, in keeping with the initial concept of the Centre. Rather than groups of participants being given the learning material as a cohort, access to the Centre and its courses was provided - 'on demand' and the pace of progression through the course was entirely under the control of the participant. Tutor support was given as needed, but there was no formal course structure, or delivery schedule. The discussion area was used not to discuss specific topics (at specific times), but to provide a forum for users of the Centre to raise their own issues. The growing archive of discussions provides an additional resource for participants; here they will find advice on using the Centre, interesting issues, further materials and so on. We expected users to progress through different courses in the Centre, gaining a range of skills through the different materials.

The screenshot below shows the course discussion area [Figure 5]. The left hand frame allows the user to decide how messages are displayed. The right hand frames show message titles (top panel) and individual messages (bottom panel).

2.4 Structure of Materials for WebCT Delivery
Our experience suggests that some materials were better suited to WebCT than others. Initially, materials were put into WebCT 'raw' and subsequently reworked. Some materials, such as the Simple HTML Authoring course worked well in this format. Course materials with a more rigid and uniform structure, such as the support texts for the 'Using the WWW in Teaching and Learning' Introductory Course worked less well. This is unsurprising, as predominantly text-based materials are known to work less well on screen. Most recently, we have created materials directly for use within WebCT. This has allowed us to create a style that meets both the constraints of WebCT and the needs of the materials themselves and these courses have proved popular with users of the Online Study Centre. [TOP]

The original online course has now been delivered on 5 different occasions to a total of 284 participants. A slightly greater number of people have signed up for the TALiSMAN Online Study Centre (333 in October 1998). Formal feedback on the TALISMAN Online Course has been collected through electronic feedback forms. For the TALiSMAN Online Study Centre, only informal feedback has been collected - no formal evaluation has yet been carried out by TALiSMAN. [TOP]

3.1 Online Course Feedback
Formal feedback for the online course attempted to elicit information about the delivery and content of the course as well as providing user profiles, views, previous experience and an indication of participant expectations. We used essentially the same feedback forms for the initial course and the subject-focus courses, but used rather different forms to evaluate the second course. This was necessary as we wished to collect feedback throughout the course rather than at the end. Completion of evaluation forms was optional and we achieved a low rate of return; never rising above one third of participants. As a result, it would be unwise to place too much emphasis on the feedback collected. Perhaps those who filled in the feedback form were those who were most happy with the course? Nevertheless we can draw some specific views and general pointers from the responses and comments collected.

Feedback from individual courses was used to revise subsequent courses. It was partly responsible for the introduction of an academic subject focus, input from external tutors and a change in use of the discussion forum. Feedback also led us to change the way we advertised the course and to clarify its aims.

Feedback was mainly positive, with most respondents indicating that the course met their expectations, that the level of tutor support was appropriate, and that they would be happy to recommend the course to others. The table below summarises the responses to these questions [Table 2].

	Yes	No	Don't Know
Did the course meet your expectations?	29	10	8
Would you recommend the course to others?	37	2	7
Did you find the level of tutor support adequate?	29	3	12
Did you find the course useful in relation to your work?	39	6	1

Table 2: Quantitative Feedback

Respondents were asked directly about a number of features of the course, the most and least useful aspects of the course, whether they would make any changes to the course, and any other general comments. The responses collected highlighted a number of issues as discussed below. [TOP]

3.2 Expectations for the Online Course
Although the online course was advertised as introductory level, concerned with exploring the potential of the WWW as an aid to teaching, a number of participants reported quite different expectations. Many participants (especially in the initial delivery) enrolled not to learn, but to experience participation on an online course. Although this was a valid reason for enrolling, these participants were largely familiar with the content of the course already and so their presence changed the behaviour of the group as a whole (they were more active in discussion groups and perhaps discussed different issues).

Some participants expected practical training in the creation of web pages. Whilst we felt that this would have constituted a logical part of the course, it was thought impractical (in such a short course) to attempt to pass on a skill which people learn in very different ways. As other TALiSMAN courses covered web authoring it was felt acceptable to omit it.

Finally, a number of participants expected to be led to teaching materials tailored for their courses. As a generic course, we felt it more appropriate to pass on transferable skills for finding and evaluating learning materials and resources, leaving the academic to use their personal expertise in their own individual field to find specific resources. The subject specific courses delivered in the second year of the project were designed to address this issue. Course participants were directed to CTI centres and Information Gateways, many of which provide libraries of links to materials they have identified as useful and appropriate. Unfortunately, such resources are not available in all subject areas. [TOP]

3.3 Participation
By far the largest subject for feedback concerned participation. Many participants felt that 2-3 hours a week was not enough to fully complete the course materials. The course materials were largely self contained, but each week, a number of examples referred to external sites. This format encourages the learner to explore the web sites and it is likely that those who spent more than 3 hours each week were those who were most easily side-tracked and explored these web sites more fully than was required. When designing web based materials that refer to external web sites, it is important to structure the materials to try to counter this, especially if the materials are objective led. Some web page designers will go as far as having no external links in their site, whilst others will keep external links to a separate section at the end of the material. A possible solution might be to link to external web sites, but to open all external links in a separate browser window; thus preventing participants from becoming completely detached from the initial material.

Many participants found that it was impossible to set aside 2-3 hours without being interrupted (e.g. by students or phone calls). Others reported that they had worked through the materials in their own time (staying late after work) to avoid interruption.

In planning the course, we were acutely aware that devoting sufficient time to the course by the participants would be a problem. We felt it appropriate to structure the delivery of the course (one lesson a week) rather than just to present the materials as self study. One advantage of this approach is that the regular delivery of chunks of materials can act as a guide through the course content. However, as our courses were not accredited, lecturers were never released from other tasks and there was no carrot which we could offer (or stick we could wield) to encourage or force participants to complete the course on time. Inevitably, a busy lecturer faced with spending 3 hours on the online course, or clearing a backlog of marking will choose the latter and our participants often emailed apologies for non-participation with reasons such as 'I've just been given extra teaching...' or 'It's exam time and I've got lots of marking...' A number of our participants asked whether the course materials would be available after the end of the course, or suggested that the courses themselves could, in future, be run over a longer period.

The main casualty of this inevitable constraint on participation time was the discussion forum. Although we tried to stress the benefits of discussion, we felt it was never used to its full potential, with a generally low level of use in all courses (though discussions in the subject specific courses were slightly more active). Some respondents felt that participants were shy and would have benefited from a face to face session at the start of the course (this was tried for some of the courses but seemed to have little effect on the usage of the discussion forum). A few users remarked that WebBoard was confusing as an interface, but this in itself did not seem to have an effect on the level of use. Realistically, six weeks is too short a time to expect students to become familiar and comfortable with any interface. The choice of discussion forum can significantly affect the way in which online discussion occurs. Although we felt that the Dialogue discussion forum was appropriate for the level of course and number of users, we were concerned that its rudimentary structuring was of little use for reading discussions retrospectively. As we knew that participants might only look in on the discussion area once a week, we felt that such structuring was an important feature. Before the subject specific courses were delivered, we decided to change to a more robust system. We chose to use WebBoard, which provided a simple and familiar interface. WebBoard was easy to integrate with the existing web pages and administration was simple. One criticism of WebBoard was that by being more rigidly structured and clearly presented, discussion may have been stifled, because participants felt that their submissions would come under greater scrutiny.

An alternative possibility might have been to use email as the main discussion medium. Email is familiar to virtually everyone in the community, so no familiarisation would have been required. As most participants check their email every day, they would be able to keep in touch easily with the discussion as it progressed. In practice, this model is unsatisfactory. Most participants tried to set aside time to work through the course in a single session per week, rather than several short sessions. Whilst single email messages might be passed over, and left unread, a single visit to the WebBoard would allow the participant to catch up on many discussions at once. Also, not every participant would be interested in every discussion thread and may come to resent the arrival of emails they considered to be irrelevant. Although we did use email to announce and pace each week of the course, but did not use it to deliver 'content'. If the level of discussion in the forum had been higher, it might have been appropriate to send an email digest of discussions to each participant, once or twice a week. [TOP]

3.4 Technical Considerations
The course was designed to be accessible by anyone with a graphical web browser. Although we felt we were careful to avoid 'technology for the sake of it' feedback indicated that our participants were most intolerant of any activity which they felt was a waste of their time. For instance, one part of the material discussed the use of plugins and how they can benefit teaching and linked to a site which used the Chime plug-in. Although it was made clear that installing and using the plugin was not necessary, several participants did so, then complained that they had spent a long time on an activity that had very little merit. Although we did not anticipate many problems with access, we did ask for information as to the platform and browser version each participant was using. As a consequence we were able to anticipate which users might have difficulty with some of the materials and act accordingly. [TOP]

3.5 Online Study Centre Feedback
Feedback from the Online Study Centre has been collected informally through discussions with users as well as through the discussion forum. This feedback is concerned more with the usability of the system than with content or whether it succeeded in its aims of providing an effective and supporting environment for the delivery of self-paced learning materials. There are incidental comments on the learning materials, but no real appraisal. Comments have been fed back to improve the usability of course materials within the Centre. Over the first eleven months of the Online Study Centre, this feedback has resulted in many improvements to course structure, making the materials more varied and active.

As the discussion forum is not used for tutor led discussions, the level of use tends to be low. Participants in the study centre realise that they are engaged in self-paced learning and tend to work independently. Although there are many messages in the system, users are given the option to display only new messages. There tend to be periods of activity as a topic is discussed, then a lull before the next point is raised. The main use of the bulletin board tends to be for technical queries, but it is also used to swap web addresses, allow users to air their own interests and to ask for general advice. As these are answered, they become part of the growing resource of the Study Centre. New users come across answers to commonly asked questions in the bulletin board. This seems to work quite well; if it weren't satisfactory, it would be an easy task to abstract the material to a FAQ (Frequently Asked Questions) page. The Online Study Centre has been running for almost one year now and will soon be internally evaluated as part of TALiSMAN's ongoing analysis of its' courses. The results of this evaluation will be made available when complete. [TOP]

3.6 External Evaluation
Under the SHEFC commissioned, external evaluation of the TALiSMAN project [13], users of the Online Study Centre were approached independently and asked for feedback. Appendix D of that report provides a case study report from one user and a questionnaire completed by another. The evaluators do point out that those replying to the questionnaire are likely to be those with a more positive attitude to the courses.

From the questionnaire responses, many positive aspects of the Centre are highlighted. This user worked the whole way through courses, re-used the materials, and contributed to the discussion forums and resource database. This user felt the level of tutor support appropriate, felt that materials were of an appropriate level and was happy to (indeed already had) recommended the Centre to others. The user was critical of some of the course structuring and provided suggestions for more interactivity which was felt to be lacking.

The simple structure adopted for the TALiSMAN online course attracted no criticism. The Online Study Centre environment created within WebCT has been criticised by many users who dislike the navigation structure and the way some of the tools function. With the extra functionality being offered, the structure of the Online Study Centre materials is necessarily more complex and perhaps requires some familiarisation or induction period.

As a further part of the external evaluation, a 'typical' user was commissioned to work through all courses in the Online Study Centre and produce a case report. This user had already participated in one of the staged delivery TALiSMAN online courses (and hinted that devoting sufficient time had been difficult). The commissioned evaluator was also generally positive about the Online Study Centre (would recommend it to others) but outlined a number of criticisms. Navigation round the courses was felt to be poor - with 'Home' buttons on different pages having different destinations - confusing to the first time user. This failing is inherent in the structure of WebCT, but we had felt that as the materials were self-paced, familiarisation with the course structure could be gradual, rather than requiring a formal induction. A similar criticism was levelled against the homepages - although potentially useful, few participants create personal homepages and this is probably because the tools provided for this purpose by WebCT are not particularly good. Making a homepage for users of the study centre was entirely voluntary. Had we wanted to make this compulsory, we would have investigated and provided more appropriate tools. The final negative comment regarded a technical issue. The 'Web Images' course required software which was not available for the platform used by this evaluator, and there was some degree of frustration at the amount of time wasted before this was realised. This highlights the need for some formal technical support. Although we do give some guidance about the recommended specification of machines used for viewing the Online Study Centre and do our best to ensure that the learning materials can be fully utilised by as many participants as possible, there can be unanticipated difficulties. Again, with experience, this could be rectified.

On a more positive note, the commissioned evaluator liked all the 'active' elements of the course, such as the discussion forum (which was utilised to solve some problems), instructions to 'TRY IT' in the 'Simple JavaScript' course, the resource database and the quizzes. Interestingly, our evaluator was acutely aware of the differing styles of the course (this argues for standard structures and standard styles) and was much more positive about courses that had been created specifically for the Online Study Centre than for those which had initially been delivered face to face. Indeed, it was recognised that courses originally delivered to a group would be better studied online as a group - with staged delivery and discussions at specific times.

In each of the cases cited above, interactivity (or at least activity) is cited as a key issue. Finding a balance between increasing interactive components and maintaining a focus for the course is vital. It is sobering to note how intolerant our users were of materials not specifically designed for online delivery. [TOP]

4 Findings and lessons Learned
Over the last two years TALiSMAN has delivered a range of learning materials (task based, self-paced, discussion centred etc.) over the WWW using a variety of delivery and administration strategies. Some of the approaches we tried have worked well, whilst others were less successful. This section shall examine some of our findings, the lessons we have learned and highlight some of the issues inherent in the use of the WWW and Virtual Learning Environments for staff development.

The special circumstances of our courses should be reiterated. Had our courses been accredited, we would have been able to explore many strategies (such as collaborative working and assessment) more fully. Also, had these courses been run over longer periods, we would have been able to investigate optimum rates of delivery (some course participants asked for longer courses, others for more time between lessons) and course structure. [TOP]

4.1 Expectations
In our experience, online learners tend to have very high expectations of the course; specifically, that the materials they utilise shall be individually tailored to their needs. Perhaps this is because they have less contact with their fellow course participants. We found that some learners seemed to expect almost personal tuition from the course tutor whereas others were reticent about communicating with the tutor at all - preferring to treat all materials as self-study. Without face to face contact, it is perhaps more difficult to manage the expectations of the group. To counter this, activities that build a community and encourage student-student interaction in addition to tutor-student communication should be developed. But this type of activity relies on full participation by students, and can only be effectively implemented in accredited courses. Another way to address high expectations from the learner is to try and customise courses to individual needs, perhaps by providing alternate lessons and allowing the user to choose. A rather simplistic example would be the subject specific online courses we delivered where one lesson was unique and the other lessons common to all courses. Rather than split up participants by subject interest (which had other benefits) we could in future offer a single course, but with 7 choices in week two. A second option is to provide an extra body of optional material to augment the core course materials. In our online course (and indeed in the OSC), we utilised a CGI script that allowed participants to add useful or interesting web site addresses (with short descriptions) to a web page. The resources collected in this way provided a large store of examples for future use. Over time, re-use of materials from discussion forums can form a knowledge base - filling in gaps and catering to individual interests. It is likely that such information would have to be edited to make it coherent, again perhaps to create a FAQ. [TOP]

4.2 Tutor Support
Support strategies for online learning are very different from those required for face to face learning. Whilst administration and delivery of the course can be very efficient, the tutor has to be prepared to communicate far more with individuals, either through email, or in discussion forums. Again, co-operative exercises set for students can be effective here (encourage them to answer each other's questions), but ultimately most communication is with many individuals, rather than a single class.

In our experience, a single tutor can support a large number of students, but the tutoring itself will require an almost full-time commitment; answering individual queries, checking regularly on discussion forums etc. Rather than all activity on a course being concentrated into a single face to face meeting each week, online courses effectively go on all day every day. The tutor will need to check the discussion areas at least twice a day and reply to email queries within a day of receipt, as well as prepare lessons and assessments, carry out administration and send out 2 or 3 messages a week to pace the learning process. For the online course, I spent about 2 days per week supporting the course, though had I been able to spare the time, I could easily have spent more time tutoring.

The way in which the discussion forum is used can also be important. If the forum is relatively unstructured, it is the tutors' responsibility to ensure that all questions raised are answered promptly. If a question is raised and not immediately responded to by other students then it becomes in danger of being lost in the discussion forum as new questions are raised. However a single tutor immediately answering all points raised in the discussion forum can stifle discussion initiated by the students. The use of a team of tutors would be helpful as students will appreciate the variety of styles and will perceive that experts are answering their questions. Tutors can adopt alternative viewpoints to encourage students to think, rather than unreservedly accepting the view of a single tutor. [TOP]

4.3 Assessment
New strategies for assessment can be used in conjunction with online learning - but they must be carefully implemented. The WWW offers great scope for objective type testing - where responses can be automatically collected, graded and logged. Careful design of materials can enable scores and limited to be feedback delivered to individual learners - perhaps integrating assessment into the learning process - formative assessment. This type of assessment appeals to the learner as he or she can benefit from the feedback provided. Of course, objective type testing is often too simplistic a measure of a learner's grasp of the course materials and computers are less effective when used for other more complex types of assessment - so the burden falls back onto the tutor. Although we carried out no formal assessment in our online courses, course activities normally involved submission to the discussion forum and these submissions could have been assessed.

The assessment facilities offered by WebCT are rudimentary, but as they are already built in, they are easily utilised. There are a number of standalone assessment tools available (see section 5) but implementing them can be time consuming and technically demanding, and this can deter course developers from integrating assessment into web based courses. [TOP]

4.4 Format of the Course
We utilised two very different course delivery formats. The very structured delivery of the online course was contrasted with the almost open-learning nature of the WebCT based learning materials in the OSC. No one delivery strategy is right for all course materials or subjects and it would likewise be wrong to suggest that there was an optimum format for all online materials. It is vital that the course content is not compromised by the delivery mechanism. The structure of the online course was governed by the educational needs of the materials, rather than constraints imposed by the delivery medium. We were able to design the exact layout of the pages, the way they were interconnected and the individual external tools utilised (such as WebBoard and customised CGI scripts). We were able to deploy the best tools for our requirements, for instance when choosing WebBoard because of its robust interface. With WebCT, the layout of the web pages is configurable by the author, but the way the pages interconnect is more rigidly controlled - there is little scope for anything other than linear progression through the teaching materials. Like other VLEs, WebCT does not force the course authors to use the tools it provides (email, conferencing, assessment), but it does encourage the author to take the easy option and use the included tool rather than the most appropriate tool.

A further issue regarding WebCT and other VLEs is their unsuitability for offline working. If the participant pays for their Internet access, then they are likely to want to minimise the time they spend online. Software such as WebWhacker allows groups of web pages to be collected for offline viewing. The learner need only log on to initially collect the pages and subsequently to take part in any online (synchronous or asynchronous) activities. This strategy does not work for WebCT because the web pages are stored in a database and cannot be collected by offline browsing software such as WebWhacker.

We should recognise that efficient delivery of accredited staff development is likely to have a significant modular component and it would make sense to consider the scalability of any system. Within reason, (again, there is no sense in compromising the materials just to make them fit a specific template) the learning materials should have a common structure (objectives learning, materials, tasks, discussion, assessment, reinforcement etc). Students will learn more efficiently if they are familiar with the structure - an important consideration if we remember that online students require more guidance. Within that structure, there should be clear sign-posting of a route through the learning material. This guidance can serve to pace the learning process. The use of simple reinforcement exercises can also reassure the student that they have satisfactorily completed that section of the material.

The online course delivered by TALiSMAN consisted of self-made web pages (conforming to a simple standard template) supported by a discussion forum (which could be changed). Administration and course announcements used email. Student records were kept in Microsoft Excel, which was also used to collect and collate feedback form data. Adding new materials would have been relatively easy, but adding large numbers of new students would have strained the administration structure. Any formal accreditation would have required more extensive administration (student tracking, collection of assessments etc.), and this would certainly have overstretched the existing structures. The WebCT environment used for the Study Centre has many administrative tools built in and once participants have been registered with the Centre, the WebCT system itself holds student records, showing access logs, student progress and the scores of any assessment carried out. One of the main advantages offered by networks for the delivery of staff development is the scalability. For efficiency, large scale courses must be modular, easy to manage and easy to administer.

Standards for these factors are being developed by the Instructional Management Systems (IMS) Project [14]. The IMS project is an international initiative whose task is to recommend and create a standard architecture for online learning. The framework will be flexible and allow interoperability without constraining developers. Materials which comply with the IMS standard criteria will be branded as IMS-enabled. IMS standards will cover not just the learning material, but also methods of administration, student tracking, communication, assessment, structure and interface. [TOP]

5 Synthesis and Conclusions
The main purpose of this report has been to review our experiences of delivering staff development over networks. Can we now make some general conclusions regarding the design and delivery of online learning materials? This section of the report will reconsider some of the key issues and provide some pointers to the role of networks in the delivery of staff development within UK Higher Education. The second report of this project will review the suitability of commercial VLEs for this purpose. [TOP]

Course Format
There are two distinct choices of format for online courses. The first is to follow a staged delivery model with cohorts of students progressing through course material at the same time (as we utilised with the Online Course). This allows close monitoring of student progress and provides opportunities for co-working and creating an community. The second option is to provide supported open-learning materials (as we did with the OSC). Although from the tutors point of view, the ideal option is to follow the first model, staged delivery is considerably less flexible than self-paced learning as it will require a regular (and rigid) time commitment from the user for completion. If participation on a course is supported through an employer, then this regular release from other duties might be possible. This would be the case for nationally accredited professional development or courses accredited internally within an institution.

It can be advantageous to have some face to face sessions in a course. A face to face induction session can activate a learning community (and can provide an opportunity for remedial training and necessary familiarisation with the online learning environment). Occasional face to face sessions can also structure open learning materials, if there are progression deadlines for each face to face event. Learner homepages, where individuals can tell the rest of their group about themselves (and if appropriate) make available some of their own work (a portfolio) are also of use in maintaining and building this community.

Course Structure
For any online study materials, consistency of structure will benefit students. A familiar structure will allow students to pace their learning. A standard, flexible structure for a lesson might include; introduction, clearly outlined objectives, course materials, activities designed to allow students to demonstrate their skills/knowledge, revision, and finally, further resources. Different sections might be colour coded to reinforce the underlying structure. Such a course structure should accommodate most materials. If some materials deviate distinctly (for instance presentation of a case study), then they should be presented in a different for instance as an 'interlude'). Interludes will also help to alleviate monotony. Whilst progression through course materials is normally linear, this can feel too regimented (and the students will feel as if they are being hand-held) so it is helpful to include sections of less rigidly structured materials, perhaps bringing together different sections of the course or presenting an alternative viewpoint.

Modularity
Any large scale use of networks for the delivery of staff development will have to be modular. Such a structure is easily scaled, allows the incorporation of materials from different authors, allows highly customised learning programs to be delivered and is easy to revise. Modular delivery is especially suited to self-study materials and is flexible enough to cater for the occasional user. For national, integrated staff development programmes, a modular basis to all courseware is essential for administration and accreditation.

Assessment
Any accredited course will have some form of assessment. Whilst network delivery of formal examinations is not practical, networks can be used to deliver (and administer) simple objective type tests and can be used to collect projects and dissertations. The CASTLE tools [15], the Clyde Virtual University Assessment Engine [16] and WebTest [17] can manage web-based assessment, whilst VLEs such as WebCT have assessment engines built in. Assessment scores can be logged and feedback returned to students. Such simple assessments can be helpful to tutors too. If score logs are analysed, tutors can see if specific parts of their course are not being understood by the student group as a whole. As well as being helpful for students' revision, simple assessment can be effectively used to structure the course materials. After completing a section of work, a student may be asked to complete an assessment which fulfils a number of roles. Assessment reinforces the learning material, can provide an indicator to the student of whether they have met the objectives of the learning material and can act as an identifiable endpoint for a section of work: once the assessment has been delivered and feedback obtained, that section of the course has been completed. If course discussion forums are used extensively, then some assessment should be made of contributions to these areas as students may have spent considerable periods of time preparing submissions. Student participation can be easily quantified by analysis of access logs (time spent online, number of submissions to bulletin boards and even number of responses elicited by a given submission). Qualitative assessment of contribution still requires the tutor to read the messages.

Activities
Active participation is the key to effective online learning, indeed it could be argued that electronic delivery is useful only in the transfer of knowledge and that any skills component of learning requires communication with others. Within an online environment, activities that work well include role-playing and debate, structured discussion, electronic seminars and collaborative projects, so VLEs that include facilities for collaborative working are especially useful. For longer term courses, students can be instructed to prepare portfolios to demonstrate their application of the skills and knowledge they have acquired. For assessment, contribution to these activities can be examined retrospectively from access logs and electronic submissions.

Administration
For large scale delivery, automation of administration is critical. WebCT and other VLEs provide student record and student tracking facilities which are useful within courses. In addition to logging scores and simple administration details, these tools can be used to examine participation rates for individual students and highlight potential problems.

Accreditation
Formal recognition of skills gained after leaving full-time education is becoming more important as we move to a culture of lifelong learning. Furthermore, the Dearing Report recommended the establishment of an Institute for Learning and Teaching in Higher Education (ILTHE) to oversee and administer staff development within HE. This Institute, now being set up and to be named ILT (Institute for Learning and Teaching) is intended to provide a 'professional standing' for Higher Education teachers [18]. All probationary lecturers would be expected to undergo some training accredited by ILT.

Technical Support
Technically, there should be nothing special about a course delivered electronically. The technology must be transparent and any temptation to use special features (plugins, audio, and video) must be justified. Technical requirements should be determined before the course and these requirements checked with participants. It is advisable to check that required plugins and special software is functioning correctly, possibly by piloting before the course starts. It is also important that students are familiar with all course tools such as the discussion forum. This familiarisation could constitute a Week 0 for the course, or perhaps form part of a face-to-face induction event.

Supplementary Materials
Whilst it is important not to include extra material if it will merely distract the students, a bank of supplementary material can be very helpful as a starting point for a student's own exploratory learning. Whereas the core learning material will be tightly controlled by objectives; extra material can be more openly thought provoking. For network based materials, the WWW provides a wonderful resource which is easily exploited. In addition to informal resource banks, more formal directories of general resources are useful and encourage exploratory learning. Students can be encouraged to contribute information on resources they have found themselves. In the longer term, this can even be extended to collecting student assignments and making these available as resources for future students.

Frequently Asked Questions (FAQ)
Although it is not always appropriate to provide an answer before a student has formulated an appropriate question, FAQs can address identified issues. If the FAQ is drawn from existing logs of discussion forums etc., then the questions and answers can be matched well to students needs.

Tutor Support
It is vital that students have some access to all tutors on the course, through access to course-wide discussion areas, but tutor support should feel personal and it is therefore important to assign tutors to individual students. Similarly, it also pays to create distinct study groups containing a single tutor and a number of students. Individual groups of students might be further split up for specific activities. For some courses, mentoring may be appropriate but for this personal email contact should be sufficient. If face to face contact is not possible, tutors should at least endeavour to make themselves more approachable, by having personal homepages etc. Learning Skills, Learning Styles and Learner Streaming

For accredited staff development in HE, most learners will be graduates and will have considerable study skills already, but these may be ill-suited to online learning. Learning material which works well 'on screen' is likely to be radically different to traditional paper-based materials and require new study skills. Basic IT skills are an essential prerequisite for any course which utilises computers and online courses obviously involve intensive use. Learners should be comfortable with navigation through learning materials, working with multiple windows and electronic note-taking. Where contribution to discussion forums is expected as part of the course, some familiarisation is essential to ensure that participants do not come to see the technology as a barrier to their participation.

Even amongst those familiar with computers, there will be some who are disinclined to take part in course discussions and co-operative work depending on their learning style. Learners who may be vocal in face to face tutorials may be inhibited by the permanence of a discussion forum submission. In contrast, learners who would normally be quiet in class might welcome the chance to reflect on their ideas and prepare a formal submission. For courses conducted entirely online, it may be possible to organise synchronous online discussions (in addition to asynchronous activities). Synchronous sessions can be used to 'brainstorm' at the start of asynchronous discussions. Also, their informality makes them attractive to those who would normally feel stifled by the rigid structure of asynchronous forums.

In order to build strong communities, it is important that learners are streamed according to their initial knowledge. Within large courses, students can be allocated to groups of similar ability. Any progressive course structure would also alleviate this problem, as learners would be expected to have fulfilled certain prerequisites for a course.

Revision
Whilst there should be little need for revision, online course materials are at least far easier to adapt and revise than traditional printed materials or even electronic materials delivered on CD. Any errors found and suggestions for improvement will be recorded in discussion forums or emails sent to the tutors and should therefore be easy to act upon. If revision is necessary, there is no cost in re-printing, re-pressing CD's or administration involved in version control. Instead, a single, definitive 'instance' of the course is always present on the network (web site). It is important that changes are logged, and it would be appropriate to provide a 'revision history' online. For the materials themselves, instead of changing the core materials for the course, revision effort should be focused on improving the knowledge base available to the learners. [TOP]

5.2 Conclusions
Online courses differ greatly from face to face courses. There is a trade off between the flexibility offered by online study and the learning culture engendered by face to face teaching. Good online delivery of staff development must retain this flexibility, and also create a community where learners can come together 'virtually' to share their learning experience.

We found that online course participants harbour very different expectations to face to face students. Managing these expectations is critical - whether by encouraging student-student communication or by anticipating the students desires and providing an extra resource for them to refer to; exaggerated expectations only become a problem if they are not addressed. Allied to this, it is important that course objectives are clearly stated, that navigation through the learning material is intuitive and also that activities (such as simple reinforcement exercises) demonstrate when/if objectives have been met.

Online students also require more explicit guidance on progress through the course - signposts which indicate the expected route through materials, and also the expected progression rate. These signposts may be concrete (as with staged delivery), formal (as with specific assessment deadlines) and informal (as with prompts from the tutor).

Assessing progress and performance in online courses requires new strategies which accurately gauge the success of the learning process. Assessment of activities is likely to be just as important as assessment of essays and projects.

At Glasgow Caledonian University, a course in learning and teaching for new lecturers is now delivered online [18]. This course uses web pages, three face to face tutorials, assignments and a discussion forum. The course has been delivered twice in 1997-1998 and changes planned for further delivery. As with our online course, the course leaders were disappointed with participation rates, and the changes they have recommended for the next delivery concur with the findings of this report: prepare participants (clarify expectations and familiarise them with components), make materials more interactive and introduce more activities, and encourage more use of discussion forums.

The new opportunities offered by networks for the delivery of staff development look capable of providing the flexibility required by workplace learning and professional development. Exploiting the World Wide Web for the delivery of these courses requires careful design of materials and courses, together with imaginative use of technology to build communities. Finally, managing the learning experience effectively is vital. Within reasonable constraints, Virtual Learning Environments can support and enhance effective learning, but although they provide new opportunities, their implementation and use must be carefully planned and adequately supported. This report has examined one (WebCT) and compared it to the facilities offered by traditionally created and administered web pages. The second report from this project will examine different VLEs and evaluate their suitability to the delivery of online staff development, given the issues we have discussed in this report. [TOP]

Launch Date :10 March 1997. 2nd cohort runs from 12 May 1997
Duration :6 weeks Hours :Approximately 3 hours each week

Overview
The Metropolitan Area Networks (MANs) offer great potential for delivery of high quality courseware and resources on the WWW. It is vital that those wishing to use WWW based resources are able to find and evaluate existing resources and courseware effectively. The ability to evaluate material is best learned by extensive practical research. This course shall encourage that research and provide a forum for the exchange of ideas - allowing all participants on the course to learn together.

Aims:
By participating in this course you will investigate how the Internet can be used for teaching and learning, evaluate the effectiveness of the Internet as a delivery mechanism and exchange ideas with other educators in your community.

Course Objectives
After completing this course, participants should be able to:

Cost
Staff from Scottish HEIs may participate in this course free of charge. Members from outside this community may participate if there are places left and will be charged course fees.

Week One - 16.5.97
Introduction; current uses, initial thoughts, and reservations, what we might hope to learn.
Week Two - 23.5.97
Finding Available Resources; avoiding search engines, knowing your sources.
Week Three - 30.5.97
Evaluating WWW Resources; quality criteria, good and bad practice.
Week Four - 11.6.97
Evaluation from the Student Perspective; technology for the sake of it, resources, relevance.
Week Five - 20.6.97
Using It; provision, implementation and integration of WWW based learning material.
Week Six - 27.6.97
A Synthesis; the future - Scotland, the MANs.

If you are not using a forms-capable browser, please email your response to this form to: colin@talisman.hw.ac.uk

Having taken part in the TALiSMAN Online Course, 'Using the WWW in Teaching and Learning', we would now like to ask for your comments on the course. Can you please answer the questions below. Some of the questions require answers. For others, response is optional.

1. Name: _________________________
2. Position: ________________________
3. Institution: _______________________

4. Would you object to your comments being used in future by TALiSMAN?
[ ] No (if attributed)
[ ] No (if unattributed)
[ ] Yes

Previous Experience
5. Have you previously used computers (inc. computer based learning material) in your teaching?
[ ] Not at all
[ ] For Preparation of Teaching Material
[ ] For Delivery of Teaching Material

6. Had you used the WWW before participating in this course?
[ ] Not at all
[ ] For Browsing
[ ] For Browsing and Authoring HTML

7. Had you previously used discussion forums before participating in this course?
[ ] Not at all
[ ] Informally
[ ] Formally (in a course etc.)
[ ] About the Course

11. Would you suggest changes to the scope or level of the course?
[ ] Yes
[ ] No
[ ] Dont Know

12. On average, how much time do you think you spent on the course each week?
[ ] less than 2 hours
[ ] 2-3 hours
[ ] more than 3 hours

15. Did you find the tutor support provided adequate?
[ ] Yes
[ ] No
[ ] Dont Know

16. Did you find the course useful in relation to your work?
[ ] Yes
[ ] No
[ ] Dont Know

17. Do you have any immediate plans to use the WWW in your Teaching and Learning? (optional)

About Dialogue
The Dialogue software provided a very simple conferencing area for course discussions.

20. Did you find 'dialogue' robust enough to cope with the level of discussion?
[ ] Yes
[ ] No
[ ] Dont Know

21. TALiSMAN is about to run this course again with modifications to content and delivery. We are keen to improve the course, can you suggest any improvements for this or future TALiSMAN OnLine courses? (optional)

Thank you for filling in the form, if you have any further questions regarding the course, this feedback, or any of TALiSMAN's activities, please do not hesitate to get in touch.

The Role of Virtual Learning Environments in the Online Delivery of Staff Development

Report 1: Review of Experiences of Delivering TALiSMAN Online Courses