|Route map: Home Publications Evaluation Studies ... Evaluation of WWW Resources ...|
Development and Evaluation of WWW Resources to Support Research Methods and Electronic Engineering: a comparisonRobert Newton, Iain Middleton, Rita Marcella, Alan MacLennan The Robert Gordon University
This paper describes the development and evaluation of two WWW based resources developed under a single funding programme during 1997 at The Robert Gordon University: 1) ReMOTE: a resource of collected materials for teaching and learning Research Methods skills; 2) Personal and Mobile Communications: a module of a postgraduate course offered by the School of Electronic and Electrical Engineering. The projects differed in their rationales for development, subject matter, manner of development and in the evaluation of their final deliverables. These issues are compared and contrasted and their effects on the outcome of their final evaluations highlighted. Problems associated with the effective evaluation of each arose for a variety of foreseeable and unexpected reasons: for example the difficulty in getting access to students in order to conduct evaluations, practical problems preventing the participation of lecturing staff, low levels of responses to surveys and in one case the presence of only a single student electing to take a course. Finally, the positive and negative elements of each experience are used to draw up a series of recommendations for future developments. These include: the need to design for a clear purpose; the need for close co-operation and input from relevant teaching staff; and a requirement for advance planning of the timetabling and logistical requirements of evaluation planning.
In January 1997 the ReMOTE (Research Methods Online Teaching Environment) project received funding for a 12 month period via the Violet Lessel Trust Fund - an internal fund to support teaching and learning initiatives within the Robert Gordon University. The original purpose of the project was to develop a World Wide Web (WWW ) site devoted to the support materials for teaching Research Methods and to integrate access to a range of in-house material with Internet based teaching materials . The project team had previously had experience of creating web sites to facilitate access to teaching and learning materials using a variety of techniques - multimedia authoring packages to develop in-house teaching materials to support a taught unit in bibliographic classification, creating a web based information skills tutorial to support open and distance learners within a public library and developing and maintaining an online directory of teaching and learning resources related to Internet skills training (NetLEARN).
During the initial discussions within the University relating to the project's remit it was decided to extend the project to include development of tailored web based materials in a specific subject discipline (Electronic and Electrical Engineering).
The project thus comprised two distinct parts associated with two schools within the University.
- School of Information and Media: the ReMOTE (Research Methods Online Teaching Environment) project's purpose was to develop a World Wide Web site devoted to the teaching of Research Methods, unifying access to materials on the topic developed in-house (primarily from an existing print based Self Study Learning Package for Research Methods). The remit covered:
- the identification and conversion of existing RGU materials to WWW format;
- collection and classification of relevant WWW links;
- the development of interface for integration of these resources within taught programmes - specifically at students on the postgraduate Information Analysis course.
- School of Electronic and Electrical Engineering: production of two postgraduate level modules to complement the teaching of these subjects in the school, giving overviews of their respective areas. The remit covered the production, conversion, acquisition and obtaining clearance for use of materials.
The funding was allocated to provide a full time research assistant to develop the materials. The Research Methods element received the bulk of the funding, and accordingly was allocated the bulk of the developer's time: nominally 4 days/week to the remainder's 1 day/week. The role of the developer was to liase with the staff at whom the resources were primarily aimed and to undertake development according to their needs. This paper will be presented in two parts and provide a discussion of both projects but will concentrate primarily on issues arising out of the ReMOTE project.
ReMOTE - Background and Rationale
Virtually all UK higher education institutions teach Research Methods both in undergraduate and postgraduate programmes. In the latter Research Methods is often seen as a key component in preparing students to undertake study leading to an MA or MSc qualification. Typically the topic is taught as a programme of lectures and seminars.
The key features of Research Methods which make it an appropriate area to develop as an Internet resource are that:
- it is generic and transferable;
- because of the wide range of potential topics which could be matched to an individual student's dissertation it is often the case that each student requires a different level of depth for different topics (this is particularly true for postgraduate programmes of study);
- the environment in which the project will be operating (the WWW) is one which is increasingly important in conducting practical research. It was thus felt that exposing students to learning using the Internet as a mechanism for delivery might in fact contribute to the learning process.
It is therefore apparent that such a widely applicable skill as Research Methods, and such a ubiquitous medium of delivery as the WWW are potentially suited for the development of a resource centre which might be of global utility.
Currently there are several WWW resources available for teaching Research Methods or facilitating the research process, the former typically concentrate on a subset of skills appropriate to a particular discipline or a single aspect of Research Methods, while the latter (e.g. SOSIG) tend to offer sites of use to researchers, and a few selected sites for teaching the subject. Pachnowski et al. provide a useful discussion of some of the sites available which provide databases and examples of surveys of resources (e.g. the US Census Bureau http://www.census.gov) and the Gallup Organization (http: //www.gallup.com) but do not provide any detailed guidance on application of these resources in teaching. Useful resources are also identified by a range of directories - the Educator's Internet Yellow Pages being a particularly good starting point. More specifically related to Research Methods work by Cozby provides an excellent guide to resources - each chapter of the book directing the student to a wide range of web resources. There are also a number of web sites which give detailed tutorials on specific Research Methods concerns or techniques (e.g. BeLue's Choosing a research design http://trochim.human.cornell.edu/tutorial/belue/belue.htm and Burn's pages on Securing internal validity http://trochim.human.cornell.edu/tutorial/burns/int.html).
The aim of ReMOTE was to provide an infrastructure which allows direct links to specialist sources which will enable the use of existing WWW materials as "plug-in teaching nodes" when developing tailored Research Methods courses. The project aimed to integrate the use of internal electronic based materials with those available on the web. For a particular taught course, therefore, academic staff could provide programmes of study which made extensive use of links to electronic 'readings' in the same way as references to the sources would normally be given in paper based documents. The researchers then tested the effectiveness of the approach and the problems which this particular type of development raised and studied the manner in which the ReMOTE pages were used by staff and students within the University.
Design and Structure
The structural and graphical design of the site required careful consideration in order to achieve a consistent and logical appearance. The factors which required to be taken into consideration included the goals of the potential users and of the university itself:
Navigability: use of logical structures to maximise the ease of use with which a user can locate the information they require, and to prevent them becoming "lost" in a complex structure of information.
Clarity: the need for all of the possibilities to be presented in a clear and comprehensible manner, making it clear at each stage - without the need for undue searching or experimentation on the user's part - exactly what information can be reached from the current point.
Attention to design: with all of the alternative sources of information on the WWW, it must be made obvious to the user that the resource has not been developed as a part-time "pet project" by an individual. A professionally designed look will be appealing on a visual level and engender confidence that this is a quality resource.
The need to establish ownership and a corporate identity; to remind users that the Robert Gordon University is providing this resource.
To provide a robust structure for the resource which will accommodate future changes and additions to the site in a logical manner without the need for excessive re-working of existing structures.
To provide quality information in a manner which: enhances the pedagogical aims of the university; and enhances the profile of the university as an innovator in the use of novel techniques for information delivery.
On a practical level, these issues fall into two broad categories: Graphic design &Structural design.
Graphic Design: Identity
It was decided that the three essential elements were:
- The pedagogical purpose of the site: i.e. the teaching of Research Methods skills;
- The use of technology (the WWW) to enable distance or independent learning;
- Institutional ownership of the resource.
The site's opening page is shown on the right.
The title ReMOTE itself is an acronym constructed from the words "Research Methods Online Teaching Environment". The use of the term "remote" also reinforces the distance learning aspect.e banner graphic uses considerable corporate and research-appropriate imagery. The underlying graphic consists of photographs taken from the 1998 prospectus which illustrate elements of research; themes represented include use of telephones, interpersonal skills, technology (computer keyboard, CD-ROMs, floppy disks), libraries, study, books, files etc. The RGU shield and corporate colours are also prominent.
The site content is overlaid on a background consisting of the undulating margin adopted on RGU publications, with the individual elements of the RGU shield (castle, boar's head, cog, sphere and torch) continuing to the right hand side. The undulating margin serves a functional purpose, containing the navigation "buttons" which enable the user to access the site.
These graphic themes run throughout the contents of the site, and provide a consistent look and feel: it is easy for a user to tell which pages are part of the ReMOTE site.
Graphic Design: Functionality
In addition to fostering a strong sense of purpose, technology and ownership ("PTO" identity), many of the graphics have a functional purpose. A decision was made that the non-use of graphics should, wherever possible, not inhibit use of the site: non-graphical browsers are still in use and browsing on standard browsers may be speeded by switching off graphics. Therefore functional graphics in the site are of a simple nature, generally only renditions of text with some embellishment e.g.
Navigation: the navigation buttons in the left hand margin, where the current page is highlighted in green with a tick beside it to make it absolutely clear to the user where they currently are in the structure. The functionality will be only marginally less clear if browsing in text only.
Pedagogy: in the actual learning materials, graphics do have an essential purpose. Where they are necessary, they have been kept to the minimum size for legibility on displays of varying resolution. However, the issues which can be discussed here are purely technical, since the use and design of these graphics are issues for the materials' authors.
The site is built on a hierarchical plan, where the most general resources contain links to the more specific ones. The home page serves to inform users of the purpose of the site and give access to the broad categories (not to the actual materials).
The pages on the next level down provide access to
i) a directory of Research Methods teaching resources available on the WWW arranged in broad subject categories,
ii) online resources produced by staff in the University and held on the university server,
iii) online course descriptions which integrate access to online resources students registered on specific courses.
Pages on the top two levels of the hierarchy link clearly to each other, so that a user does not have to backtrack to the home page every time a different section is required. The materials which have been installed on the server are structured in a manner reflecting as far as is possible that of the original materials. There have been some necessary alterations in terms of navigation e.g. the addition of hypertext contents pages.
Content of ReMOTE:
The contents will be dealt with in three parts:
- The directory of links to other WWW materials;
- The materials actually hosted on the server;
- Courses under development using the above materials.
Directory of Links
Requests for WWW Links
This was a significant part of the final deliverable of the project - it is a collection of links to already-existing materials elsewhere on the WWW. In order to identify materials for evaluation and possible inclusion in the site, input was solicited from a variety of e-mail discussion lists and online forums which have an interest in web site development and teaching Research Methods. An announcement of the project's intentions, coupled with a request for contributions, was sent out to:
- Mailbase UK Higher Education's electronic discussions host. A number of lists were identified which might have an interest in the development of such a resource, including:
- Other e-mail lists and electronic forums, including: Nettrain@listserv.acsu.buffalo.edu
A number of responses to this message were received, generally of a positive nature, although most revolved around the theme of "please let me know when it's ready". However, several worthwhile resources on the WWW were communicated, and some sites and individuals promised to donate materials, or to look into the practicalities of doing so. Similar messages were sent to individual sources which it was known had produced materials which might be suited to WWW conversion e.g. the PROCARE project at Southampton University. As in the case of the individuals who responded to the initial request offering their own materials for the site, enthusiastic initial responses have been followed by long periods of silence: in the case of PROCARE, copyright clearance deliberations at board level have thus far prevented any progress.
Searching on the WWW
In addition to asking interest groups to provide references, further materials - finally forming the bulk of the directory - were located using WWW searching techniques. Many materials were located via previously located WWW sites, which often include a list of links to other related sites ("further reading"). This has the advantage of the implicit recommendation by the link-maker in referring to the materials, but also restricts the user to what the individual author has himself already found.
Searching was therefore extended to Internet directories and search engines e.g. Yahoo, The Argus Clearinghouse, AltaVista, Excite, Infoseek. This is a less focused method of identifying suitable resources than by individual recommendations, but has the potential to identify a much wider range of resources, and the majority of the directory's contents have been located by this method.
Due to the extensive nature of the directory of links, it has been subdivided into categories for each aspect of Research Methods. Each category of the directory has a separate page, accessed from the main directory page. The format for directory entries is based on the style used in the NetLearn project. The resource title is given as a hypertext link, followed by a brief abstract describing the resource, its coverage, target group etc and finally some categorisation information. The information given should be enough to tell the user whether the resource is likely to contain the type of information which they might find useful.
STEPS Project, University of Glasgow.
Downloadable CAL modules for the teaching and learning of statistics, specifically applied to the following areas: Biology, Business, Geography, Psychology. Suitable for PCs, also some MAC material. The packages utilise graphic and interactive techniques. TYPE: materials, downloadable CAL software
An example directory entry
There are a number of drawbacks to the directory in its current form:
- It is not searchable as a database would be. The name, URL, Description and categorisation information could all be used as fields in a database, but at present this is not possible.
- Resources may fit into more than one category, resulting in duplication.
- Updating the directory must be done manually by editing the relevant HTML document: with the amount of information requiring updating: this is unwieldy and prone to errors.
If the directory is to be maintained and promoted as a feature in the future, some investigation must be made into the possibility of converting the information into a database format accessible through a WWW front-end. This would not only make the user's job searching for relevant materials easier, but also the maintainer's job of adding, amending or deleting resources. Scope would then also exist for the addition of a "user comment" feature, where users could add their own annotations as to the usefulness of any individual resource.
Materials hosted on the server
The material was currently being used in printed format, but also existed in electronic format and would not therefore require laborious verbatim retyping. The package consisted of over 100 pages of mixed text and illustrations. The approach taken was to use a conversion utility (MicroSoft Internet Assistant, or IA) which would take existing Word documents as input, converting them into HTML. The process of conversion with IA took only a few hours, but immediately presented problems: each individual Word document was converted into a single HTML document, which was not navigable and required to be broken into sections which would then have to be linked together. The HTML output by IA was also non-standard, and attempted a very literal interpretation of the look of the document when what was required was a representation of its structure and meaning. The design parameters of paper and the WWW are not compatible, and the document has required considerable restructuring to make it manageable, and the inclusion of navigational facilities for locating and moving between documents - most notably tables of contents constructed of hypertext links to all of the available pages.
A great number of further problems remained to be rectified due to the shortcomings of IA: many graphics had been discarded, others required re-sizing to make them legible on differing display resolutions and complex text formatting needed to be reinstated. An initial version was presentable after a week of work, but finding and correcting errors produced by IA required much longer, and the assistance of the original authors who were much more familiar with their material. The existence of a template into which content could be "poured" and links added automatically would have saved many days of effort: a single change in style often requires every page to be altered slightly.
The final product retains the original material's content but is structured differently. The materials are broken down into smaller sections and direct access given to each through tables of contents (a device not used in the original materials). Although changes to the material were not within the scope of the project, in the future it is envisaged that they will gradually be altered to suit their new medium more closely, for example by the inclusion of links to outside WWW materials which might be referenced in the same way as texts would be in a conventional paper. An interesting point to note here, however, is that the formative evaluation of CAL materials derived from paper based materials poses considerably more problems than developments geared specifically at the production of electronic deliverables. In part this was reflected in the project in what appeared to be a 'responsibility gap' where the division of responsibility for ensuring accurate transfer from paper to web was not clearly defined. This is also - and potentially more seriously - reflected in what may be termed a 'mind-set gap' which is evidenced by the originators attitude to the value of converting the materials from paper and unwillingness to grapple with the issues of converting material to a new medium.
Further materials developed elsewhere within the University were also to be considered for inclusion on the server where permission and copyright clearance could be obtained. It was already known that some materials developed in other Schools existed in electronic format and might be suited to conversion to HTML format and included in the 'library of resources' for ReMOTE. The amount of material, developed in isolation by lecturers can only be speculated at. Some individuals have responded enthusiastically to the possibility of making their own notes and lectures available on the WWW - although at the time of writing (more than a year after the start of the project) none have actually delivered any materials. This is clearly an area in which further work is required. We need to develop an institutional policy which will encourage a more collaborative approach to learning materials development and dissemination across the University and investigate fully the factors which are preventing collaborative approaches to developing and sharing teaching materials.
The final element of the site is the integration of materials into course modules or parts of course modules for specific subject disciplines. It was envisaged that as part of a taught programme materials from ReMOTE could be built into the course - the actual materials and the time suggested for their delivery being determined by the lecturer responsible for the taught unit. These could also be integrated with the lecturer's own materials and used either as a replacement for or extension to normal modes of tuition: it may be prescribed as an activity or coursework, or it may simply be used for "extended references".
The use of the HTML frames facility is fundamental to this section. In using this, a lefthand column on the screen is used purely as an index for the actual materials. These materials are called up on the main portion of the screen when an item from the index is chosen. Therefore, if the majority of material to be used consists of existing WWW pages, there is a minimal requirement for writing HTML - this will consist of nothing more than a plain list of URLs.
Taking the Postgraduate Diploma in Information Analysis (PGIA) as an exemplar of how this facility would work in practice the developer used the standard course information which was into HTML format - this included the course unit descriptor and bibliography, plus references to the Social Sciences package and selected lecture notes/OHPs. Ideally it was envisaged that suitable additional areas for online study will be identified in co-operation with the lecturers responsible for teaching the Research Methods module and an online programme of directed study would be created - clearly linking the learning outcomes from the unit descriptor with the aims and objectives of the online learning resources. Even in cases where the learning materials were not web based it was envisaged that such an online programme would have considerable benefit in integrating and structuring material to provide a rich learning environment for students. In practice this was only achieved to a limited extent (because of problems in gaining access to staff and the limited commitment shown by staff to the project) but the project team see this as an important feature of the interface and when the materials are used by future cohorts this feature will be considerably enhanced.
Formal evaluation procedures were carried out for the deliverables which were used in a teaching context during the project. Evaluation proved to extremely difficult partly because of technological problems which arose during the course of evaluating the teaching in class and sequencing the delivery of the online materials. In addition there were problems in motivating the students to participate fully in using the resources provided and accurately interpreting the basis on which students were assessing the materials provided as opposed to the mode of delivery. Too few students participated to allow for statistical manipulation of results and the analysis of findings are essentially qualitative and will have to be confirmed in subsequent studies using larger student populations. Evaluation took place using 4 different user groups providing a range of different situations in which it was envisaged the materials could be used:
- Tutorial sessions in the computer laboratory with students registered on the Postgraduate Information Analysis course, with immediate user feedback via questionnaire and interview. (A class of 33 students of whom 13 provided formal feedback).
- Tutorial session as a part of "Using the Internet for Research" course for PhD candidates with immediate user feedback via questionnaire and interview. (A class of 13 research students of whom 7 provided formal feedback).
- Self-study: - students from another postgraduate course within the School of Information and Media (Electronic Information Management) who used the materials in their own time after an introductory session and who were invited to respond using an online questionnaire. A class of 10 students of whom none provided formal feedback.
External use of the site with online feedback from WWW response forms. Usage statistics are provided below.
Evaluation in tutorial sessions
The ReMOTE web site was first tested during tutorials with students on the Postgraduate Diploma/MSc course in Information Analysis. One week prior to this the class had been given an initial introduction to the site. The class of 30 students was then was divided into three and each division allocated a one-hour timeslot to work through eight questions designed to take them on a "tour" of some of the more important aspects of the site. The hour was designed to acquaint students with the site, encourage them to explore it and to elicit feedback. The feedback was derived by a single questionnaire which consisted of 2 sections:
- section 1 presented students with a set of 8 questions designed to confirm that the student had understood how to navigate the site to obtain specific information;
- section 2 consisted of a series of questions to provide feedback on how the students reacted to the manner in which the course was presented.
Conditions for testing were not ideal, since to fit in with the existing schedule it was necessary to hold the tutorials early in the academic session, prior to the students' introduction to the Internet and before many of them had had the opportunity to familiarise themselves with basic computing. To compensate for this, an introductory session was held in class the previous week where the site was demonstrated, and a handout given with details of how to access the site. However, problems were further compounded on the day of the tutorials by a faulty computer network which made access to the site, for one group (the second group) in particular, almost impossibly slow. These problems persisted for the remainder of the week in question and thus prevented the completion of the exercise by these students out of class, and also prevented their receipt of e-mail requests to complete the exercise.
In spite of the difficulties, each group was able to get some experience of the site, and questionnaire returns show that the majority were able to answer most of the questions posed. Respondents had varying amounts of time in which to complete the exercise: the second group of the three in particular, which experienced such poor network performance that some had less than 20 minutes of the allocated hour after eventually logging in, whereas some students in other groups chose to continue working with the resource throughout two scheduled classes and received supervised assistance for almost 2 hours.
The 8 questions which constituted section one were assessed on the basis that an answer was deemed to be correct if it demonstrated that the respondent could locate the relevant information, whatever their subsequent interpretation of it. It was not a test of their Research Methods knowledge, rather the efficacy in identification of appropriate materials. Since it was to be used on what was likely to be the students' first contact with the site, probably even with the Internet itself, the questions were set at a level which reflected the fact that the students were novices.
In all, 14 responses were received to the request to tackle the first 8 questions. Of these, the majority (9) had succeeded in answering all 8 questions and no student achieved a score of less than 5.
Individual performances along with formal and informal feedback showed that respondents' experiences had varied widely: while some managed to complete the exercise with little assistance (and working under the poor conditions noted above), others required more time and/or help. Some language difficulties occurred, as did problems resulting from a lack of familiarity with the technology. It is interesting to note that the group which experienced by far the most severe technical problems, having the least time to complete the exercise, were in fact observed to proceed more confidently and successfully than the final group who experienced few if any technical problems, but as a group proceeded with considerable caution and lack of self-assurance, often becoming "stuck" at the same points, with and without explicit visual and verbal cues. There is no evidence, nor indeed likelihood of any major difference in skill level between these groups: it is thought that among those who are almost uniformly inexperienced in the technology and subject in hand, the atmosphere of the tutorial was a significant factor in their progress through the materials: good humour developed among the group which experienced the most technical problems, and they realised that they had little time in which to achieve a great deal. The next group did not "gel" in this fashion, and required considerable reassurance and assistance. The importance of the role of the tutor in such situations is noted, and this is not restricted to supporting students in understanding technical issues and helping them overcome problems in using the interface, but extends into the role of a "counsellor" who can provide motivation and enthuse students.
The section two 'feedback' questions were designed to provide a gauge of initial response to the resource, and were therefore brief and couched in general terms. The feedback questions were included as a continuation of the questions designed to test ability to use the site, in order to convey the idea that the feedback was an integral part of the exercise i.e. not an optional extra. It required little effort to complete: responses were given on a 4 point Likert scale, chosen to avoid the temptation of respondents to choose a "middle" answer each time. In spite of this, the preponderance of 3's suggests that inexperience prevented a highly informed critical appraisal, and what we can infer is an open minded but often unsure attitude to the site's usefulness.
In all 13 students provided feedback information via the questionnaire.
The first two questions were purely to elicit information about the individual using the resource. The substantive questions and a summary of responses given are provided below.
Figures give no. of responses & mean response (on scale 1-4, where 1= most experienced and 4=least experienced)
3. How experienced are you in using: a) Computers in general 13 2.00 b) The World Wide Web (WWW) 13 1.77 c) Research Methods 12 1.67
4.a) Did you find it easy to move around the site? 13 2.69
b) If not, where/how did confusion arise?
Comments:- inexperience and pathways to answers
- no knowledge of the meta language of Research Methods
5. Were the materials and resources sensibly structured and easy to find? 13 3
6. Was the appearance of the site (graphics, layout) clear and helpful? 13 2.92
7. Do you now have a good idea of what the site is about, what it's trying to do? 12 3.08
8. Please rate the quality of the individual materials on the basis of what you have seen today:
a) Self Study Learning Package for the Social Sciences 12 3.00
b) Links to other sites: overall rating 10 3.10
Comments? "a bit difficult to access some sites: perhaps need more than 1 hour insufficient time to analyse for Q8"
9. Overall, do you think you have benefitted from the tutorial today? 13 3.46
10. Do you see ReMOTE as being a useful resource in the future? 13 3.46
11. What do you dislike about it? Please note any improvements or additions which you think might be made.
Would prefer internet tutorial first
Perhaps better signposting for questions - more explicit
Sometimes it was slow, but overall fairly friendly
The first of the questions above (question 3) illustrated the lack of technical and subject experience of the group. While most indicated a small degree of experience with computers in general only 2 indicated a greater degree of competence. As the tutorials took place prior to their introduction to the Internet and most of the familiarisation with IT this was not surprising but this low self-perception of technical capability did not appear to produce significant difficulty in using the tutorial materials.
In response to question 4 on how easy they found navigation around the site the majority of students reported little difficulty (and where they did they related this to their own inexperience) and similarly they were highly satisfied by the structuring of the resources and the general clarity of the site (Questions 5 and 6). Generally most students responded positively to a general question asking them if they were clear about the aims and objectives of the site (Question 7). They were similarly positive about the materials presented but noted that they were given insufficient time to explore links contained within the site - generally because of the slow speed of the Internet connection (Questions 8 and 9). Overall the students were satisfied that they had benefited from the tutorial (though noting the significant role of the supervisor in helping them to achieve success) and almost uniformly they saw the resource as being useful in the future (Question 10). The main negative factors noted by respondents in question 11 were generally linked to technical issues and the need for more extensive training on using Internet resources. Informal responses made verbally included comments on the need to use computers at all (one mature student) and the importance of assistance from the supervisor.
Evaluation in PhD tutorial sessions
In order to broaden the evaluation's base, a scheduled class for the instruction of PhD candidates in "Using the Internet for Research" was used as a further testing ground. A longer period was available in which to demonstrate the site than in the previous tutorials (3 hours, as against 1 hour previously) and as PhD candidates the attendees could reasonably be expected to take a genuine interest and participate fully. The site had also been redesigned since the previous evaluation but these were largely associated with design issues and should not impinge on providing a comparison between these students and the earlier group.
13 attendees were present, and a total of seven completed questionnaires were received. The evaluation was administered in the same manner as with the previous tutorials, with the exercise portion slightly modified to be more universally applicable to the range of disciplines represented on the day.
Surprisingly the research students showed varying but generally low levels of technical experience. The WWW was entirely alien to most present, and also, more surprisingly, a general lack of familiarity with Research Methods was also evident.
Generally the research students had difficulty in orienting themselves within the site but almost uniformly found the materials on Research Methods useful, and the appearance of the site clear and helpful. They had no problem in recognising the aims and objectives of the site (as one would expect from this particular group).
Lack of time to investigate the resources fully was again noted: time was intrinsically limited, and it was also some users' first experience with the WWW, indeed in one case with IT of any sort. Overall the students felt they benefited significantly from using the materials but once again, the presence of tutors (two this time) undoubtedly enhanced the users' experience and enabled them to get much more out of the session than would otherwise have been possible. It would, however, have been unwise to let a student's first experience of the site, and in many cases the Internet - and in some cases, of computers - be unsupervised.
When asked to comment on negative aspects of using the site again the main problems noted were technical (relating to the speed of internet connection) and related to time constraints.
Evaluation through self study
This stage of evaluation was intended to test students' ability to use ReMOTE independently after a brief introduction. The same exercise and feedback form were provided to the postgraduate Electronic Information Management class, with a two week period allowed for completion. The exercise was made available by WWW download to allow those who did not attend the introductory class and anyone who might have mislaid the exercise to get it easily. The availability of this download was announced by e-mail immediately after the introductory class, and a reminder issued two days before the due date. The course lecturer also issued reminders in class. However, in spite of repeated requests from lecturer and evaluator, no students reported on their use of the site. One student did register interest through the online feedback form, but expressed no opinion of the site itself. This confirmed that purely voluntary feedback cannot be relied upon.
Online feedback and analysis of use
The ReMOTE site includes a request for online feedback asking for user opinion and suggestion. A number of users from RGU, other UK HEIs, the USA and France registered as having used the resource. As one would expect, written feedback was generally very positive.
The server log files reveal that use of the site has grown steadily since June, although it had not been formally launched at the time these statistics were gathered:
June: 158 July: 79 August: 112 September: 232 October: 225 November: 308 December: 317
Feedback from inside and outside the university reveals that the resource has been "discovered" by many at whom it was not explicitly targeted. A brief analysis shows users from all over Scotland, the UK, Europe, the Americas and Asia accessing the site. The most frequent individual site from which ReMOTE is accessed is Loughborough University, from where access to ReMOTE is almost daily.
Indirect Evidence of Use
A more encouraging development was the discovery that all of the postgraduate students had in fact returned to the resource and found it useful. This came to light during marking of assessed exercises in which some students cited ReMOTE as the source of information. Although difficult to assess quantitatively with any precision, ReMOTE evidently proved itself to students as a useful resource in that they returned to it without further encouragement and were able to extract relevant information from it. This may well be evidence of what Draper et al. have termed 'Delayed learning gains' when they state that 'both staff and students often express the view that the important part of learning is not during EIs (Educational Interventions) such as lectures or labs, but at revision time or other self-organised study times'. Arguably this application and transference of learning is perhaps a more valid indication of the success of an educational intervention than any short term gains in knowledge or skills.
Only a very limited amount can be read into the responses from the various groups who took part in testing: the sample size was small and only a "first impression" is usually possible. The formally and informally gathered feedback indicates that the resource is found to be useful, and in several cases assisted students' resource discovery during the course of the tutorial itself. The impression appears again to have been a generally positive one and the evidence of self-motivated use by students outside of any formal tutorial or evaluation process is an encouraging development. It is concluded that while this web resource may prove bemusing or difficult to assimilate in the brief time available in introductory or tutorial sessions, its value increases with longer term self-directed use when there is an actual information need. This could be maximised with further exposure to the resource e.g. through its integration into coursework assignments.
One particular point must be made with reference to the use of technology: particularly in the case of PG/MSc students. The tutorial evaluation sessions occurred at whatever point in the timetable the class was available - in the case of Information Analysis, this was prior to the students' first introduction to the Internet. As such, any evaluation will be hampered by the students' need to get to grips with technology.
Recommendations and issues for further consideration
As noted above, the directory feature in ReMOTE would be enhanced from the point of view of both of the user and maintainer by the integration of a database for storing the entries and a WWW front end which will make the directory searchable. This will enable greater flexibility, ease of use and functionality. The use of databases on the world wide web should be investigated further: it is envisaged that implementation of such a system would be inexpensive, the only major cost being the time which it would take to set up initially.
Secondly, the requirement for consistency throughout the pages points to the requirement for a system of document management: currently, pages are constructed in a simple text editor with HTML encoding done by hand. This means that any change in the overall design of pages must be individually applied to each and every page affected, a repetitive and error prone process. Software which is capable of developing and using templates which can then be filled with appropriate content would eliminate a considerable amount of low-level repetitive work and make changes much more quickly and reliably.
If large suites of web pages are in the future to be maintained with current information, both of the above issues must be considered: it will not be possible to compete with similar providers, nor to maintain extensive lists of up to date information without an automated system of document management.
Staff Input and Course Integration
The major difficulty with ReMOTE has been finding a place for it within the taught course structure. Pressures on the time of the relevant members of staff and their lack of good quality, reliable access to the WWW has limited their input and ability to decide on how best to exploit the resource. This contrasts with the Electrical Engineering experience described below where close lecturer involvement enabled full integration into a taught course.
Electronic and Electrical Engineering Modules
The School of Electronic and Electrical Engineering has been at the forefront of developing educational uses for the Internet in the Robert Gordon University, producing a wide range of materials under the auspices of the UMI (Use of MANs Initiative) and other initiatives. With a number of new course modules to teach in the approaching academic year, it was found desirable to try to use the WWW and the prior lessons learned in developing web based applications in order to assist in the teaching and learning process: to alleviate some of the load on lecturing staff by providing materials which could convey information which would normally require an in-person explanation or demonstration; and to provide a resource for the self-study of Personal and Mobile Communications which would enable supplementary work to be done outside of class time, giving access to basic, essential information, to illustrate through the use of the capabilities of the WWW certain concepts, and to provide access to sources for further reading.
The modules in question: Personal and Mobile Communications and Broadband Networks form an essential part of postgraduate MSc courses in the School of Electronic and Electrical Engineering. While much material already existed for Broadband Networks, the Personal and Mobile Communications Course was essentially undefined at this stage, and the WWW module was to be developed in parallel with the taught course.
In many ways the creation of these units was seen as a more straightforward process - although in terms of work involved in developing the teaching materials they posed a considerably greater workload. As Benyon notes there is still little published material which provides specific guidance on designing teaching materials for the WWW although the existing body of publications on hypertext and hypermedia should in theory provide a solid foundation for this. In order to ensure that the development could be completed within the time constraints imposed by the project an existing framework for unit development was adopted. This was not an ideal solution and ideally the project team would like to have explored and experimented with some of the ideas and issues raised by Mumford, Andrews, Schneidermann and Uys which provide general advice and guidelines. The development approach was based on rapid prototyping and effectively formative evaluation was ongoing throughout the construction of the CAL materials. This obviously was to have a marked impact on evaluation of the packages.
The first module to be tackled was Broadband Networks: much of the content and structure of this was already designed, with the overall design adhering closely to the guidelines for module development drawn up by the School during their involvement with the UMI initiative. Essentially, the bulk of the task was a raw conversion of paper based materials onto the WWW and the incorporation of WWW features which would exploit the medium more effectively e.g. use of colour, graphics, animation.
Much of the material was able to be gathered from texts on the subject, as the lecturer concerned has published extensively in the field. There were therefore few problems in identifying suitable materials or deciding on structure or content. The major requirement was for the identification of alternative methods of representing information which exploited the WWW to its best effect.
Personal and Mobile Communications
The initial expectation was that existing paper based materials were to be used as the basis of the resource. However, due to the departure of the lecturer responsible for teaching the subject, the material had largely to be gathered and selected from a variety of texts available on the subject and from whatever other sources could be found. The outline specification of the module was made by the lecturer and was subsequently adapted throughout the development process in conjunction with the developer.
The work required from the developer was more extensive and in-depth than had first been envisaged, since it now involved a considerable amount of resource selection, acquisition and evaluation in an unfamiliar field. The lack of development guidelines in this area is a serious problem.
This unit was only undertaken by 1 student - an event which could not have been foreseen by teaching staff when the unit was being developed. However, it was felt that it was important nonetheless to attempt to gain some feedback. Evaluation was conducted by interview with the student taking the module, using an LTDI evaluation proforma, and supplementary feedback proforma (also from LTDI). The student proved willing to participate extensively in evaluation and offered in depth observations and opinions on all aspects of the resource. The interview session extended into a valuable 3 way discussion between student, lecturer and interviewer on the merits of the resource and areas for greater emphasis and further development. It was noted that the student copied and took the WWW pages home to study on his own computer, thus negating any design issues dictated by the specification of computers available at the university and communications problems.
Student Overall Impression
The student's overall opinion was that the resource was good, with a few aspects which required sorting. The lecturer and interviewer agreed broadly on this evaluation, and were happy with the module and the recommendations made for its improvement. Some changes were technical, textual or layout alterations which could be effected immediately, others were valid suggestions for extensions in coverage which could be addressed with only a little more research and effort in implementation.
The lecturer's own opinions and impressions were also sought. Apart from the obvious continuous feedback process resulting from his close involvement with the module's development and use, it was decided to formalise these. A number of feedback forms (LTDI proformas) were completed. One such proforma is reproduced here.
Software usability evaluation
Lecturer's responses in bold, underlined text: (1 = strongly agree, 5 = strongly disagree)
I found this package easy to use 1 2 3 4 5 6 I would like to use this package again 1 2 3 4 5 6 I would feel confident using this package with students 1 2 3 4 5 6 I think my students would enjoy using this package 1 2 3 4 5 6 I think my students would benefit from using this package 1 2 3 4 5 6 I think that my students would find this package easy to use 1 2 3 4 5 6 I think that the material in this package would be challenging for my students 1 2 3 4 5 6 I think that this package contains material similar to my course content 1 2 3 4 5 6 I think that this package contains material which is desirable pedagogically 1 2 3 4 5 6 I think that this package could be useful as a tutorial aid 1 2 3 4 5 6 I think that this package could be useful as a lecture aid 1 2 3 4 5 6 I think that this package could be useful as part of students' directed study 1 2 3 4 5 6 I think that this package could be useful for students encountering difficulties with the subject 1 2 3 4 5 6 I think that this package could be useful as a revision aid 1 2 3 4 5 6 I think that this package could be easily be integrated into my course 1 2 3 4 5 6 This package could be useful for more than one course 1 2 3 4 5 6
The lower scores concern issues of ease of use, appropriateness for use in other scenarios and the level of "challenge" for students (score on this final issue concurs exactly with student's own opinion). Given that the materials are designed for a specific scenario and the content has in large part been determined by the needs of the course itself, the only area of actual deficiency in the lecturer's eyes appears to be some awkwardness in using the package: this may be an issue of clarity, given that some screens contain a large amount of information.
Usefulness as a revision aid is borne out by the student's stated response that he found it most useful "as a backup, reference and revision source... at any time and place in the future".
Evaluation of the resource was positive from all sides, with enthusiasm from the student and a commitment from the lecturer to continue updating and using the resource as an integral part of the taught module.
P&MC's development and implementation illustrate the other side of the coin: in contrast to ReMOTE, P& MC was developed with the full involvement of the lecturer concerned with a definite purpose and end product in mind. This has resulted in a deliverable which matches his needs closely and will form an integral part of the course in the future.
One drawback was evident throughout the process: the bulk of development work required the acquisition of suitable materials, and selecting these required an understanding of the subject, which was not a part of the developer's expertise. Consequently, the developer spent much time trying to "become an expert", a time consuming process which meant that both lecturer and developer needed an extensive subject understanding. The division of labour was much less clear than initially envisaged: the intention was to convert existing materials into another format, but proved much more complex, time consuming and demanding. In essence, it was found that the representation of existing materials (or more relevantly, knowledge) in alternative formats requires the conversion to be undertaken by a person or persons with a good understanding of both the subject in hand and the medium in which it is to be represented. The implications are not all negative: in the case of P& MC, the developer acquired much knowledge of a subject which would otherwise have been unavailable. This might be exploited in future collaborative projects, where knowledge or skill transfer is a desired outcome - persons with skills in resource development might be given conversion/development tasks in subjects in which they need to increase their understanding. A useful knowledge transfer between the resource developer and subject expert is possible in such cases, and, it could be argued this provides an indirect measure of the success of the project.
General conclusions and recommendations on Conducting Evaluation Procedures
1. More than one evaluation stage is required. "First impressions" gained in a particular environment under certain constraints, plus a raw analysis of usage statistics do not give a full picture. Only through encouraging the further use of the site through integration with taught courses and examining student attitudes after a period of exposure will an accurate and meaningful evaluation be possible.
2. In-class tutorials are subject to the level of student attendance, reliability of the technology and time restrictions. An alternative action plan must be available to cater for non-attendees and those who wish to complete work at a more leisurely pace. This plan must be specified and made known to the class (present and not present).
3. A deadline for post-class completion must be set and any materials required on the day must be announced and available to the students to pick up if they miss the class or lose the materials (e.g. downloadable from WWW, or better still, mailed personally to each student).
4. Testing should take account of all the scenarios in which the resource is to be used e.g. in class and self-study. What may be achievable in class may be more difficult for independent learners. Advance consideration of this is required e.g. class announcements, reminders, integration into homework etc.
5. Where user performance is tested as a means of evaluating a resource, careful thought must be given to the test itself to ensure that it does not contribute to user confusion: a requirement for plain language with adequate signposting which explains its purpose clearly. Ideally we should be using a student's reflection on his/her performance as a critical measure and build evaluative systems which take this into account.
6. Such developments should be introduced as an integral part of the course, taking account of existing schedules and any prerequisite learning e.g. introduction to the WWW should occur before student evaluation of, or introduction to, a WWW resource.
7. Evaluations are subject to student participation: it is often difficult to enforce the filling in of feedback sheets, and this will only be done if the student feels he/she has good reason - and the time - to do so. We need to look at mechanisms to get critical evaluation of an assured level of quality.
There are ongoing problems in accessing the materials, primarily server availability. There must be some consideration given to solving the technical problems which inevitably arise during WWW use. In a tutorial scenario, services must be reliably available at the time allocated; in a self-study scenario, access may be required at any time and use must not be discouraged by loss of service. Communication between users and those responsible for server administration is essential.
The projects described above do not claim to have 'revolutionised teaching'. They have, however, introduced technology to staff and students and explored some of the issues of integrating that technology to provide sound learning outcomes. The project has given us considerable experience of how to produce materials efficiently but the real challenge which remains is how we can use such resources effectively.
Accessing the sites
At the time of writing the sites are in their initial release states: they have been announced to, and are in use by staff and students within RGU. There are some access restrictions placed on certain materials which are for use within RGU only, in order to ensure copyright compliance.
- The ReMOTE site can be found at URL: http://jura2.eee.rgu.ac.uk/dsk5/research
- The Personal and Mobile Communications site can be found at URL: http://jura2.eee.rgu.ac.uk/dsk5/eee/mobile%20comms/pmc1.html
- The Broadband Networks site can be found at: http://jura2.eee.rgu.ac.uk/dsk5/eee/bbnhome.html
Anderson, F.D. & Newton R., Evaluation of multimedia in higher education. In Proceedings of the first British-Nordic Conference on Information and library studies. Copenhagen, 24 May 1995, pp181-194
Andrews. K., Applying hypermedia research to the World Wide Web. URL : http://hyperg.iicm.tu-graz.ac.at/apphrweb/
Benyon, D., Stone, D. & Woodroffe,M. (1997) Experience with developing multimedia courseware for the world wide web: the need for better tools and clear pedagogy. International Journal of Human Computer Studies Vol 47 pp 197-218
Denham, B (1997) Teaching research methods to undergraduates. Journalism and mass communication educator Vol 51 (4) pp 54-62
Cozby, P.C. (1997) Methods in behavioural research 6th edition
Draper, S. et al. (1996) Integrative Evaluation: an emerging role for classroom studies of CAL. Computers and Education Vol 26 No.1/3 pp.17-32 Available at: http://www/psy.gla.ac.uk/-steve/IE.html
Mumford, A., The world wide web - a strategic tool for UK higher education. In J.Darby & J. Martin (eds.)Using the Internet for Teaching. Active Learning. Oxford: CTISS 1995
Newton, R., Sutton, A. & McConnell M.(1998) Information Skills for open learning: a public library initiative. Library review January
Newton, R., Marcella R. & Middleton, I. (1998) Netlearning: creation of an online directory of Internet learning resources. British Journal of Educational Technology Vol 29, No.2, pp173-176
Pachnowski, L.M. et al. Immediate Data: The World Wide Web as a resource for teaching research methods URL: http://junior.apk.net/jurczyk/eera.html
Place,R. et al (1996) Educator's Internet Yellow Pages. Prentice Hall, Englewood Cliffs
Schneidermann, B. (1997) Designing information-abundant web sites: issues and recommendations. International Journal of Human Computer Studies Vol 47
Stoner, G. (editor) (1996) Implementing Learning Technology, LTDI Edinburgh
Uys, P., Supporting cyber students over the web: the online campus of Wellington Polytechnic. Paper presented in June 1997 at the 18th ICDE World Conference: "The New Learning Environment - a global perspective"
To contact the maintainers - mail (email@example.com)
HTML by Phil Barker
© All rights reserved LTDI.
Last modified: 29 September 1998.