Evaluation Cookbook

Where and when was the study carried out?
This research was undertaken at Manchester Metropolitan University between 1994 and 1996. The course unit observed was part of the BA in Information Technology and Society Degree. The course "Technology in Communications" was a second year option taught on-line using the FirstClass computer conferencing system. The approach was to deliver the course 'online' as much as possible. The course unit is still being run in a slightly modified form, but it is now likely to migrate to the FirstClass intranet server which will provide an internet gateway.

What were the aims and objectives of the study?
The evaluation was 'illuminative' in that the question asked was 'what happened when a course with an explicitly collaborative aim was taught using a conferencing system as the means of delivery?'. An ethnographic methodology was employed to generate an adequate description of 'just what' happened when university education was transposed from a traditional setting into the new technology.

What Methods are Relevant?
 Ethnography and
 Supplemental Observation

How many students were involved in the study?
Manchester Metropolitan University (MMU) is situated close to Manchester City centre. The University recruits students from a wide range of academic and social backgrounds and has a large number of home students from Greater Manchester and the surrounding region. The course ran in 1994/5 with 8 registered students. The students were mainly native English speakers five of whom were male, and three were female. In the second year, the registration included a large number of students whose first language was not English. The total registration for the course was 20 students, 7 female and 13 male. The English language competence and experience of this group varied considerably, especially their written fluency.

What were the main findings?
During the two years of observation, the work that was done was organised in part on-line, though much of the activity was off-line and ratified rather than conducted on the system. Students when working amongst themselves, distributing their work between on and off-line working. Work carried out in the system, without physical contact, was often a case of division of labour. The system would be used to allocate tasks to individuals writing sections and later a named individual would take the sections and combine them into a final text. Often there was little or no consideration given to re-drafting. These submissions were essentially composite, more or less coherent depending upon the skill of the student compiling the final copy.

The expectation for the course unit was that it would involve the development of collaborative skills through group working and the development of peer interaction. Students were observed orienting themselves to course requirements so that the conference transcripts reflected the students and tutors understanding of what they ought to do. It was found that the transcript served as an official document. In many instances, on-line activity was an artificial construct consciously produced as material for assessment.

Interestingly, collaboration was not achieved in the expected form: students did collaborate, they were observed off-line working in groups talking over and around machines. The work would then be presented in the conference as individual messages, a division of labour. However, the expectation, of peer interaction was that students would draft and redraft work on the system and that the process of collaboration would be visible, did not happen.

Reflections on the study
The research cast doubt on the degree of collaboration and the character of the collaboration that took place.

It also questioned the reliability of the use of transcripts for research and assessment purposes. Transcripts were often incomplete and only provided partial records of the activity of the conference. On-line text was often a public display. Therefore, the transcribed record was not so much about what happened but more a reflection of what participants thought ought to be happening. Students and tutors recorded those things they believed were required to fulfil the requirements of the course.

Ethnographic evaluation, by adopting an 'indifferent' stance to the course and observing informally the practice of students' and tutor's on and off-line, provided information not readily accessible by other methods of evaluation. It revealed the detailed way in which students co-operated together in order to achieve what they took to be the course requirements. A more formal evaluation procedure may have concentrated on the 'learning process' and not have considered some of the observed interaction might relevant to education.

Chris Jones
Liverpool John Moores University