1. Define the circumstances under which observation will occur
This will involve users, often these will be students. Here, it is assumed that they will already be performing a predefined task in an arranged session, such as having been invited to an evaluation session for a developmental piece of software, or be participating in a class using new technology-based materials or techniques. For most evaluations, the main use of observation is as a supplemental technique to complement other standard tools. One exception when observation may be the primary tool is where it is specifically employed to provide information about a known gap in existing provision. The technicalities of setting up software evaluation sessions or courses incorporating technologies are important, but are discussed elsewhere. Hint
2. Identify the key areas of interest
Areas you might consider include:
How do users interact with the interface of a developmental piece of software? Are they using it in the way expected? Are they using it in innovative or unexpected ways? Are they having difficulty? In what areas, or with what tasks?
How do students interact with the new piece of learning technology? Do they discuss the task in question? Do they develop a working strategy for the task or just plunge in? Are the instructions adequate? How long does each section take them? What is their attitude while working? How do they cope with the interface?
3. Plan your observation session
Will the observation be active or passive? Will you be able to interact with the students or users during the session or will this be intrusive? Hint
4. Design the session proformas for the observer
These should be self-explanatory in style, and act as a prompt sheet for the observer during a session. They should include a reminder of all the key points to watch for and provide a straightforward way to quickly mark down all findings of interest plus any additional information. Hint
It may be appropriate to design a number of different proformas for different types of session and circumstances. Or the original version could be adapted.
Provide a mechanism for identifying individual participants across different evaluation instruments, but retain the anonymity of participants unless specific permission is obtained.
5. Decide when your observation sessions are going to take place
What stages of a taught course will provide the most information? For example, you may choose to cover all sessions: the first, middle, and last sessions; or to select several at random. Which are most likely to be key in students' learning or gaining technological familiarity? When will the key material be covered? Missed sessions can benefit from feedback by the tutor. Hint
How often is it necessary to monitor? It is uncommon to need to observe every session of a course, but if there are a limited number of invited evaluation sessions, particularly if these involve different user groups, it may be beneficial to observe all. Hint
6. Arrangements on the day Hint
To achieve the best results:
Make sure the tutor or session organiser explains why an observer is there. Make it clear to the participants that the reason they are being observed is to help improve the course and/or software (whatever is the focus of the evaluation), and that it is not a judgement of their skill or ability. Hint
In the case of courses, make sure participants know that the observation is independent, and will not affect their course grades. Emphasise anonymity for any records, published material, or subsequent reports, unless specific permission is granted by an individual participant. Hint
Where active observations are agreed and are practical, adopt the technique suggested in the "Split Screen Video" section. Remember that students are likely to be more open with another student or an outsider than to another member of staff from their own department.
7. Analysis and feedback
Generally a few key points immediately emerge from an observation exercise. Immediate feedback to the development team or course tutor is often most effective in these cases, as it allows direct action.
More detailed analysis should draw out the key issues for each theme, and these can be compared to the corresponding findings from using the other evaluation instruments.
Some themes will yield little or no information and can be dropped. Others can be adapted for more detailed observation. For an ongoing series of observations using the same group, such as in a course, it is often more useful to immediately adapt proformas before the next session. However, where different groups are being observed under the same circumstances, it is more valuable to keep the exact format for each group, to allow direct comparison.