Evaluation Cookbook

1. Identify situation to investigate
Confidence logs should be applied to identified situations. Hint. These can be:
 the particular point(s) in the class at which you wish to investigate the students' development - snapshot(s);
 an activity to investigate e.g. a scheduled lab session using a computer simulated experiment - comparative study.

For a comparative study, note that you will need to apply the logs before the identified situation as well as after.

2. Identify key areas of interest
You may be seeking information that is either general or specific.
General - Relating to the knowledge/skills objectives of the activity/course.
Specific - Relating to areas of interest (e.g. where you suspect there may be an area of weakness in the material).
In either case, define these interests before designing the statements for the log.

3. Construct statements
Many of the issues relating to the construction of questions for a questionnaire are equally valid in the context of confidence logs. However, it is important to remember the statements are not questions and should not be constructed as such. Hint

Generally, they should be short, concise and unambiguous. Keep each statement simple; avoid combining areas together in one statement (or using high level statements) - if the student feels very confident in one aspect and not in the other it is very difficult for them to rate their confidence and fairly meaningless for you. Hint

Remember to use language which the students will understand, particularly if you think they may have experience of the material in another context. Also, remember that there is a difference between an understanding of a theoretical and a technical competency so be clear what you are interested in, and communicate this to the student! (See note at end.)

4. Practicalities of application
There are various practical issues to consider. They can be summarised as when, where and how. Hint

When
Comparative - For comparative studies, the when will be before and after a specific activity. Some issues to consider are: Is the activity timetabled? If it isn't, do you know when the students will do the activity? Do all the students work together or are they split into groups? Snapshots - For a snapshot study, the when is related not to a specific activity but to a point in the class. As well as issues similar to those addressed for a comparative study, you should also consider whether or not all the students are expected to reach the specified point at the same time.

Where will you be able to get access to the students? In an existing lecture or lab., or will you need to arrange something?

How will the logs be presented? Part of the how is the introduction/instructions you give to the students. Some people use a short introductory paragraph to explain what is required of the student, others give these instructions verbally, some use both. This is really dependant on the situation but the students should receive some guidance. Other aspects you might consider include: whether the logs are going to be electronic- or paper-based; who, if anyone, will need to be there to administer the logs; and how long will it take to complete the logs? Hint

5. Analysis
There are many ways to visually present the information from logs. Bar charts can be recommended as a simple and informative option.

Comparative - If you have anonymous logs you can look at the spread of confidence in a standard bar chart format for each of the statements. Comparing the bar charts before and after an activity will give you a general indication of any shift.
If you can pair before and after confidence logs, this will allow you to chart changes in confidence which is generally more informative.

Snapshots - By constructing bar charts for each statement (as above) you can gain an overall impression of the confidence of the class at a given moment which can be compared with your expectations.

Note: Theory v Practice.