Evaluation Cookbook

1. Define your sample
State your objectives clearly.

Decide whether data can be collected on everyone involved (a census) or whether you are restricted to collecting information from a smaller group (a survey).

For surveys:
 define clearly your population of interest;
 think carefully how you can obtain a 'representative' sample (it is very easy to introduce bias into a survey if the sample is not chosen carefully) and how big your sample should be (the size will affect the reliability of the results).

2. Clarify your objectives
Be very clear about your objectives and this will make it easier to write down precisely what information you require.

3. Develop your questionnaire
a. Construct statements
Think about whether you want to use open-ended questions or closed questions or a mixture of the two.

Open-ended questions allow the respondent to express an opinion on some matter and could be a single word or a long statement; closed questions require a specific answer e.g. a box to be ticked, items to be ranked, etc. Hint

Likert-type scales are useful for assessing a respondent's attitude to a statement, e.g. strongly agree, agree, neutral, disagree, strongly disagree. In order to avoid respondents agreeing with a set of statements without thinking clearly about each one, vary positive statements (e.g. I found the package easy to use)with negative statements (e.g. the screens were too cluttered).

b. Design the questionnaire: Hint
 Include an introduction explaining the purpose of the questionnaire
 Give clear instructions on how to fill it in
 Keep questions simple and unambiguous
 Use a consistent style (not a mixture of ticking boxes, circling answers, etc.)
 Arrange questions in a logical order putting sensitive issues towards the end
 Include a thank you at the end
 Including codes for responses can be useful if the analysis is to be carried out using a computer package
Hint Hint

4. Practicalities of application
Before using the questionnaire in earnest, pilot it with 4 or 5 respondents to uncover any potential problems in the questionnaire and in the analysis.

Questionnaires can be:
 handed out and completed there and then (ensuring a good response); Hint
 handed out and collected in at some later date (giving respondents more time to consider their answers);
 posted out (generally results in a poor response rate);
 administered in person or by 'phone (takes much longer but acheives a good response rate and allows clarification).

5. Analysis
Check answers for values outwith expected ranges and inconsistencies. Hint

Check the answers to open-ended questions to see if they have common responses that can be coded up in some way.

There are many ways to usefully present the information from questionnaires visually - bar charts, histograms, scatterplots, etc. Most statistical analysis packages allow these to be produced easily. Spreadsheets can also be used to analyse questionnaire data. However, if a very large survey is being undertaken, it may be worth considering a specialist package such as SphinxSurvey, distributed by Sage Publications (http://www.sagepub.co.uk).

example