Basic Survey Design
| Chapter 02 | The Set-up Stage of a Survey |
The set-up stage is a very important part of any research, yet its value is often underestimated. Most research projects (and therefore, most surveys) which go badly wrong, do so because of inadequate attention given to setting the project up properly. For this reason, any effort to collect information, should not occur until the issues discussed in this Section are dealt with.
This Section is intended to assist researchers and managers in identifying the key factors in planning the research project, analyse their requirements and select the most appropriate method of collecting the necessary information.
Aims and Objectives
The first and most important step, in the collection of any information, is to establish the research objectives clearly. From this basis, it is then possible to translate these objectives into establishing specific information requirements.
The objectives should define what the survey results will be used for and may include such uses as the basis for decision-making, the allocation of funds, to analyse the outcome of policies or programs (program evaluation) or to determine the direction of future operations. It is highly recommended that the number of objectives be kept to a minimum. With too many, the information gathering exercise can easily become too unwieldy and costly.
The Target Population
Once our uses for the data have been determined, our next requirement is to define the target population. The target population is the group about which we wish to make inferences from the survey data. The target population should be defined in terms of :
- content (eg. all persons),
- units (eg. in households),
- extent (eg. in Australia), and
- time frame (eg. 6 September 1991).
Note that the target population may not be easily accessible due to logistical or financial restraints. For example, in the Census of Population and Housing, it is difficult to obtain data on the homeless or people who are in transit from one place to another during Census day.
When considering the population that is to be studied, an important factor is the units that are to respond to the collection (the reporting units) and the units that are to be selected to access the reporting units (the selection units). An example of the difference between reporting units and selection units is given in Aims and Objectives, where the selection unit is a household and the reporting unit is any responsible adult resident at that household.
Methods of accessing populations are dealt with further in the chapter on 05 Frames & Population .
Conducting Preliminary Research
This step involves becoming familiar with the issues to be researched. By doing so, researchers obtain a stronger focus for their study, as well as some insights on how best to collect the desired information. Preliminary research is done by gathering together relevant reports, information papers, academic journals, books, newspapers and government files. Similarly, talking to people who are familiar with the issues should also be undertaken.
Establishing Data Requirements
The first consideration when clarifying information requirements is to clearly define the data items to be collected. Only with clear definitions of the terms and concepts involved can the researcher be assured that the data will be relevant to the project's aims and objectives. Standard definitions for data items should be investigated as the use of standards allows comparability between the results from the current investigation and previous data.
A common way to ensure all the appropriate data items are collected is to specify, as early as possible in the planning stages of the collection, the detailed tables to be derived. Such specifications not only help clarify the data items to be collected, but the levels or subgroups for which the data is required. Assuming an appropriate collection methodology, defining the levels of interest beforehand will help to ensure an adequate sample size is chosen to achieve the desired levels of accuracy.
The definition of concepts also plays a part in identifying the population. The survey objectives define the target population, but the specific data items and output requirements determine the make-up of subgroups inside the population. These subgroups make up the cells on the tables that have been determined in describing the output requirements.
All data items proposed for the survey should be scrutinised closely to establish whether they are consistent with the aims and objectives of the project. It is vital that extra information is not sought because it is of interest or is related to the issues being examined. This may lead to the survey losing its focus and add considerably to the cost and time to do the project.
Ideally the information requirements, whether they be factual data (eg. unemployment or inflation rates) or attitudinal data (eg. opinions on a particular topic), and the proposed methods of analysis should be put down in writing, as this can help to focus the researcher's attention on important considerations affecting the design of the collection.
Financial and Time Constraints and Other Considerations
Apart from the important considerations in survey design outlined above, there are several other aspects that need to be accounted for. The financial and time constraints placed on a survey are often a deciding factor influencing survey design, as well as the size of the survey and the level of accuracy sought. It is therefore important to make both time and financial estimates for each aspect of a survey and monitor them closely.
Other aspects include creating or maintaining the survey frame, designing the sample selection, choosing a data collection method, designing the questionnaire and determining the field and data processing procedures.
Return to top
As described above, the characteristics of our data items often point at the best way to collect our information. How expensive and time-consuming the data are to collect are also often important considerations when choosing whether to run a survey. The availability of similar data from other sources also plays a part, but collating data from many sources may also prove time-consuming and expensive.
Existing Data Sources
The information that is required may already be available from another data source. Research of subject matter documentation, as discussed in Conducting Preliminary Research above, may provide the necessary information on some topics. Some data may also be available from government departments and agencies as a by-product of their administrative information (eg. Registrar of Births, Deaths and Marriages). Previous surveys may also have been carried out in the field of interest. Care must be taken that the recording of the data comes from reliable sources and that the definitions of the data items used closely approximate those currently of interest. This is a cheap and fast way of obtaining data so long as the existing data meet the requirements of the research.
Case Study
In preparing a case study the researcher seeks to collect and analyse as much data about the chosen subject as possible from a relatively small number of cases. Case studies allow greater detail to be gathered for less cost than in most surveys, but the data is less likely to reflect the whole population.
Participatory Observation
Participant observation is a method of data collection whereby the researcher seeks to become a member of the group under study, and records data from within. This method is often used in qualitative studies ie. studies that deal with variables which cannot be readily measured such as peoples perception, group dynamics, etc.
Advantages and Disadvantages
The main advantages of non-survey methodologies is that they provide data, often quite detailed, in-depth data, at a relatively low cost and that they can result in substantial reductions in the time taken to produce results. However, statistical accuracy may be traded off to achieve these gains. Existing data sources may use definitions that do not match the required definitions very well, and the other non-survey methodologies use only very small numbers of subjects in the study and are open to the subjects not being representative of the target population.
This does not mean that non-survey methodologies should be overlooked entirely - it only means that care should be taken when analysing their results. For example, if we chose to look at the effect of overcrowding in our hospitals, we may decide to perform a case study or control group experiment rather than a survey as these methods allow the researcher to collect information for a comparatively modest cost.
Return to top
After considering all the options available to you, should you decide that a survey is the option most suited to your study, your next task will be to decide upon the survey design.
The three objectives of sample surveys are to
- use the data collected from a sample of population units (using the sample statistics calculated),
- to describe a population, i.e. estimate certain population parameters (descriptive statistics), and
- to test statistical hypotheses about the population.
Issues relating to the survey design encompass all the methodological and organisational aspects of the survey to be conducted and involve making decisions about how these will be carried out. In doing this, decisions that are technically desirable need to be balanced off against what is practically feasible, and need to take into account the purposes of the survey, the accuracy required in the results, the resources available (ie financial, timing, staffing, and other resources) and other practical considerations. The aim of survey design is to best meet the aims of the survey within the boundaries of the resources that are available. Survey design issues to be considered include :
- the survey objectives,
- the population of interest (or target population),
- the reference period for the data,
- Geographic, demographic boundaries,
- the frame and the units,
- the sample design and sample size,
- the data collection method,
- the questionnaire,
- non-sampling errors,
- non-response,
- the field procedures and data processing system, and
- the tabulations and analyses required.
Some of these points have been dealt with in this Chapter. The rest are dealt with in the remaining Chapters.
Return to top
Survey management involves organising and controlling each aspect of the survey. The staff responsible for the survey need to be skilled and experienced in order to avoid the many pitfalls that confront the researcher new to the field. The operational aspects of a survey require thorough planning and efficient management of financial and human resources. In the course of addressing survey management issues, a researcher may find that the survey needs to be modified before it can be conducted feasibly. Since survey management is an integral part of a survey, sufficient resources should be allocated to it to ensure that the survey runs smoothly and produces reliable and timely results.
Advertising
There are several parts of the survey where advertising is necessary. These include advertising for staff which can be done through the daily newspapers or through Centrelink and pre-survey publicity to potential respondents which can be done through an introductory letter sent to respondents. After the survey, the availability of survey results can be advertised through the media or by contacting interested parties. Publicity of the implementation of survey results can also raise the usefulness of the study.
When advertising for staff, you should specify what the job is, subject of the survey, amount of interviewing work, need for prior experience if required, rate of pay, number of days interviewer is required for, whether evening/weekend work is involved and area where interviews will be conducted.
Finance
The financial constraints placed on a survey is often a deciding factor influencing the survey. It is therefore important to make financial estimates for each aspect of a survey and to monitor spending (and therefore the progress of the survey) closely.
Adequate resources should be allocated to survey management as it plays a central role in the conduct of a survey, particularly the most labour intensive part. Financial estimates for other aspects of a survey generally fall into three categories:
- overheads,
- salaries, and
- survey and processing costs.
Recruitment
Regardless of particular collection methods used, personnel need to be recruited and trained to collect, despatch and process the survey questionnaires as well as interviewing respondents. When recruiting staff, it is necessary that they have the basic skills and attributes required to do the work, and in the case of a national survey, they must be in or near the areas where they are expected to work. The number of staff that you need to recruit depends on the size of the sample, the length and complexity of questionnaire, the timetable and the availability of funds. Staffing levels may vary during the research period depending on staff retention rates or whether there are peak periods or low periods.
Training
The adequacy of training given to interviewers and processing staff has a strong influence on the quality of results obtained from the survey. Thorough training of interviewers is important because they have a wide range of tasks to perform and are the main point of contact between the respondents and the researcher. Comprehensive training of office staff should enable them to process the survey questionnaires as accurately and as quickly as possible. Training can be provided in the form of:
- manuals,
- formal training courses, and
- "on the job" training.
Interviewer Training
Training for interviewers includes the purpose of the survey; the scope and coverage of the survey; a general outline of the adopted sampling approach; the format of the questionnaire; recording of the responses; correct interviewing techniques; techniques for avoiding or reducing non-response and maintaining respondent co-operation; field practice; quality control; editing; planning the workload; and administrative arrangements.
Training for Processing Staff
For processing staff the training should cover the purpose of the survey; the scope and coverage of the survey; the questionnaire; the way the responses are recorded; coding; editing; key-punching instructions if the survey responses are to be punched onto a computer database; quality control; and administrative arrangements.
Return to top
