Basic Survey Design
Introduction
The following discussion will give a brief introduction to some basic terms and ideas in sampling and an outline of sample designs commonly used. The main focus of the discussion will be on determining an appropriate sampling method. Return to top Probability and NonProbability If the probability of selection for each unit is unknown, or cannot be calculated, the sample is called a nonprobability sample. Nonprobability samples are often less expensive, easier to run and don't require a frame. A probability sample is one in which every unit of the population has a known nonzero probability of selection and is randomly selected. Choosing Between Probability and NonProbability Samples The choice between using a probability or a nonprobability approach to sampling depends on a variety of factors:
Probability sampling is normally preferred when conducting major surveys, especially when a population frame is available ensuring that we are able to select and contact each unit in the (frame) population. However, where time and financial constraints make probability sampling infeasible, or where knowing the level of accuracy in the results is not an important consideration, nonprobability samples do have a role to play since they are inexpensive, easy to run and no frame is required. For this reason, when conducting qualitative (investigative), rather than quantitative research, nonprobability samples & techniques such as case studies are generally superior to probability samples & quantitative estimation. Nonprobability sampling can also be useful when pilot testing surveys. If a nonprobability sample is carried out carefully, then the bias in the results can be reduced. Note that with nonprobability methods it is dangerous to make inferences about the whole population. Quota sampling may be appropriate when response rates are expected to be low. True probability sampling would be more expensive and may require top up units to be selected. If quota sampling is used, selection of units should be as random as possible and care should be taken to avoid introducing a bias. Unlike certain nonprobability samples, probability sampling involves a random selection of units. This allows us to quantify the standard error of estimates and hence allow confidence intervals to be formed and hypotheses to be formally tested. The main disadvantages with probability sampling involve cost, such as the costs involved with frame maintenance and surveying units which are difficult to contact. Return to top Simple random sampling (SRS) is a probability selection scheme where each unit in the population is given an equal probability of selection, and thus every possible sample of a given size has the same probability of being selected. One possible method of selecting a simple random sample is to number each unit on the sampling frame sequentially, and make the selections by generating "selection numbers" from a random number table or, from some form of random number generator. We have discussed methods of drawing simple random samples. Systematic sampling provides a simple method of selecting the sample when the sampling frame exists in the form of an explicit list. Where the frame contains auxiliary information then the units in the frame are ordered with respect to that auxiliary data (eg employment size of a business). A fixed interval (referred to as the skip) is then used to select units from the sampling frame. Systematic sampling is best explained by describing how the sample selections are made.
The value of k is usually not an integer. In this case we either
Example 7.1: Calculating the Skip Interval Say that we wanted to take a systematic sample of size 5 from a population of 37 units. The sample size does not divide evenly into the population. The two options for coping with this are discussed below. Order the population units in some way and number them from 1 to 37.
n = 5 k = 37/5 = 7.4 k = 7 r = 4 Then the sample units are : 1st unit=4, 2nd unit=4+7=11, 3rd unit=4+14=18, 4th unit=4+21=25 and 5th unit=4+28=32. Therefore, sample = ( 4, 11, 18, 25, 32) k = 7.4 r = 4.2 sample = ( 4.2, 11.6, 19, 26.4, 33.8) = ( 4, 12, 19, 26, 34) Features of Systematic Random Sampling The usefulness of systematic random sampling depends upon the strength of the relationship between the variable of interest and the benchmark variable/s. The more highly correlated they are, the greater the gains in accuracy achieved over simple random sampling. This is because we are ensuring a more representative sample of population units are selected. If there is a strong relationship between the variable of interest and the benchmark variable/s then ordering the list by the variable of interest will yield more accurate results using systematic sampling than simple random sampling. Systematic random sampling using ordered lists ensures a range of units will be selected in the sample. Advantages
Disadvantages
Return to top Stratified sampling is a technique which uses auxiliary information which is referred to as stratification variables to increase the efficiency of a sample design. Stratification variables may be geographical (eg. state, rural/urban) or nongeographical (eg. age, sex, number of employees).
Stratification almost always improves the accuracy of estimates. This is because the population variability can be thought of as having components within strata and between strata. By independently sampling within each stratum we ensure each stratum is appropriately reflected in the sample, so between stratum variability is eliminated and we are left only with the within stratum component. With this factor in mind we see that the most efficient way to stratify is to have strata which are as different from each other as possible (to maximise the variance which is being eliminated) while being internally as homogeneous as possible (to minimise the variance remaining). Practical Considerations When planning a stratified sample, a number of practical considerations should be kept in mind:
Example of Stratification As an example of stratification, if we were interested in the educational background of members of a Science faculty at a University, we could select a sample from the faculty as a whole or select samples independently from each of the departments within the faculty, such as mathematics, physics, chemistry etc. This latter method would ensure that each department was adequately represented (which would not necessarily happen otherwise), and should increase the precision of the overall estimate. If on the other hand, we were interested in the level of education (PhD, Masters, Bachelor) rather than the background we should stratify the faculty by level (Professor, Senior Lecturer, Lecturer) rather than by the department. Using this stratification we are more likely to find uniformity of educational standards within a level rather than an area of work, and we are also more likely to separate the better qualified from the less qualified. Advantages The four main benefits of stratified sampling are:
Disadvantages
Number of Strata There is no rule as to how many strata the population should be divided into. This depends on the population size and homogeneity and the format in which the output is required. If output is required for some subgroups of the population these subgroups must be considered as separate strata. ABS Surveys All surveys conducted by the Australian Bureau of Statistics employ stratification. Household surveys (such as the Monthly Population Survey and the Household Expenditure Survey) use geographic strata. Business surveys use variables such as state and industry strata and use some measure of size (eg employment) to form size strata. Allocation of Sample An important consideration after deciding on the appropriate stratification is the way in which the total sample is to be allocated to each stratum. There are three common methods of calculating the number of units required from each stratum.
Estimation As samples are selected independently from each stratum, estimates are also usually made separately for each stratum, then added to give the overall estimate (eg estimated unemployment for Australia will be the sum of the state unemployment estimates). Similarly, standard errors or variances (measures of sample variability) are calculated for each stratum and then all strata specific variances are added up to obtain the overall variance. The addition of variances is possible because the sample is selected independently from each strata. This overall variance can then be used to calculate an overall standard error. Post Stratification There will be occasions when we may like to stratify by a certain variable, say age or sex, but we cannot because we do not know the age and sex of our population units until we select them. Poststratification is a method used when stratification is not possible before the survey. The stratification variable can then be used after the survey is conducted, to improve the efficiency of estimates or, to obtain estimates corresponding to different categories of that variable (eg. sex) by stratifying the sample as if the benchmark information had been available previously. Return to top Cluster and Multistage Sampling So far we have considered a number of ways which a sample of population units can be selected and population characteristics estimated on the basis of this sample. In this section consideration is given to a sampling scheme where the selection of population units is made by selecting particular groups (or clusters) of such units and then selecting all or some of the population units within selected groups for inclusion in the sample. Cluster sampling involves selecting a sample in a number of stages (usually two). The units in the population are grouped into convenient, usually naturally occurring clusters. These clusters are nonoverlapping, welldefined groups which usually represent geographic areas. At the first stage of selection, a number of clusters are selected. At the second stage, all the units in the chosen clusters are selected to form the sample.
Advantages Cluster sampling involves selecting population units that are "close" together and does not require all the population units to be listed. Cluster sampling has two advantages:
Disadvantages In general, cluster sampling is less accurate than SRS (for samples of the same size) because the sample obtained does not cover the population as evenly as in the case of SRS. However it is often preferred because it is more economical. For example, if we take a simple random sample of 10,000 households across the whole of Australia then we are more likely to cover the population more evenly, but it is more expensive than sampling 50 clusters of 200 households. Return to top
Multistage sampling involves selecting a sample in at least two stages. At the first stage, large groups or clusters of population units are selected. These clusters are designed to contain more units than are required for a final sample.
Uses of Multistage Sampling Multistage sampling is generally used when it is costly or impossible to form a list of all the units in the target population. Typically, a multistage sample gives less precise estimates than a simple random sample of the same size. However, a multistage sample is often more precise than a simple random sample of the same cost, and it is for this reason that the method is employed. Advantages and Disadvantages The advantages and disadvantages of multistage sampling are similar to those for cluster sampling. However, to compensate for the lower accuracy, either the number of clusters selected in the first stage should be relatively large (but this increases the cost of the survey) or the sampling fraction for later stages should be high (ie a large percentage of each cluster should be selected). Return to top Sample Size Issues and Determination An important aspect of sample design is deciding upon the sample size given the objectives and constraints that exist. Since every survey is different there are no fixed rules for determining sample size. However, factors to be considered include
Once these issues have been addressed, you are in a better position to decide on the size of the sample. Variability The more variable the population is, the larger the sample required to achieve specific levels of accuracy. However, actual population variability is generally not known in advance; information from a previous survey or a pilot test may be used to give an indication of the variability of the population. When the characteristic being measured is comparatively rare, a larger sample size will be required to ensure that sufficient units having that characteristic are included in the sample. Population Size An aspect that affects the sample size required is the population size. When the population size is small, it needs to be considered carefully in determining the sample size, but when the population size is large it has little effect on the sample size. Gains in precision from increasing the sample size are by no means proportional to population size. Resources and Accuracy As discussed earlier, the estimates are obtained from a sample rather than a census, therefore the estimates are different to the true population value. A measure of the accuracy of the estimate is the standard error. A large sample is more likely to have a smaller standard error or greater accuracy than a small sample. When planning a survey, you might wish to minimise the size of the standard error to maximise the accuracy of the estimates. This can be done by choosing as large a sample as resources permit. Alternatively, you might specify the size of the standard error to be achieved and choose a sample size designed to achieve that. In some cases it will cost too much to take the sample size required to achieve a certain level of accuracy. Decisions then need to be made on whether to relax the accuracy levels, reduce data requirements, increase the budget or reduce the cost of other areas in the survey process. Level of Detail Required If we divide the population into subgroups (strata) and we are choosing a sample from each of these strata then a sufficient sample size is required in each of the subgroups to ensure reliable estimates at this level. The overall sample size would be equal to the sum of the sample sizes for the subgroups. A good approach is to draw a blank table that shows all characteristics to be crossclassified. The more cells there are in the table, the larger the sample size needed to ensure reliable estimates. Likely level of Nonresponse Nonresponse can cause problems for the researcher in two ways. The higher the nonresponse the larger the standard errors will be for a fixed initial sample size. This can be compensated for by assigning a larger sample size based on an expected response rate, or by using quota sampling. The second problem with nonrespondents is that the characteristics of nonrespondents may differ markedly from those of respondents. The survey results will still be biased even with an increase in sample size (ie. increasing the sample size will have no effect on the nonresponse bias). The lower the response rate, the less representative the final sample will be of the total population, and the bigger the bias of sample estimates. Nonresponse bias can sometimes be reduced by poststratification as well as through intensive follow up of nonrespondents, particularly in strata with poor response rates. Sampling Method Many surveys involve complex sampling and estimation procedures. An example of this is a multistage design. A multistage design can often lead to higher variance in resulting estimates than might be achieved by a simple random sample design. If, then, the same degree of precision is desired, it is necessary to inflate the sample size to take into account the fact that simple random sampling is not being used. Relative importance of the variables of interest Generally, surveys are used to collect a range of data on a number of variables of interest. A sample size that will result in sufficiently precise information for one variable may not result in sufficiently precise information for another variable. It is not normally feasible to select a sample that is large enough to cover all variables to the desired level of precision. In practice therefore, the relative importance of the variables of interest are considered, priorities are set and the appropriate sample size determined accordingly. Calculation of sample size When determining an appropriate sample size, we take as a general rule, the more variable a population is, the larger the sample required in order to achieve specific levels of accuracy in survey estimates. However, actual population variability is not known and must be estimated using information from a previous survey or a pilot test. It is worthwhile keeping in mind that the gains in precision of estimates are not directly proportional to increases in sample size (i.e doubling the sample size will not halve the standard error, generally the sample has to be increased by a factor of 4 to halve the SE). In practice, cost is a major consideration. Many surveys opt to maximise the accuracy of population estimates by choosing as large a sample as resources permit. In complex surveys, where estimates are required for population subgroups, enough units must be sampled from each subgroup to ensure reliable estimates at these levels. To select a sample in this case, you might specify the size of the standard error to be achieved within each subgroup and choose a sample size to produce that level of accuracy. The total sample is then formed by aggregating this sample over the subgroups. Sample size should also take into account the expected level of nonresponse from surveyed units. When the characteristic being measured is comparatively rare, a larger sample size will be required to ensure that sufficient units having that characteristic are included in the sample. Sample Size Formulae If a survey is designed to estimate simple proportions without any crossclassifications in a large population (approximately over 10,000 units), the following formulae can be used to determine the size of the sample: where n = sample size, p = sample proportion, SE(p) = required standard error of the sample proportion However, to be able to use this formula, the proportion being estimated needs to be roughly known from supplementary information or a similar study conducted elsewhere. For example, suppose a survey seeks to estimate the proportion of Richmond residents in favour of Sunday night football at the MCG. The standard error (SE) desired is 0.04, while the proportion (p) in favour of the proposal is thought to be about 0.40. The size of the sample would need to be n=150. If this survey was then completed with a sample size of n=150 and it was a found that the sample proportion (p) in favour of the proposal was 0.8 (not 0.4 as guessed), then the standard error of this sample proportion of 0.8 would be 0.033 not 0.04 as originally planned for. A proportion of 0.5 gives the highest standard error for a fixed sample size or, requires the highest sample size for a fixed standard error, hence p=0.5 is the worst case scenario. It is for this reason that an estimate of p=0.5 is often used when calculating sample sizes when there is no information on the proportion to be estimated. Example 7.3: Gains From Sampling Suppose we wish to take a sample from a population. We have a preliminary estimate of the proportion of the population having the characteristic we are interested in measuring (50%). The level of accuracy we require from our survey is an RSE of 5%. Using the formulae for the sample size in a finite population. For various population sizes, the sample size that we would need is:
The gains from employing sampling are greatest when working with large populations. Return to top
