Evaluation of Food Safety Standards Benchmark - EHO, Year Ending February 2001
Contents
- Overview *** You are here ***
- Quality Framework
- A. General Information
- B. User Requirements, Survey Objectives and Budget
- C. Alternative Sources of Data
- D. Data Outputs
- E. Questionnaire Design and Data Collection
- F. Respondent Load
- G. Population / Frame
- H. Sample Design
- I. Estimation and Analysis
Overview
 | Portfolio: | Health and Aged Care | ||
| Agency: | ANZFA - Australia New Zealand Food Authority | ||
| Contact Person... | Name: | Janis Baines | |
| Email: | Janis.Baines@anzfa.gov.au | ||
| Phone: | 02 6271 2234 (International +61) | | |
Summary
|
This survey is part of an evaluation strategy to assess the effectiveness of the new food safety standards. The current surveys will provide benchmark, baseline data of knowledge and practices of food safety standards. As States and Territories change their laws to incorporate the new standards additional surveys will be undertaken to assess effectiveness of the food standards setting system. Information gained from these surveys will be used to inform future policy decisions and adjustments to the food safety standards setting system. EHO inspection survey This survey will examine the extent to which safe food handling practices are in place. 1. Survey topics: The telephone and EHO inspector surveys cover the same topics and have been chosen to address the key areas of the new food safety standards. Topics in each are cover risk of contamination and correct handling procedures. The areas of interest include: Receiving food (cold and hot) Storing food (dry, cold and hot) Preparing and processing of food Cooking and/or reheating food Cooling cooked foods Display of food (buffets, salad bars, counter) Transport of food Recording of food recall Washing and sanitising food preparation / processing materials Washing and sanitising employee hands Handling staff illness and food contact 2. Organisation comparison: In order to correctly categorise the food businesses surveyed we have incorporated a series of questions to assess the type of food business and risk of contamination. Classification includes the type of food handled, type of food preparation (eg manufacturing, preparation and processing) and whether the business regularly serves people at greater risk of food borne illness (children less than 5 years, pregnant women and elderly or sick persons). 3. Interviewee involvement in food preparation: Survey to be conducted with the proprietor or primary manager involved in food preparation if this is not the proprietor. |
Return to top
Reference Period & Frequency
| Reference Period: | Year Ending February 2001 |
| Frequency: | Once only |
| | |
Return to top
Response Rates
| | | As planned | Actual |
| Number of businesses... | | ||
| | | | |
|
| | 3100 | 1950 |
|
| | 1550 | 1410 |
|
| | 620 | 499 |
| | | | |
| Average time to complete questionnaire: | | 60 mins | 54 mins |
| Statistics... | | | |
| Response rate: | | 40% | 35% |
Return to top
Clearance Information
| Survey reviewed by | |
| Stat. Clearing House: | Year Ending February 2001 |
| Status: | Approved |
| Approval period: | Year ending February 2001 |
| For additional clearance information, please contact the Statistical Clearing House by phone, fax, post or email. | |
Return to top
|
Evaluation of Food Safety Standards Benchmark - EHO, |
 |
 |
|||
 |
 |
|
|||
 |
 |
 |
 |
|
|
Quality Framework
This data quality framework has been published internationally (Brackstone G., Managing Data Quality in a Statistical Agency, (1999) Survey Methodology, Vol. 25, no. 2, Statistics Canada) and has been recommended by the ANAO as 'better practice' in specifying performance measures (ATO Performance Reporting under the Outcomes and Outputs Framework, Australian Taxation Office, Audit Report No.46 2000-01, pp63-64.) on advice from the ABS Statistical Consultancy Unit.
Return to Survey Contents
Relevance
The relevance of statistical information reflects the degree to which it meets the needs of the clients. It is usually described in terms of key user needs, key concepts and classifications used, the scope of the collection and the reference period. |
Why do you need to conduct a survey? (B1)
- Overall: A new approach to food safety is being implemented by ANZFA. Consistent food safety standards are being adopted nationally beginning February 24, 2001.
This survey is part of an evaluation strategy to assess the effectiveness of the new food safety standards. The current surveys will provide benchmark, baseline data of knowledge and practices of food safety standards. As States and Territories change their laws to incorporate the new standards additional surveys will be undertaken to assess effectiveness of the food standards setting system.
Information gained from these surveys will be used to inform future policy decisions and adjustments to the food safety standards setting system.
EHO inspection survey
This survey will examine the extent to which safe food handling practices are in place.
- 1. Survey topics: The telephone and EHO inspector surveys cover the same topics and have been chosen to address the key areas of the new food safety standards. Topics in each are cover risk of contamination and correct handling procedures. The areas of interest include:
Receiving food (cold and hot)
Storing food (dry, cold and hot)
Preparing and processing of food
Cooking and/or reheating food
Cooling cooked foods
Display of food (buffets, salad bars, counter)
Transport of food
Recording of food recall
Washing and sanitising food preparation / processing materials
Washing and sanitising employee hands
Handling staff illness and food contact
2. Organisation comparison: In order to correctly categorise the food businesses surveyed we have incorporated a series of questions to assess the type of food business and risk of contamination. Classification includes the type of food handled, type of food preparation (eg manufacturing, preparation and processing) and whether the business regularly serves people at greater risk of food borne illness (children less than 5 years, pregnant women and elderly or sick persons).
3. Interviewee involvement in food preparation: Survey to be conducted with the proprietor or primary manager involved in food preparation if this is not the proprietor.
- Food business classification: Businesses will be classified according to risk of food borne illness contamination (high, medium and low) based on standard risk classification questions incorporated into the surveys. Questions will be analysed within each risk group and results will also be compared across groups.
The two surveys will be cross analysed where possible to measure level of knowledge against practices in place for businesses of varying risk.
All survey questions will be analysed by the following:
Level of risk of food borne illness
Business size (small or not small)
Business type (supplies ready to eat food, food processing / manufacturing, food service / retail business, serves high risk consumers)
State / Territory
Area (Capital city, Ex-capital city)
- We anticipate only simple cross-tabulations and percentages will be utilised.
Statistical significance testing will be performed (probably Z score) to measure differences between states or business risk classification.
- Standard risk classification (based on food and business type) and as currently implemented by ANZFA will be utilised.
Data will be collected by LGAs and will be able to be analysed by regions. The small number of surveys conducted in each LGA will not allow LGA comparisons.
Where standards are similar we have attempted to use comparable questions in order to measure results against an international survey.
- Food based businesses across all Australian states / Territories including manufacturers, food retailers, food service establishments and hospitals. Businesses that handle only pre-packaged, long life products such as soft drinks will be excluded.
- EHO inspection survey: The sample frame will be taken from a list of food businesses within a sample of 62 Local Government Areas (LGAs).
- Proprietor of food business or, if the proprietor is not involved in overseeing food handling practices we will target the senior manager who is directly involved in overseeing food handling practices.
- EHO inspection survey chosen to assess safe food handling in practice. EHOs are the food safety inspectors for LGAs and have in-depth knowledge about food safety regulations. They are therefore the ideal persons for conducting this survey with minimal impact on the business and within an efficient manner. The survey will not be conducted as an official ‘inspection’ and EHOs will be clearly informed that they are conducting a voluntary survey. (One-third of sample)
- Information is sought for only the day and time the survey is conducted, that is, what the respondent knows or
is doing at that time. This will be through the month of March 2001.
Respondents will not be required to gather past records of information
- Repeated in 2002. Frequency yet to be determined and will be influenced by the rate of uptake of new food safety standards across various states/territories.
Return to Quality Framework
Accessibility
The accessibility of statistical information refers to the ease with which it can be referenced by users. It includes the ease with which the existence of information can be ascertained, as well as the suitability of the form or medium through which the information can be accessed. |
How will the results of the survey be explained / presented to the major users? (I8)
- Tables and graphs will be provided with simple per cent scores.
Sample bases are always provided on the analysis tables.
Results reported on will be those that have achieved statistical significance with suitable sample sizes.
Campbell Research & Consulting always offers a separate section in our reports outlining how to read the tables and graphs to avoid inappropriate conclusions being drawn.
Business specific information will not be reported on, it will not be in the data set. No individual business will be identified in the report or results.
- Australia New Zealand Food Authority (ANZFA)
Monitoring and Evaluation Program
Food Safety Program
- Survey Manager at ANZFA
Janis Baines
Senior nutritionist
Monitoring and Evaluation Program ANZFA
Phone: 02 6271 2234
E-mail: janis.baines@anzfa.gov.au
Primary contact at Campbell Research & Consulting (day to day correspondence for SCH approval)
Kristin Diemer
Project Director
Phone: 03 9482 4216
E-mail: kristin.diemer@campbellresearch.com.au
- At then end of the project all contact details are securely destroyed from the contact information sheets provided to interviewers.
After each telephone interview is conducted, and during analysis of the EHO surveys, the personal details are disassociated from the questionnaire.
Return to Quality Framework
Timeliness
The timeliness of statistical information refers to the delay between the reference point to which it pertains and the date on which the information becomes available. |
To what reference peiod(s) does the survey refer? (A6)
- Information is sought for only the day and time the survey is conducted, that is, what the respondent knows or
is doing at that time. This will be through the month of March 2001.
Respondents will not be required to gather past records of information
- Our anticipated timeline is as follows:
| Pilot surveys: | January 15 to January 19 2001 |
| Clearing House approval: | obtained by February 5 2001 |
| EHO inspection surveys printed: | February 8 |
| Data collection commences: | March 6 2001 |
| Data collection completed: | April 5 2001 |
| Preliminary data tables available: | April 28 2001 |
Return to Quality Framework
Accuracy
The accuracy of statistical information is the degree to which the information correctly describes the phenomena it was designed to measure. It includes measures of both sampling and non-sampling error. |
What is the expected total sample size to be approached and how has it been set? (H4)
- 600 EHO surveys. We have included an extra 2 LGAs to allow for incomplete surveys (potential 620 EHO surveys).
- We expect the sample frame to be the most up-to-date available and therefore will reduce sampling error.
Due to random selection procedures we anticipate we will be able to generalise results across business size and risk classification. Generalising at a broader level will be done with caution, explicitly stating limitations of the sample.
- EHO surveys: We expect 20% of contact details to be incorrect and 20% of businesses to be ineligible due to recent previous inspections. We expect a 60% response rate out of the eligible sample population.
These non-contact and refusal units will be replaced with other sample units from within the strata to achieve the desired number of 600 EHO surveys.
- Sufficient sample populations are being obtained to work through business numbers to achieve our target sample per strata.
- We estimate that all businesses will be in scope.
We will exclude businesses who have recently had a formal food safety inspection to avoid undue hardship. Because the survey is voluntary we expect a 40% refusal rate.
- EHO survey:
The first survey that inspectors are required to send back for assessment will identify most potential problems with data collection. Each of the returned surveys will be checked by research staff at Campbell Research and Consulting prior to data entry. Thus, questions can be recoded if unclear.
Data entry processes include double entry of the information and cross-verification between the two data files.
Overall:
Frequencies will be examined by data analysts and project managers to identify potential data processing errors.
- Testing to be undertaken the week of January 15-17 2001
- ANZFA and LGAs produce materials addressing all of the issues contained in the survey. Food businesses are required to comply with regulations matching the questions in the surveys in order to maintain their business. Information regarding all questions should be known by the person responding.
EHOs already perform regular, similar inspections of food businesses of much more detailed regulations. This survey differs only in that the information required is less specific and therefore the inspection should take less time to conduct than a regular inspection. The aim here is to measure extent of implementation of safe food handling practices as broadly defined in "Safe Food Australia", 2nd edition, ANZFA.
- Unknown - to date we have been unable to identify the number of units.
- EHO survey: EHOs across 62 LGAs will be asked to conduct 10 surveys. Adequate sample will be generated to allow for 40% refusal rate, 20% inaccurate information and 20% previous inspections. We have also allowed for sampling in two additional LGAs as we anticipate 20 surveys (up to two LGAs) may not be conducted within the survey period. Staff shortages and in some cases extensive travel distances may prevent conduct of surveys.
- 600 EHO surveys, all of which are expected to be live and in scope.
We are targeting 600 EHO surveys. 620 surveys will be sent out to EHOs to allow for a possible 20 incomplete surveys.
- Random sampling processes have been put in place both for generation of sample population and for target sample achievement.
Return to Quality Framework
Interpretability
The interpretability of statistical information reflects the availability of the supplementary information and metadata necessary to interpret and utilise it appropriately. It includes appropriate presentation of data such that it aids correction interpretation. |
How will the results of the survey be explained / presented to the major users? (I8)
- Tables and graphs will be provided with simple per cent scores.
Sample bases are always provided on the analysis tables.
Results reported on will be those that have achieved statistical significance with suitable sample sizes.
Campbell Research & Consulting always offers a separate section in our reports outlining how to read the tables and graphs to avoid inappropriate conclusions being drawn.
Business specific information will not be reported on, it will not be in the data set. No individual business will be identified in the report or results.
- Simple stratified random sample by State / Territory and area (metro, rural, regional). We will not be conducting a census.
NSW (12 LGAs): EHO = 120 (6 metro, 6 rural/regional) true sample size 600
VIC (12 LGAs): EHO = 120 (6 metro, 6 rural/regional) true sample size 600
QLD (12 LGAs): EHO = 120 (6 metro, 6 rural/regional) true sample size 600
SA (8 LGAs): EHO = 80 (4 metro, 4 rural/regional) true sample size 400
WA (8 LGAs): EHO = 80 (4 metro, 4 rural/regional) true sample size 400
TAS (4 LGAs): EHO = 40 (2 metro, 2 rural/regional) true sample size 200
NT (3 LGAs): EHO = 30 (2 metro, 1 rural/regional) true sample size 150
ACT (1 LGA): EHO = 30 true sample size 150
Total 62 LGAs, 620 EHO true sample size 3100
- Stratification by State / Territory and region (Capital city, Ex-capital city) to enable an even representation and comparison of businesses across all areas of Australia.
- Sufficient sample populations are being obtained to work through business numbers to achieve our target sample per strata.
- Outliers will be identified simply through examination of frequencies of data. Where suspicions arise we will have the data re-checked for data entry problems.
Where necessary outliers may be excluded on a question by question basis.
- Sample is being generated randomly and we anticipate a ‘normal’ representation of business size and risk classification within LGAs.
If we decide to aggregate data, weights will be applied based on available ABS data on the breakdown of industry size and type by state.
We do not anticipate a need to weight the data.
- We expect the sample frame to be the most up-to-date available and therefore will reduce sampling error.
Due to random selection procedures we anticipate we will be able to generalise results across business size and risk classification. Generalising at a broader level will be done with caution, explicitly stating limitations of the sample.
- We anticipate only simple cross-tabulations and percentages will be utilised.
Statistical significance testing will be performed (probably Z score) to measure differences between states or business risk classification.
Return to Quality Framework
Coherence
The coherence of statistical information reflects the degree to which it can be successfully brought together with other statistical information within a broad analytical framework and over time. The use of concepts, classifications and target populations promotes coherence, as does the use of common methodology across surveys. |
Is the survey a new survey or a cycle of an existing repeated survey? (A7)
- New survey – never previously conducted
- Repeated in 2002. Frequency yet to be determined and will be influenced by the rate of uptake of new food safety standards across various states/territories.
- Electronic yellow pages is continually updated.
- Definitions utilised are taken from ANZFA inspection forms which food businesses and EHO inspectors are already familiar with. Data item definitions include: Classification of business and food type into risk categories, as well as definition of ‘small business’.
Where standards are similar we have attempted to use comparable questions in order to measure results against an international survey.
- To the best of our knowledge, based on extensive and lengthy work in the area, there have been no other surveys similar to this conducted in Australia as the food safety regulations have differed between the States and Territories. This is the first attempt to standardise the regulations and thus conduct a national benchmark survey.
A survey similar to the proposed EHO inspection survey was conducted by the Food and Drug Administration of the United States and reported on in 1996. That survey report is referred to as the FDA Retail Food Program Database of Foodborne Illness Risk Factors.
Where standards are similar we have attempted to use comparable questions in order to measure results against an international survey.
- Results can not be generalised to Australia due to variances in standards and regulations
- Standard risk classification (based on food and business type) and as currently implemented by ANZFA will be utilised.
Data will be collected by LGAs and will be able to be analysed by regions. The small number of surveys conducted in each LGA will not allow LGA comparisons.
Where standards are similar we have attempted to use comparable questions in order to measure results against an international survey.
Not applicable
Return to Quality Framework
