Statistical collections, like all other projects, do not always go according to plan. Statistical collection projects do suffer from many uncertainties which may arise at any stage in any form. For example, large scale non-response to a survey, loss of vital data from computer malfunction, loss of skilled staff during the collection process can all impact on the collection process or the continuity of the project itself. While it is not always possible to avoid all such risks, it is important to recognise potential risks and develop plans to identify, analyse, avoid, eliminate, minimise or manage adverse impact of risks to the project.
13.1 RISK MANAGEMENT STRATEGY
Risk management is a set of activities concerned with identifying potential risks, analysing their impacts, and devising and implementing responses to ensure the project’s objectives are achieved.
Risk management can be applied at every level. However the effort needs to be commensurate with the associated risk. At some levels the process can be informal process, while at others it needs to be formalised, documented and managed.
Risk management is an essential element of good corporate governance and a management tool to assist strategic and operational planning.
In regard to statistical activities, some potential risks include:
· poorly defined scope
· use of inappropriate methodologies
· data does not meet user needs
· analysis not viewed as objective
· quality control measures not sufficient
· data loss from IT malfunctions
· insufficient staff skills
· project committees and boards do not operate effectively
13.2 RISK MANAGEMENT IN DECISION-MAKING
Risk management should be incorporated into the decision-making process. This is easier if decisions are being made based on reliable data. If the data quality is high greater confidence can be placed in the decision making process.
To facilitate the decision-making process for data, risk management has been subdivided into three steps which are broadly summarised in the diagram below.
Figure 1 – A Risk Management Model
13.2.1 Step 1 - Understanding the Risks
The first step in risk management is to understand the risks, and in using statistical data this involves two key steps: sensitivity analysis, and classifying risks using Quality Assessments.
The purpose of sensitivity analysis is to identify the various levels of risk associated with using a specific data source for a given data need. This is achieved by examining the data against the data quality analysis framework and criteria outlined in Section 2.5 – Data Quality in the Handbook and understanding the potential impact on the underlying decision of using the data.
As a result of this analysis, each characteristic is classified according to the degree of match between the data need and the data source, ranging from 'the data collection significantly falls short of requirements' to 'the data collection significantly exceeds requirements'.
Sensitivity analysis can be as simple as going through a process of asking two key questions:
· How different would the data need to be for me to make a different decision?
I. Improving the data quality; and
II. Making conservative decisions.
Improving the data quality
Improving the data quality primarily deals with looking for opportunities to improve the match between the data need and the data source. There are a range of opportunities to improve the quality of the data source.
1. Apply a data collection to only part of the problem.
This option accepts that the data source is sufficient for answering part of the question, but not all of it. For example, the scope of the data collection might only cover specific States but information is required for all States and Territories.
2. Accessing multiple data sources.
Accessing multiple data sources provides the opportunity to either validate existing data sources or use each data source for its respective strengths and alternative data sources to cover the weaknesses. This may be particularly useful when a data collection has been found to be suitable to apply to only part of the problem. Multiple data sources may also provide information for small area estimates where these are not otherwise available.
3. Deciding more information on the data collection is required.
Risks that are identified in the Quality Assessment with the rating of 'There is insufficient information to judge the suitability of this characteristic' can be mitigated by undertaking further research to provide sufficient information to assess the suitability of the characteristic in question. These risks can then be adequately assessed and identified.
4. Modifying an existing data source
Any decision to modify the data collection will need to assess the cost of implementing changes against the benefits. The Quality Assessment should provide a key indication of the type and extent of modification required. The data characteristic measured against the Quality Assessment, as set out below, will identify the issues that need to be answered.
· The data collection significantly falls short of requirements: the data collection may require a significant structural change to better meet data need requirements. For example, the scope of the data collection might be changed to include a broader geographical scope or the sample size might be increased to meet specific user requirements.
· The data collection is sufficient with some areas of reservations: the assessment has already identified that the data collection is already nearly sufficient to meet needs so this may indicate a smaller structural change to the collection or minor alterations to collection processing or procedures.
· The data collection is sufficient for the requirements: while the data collection may meet the specific needs of the user, the user may have identified some potential areas for improvement, which may involve only marginal additional costs or allow for the data to be used more effectively across a wider range of purposes.
· The data collection significantly exceeds requirements: this rating may indicate that some savings might be achieved by reducing what is being offered. For example, the amount of editing might be reduced or the sample size might be reduced.
· There is insufficient information to judge the suitability of this characteristic: this reflects a need to improve the level of metadata available to provide a better mechanism to assess the quality of the collection and provide a suitable assessment.
5. Deciding a new data source is required
It may be decided that the risks associated with existing data sources are too high and cannot be sufficiently mitigated. In this case, it might be necessary to develop a new data source. The same process of assessing the proposed data source against the specified data need using the data quality framework should be followed (See Section 2.5 – Data Quality in the Handbook for ABS’ data quality framework).
While the above options are available to enhance the overall quality of any given data source, resource limitations may mean that compromises need to be made to achieve an 'affordable level of quality'. However, in making these compromises, two issues need to be considered:
· Where compromises should be made?
· Once compromises have been made, will the data still meet data requirements sufficiently?
Decision Making Issues
Having identified the risks, it is important that the underlying decision takes these risks into account. In other words, these decisions should take into account the quality of the data as well as the value of the data.
It is difficult to provide specific options, as they are dependent on the underlying decision and the corresponding areas of risk. For example, a decision may need to be made on whether an allocated budget will support the introduction of a localised program to support unemployed persons. Rather than initiating the complete program, more conservative decisions might include implementing localised trials, restricting eligibility criteria for program assistance (e.g. based on age or length of unemployment), delaying a decision pending more information or even deciding to use the money to expand existing programs.
13.2.3 Step 3 - Plan for the Unexpected
Form Contingency Plans
In Understanding the Risks, Quality Assessment was used to classify the risks. The areas which were identified as higher risks are the same areas where contingency plans are required (unless the risks were later mitigated sufficiently).
Contingency plans are simply strategies of what to do if certain risks eventuate. For example, a low response rate for a survey generates a risk that the survey results are significantly influenced by non-response bias. As a result, inappropriate decisions might be made on the basis of the biased results. A contingency plan will have answers in place to solve problems associated with a higher than expected degree of non-response bias.
These contingency plans should relate to the underlying decision. Using the unemployment program example referenced earlier, a contingency plan might be to reduce or drop the program if uptake proves to be much lower than expected. Similarly, the program may be moved to a different location or eligibility criteria expanded. These plans can be very similar to those considered early at the risk mitigation stage. However, instead of mitigating the risk immediately through making a more conservative decision, the decision might not fully take into account the associated risks. Rather, the risk mitigation option would only be implemented if further information suggested that the risks had been realised.
Monitor and React
Monitoring is a key part of planning for the unexpected. Having formed contingency plans, it is important that the information is available which will trigger the implementation of them.
While it may be possible to continue to monitor data from a regular survey or an ongoing administrative data system, this will not always be possible. As such, other ways to monitor the impact on the underlying decision should also be considered. For example, monitoring budgets would assist in avoiding. Similarly, a decision to run specialised training programs for the unemployed would benefit from monitoring both participation levels in training programs, participant comments on the training and overall levels of unemployment.