Data management is the handling of data from acquisition through processing, output and storage. The objective of data management is to ensure that data holdings are of acceptable quality to meet user needs. Effective data management makes data more likely to be:
· relevant to the information needs of users;
· easily accessible to users;
· interpretable to users; and
· coherent with other data sources.
Data is managed more efficiently and effectively if systems and procedures have been set up for the cataloguing and storing of datasets. There are many different methods available to manage data holdings. A data warehouse is one method that can be considered.
Metadata is information about the data. It includes information about concepts, classifications, scope and coverage, sampling methodology, units and measurements, derivations and transformations, estimations and adjustments and processing techniques (e.g. editing) used in the collection.
Metadata assists data users to decide whether a particular dataset is fit for a given purpose. It can also be used as a guide or reference manual, or to evaluate system or procedural performance.
Suggested Elements of Metadata
· name of the organisation or organisational unit that collected or own the data;
· contact details of the organisation and staff responsible for managing the data; and
· further details about the data source (e.g. data website).
Name and Overview
· name of the dataset
· frequency of collection
· scope and coverage
· statistical unit(s) (e.g. persons, households, businesses)
· types of information produced
· geographic, demographic or population levels at which data was collected
· purpose of the collection (e.g. to meet specific policy needs)
· key users and uses of the data.
Back to Top of Page
· Concepts, terms and definitions used in the collection.
· Standards and classifications used in the collection.
· sources of error e.g. processing error, coding error
· sample size, response rate, and sampling error (for survey data)
· any specific areas where careful interpretation is required.
· units of collection e.g. persons, retail establishments, transport etc
· data breakdowns - levels at which the data is to be disaggregated e.g. by state, by industry, by sex etc
· type of statistics, i.e. level or movement, totals, means, medians, proportions or indexes, seasonally adjusted or trend, constant or current prices.
Data source history
· the history of the data source, including time series data available
· major changes in the past to the data collection that impacted on outputs and comparability over time
· comparability with other national and international data sources
· concordances with previous versions of classifications
Data available for dissemination
· forms of data release (e.g. paper publication, internet)
· time lag between collection and release
For additional information on metadata and case studies showing how metadata can add value to your statistical collections, visit the National Statistical Service website www.nss.gov.au.
9.2 ADDITIONAL DOCUMENTATION
In addition to metadata, more general documentation relating to various aspects of data collections should be kept and stored for future reference. For example, documentation covering the reasons why certain decisions were made, minutes from meetings, project proposals, changes to collections, etc can serve as a record for users and producers to ensure that data is interpreted and used correctly.
9.3 DATA DIRECTORY/ CATALOGUE / DICTIONARY
Recording data collections and databases in a directory, catalogue or dictionary makes the retrieval of information easier and promotes the transparency of data holdings. The directory could include features for searching by topic or theme, geographic area and keywords.
9.4 DATA STORAGE
Data storage systems are used as repositories for the data collected in statistical or administrative collections. Simple data collections may be held in a spreadsheet, while more complex data holdings may be kept in a database or data warehouse.
Some key issues to consider in designing a data storage system:
· access controls to the system
· documentation of systems and procedures
· the use of standard classifications across collections
· backup and retrieval policies
· data retention and archiving policies
· management of data storage space
· availability and accessibility of metadata
· naming conventions and cataloguing for identification of file contents.
9.5 DATA WAREHOUSE
A data warehouse is a central integrated repository of data brought together for efficient management, querying and reporting. Outputs can vary from simple tables to highly complex reports.
The benefits in having data stored as part of a data warehouse include providing:
· a platform for manipulation of data
· a basis for extraction and dissemination of statistics
· a place where metadata can be integrated.
For the data collection agency, a data warehouse can make it easier to maintain standards for storage, access, use and dissemination of data. It can also provide a central vehicle for the storage of concepts, definitions and procedures for agencies collections. A warehouse brings efficiencies to data storage by supporting the sharing and re-use of metadata across different collections.
9.6 DATA RETENTION
Agencies that conduct statistical collections should develop a data retention policy for their own needs and to meet the needs of external users.
The objectives of a data retention policy are to ensure that:
The retention and/or destruction of Commonwealth records is governed by the National Archives of Australia (Archives) under the "Archives Act 1983". Section 24 of that Act provides that agencies can only dispose of records (i.e. destroy, transfer custody or ownership, or damage or alter):
· if required by another law
· with the approval of the National Archives of Australia
· as a normal administrative practice (i.e. routine procedures that dispose of records of only ephemeral value - duplicates, telephone messages, compliments slips etc)
· for the purpose of returning records to the custody of the Commonwealth.
National Archives of Australia approves the disposal of records by issuing disposal authorities. These are legal instruments that describe groups of records and state the minimum periods that they should be kept. The Administrative Functions Disposal Authority was developed by National Archives of Australia to cover records that are common to all Commonwealth Agencies. Specific Disposal Authorities for an agency's operational records are developed by agencies in consultation with National Archives of Australia.
Further information on archiving policies of Australian and State/Territory governments can be found at:
National Archives of Australia
State Records Authority of New South Wales
Public Record Office Victoria
Queensland State Archives
State Records of South Australia
State Records Office of Western Australia
Archives Office of Tasmania
Northern Territory Archives Service
ACT -Territory Records Office