Access to Public Hospitals: Measuring Performance

DHS sets out this performance monitoring framework, and its indicators, in annual Statements of Priorities (SOP). The SOPs outline government and hospital policy priorities and expected performance levels. DHS then monitors and assesses the performance of a hospital, and its management, against the framework. The DHS performance monitoring framework also includes 11 service and 5 financial indicators that are outside the scope of the audit.

Many of these indicators contribute to statewide performance measures set for DHS in the State Budget and in DHS’s Victorian Public Hospitals and Mental Health Services: Policy and Funding Guidelines. These measures report on performance at the health system level and are a tool to judge the state’s performance against the departmental objective of timely and accessible human (health) services.

The access indicators also provide individual hospitals, and in aggregated format the overall health system, performance information to the public. DHS’s annual report and the publication, Your Hospitals are the main sources of this information.

Because the access indicators are used to assess hospital performance, allocate funds, and report to government and the public on this core component of hospital care, they need to be fit for purpose.

Access indicators need to be:

The objective of this audit was to determine whether reported access indicators, used by DHS, are relevant, appropriate and fairly represent hospital performance.

The audit examined four hospitals, DHS and Ambulance Victoria to determine whether the indicators:

• reflect the government objective of ‘timely and accessible human (health) services’
• clearly demonstrate performance in providing timely access to hospital care through:
• the use of appropriate targets and benchmarks
• transparent public reporting
• consistently captured
• reported accurately.

1.2 Key findings

1.2.1 Fair representation of access performance

It was not possible to assure that reported performance against the majority of the access indicators fairly represented actual performance.

Emergency access indicators

The hospitals inconsistently interpreted reporting rules, data capture methods were susceptible to error, and the accuracy of some data was impossible to check. This means incorrect data can go undetected. In one hospital, data manipulation had occurred. This hospital has now acted appropriately to better assure accurate reporting.

Poor security of emergency department data, no computer audit logs and failure to audit the Victorian Emergency Minimum Dataset (VEMD), the DHS database used for emergency access indicator reporting, has contributed to this situation. DHS’s implementation of an audit program for the VEMD will help to address this. 

Elective access indicators

The accuracy of these indicators is uncertain because of:

• limitations in the ability to audit them
• variability in how urgency categories are assigned
• evidence of some inappropriate recording of patients as ‘not ready for care’.

At the hospital using the HealthSMART patient manager tool, staff had trouble in reporting accurate elective surgery waiting list data to DHS. They were unable to extract an accurate report from the HealthSMART waiting list module for submission to the Elective Surgery Information System (ESIS). This raises doubt about the ability to obtain accurate elective surgery data as the system rolls out across the state.

1.2.2 Relevance of the access indicator suite

The majority of the access indicators used are relevant as they relate to timeliness of access to hospital care and hospitals are properly accountable for their performance.

Two indicators, however, are not considered relevant. These are:

• the percentage of time spent on bypass
• the total numbers of patients on the elective surgery waiting list.

Bypass occurs when an emergency department is full and the hospital calls for a period where ambulance patients are diverted elsewhere. When one hospital commences bypass the remaining hospitals experience increased ambulance arrivals. The increase can cause subsequent hospitals to call for bypass, creating a bypass cycle.

Bypass for this reason does not reflect hospital performance, but a failure of bypass as a method to manage ambulance arrivals. Performance against the indicator is, therefore, not always within the control of individual hospitals.

Hospitals also have limited control over the number of patients on their elective surgery waiting list. While they can manage patient removals, new registrations on the list represent public demand for elective surgery. While a useful indicator of demand, the indicator is not considered a relevant measure of timely access to hospital care.

It is not productive to devote time and resources to collect and report data for irrelevant indicators.

The access indicators also omit some key patient groups and aspects of timely access to care. These are the timeliness:

• with which a hospital emergency department accepts patients who have arrived by ambulance
• of access to emergency department care for triage category 4 and 5 patients
•  of access to specialist outpatient appointments.

Reflecting these aspects of timely access to care within the performance monitoring framework will improve its balance, better reflecting access along a patient’s journey through the hospital system. 

1.2.3 Appropriateness of access performance reporting

DHS pioneered many of the access indicators and, in 2005, reviewed them. However, it could not provide evidence-based rationales for the selection of indicator targets and benchmarks measuring:

• time spent on bypass
• time taken until admission or discharge from the emergency department
• time spent waiting for elective surgery by urgency category
• the rate of HIPs of surgery.

Without evidence for the particular timelines, targets and benchmarks included in these indicators, it is impossible to make an informed appraisal of whether access to hospital services is good or bad. Given that reporting against access indicators involves significant resources and that performance is judged on them, targets and benchmarks warrant evidence-based rationales.

Open and transparent reporting is core to a fair assessment of performance. Your Hospitals reports the timeliness of access to elective surgery for the period, and over time, openly and transparently. However, the method chosen for presenting performance over time for emergency access indicators does not provide the reader with a clear view of performance trends. Instead, to obtain this information the readers must undertake their own calculations. Such calculations show declining performance for waiting times, for triage categories 2, 3 and 4 patients, since 2003–04, which is not clearly shown in the report.  

Your Hospitals is also limited in that it excludes HIP rates and the indicator measuring waits of more than 24 hours in the emergency department. These indicators report against experiences the public can readily understand and are useful in presenting a comprehensive picture of health system performance.

1.3 Audit conclusions

Access indicators assist in assessing achievement against DHS’s stated objective of timely and accessible human (health) services. Systemic problems with the access indicators, however, limit their usefulness. It is not possible to assure the accuracy of actual results reported by hospitals, and while most of the indicators are relevant, the appropriateness of some of the benchmarks and targets used need further justification.

Inability to provide assurance about the fair representation of access indicator performance stems from the lack of effective quality control regimes at the hospitals, and at DHS. Similarly, limitations to the relevance and appropriateness of aspects of the access indicators reflect the need for further research to validate indicators and greater transparency about how indicators are chosen, developed and reviewed.

Effort towards data quality and validation of indicators is disproportionately low in comparison to the resources and effort put into collecting and reporting access indicators, and to the importance placed on their results. These conditions have opened the way for inappropriate practices such as data manipulation, which undermine the integrity of hospital performance monitoring.

If access indicators are to play a key role in measuring hospital performance, then this situation needs attention to assure a reliable governance and accountability framework for public hospitals in Victoria. We acknowledge that measuring performance in an environment as complex as a hospital is challenging. However, reports of hospital performance against indicators that are meaningful to both hospital staff and the public, and where their levels of accuracy are transparent, are not only achievable, but warranted. Work begun by DHS, such as implementation of an audit program for emergency access data, will assist in meeting this challenge.     

1.4 Recommendations

Fair representation of access performance

The Department of Human Services needs to:

•  review and clarify definitions and rules for reporting of access indicator data (Recommendation 6.1)
• routinely audit both the Victorian Emergency Minimum Dataset (VEMD) and the Elective Surgery Information System (ESIS) for compliance with reporting rules and data accuracy (Recommendation 6.2)
• facilitate implementation of information technology systems that support simple, real-time data capture within hospital emergency departments (Recommendation 6.3)
• review the reporting capability of the iPM waiting list module and facilitate improvements as required (Recommendation 6.4).

Hospitals need to:

• improve security controls on computer systems used for recording VEMD data and utilise audit log systems (Recommendation 6.5)
• internally monitor compliance with policy regarding reporting of access indicators and provide appropriate instruction and training to staff submitting data (Recommendation 6.6)
• conduct internal audits of accuracy of VEMD and ESIS data
  (Recommendation 6.7).

Relevance of the access indicator suite

The Department of Human Services needs to:

• improve the measurement of access to emergency care by ambulance by:
• implementing a ‘destination decision support system’ to manage ambulance arrivals thereby eliminating the need for bypass
• addressing the need to measure hospital performance in both their ability to be available to ambulance arrivals, as well as the timeliness with which they accept patients arrived by ambulance (Recommendation 4.1)
• include indicators and targets for emergency patients in triage categories four and five, reflecting the Australasian College of Emergency Medicine’s (ACEM) policy and National Health and Hospital Reform Commission (NHHRC) recommendation (Recommendation 4.2)
• continue to monitor total numbers of patients on the elective surgery waiting list as a measure of demand, but remove this indicator from the performance monitoring framework (Recommendation 4.3)
• address the need to measure hospital performance in providing access to specialist outpatient appointments (Recommendation 4.4).

Appropriateness of access performance reporting

The Department of Human Services needs to:

• review the use of improvement targets for elective surgery indicators and set specific action plans and timelines for when poor performing hospitals should achieve improved performance (Recommendation 5.1)
• conduct research and analysis to determine evidence-based targets and benchmarks for access indicators (Recommendation 5.2)
• present emergency access performance over time as the percentage of patients seen, admitted or discharged within time (Recommendation 5.3)
• include performance against access indicators measuring the number of patients with emergency department stays of more than 24 hours and rates of HIPs of surgery in Your Hospitals (Recommendation 5.4).