Managing the Performance of Rail Franchisees

Tabled: 7 December 2016

2 Contract and performance management

The current metropolitan train and tram franchise agreements (MR3) include an operational performance regime (OPR) and a customer experience performance regime (CEPR).

The former Department of Transport designed these regimes and negotiated the MR3 agreements with the franchisees, Metro Trains Melbourne (MTM) and Yarra Trams. Since 2012, Public Transport Victoria (PTV) has been responsible for managing the franchisees and motivating them to improve their performance.

This Part examines PTV's contract management and how well PTV has implemented these performance regimes. In particular, we assessed whether PTV has addressed the weaknesses in the integrity of performance data and performance monitoring systems that were identified in previous audits.

2.1 Conclusion

Weaknesses in the design of the performance regimes and in PTV's contract management mean that it is not extracting maximum value for money from its franchise agreements.

PTV is aware of the gaps in its contract management approach and has begun work to address these deficiencies.

PTV has also comprehensively reviewed the OPR and CEPR in preparation for future franchisee agreements. It has identified several improvements that should address current weaknesses if they can be successfully negotiated and implemented.

2.2 Contract management

An effective contract management system:

  • clarifies roles and responsibilities for contract management and decision-making
  • is managed by appropriately skilled and well-trained staff who understand the contract
  • consistently applies the provisions of the contract, including payments and performance regimes
  • incorporates ongoing reviews of the contract, adapts to changing circumstances and manages contract variations
  • guides relationship management, dispute resolution and issue management
  • outlines contingency planning and risk management.

The contract management plan for the train and tram franchise agreements was established in November 2009. It defines the overall approach, including contract objectives, roles and responsibilities, reporting and communication protocols, and governance requirements. The plan is supported by 36 contract administration guides that define how PTV will monitor, manage and enforce the obligations of the agreements.

We found that the contract management plan and guides were useful resources for day-to-day contract management activities, but there were two areas that needed to be strengthened:

  • First, there was no provision in the plan for a rigorous, systematic review of the agreements during their lifetimes. Consequently, PTV has not undertaken such reviews and it has not systematically captured issues and risks as they arose, identified opportunities to improve contract management, or routinely tracked the benefits of the agreements and evaluated whether they are achieving their objectives.
  • Second, PTV's record keeping and document control has not been adequate. This creates a risk that knowledge needed for effective contract management may be lost when key staff leave PTV.

In October 2015, PTV completed an internal audit of its contract management of the tram franchise agreement. PTV believes that the gaps identified could equally be applied to the management of the train franchise agreement. The gaps include:

  • unclear roles and responsibilities for contract management
  • no designated contract manager
  • no systematic approach to identifying, mitigating and monitoring risks
  • ad hoc quality assurance processes
  • out-of-date contract management plans and associated documentation
  • no strategy to capture the lessons learnt.

We found similar weaknesses in PTV's contract management practices, as did our May 2015 audit Tendering of Metropolitan Bus Contracts.

It is encouraging that PTV has acknowledged these weaknesses and plans to address them by developing an organisation-wide contract management framework, to be implemented by November 2017.

2.3 Operational performance regime

Weaknesses in the current OPR compromise PTV's ability to rigorously assess and drive franchisee performance.

A well-designed performance regime starts with clearly defined measures and targets that are relevant to the franchise agreement's stated aims. It is important that these measures and targets can reliably assess actual performance. The performance regime also needs to be cost effective to administer and should create positive incentives for operators to continually improve their performance.

When a performance regime has been effectively implemented, it produces complete and accurate information about the franchisee's performance. PTV can then use this information to identify and manage underperformance and to inform decisions about investment in, and the design of, future agreements.

Because the performance regimes for Victoria's train and tram system are embedded in the franchise arrangements, their ability to effectively measure the franchisee's performance depends on how well PTV manages the franchise agreements. Appendix C provides further detail on the contractual framework for the OPR.

2.3.1 Reliability of performance data

Trains

Until March 2015, PTV relied on manually recorded and self-reported reliability and punctuality data from MTM. This information was used to determine bonus payments and penalties under the OPR and to inform publicly reported performance results.

Manual recording of this information has obvious and inherent risks—such as franchisee error and bias—which affect the reliability of the data. PTV managed this risk by conducting regular field surveys to assess the accuracy of MTM's reporting. From 2011–12 to 2014–15, PTV's surveys identified instances of under-reporting of late trains. This led to PTV issuing approximately $2 million in penalties to MTM.

Station skipping

Although PTV's field surveys successfully detected under-reporting of trains arriving late at their destination, they did not detect instances of trains skipping stations and did not monitor train movements throughout the course of an entire trip. Train operators may decide to skip stations to recover time from a previous disruption to avoid penalties imposed when trains arrive late at their destination. In March 2013, PTV asked franchisees to begin self-reporting discretionary station skipping to help address this problem.

MTM's self-reporting of trains skipping stations excludes instances when a train was forced to skip a station to make up time for reasons beyond its control—for example, a delay caused by a passenger requiring medical attention. The publicly reported figures on station skipping—labelled as 'unplanned expresses' in the reports—show a significant reduction in the practice from June 2014 to June 2016.

PTV relies on honest and accurate self-reporting of station skipping by MTM. PTV also relies on franchisees to report the causes of service delays accurately and in good faith, as these reports form the basis for bonus and penalty payments under the OPR.

Automated Performance Reporting System

In March 2015 PTV replaced the manual data recording system for trains with an automatic train monitoring system, the Performance Reporting System (PRS). The PRS reduces the risk of errors that could arise from manual recording, significantly increases the number of monitoring points on the network, and allows for more sophisticated data analysis. However, the PRS still does not provide comprehensive train journey information, because it does not measure train punctuality at all stations. PTV reports that it is not feasible for the PRS to be implemented at all stations, due to technical limitations and costs.

PTV carried out field surveys in June 2015 to test the accuracy of the PRS. These surveys did not identify any significant discrepancies between manually collected data and the data generated by the PRS. However, PTV has not done any further audits or reviews of the PRS and, therefore, has limited assurance about the ongoing integrity of the information generated by this system.

Trams

Data about tram performance is recorded in the Automatic Vehicle Monitoring (AVM) system. This state-owned system records tram journey information and is operated by Yarra Trams.

Each tram route has three to five monitoring points to identify early departures, late arrivals, short runs or cancellations. When a tram passes a monitoring point, information is captured in a central database and sent to PTV so that it can analyse tram services' punctuality and reliability.

PTV acknowledges that the monitoring sites along each route do not provide enough data to accurately assess service performance for the entire length of a journey.

PTV also acknowledges that there are longstanding problems and risks that affect the reliability of the AVM system. It uses technology from 1987, and the risks and limitations of the technology may affect safety, operational performance and passenger information. Our June 2014 report Using ICT to Improve Traffic Management and our August 2014 report Coordinating Public Transport found that the system was obsolete and unable to determine the precise position of each tram.

PTV has completed a project to extend the life of the AVM until 2020 and has recently begun another project to identify potential replacement options. PTV acknowledges that the life extension of the AVM is an interim measure, and that there are problems and risks with the current system. Despite these risks, PTV relies solely on Yarra Trams to assure the ongoing integrity of the system and the data it produces. This means that PTV is publicly reporting performance results and determining bonus and penalty payments with only limited assurance about the reliability of the performance data that underpins these results.

2.3.2 Public reporting of operational performance

PTV publishes daily results of train and tram reliability and punctuality on its website. It collates these results and other operational information into monthly and quarterly reports titled Track Record, which provide the public with an indication of train and tram performance. Appendix C provides further detail on the percentage‑based measures used to produce the data in Track Record and how they differ from the operational performance regime.

There are several problems with how the agreements were designed to measure performance and how PTV reports on performance:

  • Lack of performance targets—there are no targets for the reliability and punctuality of trains and trams, but this is not made clear in the way results are reported. Track Record reports the minimum performance standards, known as thresholds, mandated in the franchise agreements. If the franchisee's performance falls below this threshold, the agreements state that a 'call-in' is triggered. A 'call-in' event requires the franchisee to explain their poor performance and how it will be addressed. This means that the thresholds represent the minimum acceptable performance for franchisees rather than performance targets. They are based on the average performance of the previous franchisee for the period May 2008 to June 2009 minus two percentage points for punctuality and one percentage point for reliability.
  • Fixed thresholds—the franchise agreements do not allow for periodic resetting of the reliability and punctuality thresholds, to reflect service or network improvements or actual performance.
  • Unavoidable service delays—the publicly reported reliability and punctuality figures take into account events that are outside the control of the franchisees but these events are excluded from the OPR.
  • Bonus and penalty payments not aligned to performance results—the publicly reported reliability and punctuality figures are not directly connected to the bonus and penalty payments that franchisees receive under the OPR, but this is not made clear in Track Record. For example, there are instances where the franchisee performs above the threshold, but is still penalised under the OPR because it has failed to meet performance targets. This means that the figures in Track Record do not give the public a sound understanding of franchisee performance.

PTV's review of public reporting in preparation for negotiating the metropolitan train and tram franchise agreements from 2017 onwards (MR4) has identified the need for several key improvements. It is important that public reporting of targets, thresholds and payments is simple and transparent.

2.4 Customer experience performance regime

Under the franchise agreements, the CEPR measures customer experience with train and tram services. This regime, linked to bonus and penalty payments, was first introduced in MR3 franchise agreements.

The CEPR includes ten components, detailed in Figure 2A, monitored by PTV (and previously by the former Department of Transport) through quarterly network inspections and customer satisfaction phone surveys.

Figure 2A
Summary of CEPR components

Asset

CEPR component

Description

Tram and train stop precincts

Graffiti removal and personal safety (train)

Graffiti removal and, for trains, availability of an emergency stop button

Customer information

Fit-for-purpose customer information

Asset condition

General asset condition, including vandalism rectification

Cleanliness

Cleaning and de-littering

Rolling stock

Graffiti removal

Graffiti removal

Customer information

Fit-for-purpose static information, announcements and passenger information displays (electronic)

Asset condition

General asset condition including vandalism rectification

Cleanliness

Cleaning and removal of litter

Train and tram reserves and hot‑spots

Cleanliness

Removal of graffiti, litter and refuse

Customer satisfaction

Overall satisfaction (measured by customer satisfaction survey)

n/a

Source: VAGO, based on information provided by PTV.

Until July 2012, PTV used the quarterly network inspections and phone survey results to assess franchisees' annual performance against the thresholds in the CEPR. Franchisees received a bonus payment when performance for each component exceeded an upper threshold, and incurred a financial penalty when performance was below the lower threshold. The annual payments were capped at $500 000 for trams and $1 million for trains.

In July 2012, PTV abandoned the CEPR components that required network inspections. PTV determined that the CEPR did not provide value for money because:

  • the cost of physical inspections was high
  • the threshold range was too wide, making it hard to perform below the threshold, but also difficult to exceed it in many areas, which discouraged improvement
  • the contracts lacked a provision to reset the benchmarks as more performance data was gathered
  • there were some unintended outcomes—for example, during the three years the CEPR was in place:
    • Yarra Trams received more than $1 million for exceeding set thresholds even though its performance declined in some of the CEPR components
    • MTM received over $0.7 million in incentive payments, even though it did not consistently exceed the thresholds
    • the incentives and penalties were relatively small given the investment required to improve some areas, such as the condition of rolling stock
    • the former Department of Transport did not proactively share the CEPR performance data with the franchisees to help them improve their performance.

From July 2012, PTV continued incentive payments and penalties based solely on the results of quarterly customer satisfaction surveys. It reviewed past survey results to develop upper and lower thresholds for the incentive payments and penalties, and discounted factors that were beyond the franchisees' control but which could negatively impact on results, such as rises in ticket prices.

Although PTV's decision to abandon network inspections was reasonable, surveys only give a very subjective assessment of how well the franchisees have been fulfilling their customer service obligations. Since July 2012, the train and tram franchisees have received incentive payments of $0.6 million and $1.1 million respectively, and neither has been penalised.

This lack of objective performance data will continue at least until the current agreement ends in November 2017.

2.4.1 Customer service standards

As well as meeting the requirements of the CEPR, franchisees are also expected to meet agreed customer service standards under the franchise agreements. We found that PTV has not held the franchisees to account for their compliance with these standards.

The MR3 agreements require that the franchisees 'make reasonable endeavours' to comply with the standards, which include cleanliness, graffiti removal, provision of information, staffing and fault rectification times. The franchisees are required to submit annual system upkeep plans, which outline how they will comply, and PTV has the right to audit franchisees' compliance with the plans. There are no financial penalties or incentives attached to the upkeep plans, but failing to comply triggers a warning and a requirement for franchisees to explain their performance and how problems will be addressed.

In preparing for MR4, PTV has identified that full compliance with the current standards has not always been possible because of their ambiguity. For example:

  • The train franchisee is required to remove internal and external graffiti from rolling stock within 24 hours of notification or, where not reported, within 72 hours of occurrence. However, because specific record-keeping practices and inspection regimes are not mandated in the franchise agreement, it is difficult to define when the graffiti 'occurred' or how long it was visible to customers.
  • The train franchisee is required to wash the exterior of rolling stock once every 14days, but PTV and the franchisee have recently agreed that this target was never achievable.

PTV has not proactively reviewed system upkeep plans, nor has it completed any audits or inspections of franchisee compliance with the customer service standards since July 2012. Instead, PTV has relied on customer feedback provided through surveys, call centre and website enquiries, and 'mystery shopper' programs operated by the franchisees. As a result, PTV has not objectively assessed franchisees' compliance with their customer service obligations.

2.4.2 Revised customer experience regime

In 2014, after consulting customers and franchisees, and completing other research, PTV prepared a revised set of customer experience standards that sought to capture the end-to-end customer experience. The standards include achievable, desirable and aspirational standards based on current performance.

PTV has recognised that capital investment is required to achieve some of these standards, and that effective and efficient measurement of performance against every standard is not achievable. PTV plans to use these standards, together with the lessons learnt from the CEPR and the current franchise agreement provisions, to develop a revised CEPR to be included in the MR4 franchise agreements. The success of this new regime will depend on the forthcoming negotiations and PTV's implementation of the new agreements.

Back to Top