[ Skip to content | Skip to Table of Contents | PDF of this document | VAGO Home | VAGO Publications | Reduce Text Size | Enlarge Text Size ]
Victorian Coat of Arms

Managing the Performance of Rail Franchisees

Ordered to be published

VICTORIAN GOVERNMENT PRINTER December 2016

PP No 228, Session 2014–16

The Hon. Bruce Atkinson MLC
President
Legislative Council
Parliament House
Melbourne
The Hon. Telmo Languiller MP
Speaker
Legislative Assembly
Parliament House
Melbourne

Dear Presiding Officers

Under the provisions of section 16AB of the Audit Act 1994, I transmit my report Managing the Performance of Rail Franchisees.

Yours faithfully

Signature of Andrew Greaves (Auditor-General)

Andrew Greaves
Auditor-General

7 December 2016

Audit overview

Public transport is an essential service for many Victorians, and an effective transport system is critical to the state's economic development. Public Transport Victoria (PTV) is responsible for managing the contracts with the operators that run the metropolitan passenger train and tram networks, known as franchise agreements. Agreements must be well designed, and effective contract management policies, processes and systems must be in place to ensure these agreements deliver value for money to the state of Victoria.

In this audit, we looked at how PTV manages its franchise agreements and its assets, to improve the performance of the train and tram networks. We also assessed how PTV and the Department of Economic Development, Jobs, Transport & Resources have prepared for future franchise agreements.

Conclusion

Train and tram services have become more reliable and punctual under the current franchise agreements, and customer satisfaction has improved. However, weaknesses in the design and implementation of the performance regimes prevent PTV from maximising value from the current agreements.

It is positive to see that PTV has gained a good understanding of the strengths and weaknesses of the current agreements and is well prepared to negotiate better outcomes in future agreements. The challenge for PTV is to successfully negotiate and properly implement the new agreements and improve its management of contracts.

Our key area of concern has been PTV's lack of focus on managing its train and tram system assets, which it has only recently begun to address. As a result, PTV has limited assurance that these critical assets are being optimally maintained and used by operators to maximise the efficiency and effectiveness of train and tram services.

Findings

Managing contracts and performance

Measuring performance

PTV measures the performance of its train and tram operators, or franchisees, in several ways. It measures and publicly reports on the reliability and punctuality of train and tram services, and surveys customer satisfaction with these services.

One of the challenges for PTV has been that the current arrangements do not enable it to periodically review and reset reliability and punctuality performance thresholds. These thresholds set minimum standards for performance and outline consequences if performance falls below the threshold.

The franchise agreements also include operational and customer experience performance regimes that are linked to financial incentives and penalties.

Until recently, PTV has relied heavily on manual systems and on data reported by train operators to assess the performance of trains. In 2015, it introduced an automatic train position monitoring system to improve the way it keeps track of franchisees' performance.

PTV's automated system to monitor tram performance, introduced in 1987, uses old technology with significant limitations that reduce its effectiveness. PTV's own checking of the reliability of both systems is limited. It largely relies on the franchisees to assure the ongoing integrity of the monitoring systems and the data they produce. This means PTV has only limited assurance about the reliability of the underlying performance data that it reports publicly and uses to determine bonus and penalty payments for franchisees.

The customer experience performance regime under the current franchise agreements was poorly designed and costly to administer, making it ineffective. PTV decided that it was not delivering value for money, and largely abandoned it in 2012. Since then, PTV has relied on limited subjective data to assess the way franchisees fulfil their customer experience obligations and to determine the payment of associated bonuses and penalties.

Reporting performance

PTV's public reports identify reliability and punctuality performance thresholds for franchisees. However, they do not clearly explain that these thresholds actually represent underperformance levels instead of performance targets. PTV needs to ensure that its public reporting is easy to understand and clearly reflects franchisee performance.

Managing performance

While there have been improvement in train and tram reliability and punctuality, we found gaps in how PTV manages its franchise agreements:

The current agreements give franchisees an exclusive right to negotiate a seven‑year extension of their agreement with the state, as long as they meet certain performance benchmarks. In December 2015, PTV assessed that both franchisees met the required benchmarks. However, PTV could not provide evidence to demonstrate the rationale for some of the benchmarks and how they were assessed.

PTV recognises the need for a more comprehensive contract management framework supported by effective systems and adequate resources. It has started to address these issues.

Managing assets

We found significant weaknesses in how PTV oversees the maintenance and renewal of assets leased to its franchisees under the current agreements. In particular, PTV does not have adequate medium- to long-term asset strategies for its train and tram network assets.

PTV also does not know enough about the condition of the assets used across both the tram and train networks. The lack of meaningful information about asset condition reduces PTV's capacity to appropriately plan, budget and prioritise maintenance and renewal work.

It is positive that PTV and the Department of Economic Development, Jobs, Transport & Resources have begun to address these weaknesses.

Extending the current franchise agreements

PTV has prepared for future franchise agreements by thoroughly examining the strengths and weaknesses of its current agreements. To help it negotiate with the current franchisees, PTV has developed an in-depth understanding of operating costs and has identified opportunities to improve the franchisee's financial reporting. This puts PTV in a good position to negotiate important improvements with franchisees.

Recommendations

We recommend that Public Transport Victoria:

  1. finalise and introduce a contract management framework with appropriate processes and systems to better:
    • manage knowledge and improve record keeping
    • define roles and responsibilities and decision-making processes
    • manage contract risks
    • track benefits and evaluate performance throughout the life of the contract to identify problems, opportunities and weaknesses

    (see Section 2.2).

  2. improve how it monitors and manages the performance of train and tram franchisees by:
    • periodically auditing franchisees' performance data and related systems (see Section 2.3.1)
    • planning and implementing monitoring systems for the train and tram networks that provide comprehensive real-time journey information (see Section 2.3.1)
    • preparing performance benchmarks for future franchise agreements that drive improvement and are periodically reviewed and reset where necessary (see Section 2.3.2)
    • clearly explaining performance thresholds and how they relate to incentives and penalties in public reporting on train and tram performance (see Section 2.3.2)
    • developing an effective way to monitor customer experience performance, using lessons from the current agreements (see Section 2.4)
  3. before the next franchise agreement starts, work collaboratively with Victorian Rail Track (VicTrack) and the Department of Economic Development, Jobs, Transport & Resources to prepare and introduce a comprehensive framework for managing assets that is consistent with the Department of Treasury & Finance's Asset Management Accountability Framework and includes:
    • clearly defined roles and responsibilities for all agencies and franchisees, including responsibility for operational control and management system assets
    • systems and practices to capture and analyse comprehensive data about the condition of train and tram assets, including a baseline condition survey when the new agreements start and periodic reviews from then on
    • clearly defined and ranked standards for asset management
    • strategies for managing each groups of assets

    (see Section 3.2)

  4. evaluate the performance-based right to a contract extension under the current franchise agreement, to understand the benefits and weaknesses of this approach for future agreements (see Section 4.2.1)
  5. improve its systems and capability to collect and analyse comprehensive information on franchisee costs throughout the life of franchise agreements (see Section 4.3.1).

In June 2016, the government announced a new agency—Transport for Victoria (TFV)—that will plan, coordinate and operate Victoria's transport system and associated agencies. We acknowledge that responsibilities for coordinating and implementing the above recommendations may be affected by this change.

Responses to recommendations

We have consulted with the Department of Premier & Cabinet, the Department of Economic Development, Jobs & Transport, Public Transport Victoria, and Victorian Rail Track and we considered their views when reaching our audit conclusions. As required by section 16(3) of the Audit Act 1994, we gave a draft copy of this report to those agencies and asked for their submissions and comments.

The following is a summary of those responses. The full responses are included in Appendix A.

Public Transport Victoria accepted all of the recommendations in the report and provided a detailed action plan on how it plans to address these recommendations. The Department of Economic Development, Jobs & Transport responded with a commitment to support Public Transport Victoria in implementing the recommendations.

[ Contents of this report | VAGO Home | VAGO Publications ]

1 Audit context

Efficient and reliable public transport is critical for the state's economic prosperity and liveability. In Victoria, metropolitan trains and trams have been run by private operators since 1999, operating under various franchise agreements with the state government.

Operating train and tram services through franchise agreements introduces risks for the state—private operators may not deliver the best possible performance and value for money.

As the agency responsible for managing the franchise agreements, Public Transport Victoria (PTV) is also responsible for managing this risk. It has a central role in monitoring the performance of franchisees and driving improvements in the way they operate. PTV is also responsible for making sure that franchisees satisfactorily use and maintain the train and tram assets that they lease from the state.

1.1 Train and tram franchise agreements

The current train and tram franchise agreements—known as MR3—were established in 2009 through a competitive tender process. This was the third time that franchise agreements had been put in place for Melbourne's train and train services.

Figure 1A summarises the history of the franchise agreements for Victoria's train and tram services.

Figure 1A
Phases of Victorian metropolitan rail franchising

Phases of Victorian metropolitan rail franchising

Source: VAGO, based on information from PTV.

Under MR3, the former Department of Transport appointed:

Both agreements were for an initial term of eight years, and expire in November 2017. Regional train services (V/Line) are not part of these agreements.

When PTV was established in April 2012, it took responsibility for these franchise agreements. Figure 1B shows the various parties involved in the agreements and Figure 1C explains the contractual arrangements.

Figure 1B
MR3 contractual framework

MR3 contractual framework

Source: VAGO, based on information from PTV.

Figure 1C
Summary of MR3 contractual arrangements

Agreement

Summary

Franchise agreement

The franchise agreement details the obligations for service delivery, network development and service planning, rolling stock, payments and variations, operational incentives, enforcement and how the franchise will be re-tendered.

Infrastructure lease

Victorian Rail Track (VicTrack) owns the state's rail assets and leases them to PTV under an overarching infrastructure lease, known as a headlease. PTV then subleases the assets to the franchisees under individual infrastructure leases. The franchisees are obligated to manage, maintain, repair and replace the assets, which include track, signals, stations, land and information systems.

Rolling stock lease

Rolling Stock Holdings (a subsidiary of VicTrack) owns the majority of Victoria's rolling stock. It leases rolling stock to PTV under an overarching lease. PTV then subleases the rolling stock to the franchisees under individual rolling stock leases.

Projects agreement

The projects agreement establishes a consultative process for the state and franchisees to work together to plan, develop and deliver projects.

Telecommunications service agreement

VicTrack owns the state's transport telecommunications network. VicTrack provides managed telecommunication services to the franchisees through this infrastructure under these agreements.

Inter-operator agreements

Inter-operator agreements specify how franchisees and other operators (such as V/Line) are required to work together.

Source: VAGO, based on information from PTV.

1.2 Train and tram performance under the current franchise agreements

To manage the performance of the train and tram franchisees, PTV uses:

The MR3 agreements specify minimum service standards for punctuality and reliability—known as thresholds—which are not performance targets.

Under the OPR, PTV determines incentive and penalty payments by measuring the impact of an individual service delay or cancellation on passengers. This is a different measure to the publicly reported figures, which reflect whether trains and trams run on time. Appendix C outlines the OPR in more detail.

1.2.1 Train performance

In general, MTM has stabilised and improved train performance under the current agreement. Figure 1D shows that performance has significantly improved—it has exceeded the punctuality threshold—87 per cent—since late 2011 and has met or exceeded the reliability threshold since 2009 (the entire period of the agreement).

Figure 1D
Trains quarterly punctuality and reliability, December 2009 to June 2016

Trains quarterly punctuality and reliability, December 2009 to June 2016

Note: The December 2009 quarter covers only one month of operations.

Source: VAGO, based on information from PTV published in Track Record.

PTV attributes improvements in train punctuality to more state investment in network upgrades, new rolling stock, franchisee performance initiatives and stable patronage growth.

The satisfaction of train customers is measured by a quarterly phone survey. As shown in Figure 1E, customer satisfaction has improved under the current agreement and has met the performance targets that were determined when the agreement was negotiated.

Figure 1E
Train customer satisfaction monitor results (phone survey), March 2010 to June 2016

Train customer satisfaction monitor results (phone survey),

March 2010 to June 2016

Note: Under the current agreement, the train franchisee was required to achieve and maintain customer satisfaction index of at least 70 by 1 December 2014 and 80 per cent by 1 December 2018.

Source: VAGO, based on information from PTV.

The train franchisee is eligible for incentive payments and may incur penalties under the OPR and CEPR. Under the OPR, MTM received $28.6 million in total (comprising incentive payments less penalties) between December 2009 and June 2016. Under the CEPR, MTM received $1.3 million in incentive payments and had not incurred any penalties at the time of the audit.

1.2.2 Tram performance

Under the current agreements, tram performance has been steady and above the service thresholds—98 per cent for reliability and 77 per cent for punctuality—as shown in Figure 1F.

Figure 1F
Trams quarterly punctuality and reliability, December 2009 to June 2016

Trams quarterly punctuality and reliability,

December 2009 to June 2016

Note: The December 2009 quarter covers only one month of operations.

Source: VAGO, based on information from PTV published in Track Record.

PTV attributes improvements in tram punctuality to state initiatives, such as new timetables, and franchisee initiatives to improve communication and efficiency. Tram reliability and punctuality is affected by hot weather and the special events held from January to March. The introduction of the free tram zone in Melbourne's central business district in January 2015 resulted in a decrease in performance due to increased patronage. Tram performance is also increasingly affected by road traffic congestion.

The satisfaction of tram customers is measured by a quarterly phone survey. As shown in Figure 1G, customer satisfaction has improved under the current agreement and has met the performance targets that were determined when the agreement was negotiated.

Figure 1G
Tram customer satisfaction monitor results (phone survey), March 2010 to June 2016

Tram customer satisfaction monitor results (phone survey),

March 2010 to June 2016

Source: VAGO, based on information from Public Transport Victoria.

The tram franchisee is eligible for incentive payments and may incur penalties under the OPR and CEPR. Under the OPR, Yarra Trams was penalised $4.7 million in total for the period December 2009 to 30 June 2016. Under the CEPR, Yarra Trams received $2.2 million in incentive payments and has not incurred any penalties to date.

1.3 Patronage

Increasing numbers of passengers also affects the operational performance of trains and trams. When more passengers are on board a tram or train, it must stop for longer—known as dwell time—to allow passengers to board or exit.

From 2006 to 2008, the number of train passengers grew by 12 per cent annually, which had a negative impact on punctuality. Since 2009, the number of passengers has stabilised, which has helped to improve operational performance under MR3.

PTV forecasts that passenger numbers will grow by 3.3 per cent annually for trains and 3.8 per cent annually for trams, from 2015 to 2021. This increased patronage, and the major construction projects planned during this period, will present operational challenges for the franchisees.

1.4 Payments to the franchisees

Since 2009, the state has paid over $7.6 billion to the franchisees to run Victoria's train and tram networks—$5.4 billion to MTM and $2.2 billion to Yarra Trams, up to 30 June 2016. These payments cover a range of services and activities, summarised in Figure 1H and do not include the payments for rolling stock lease amounts made by the franchisees on behalf of the state.

Figure 1H
Summary of franchise payment types

Summary of franchise payment types

Source: PTV.

The service fee for running the networks (base contract and maintenance, in Figure 1H) and a share of the revenue from metropolitan ticket sales make up the major part of payments to the franchisees. The franchisees carry the risk that their running costs might exceed the combined service fee and share of ticket revenue.

Until 2014, the state guaranteed ticket revenue due to uncertainties with the implementation of the myki ticketing system. From 2014 onwards, franchisees have been exposed to the risk of variable ticket revenue. This risk is mitigated through periodic resetting of franchisee forecasts of ticket revenue to align with actual revenue, and having a risk-sharing regime that sets an upper and lower limit on the amount of risk the franchisee would bear. Figures 1I and 1J include the actual ticket revenue and risk mitigation payments after January 2014. However, PTV does not include actual ticket revenue in its public reporting in Track Record.

Figure 1I
Payments to MTM, December 2009 to June 2016

Payments to MTM, December 2009 to June 2016

Note: 2009–10 only includes the period December 2009 to June 2010.

Note: Excludes rolling stock lease payments.

Source: VAGO, based on information from PTV.

Figure 1J
Payments to Yarra Trams, December 2009 to June 2016

Payments to Yarra Trams, December 2009 to June 2016

Note: 2009–10 only includes the period December 2009 to June 2010.

Note: Excludes rolling stock lease payments.

Source: VAGO, based on information from PTV.

Separate to these payments, franchisees can earn commercial revenue from advertising and renting out surplus land or buildings. Franchisees also receive payments for their involvement in projects beyond their project agreements with PTV—for example, the level crossing removal program. The scope of this audit did not cover these project agreements.

1.5 Managing train and tram assets

Victoria's public transport assets were valued at $29.8 billion at 30 June 2016, according to VicTrack's annual report for 2015–16. This estimate includes tracks, stations, bridges, buildings, structures, plant and equipment, signalling and communications, land and the Melbourne Underground Rail Loop.

Asset management involves managing all of the activities carried out throughout the life of the asset, including maintenance and renewal, to make sure that it delivers its full value and contributes to the service objective.

Managing Victoria's train and tram assets is complex because of the range of agencies involved—the assets are owned by VicTrack, leased to PTV, and then subleased to the franchisees. Figure 1K describes the roles and responsibilities of these agencies for maintaining and renewing Victoria's train and tram assets under MR3.

Figure 1K
Summary of maintenance and renewal responsibilities in MR3

Agency

Key responsibilities

Franchisees

  • Ensure infrastructure is always in a condition that allows the network to operate safely.
  • Maintain the functionality and capacity of the network.
  • Undertake a predetermined number of renewals specified in the franchise agreement. The train franchisee introduced a renewals prioritisation tool in 2014 that removes this requirement and allows them to determine which renewals will be undertaken based on a range of factors, with the goal of achieving the best outcome for the network.
  • Undertake maintenance and renewal work in accordance with agreed asset management plans and annual works plans.
  • Submit reports on maintenance and renewal key performance indicators and completed work to PTV.

PTV

  • Manage all maintenance and renewal aspects of the franchise agreements and ensure franchisees comply with them. This includes assessing and approving franchisees' asset management plans and annual works plans.
  • Ensure a sound asset management framework is in place to guide franchisees' maintenance and renewal work.
  • Pay franchisees for maintenance and renewal work according to the franchise agreements.
  • Maintain sound knowledge of asset condition by collecting relevant data and information.

Department of Economic Development, Jobs, Transport & Resources

  • Implement the state's policy objectives and frameworks for managing transport assets and infrastructure.

Source: VAGO.

The train and tram network is a significant asset for the state, so it warrants rigorous oversight and management by PTV to make sure that the assets enable the delivery of effective and efficient public transport.

1.6 Extending train and tram franchise agreements

MR3 includes a provision for the current franchisees to negotiate exclusively with the state for a seven-year extension if they meet certain performance benchmarks.

PTV has determined that both MTM and Yarra Trams have met the performance benchmarks specified in the agreements, enabling them to negotiate a new franchise agreement, which will be known as MR4. If these negotiations fail, the state could extend the agreement at a fixed price for up to three years, while it undertakes a tender process for new franchisees.

In preparation for these MR4 negotiations, PTV has undertaken a project to examine the strengths and weaknesses of the current agreements and identify opportunities to improve value for money in future franchise agreements.

1.7 Agency responsibilities

Public Transport Victoria

PTV was established in April 2012. Its primary objective is to plan, coordinate, provide, and maintain a safe, punctual, reliable and clean public transport system. PTV oversees all aspects of the franchisees' day-to-day operation of the train and tram networks, including performance requirements.

Department of Economic Development, Jobs, Transport & Resources

The former Department of Transport, which initiated the MR3 franchise agreements, was disbanded in 2013 and its functions were distributed between the now Transport Division of the now Department of Economic Development, Jobs, Transport & Resources (DEDJTR) and PTV.

DEDJTR is responsible for the state's transport and infrastructure policy and planning. It is overseeing the negotiations of new franchise agreements, and the secretary of DEDJTR is chairing the primary governance body for this project.

In June 2016, the Victorian Government announced the creation of Transport for Victoria (TFV). This new agency will plan, coordinate and operate Victoria's transport systems and associated agencies.

TFV will sit within DEDJTR and have overarching responsibility for the transport portfolio. Under this arrangement, PTV will remain as the agency responsible for managing and administering the train and tram franchise agreements.

VicTrack

Created in 1997, VicTrack is a state-owned business operating under the Transport Integration Act 2010. It owns Victoria's railway land, infrastructure and assets. Through a subsidiary (the Rolling Stock Holdings group of companies), it also owns much of the state's rolling stock. VicTrack also has a role in providing telecommunications services to franchisees.

1.8 Previous VAGO audits

Since the 1990s, we have regularly scrutinised Melbourne's public transport operations. We have conducted audits examining various aspects of the train and tram system, including franchising arrangements, financial performance, asset management and operational performance. A summary of these audits and their findings is provided in Appendix B.

1.9 What this audit examined and how

We examined PTV's effectiveness in managing the performance of Melbourne's metropolitan train and tram franchisees, by assessing whether:

Achieving value for money in tram and train services involves consideration of a wide range of factors and is not only a financial question. We focused on contract management, oversight of franchisee operational performance, and asset management and whether they contributed to value for money.

The agencies included in this audit were:

We conducted the audit in accordance with section 15 of the Audit Act 1994 and the Australian Auditing and Assurance Standards.

The total cost of the audit was $555 000.

1.10 Report structure

The report is structured as follows:

[ Contents of this report | VAGO Home | VAGO Publications ]

2 Contract and performance management

The current metropolitan train and tram franchise agreements (MR3) include an operational performance regime (OPR) and a customer experience performance regime (CEPR).

The former Department of Transport designed these regimes and negotiated the MR3 agreements with the franchisees, Metro Trains Melbourne (MTM) and Yarra Trams. Since 2012, Public Transport Victoria (PTV) has been responsible for managing the franchisees and motivating them to improve their performance.

This Part examines PTV's contract management and how well PTV has implemented these performance regimes. In particular, we assessed whether PTV has addressed the weaknesses in the integrity of performance data and performance monitoring systems that were identified in previous audits.

2.1 Conclusion

Weaknesses in the design of the performance regimes and in PTV's contract management mean that it is not extracting maximum value for money from its franchise agreements.

PTV is aware of the gaps in its contract management approach and has begun work to address these deficiencies.

PTV has also comprehensively reviewed the OPR and CEPR in preparation for future franchisee agreements. It has identified several improvements that should address current weaknesses if they can be successfully negotiated and implemented.

2.2 Contract management

An effective contract management system:

The contract management plan for the train and tram franchise agreements was established in November 2009. It defines the overall approach, including contract objectives, roles and responsibilities, reporting and communication protocols, and governance requirements. The plan is supported by 36 contract administration guides that define how PTV will monitor, manage and enforce the obligations of the agreements.

We found that the contract management plan and guides were useful resources for day-to-day contract management activities, but there were two areas that needed to be strengthened:

In October 2015, PTV completed an internal audit of its contract management of the tram franchise agreement. PTV believes that the gaps identified could equally be applied to the management of the train franchise agreement. The gaps include:

We found similar weaknesses in PTV's contract management practices, as did our May 2015 audit Tendering of Metropolitan Bus Contracts.

It is encouraging that PTV has acknowledged these weaknesses and plans to address them by developing an organisation-wide contract management framework, to be implemented by November 2017.

2.3 Operational performance regime

Weaknesses in the current OPR compromise PTV's ability to rigorously assess and drive franchisee performance.

A well-designed performance regime starts with clearly defined measures and targets that are relevant to the franchise agreement's stated aims. It is important that these measures and targets can reliably assess actual performance. The performance regime also needs to be cost effective to administer and should create positive incentives for operators to continually improve their performance.

When a performance regime has been effectively implemented, it produces complete and accurate information about the franchisee's performance. PTV can then use this information to identify and manage underperformance and to inform decisions about investment in, and the design of, future agreements.

Because the performance regimes for Victoria's train and tram system are embedded in the franchise arrangements, their ability to effectively measure the franchisee's performance depends on how well PTV manages the franchise agreements. Appendix C provides further detail on the contractual framework for the OPR.

2.3.1 Reliability of performance data

Trains

Until March 2015, PTV relied on manually recorded and self-reported reliability and punctuality data from MTM. This information was used to determine bonus payments and penalties under the OPR and to inform publicly reported performance results.

Manual recording of this information has obvious and inherent risks—such as franchisee error and bias—which affect the reliability of the data. PTV managed this risk by conducting regular field surveys to assess the accuracy of MTM's reporting. From 2011–12 to 2014–15, PTV's surveys identified instances of under-reporting of late trains. This led to PTV issuing approximately $2 million in penalties to MTM.

Station skipping

Although PTV's field surveys successfully detected under-reporting of trains arriving late at their destination, they did not detect instances of trains skipping stations and did not monitor train movements throughout the course of an entire trip. Train operators may decide to skip stations to recover time from a previous disruption to avoid penalties imposed when trains arrive late at their destination. In March 2013, PTV asked franchisees to begin self-reporting discretionary station skipping to help address this problem.

MTM's self-reporting of trains skipping stations excludes instances when a train was forced to skip a station to make up time for reasons beyond its control—for example, a delay caused by a passenger requiring medical attention. The publicly reported figures on station skipping—labelled as 'unplanned expresses' in the reports—show a significant reduction in the practice from June 2014 to June 2016.

PTV relies on honest and accurate self-reporting of station skipping by MTM. PTV also relies on franchisees to report the causes of service delays accurately and in good faith, as these reports form the basis for bonus and penalty payments under the OPR.

Automated Performance Reporting System

In March 2015 PTV replaced the manual data recording system for trains with an automatic train monitoring system, the Performance Reporting System (PRS). The PRS reduces the risk of errors that could arise from manual recording, significantly increases the number of monitoring points on the network, and allows for more sophisticated data analysis. However, the PRS still does not provide comprehensive train journey information, because it does not measure train punctuality at all stations. PTV reports that it is not feasible for the PRS to be implemented at all stations, due to technical limitations and costs.

PTV carried out field surveys in June 2015 to test the accuracy of the PRS. These surveys did not identify any significant discrepancies between manually collected data and the data generated by the PRS. However, PTV has not done any further audits or reviews of the PRS and, therefore, has limited assurance about the ongoing integrity of the information generated by this system.

Trams

Data about tram performance is recorded in the Automatic Vehicle Monitoring (AVM) system. This state-owned system records tram journey information and is operated by Yarra Trams.

Each tram route has three to five monitoring points to identify early departures, late arrivals, short runs or cancellations. When a tram passes a monitoring point, information is captured in a central database and sent to PTV so that it can analyse tram services' punctuality and reliability.

PTV acknowledges that the monitoring sites along each route do not provide enough data to accurately assess service performance for the entire length of a journey.

PTV also acknowledges that there are longstanding problems and risks that affect the reliability of the AVM system. It uses technology from 1987, and the risks and limitations of the technology may affect safety, operational performance and passenger information. Our June 2014 report Using ICT to Improve Traffic Management and our August 2014 report Coordinating Public Transport found that the system was obsolete and unable to determine the precise position of each tram.

PTV has completed a project to extend the life of the AVM until 2020 and has recently begun another project to identify potential replacement options. PTV acknowledges that the life extension of the AVM is an interim measure, and that there are problems and risks with the current system. Despite these risks, PTV relies solely on Yarra Trams to assure the ongoing integrity of the system and the data it produces. This means that PTV is publicly reporting performance results and determining bonus and penalty payments with only limited assurance about the reliability of the performance data that underpins these results.

2.3.2 Public reporting of operational performance

PTV publishes daily results of train and tram reliability and punctuality on its website. It collates these results and other operational information into monthly and quarterly reports titled Track Record, which provide the public with an indication of train and tram performance. Appendix C provides further detail on the percentage‑based measures used to produce the data in Track Record and how they differ from the operational performance regime.

There are several problems with how the agreements were designed to measure performance and how PTV reports on performance:

PTV's review of public reporting in preparation for negotiating the metropolitan train and tram franchise agreements from 2017 onwards (MR4) has identified the need for several key improvements. It is important that public reporting of targets, thresholds and payments is simple and transparent.

2.4 Customer experience performance regime

Under the franchise agreements, the CEPR measures customer experience with train and tram services. This regime, linked to bonus and penalty payments, was first introduced in MR3 franchise agreements.

The CEPR includes ten components, detailed in Figure 2A, monitored by PTV (and previously by the former Department of Transport) through quarterly network inspections and customer satisfaction phone surveys.

Figure 2A
Summary of CEPR components

Asset

CEPR component

Description

Tram and train stop precincts

Graffiti removal and personal safety (train)

Graffiti removal and, for trains, availability of an emergency stop button

Customer information

Fit-for-purpose customer information

Asset condition

General asset condition, including vandalism rectification

Cleanliness

Cleaning and de-littering

Rolling stock

Graffiti removal

Graffiti removal

Customer information

Fit-for-purpose static information, announcements and passenger information displays (electronic)

Asset condition

General asset condition including vandalism rectification

Cleanliness

Cleaning and removal of litter

Train and tram reserves and hot‑spots

Cleanliness

Removal of graffiti, litter and refuse

Customer satisfaction

Overall satisfaction (measured by customer satisfaction survey)

n/a

Source: VAGO, based on information provided by PTV.

Until July 2012, PTV used the quarterly network inspections and phone survey results to assess franchisees' annual performance against the thresholds in the CEPR. Franchisees received a bonus payment when performance for each component exceeded an upper threshold, and incurred a financial penalty when performance was below the lower threshold. The annual payments were capped at $500 000 for trams and $1 million for trains.

In July 2012, PTV abandoned the CEPR components that required network inspections. PTV determined that the CEPR did not provide value for money because:

From July 2012, PTV continued incentive payments and penalties based solely on the results of quarterly customer satisfaction surveys. It reviewed past survey results to develop upper and lower thresholds for the incentive payments and penalties, and discounted factors that were beyond the franchisees' control but which could negatively impact on results, such as rises in ticket prices.

Although PTV's decision to abandon network inspections was reasonable, surveys only give a very subjective assessment of how well the franchisees have been fulfilling their customer service obligations. Since July 2012, the train and tram franchisees have received incentive payments of $0.6 million and $1.1 million respectively, and neither has been penalised.

This lack of objective performance data will continue at least until the current agreement ends in November 2017.

2.4.1 Customer service standards

As well as meeting the requirements of the CEPR, franchisees are also expected to meet agreed customer service standards under the franchise agreements. We found that PTV has not held the franchisees to account for their compliance with these standards.

The MR3 agreements require that the franchisees 'make reasonable endeavours' to comply with the standards, which include cleanliness, graffiti removal, provision of information, staffing and fault rectification times. The franchisees are required to submit annual system upkeep plans, which outline how they will comply, and PTV has the right to audit franchisees' compliance with the plans. There are no financial penalties or incentives attached to the upkeep plans, but failing to comply triggers a warning and a requirement for franchisees to explain their performance and how problems will be addressed.

In preparing for MR4, PTV has identified that full compliance with the current standards has not always been possible because of their ambiguity. For example:

PTV has not proactively reviewed system upkeep plans, nor has it completed any audits or inspections of franchisee compliance with the customer service standards since July 2012. Instead, PTV has relied on customer feedback provided through surveys, call centre and website enquiries, and 'mystery shopper' programs operated by the franchisees. As a result, PTV has not objectively assessed franchisees' compliance with their customer service obligations.

2.4.2 Revised customer experience regime

In 2014, after consulting customers and franchisees, and completing other research, PTV prepared a revised set of customer experience standards that sought to capture the end-to-end customer experience. The standards include achievable, desirable and aspirational standards based on current performance.

PTV has recognised that capital investment is required to achieve some of these standards, and that effective and efficient measurement of performance against every standard is not achievable. PTV plans to use these standards, together with the lessons learnt from the CEPR and the current franchise agreement provisions, to develop a revised CEPR to be included in the MR4 franchise agreements. The success of this new regime will depend on the forthcoming negotiations and PTV's implementation of the new agreements.

[ Contents of this report | VAGO Home | VAGO Publications ]

3 Maintenance and renewal of assets

The delivery of reliable public transport services in Melbourne depends on the train and tram networks' assets being adequately maintained. Victorian Rail Track (VicTrack) owns these assets, which it leases to Public Transport Victoria (PTV). PTV then leases the assets to the train and tram franchisees, Metro Trains Melbourne (MTM) and Yarra Trams, and pays them for maintenance and renewal works.

To make informed decisions about funding and prioritising asset maintenance and renewal, PTV needs to understand the condition of train and tram assets. In this Part, we examine PTV's strategic management of these important assets and their maintenance and renewal.

3.1 Conclusion

PTV does not have enough assurance that train and tram assets are performing at their optimum level and delivering value for Victoria.

There are longstanding gaps and deficiencies in the way PTV and former responsible agencies have managed the state's train and tram network assets. Deficiencies in PTV's knowledge of the condition of train and tram assets hamper its efforts to make sure that the franchisees are effectively managing the assets. This compromises PTV's planning of maintenance and renewal works and asset funding.

PTV and the Department of Economic Development, Jobs, Transport & Resources (DEDJTR) are now addressing the absence of long-term asset management strategies. In preparing for the next franchise agreements, known as MR4, PTV has acknowledged the inadequate attention it has given to asset management over the course of the current franchise agreements and has started to address these longstanding deficiencies.

3.2 Strategic asset management

Effective maintenance and renewal practices are integral to strategic, whole-of-life asset management to ensure the assets deliver their full service potential.

We found that PTV has not been managing the state's train and tram assets strategically. PTV does not have a comprehensive understanding of the condition of its assets. Therefore PTV has limited capacity to make evidence-based decisions on how it funds and prioritises maintenance and renewal work and limited ability to develop sound long-term asset management strategies.

3.2.1 Asset management strategies

Since the mid-1990s, the Department of Treasury & Finance (DTF) has set clear expectations for Victorian public sector agencies to have a whole-of-life approach to asset management that focuses on the services the assets need to deliver.

Asset management strategies are critical features of this approach—they set out how assets will help to deliver service objectives and include planning for asset maintenance, upgrades, acquisitions and disposals.

A sound asset management strategy requires a robust understanding of the base condition of the existing assets and their capacity, capability and usage. The strategy should take into account available resources, funding constraints, and competing service and asset priorities.

In our 2007 report Managing Victoria's Rail Infrastructure Assets, we recommended that the former Department of Infrastructure develop a long-term asset management strategy for metropolitan rail infrastructure, to help identify and prioritise the maintenance and renewal of these assets. The department did not address this recommendation. Consequently, there was no long-term asset management strategy in place for the train and tram networks at the beginning of the current franchise agreements—known as MR3—in 2009.

PTV acknowledged this gap and began work to address it in November 2015. It published a strategic asset management policy and guide, which outlines the need for medium- and long-term asset management strategies to ensure that train and tram assets deliver the required services. PTV has also committed resources to planning its asset management.

Although these are positive steps, PTV has significant work to do before it can show that it has robust asset management strategies in place and can comply with DTF's 2016 Asset Management Accountability Framework. This framework requires PTV and other public sector agencies to attest to their compliance with a range of mandatory requirements, including the development of asset management strategies, by 30 June 2018.

DEDJTR has also started developing a broad transport asset management program, including a draft transport asset management policy. This will guide all transport agencies as they develop strategies for specific modes of transport, to ensure they align with DEDJTR's draft Network Development Strategy, which is currently in development.

3.2.2 Asset condition information

To fully understand the condition of assets, agencies need to continually collect and analyse information throughout the asset's life cycle. This information can come from:

PTV collects a large amount of information on Victoria's train and tram assets from the franchisees in shared databases, as required by the franchise agreements. PTV also meets regularly with the franchisees to specifically discuss maintenance and renewal of assets.

Figure 3A summarises the main maintenance and renewal reporting that franchisees provide to PTV and any issues we identified with each report.

Figure 3A
Maintenance and renewal reporting requirements

Report

Report content

Issues

Monthly maintenance and renewal report

Includes information on the quantity of work in the annual works plan, categorised by asset class. Submitted with an invoice for payment.

Includes quantity data only, without any analysis of trends.

Quarterly maintenance and renewal report

Quarterly summary of monthly maintenance and renewal reports.

As above.

Monthly key performance indicator (KPI) report

Reports results against the KPIs set at the beginning of the agreements, including:

  • service delays caused by assets
  • types of asset failure and frequency
  • track condition indicators measured by remote monitoring equipment
  • number of temporary speed restrictions imposed due to asset condition
  • amount of maintenance work completed and outstanding.

Contains limited overall analysis of data and trends that would build understanding of asset condition.

KPIs primarily aligned to reliability and performance, rather than the underlying asset condition and useful life.

Maintenance and renewal review group dashboard report

Broader report including achievements and activities, safety, finance, personnel and operational information. The maintenance and renewal component of the report includes a summary of the quantity of maintenance work and inspections, and progress against planned works.

Limited meaningful analysis of trends or data.

Source: VAGO, based on information from PTV.

When preparing for the MR4 franchise agreements, PTV identified weaknesses in the information it collects. These include:

These issues created gaps in PTV's knowledge of the overall health and condition of the assets.

PTV has recently started to address these gaps by planning an asset condition assessment, which is subject to funding approval. PTV plans to document the baseline condition of assets through a further comprehensive survey, which will need to be completed periodically throughout MR4.

VicTrack asset information

As owner of the state's rail assets and infrastructure, VicTrack has an obligation to know the condition and value of these assets in order to comply with applicable accounting standards. Under DTF's new Asset Management Accountability Framework, VicTrack's obligations for asset management reporting will increase.

Although PTV has provided VicTrack with the information required to comply with their accounting and financial reporting obligations, PTV and VicTrack have not worked collaboratively to share more detailed information about assets, and VicTrack was not involved in the asset management meetings between PTV and the franchisees.

For this process to be successful, VicTrack needs to be involved in the strategic asset management planning. The Public Transport Asset Framework Steering Committee was recently established, and VicTrack's inclusion is a positive step. This committee aims to oversee and approve the development and implementation of an asset management framework for Victoria's public transport assets.

3.2.3 Network condition

The limited information available about Melbourne's train and tram networks indicates that the condition of these assets has deteriorated since 2009.

Tram

PTV collects information twice a year on the condition of the tram network by running specially equipped test trams across the network, which collect data on track condition as they go. Although this information is limited, it shows that the condition of Melbourne's tram network deteriorated between 2012 and 2016.

PTV has attributed the degradation in the condition of tram tracks to a range of factors, primarily the recent increase in the use of heavier trams with lower floors. As the use of this type of tram is likely to increase in the future, PTV will need to address this challenge when managing tram network assets.

Train

In May 2015, the Office of the National Rail Safety Regulator (ONRSR) reported that Victoria's metropolitan and regional rail infrastructure and rolling stock (for the train network only) was generally fit for purpose, but much of it had deteriorated and was in poor condition. The ONRSR also reported that the track infrastructure in Melbourne was significantly below the standard of track in equivalent interstate networks and, although this did not present any immediate safety risks, PTV's medium- to long-term management of asset-related risks was inadequate.

In January 2016, PTV responded to this report by committing to systematically apply a more comprehensive approach to managing train assets. To achieve this, PTV plans to implement a range of initiatives, including:

Adequacy of maintenance and renewal funding under MR3

Between the previous franchise agreement and the current MR3 agreement, annual funding for maintenance of tram network assets and renewal works has been relatively constant.

Under the MR3 agreement, annual funding for work on train network assets increased by around $42 million. PTV has reviewed the adequacy of funding levels for asset maintenance and renewal works on the train and tram networks and found that it cannot accurately determine whether current funding is sufficient, partly because it lacks comprehensive information on the condition of the assets.

PTV's benchmarking against other jurisdictions suggests that its funding for asset maintenance and renewal as a percentage of asset renewal costs is relatively low. However, equivalent networks were difficult to identify—particularly for trams—and the results were therefore of limited use in determining required funding.

Given the importance of train and tram assets, PTV's inability to demonstrate a comprehensive understanding of required funding levels is significant.

3.3 PTV's oversight of asset maintenance and renewal

PTV pays franchisees for maintenance and renewal work that they complete. PTV does some limited checking of this work but does not analyse the information provided or how effectively franchisees are tracking and managing issues with assets.

As a result, PTV has limited assurance that franchisees' maintenance and renewal work is keeping train and tram assets in optimal working order throughout their life.

3.3.1 Payments

During the current agreements, up to June 2016, the state had paid $1.96 billion to the franchisees to cover the cost of maintaining and renewing train and tram assets—$1.6 billion to MTM and $362 million to Yarra Trams.

The franchisees submit monthly invoices to PTV for the maintenance and renewal work they have undertaken, along with activity reports and other evidence to support their invoices. The franchise agreements prescribe the format and content of these activity reports.

The franchisees have not consistently provided the required information about asset maintenance and renewal on time—in some cases, they were over a year late in providing this information to PTV. PTV withheld payments in these instances, but the delays in providing information have contributed to PTV's lack of knowledge about the actual condition of rail network assets.

3.3.2 Quality assurance over maintenance and renewal work

PTV gains assurance on the quality of franchisees' maintenance and renewal works by:

We found that PTV's checks of the quantity of maintenance and renewal work were adequate, but that it rarely checked the quality of these works because some assets could not be accessed and checked without interfering with the operation of the train and tram network, and others could not be accessed at all. We also found that PTV has not analysed the information and results from these activities to identify trends and issues.

3.3.3 Managing maintenance and renewal issues

PTV has not effectively managed maintenance and renewal issues throughout the current agreements.

Over the course of the MR3 agreements, franchisees have submitted reports about maintenance and renewal work to PTV, which they discussed together at monthly meetings. However, until recently these reports and meetings primarily covered operational issues and progress with maintenance and renewal works.

PTV did not adequately track and resolve maintenance and renewal issues at the meetings, and PTV has not sufficiently analysed maintenance and renewal trends or issues to inform its strategic asset maintenance and renewal planning.

Figure 3B is a case study about the management of operational control and management systems (OCMS) under the MR3 franchise agreements. It highlights the challenges of resolving issues when roles and responsibilities have not been clearly defined.

Figure 3B
Case study: Managing OCMS—black diamond switches

What is a black diamond switch?

Black diamond switches (BDS) are used to control CCTV data transmissions through an optical fibre network. There are four BDS on the train network. They were installed in 2006 and have a life span of between five and seven years. The BDS are part of VicTrack's security network.

Who is responsible for BDS?

VicTrack owns and manages the underlying fibre network and the BDS, but it does not own or manage the peripherals that 'plug' into the network, which are the responsibility of the franchisees, such as cameras, decoders, storage computers and workstations.

Under the telecommunications services agreement (TSA), VicTrack manages, monitors and maintains the network and BDS, and grants the franchisees access so they can plug in their equipment.

What work was required on the BDS?

As demand for CCTV increased across the train network—the number of cameras grew from 2 500 in 2006 to 6 000 in 2016—the BDS were no longer able to transmit the increasing volume of data. In 2015, VicTrack submitted a project proposal to PTV to replace the BDS, at a cost of $1.6 million. At the same time, VicTrack was investigating options for a totally different system, which would mean that the proposed replacement of the BDS would be irrelevant in two to three years. Based on this, PTV asked VicTrack to consider other options for BDS replacement.

At around the same time, MTM also submitted a proposal to PTV for a lower-cost replacement of the BDS, but VicTrack disputed the accuracy of MTM's costing of the proposal.

What was the outcome?

Before the parties could agree on the best solution, MTM went ahead and replaced three of the four BDS. VicTrack prevented MTM from replacing the fourth BDS.

VicTrack and PTV have not been able to confirm how the project was funded or whether MTM received any formal approval, and the dispute remains unresolved. VicTrack advised that PTV is trying to resolve the dispute, despite not having a formal mediation role under the TSA.

What were the issues and findings?

  • VicTrack reported that MTM's actions breached the TSA. VicTrack is aware of the need to balance any penalties or consequences for MTM with the need to continue effective operation of the train network. To date, there is no evidence of any consequences for MTM.
  • It is clear that the BDS needed to be replaced. However, VicTrack has stated that due to MTM's actions it does not have adequate knowledge of the BDS'condition.
  • The BDS are part of a network-wide security system, and it is important that the state, through VicTrack, can adequately monitor and control this system. However, VicTrack says that it is not currently able to do this.

What were the contributing factors?

  • There is inadequate governance of the maintenance and renewal of OCMS.Although there are regular coordination meetings between VicTrack and MTM at officer and executive levels, there is no defined process to resolve issues, and PTV does not attend these meetings.
  • Renewal of OCMS is funded separately to other types of assets. This means that OCMS renewals are managed as individual projects and assessed on a case-by-case basis. In this case, PTV's project management was ineffective.
  • There is no agreed process for approving projects that will be directly funded by the franchisees.
  • VicTrack is not represented in meetings of the Network Development Partnership, in which network-wide interests are discussed.
  • There is no clear and well-managed process for resolving disputes between VicTrack and MTM.

Source: VAGO, based on information from PTV and VicTrack.

While preparing for negotiating the MR4 franchise agreement, PTV has identified opportunities to improve the way OCMS are managed in the next agreement by:

[ Contents of this report | VAGO Home | VAGO Publications ]

4 Extending the franchise agreements

The current train and tram franchise agreements—known as MR3—expire in November 2017. The agreements give franchisees the exclusive right to negotiate with the state to extend the agreement for up to seven years, if they meet certain performance benchmarks.

Public Transport Victoria (PTV) has assessed the current franchisees—Metro Trains Melbourne (MTM) and Yarra Trams—as meeting the required performance benchmarks. PTV and the Department of Economic Development, Jobs, Transport & Resources (DEDJTR) are responsible for negotiating the new agreements with the franchisees. Their primary objective is to 'deliver sustainable franchise agreements and appropriately allocate risk in order to maximise value for money for the state and optimise the operation of the train and tram systems'.

This Part examines whether PTV:

4.1 Conclusion

PTV could not provide evidence to support the rationale and assessment process for some of the performance benchmarks that led to the franchisees earning the right to exclusively negotiate an extension of the franchise agreements. Given the commercial value of such a right, PTV needs to carefully scrutinise the way franchisees have been assessed against the performance benchmarks so that lessons learnt are applied when negotiating future arrangements.

It is encouraging to see that PTV's planning for the negotiations with franchisees has been thorough. PTV has developed an in-depth understanding of the strengths and weaknesses of the current agreements and has identified important, achievable opportunities for improvement.

4.2 The right to negotiate a franchise extension

Under the train and tram franchise agreements, PTV (and, at the time, the former Department of Transport) set benchmarks annually and advised each franchisee of whether they had been met.

However, PTV has not been able to provide sufficient evidence to demonstrate how it or the former Department of Transport determined some of these benchmarks, nor could it demonstrate how franchisees' performance was measured against them.

4.2.1 Contractual arrangements for franchise extension

To earn the right to negotiate a contract extension, franchisees must meet both fixed and flexible benchmarks, and comply with contract and safety requirements. Figure 4A summarises these benchmarks and contractual arrangements.

Figure 4A
Contractual arrangements for franchise extension

Contractual arrangements for franchise extension

Source: VAGO.

If PTV successfully negotiates further seven-year contracts with the existing franchisees, this will bring to an end the current 15-year arrangements. There is no further performance-based right to extend them. PTV must then open the tendering process to other prospective operators.

The MR4 project does not include an evaluation of whether a performance-based right to extend these types of agreements is an effective way to improve franchisees' performance. Without evaluating the effect of this right, PTV will not have a full understanding of whether this approach is useful and whether it should be included in future agreements.

4.2.2 Annual review process and results

The current agreements include a six-year assessment period to determine whether franchisees met the criteria to negotiate a contract extension. At the end of this period, PTV assessed the overall performance of both franchisees and determined that they had met the benchmarks specified in the MR3 agreements. Figures 4B and 4C show the outcome of this assessment.

Figure 4B
MTM's performance against extension benchmarks

Performance benchmark

2009–10(a)

2010–11

2011–12

2012–13

2013–14

2014–15

Fixed benchmarks

Reliability

Customer experience

Flexible benchmarks

Flexible benchmark 1

Flexible benchmark 2

Flexible benchmark 3

Other criteria

No breach of franchise agreement that has not been waived or corrected in assessment period (at 31 December 2015)

Written confirmation from regulator that there are no material matters concerning the franchisee's safety record (at 31 December 2015)

(a) For seven months from 30 November 2009 to 30 June 2010.

Note: achieved, not achieved.

Source: VAGO, based on information provided by PTV.

Figure 4C
Yarra Trams' performance against extension benchmarks

Performance benchmark

2009–10(a)

2010–11

2011–12

2012–13

2013–14

2014–15

Fixed benchmarks

Reliability

Customer experience

Flexible benchmarks

Flexible benchmark 1

Flexible benchmark 2

Flexible benchmark 3

Other criteria

No breach of franchise agreement that has not been waived or corrected in assessment period (at 31 December 2015)

Written confirmation from regulator that there are no material matters concerning the franchisee's safety record (at 31 December 2015)

(a) For seven months from 30 November 2009 to 30 June 2010.

Note: achieved, not achieved.

Source: VAGO, based on information provided by PTV.

On 31 December 2015, PTV informed franchisees of the outcome of its assessment. The state was then obliged to enter into good-faith negotiations with the franchisees, which began in October 2016.

Fixed benchmarks

In 2009, at the beginning of the current agreements, the former Department of Transport determined the two annual fixed performance benchmarks—service reliability and customer experience—based on the performance of the previous franchisees in these areas during the previous year.

The current agreements do not allow for periodic review and resetting of the benchmarks. As a result, PTV has had only a limited ability to use these benchmarks to drive continuous improvement.

Reliability benchmark

The first fixed performance benchmark was a reliability target—that 98.5 per cent of daily services listed in an agreed timetable be delivered. This target was based on the minimum service thresholds explained in Section 2.3 of this report. The thresholds included a provision for PTV to request a meeting and to seek an explanation from the franchisee if their performance fell below a 'call-in' threshold of 98 per cent.

This threshold was determined using data from the 12 months before the commencement of the MR3 franchise agreements. It was based on the franchisees' average performance over this period minus one percentage point. This meant that the performance target of 98.5 per cent was in fact half a percentage point below the average performance achieved by the previous franchisee. As a result, this benchmark was easier for franchisees to achieve and did not act as an incentive for franchisees to improve their performance.

Customer experience benchmark

The second fixed benchmark was that franchisees should not incur a financial penalty under the customer experience performance regime (CEPR). The CEPR was largely abandoned in 2012 due to a range of problems—see Section 2.4 for further discussion.

From 2012, PTV measured franchisees' performance against this benchmark using the results of the quarterly phone survey. PTV's evaluation excluded survey results related to factors that were beyond the franchisees' control, such as ticket prices.

PTV's decision to continue with incentive payments and penalties using subjective phone surveys meant that achieving this benchmark required significantly less effort by each franchisee.

Flexible benchmarks

Flexible benchmarks were intended to provide PTV with the flexibility to focus franchisees on key problem areas or particular initiatives.

Under the franchise agreements, the flexible benchmarks must:

Figure 4D provides some examples of flexible benchmarks for trains and trams.

Figure 4D
Flexible benchmarks for train and tram franchise

Flexible benchmark

Year

Summary

Outcome

Train

Sydenham line performance

2010–11 (Year 2)

Run more trains on time compared to previous year and have trains stop at all stations.

Achieved

Journey Planner submissions

2011–12 (Year 3)

Submit all planned service alterations by noon on a Tuesday—that is, at least 72 hours before first affected service. Submit Christmas and January holiday period timetables by specified dates.

Achieved

On-board and station announcements

2014–15 (Year 6)

Plan an independent audit program by a specified date. Set key performance indicators for improvement based on the outcome of the audit by an agreed date.

Not achieved—audit 'mystery shopper' program introduced and data collected but targets not achieved

Tram

Driver complaints

2009–10 (Year 1)

Receive fewer complaints about drivers compared with previous benchmark period.

Achieved

Passenger falls and pedestrian knockdowns

2010–11 (Year 2)

Have fewer passenger falls and pedestrian knockdowns compared with previous year.

Not achieved—more falls, fewer knockdowns

Special events management

2014–15 (Year 6)

Develop and submit operation and project management plans by specified dates to protect revenue from any adverse impacts from special events (such as the Grand Prix and White Night Festival).

Achieved

Source: VAGO, based on information provided by PTV.

PTV developed the flexible benchmarks and assessed franchisees' performance as part of its day-to-day contract management activities. The process was not subject to formal consultation and assurance processes, and governance was limited to PTV's chief executive officer endorsing the benchmarks and associated outcomes.

Correspondence about the flexible benchmark process shows that it did occur within the required time frames. However, PTV could not provide evidence for the rationale behind some of the benchmarks or how performance had been assessed against them. For example, for three of the six assessment years, PTV could not provide evidence that explained why flexible benchmarks were chosen for MTM or give sufficient detail about how it had assessed MTM's performance.

We found that PTV's consultation and communication with the franchisees about setting and assessing the performance benchmarks was either conducted via email or was not recorded. Given the important role of the performance benchmarks in the franchise agreements, it is surprising that there is no documentation to show how they were established and assessed. It is also contrary to the principles of evidenced-based decision-making.

Despite these shortcomings, there is evidence that some of the flexible benchmarks helped PTV to motivate franchisees to improve their performance in particular areas. The case study in Figure 4E provides an example of a flexible benchmark that resulted in network-wide benefits.

Figure 4E
Case study: Flexible benchmarks in the tram franchise, 2014–2015

Aim of the benchmark

Develop a planning and resourcing methodology for special events:

  • to minimise the impact on regular users
  • to more proactively plan and deploy resources for special events.

Criteria

PTV required Yarra Trams to develop an operational plan for each of the following events in 2014−15:

  • Australian Open
  • St Kilda Festival
  • White Night festival
  • Grand Prix
  • Flower and Garden Show weekend
  • AFL football season to 30 June 2015.

PTV required Yarra Trams to produce the plans four months before each event, and PTV provided feedback within one month of receipt of the plans. The final operations plan had to be finalised one month before the event.

Rationale

PTV reviewed performance during special events and found that events in the CBD put a strain on resources, which had a flow-on effect for regular users. PTV wanted to ensure the delivery of more consistent and reliable core services.

Outcome

PTV assessed Yarra Trams as having met the benchmark because it submitted the special event plans within the specified time frames. However, during the assessment period, Yarra Trams' regular services during special events did not improve. This was due to a number of challenges, including the introduction of the 'Free Tram Zone' and new rolling stock.

To improve its performance, Yarra Trams undertook a detailed analysis of its performance, and implemented an improvement plan in 2015. This resulted in record punctuality in the summer period of 2015–16.

This flexible benchmark resulted in a number of benefits, including:

  • implementation of an event planning committee to resolve strategic issues
  • application of the event planning methodology to the train network
  • improved coordination between PTV and the franchisees during special events.

Source: VAGO, based on information provided by PTV.

4.3 Preparing for the next agreements

PTV's project to prepare for the MR4 franchise agreements has been thorough and gives a sound basis for achieving value for money from the new agreements. However, achieving value for money from these agreements depends on successful negotiation and effective implementation and management of any future franchise arrangements.

4.3.1 Value for money

PTV has a detailed understanding of the financial and non-financial factors that affect the value for money of the current MR3 agreements. It has identified important improvements for the MR4 agreements.

Understanding network operating costs

PTV's analysis of the MR3 agreement found that operating costs increased over the course of the current agreements—particularly for the train network—due to:

Some of these increased costs are funded by the state—such as the costs associated with additional train services—but others have been absorbed by the franchisees.

PTV expects that operating costs will continue to increase throughout the MR4 franchise agreements, largely due to emerging cost pressures, such as:

PTV expects the operating environment to become more challenging during the course of the MR4 franchise agreements—the state will implement a significant transport infrastructure program that may result in disruptions to train and tram services. PTV also estimates that patronage will continue to increase by 3 to 4 per cent annually.

Efficient Operator Comparator

PTV has developed an Efficient Operator Comparator (EOC) to help it demonstrate that the MR4 agreements will deliver value for money. The EOC compares operating costs estimated by the franchisees against those of comparable efficient rail operators. The EOC is an important tool—PTV has recognised that the lack of competitive tension in the negotiation process has increased the need to establish value for money.

PTV's development of the EOC has been guided by Partnerships Victoria's guidelines for public sector comparison tools. Although the EOC uses a different model, it shares some fundamental similarities and principles with Partnerships Victoria's framework. We found that PTV used this framework appropriately, modifying it where needed to suit its particular circumstances.

To understand franchisees' operating costs, PTV worked with MTM and Yarra Trams and used a transparent and collaborative 'open book' approach. PTV identified weaknesses in the way it has collected financial information from franchisees under the MR3 agreements. PTV also identified opportunities to improve franchisees' financial reporting—for example, by requiring more comprehensive information from franchisees and making sure that it has the systems and capability to analyse and understand trends.

Allocating and managing risks appropriately throughout the life of the agreement is fundamental to achieving value for money from the agreements. PTV and the franchisees need to be clear about who bears the cost if a risk to the operation of train and tram services eventuates—for example, if ticket revenue does not increase as predicted or if a new timetable is delayed.

In our 2005 audit Franchising Melbourne's Train and Tram System, we recommended that the former Department of Infrastructure clearly outline which risks were to be retained by the state and which were to be transferred to the franchisee. We found that PTV has addressed this recommendation. It engaged external contractors to assess the rigour of the EOC, and this continuing work will result in ongoing adjustments to the EOC.

We found that the EOC documents demonstrated detailed analysis of past performance, franchisee costs, environmental context and an understanding of the causes of any increasing or decreasing cost. This work—completed by PTV's subject matter experts, project leads and DEDJTR—has given PTV greater assurance that its assumptions about operating costs are well informed.

Identifying strengths and weaknesses in the franchise agreements

PTV's MR4 project was divided into streams that created clear accountability and use of subject-matter expertise in areas of commercial significance. After consultation and research, PTV developed position papers that were endorsed at the relevant governance forums.

The position papers include detailed background, identify weaknesses in the MR3 agreements and provide options to address them. PTV has appropriately benchmarked franchisees' performance against past data and the performance of other national and international networks.

PTV has identified a range of ways that value for money can be improved in the MR4 agreements, such as:

Some examples of strengths, weaknesses and opportunities for improvement are summarised in Figure 4F.

Figure 4F
Priority issues for the MR4 agreements

Position paper/ priority issue

Summary of issue

Factors to consider for MR4

Ticket revenue

Ticket revenue is currently shared with franchisees but it is affected by a range of factors, many of which cannot be controlled by franchisees. Current ticket revenue sharing mechanisms include setting upper and lower limits on revenue sharing, and conducting periodic resets to mitigate risks. Ticket revenue was lower than predicted during MR3.

  • There is no compelling evidence that sharing the risks connected with ticket revenue contributes to growth in patronage.
  • It will be challenging to increase ticket revenue under the MR4 agreements.
  • It is important that franchisees have incentives to increase patronage.

Accessibility

Disability legislative requirements and compliance targets presented a challenge in MR3 and this is likely to continue. The state retains responsibility for the capital expenditure needed to comply with accessibility requirements, and franchisees are responsible for operational requirements.

  • Develop an accessibility action plan, increased reporting and risk management requirements.
  • Consider incorporating some franchisee requirements in the agreements.

Management of projects

Project agreements have been largely successful, but need some fine-tuning of profit margins and adjustment to PTV's oversight of franchisee project management.

  • Explore opportunities to review franchisee profit margins.
  • Improve oversight of project management by franchisees.

Source: VAGO, based on information provided by PTV.

PTV's ability to improve train and tram services will depend on the success of its contract negotiations and its implementation of the new agreements. PTV's new contract management framework, described in Part 2, includes periodic reviews of the MR4 agreements throughout their life span to determine whether the agreements are meeting their objectives and providing the anticipated benefits. This will help PTV determine any changes needed to the MR4 agreements and prepare for future agreements.

4.3.2 MR4 project management

PTV began planning for the MR4 franchise negotiations in early 2015, with a budget of $25.7 million, plus contingency, over three years. To manage this project effectively, PTV requires considerable expertise, rigorous planning and clearly defined objectives, scope and governance structure.

The MR4 franchise negotiation is not a standard procurement process because it involves exclusive negotiations, so it does not fit within the policies and guidelines of the Victorian Government Procurement Board or Partnerships Victoria. It is also not subject to the Department of Treasury & Finance's (DTF) High Value High Risk process.

Nevertheless, the principles of effective procurement and project management still apply, as does PTV's Investment Lifecycle Framework process, which is consistent with DTF's Investment Lifecycle Framework.

We found that PTV's project team has developed a detailed strategic procurement plan, which clearly outlines objectives, time frames and a structure for the project's governance. Further details on the governance structure and roles and responsibilities of key groups are provided in Appendix D.

One gap we identified in the governance structure was the exclusion of VicTrack, the owner of the state's rail assets and infrastructure. DEDJTR explained that VicTrack was not included in the governance structure because the meetings covered broad policy matters that were not related to assets. However, because rail assets and infrastructure play an important role in the functioning of the network, VicTrack should have been formally included in the MR4 project governance structure. DEDJTR has partly addressed this gap by establishing the Public Transport Asset Management Steering Committee, which includes representatives from VicTrack.

PTV has adequately considered probity requirements by engaging a probity adviser, and it has also committed to hiring an independent probity auditor.

We also found PTV has developed an appropriate stakeholder engagement and communication plan and kept a detailed stakeholder engagement record. The governance bodies and workshops have facilitated effective consultation within PTV and between PTV and other departments and external stakeholders. This stakeholder engagement has helped PTV to develop a detailed understanding of the strengths and weaknesses of the MR3 agreements and identify opportunities for improvement.

[ Contents of this report | VAGO Home | VAGO Publications ]

Appendix A. Audit Act 1994 section 16—submissions and comments

We have consulted with the Department of Economic Development, Jobs & Transport, Public Transport Victoria, and Victorian Rail Track throughout the course of our audits, and we considered their views when reaching our audit conclusions. As required by section 16(3) of the Audit Act 1994, we gave a draft copy of this report, or relevant extracts, to those agencies and asked for their submissions and comments.

Responsibility for the accuracy, fairness and balance of those comments rests solely with the agency head.

Responses were received as follows:

RESPONSE provided by the Secretary, Department of Premier & Cabinet

RESPONSE provided by the Secretary, Department of Premier & Cabinet

RESPONSE provided by the Secretary, Department of Economic Development, Jobs, Transport & Resources

RESPONSE provided by the Secretary, Department of Economic Development, Jobs, Transport & Resources

RESPONSE provided by the Chief Executive Officer, Public Transport Victoria

RESPONSE provided by the Secretary, Department of Economic Development, Jobs, Transport & Resources

RESPONSE provided by the Chief Executive Officer, Public Transport Victoria – continued

RESPONSE provided by the Chief Executive Officer, Public Transport Victoria – continued

RESPONSE provided by the Chief Executive Officer, Public Transport Victoria – continued

RESPONSE provided by the Chief Executive Officer, Public Transport Victoria – continued
[ Contents of this report | VAGO Home | VAGO Publications ]

Appendix B. Past VAGO audits

Figure B1 summarises past audits with similar themes and findings.

Figure B1 Past VAGO audits and findings

Audit

Audit objective

Findings relevant to this audit

Operational Effectiveness of myki Ticketing June 2015

Examine myki's operational effectiveness, and whether the outcomes and benefits expected from its introduction were being achieved, and how well Public Transport Victoria (PTV) was preparing for re-tendering.

  • PTV does not adequately assure the reliability of results reported by the contractor underpinning incentive payments.
  • PTV needs to strengthen its monitoring of myki prior to awarding the new contract.
  • There was no assessment of whether myki had achieved its proposed benefits and objectives.

Tendering of Metropolitan Bus Contracts May 2015

Assess the planning and conduct of the bus tender process and PTV's management of the contracts.

  • The contracts did not achieve value for money due to PTV's failure to properly implement the performance regime and long-standing issues with data reliability.
  • We identified gaps in PTV's contract management system and processes, including the transactional focus and lack of clear accountabilities.

Coordinating Public Transport August 2014

Assess PTV's effectiveness in coordinating public transport services.

  • Performance monitoring is limited by an over-reliance on self-reporting by operators, inefficient manual systems and minimal quality assurance.

Using ICT to Improve Traffic Management June 2014

Assess the effectiveness of the use of ICT systems to manage traffic flow.

  • There is no statewide approach to managing traffic congestion, which affects tram performance.
  • The tram tracking system is obsolete and is unable to interact with traffic signals, causing ineffective traffic flow.

Public Transport Performance February 2012

Examine effectiveness of the state's public transport services by assessing whether agencies had delivered effective public transport services and were adequately prepared to improve and sustain performance in the future

  • The Department of Transport (the department) was not prepared to effectively manage the increase in public transport patronage that occurred between 2004 and 2009.
  • There were weaknesses in performance measurement and reporting, and some aspects of performance were not being measured at all.
  • The department managed public transport as a collection of separate modes of transport rather than as an integrated system.

Maintaining Victoria's Rail Infrastructure Assets May 2007

 

Assess how effectively the state's rail infrastructure was being maintained and renewed.

  • Overall maintenance arrangements for metropolitan rail infrastructure were satisfactory, and the condition and performance of metropolitan infrastructure was 'fit for purpose' at the current level of operation.
  • Improvements could be made by the development of a long-term asset management strategy detailing the rationale for maintenance and renewal of assets, clear documentation of the methodology for prioritising maintenance and renewal, and review of key performance indicators so that they better describe infrastructure condition, performance and the effectiveness of maintenance and renewal works.

Franchising Melbourne's Train and Tram System September 2005

Determine whether 2004 franchise agreements for the metropolitan train and tram system represented value for money.

  • The renegotiated contract arrangements did represent value for money, assuming that the service providers met contractual requirements.
  • The government did not adequately consider risk allocation.

Source: VAGO.

[ Contents of this report | VAGO Home | VAGO Publications ]

Appendix C. Monitoring franchisee performance

Operational performance

Public Transport Victoria (PTV) measures operational performance in two main ways, summarised in Figure C1.

Figure C1
Train and tram operational performance measures

Publicly reported service reliability and punctuality—percentage-based measures

Operational performance regime used for incentive payments and penalties

Measures

Reliability—percentage of timetable delivered.

Punctuality—percentage of services that arrive no more than one minute early or four minutes 59 seconds late.

The impact on passengers of an individual service delay or cancellation, taking into account:

  • the type of service disruption (such as cancellation, late arrival, early departure)
  • the extent of the disruption (in minutes)
  • the number of people affected.

Measurement method

Trains—franchisee records on paper the arrival times of trains at the destination, which is periodically audited by PTV. This continued until March 2015 when an automated system was introduced, which increased monitoring points.

Trams—measured using Automatic Vehicle Monitoring system, which records information at several points along each tram journey.

Relies on same service delivery and punctuality measurements.

Franchisee self-reports the causes of delays.

Publicly reported

Publicly reported in PTV's Track Record.

Not publicly reported.

Incentives

No financial incentive or reward.

Failure to meet thresholds can lead to:

  • call in (please explain)
  • contract breach
  • contract termination.

Linked to financial incentive and penalty scheme (capped at $1.1 million per month for trains, $0.55 million per month for trams).

External events

Does not exclude events that are outside the franchisee's control.

Excludes unusual, material events outside the franchisee's control (such as extreme storms, customer incidents).

Assessment, thresholds and targets

Results are assessed quarterly against established minimum thresholds equal to the 'call in' threshold. These were set at the commencement of MR3 based on past performance. These are not subject to any periodic reviews or resets.

Reliability—98 per cent (train and tram).

Punctuality—train 87 per cent and tram 77 per cent.

Assessed monthly against thresholds and benchmarks that were set at the commencement of MR3 based on past performance. The agreement stipulates that benchmarks can be reviewed and varied in certain circumstances, such as timetable changes.

Source: VAGO.

Operational Performance Regime

The Operational Performance Regime (OPR) bonus and penalty payments are determined based on passenger weighted minutes (PWM) of delay. PWM measures the impact of an individual service delay or cancellation, is weighted for peak times and makes allowances for events outside the control of franchisees, such as public trespassing or illness.

Figure C2 provides a summary of how PWM is calculated.

Figure C2
Summary of payment calculation for OPR

Summary of payment calculation for OPR

Source: VAGO, based on information provided by PTV.

At the beginning of MR3 an annual target was developed based on the franchisee performance for the 12 months ending 31 June 2009.

The franchise agreements specify the procedures for periodic adjustments of the annual targets, typically aligned with service changes such as route alterations or journey time changes.

[ Contents of this report | VAGO Home | VAGO Publications ]

Appendix D. MR4 project governance

Public Transport Victoria (PTV) established a project in 2015 to prepare for the next franchise agreements—known as the MR4 project.

Figure D1
MR4 project governance structure

MR4 project governance structure

Source: PTV.

Roles and responsibilities

The Network Strategies and Policy Alignment Group is the main inter-departmental governance body and includes representatives from PTV, the Department of Economic Development, Jobs, Transport & Resources, the Department of Treasury & Finance and the Department of Premier & Cabinet. This group has functioned effectively to monitor project progress and risks, and provide support and direction to the MR4 director. This group also endorses positions and makes recommendations to the minister.

The Train, Tram & Bus Recontracting Committee provided an important forum for testing ideas, ensuring alignment with government policy and encouraging collaboration between departments. After initial meetings, the function of the committee was reassessed and it was replaced by other inter-agency working groups, such as the contract design groups.

PTV's internal governance of the MR4 project occurs at the Franchising and Major Projects Executive Committee, which is chaired by the PTV chief executive and comprises senior PTV executives. This group met monthly until February 2016, after which oversight of the project has been provided mainly by the Network Strategies and Policy Alignment Group. Project status reports are provided to the Enterprise Project Management Office at PTV where they are collated. Bi-monthly project updates are provided to the PTV board.

[ Contents of this report | VAGO Home | VAGO Publications ]