Managing the Performance of Rail Franchisees

Tabled: 7 December 2016

4 Extending the franchise agreements

The current train and tram franchise agreements—known as MR3—expire in November 2017. The agreements give franchisees the exclusive right to negotiate with the state to extend the agreement for up to seven years, if they meet certain performance benchmarks.

Public Transport Victoria (PTV) has assessed the current franchisees—Metro Trains Melbourne (MTM) and Yarra Trams—as meeting the required performance benchmarks. PTV and the Department of Economic Development, Jobs, Transport & Resources (DEDJTR) are responsible for negotiating the new agreements with the franchisees. Their primary objective is to 'deliver sustainable franchise agreements and appropriately allocate risk in order to maximise value for money for the state and optimise the operation of the train and tram systems'.

This Part examines whether PTV:

  • has effectively managed its assessment of franchisee performance
  • is effectively managing its preparation for the next franchise agreements, known as MR4
  • is well positioned to improve services and value for money in the MR4 agreements.

4.1 Conclusion

PTV could not provide evidence to support the rationale and assessment process for some of the performance benchmarks that led to the franchisees earning the right to exclusively negotiate an extension of the franchise agreements. Given the commercial value of such a right, PTV needs to carefully scrutinise the way franchisees have been assessed against the performance benchmarks so that lessons learnt are applied when negotiating future arrangements.

It is encouraging to see that PTV's planning for the negotiations with franchisees has been thorough. PTV has developed an in-depth understanding of the strengths and weaknesses of the current agreements and has identified important, achievable opportunities for improvement.

4.2 The right to negotiate a franchise extension

Under the train and tram franchise agreements, PTV (and, at the time, the former Department of Transport) set benchmarks annually and advised each franchisee of whether they had been met.

However, PTV has not been able to provide sufficient evidence to demonstrate how it or the former Department of Transport determined some of these benchmarks, nor could it demonstrate how franchisees' performance was measured against them.

4.2.1 Contractual arrangements for franchise extension

To earn the right to negotiate a contract extension, franchisees must meet both fixed and flexible benchmarks, and comply with contract and safety requirements. Figure 4A summarises these benchmarks and contractual arrangements.

Figure 4A
Contractual arrangements for franchise extension

Contractual arrangements for franchise extension

Source: VAGO.

If PTV successfully negotiates further seven-year contracts with the existing franchisees, this will bring to an end the current 15-year arrangements. There is no further performance-based right to extend them. PTV must then open the tendering process to other prospective operators.

The MR4 project does not include an evaluation of whether a performance-based right to extend these types of agreements is an effective way to improve franchisees' performance. Without evaluating the effect of this right, PTV will not have a full understanding of whether this approach is useful and whether it should be included in future agreements.

4.2.2 Annual review process and results

The current agreements include a six-year assessment period to determine whether franchisees met the criteria to negotiate a contract extension. At the end of this period, PTV assessed the overall performance of both franchisees and determined that they had met the benchmarks specified in the MR3 agreements. Figures 4B and 4C show the outcome of this assessment.

Figure 4B
MTM's performance against extension benchmarks

Performance benchmark

2009–10(a)

2010–11

2011–12

2012–13

2013–14

2014–15

Fixed benchmarks

Reliability

Customer experience

Flexible benchmarks

Flexible benchmark 1

Flexible benchmark 2

Flexible benchmark 3

Other criteria

No breach of franchise agreement that has not been waived or corrected in assessment period (at 31 December 2015)

Written confirmation from regulator that there are no material matters concerning the franchisee's safety record (at 31 December 2015)

(a) For seven months from 30 November 2009 to 30 June 2010.

Note: achieved, not achieved.

Source: VAGO, based on information provided by PTV.

Figure 4C
Yarra Trams' performance against extension benchmarks

Performance benchmark

2009–10(a)

2010–11

2011–12

2012–13

2013–14

2014–15

Fixed benchmarks

Reliability

Customer experience

Flexible benchmarks

Flexible benchmark 1

Flexible benchmark 2

Flexible benchmark 3

Other criteria

No breach of franchise agreement that has not been waived or corrected in assessment period (at 31 December 2015)

Written confirmation from regulator that there are no material matters concerning the franchisee's safety record (at 31 December 2015)

(a) For seven months from 30 November 2009 to 30 June 2010.

Note: achieved, not achieved.

Source: VAGO, based on information provided by PTV.

On 31 December 2015, PTV informed franchisees of the outcome of its assessment. The state was then obliged to enter into good-faith negotiations with the franchisees, which began in October 2016.

Fixed benchmarks

In 2009, at the beginning of the current agreements, the former Department of Transport determined the two annual fixed performance benchmarks—service reliability and customer experience—based on the performance of the previous franchisees in these areas during the previous year.

The current agreements do not allow for periodic review and resetting of the benchmarks. As a result, PTV has had only a limited ability to use these benchmarks to drive continuous improvement.

Reliability benchmark

The first fixed performance benchmark was a reliability target—that 98.5 per cent of daily services listed in an agreed timetable be delivered. This target was based on the minimum service thresholds explained in Section 2.3 of this report. The thresholds included a provision for PTV to request a meeting and to seek an explanation from the franchisee if their performance fell below a 'call-in' threshold of 98 per cent.

This threshold was determined using data from the 12 months before the commencement of the MR3 franchise agreements. It was based on the franchisees' average performance over this period minus one percentage point. This meant that the performance target of 98.5 per cent was in fact half a percentage point below the average performance achieved by the previous franchisee. As a result, this benchmark was easier for franchisees to achieve and did not act as an incentive for franchisees to improve their performance.

Customer experience benchmark

The second fixed benchmark was that franchisees should not incur a financial penalty under the customer experience performance regime (CEPR). The CEPR was largely abandoned in 2012 due to a range of problems—see Section 2.4 for further discussion.

From 2012, PTV measured franchisees' performance against this benchmark using the results of the quarterly phone survey. PTV's evaluation excluded survey results related to factors that were beyond the franchisees' control, such as ticket prices.

PTV's decision to continue with incentive payments and penalties using subjective phone surveys meant that achieving this benchmark required significantly less effort by each franchisee.

Flexible benchmarks

Flexible benchmarks were intended to provide PTV with the flexibility to focus franchisees on key problem areas or particular initiatives.

Under the franchise agreements, the flexible benchmarks must:

  • reflect the franchisees' role in meeting the passenger-related objectives included in Victoria's public transport policy during the franchise period
  • be reasonably achievable in the relevant assessment year without requiring any additional capital investment
  • be developed in consultation with the franchisee
  • must specify a reasonable way for the franchisee to demonstrate compliance if objective measurement is not possible.

Figure 4D provides some examples of flexible benchmarks for trains and trams.

Figure 4D
Flexible benchmarks for train and tram franchise

Flexible benchmark

Year

Summary

Outcome

Train

Sydenham line performance

2010–11 (Year 2)

Run more trains on time compared to previous year and have trains stop at all stations.

Achieved

Journey Planner submissions

2011–12 (Year 3)

Submit all planned service alterations by noon on a Tuesday—that is, at least 72 hours before first affected service. Submit Christmas and January holiday period timetables by specified dates.

Achieved

On-board and station announcements

2014–15 (Year 6)

Plan an independent audit program by a specified date. Set key performance indicators for improvement based on the outcome of the audit by an agreed date.

Not achieved—audit 'mystery shopper' program introduced and data collected but targets not achieved

Tram

Driver complaints

2009–10 (Year 1)

Receive fewer complaints about drivers compared with previous benchmark period.

Achieved

Passenger falls and pedestrian knockdowns

2010–11 (Year 2)

Have fewer passenger falls and pedestrian knockdowns compared with previous year.

Not achieved—more falls, fewer knockdowns

Special events management

2014–15 (Year 6)

Develop and submit operation and project management plans by specified dates to protect revenue from any adverse impacts from special events (such as the Grand Prix and White Night Festival).

Achieved

Source: VAGO, based on information provided by PTV.

PTV developed the flexible benchmarks and assessed franchisees' performance as part of its day-to-day contract management activities. The process was not subject to formal consultation and assurance processes, and governance was limited to PTV's chief executive officer endorsing the benchmarks and associated outcomes.

Correspondence about the flexible benchmark process shows that it did occur within the required time frames. However, PTV could not provide evidence for the rationale behind some of the benchmarks or how performance had been assessed against them. For example, for three of the six assessment years, PTV could not provide evidence that explained why flexible benchmarks were chosen for MTM or give sufficient detail about how it had assessed MTM's performance.

We found that PTV's consultation and communication with the franchisees about setting and assessing the performance benchmarks was either conducted via email or was not recorded. Given the important role of the performance benchmarks in the franchise agreements, it is surprising that there is no documentation to show how they were established and assessed. It is also contrary to the principles of evidenced-based decision-making.

Despite these shortcomings, there is evidence that some of the flexible benchmarks helped PTV to motivate franchisees to improve their performance in particular areas. The case study in Figure 4E provides an example of a flexible benchmark that resulted in network-wide benefits.

Figure 4E
Case study: Flexible benchmarks in the tram franchise, 2014–2015

Aim of the benchmark

Develop a planning and resourcing methodology for special events:

  • to minimise the impact on regular users
  • to more proactively plan and deploy resources for special events.

Criteria

PTV required Yarra Trams to develop an operational plan for each of the following events in 2014−15:

  • Australian Open
  • St Kilda Festival
  • White Night festival
  • Grand Prix
  • Flower and Garden Show weekend
  • AFL football season to 30 June 2015.

PTV required Yarra Trams to produce the plans four months before each event, and PTV provided feedback within one month of receipt of the plans. The final operations plan had to be finalised one month before the event.

Rationale

PTV reviewed performance during special events and found that events in the CBD put a strain on resources, which had a flow-on effect for regular users. PTV wanted to ensure the delivery of more consistent and reliable core services.

Outcome

PTV assessed Yarra Trams as having met the benchmark because it submitted the special event plans within the specified time frames. However, during the assessment period, Yarra Trams' regular services during special events did not improve. This was due to a number of challenges, including the introduction of the 'Free Tram Zone' and new rolling stock.

To improve its performance, Yarra Trams undertook a detailed analysis of its performance, and implemented an improvement plan in 2015. This resulted in record punctuality in the summer period of 2015–16.

This flexible benchmark resulted in a number of benefits, including:

  • implementation of an event planning committee to resolve strategic issues
  • application of the event planning methodology to the train network
  • improved coordination between PTV and the franchisees during special events.

Source: VAGO, based on information provided by PTV.

4.3 Preparing for the next agreements

PTV's project to prepare for the MR4 franchise agreements has been thorough and gives a sound basis for achieving value for money from the new agreements. However, achieving value for money from these agreements depends on successful negotiation and effective implementation and management of any future franchise arrangements.

4.3.1 Value for money

PTV has a detailed understanding of the financial and non-financial factors that affect the value for money of the current MR3 agreements. It has identified important improvements for the MR4 agreements.

Understanding network operating costs

PTV's analysis of the MR3 agreement found that operating costs increased over the course of the current agreements—particularly for the train network—due to:

  • increases in wages of franchisee staff and the introduction of new roles to support service performance
  • increases in the number of kilometres travelled by trains and trams
  • increases in the cost of infrastructure compared to the MR2 agreements
  • the operation of additional premium and host stations, line extensions, and maintenance of additional facilities on the train network.

Some of these increased costs are funded by the state—such as the costs associated with additional train services—but others have been absorbed by the franchisees.

PTV expects that operating costs will continue to increase throughout the MR4 franchise agreements, largely due to emerging cost pressures, such as:

  • wage increases agreed in recent industrial agreements
  • the cost of maintaining and renewing ageing train and tram fleets
  • increased staffing levels to improve service delivery—for example, the introduction of driver managers to monitor and support driver performance
  • uncertainty about how much funding will be required to maintain and renew assets
  • the need to renew operational control systems
  • uncertainty about ticket revenue because of significant future transport projects, such as Melbourne Metro Rail.

PTV expects the operating environment to become more challenging during the course of the MR4 franchise agreements—the state will implement a significant transport infrastructure program that may result in disruptions to train and tram services. PTV also estimates that patronage will continue to increase by 3 to 4 per cent annually.

Efficient Operator Comparator

PTV has developed an Efficient Operator Comparator (EOC) to help it demonstrate that the MR4 agreements will deliver value for money. The EOC compares operating costs estimated by the franchisees against those of comparable efficient rail operators. The EOC is an important tool—PTV has recognised that the lack of competitive tension in the negotiation process has increased the need to establish value for money.

PTV's development of the EOC has been guided by Partnerships Victoria's guidelines for public sector comparison tools. Although the EOC uses a different model, it shares some fundamental similarities and principles with Partnerships Victoria's framework. We found that PTV used this framework appropriately, modifying it where needed to suit its particular circumstances.

To understand franchisees' operating costs, PTV worked with MTM and Yarra Trams and used a transparent and collaborative 'open book' approach. PTV identified weaknesses in the way it has collected financial information from franchisees under the MR3 agreements. PTV also identified opportunities to improve franchisees' financial reporting—for example, by requiring more comprehensive information from franchisees and making sure that it has the systems and capability to analyse and understand trends.

Allocating and managing risks appropriately throughout the life of the agreement is fundamental to achieving value for money from the agreements. PTV and the franchisees need to be clear about who bears the cost if a risk to the operation of train and tram services eventuates—for example, if ticket revenue does not increase as predicted or if a new timetable is delayed.

In our 2005 audit Franchising Melbourne's Train and Tram System, we recommended that the former Department of Infrastructure clearly outline which risks were to be retained by the state and which were to be transferred to the franchisee. We found that PTV has addressed this recommendation. It engaged external contractors to assess the rigour of the EOC, and this continuing work will result in ongoing adjustments to the EOC.

We found that the EOC documents demonstrated detailed analysis of past performance, franchisee costs, environmental context and an understanding of the causes of any increasing or decreasing cost. This work—completed by PTV's subject matter experts, project leads and DEDJTR—has given PTV greater assurance that its assumptions about operating costs are well informed.

Identifying strengths and weaknesses in the franchise agreements

PTV's MR4 project was divided into streams that created clear accountability and use of subject-matter expertise in areas of commercial significance. After consultation and research, PTV developed position papers that were endorsed at the relevant governance forums.

The position papers include detailed background, identify weaknesses in the MR3 agreements and provide options to address them. PTV has appropriately benchmarked franchisees' performance against past data and the performance of other national and international networks.

PTV has identified a range of ways that value for money can be improved in the MR4 agreements, such as:

  • measuring franchisees' operational performance at a greater number of monitoring points, allowing PTV to address and penalise undesirable practices such as skipping stations
  • setting higher reliability and punctuality thresholds than those set for the MR3 agreements
  • focusing on overall customer experience, including better fleet and station presentation, customer information and assistance, achievement of minimum standards and a renewed incentive regime
  • implementing a more strategic approach to maintaining, renewing and managing assets to improve understanding and monitoring of asset condition and the associated risks
  • reducing the risks associated with the current disparate operational control and management systems
  • facilitating major network changes (for example, the Melbourne Metro Tunnel) to improve the operation of the broader network
  • developing more restrictive standards for advertising to align with community expectations.

Some examples of strengths, weaknesses and opportunities for improvement are summarised in Figure 4F.

Figure 4F
Priority issues for the MR4 agreements

Position paper/ priority issue

Summary of issue

Factors to consider for MR4

Ticket revenue

Ticket revenue is currently shared with franchisees but it is affected by a range of factors, many of which cannot be controlled by franchisees. Current ticket revenue sharing mechanisms include setting upper and lower limits on revenue sharing, and conducting periodic resets to mitigate risks. Ticket revenue was lower than predicted during MR3.

  • There is no compelling evidence that sharing the risks connected with ticket revenue contributes to growth in patronage.
  • It will be challenging to increase ticket revenue under the MR4 agreements.
  • It is important that franchisees have incentives to increase patronage.

Accessibility

Disability legislative requirements and compliance targets presented a challenge in MR3 and this is likely to continue. The state retains responsibility for the capital expenditure needed to comply with accessibility requirements, and franchisees are responsible for operational requirements.

  • Develop an accessibility action plan, increased reporting and risk management requirements.
  • Consider incorporating some franchisee requirements in the agreements.

Management of projects

Project agreements have been largely successful, but need some fine-tuning of profit margins and adjustment to PTV's oversight of franchisee project management.

  • Explore opportunities to review franchisee profit margins.
  • Improve oversight of project management by franchisees.

Source: VAGO, based on information provided by PTV.

PTV's ability to improve train and tram services will depend on the success of its contract negotiations and its implementation of the new agreements. PTV's new contract management framework, described in Part 2, includes periodic reviews of the MR4 agreements throughout their life span to determine whether the agreements are meeting their objectives and providing the anticipated benefits. This will help PTV determine any changes needed to the MR4 agreements and prepare for future agreements.

4.3.2 MR4 project management

PTV began planning for the MR4 franchise negotiations in early 2015, with a budget of $25.7 million, plus contingency, over three years. To manage this project effectively, PTV requires considerable expertise, rigorous planning and clearly defined objectives, scope and governance structure.

The MR4 franchise negotiation is not a standard procurement process because it involves exclusive negotiations, so it does not fit within the policies and guidelines of the Victorian Government Procurement Board or Partnerships Victoria. It is also not subject to the Department of Treasury & Finance's (DTF) High Value High Risk process.

Nevertheless, the principles of effective procurement and project management still apply, as does PTV's Investment Lifecycle Framework process, which is consistent with DTF's Investment Lifecycle Framework.

We found that PTV's project team has developed a detailed strategic procurement plan, which clearly outlines objectives, time frames and a structure for the project's governance. Further details on the governance structure and roles and responsibilities of key groups are provided in Appendix D.

One gap we identified in the governance structure was the exclusion of VicTrack, the owner of the state's rail assets and infrastructure. DEDJTR explained that VicTrack was not included in the governance structure because the meetings covered broad policy matters that were not related to assets. However, because rail assets and infrastructure play an important role in the functioning of the network, VicTrack should have been formally included in the MR4 project governance structure. DEDJTR has partly addressed this gap by establishing the Public Transport Asset Management Steering Committee, which includes representatives from VicTrack.

PTV has adequately considered probity requirements by engaging a probity adviser, and it has also committed to hiring an independent probity auditor.

We also found PTV has developed an appropriate stakeholder engagement and communication plan and kept a detailed stakeholder engagement record. The governance bodies and workshops have facilitated effective consultation within PTV and between PTV and other departments and external stakeholders. This stakeholder engagement has helped PTV to develop a detailed understanding of the strengths and weaknesses of the MR3 agreements and identify opportunities for improvement.

Back to Top