We examined if VicRoads has implemented an evaluation framework to measure if the Top 20 Program is reducing run-off-road and head-on serious casualty crashes as intended. We also undertook our own analysis of how effective flexible barriers are on the top 20 roads.
VicRoads cannot determine how effective flexible barriers are at reducing run off-road and head-on serious casualty crashes on the top 20 roads. This is because VicRoads has not yet evaluated any of the completed Top 20 Program projects and does not plan to until 2026. This means that VicRoads does not know if TAC’s significant investment in flexible barriers is achieving the intended safety outcomes, or that it will be as cost-effective as intended.
VicRoads’ inadequate record keeping will hinder its ability to evaluate the Top 20 Program. VicRoads does not record the exact location of flexible barriers and its crash data does not track if and when a vehicle hits them. Consequently, VicRoads cannot determine which serious casualty crashes have involved flexible barriers or identify when a flexible barrier has not worked correctly.
Our limited evaluation found that the Top 20 Program is not on target to achieve the expected reductions in run-off-road and head-on serious casualty crashes stated in its investment plans and project proposals. Our assessment of 18 completed projects determined that VicRoads’ safety treatments (including flexible barriers, wide centrelines and rumble strips) have reduced run-off-road and head-on serious casualty crashes by 46.5 per cent (with a 95 per cent confidence interval of 25.6 to 65.6 per cent). If these results persist, then the program’s projects are not likely to meet their stated benefit cost ratios.
4.2 Key performance indicators
DTF’s Investment Lifecycle and High Value High Risk Guidelines expect agencies to monitor if their projects are achieving the intended benefits. To do this, DTF recommends that agencies develop a benefits management plan. This plan should include:
- KPIs, including the baseline, targets, time frames and how the agency will measure them
- a schedule for reporting results.
VicRoads did not develop a benefits management plan or equivalent for the Top 20 Program. This means that there is no evidence of how and when it plans to evaluate the program’s outcomes.
VicRoads did not outline formal KPIs for its Top 20 Program projects. The crash reduction factors in its investment plans and project proposals could function as KPIs to an extent though.
Despite this, crash reduction factors are not the equivalent of KPIs because their measures, baselines and time frames lack detail. Without this detail, TAC and VicRoads cannot reliably measure if flexible barriers are achieving their intended safety benefits.
If VicRoads uses crash reduction factors to measure its Top 20 Program projects’ success, then it will need to address our findings in Part 2 about the validity of crash reduction factors and how it calculates them.
Reporting against KPIs
According to DTF, agencies should outline how they will report on KPIs in their benefits management plan, including:
- who the reports should go to
- when reporting will start and end
- what the reporting frequency will be
- who is responsible for reporting.
By outlining these requirements at the start of a project, the responsible agency holds itself accountable for reporting on the expected benefits.
VicRoads internally reports on fatality crashes and tracks the number of FSI crashes that occur on the top 20 roads. These reports are graphs that show the number of FSI crashes on a treated road before and after it installed flexible barriers (see Figure 4A as an example). VicRoads does not consistently and regularly report this information though.
Total FSI crashes for Calder Freeway (Melbourne to Bendigo) between 2014 and September 2019
Source: VAGO, based on VicRoads’ information.
VicRoads develops these reports when the Minister for Road Safety, Regional Roads Victoria or DoT and VicRoads’ executive teams request them. VicRoads advised that it does not distribute these graphs publicly because they are not formal statistically valid evaluations. Instead, they are simplistic before and after snapshots of crash numbers that uninformed readers could misinterpret. This is because they do not explain or account for the tendency for the number of crashes per year to move closer to the average over time.
4.3 Program evaluation
VicRoads has not yet developed a plan to evaluate its Top 20 Program projects.
In 2017, VicRoads commissioned a consulting firm to develop an evaluation framework for the Top 20 Program. The consultant developed guidance for a short to medium-term evaluation and a long-term evaluation. This guidance included key evaluation questions and sampling methods. VicRoads is yet to document how it will apply this guidance to evaluate the program. It also lacks documented plans for what data it will need to evaluate the program or how it will obtain it.
Short to medium-term evaluation
While VicRoads has not defined its evaluation plan for its Top 20 Program projects, it has started contracting for the formal short to medium-term evaluation. This evaluation focuses on the program’s behavioural effects, such as lane changing, overtaking and tailgating behaviour, rather than FSI crash rates.
As VicRoads has not finished the short to medium term evaluation, no results are available yet.
VicRoads has not commenced its long-term evaluation of the Top 20 Program. VicRoads advised that it expects to start this in June 2021 and complete it in June 2026. VicRoads has identified some aspects of the long-term evaluation, such as KPIs, approvals and procurement, but could not tell us what methodology it will use.
VicRoads advised that it needs to wait for three to five years after it completes a project before it can start a long-term evaluation of it. This is to make sure that it has a sufficient amount of post-treatment data to analyse.
VicRoads has a history of completing long-term evaluations for previous road safety infrastructure programs. VicRoads completed long term evaluations for the earlier SRIP 1, 2 and 3 programs.
4.4 VAGO evaluation
The Towards Zero Strategy states that flexible barriers have been shown to reduce run-off-road and head-on serious casualty crashes by up to 85 per cent. Each of VicRoads’ project proposals has a crash reduction factor that outlines the expected effectiveness of the project’s treatments. As shown in Figure 4B, 25 of the 27 project proposals have a crash reduction factor of 70 or higher.
Crash reduction factors stated in Top 20 Program project proposals
Crash reduction factor (per cent)
Number of projects
Source: VAGO analysis of Top 20 Program project proposals.
As VicRoads has not evaluated the effectiveness of flexible barriers on the 21 projects it has completed as at April 2020, we completed our own evaluation. Our evaluation is limited due to VicRoads’ poor record keeping and data quality, which we discuss in the next section. VicRoads could not provide us with accurate construction dates for three of the completed projects, so we were only able to evaluate 18 of the 21 completed projects.
We looked at between 20 and 55 months of crash data aggregated from 21 projects for the period from 1 July 2015 to when VicRoads installed the safety treatments for each project. We compared this to between one and 28 months of crash data aggregated from 18 of these projects that had been completed by the end of January 2020 to estimate their effectiveness in reducing serious casualty crashes.
Our limited evaluation found that for the 18 completed projects, installing flexible barriers, wide centrelines and rumble strips has led to a 46.5 per cent reduction in run-off-road and head-on serious casualty crashes. Within a 95 per cent confidence interval, this reduction could be between 25.6 and 65.6 per cent. We were not able to determine how much of this reduction can be attributed to flexible barriers alone because VicRoads did not have enough data on barrier location.
4.5 Data quality
VicRoads has inadequate record keeping and poor data quality, which impacted our evaluation of the Top 20 Program and will impact VicRoads’ own evaluation. These issues included incomplete:
- record keeping of construction start and end dates
- barrier location data
- barrier failure data.
Record keeping of construction start and end dates
To evaluate how effective flexible barriers are, we asked VicRoads to provide construction start and end dates for the 21 Top 20 Program projects it had completed as at April 2020. VicRoads could only provide this information for 18 of the completed projects.
The success of VicRoads’ evaluations will depend on the quality of its record keeping. Without knowing key information, such as when barriers were installed, VicRoads will not be able to accurately determine their effectiveness at reducing FSI rates.
Quality of location data
VicRoads does not have reliable data on the location of flexible barriers.
In August 2018, VicRoads started collecting location data for flexible barriers that it installed on the top 20 roads before September 2019. VicRoads advised that it completed this project in August 2019. VicRoads will need to collect location data for the Top 20 Program projects completed after August 2019.
Barrier failure data
There are no crash statistics or reports from TAC and VicRoads that easily identify FSI crashes where flexible barriers have not worked as intended. This is because crash statistics do not contain a field that identifies if a vehicle has hit a flexible barrier.
The evaluation framework that the consultant developed for VicRoads included barrier failure as an area for evaluation. However, as of October 2019, this evaluation had not been formalised or progressed.