Outcomes of Investing in Regional Victoria

Tabled: 2 May 2019

3 Delivering regional outcomes

In this part, we assess whether RDV can demonstrate the outcomes the grants it has administered have achieved in improving outcomes in regional Victoria.

We also highlight the way RDV could use available social and economic data to target fund expenditure to drive better outcomes for regional Victorians.

3.1 Conclusion

Despite the significant investment to date, RDV cannot demonstrate its impact on social and economic outcomes in regional Victoria. Consultants engaged to assess PVGF and RGF outcomes have been unable to comprehensively evaluate those funds due to deficiencies in their design and inconsistent record-keeping. Due to this lack of effective evaluation, RDV did not have an evidence base to inform the design of the RJIF.

In response to these historical issues, RDV has put effort into improving its monitoring and reporting of RJIF and developing RJIF's evaluation framework. However, significant risk remains that it will not be able to evaluate RJIF's outcomes after the fund finishes in 2019. These risks include:

  • the evaluation framework not incorporating a complete program logic approach and failing to align the fund's objectives with its activities and evaluation measures
  • a lack of effective measures for some of RJIF's objectives
  • a lack of targets for all of RJIF's objectives
  • not having set criteria for how it will undertake the CBA analyses, which RDV is relying on as a key element of its fund-level evaluation framework
  • not having developed a data plan to specify and control the data it needs from grant recipients to ensure it will be able to aggregate the information to conduct an analysis at the fund-level.

Unless RDV addresses these challenges, it may be again unable to fully evaluate RJIF and assess whether it has achieved its objectives.

RDV has also not undertaken a local or regional level needs analysis using social and economic data to consider how to best target its efforts and grants to maximise RJIF's intended outcomes. RDV has not linked the distribution of RJIF funds to any identified areas—for example, LGAs or industries—and instead over half its funds have been given to projects delivering election or budget commitments. A thorough analysis of available data, along with RDV's regional networks and greater authority to distribute funds based on need, would enable RDV to target grants to areas and projects that help achieve the greatest impact in regional Victoria.

3.2 Outcomes of RDV's previous funds

Our two audits of RDV's previous funds and several consultant reports commissioned by RDV have identified persistent issues affecting evaluation. RDV could not evaluate PVGF and RGF due to failures in program design, data retention and documentation. These issues persisted from PVGF into RGF, and some remain in RJIF despite RDV, consultants and VAGO identifying the issues as early as 2009.

Previous attempts at evaluation

In 2009 RDV engaged a consultant to evaluate PVGF and the Regional Infrastructure Development Fund (RIDF). It found that PVGF had generated some economic and social improvements, but there were challenges in attributing outcomes to the funds. The consultant found that issues with the alignment of projects to the fund's objectives, a lack of measures, and data quality issues affected its ability to evaluate PVGF's outcomes. RDV did not undertake any further analysis of PVGF outcomes. The evaluation of RIDF found issues with data quality, a lack of baseline data and challenges in attributing outcomes to the fund.

RDV engaged a consultant in 2013 to undertake an interim evaluation of RGF, but RDV was not satisfied with the report. During our 2015 audit, RDV committed to undertaking a comprehensive review of RGF after the fund finished and engaged another consultant in 2016 to undertake this review. The consultant concluded it could not comprehensively evaluate RGF due to the poor design of the fund's evaluation framework and the lack of data—issues similar to those that impacted the PVGF evaluation.

The consultant recommended instead that RDV evaluate a selection of grants to identify the lessons it could learn from RGF. RDV accepted this advice and engaged another consultant in 2017 to undertake this sample-based evaluation. This evaluation concluded that the grants usually achieved their output goals and recommended that RDV measure a project's success using a cost-benefit analysis. This review could not provide a comprehensive, fund-level assessment of RGF's impact in regional Victoria.

As a result, RDV has been unable to fully evaluate the outcomes of PVGF and RGF and cannot demonstrate the outcomes of more than $600 million worth of investment in regional Victoria. This is a missed opportunity for RDV to assess how and in what form grants can improve regional Victorian outcomes, which it could use to design future funds.

3.3 RJIF's outcomes

Good program design includes clear objectives, measures, targets and methods to conduct the evaluation. For RJIF to successfully deliver outcomes—and prove its success—its design and implementation needs these key components.

Similar to our experience at the time of the RGF audit, many of RJIF's projects are still ongoing, so it is difficult to evaluate all its outcomes. However, the consistent lack of a detailed evaluation framework is a concern.

It is possible to apply a program logic model to assess whether RJIF's design and implementation mean it is likely to achieve its intended outputs and outcomes, and whether RDV will be able to evaluate them. This approach requires us to consider:

  • RJIF's strategic objectives
  • the resources available to RDV to achieve those objectives
  • the outputs that RJIF has delivered and how they align with its objectives
  • the outcomes that RJIF will deliver and how RDV will evaluate them.

RJIF's strategic objectives

Similar to earlier funds, the government established RJIF to deliver social and economic outcomes in regional Victoria, including enhancing job opportunities, improving the economic environment and increasing the liveability of regional Victoria. RJIF's three programs have individual objectives which align with the fund's overarching objectives.

RDV developed a business case for RJIF, outlining its objectives, how it would divide the funds between the programs and how it would evaluate the outcomes of the fund. However, as the government had already committed to a fund, the business case did not consider different options to achieve its goals. As RDV has been unable to evaluate its former funds, the business case could not draw on an evidence base that linked RJIF's design with successful practices identified in the previous funds.

Needs analysis

Program logic design begins with a needs analysis—identifying who should benefit from the program in order to best achieve the program's objectives. When establishing RJIF, RDV did not conduct such an analysis, which would rely on inputs such as population and jobs data, to identify areas or project types that would produce the greatest positive impact. A thorough needs analysis would enable the effective targeting of RJIF's finite resources.

A needs analysis is particularly important considering the challenge RJIF has in creating a statewide impact. The relatively small size of the fund and the many issues RJIF seeks to address, such as jobs and population growth in regional areas, mean it is unlikely that RDV can attribute changes to state-level measures, like unemployment, directly to RJIF. The large number and different types of projects supported through RJIF increases the complexity of this attribution. Therefore, targeting grants to areas of identified need, and evaluating those interventions collectively, may enable RJIF to demonstrate real impact, for example in a particular location or in a specific occupation of high need.

Resources available to RDV

Of the $500 million allocated to RJIF, $354 million was committed before it began considering grant applications from the public, meaning RDV was unable to target the full $500 million to activities that best met its strategic objectives.

The pre-committed funds included funding for RDV and the delivery of election or budget commitments the government allocated to RDV to deliver through RJIF. For example, the government made an election commitment of $5.2 million to Ararat Rural City Council to redevelop its art precinct and decided to fund this project from RJIF. Only $146 million over four years was fully available to RDV to allocate to applications for RJIF grants.

Figure 3A shows the allocation of RJIF funds.

Figure 3A
RJIF fund allocation to 31 December 2018

Pie chart shows that 38% of funds were allocated to election commitments, 20% to budget commitments, 12% to RDV costs from RJIF, 27% to other applications and the remained 3% are "to be allocated".

Source: VAGO, based on RDV data.

What outputs has RJIF delivered?

RDV reporting indicates that at the end of RJIF it will have supported a wide range of infrastructure projects, including airport, stadium, and streetscape upgrades and social improvement grants. RJIF has also reported it has contributed to the creation of more than 7 000 jobs. RDV currently reports on RJIF activity and outputs, as shown in Figure 3B.

Figure 3B
RJIF activity and outputs at 31 December 2018

Infographic informs that RJIF had 694 approved projects, $1.7 billion in total projects facilitated, $426 million in approved grants, and at least one project in all 48 regional LGAs

Source: VAGO, based on RDV data.

RDV requires grant recipients to identify the outputs they will deliver and includes them in the grant agreement. Payment of funds is contingent on the organisation completing milestones and submitting evidence of agreed milestones to RDV.

For infrastructure projects, milestones link to the completion of the agreed project which RDV can readily verify. For jobs fund projects, RDV links payments to the delivery of job-creating projects—for example expanding a factory—and the establishment of the jobs the organisation agrees to create. RDV's verification involves various methods including receiving statutory declarations and independent payroll verification, depending on the size of the grant.

From a reporting perspective, RJIF jobs-creation projects' outputs are standardised, which allows RDV to aggregate and evaluate them. For social engagement projects, however, the agreed outputs are difficult to measure. These measures can be specific to a project—for example, increasing participation at an arts precinct—so it may be challenging to aggregate outputs across the fund, or to draw conclusions based on a small sample size.

Evaluating RJIF's outcomes

As discussed, RDV has been unable to show the outcomes of previous funds, so it is essential it can demonstrate the value of RJIF and establish an evidence base for future fund design.

RDV has established a framework of documents that outline RJIF's objectives, outputs and outcomes and how it plans to evaluate RJIF. We refer to the collection of documents as the 'evaluation framework'. It includes:

  • the Implementation and Outcomes Framework, which outlines the key objectives of the fund, how RDV will assess grant applications and refers to the Monitoring and Evaluation Framework
  • output and outcome measures and performance indicators, and notes the use of a CBA approach to evaluate the RJIF, but does not directly link the indicators to the fund's five objectives
  • the Monitoring and Evaluation Framework, which outlines the method and time lines for RJIF's evaluation. This document is a single PowerPoint slide and does not provide detailed evaluation methodologies.

RJIF's business case promised a detailed evaluation methodology, which the government accepted as part of agreeing to the fund's business case. It included:

  • a program logic model to inform the evaluation design
  • assessing outcomes at state and local levels
  • undertaking evaluation across short, medium and long-term time frames
  • developing a data plan to support the evaluation.

RDV's evaluation planning to date does not contain this level of detail. Although RDV plans to evaluate RJIF's outcomes at the state level, there is no clear plan of how it would do so at the local level, or its methods for collecting data to enable an aggregation of results across the whole fund. RDV's evaluation plan also does not detail how RDV will evaluate the fund's short, medium or long-term outcomes.

Figure 3C shows the gaps in the existing evaluation framework compared to the business plan.

Figure 3C
Comparison of RJIF evaluation methodology noted in the business case with RJIF's current evaluation plan

RJIF business case evaluation plan

VAGO's analysis of RJIF's evaluation framework

A program logic model (incorporating an investment logic model) which sets out the goals, objectives and targeted actions of RJIF and demonstrates how specific activities contribute to short, medium and long-term program outcomes.

The evaluation framework does not fully implement a program logic model. It does not identify how grants contribute to the fund's objectives or clearly link the fund's objectives to the measures RDV has included in its evaluation framework. There are no separate short, medium or long-term outcomes.

A set of key performance measures derived from the program logic model, which will assist in measuring the progress of RJIF in meeting its goals and objectives.

The framework identifies high-level measures and a plan to use CBAs to evaluate the fund. However, RDV has not linked these to fund objectives.

As a result, some objectives do not have measures or have measures that are only loosely attributable to the fund. While RDV plan to use CBA analysis to fill in these gaps, the documentation lacks enough detail to provide assurance that this approach will be successful.

A program map that provides a snapshot of the spread of programs within RJIF and demonstrates alignment between specific activities, key performance measures and RJIF outcomes.

The framework does not have a program map linking outcomes, activities and measures. At best, RDV's reporting shows allocation at the LGA level and on a per person basis. However, this report does not link funded projects to the specific objectives and measures outlined in the evaluation framework.

A data collection, reporting and storage plan to ensure RJIF reporting and evaluation occurs in a structured, consistent and systematic way.

The framework does not have a data management plan. RDV therefore risks having incomplete, inconsistent and inaccurate data that will prevent it from evaluating RJIF.

Source: VAGO, based on RJIF's business plan developed by RDV.

RDV refined the objectives highlighted in the business case into five specific objectives in RJIF's guidelines. RDV intends to measure RJIF's outcomes against those objectives in two ways—assessing its impact on high-level indicators and undertaking CBAs of individual grants that it will aggregate at the fund, program and stream levels.

RDV's evaluation framework identifies 14 high-level outcome measures it will use to evaluate RJIF against its five objectives. However, the evaluation framework does not align the indicators and the objectives.

We have aligned these in Figure 3D and, in doing so, have identified gaps and attribution issues that would lead to an incomplete evaluation. We have grouped objectives two and three together because they both focus on improving business conditions and RDV could assess them using the same indicators.

Figure 3D
Measuring RJIF's outcomes by high-level indicators

From RDV evaluation framework

VAGO analysis



Data source

Attribution to RJIF


Create jobs of the future and diversify the regional employment base.

Number of direct, indirect, construction and retained jobs

Review jobs figures after completion of project


These are relevant indicators that identify the impact of RJIF.

Information sources exist that will enable RDV to evaluate these indicators.

However, RDV has not developed a methodology for its CBA implementation.

Number of jobs created in high‑growth and emerging sectors

Analysis of jobs created in RJIF


Unemployment rate in regional Victoria

Government data


Change in average regional wage

Government data


Average Benefit: Cost ratio at the fund, program and stream levels

RDV cost-benefit analysis


Improve productivity and enhance the long‑term competitiveness of regional Victoria through innovation and a transition to new growth opportunities.

Create the conditions for business growth by enhancing workforce skills, providing enabling economic infrastructure and facilitating expansion into new markets.

Change in Gross Value Add in regional Victoria

Government data


These are relevant indicators that identify the impact of RJIF on most components of the objectives.

However, it is not clear how RDV will evaluate competitiveness or enhanced workforce skills using these measures.

Information sources exist that will enable RDV to evaluate these indicators.

However, RDV has not developed a methodology for its CBA implementation.

Change in business diversity in regional Victoria

Government data


Change in turnover of RJIF‑supported businesses

RJIF-funded businesses' data


Change in export sales of RJIF‑supported businesses

RJIF-funded businesses' data


Level of investment in RJIF‑supported projects

RJIF-funded businesses' data


Change in annual visitor numbers

Government data


Change in total number of visitor nights

Government data


Average Benefit: Cost ratio at the fund, program and stream levels

RDV cost-benefit analysis


Improve the liveability of our cities, centres and towns to attract and retain families and young people to live and work.

Change in net migration

Government data


As there is only a limited attribution between the objective and the indicators, RDV may be unable to conclude whether it has achieved this objective.

Change in life satisfaction index

Government data


Change in community wellbeing index

Government data


Enhance community capacity through collaboration, leadership development and regional planning.

No indicators

No indicators

RDV will be unable to evaluate this objective given the lack of measures.

Source: VAGO, based on RDV's evaluation framework.

Challenges to attribution

Attributing outcomes to RJIF is challenging as its funding is small in comparison to the regional population and economy—RJIF's average annual spend of $125 million represents just 0.16 per cent of regional Victoria's GRP for 2018. External factors, such as migration levels and Australia's overall economic performance, also influence outcomes.

As such, RDV's state-level measures against these outcomes can have limited attribution to RJIF, leaving RDV at risk that it will be unable to effectively evaluate achievement. RDV recognises this issue and advise that CBAs will instead provide more meaningful information for evaluating RJIF's achievements.

RDV's evaluation framework identifies that the CBAs should enable it to:

  • assess the relative value of the benefits and costs of RJIF, wherever possible
  • assess whether RJIF investment has been in the right areas
  • inform future decision-making in investment and program development.

However, RDV has not developed a policy to ensure it applies a consistent CBA methodology—for example, evaluation criteria and data used—in its grant evaluation framework. Consequently, it cannot be sure that it will be able to aggregate the grants' outcomes to provide fund-level results.

Further, despite its stated reliance on CBAs to evaluate RJIF, RDV itself has not undertaken test-CBA evaluations of completed grants to assess the effectiveness of its approach. Creative Victoria received RJIF funds for the Regional Arts Program, a program that also required evaluation under DTF's 'lapsing program' requirements. As part of that evaluation, the contractors Creative Victoria hired completed CBAs of the seven individual projects within the Regional Arts Program. However, RDV does not have access to the methodology, assumptions made, or data used by the consultant in those CBAs. As such, RDV cannot replicate the method across other similar RJIF-funded projects and may not be able to aggregate these CBA results with those for other projects.

Lapsing program—a program where government provided funding for a specified period and which will conclude by the end of the financial year.

Measuring outputs

RDV plans to measure 13 outputs of RJIF, which we have grouped into three categories:

  • process measures—number of grants approved, total funds distributed, number of successfully completed projects, percentage of cases that do not proceed to application, percentage of cases withdrawn or rejected, client satisfaction survey
  • value for money—total funds leveraged by RJIF, funding ratio
  • distribution—per person funding by LGA, percentage of Regional Partnership projects funded through RJIF, total funds to regional cities, total funds to rural councils, number of LGAs that received a grant.

However, these measures are incomplete because they do not contain outputs against RJIF's productivity objectives or the efficiency criteria RDV has established to assess value for money.

The outputs also lack targets to determine whether RDV has been successful in delivering RJIF. For example, RDV will measure the value of grants to each LGA, but does not have a target to show what amount represents a good outcome. At a more strategic level, many of the output measures also fail to measure performance. For example, counting the number of grants approved will not enable RDV to determine whether many small grants or fewer large grants are better for regional Victoria.

Evaluation risk mitigation

RDV has been unable to evaluate previous funds due to deficiencies in their administration and particularly data collection. RDV is aware of this issue and is taking steps to mitigate the risk of being unable to evaluate RJIF, but several risks remain. Figure 3E describes the key mitigation activities devised by RDV to reduce the risk of being unable to evaluate RJIF's outcomes.

Figure 3E
RJIF evaluation risk mitigation approach


RDV description

VAGO analysis

Data probes

Testing/sampling grants to determine whether they have measurable objectives and baseline data and if RDV has collected the necessary data during project delivery

If managed effectively, the data probes should assure RDV that it will have enough, aggregable data to undertake a fund-level analysis.

However, several key issues remain that can limit the probes in achieving their intended goals:

  • RDV is not defining the data it requires from grant recipients, so similar projects may collect different data
  • RDV has no control over the quality of data collected
  • RDV hasn't linked the data collected to the fund's objectives to show how it is relevant to assessing RJIF's overall outcomes.

Project profiles

Detailed examinations of individual projects

These should provide case-study and qualitative analysis of grants but does not guarantee a state-level evaluation.

Evaluation Readiness Review

Scheduled near the conclusion of RJIF to test whether RDV has collected enough data and information prior to the commissioning of RJIF's overall evaluation

While the review is of benefit, the timing assumes that RDV has successfully implemented RJIF's evaluation strategy and collected the required data. If the review finds that the fund is not ready for evaluation, it is not clear how RDV would resolve this issue after the fund has concluded. A clearly articulated data plan would have mitigated this risk.

Source: VAGO, based on RDV information.

Strategic evaluation of RJIF

RDV's evaluation plan focuses on the delivery of RJIF's outputs and outcomes and does not consider its overall strategy or structure. RJIF's evaluation could provide evidence for future fund design by considering:

  • whether a grant scheme is the best way to achieve outcomes in regional Victoria as opposed to alternate methods, such as direct investment in projects or ongoing programs
  • whether an allocative or competitive grant program is the best design choice
  • which strategic approach to distributing funds achieves the best outcomes—for example, equal distribution or targeting disadvantaged areas
  • what grant size achieves the best outcomes—for example, a few large or many small grants
  • which outputs best link to the outcomes the fund wants to achieve, such as job creation or infrastructure investment
  • the scale of investment needed to influence outcomes in regional Victoria.

Answering these questions is fundamental to inform future investment supporting regional Victoria's development.

3.4 Using data to plan grant distribution

RJIF's business case highlights that 'Regional Victoria has a number of pockets of entrenched disadvantage, with a high demand for services' and that 'this place‑based disadvantage reduces the growth potential of regions'. In providing this advice, RDV recognised that its interventions needed to address specific local issues and that it needed to undertake evidence-based decision-making to prioritise investment.

The government consequently approved RJIF with the objectives of improving job, economic and social outcomes in regional Victoria. RJIF's objectives and guidelines do not explicitly identify reducing social or economic disadvantage as a goal. However, targeting resources to areas of disadvantage would support the overall objective.

RDV uses its networks, Regional Partnerships and officers' experience to understand the challenges facing regional Victoria, giving it context to assess grant applications. Using data analysis would bring an additional perspective and assist RDV to target funds to areas where they could make the most impact. Linking RDV's data with social and economic indicators—and assessing change over time—could form a key part of the RJIF's formal evaluation.

To test whether RDV has linked RJIF's grants with local needs, we compared the distribution of grants to datasets focused on jobs and social outcomes. This type of analysis is available to RDV and aligns with the 2018 Victorian Government's A Data Reform Strategy for the Victorian Public Service. The strategy advocates harnessing data to improve government decision-making, including linking datasets to understand issues affecting citizens and the economy.

We tested RJIF's distribution of grants against:

  • population
  • unemployment rates and job growth
  • indexes of relative socio-economic advantage and disadvantage.


RDV has distributed at least one RJIF grant to each of the 48 eligible LGAs. However, the spread of RJIF grants is uneven geographically and on a per person basis.

Of the funds distributed to 31 December 2018, 29.7 per cent went to Ballarat and Geelong LGAs. Election and budget commitments drove most of the allocation to these LGAs—81 per cent for Geelong and 91 per cent for Ballarat.

Figure 3F shows the distribution of grants across the regional LGAs.

Figure 3F
RJIF grant allocation by local government area

Image is a map of Victoria outlining the information described in this Section.

Source: VAGO, based on RDV data.

This contrasts with RGF, which RDV distributed more evenly—only six LGAs have received more than $10 million in RJIF funds, compared to 16 LGAs for RGF.

Figure 3G compares the total value of grants received against LGA population for the two funds and shows that small LGAs received higher levels of funding under RGF.

Figure 3G
RGF and RJIF funds by local government area

Figure 3G is a dot chart displaying the information stipulated in this part.

Source: VAGO, based on RDV and the Australian Bureau of Statistics (ABS) data.

One way to measure RJIF distribution is to compare the funds per person that each LGA received. We found that the per person rate ranged from $15 for Surf Coast to $2 101 for Northern Grampians LGAs. RDV now includes data on the per person rate of RJIF distribution in its quarterly report to government, which it did not include for earlier funds, which increases its ability to oversee the equity of RJIF distribution around the state. This is a positive improvement because it gives important context to decision-makers about overall RJIF distribution. RDV does not make this information publicly available.

Unemployment rates and job growth

We found no consistent relationship between RJIF distribution and unemployment or job growth, despite a RJIF outcome measure to lower unemployment. Figure 3H shows funding distribution per person for LGAs with the highest and lowest average job growth rates. There is no clear link between job growth and RJIF distribution—for example, Mount Alexander and Loddon have each received similar per person funding despite significantly different job growth figures.

Figure 3H
Job growth and RJIF distribution


Average job growth 2007–08 to 2016–17 (%)

RJIF funding per person

Lowest job growth







West Wimmera









Highest job growth

Macedon Ranges



Greater Geelong



Mount Alexander






Surf Coast



Median of all LGAs



Note: The RJIF funding per person is based on an average population from 2007–17.

Relative socio-economic advantage and disadvantage refers to people's access to material and social resources and their ability to take part in society. The indicators used to calculate it include income, education, employment, occupation and housing.

Source: VAGO, based on RDV and ABS data.

Socio-Economic Indexes for Areas

We measured RJIF distribution against Socio-Economic Indexes for Areas (SEIFA) data, which is an ABS measure of relative socio‑economic advantage and disadvantage. It is a useful tool that can help governments to direct funding and services where they are most needed.

Figure 3I shows the five most and least disadvantaged regional LGAs and the per capita RJIF funding they have received.


Figure 3I
SEIFA and RJIF distribution


SEIFA disadvantage rank (statewide)

RJIF funding per person (average population 2007–17)

Total RJIF funding to 31 December 2018

Most disadvantaged

Central Goldfields



$4 403 666




$24 414 194




$8 116 945

Northern Grampians


$2 101

$24 902 500




$1 032 700

Least disadvantaged




$1 988 284

Golden Plains



$566 145

Macedon Ranges



$1 988 284




$3 240 000

Surf Coast



$414 500

Median of all LGAs




Source: VAGO, based on RDV and ABS data.

This data shows that mostly RJIF funds were distributed to LGAs with greater need, but not consistently. For example, Yarriambiack is the seventh most disadvantaged LGA and has only received $147 per person, which is comparable to Moyne's $121 per person, despite Moyne being the 58th least disadvantaged LGA in Victoria.

These analyses of population, unemployment and disadvantage are some of the ways in which RDV could give evidence-based advice to government on effectively investing in regional Victoria and achieving the outcomes of grant programs like RJIF.

Back to Top