Outcomes of Investing in Regional Victoria

Tabled: 2 May 2019

Audit overview

Gross Regional Product—the value of goods and services produced in an area.

Regional Victoria is home to strong communities and plays a significant role in Victoria's economy—it created 19 per cent of the state's Gross Regional Product (GRP) in 2017–18. However, regional Victoria also faces challenges compared with metropolitan Melbourne, including slow or negative economic growth, low job creation rates and slow population growth.

Successive state governments have focused on improving economic and social outcomes in regional Victoria through significant programs and grants. Regional Development Victoria (RDV)—a statutory agency in the Department of Jobs, Precincts and Regions (DJPR), formerly the Department of Economic Development, Jobs, Transport and Resources—is the lead agency responsible for developing regional Victoria and administering key regional development grant programs.

Since 1999, RDV has administered a series of regional development grant funds. The current fund, the Regional Jobs and Infrastructure Fund (RJIF), was set up in 2015–16 with the objective of 'growing jobs, building infrastructure and strengthening communities in regional Victoria'.

We audited two of the previous funds—the Regional Growth Fund (RGF) and Provincial Victoria Growth Fund (PVGF). These audits found that RDV could not demonstrate that the grants had supported regional growth outcomes due to deficiencies in the funds' administration. Our 2015 RGF audit made nine recommendations to RDV about the design and implementation of RJIF.

This report examines whether investment in regional Victoria through these three funds has improved economic and social outcomes for regional Victoria.

Conclusion

Program logic—an analysis approach that sets out the relationships between the objectives, resources, activities, outputs and outcomes a program is designed to achieve.

Output—the services provided or goods produced by an entity.

Outcome—the impacts on society of an organisation's outputs.

Having administered more than $1 billion in allocated funds since 2011, RDV still cannot reliably determine whether their grants have improved economic or social outcomes directly or indirectly, or whether any benefits have been sustained beyond the immediate injection of funds into a community. There is a high risk that its next evaluation, scheduled for 2021, also will not provide the answers to these questions.

RDV has improved its management of RJIF compared to previous funds, which was necessary given the serious issues identified in our two prior audits. To do so, it improved its organisational culture, increased its focus on program monitoring and better promoted the fund to intended applicants. However, several issues still hamper the successful management of RJIF:

  • RJIF's evaluation framework has incomplete measures and targets and lacks a full program logic model and data plan.
  • RDV advises it has spent less than the state's benchmark of 5 per cent of RJIF's funds on its administration, however it cannot substantiate this claim.
  • RDV struggles to assess grants promptly, in part because it has not adopted a risk‑based approach to grant assessment.

RDV can also improve its transparency as it currently does not publish a comprehensive list of the grants awarded or complete information about how it assesses applications.

Findings

Achieving social and economic outcomes

RDV cannot show that its grant programs have improved social and economic outcomes because it has not successfully evaluated them. RDV has engaged multiple consultants to evaluate the various funds since 2009, each of which concluded that they could not systematically evaluate the funds due to issues including:

  • inconsistencies between objectives and implementation
  • a lack of targets and measures
  • poor data quality
  • challenges attributing change in social and economic indicators to the grant programs.

During our 2015 audit, RDV advised it would undertake a 'full-term evaluation' of RGF, however it was unable to do so due to various issues including a lack of appropriate data. Instead, RDV relied on a limited, case-study based evaluation of the $570 million RGF that could not demonstrate whether it had achieved its objectives.

RDV has developed an evaluation framework to assess RJIF outputs and outcomes and has dedicated resources to its implementation. This framework uses the dual approach of assessing changes in high-level indicators and undertaking a cost-benefit analysis (CBA) of individual RJIF-funded projects. It includes activities to mitigate the risk that RDV will be unable to evaluate RJIF, including data probes and an evaluation readiness report to determine whether individual projects can access necessary evaluation data. However, significant risks to a successful evaluation of RJIF remain:

  • The evaluation framework uses an incomplete program logic approach, despite RDV committing to this in the RJIF business case. The program logic model does not clearly articulate how the fund's objectives link with its grant activity and evaluation framework. This has resulted in the fund's objectives having incomplete evaluation measures.
  • RDV has not developed a consistent, comprehensive set of measures, with business rules for data capture. RDV therefore cannot be sure that its grant recipients will collect consistent data that will enable an overall evaluation of the fund.
  • RDV has not set targets against which to measure the fund's success in delivering on its objectives.
  • RDV plans to evaluate grants using a CBA approach to supplement its analysis of high-level indicators, but it has not outlined a methodology for how it will conduct this analysis to ensure consistency and enable an analysis of the whole fund.
  • RDV has not developed a data plan—in line with its commitment in the RJIF business case—to ensure it consistently collects data about the grants and the baseline data needed to evaluate the fund.

Overall, RDV's evaluation framework carries significant risks that indicate RDV is unlikely to be able to undertake a localised, program specific or fund-level analysis of RJIF. Unless RDV addresses these issues, it will once again be unable to demonstrate the benefits of its regional grant fund and set up an evidence base for any future fund development.

Management of RJIF

RDV has undertaken a significant amount of work to improve its management of RJIF compared to earlier funds by focusing on cultural change and improving the content of its reporting framework. Timeliness issues have affected RDV's administration of RJIF and RDV cannot prove it has managed the fund's administration costs in line with state guidance.

Timeliness

RJIF grants support councils, businesses and other organisations to implement new projects to improve outcomes in regional Victoria. Many projects are time sensitive and rely on RJIF funding to proceed, but RDV is slow to process grants.

Overall, grants take nine to 12 months to assess and process, from expression of interest (EOI) submission to a funding agreement. RDV's reports show the assessment of a formal grant application and signing the agreement each take approximately three months. There are also additional steps prior to lodging a formal application, including:

  • the applicant writing an EOI
  • RDV assessing the EOI
  • the applicant developing the formal application with RDV's support.

RDV advises that the time from formal application to approval is the best measure of performance because this is within their control. However, the full timeframe is relevant because a project cannot commence until RDV and the funded organisation sign the grant agreement, and the long lead-times caused significant dissatisfaction among grant recipients.

A key reason for this delay is that RDV applies the same grant assessment processes regardless of a grant's size or risk, despite significant variations in grant size—of the 694 grants approved to 31 December 2018, 266 were for less than $50 000 and five exceeded $20 million. RDV's processes, which assess some applications multiple times, and the need to seek the Minister for Regional Development's (the Minister) approval for all grants, increases the work required of applicants and RDV. In other areas of their regular operations, RDV regional directors have expenditure approval delegations of up to $150 000 while the Chief Executive Officer has $500 000, but they have no authority to approve RJIF grants regardless of value.

Adopting a risk‑based approach to assessing applications would enable RDV to improve its timeliness and streamline the application process for small grants, while dedicating more resources to higher-risk applications.

Grant administration costs

RDV was allocated 12.3 per cent of RJIF's funds to administer the grant program and some of its other service delivery activities. Department of Treasury and Finance (DTF) guidelines on grant administration recommend that agencies keep administration costs below 5 per cent. RDV advises that it meets this target but does not have documentation or data to support this claim.

In 2017, DTF recommended an interdepartmental capability review of RDV's grant management. DTF's view was that RDV needed to improve 'RJIF process and control processes to prevent funding shortfalls from occurring again' and 'to identify drivers of RDV's significant administration and project costs'. The government and RDV accepted this recommendation; however, RDV advised that the government did not approve the terms of reference and the review did not occur. This is a missed opportunity to evaluate RDV's funding and drive efficiency in its grant administration.

As part of its 2019 budget bid, RDV plans to separate its grant administration costs, capped at 5 per cent, from its other funding. Separating these costs would increase the transparency of its grant administration.

Data quality and documentation

Good quality data enables agencies to monitor and evaluate their performance against stated objectives. RDV uses a case management system—the Global Engagement Management System (GEMS)—to manage RJIF. RDV captures all grant-related activity in GEMS to provide data for internal monitoring and reporting. However, RDV does not have effective quality assurance processes to ensure it is accurately reporting on the outcomes of its grants.

We reviewed a selection of grants and identified several types of errors that have affected the quality of information RDV uses to monitor the program and report to government. One issue is that RDV incorrectly reports the number of jobs it expects to create through RJIF.

We sampled 10 cases and found errors in RDV's reporting of the number of jobs created in eight cases. After we raised this issue with RDV during the audit, RDV reviewed its data and increased the reported number of jobs RJIF created by 2 239. We retested the same sample after RDV's changes and found that RDV was still incorrectly reporting data on four of the eight cases originally identified.

We also found that RDV consistently documents its advice to the Minister and grant approvals, however, it does not consistently keep records for other aspects of its grant management, including:

  • engagement with potential applicants, such as emails and meeting minutes
  • grant application assessments and analyses of supporting documentation
  • internal decisions to progress or not progress an application.

This lack of documentation reduces RDV's ability to effectively monitor RJIF's implementation and ensure the transparency and accountability of decision‑making.

An issue identified in the 2015 RGF audit was that RDV attributed the total number of jobs created and investment leveraged as RDV's achievement—even when its contribution was a small part of the overall project. We recommended that RDV develop guidelines for making appropriate adjustments to outcomes, to ensure accurate and fair reporting. RDV did not develop these guidelines and continues to attribute the full number of jobs created by a project to RJIF, regardless of the size of the grant relative to the project's total cost.

For example, RJIF contributed $1.5 million to a $55 million project—2.7 per cent of the total project cost—yet included all 357 expected jobs in RJIF output figures. This means RDV is overstating RJIF's impact. RDV advises it uses the phrase 'facilitated by RJIF' to communicate that multiple parties are involved in the project. However, this does not clearly communicate RDV's level of investment and its influence in creating the jobs for which it takes full credit in its internal and public reporting.

Distribution of RJIF grants

The most funded projects were those delivering election and budget commitments that were allocated to RJIF. Election commitment projects consumed $191 million and budget projects $101 million—58.5 per cent of the fund—to 31 December 2018.

Those grants were the key driver of 50 per cent of funds being delivered to regional city local government areas (LGA) to 31 December 2018—areas which may be more able to independently deliver growth through increasing populations, transport connections and business opportunities. In particular, 29.7 per cent of RJIF funding went to organisations in the Ballarat and Geelong LGAs. Most of those grants were to deliver election or budget commitments—81 per cent for Geelong and 91 per cent for Ballarat.

While a sizable part of RJIF was allocated to deliver election or budget commitments, we found no evidence that RDV uses available social and economic data—for example population and job growth figures—to target the remaining funds to where they might have the greatest impact.

RDV advises that targeting areas of greatest need in regional Victoria is not an objective of the fund. However, RJIF's business case identified the need for a nuanced approach as 'Regional Victoria has a number of pockets of entrenched disadvantage' and 'there is variation in how these challenges and opportunities impact upon different parts of country Victoria'.

RDV relies on its networks for information and government policy for direction. By omitting available data, it misses an opportunity to enhance its understanding of regional need. Incorporating data analysis would enable RDV to target grants to local needs and increase the likelihood of delivering strong outputs that align with the fund's objectives.

Transparency and accountability

Given that grants are public funds, it is important that the public has access to enough information to understand how the grants are allocated. We found two key issues that diminish the transparency and public accountability of RDV's management of RJIF.

Public information about awarded grants

At the commencement of our audit in September 2018—more than three years into RJIF—RDV did not publish a list of awarded grants on its website. RDV has since added a searchable list of grants. However, RDV has only published information about 335 grants to April 2019 despite the Minister having approved over 600 grants and it does not include the value of grants given to businesses. This omission contrasts with its earlier practice of publishing the value of RGF grants to businesses. RDV's annual report also includes an incomplete list of grants awarded in the relevant year and again does not include the value of grants to businesses.

RDV advises that it only publishes information about grants that the Minister has announced. It further advises that it does not include the value of grants to businesses to protect commercial information and avoid inadvertently establishing a benchmark for grants awarded to businesses.

The state's grant policy does not provide advice on disclosure. In the absence of state policy, the commonwealth guidelines are informative. They require disclosure unless specific reasons exist. These guidelines detail that if not fully disclosed, the agency must publish as much information as possible and the reasons why it did not make the remaining information public.

RDV's approach means the public and stakeholders cannot obtain a comprehensive understanding of the money distributed through RJIF. This approach diminishes the transparency of the fund and its accountability.

Making accountable decisions

RJIF has three programs.

The Regional Infrastructure Fund invests in major infrastructure projects.

The Regional Jobs Fund helps create and retain jobs.

The Stronger Regional Communities Plan supports towns to attract families and young people.

The three programs have 12 different streams.

RDV needs to accurately apply RJIF guidelines to ensure consistency and public accountability when it recommends grants for approval to the Minister. RDV assesses RJIF applications against the Regional Development Victoria Act 2002 (the RDV Act) and the relevant RJIF program and stream guidelines, which are available online. However, in recommending grants to the Minister, RDV deviated from the published guidelines in two key areas.

Firstly, while knowing that the government had allocated large projects to the Stronger Regional Communities Plan (the communities plan), RDV developed public guidelines capping the value of grants at $50 000—much less than the cost of the already committed projects. In its advice to government on the communities plan's guidelines, RDV did not include advice on this issue and did not make provisions to exceed the cap for projects allocated to the program. When RDV later assessed the applications for these grants, it did not advise the Minister that they would exceed the cap set out in the program's publicly available guidelines.

Based on RDV's guidelines, to the public it appears that $50 million worth of $50 000 grants were available, however this is not the case. Instead, $20.1 million was committed to 34 grants above $50 000 to deliver election or budget commitments—16 of which were over $500 000 and the largest of which was $3.4 million. RDV could have avoided this lack of clear and appropriate information by including the exceptions in the guidelines.

Secondly, RDV applies unwritten, subjective weightings in its assessment of grant applications that preference job creation and value for money outcomes. Applying weightings is relevant and can help guide decision-makers. However, by not documenting and publicising the weightings, RDV limits the transparency of its decision-making. Further, grant applicants cannot access all assessment criteria when preparing their submission, which potentially reduces their chance of success.

Recommendations

We recommend that the Department of Jobs, Precincts and Regions:

1. strengthen the Regional Jobs and Infrastructure Fund's evaluation framework to ensure it:

  • has measures, supported by clear business rules and data definitions, against all the Regional Jobs and Infrastructure Fund's objectives
  • supports consistent data collection from grant recipients to allow for the aggregation of results at state, local, stream and program levels
  • details its cost-benefit analysis method (see Section 3.3)

2. improve data quality and the documentation of decision-making by:

  • fully recording its engagement with stakeholders in its case management system, including calls, emails and meeting minutes
  • using its case management system to record all grant documentation
  • improving staff training on data quality, documenting decisions and using the Global Engagement Management System
  • implementing a policy to guide data entry and documentation requirements
  • implementing effective quality assurance processes, specifically for Regional Jobs and Infrastructure Fund output data (see Sections 2.3)

3. review and ensure that grant administration costs:

  • have not exceeded 5 per cent of the Regional Jobs and Infrastructure Fund costs
  • are effectively monitored and reported on in the future and remain below 5 percent of the fund's value (see Section 2.3)

4. streamline application and approval processes by adopting a risk-based approach (see Section 2.3)

5. improve transparency of grant programs by:

  • publishing all assessment policies, criteria and weightings
  • following its grant guidelines and processes, or reporting publicly when and why it diverts from its standard approach
  • fairly and accurately reporting fund outputs and outcomes including the number of jobs the grant program creates prior to reporting (see Section 2.4)

6. maintain a list of all awarded grants including Regional Jobs and Infrastructure Fund and Regional Growth Fund that:

  • is complete and up to date, by announcing all grants within 60 days of RDV and the recipient signing the grant agreement
  • discloses information publicly, including the value of the grant awarded, unless the recipient could suffer commercial harm, determined in line with the considerations included in Victoria's Freedom of Information Act 1982
  • clearly identifies rationale and evidence, on a case by case basis, for the decision of not releasing information about a grant (see Section 2.4)

7. use available social and economic data to understand the areas of greatest need and use this information when assessing grants, providing advice to government and undertaking strategic planning for any future fund (see Section 3.4).

Responses to recommendations

We have consulted with DJPR and we considered its views when reaching our audit conclusions. As required by section 16(3) of the Audit Act 1994, we gave a draft copy of this report to the agency and asked for its submissions or comments. We also provided a copy of the report to the Department of Premier and Cabinet (DPC).

The following is a summary of those responses. The full responses are included in Appendix A.

DJPR has accepted the seven recommendations from this audit and has undertaken to implement them by 31 December 2019.

Back to Top