Managing the Municipal and Industrial Landfill Levy

Tabled: 25 July 2018

4 Program outcomes

Trust in government is important for the success of many policies, programs and regulations that depend on the cooperation and compliance of citizens. Transparent public reporting on government performance is an important way to build and maintain public trust. This is particularly relevant where money is collected from the public for a particular purpose as is the case with the MILL. It is important to explain to the public not just how this money is being spent, but also the extent to which it is achieving intended outcomes.

In this part of the report, we examine how the team manages the establishment of funding agreements, the delivery of funded initiatives, reporting and evaluation of outcomes. We also look at how DELWP measures and publicly reports on the outcomes of individual projects and how it uses these reports to understand how individual project outcomes contribute to the overall fund outcomes.

4.1 Conclusion

The MILL has been collected from the public and set aside to address two important environmental issues. Despite the fund being established in 2005, DELWP has not been able to demonstrate the extent to which the allocation of fund money has contributed to achieving the objectives of the fund. Without demonstrable and publicly reported outcomes the public may reasonably question the rationale for paying the MILL.

While DELWP's annual reports document expenditure from the MILL Trust Account and the fund they do not demonstrate to the community how this has contributed to the achievement of the fund's legislative purposes, explain the fund's governance arrangements, or the extent to which funded programs have met their expected outcomes.

Without established performance measures for the fund it is not possible to evaluate its success. DELWP cannot fully capitalise on its success or failures because it does not measure overall fund outcomes or utilise lessons learned from individual funded initiatives. Further, individual funded initiatives do not always achieve their intended outcomes or realise their expected benefits.

The team and the committee have focused on establishing the governance framework and administrative processes for the fund. However, the lack of clarity about what role they have in setting up and overseeing projects once approved has led to potential duplication in effort.

4.2 Project delivery

Funded initiatives may consist of more than one program and some have many projects under each program. The team establishes separate funding agreements between DELWP and the funding recipient for each project.

A range of agencies are responsible for delivery of programs funded from the fund. DELWP is responsible for those that involve policy development or have statewide application. Other delivery partners include DTF, the Department of Health and Human Services (DHHS), PV, EPA and Trust for Nature.

Project plan development

Following approval by the minister and Premier, the relevant department policy area or agency develops a project plan. The team uses the project plan to develop the funding agreement. The recipient does not receive any money until the team establishes a funding agreement.

Funding agreements

Funding agreements for other departments or for initiatives managed by other agencies are set out in an MoU. DELWP-managed projects use an Internal Funding Memorandum.

The team uses the project plan to populate the funding agreement milestones and deliverables. The Funding Agreement Team operating procedures state that the team should use the project plan to populate the Agreement Schedules template ensuring that the project aligns to at least one of the fund objectives and that the deliverables are clear. It checks to ensure that:

  • 'the payment schedule reflects the amount of budget approved through ERSC
  • the proposed milestones and payments can be processed through DELWP's GEMS and finance system as required
  • the milestones and payments are connected and realistic.'

It is important that there is clarity about each party's role so that the team can construct the funding agreements to ensure that risks are appropriately apportioned to those best placed to manage them.

We examined a selection of funding agreements and assessed whether:

  • they established appropriate requirements with agencies
  • the team effectively monitored agency compliance with the funding agreement or MoU.

We found all funding agreements we assessed are appropriate and include key elements such as:

  • funding objectives
  • milestones and a requirement for payment to be made in accordance with agreed performance milestones
  • a requirement to submit a funding variation if milestones are not achieved in the financial year
  • a requirement for performance measures
  • reporting requirements, including quarterly progress reporting and a Final Evaluation Report
  • acquittal processes, including certification of financial expenditure.
Role of the team

The team's role in the oversight of approved projects should be clarified to prevent potential duplication of effort and address any questions about the effectiveness and efficiency of the allocation of the team's resources. Funding recipients advised that the team can overstep its role when it comes to its involvement and influence in the development of the project plan.

There is also a question as to the amount of oversight needed from the team for initiatives once they have commenced. For example, SV is an independent statutory authority established by legislation, with an independent board appointed by the minister who is accountable to Parliament. Accordingly, SV has existing statutory obligations to deliver on behalf of the Victorian Government and the minister, therefore additional oversight by the team may be inefficient.

Monitoring

Funding agreements specify the monitoring, reporting and acquittal requirements for each initiative over the approved funding period. Funding agreements are stored in GEMS, which assists with performance monitoring and reporting. The team uses funding agreements to:

  • demonstrate the alignment of funded projects with the legislation and priorities
  • facilitate and reconcile payments
  • monitor and evaluate the extent to which projects meet the legislated objectives.

The team monitors and ensures recipients, with some exceptions, comply with requirements included in funding agreements. Recipients do not always provide their quarterly reports, however, where this occurs we saw evidence that the team follows this up.

Performance measures

We reviewed 33 funding agreements established between 2015 and 2017 and found that six did not include performance measures. SV established all of these agreements before DELWP took over administration of the fund. SV advised that performance measures for these projects are included in their project plans.

Milestones and requests for payment
Requesting milestone payments

Recipients can request payments when a specified milestone is completed or when lodging a quarterly progress report. Recipients must provide evidence of completion of the payment milestone and a completed request for payment (RFP).

The team runs a monthly due payments report using GEMS to identify projects that have missed RFP due dates. It follows up with project managers to encourage them to submit their RFPs.

Our analysis of a selection of 49 funding agreements found that the average number of milestones was 15, with the highest number of milestones for a single project being 71 and the lowest being three. There is no relationship between the number of milestones and the project cost or length. The rationale for some projects having significantly more milestones than others is not clear. DELWP advised that the recipient's level of activity determines the number of milestones. The administrative issue raised by funding recipients illustrates the need for the team to avoid the creation of unnecessary milestones during the development of project plans.

The team establishes funding agreements prior to the first payment. However, 14 recipients stated that this is an issue because it often takes them significant resources to develop the project plan, which is a prerequisite to the establishment of a funding agreement.

We reviewed the time frames for establishing funding agreements for 16 projects from 2015 to 2018. DELWP took 213 days on average to develop a funding agreement from the point of approval. One project took a year to establish a funding agreement after the minister and Premier approved its funding. DELWP should examine mechanisms to encourage the earlier development of project plans for approved programs or review the requirements for, and format of, funding agreements.

DELWP aims to have projects plans in place by 1 July 2018 for initiatives approved in the 2018–19 Budget and has engaged a consultant to assist with this.

Financial acquittal

The team relies on the approval of the funding recipient's financial officer as evidence of financial acquittal. The team received a financial acquittal for all 17 projects that we tested. All included an expenditure report designed to include information detailing how funding was accounted for, what recipients spent funds on and what financial year they spent the funds in. The expenditure reports vary in the level of detail provided and may not always enable the team to confirm that recipients use the funds for their intended purpose.

4.3 Project reporting

Recipient reporting

Recipients do not always comply with their reporting obligations.

Recipients are required to submit a quarterly progress report to the team 10 working days after the end of each financial quarter.

These reports provide sufficient information to the team to allow them to raise any project issues to the committee. However, the committee does not have a clear process to manage these issues. This is discussed further in Section 4.4.

According to the November 2017 quarterly report to the committee, 16 of the 56 project managers (29 per cent) had not submitted their most recent quarterly progress report. The team could not accurately assess and report on the status of these projects. DELWP manages 11 of the 16 projects (69 per cent) that did not submit their report. Figure 4A shows the reporting rates of funding recipients for the September 2017 quarterly progress reports.

DELWP advised that for both the December and March quarters there were a total of three projects where quarterly reports were not provided by the deadline.

Figure 4A
Quarter ending September 2017 progress reporting from funding recipients

Figure 4A shows quarter ending September 2017 progress reporting from funding recipients

Source: VAGO.

As part of their quarterly progress reporting, recipients self-report their project status based on whether or not they are meeting milestones. The team reviews this information and provides its own assessment of the project status. The committee receives the recipients' and the team's assessments side-by-side so that it can see any difference in assessment, as part of the fund quarterly report. We compared these in the November 2017 report and found the ratings varied for only one project.

The fund quarterly report provides the committee with the opportunity to review key aspects of project and financial management. This report includes:

  • progress of the development of funding agreements
  • details of issues that may be delaying payment and appear as an underspend in the fund budget
  • a traffic light report showing progress of active projects
  • an exception report providing more detail on projects that may need attention or support
  • details of any project closures
  • a snapshot of the current financial position based on the known information
  • a summary of the range of public enquiries that the team receives.

DELWP reporting

DELWP has its own internal reporting requirements. Program managers must report progress monthly by providing status updates in Sycle, DELWP's performance management system. This informs the monthly Senior Executive Team (SET) performance dashboard and SET performance reporting.

The performance reporting includes a traffic light report for each program, however, this rating is for each program overall rather than each individual project. For example, 21 individual projects make up the overall program Protecting Victoria's Environment—Biodiversity 2037. DELWP has clear criteria for how to determine the overall status of a program.

Alignment of reporting

DELWP ensures consistency between its internal management report and the committee's quarterly reporting. Its internal management reports now contain reports of the status of funded programs being delivered by DELWP, its portfolio agencies and DHHS as in the committee's February 2018 quarterly report. However, this report does not include reporting on the progress of projects delivered by DTF.

The traffic light rating used for DELWP executive reporting differs in that it details the status of overall programs whereas the team's traffic light report provided to the committee reports on the status of the overall program and at an individual project level.

The detail and content of quarterly progress reports are adequate and DELWP and SV advised us that they were easy to use and did not provide a significant administrative burden.

4.4 Escalation of project risks

There is no clear process to manage underperformance or risks of all projects that receive fund money. Despite this, there is evidence that the team withholds payments where recipients are not meeting their milestones. The team may recommend program termination to the minister, under the terms of the funding agreement, where:

  • there has been a substantial breach of the funding agreement
  • the team has issued a notice giving 14 days to rectify a breach and the program manager has failed to do so
  • progress is suspended for a period of more than six months without written consent of the team
  • there is any other reason that the team thinks is appropriate.

There is no evidence this has ever occurred despite the team raising project issues with the committee as part of its quarterly reports.

DELWP advise that senior management is currently reviewing roles and responsibilities for escalation of projects not achieving milestones or objectives.

Exception reporting

The team produces an exception report before every quarterly meeting that reviews and flags projects that may need attention or support and raises any issues for escalation to the committee.

Examples of issues identified for consideration by the committee include:

  • grant programs that are undersubscribed
  • lack of evaluation of failed programs or the use of this to inform new programs
  • broader issues of how DELWP and SV work together to develop concept and project planning
  • funding or prioritisation of funding to projects that are the same as previously failed projects without clear evidence of application of lessons learned or of mitigation actions for new programs
  • lack of clear mitigation actions or strategies to deal with significant time line extensions, under delivery and missing documentation.

The exception reports we reviewed also include suggested committee actions such as:

  • seeking clarification of project status and delivery issues from SV
  • considering how to best encourage quality and transparent reporting.

The actions identified in these exception reports need to be clearer to ensure that project performance issues are being addressed. DELWP should document a clear process to manage underperformance or program risks of all recipients of fund distributions.

The team identifies and escalates project issues but the committee's role and powers to address these are unclear. There are potential issues with conflicts of interest as DELWP committee members who manage DELWP-funded programs also advise on program termination and are responsible for performance issues.

Committee meeting minutes from 20 March 2017 do not clearly record the Committee's discussion of the performance issues raised in the quarterly report or show that the committee endorsed suggested actions to address the issues identified.

DELWP advised that it manages program issues through its internal reporting process but this does not include reporting on funded programs that DTF delivers. We expect the escalation process for management of funded project risks to incorporate all relevant programs while making use of existing processes to minimise duplication.

4.5 Program management

In response to concerns across DELWP about under-expenditure and fund governance processes, DELWP's Finance and Performance Sub-Committee (FPSC) requested that it receive a 'deep dive' report into the fund.

FPSC requested that DELWP's Finance and Planning Division report on the following issues:

  • Are the governance mechanisms and processes, such as project plans and funding agreements, appropriate?
  • Do the relevant areas of DELWP and other agencies have the capabilities required to deliver under these mechanisms and processes?

DELWP's Finance and Planning Division presented the report to the FPSC in October 2017. It found:

  • issues with the quality of project plans and the time it takes to establish funding agreements
  • various delays and issues in projects, leading to significant issues with budget phasing, milestones not aligning with actual expenditure and projects failing to deliver expected outcomes
  • gaps in staff capability
  • lack of an evaluation process for the fund.

Review of committee quarterly reports shows that various projects are experiencing delays, with only 50 per cent of projects reported as being on track in the November 2017 report. The team's November 2017 exception report notes that delays in implementation have resulted in an increase in requests for contract variations. Most delays linked to deliverables resulting in re-phasing of payments to future years.

The 'deep dive' report concludes that the root cause of the issue was project design and delivery issues within DELWP more broadly.

It proposed 11 recommendations to address issues affecting the timely delivery of projects within budget, including to:

  • undertake an immediate assessment of gaps in project management skills
  • facilitate engagement with the team at the start of project planning to maximise alignment of the project plan and funding agreement
  • finalise the evaluation framework as a matter of urgency.

As a result of DELWP's subsequent skills gap analysis, it developed a Project Management Strategic Framework and rolled out project management training modules across the department. The team, where possible, engages early with policy and recipient areas in the development of business cases and project plans respectively.

The committee approved the evaluation framework in its December 2017 meeting.

Underspend

Figure 4B shows the quarterly program underspend against the combined budget of the approved programs.

Figure 4B
Quarterly program underspend

Figure 4B shows quarterly program underspend

Source: VAGO based on DELWP's quarterly compliance reports.

The highest quarterly underspend was in December 2016, at 79.8 per cent. DELWP advised that underspend occurs as a result of factors such as:

  • using preliminary financials from programs as they are announced in the Budget to track performance instead of updated figures from the final budget in the project plan
  • delays in establishment of funding agreements due to the time it takes agencies to develop project plans
  • increased government targets and commitments to spend the money but its capacity to ramp up delivery is limited.

DELWP advised that it reviews alignment of funding agreements and approved budgets on a regular basis. DELWP significantly reworked budgeted cash flows as part of the mid-cycle budget in December 2017 to align them to contracted milestones and reflect contract variations. As at 31 December 2017, year to date program expense was $52.6 million, in line with forecast expenditure of $52.1 million. However, GEMS indicated that $60.2 million was expected to be paid in milestone payments for this period. This underspend is due to delays from program delivery areas seeking milestone payments. The improvement in underspend therefore does not necessarily reflect an improvement in the timeliness of program delivery.

4.6 Evaluation

The funding agreement requires each recipient to submit a Final Evaluation Report to the team within six weeks of project completion. The report must include as a minimum:

  • a detailed breakdown of expenditure certified by the recipient's chief financial officer (or delegated equivalent)
  • a description of the quantifiable outcomes of the project and an assessment of the effectiveness of the project in meeting its objectives
  • an assessment of the project's achievement of key performance indicators
  • an assessment of why targets were not met or exceeded
  • a summary of any consultation or communication that the recipient has undertaken with the community and any stakeholders during the duration of this MoU and an assessment of the effectiveness of these activities and the community's and stakeholders' responses to the project
  • an assessment of the performance of the project in terms of time, cost and quality (planned versus actual)
  • a discussion of the recipient's key lessons learned from the project in terms of planning and implementation
  • a discussion of what would/could be done differently to achieve a better outcome if the process were to begin again.

We reviewed all 18 projects that have closed in the past two years and found that all of them completed a project evaluation. The team completes a Closure Report that assesses the evaluation, where available, to determine whether closed projects have met their objectives. Of the 18 projects we reviewed, only seven (39 per cent) clearly met their objectives. Of the remaining projects, four did not meet objectives, four partly met objectives and three were not clearly assessed against objectives as part of their evaluation. Only two of these projects commenced during DELWP's administration of the fund. One of these was terminated early and the other was designed to achieve long-term behaviour change and its outcomes have not yet been able to be assessed. DELWP has worked to enhance the clarity of project outcomes in funding agreements with the aim of improving project evaluations on completion.

Evaluation framework

At its December 2017 meeting, the committee endorsed an evaluation framework for the fund. The committee noted that, currently, the fund receives project evaluations and the content and quality of them vary greatly depending on the experience of the program manager. The framework aims to improve this by requiring clear articulation of project objectives and the demonstration of how they link to the legislated objectives of the fund. This is required at the outset of the project to ensure recipients report this in Final Evaluation Reports.

A key new addition to the evaluation framework is the introduction of a Priority Statement benefits table that will support budget submission authors and program managers to understand and articulate the line of sight between their project's benefits/objectives and the legislated objectives and priorities of the fund.

The Priority Statement notes that Victorian Government policies and other relevant strategies, all of which include their own objectives, will guide investment. However, DELWP does not use these to evaluate the success of the use of fund monies. DELWP advised that policy guides program development and recipients identify the measures that will be used to show alignment and performance.

DELWP advised the evaluation framework is in the process of being implemented and will be detailed in the 2017–18 Activity Report due to be released alongside the DELWP annual report. The committee advised that the evaluation framework will support a whole-of-fund evaluation, which it intends to complete every four years. However, DELWP has not applied the framework to any programs and it is not clear how this will work in practice.

4.7 Lessons learned

Recipients are required to include a section on lessons learned in the Final Evaluation Report.

DELWP does not have a formal process to consistently apply lessons learned from previous projects during the development or selection of new projects funded from the fund. Lessons learned are not stored in a central location to enable them to be considered easily by the committee when new but similar programs come up for assessment. This process relies on the team to raise these issues in its summary assessment of proposals or on the organisational knowledge of the committee members.

There is no debrief held between the team and recipients at project close. Funding recipients provided feedback to us that the team does not share lessons learned from projects to inform the development of future initiatives seeking funding. Recipients advised that feedback about past programs is dependent solely on organisational knowledge. This represents a lost opportunity as there is a risk that corporate knowledge will be lost.

DELWP advised that the committee and the team have identified this as an issue and it intends to address it across the department.

4.8 Public reporting

Annual report

The MAC report recommended that there be greater transparency of the collection and distribution of the MILL and the fund. Previously, SV and EPA reported information about expenditure and balances separately in their annual reports. MAC recommended consolidating this information in DELWP's annual report. DELWP now publishes this information in appendices 7 and 8 of its annual report.

DELWP's annual report includes the fund movements and a brief description as an appendix to the financial statements. DELWP provides a brief summary of the purpose of each fund and fund movements. Details of payments made during the year to funding recipients by program are also included. This information is insufficient to inform the public about the governance, activities of the fund, programs funded and how program outcomes deliver against the legislative purposes.

Budget Papers

The 2016–17 Budget Papers were the first to clearly identify the fund as the source of funding of budget initiatives. This shows an improvement in the transparency of reporting programs that receive funding from the fund.

Website and media releases

DELWP's website includes background information on the fund, details of budget commitments funded from the fund and links to the Priority Statement and Guidelines. The website lacks detail about the composition of the committee, the role of the committee and outcomes of funded activities. Media releases inconsistently refer to the fund as the source of funding. The government did not announce that the Stepping In To Support Industry And Councils With Recycling program was funded by the fund in a media release in February 2018.

Activity report

SV produced an annual sustainability fund activity report. This report provided details of the fund's governance arrangements, financial performance and funding provided during the period. It included case studies of programs funded by the fund and the program outcomes. This activity report was not a statutory requirement, but SV published it to improve the transparency of fund management.

Since the administration of the fund moved to DELWP, there has been limited public reporting about the activities of the fund. There has been no reporting of the fund's governance arrangements or information about the nature and outcomes of funded programs.

DELWP published an activity report for 2015–17 in May 2018, Investing in a more sustainable future. DELWP intends to provide such an activity report annually in the future. The report details the fund governance arrangements, including the role and membership of the committee.

The report provides a good description of the MILL and the MILL Trust Account and the flow of money to the fund. It details the legislative purpose of the fund and Priority Statement. Analysis of funding recipients is included with a list of programs that received funding. Detailed project case studies outline the purpose of projects funded and the outcomes delivered.

There are clear similarities between DELWP's activity report and SV's annual activity reports. It is unclear why it has taken DELWP nearly two years to produce it. The delay in its publication affects its relevance and usefulness as a tool to improve the transparency of the fund and its management.

Back to Top