State Purchase Contracts

Tabled: 20 September 2018

4 Realisation of SPC benefits

Tracking and reporting SPC benefits is a key part of measuring the performance of an SPC. Lead agencies should be able to show whether SPCs are delivering the expected benefits.

This part examines whether effective arrangements are in place to monitor and evaluate the achievement of expected savings and benefits from SPCs.

4.1 Conclusion

Lead agencies report that SPCs deliver significant financial benefits. However, without individual targets for each SPC to measure these reported benefits against, it is difficult to determine if they represent value for money.

For some SPCs managed by DPC, we disagree with the methodology used to calculate these benefits and we believe the calculation has resulted in the overstatement of benefits.

For seven SPCs, lead agencies did not report any financial or non-financial benefits.The lead agencies are unable to show that these SPCs deliver a benefit to the state.

VGPB does not publicly report the overall financial benefits of SPCs and there is no evidence of central oversight of the way lead agencies calculate financial benefits for different SPCs.

4.2 Targets

Tracking and reporting SPC benefits is the key way lead agencies show the performance of an SPC and the impact of contract management activities.

We assessed whether lead agencies reported on the overall performance of the SPCs they managed, and whether lead agencies had targets to measure the benefits of each SPC.

Overall targets for lead agencies

DTF is the only lead agency that publicly reports on the performance of SPCs against a target.DTF is required to meet a financial benefit performance target specified in the State Budget papers—'Benefits delivered as a percentage of expenditure by mandated agencies under DTF managed SPCs'.

Figure 4A shows how DTF-managed SPCs performed for 2013–14 to 2016–17 against the target of 5 per cent.

Figure 4A
Reported financial benefits delivered as a percentage of expenditure by mandated agencies on DTF-managed SPCs, 2013–14 to 2016–17

Figure 4A shows how DTF-managed SPCs performed for 2013–14 to 2016–17 against the target of 5 per cent.

Source: VAGO, based on information from 2013–14 to 2016–17 State Budget papers and DTF.

DTF reported that it exceeded the target each year. DTF advised that the 5 per cent target was derived in 2013 from past performance data. However, DTF has not documented the basis for the target and it is unclear whether this is a reasonable measure against which to judge performance. There is also no documentation that outlines how each of DTF's 17 SPCs should contribute to the target. Further, the target has not changed over four years, despite changes to market environments and SPCs across this period.

DPC and Cenitex do not have overall performance targets to meet for their SPCs, which makes it difficult to assess reported benefits and track performance. DJR has a target but only manages one SPC so is discussed in the next section.

Individual targets for SPCs

Measuring and monitoring benefits against pre-determined targets enables lead agencies to determine if SPCs are performing as expected. Following the sourcing process, the expected benefits of an SPC should be 'locked in' as documented targets. Lead agencies should then report against these targets to demonstrate that SPCs are achieving expected value for money.

The Legal Services Panel is the only SPC with a 'locked-in' target against which the lead agency, DJR, can track financial benefits. DJR's target reflects the discount in hourly rate that the SPC hopes to achieve compared to the market. DJR undertakes an annual benchmark activity to measure whether SPC rates have achieved this. Since the Legal Services Panel's establishment in March 2016, DJR has consistently exceeded this target. Given this is the only SPC that DJR manages, it does not report publicly against this target to ensure the financial benefit achieved remains confidential.

Although DTF identifies the expected financial benefit following the sourcing process for an SPC, they do not lock in targets and measure against these. DPC and Cenitex also do not have documented individual financial benefit targets for each SPC they manage. Without 'locked-in' targets, DPC and Cenitex cannot demonstrate that their SPCs are performing well.

4.3 Measuring financial benefits

Lead agencies report the achievement of financial benefits for 26 of the 34 SPCs. Figure 4B shows all reported SPC spend and benefits for 2012–13 to 2016–17.

Figure 4B
Overall reported spend and benefits, 2012–13 to 2016–17

Figure 4B shows overall reported spend and benefits, 2012–13 to 2016–17

Source: VAGO, based on data provided by DTF, DPC and DJR.

The total reported financial benefits as a percentage of expenditure ranged from 14 to 20 per cent. This provided reported savings to SPC users of between $192 million in 2014–15 and $272 million in 2016–17. Appendix H provides the SPC spend and benefits reported by DTF and DPC for 2012–13 to 2016–17.

Financial benefits achieved by user entities

The reported financial benefits in 2016–17, as a percentage of spend, varied between users and ranged from 9.5 per cent at DPC to 34.2 per cent at DET, as Figure 4C shows. This variation can be due the nature of goods and services purchased by entities and the different methodologies used to calculate benefits for each SPC.

Figure 4C
Reported financial benefit as a percentage of spend by user entity, 2016–17

Figure 4C shows reported financial benefit as a percentage of spend by user entity, 2016–17

Source: VAGO, based on data provided by DTF, DPC and DJR.

Financial benefits achieved by SPCs

Without targets we were unable to determine whether the reported financial benefits for the SPCs managed by DTF, DPC and Cenitex were appropriate. These SPCs exist in very different market environments and are in place for very different goods and services. For this reason, it is hard to assess the reported financial benefits for each SPC without an associated target.

For example, Figure 4D shows the reported financial benefits as a percentage of spend for each of the 26 SPCs that report financial benefits. These percentages vary widely, from 0.1 to 154 per cent. Without a target for each SPC it is difficult to judge whether these percentages represent a good outcome for each SPC in its own market environment.

Figure 4D
Reported financial benefit as a percentage of spend by SPC, 2016–17

Figure 4D shows reported financial benefit as a percentage of spend by SPC, 2016–17

Source: VAGO, based on data provided by DTF, DPC and DJR.

Some SPCs—such as Stationery and Workplace Consumables , and Travel Management Services —may report small financial benefits in relation to spend. However, this may be a good result if the objective was to contain procurement costs rather than achieve large savings, given they have been previously market tested. Without a target we are unable to make this judgement.

SPCs with no financial benefits

Lead agencies do not track financial benefits for all SPCs. As Figure 4E shows, seven SPCs, with a total spend more than $176 million in 2016–17, do not have their financial benefits tracked.

Figure 4E
SPCs with no financial benefits tracking

SPC

SPC spend
2016–17
($ millions)

Reason financial benefits are not tracked

Marketing Services Register

5.5 (a)

The system used for monitoring the Marketing Services Register does not capture spend. Expenditure reports from the 200-plus suppliers are not received.

PAS

71.0

Savings/benefits not measured.

Security Services

46.5

The business case has no savings objectives, so financial benefits are not measured or reported.

According to the business case the focus is on non-financial benefits with the key objective being supporting the security services industry and providing high‑quality services.

IT Infrastructure Register

44.6

Savings/benefits not measured.

eServices Register

(b)

Savings/benefits not measured.

Multifunction Devices and Printers

7.7

Benefits have been difficult to quantify as not all large mandated departments and agencies have completed the transition to Multifunction Devices and Printers.

Rosetta

0.7

Savings/benefits not measured.

(a) DTF estimated spend not actual reported spend by suppliers.
(b) Spend data not collected.
Note: Financial benefits for the Microsoft Licensing Solution SPC are realised through the Microsoft Enterprise Agreement SPC.
Source: VAGO, provided by DTF.

In some cases, it can be difficult to calculate the full financial benefit of an SPC compared to if the SPC did not exist. For example, for an engagement of consultancy services under the PAS SPC, no means is available to accurately know the rates a department would have been able to negotiate for the same engagement if the SPC was not in place. However, without attempting to measure the financial benefits of these SPCS, lead agencies are unable to determine if they are providing value for money.

4.4 Benefits methodology

The method for calculating a financial benefit is different for each SPC since contracts deliver different goods and services and exist in distinct markets.

DTF and DPC manage different types of SPCs and use different methods to calculate financial benefits. Figure 4F shows the difference in reported financial benefits between SPCs managed by DTF and those managed by DPC.

Figure 4F
Reported financial benefit as a percentage of spend by DTF and DPC, 2016–17

Figure 4F shows the reported financial benefit as a percentage of spend by DTF and DPC, 2016–17

Note: Cenitex did not report any financial benefits for the Rosetta SPC. We have not reported benefits data for the SPC managed by DJR due to commercial sensitivity.
Source: VAGO, based on data provided by DTF and DPC.

As Figure 4F shows, DPC reports significantly higher financial benefits than DTF.

We examined how lead agencies calculate financial benefits for 11 SPCs and found inconsistencies and irregularities in the methods for six SPCs managed by DPC. Figure 4G outlines these examples.

Figure 4G
Financial benefit method for a selection of SPCs managed by DPC

SPC

Financial benefit method

End User Computing Equipment Panel

This SPC provides a list of set prices for computing equipment such as desktop computers and laptops.

DPC engages an international market-intelligence advisory firm to benchmark SPC prices to what similar sized organisations are paying for the same computing equipment. The difference between these two prices is reported as the financial benefit for purchases on this SPC.

The benchmarking captures prices in the market at a point-in-time. However, DPC uses these benchmarked prices to calculate the financial benefits for purchases made past this point. The last benchmarking activity for this contract was conducted in October 2015. Prices in this industry move rapidly, so it is not a robust method to calculate the financial benefit attributable to equipment purchased in 2018 based on prices identified in October 2015.

The reported financial benefits in 2016–17 for mandated agencies was $63 million, which is higher than the $55.6 million spent by these agencies on the SPC during this year. In some cases, the benchmarking in 2015 identified discounts on computing equipment of up to 62 per cent compared to what a similar size entity was achieving. Based on the 2015 benchmarking activity, it is unclear how the Victorian Government received such a high financial benefit compared to similar-sized organisations. For these cases, we question the appropriateness of the entities selected for comparison. DPC was not able to provide details on which similar sized entities had been selected by the market-intelligence firm for benchmarking.

DPC has advised that it will explore alternative methods to calculate financial benefits for this SPC, including the frequency of benchmarking given the rate at which prices in this market change.

Software SPCs:

  • Oracle Software and Support
  • Salesforce Customer Relationship Management
  • Citrix Products and Services
  • IBM Enterprise Licencing Agreement
  • VMware Enterprise Licensing Agreement

The calculations of financial benefits for the five software SPCs are based on a set discount determined at the start of the contract. However, the price from which the discount should apply is unclear. Large organisations such as government departments would be able to negotiate savings and discounts without the existence of an SPC, due to their size and purchasing power. For the Oracle Systems SPC, DPC, as the lead agency, recognises this and has stated that it considers only 30 per cent of reported financial benefits as being truly attributable to the SPC being in place.

Source: VAGO, based on information provided by DPC.

4.5 Non-financial benefits

Lead agencies also establish SPCs to deliver non-financial benefits. For example, the establishment of a consistent set of legally endorsed standard terms and conditions—instead of several independent agency contracts—can reduce commercial risk and legal exposure. While some of these non-financial benefits are difficult to accurately measure or even estimate, lead agencies should capture and measure them where possible.

DTF and DPC advised that they monitor the realisation of non-financial benefits identified in SPC business cases through ongoing category management. However, they could not provide evidence to demonstrate reporting on the achievement of these non-financial benefits.

DJR measures and reports on the non-financial benefits pertaining to the operation of social justice policies incorporated in the Legal Services Panel SPC. For example, DJR monitors whether suppliers are promoting corporate social responsibility practices in the workplace, including:

  • pro bono activities—suppliers have committed to provide pro bono services equivalent to 20 per cent on average of the net legal fees they earned during the previous financial year
  • implementation of diverse work practices that promote gender equality and flexible work arrangements
  • the briefing of female barristers by both panel firms and clients in accordance with the Victorian Bar's equal opportunity briefing policy.

As Figure 4H shows, the 2016–17 Legal Services Panel Annual Report analyses and reports this information annually.

Figure 4H
Example of the non-financial benefits monitored and reported by DJR in the 2016–17 Legal Services Panel Annual Report

Figure 4H shows an example of the non-financial benefits monitored and reported by DJR in the 2016–17 Legal Services Panel Annual Report

Figure 4H shows an example of the non-financial benefits monitored and reported by DJR in the 2016–17 Legal Services Panel Annual Report

This data indicates more female than male lawyers are receiving promotion, regardless of whether they work full-time or have flexible work arrangements.

Source: DJR 2016–17 Legal Services Panel Annual Report.

4.6 Reliability of data

In analysing spend and savings data for SPCs, significant data issues hamper the VPS, both in terms of the quality of systems in departments and agencies, and the unavailability of comprehensive data on spend, price and volume.

Because SPC spend is fragmented across many users, and there is no whole‑of‑government procurement platform to provide overall visibility, lead agencies rely on suppliers to capture and self-report spend data. This is done monthly for the Legal Service Panel SPC and quarterly for the remaining SPCs, as Figure 4I shows.

Figure 4I
Capture and recording of supplier-reported spend and lead agency-reported savings data

Figure 4I shows a capture and recording of supplier-reported spend and lead agency-reported savings data

Source: VAGO based on information provided by DPC, DTF and DJR.

Lead agencies cannot easily validate self-reported supplier data and the quality of this data varies. Lead agencies undertake ad hoc checks of the data and require SPC users to confirm spend for some SPCs. However, these verification activities are limited, given lead agencies do not have all the information needed to sufficiently assess and validate the supplier reports, including access to the finance systems of SPC users.

Lead agencies should periodically audit provider reports to confirm their accuracy. However, DTF and DPC have not done so, despite the current contracts allowing for this. DTF and DPC need to address this because it affects the effectiveness and integrity of spend and benefits monitoring for SPCs.

Our review of information maintained by DTF and DPC also highlighted errors in the spreadsheets they use to track spend and benefits for SPCs. We found under-reporting of:

  • $5.2 million in non-mandated entity spend from 2015 to 2018 for the Stationery and Workplace Consumables SPC
  • $0.89 million in financial benefits during 2016–17 for the Travel Management Services SPC
  • $0.11 million in financial benefits in 2016–17 for the Salesforce Customer Relationship Management SPC.

We also found a further issue with the accuracy of financial benefits calculated for the Travel Management Services SPC. Spend for this SPC in 2016–17 was over-reported by $5.9 million due to changes in internal processes.

Given these data issues, the actual levels of savings achieved by SPC users is very likely to be different from those reported by lead agencies.

DTF uses this information as the basis for publicly reported performance information in the State Budget papers and its annual report, so it is important that it has quality assurance measures to ensure the reported information is accurate. DTF advised that it has rectified all errors identified by this audit.

Back to Top