Learning Technologies in Government Schools

Tabled: 12 December 2012

Overview

This audit reviewed the effectiveness of learning technologies in government schools. In particular, the audit focused on two key Department of Education and Early Childhood Development (DEECD) learning technology initiatives: VicSmart and the Ultranet.

The government's current policy for ICT in school education, The Digital Learning Statement (the Statement), does not provide a clear action plan or framework for investment in learning technologies. This means that departmental staff and school leaders have little guidance on how future learning technology initiatives can be appropriately planned and integrated.

Planning for the VicSmart high-speed fibre-optic network for all government schools was underpinned by a robust needs and options analysis. The high-speed connectivity that VicSmart provides is a key enabler of current and future digital learning in government schools.

In contrast, the Ultranet was poorly planned and implemented. Six years after its announcement, it is yet to achieve expected benefits for students, parents and schools. It is significantly late, more than 80 per cent over its first announced budget, has very low uptake, and does not have originally intended functionality.

This audit identified a number of serious probity, procurement and financial management issues surrounding the Ultranet project. DEECD’s tender process lacked rigour and was seriously flawed. There is little confidence in the financial management practices around the Ultranet project, and limited assurance that the selected outcome represented value for money.

Back to top

Learning Technologies in Government Schools: Message

Ordered to be printed

VICTORIAN GOVERNMENT PRINTER December 2012

PP No 207, Session 2010–12

The Hon. Bruce Atkinson MLC

President

Legislative Council

Parliament House

Melbourne

The Hon. Ken Smith MP

Speaker

Legislative Assembly

Parliament House

Melbourne

Dear Presiding Officers

Under the provisions of section 16AB of the Audit Act 1994, I transmit my report on the audit Learning Technologies in Government Schools.

Yours faithfully

Signature of D D R PEARSON (Auditor-General)

D D R PEARSON

Auditor-General

12 December 2012

Back to top

Audit summary

Information and communications technologies (ICT) are now extensively embedded into our modern lifestyles. Many innovative technologies are emerging, and teaching practice and content is now increasingly expected to be delivered ‘anytime, anywhere’.

Governments across Australia and in most other developed countries invest heavily in ICT for use in schools.

Over the past nine years, the Department of Education and Early Childhood Development (DEECD) has spent around $1.6 billion on the provision of ICT services for government schools. This amount excludes Commonwealth funding and schools’ own direct ICT expenditure. DEECD estimates that it will spend a further $2.6 billion over the next 10 years on ICT investment for government schools.

The term ‘learning technologies’ refers to a broad range of ICT tools that can be used to support learning, teaching and assessment. These tools include interactive boards and learning materials as well as networked technology such as online learning environments. Learning technologies also include devices such as handheld, desktop or laptop computers.

This audit reviewed the effectiveness of the application of learning technologies in government schools. In particular, the audit focused on two key DEECD learning technologies initiatives: VicSmart and the Ultranet.

The audit assessed in detail whether DEECD has:

  • conducted robust research on the most appropriate use of ICT in government school teaching environments
  • developed effective strategies to implement learning technologies in government schools and supported teaching staff to adopt these technologies
  • monitored achievement of desired outcomes, including realisation of purported benefits described in business cases used to support funding decisions for VicSmart and the Ultranet.

Conclusion

DEECD’s Digital Learning Statement (the Statement) does not provide a clear action plan or framework for investment in learning technologies. There is no supporting strategy or further detail to the Statement. This means that departmental staff and school leaders have little guidance on how future learning technology initiatives can be appropriately planned and integrated to build upon past and present ICT investments.

Planning for the VicSmart high-speed fibre-optic network for all government schools was underpinned by a robust needs and options analysis. Although the project was a less complex infrastructure rollout, it was well executed and is delivering its desired benefits. The high-speed connectivity that VicSmart provides is a key enabler of current and future digital learning in government schools.

In contrast, the Ultranet, the Statement’s key foundation plank and key enabler, was poorly planned and implemented. Six years after its announcement as a government priority, it is yet to achieve expected benefits for students, parents and schools. It is significantly late, more than 80 per cent over its first announced budget, has very low uptake by users, and does not have the functionality originally intended.

This audit identified a number of serious probity, procurement and financial management issues surrounding the Ultranet project. DEECD’s tender process lacked rigour and was seriously flawed. There is little confidence in the costing and financial management practices around the Ultranet project, and limited assurance that the selected outcome represented value for money.

Findings

Digital Learning Statement

The Digital Learning Statement—the government’s current policy document on the use of learning technologies—was not informed by robust and comprehensive research and does not make a clear and cogent case for government investment in learning technologies.

The Statement does not deliver on the directive in the 2008 Blueprint for Education and Early Childhood Development to provide a plan of action to use learning technologies in teaching and learning. A review of DEECD’s advice to the Minister for Education shows that the government was not advised that the Statement did not comply with the Blueprint’s directive to develop and deliver a strategy.

To date, there has been no accompanying detailed strategy developed to support the Statement, even though this was originally planned.

VicSmart high-speed broadband for schools

Planning for VicSmart was underpinned by a robust needs and options analysis, as demonstrated by its 2005 business case, which articulated the needs to be addressed and provided a clear rationale for the purchase of high-speed fibre-optic broadband connectivity. The business case provided confidence that the project was achievable and could be delivered as planned.

The VicSmart procurement process was streamlined by using a mandated whole-of-government single-source provider. The fibre-optic system is performing as expected and has been upgraded incrementally to meet emerging data and connectivity needs across the government school system.

Ultranet e-learning system

The Ultranet project was poorly planned and implemented. None of its three business cases had a well thought out needs analysis or gave considered options to deliver the project. The various business cases did not answer the ‘Why invest?’ question for the Ultranet, nor did they provide a sound basis for the project’s approval.

Some six years since its announcement as a government priority, the Ultranet has not delivered its main objectives:

  • to improve responsiveness to individual learning needs
  • to provide better information to parents, the school system and government
  • to improve the efficiency of the learning environment and school administration.

Consistent with public sector practice, the Department of Treasury and Finance (DTF) and the Department of Premier and Cabinet (DPC) provided advice to government at key decision points over the life of the Ultranet project.

The project continued despite advice from central agencies that it should cease or be delayed. Further, there is no trail of documentary evidence to explain whether or how DEECD addressed the many critical issues raised by DPC and DTF.

It is difficult to understand why the Ultranet procurement was able to proceed to contract execution, given the significant concerns raised by DPC and DTF, as well as the many adverse ratings that DEECD had received from various Gateway reviews since the project first commenced.

Further, this audit detected a number of serious process and probity issues in relation to tendering and procurement for the Ultranet. DEECD has advised that it has commenced a number of actions and further detailed investigations in response to these matters.

There is little confidence that the financial management practices relating to the Ultranet were sound and that full costs have been adequately recorded. VAGO estimates that actual capital and operating expenditure for the Ultranet was approximately $162 million as at June 2012, and by June 2013 it is likely to have cost approximately $180 million. DEECD has advised that it is currently investigating the financial practices in relation to this major ICT project.

Despite this significant expenditure, no cost-benefit analysis has been conducted to determine whether the Ultranet provides value for money, or whether the same functionality could have been delivered more cost effectively.

Performance indicators for the Ultranet have been revised down over time and do not provide appropriate measures of whether the Ultranet is achieving what the government expected when it funded the project.

Use of the Ultranet is low, and declining. On average, only 10 per cent of students and 27 per cent of teachers logged in on a monthly basis from July 2011 to May 2012.

An underlying factor which has limited the effective implementation of the Ultranet is the significant discrepancy between the original scope of the project and expected benefits and what has actually been implemented and delivered. This underscores the urgent need for DEECD to review whether it should continue to invest in this project.

Further, DEECD did not adequately manage the change processes required to maximise the Ultranet’s acceptance and, therefore, the state’s return on investment. Teachers and parents were not appropriately trained and supported to use the Ultranet. Ultimately, the Ultranet is only a technology tool, and cannot by itself deliver the benefits intended from it.

Recommendations

The Department of Education and Early Childhood Development should:

  1. develop a comprehensive and evidence‑based strategy or plan of action for use of learning technologies to underpin and guide the significant investment in ICT for government schools
  2. develop performance indicators that measure both VicSmart’s ongoing operational performance and its achievement of intended benefits
  3. urgently review its investment in the Ultranet, with a particular focus on:
    • assessing whether the contractor has delivered all functionality as required by the contract and what action, if any, needs to be taken to enforce the state’s rights
    • rigorously assessing its financial management practices and identifying the real, current cost of the Ultranet to determine the extent to which further investment is warranted
    • identifying and addressing the underlying causes of low take‑up rates across the school system by teachers, students and parents
    • providing advice to government on the cost-benefit of decommissioning the system now against continuing to fund and rectify the system so that it can be implemented as originally expected
  4. conduct an agency-wide review of its internal tendering, probity and financial management practices in light of the serious issues identified by this audit
  5. expedite the provision of guidance to schools on the current status of the Ultranet as the department’s key learning technology investment, and clarify the policy context of schools’ autonomy in purchasing non-Ultranet learning technologies.

Submissions and comments received

In addition to progressive engagement during the course of the audit, in accordance with section 16(3) of the Audit Act 1994 a copy of this report, or relevant extracts, were provided to the Department of Education and Early Childhood Development, the Department of Premier and Cabinet and the Department of Treasury and Finance, with a request for submissions or comments.

Agency views have been considered in reaching our audit conclusions and are represented to the extent relevant and warranted in preparing this report. Their full section 16(3) submissions and comments are included in Appendix A.

1. Background

1.1 Introduction

The role of information and communications technologies (ICT) in teaching and learning continues to grow. Many innovative technologies are emerging, and teaching practice and content is now increasingly expected to be delivered ‘anytime, anywhere’.

Over the past nine years, the Department of Education and Early Childhood Development (DEECD) has spent around $1.6 billion on the provision of ICT services in government schools. This amount excludes Commonwealth funding and schools’ own direct ICT expenditure. DEECD estimates that it will spend a further $2.6 billion over the next 10 years on ICT investment for government schools.

Identifying the impact of ICT on student learning is a challenging exercise. In 2010, the Organisation for Economic Cooperation and Development reported that there is no ‘clear cut evidence on the expected benefits’ from learning technologies.

In light of this, it is important that government investment in learning technologies is underpinned by rigorous business cases and planning, careful project implementation, and effective evaluation of achievement against expected outcomes.

1.2 Learning technologies

The term ‘learning technologies’ refers to a broad range of ICT tools that can be used to support learning, teaching and assessment. These tools include interactive boards and learning materials as well as networked technology such as online learning environments. Learning technologies also include devices such as handheld, desktop or laptop computers.

DEECD defines the role of learning technologies as tools or resources ‘to support teaching and learning by enabling users to communicate, create, collaborate, disseminate, store and manage information’.

1.3 Digital Learning Statement

The government’s Digital Learning Statement (the Statement) was released in October 2010 and is the current policy relating to ICT use in school education.

The Statement’s vision is for all teachers and students to have access to contemporary technology and world-class digital content with which to create, communicate and collaborate locally and globally.

The Statement notes that learning technologies in the school environment can:

  • improve students’ confidence and attitudes towards their own learning, behaviour and attendance
  • promote improved opportunities for students to learn through collaboration and conversation
  • improve connections with the real world and provide access to global communities with expertise and perspectives that can enrich learning.

The Statement identified three objectives:

  • provision of ‘anywhere, anytime’ access
  • advancing teacher practice in the use of digital learning in teaching
  • access to high quality resources, tools and data.

1.4 VicSmart broadband network

The 2005–06 State Budget allocated $89.3 million to VicSmart (first referred to as SmartOne when it was announced in April 2005) to provide at least 2 megabits per second (Mbps) broadband data communication services to all government schools.

Full implementation of VicSmart required the laying of fibre-optic cables to the ‘school gate’. The network was designed to allow schools to effectively access online information and resources. Prior to VicSmart, most government school bandwidth was limited to 64 kilobits per second (Kbps), which did not allow for quick access to online materials.

VicSmart was delivered through the whole-of-government 2002 Telecommunications Procurement and Management Strategy. This strategy consolidated the purchase and management of fixed voice, mobile, telephony and data carriage services across government.

The contracts for the provision of voice, mobile and data carriage services are jointly referred to as telecommunications carriage services. In July 2004, the telecommunications carriage services contract was awarded for the provision of broadband data services, including to all government schools.

1.4.1 Rollout of VicSmart

The VicSmart rollout started in October 2005. Haddon Primary School near Ballarat and North Fitzroy Primary School were the first beneficiaries. Internet connectivity increased roughly 60 times—from 64 Kbps to 4 Mbps.

Since its first rollout in 2005, VicSmart had been upgraded four times:

  • In 2009, all government schools were upgraded from 4 Mbps to at least 10 Mbps.
  • In 2010, larger school campuses received faster bandwidth, at 20 Mbps and 50 Mbps. The basis for this upgrade was student population. Schools with enrolments of 500 to 999 students were upgraded to 20 Mbps, and those with enrolments greater than 1 000 received 50 Mbps bandwidth.
  • Later in the same year, all metropolitan government schools with 10 Mbps broadband bandwidth were upgraded to 20 Mbps. The bandwidth allocation for non-metropolitan schools remained at 10 Mbps due to additional costs estimated at $4.5 million per annum.

In 2012, a further 244 schools were upgraded, including 11 that received up to 100 Mbps bandwidth. The main driver for this upgrade was the need to support access to network applications such as videoconferencing and voice over internet protocol, which require Quality of Service network technology. Until this time, VicSmart provided base bandwidth only, and did not support Quality of Service traffic.

1.5 The Ultranet e-learning system

The Ultranet is a Victoria-wide learning platform and ICT system that students, parents and teachers of government schools can access via the internet.

The DEECD website says that students and teachers can use the Ultranet for online learning activities and for collaborating and communicating with students from within their school and across Victorian government schools.

The system is also designed to allow parents to access information that will help them keep up to date with their child’s learning.

The Ultranet was described in the Statement as the key foundation plank or key enabler of the three components of the vision for ICT in the classroom.

1.5.1 Election commitment

In November 2006, Victorian parents were told by the government that they would get virtual access to their child’s classroom 24 hours a day, seven days a week under a $60.5 million plan to revolutionise learning.

Figure 1A shows the components of the Ultranet announced in November 2006.

Figure 1A
Election commitment for Ultranet components

Commitment

Connect the entire school community—parents, students, teachers and administrators—in a way never before possible.

Consolidate school administrative functions into the one system and lighten teachers’ burden.

Allow parents to:

  • log into their child’s classroom
  • check teachers’ lesson plans and their child’s homework and results
  • communicate directly with teachers by email
  • monitor school attendance, which will be recorded twice a day
  • receive automatic notification of non-attendance via text, email or phone call.

Give students:

  • their own virtual work spaces with homepages and personally tailored lesson plans, which will be accessible via the internet
  • the ability to keep up to date with their education while away from school due to illness or family reasons and get teacher feedback much faster.

Source: Victorian Auditor-General’s Office, based on Department of Education and Early Childhood Development media release, issued 13 November 2006.

The Ultranet was presented as a revolutionary educational tool that would ‘connect the entire school community—parents, students, teachers and administrators—starting in 2007’.

1.5.2 The 2007 Ultranet business case

The 2007 business case vision for the Ultranet was for an intuitive, student-centred electronic learning environment that supports high quality learning and teaching. Its objectives were to:

  • improve responsiveness to individual learning needs
  • provide better information to parents, the school system and government
  • improve the efficiency of the learning environment and school administration
  • adopt an enterprise approach to intranet development
  • exploit previous ICT investments.

The business case noted that there were many commercial ICT solutions available in the market, and that many schools were already using these. However, it emphasised that no single solution could provide the full level of scalability, security, interoperability and functionality that is needed to meet the identified business needs and objectives of the Ultranet.

According to the Ultranet business case, some 1.5 million Victorian students, teachers and parents were expected to use the Ultranet when fully operational. Figure 1B summarises the Ultranet’s identified eight key performance indicators.

Figure 1B
Ultranet key performance indicators

Benefit

Key performance indicators

Establish a school environment for the future

Number of students using Ultranet (target 70 per cent)

Improve the educational outcomes for Victorians

Number of students accessing online learning resources in the Ultranet (target 65 per cent)

 

Number of teachers accessing resources, preparing curriculum plans, delivering Ultranet-based resources (target 65 per cent)

Reduce the administrative burden on teachers and school leaders

Efficiency in administrative processes (target 20 per cent over baseline data)

Reduce the future cost of education

Number of external software packages used to assist school administration (target 30 per cent decrease over baseline data)

 

Cost of external software packages used by government schools to assist in school administration (target 15 per cent over baseline)

Improve educational opportunities for regional, rural and remote Victorians

Number of students participating in online collaborative technology via the Ultranet (target 30 per cent of regional and rural schools’ students)

 

Number of teachers accessing online content and communities via the Ultranet (target 30 per cent of regional and rural schools’ teachers)

Source: Victorian Auditor-General’s Office based on Department of Education and Early Childhood Development documentation and the 2007 Ultranet business case.

1.5.3 Failed tender in 2007 and subsequent re-scoping

A request-for-tender to develop and manage the Ultranet was released to the market on 1 August 2007. All bids received were in excess of $100 million and resulted in DEECD being unable to nominate a successful tenderer.

Following this failed tender, the Ultranet project was re-scoped to fit the approved budget of $60.5 million. Consequently, the Ultranet’s business requirements were reduced by 90 per cent from the 1 260 functions stated in the 2007 business case to 131 functions contained in the 2008 request-for-tender specifications.

The revised Ultranet project focused on the delivery of learning and teaching components, a parent portal and ‘sufficient’ functionality to support these areas. When re-scoping the Ultranet, consideration was also given to purchasing an off-the-shelf product rather than developing a bespoke solution.

In addition to re-scoping and reducing the Ultranet’s functionality, the concurrent user requirement was reduced from 250 000 users to 78 000 users and total storage was reduced from 330 terabytes (TB) to 160 TB, reductions of 69 per cent and 51 per cent respectively.

1.5.4 Procurement and deployment

DEECD released a selective request for tender in November 2008. Four bids were received when the tender closed on 23 December 2008.

The Ultranet Master Agreement was signed with the successful tenderer on 30 June 2009 for the contract price of $64.6 million. The price included $47.5 million for the build and delivery, and $17.1 million for application hosting, software maintenance, disaster recovery and help desk support until 30 June 2013.

The Ultranet rollout to government schools commenced in May 2010. On 9 August 2010, government school principals and assistant principals attended the Ultranet ‘Big Day Out’ at the Melbourne Convention and Exhibition Centre. Government school teachers across the state were asked to sit at their desks to participate in a trial of the Ultranet.

However, the Ultranet suffered user access issues and teachers waited all morning for the system to become available. DEECD explained to the participants that technical issues were affecting the Ultranet and that work was being done to correct the problems. When the Ultranet went live in the afternoon, it suffered further login issues and performed slowly and unreliably.

Even in its reduced form, significant Ultranet functionalities have not been fully delivered, and they continue to be deployed to government schools at the time of finalising this report.

1.5.5 Ultranet oversight

Executive sponsorship and management of the Ultranet rests with the Ultranet Project Board, comprised of executives from various DEECD divisions and some private sector representatives. The Ultranet Project Board meets monthly and is chaired by the DEECD Deputy Secretary, School Education Group.

The Ultranet Project Board is responsible for the Ultranet’s business management and project delivery functions. The project team reports to the board and its responsibilities include giving advice on Ultranet strategy and policy, change management within schools, and DEECD’s relationship with the contractor. Project delivery functions include contract and service level management, data analysis, major incident management, and solution architecture and integration.

Software development, release management and application support, are the contractor’s responsibilities.

Operational functions relating to the live ICT system are managed either by the contractor or by DEECD’s Information and Technology Division. A contractor-provided help desk is currently available to users until 30 June 2013.

1.6 Audit objective and scope

The objective of this audit was to examine the effectiveness of the application of learning technologies in government schools.

In particular, the audit examined DEECD’s planning, delivery and management of learning technologies across the government school system.

To address the objective, the audit examined whether DEECD:

  • conducted robust research on the most appropriate use of ICT in government school teaching environments
  • developed effective strategies to implement learning technologies in government schools and supported teaching staff to adopt these technologies
  • is monitoring achievement of desired outcomes, including realisation of purported benefits described in the business cases used to support funding decisions for the Ultranet and VicSmart.

1.7 Audit method and cost

The audit was performed in accordance with section 15 of the Audit Act 1994 and the Australian Auditing and Assurance Standards.

The total cost was $310 000.

1.8 Report structure

The report is structured as follows:

  • Part 2 – Digital Learning Statement
  • Part 3 – Case study: VicSmart broadband connectivity for schools
  • Part 4 – Case study: the Ultranet e-learning system.

2. Digital Learning Statement

At a glance

Background

The government’s Digital Learning Statement (the Statement) was released in October 2010 and remains the current policy for information and communications technology (ICT) in school education. Its vision is for all teachers and students to have access to contemporary technology and world-class digital content with which to create, communicate and collaborate locally and globally.

Conclusion

The Statement does not provide a strategy, action plan or framework for investing in learning technologies.

This means that departmental staff and school leaders have little guidance on how learning technology initiatives can be appropriately planned and integrated to build upon past and present ICT investments.

Findings

  • The Statement was not informed by robust and comprehensive research and does not make a clear and cogent case for government investment in learning technologies.
  • The Statement did not deliver on the directive of the 2008 Blueprint for Education and Early Childhood Development for the Department of Education and Early Childhood Development (DEECD) to develop a strategy or plan of action to use learning technologies in teaching and learning.

Recommendation

DEECD should develop a comprehensive and evidence-based strategy or plan of action for use of learning technologies to underpin and guide the significant investment in ICT for government schools.

2.1 Introduction

The government’s Digital Learning Statement (the Statement) was released in October 2010 and is the current policy information and communication technologies (ICT) use in school education. It was developed by the Department of Education and Early Childhood Development (DEECD) and was released in response to the 2008 Blueprint for Education and Early Childhood Development (the Blueprint).

The Blueprint required DEECD to develop and release an overarching ICT strategy for teaching and learning.

Action 7 of the Blueprint referred to the government’s commitment to improve Victorian students’ performance in literacy and numeracy across all sectors. The first step in furthering this commitment was to clearly set out the government’s approach to unlocking the full potential of ICT usage in schools.

2.2 Conclusion

The Statement has no clear action plan or framework for investment in learning technologies in the government school system.

Consequently, departmental staff and school leaders have little guidance on how learning technology initiatives can be appropriately planned and integrated to build upon past and present ICT investments.

2.3 Vision and objectives for learning technologies

The Statement describes the government’s vision and objectives for learning technologies in Victorian government schools. The vision is that:

  • all teachers and students have access to contemporary technology and world‑class digital content with which to create, communicate and collaborate locally and globally
  • student learning is engaging, personalised and authentic to enable them to become confident and creative individuals and active and informed citizens of the 21st century.

The three objectives outlined by the Statement are:

  • Anywhere, anytime access—all students and teachers have appropriate and equitable opportunities to use digital devices and systems, including access to a range of contemporary digital devices and sophisticated online systems that support learning, teaching and knowledge sharing.
  • Advancing teaching practice—all teachers and school leaders build capabilities to use digital technology to improve student learning.
  • Access to high quality resources, tools and data—all teachers and students have access to quality assured digital resources, tools and rich data sources.

2.4 Lack of strategy or action plan

The Statement sets out the vision and objectives for the use of learning technologies in government schools. It also highlights previous and ongoing government investments in learning technologies.

However, it does not provide a strategy, clear action plan or framework to achieve the stated vision and objectives and to align future investments.

DEECD intended to develop a strategy or framework for future ICT initiatives when the department was developing the Statement. A proposal was made to identify the appropriate initiatives and investments required to address gaps and barriers identified in the then current delivery of learning technologies in government schools.

However, this work was not undertaken, and the Statement did not include any reference to identified barriers, or a strategy for future action.

As a consequence, the Statement provides little guidance to DEECD staff and school leaders on how future initiatives can and should be appropriately planned to build upon past and present investments.

This also means that the Statement did not deliver on the Blueprint’s expectation to develop a strategy to use learning technologies in teaching and learning.

DEECD asserts that although the Blueprint committed to develop and release a strategy, it was the government’s decision to release the Statement, which did not include a strategy.

However, a review of DEECD’s advice to the Minister for Education shows that the government was not advised that the Statement did not comply with the Blueprint’s directive to develop and deliver a strategy. No further action was taken to develop a strategy.

2.5 Research for the Statement

The Statement used some national and international research to inform its development. It cites a 2007 UK research report suggesting that learning technologies can improve learning outcomes.

However, the cited literature states that the evidence on the impact of ICT on learning is ‘as yet inconsistent’, and that there is not a sufficient body of evidence from which ‘to draw firm conclusions’.

This view is shared by the Organisation for Economic Cooperation and Development (OECD), which states in its March 2010 research report, Technology Use and Educational Performance in PISA 2006, that policy makers in OECD countries have had to adjust their expectations on the impact of ICT use in teaching and learning because they could not see clear-cut evidence of the expected benefits.

An acknowledgement in the Statement of this more cautious view would have provided a clearer picture of the challenging context for the use of learning technologies in schools and given greater scrutiny of the purported benefits of any investment.

The lack of clear evidence on the effectiveness of learning technologies in enhancing teaching and learning could also have prompted the development of a clear action plan for learning technologies in Victorian government schools that focused more realistically on how best to minimise cost and risk while maximising benefits.

The Statement’s absence of robust and comprehensive research means that it does not make a cogent case for government investment in learning technologies.

2.5.1 Evaluation of programs

In the absence of clear-cut evidence, some reference to outcomes or lessons learned from previous government investments in learning technologies would have provided a stronger basis or rationale for the Statement.

DEECD has not been evaluating the effectiveness of most of its learning technology initiatives. Therefore, there is no objective evidence that government’s significant investment in learning technologies has delivered the intended benefits for Victorian students.

2.5.2 Stakeholder consultation

The Statement was not informed by consultation with key stakeholders beyond departmental staff.

Input from teachers, parents and student representative groups was not sought to clarify expectations or perspectives on what does and doesn’t work with learning technologies.

Recommendation

  1. The Department of Education and Early Childhood Development should develop a comprehensive and evidence-based strategy or plan of action for use of learning technologies to underpin and guide the significant investment in ICT for government schools.

3. Case study: VicSmart broadband connectivity for schools

At a glance

Background

The VicSmart project was rolled out between October 2005 and December 2011. Internet connectivity speeds in schools increased roughly 60 times from the previous 64 Kbps (kilobits per second) to 4 Mbps (megabits per second). All government schools are now connected to VicSmart.

The VicSmart procurement process was relatively straightforward because the Department of Education and Early Childhood Development (DEECD) was required to use the whole-of-government single-source provider.

Conclusion

VicSmart was well planned and underpinned by a business case that provided a robust needs and options analysis.

Although no performance indicators have been developed to measure VicSmart’s intended benefits, it is evident that VicSmart is a key enabler of digital learning in government schools. It achieves this by providing high quality broadband data access to online DEECD and internet-based digital learning resources.

Findings

  • VicSmart has provided high-capacity and high-speed fibre-optic connectivity to all Victorian government schools.
  • Use of VicSmart is increasing in line with the increase in end-user computing devices in government schools.
  • Network capacity has been appropriately upgraded to meet increasing bandwidth needs over time, with much of this increase matching the increase in internet access and usage by government schools.

Recommendation

DEECD should develop performance indicators that measure both VicSmart’s ongoing operational performance and its achievement of intended benefits.

3.1 Introduction

The rollout of the VicSmart project began in October 2005 and was completed in December 2011.

After installation of the fibre-optic cable to the ‘school gate’, internet connectivity speeds increased roughly 60 times from the previous 64 Kbps (kilobits per second) to 4 Mbps (megabits per second).

All Victorian government schools are connected to the VicSmart fibre-optic network.

3.2 Conclusion

The VicSmart project was well planned and underpinned by a business case providing a robust needs and options analysis.

The Department of Education and Early Childhood Development (DEECD) delivered VicSmart on time and on budget. The project has delivered a high-speed broadband fibre-optic network providing internet access, email and a range of digital applications. Schools report very few problems with VicSmart and its quality, functionality and performance parameters are meeting expectations.

Although no specific performance indicators have been developed to measure the realisation of VicSmart’s intended benefits, it is evident that VicSmart is a key enabler of digital learning in government schools by providing high quality broadband data access to online DEECD and internet-based digital learning resources.

3.3 Findings

3.3.1 Planning for the project

Prior to VicSmart, all Victorian government schools received a narrowband internet connection through the VicOne network. Ninety per cent of government schools were provided with a 64 Kbps connection speed, while the remaining schools received slightly higher bandwidth provisions.

When first implemented in 1998, VicOne delivered productivity improvements as emails replaced surface mail and many administrative reports moved online.

However, increased demand for internet connectivity placed significant pressure on the VicOne network. By 2004, online access became impractical due to the considerable length of time required to download emails and web-based reference materials. DEECD data shows that in October 2004, Victoria was providing the lowest internet bandwidth connectivity among all states and territories.

Planning to replace VicOne was underpinned by a robust needs and options analysis which was contained in the VicSmart project’s 2005 business case. This effectively articulated the needs to be addressed and provided a clear rationale for the chosen option.

It also clearly established a case for government expenditure on the purchase of high‑speed fibre-optic broadband connectivity as well as providing confidence that the project was achievable and could be delivered as planned.

The business case also noted that broadband internet connections available to most Victorian households in 2005 had raised expectations about the performance of online services. This placed additional pressure on the government to provide comparable access to online services at school.

Three options were considered for the VicSmart project. A clear analysis of the strengths and weaknesses of each option was provided in the business case.

The preferred option was for all schools to receive an entry-level broadband fibre-optic service upgradeable to 100 Mbps.

The business case explained that this was the best option available because:

  • it provided a scalable platform for the growing data services needs of schools
  • it avoided future costs of up to $60 million by taking advantage of previously negotiated pricing for installation of optical fibre in non-metropolitan schools
  • it delivered high quality data services to both metropolitan and non-metropolitan schools.

3.3.2 Project delivery

In February 2005 the government formally approved VicSmart as an $89 million project to be run over four years. This approved budget was in addition to DEECD’s existing $18.6 million recurrent annual bandwidth budget.

VicSmart-related costs have therefore been $163 million for the period 2005–06 to 2008–09, plus $48 million in ongoing recurrent costs from 2009–10.

Since it was compulsory for DEECD to use a sole-source contractor, the procurement process for VicSmart was relatively straightforward. This saved DEECD time and resources by not having to develop request-for-tender documentation, solicit bids and evaluate tender proposals. The procurement process was limited to organising purchase orders with the sole-source provider in accordance with the VicSmart project specification.

Since completion of rollout, the VicSmart project has provided high-capacity and high‑speed fibre-optic connectivity to all Victorian government schools. Use of VicSmart is increasing in line with the increase in end-user computing devices in government schools.

Network capacity has also scaled-up appropriately to meet increasing bandwidth needs. Much of this increase matches the increase in internet access and usage by government schools. From the initial 2 Mbps broadband service, upgrades have been achieved progressively, as indicated in Figure 3A.

Figure 3A
VicSmart upgrades over time

Upgrade

Description

Phase 1: 2005

All schools transitioned from VicOne to VicSmart (minimum 2 Mbps in each school)

Phase 2: 2008

Scalable 4 Mbps to all schools connected by fibre-optic cable

Phase 3: 2010

Upgrade to 10 Mbps for all schools connected by fibre-optic cable

Phase 4: 2010

Enhanced provisioning of 20 Mbps to 50 Mbps for larger school campuses

Phase 5: 2012

Enhanced provisioning to up to 100 Mbps for larger school campuses

Source: Victorian Auditor-General’s Office review of Department of Education and Early Childhood Development documents.

Timely communication with schools on VicSmart’s implementation and upgrade has meant little disruption to school activities.

During the rollout, regional schools were not disadvantaged and generally received the service at about the same time as metropolitan schools.

DEECD also advised all schools prior to the rollout of VicSmart that the significantly higher download speeds would affect their internet service provider (ISP) costs. Schools were asked to develop strategies to monitor and manage ISP costs, as DEECD currently pays all ISP-related costs for government schools.

Continuing to benchmark VicSmart against national and international standards, DEECD has upgraded speeds to up to 100 Mbps (depending on school enrolments), making Victoria compliant with the Ministerial Council for Education, Early Childhood Development and Youth Affairs bandwidth provisioning target of 8.7 Kbps per student.

3.3.3 Benefits realisation

VicSmart is delivering the intended benefits of providing reliable high-speed internet connectivity to all Victorian government schools.

Although no performance indicators have been developed to measure VicSmart’s intended benefits, it is evident that VicSmart is a key enabler of digital learning and provides high quality broadband data access to both online DEECD and internet-based digital learning resources.

Figure 3B
Assessment of VicSmart’s expected benefits

VicSmart benefits as defined in the 2005 business case

Achieved

Learning and teaching benefits

 
  • Provide access to online multimedia learning materials from any school computer.

Yes

  • Capacity to deliver lesson via videoconference between all schools, leading to increased staffing flexibility.

Yes

  • Enable online testing and examinations from any school site

Yes

Administration and management benefits

 
  • Able to centralise data and software, lowering cost and improving data quality and reporting.

Yes

Schools infrastructure benefits

 
  • Future cost reduction. All schools provided with scalable optical fibre‑based broadband. Affordable upgrades available as required.

Yes

  • Improved productivity of information and communications technologies resources by enabling support and configuration from clusters or centrally to optimise service delivery, performance and efficiency.

Yes

  • Improved productivity. Software will be deployed and supported across the network rather than onsite.

Yes

Source: Victorian Auditor-General’s Office.

DEECD receives monthly reports on VicSmart’s operational performance. Information from the service provider’s monthly reports is used to inform key management decisions, including phased upgrades of VicSmart over time.

Principals and teachers at the schools visited during the audit, as well as other stakeholders, have confirmed that VicSmart is enabling digital learning by providing access to online resources and enabling videoconferencing between schools.

A teacher in one of the visited schools said that if VicSmart was to be turned off, ‘digital learning in Victorian government schools would come to a halt’.

Recommendation

  1. The Department of Education and Early Childhood Development should develop performance indicators that measure both VicSmart’s ongoing operational performance and its achievement of intended benefits.

4. Case Study: the Ultranet e‑learning system

At a glance

Background

The Ultranet is a Victoria-wide digital learning platform and ICT system. Government school students, parents and teachers can access the Ultranet via the internet.

Conclusion

The Ultranet was poorly planned and implemented. Some six years after its announcement as a government priority, it is yet to deliver the benefits expected for students, parents and schools. It is significantly late and over budget, has very low uptake by users, and has reduced functionality compared with what was originally announced.

This audit also identified a number of serious probity, financial management and procurement issues.

Findings

  • The Ultranet has not delivered its main objectives or expected benefits.
  • There were a number of serious process issues and apparent probity breaches in relation to the Ultranet procurement.
  • No cost-benefit analysis has been conducted to determine whether the Ultranet provides value for money, or whether the same functionalities could have been delivered more cost effectively.
  • Use of the Ultranet is low, and declining.
  • Poor quality financial data means that actual capital and operating expenditure for the Ultranet is unclear.

Recommendations

The Department of Education and Early Childhood Development should:

  • urgently review its investment in the Ultranet
  • conduct an agency-wide review of its internal tendering, probity and financial management practices in light of the serious issues identified by this audit
  • expedite the provision of guidance to schools on the current status of the Ultranet as the department’s key learning technology investment, and clarify the policy context of schools’ autonomy in purchasing non-Ultranet learning technologies.

4.1 Introduction

The Ultranet was announced in November 2006 as a $60.5 million project. It was expected to provide an online teaching, learning and assessment system for all government schools.

Over its life, the Ultranet has been the subject of three business cases and two tender processes.

The project is in a ‘business as usual’ phase of the contract, which expires in June 2013.

4.2 Conclusion

The Ultranet has been poorly planned and implemented. None of its three business cases had a well thought out needs analysis or gave comprehensive options to deliver the project.

Some six years since its announcement as a government priority, the Ultranet has not delivered any of the main objectives that were expected when the project was funded.

The Ultranet is significantly late and over budget—and with limited functionality—when compared with what was originally announced. The full costs of the project are poorly recorded by the Department of Education and Early Childhood Development (DEECD) and a conservative estimate of actual costs by June 2013 is $180 million, or close to 300 per cent above the first announced budget.

This audit has detected a number of serious procurement and probity lapses that have triggered further review and action by the department. Although the department’s work is not yet complete, there is no assurance that the tender process was rigorous, or that the chosen delivery option represented value for money.

Use of the Ultranet is well below expectations, with only 10 per cent of students and 27 per cent of teachers logging into the system. Because of its very low and declining usage, and the ambiguous guidance from DEECD about whether schools can opt out of the system, the viability of the Ultranet as the government’s key learning technology investment is now under serious threat.

DEECD has not given sufficient attention to understanding the negative effect that the continuing low usage, as well as the opt-out issue, is having on delivery of expected benefits from the Ultranet. These shortcomings are also diluting the return on investment for the significant public funds that have already been spent.

4.3 Findings

4.3.1 Planning for the project

Three business cases

Over time, the Ultranet project has been the subject of three business cases, with one rejected and two endorsed by government.

None of these business cases provided a robust needs analysis and also did not provide a sound basis for approval of the project. They did not answer the ‘Why invest?’ question for the Ultranet.

The first business case was developed in early 2007 and was rejected by government. The second business case, submitted in July 2007, was a revised version of the first. This formed the basis of the (failed) tender process initiated in 2007.

The third business case was presented to government in May 2009 after the second procurement process had been initiated and almost immediately before the Ultranet contract was awarded to the service provider in June 2009.

This was a serious departure from the usual process of developing a business case prior to embarking on a procurement process. The Department of Treasury and Finance’s (DTF) Lifecycle Investment Framework states that the development of a business case must occur before the procurement phase.

Needs analysis

The 2007 and 2009 business cases stated a number of business drivers for the development of the Ultranet described in Figure 4A.

Figure 4A
Identified needs for the Ultranet

2007 business case

2009 business case

Critical need for a customised learning environment, i.e. teachers are unable to centrally source and share learning resources.

Same

Poor quality of information available to education stakeholders, i.e. teachers and school leaders have difficulties in tracking student progress from year to year and between schools.

Same

Failure to maximise return on previous information and communications technologies (ICT) investments, i.e. existing disparate school systems inhibit interoperability with other systems.

Same

Massive inefficiencies in administration for schools, i.e. too much time is spent on time‑consuming tasks that take the focus away from learning and teaching.

Not included – administration was dropped as an Ultranet functionality in the 2009 business case.

Source: Victorian Auditor-General’s Office.

The purpose of a needs analysis in a business case is to identify the fundamental need for an investment including the problems, issues or circumstances of the current situation that needs to be addressed.

Both business cases failed to provide sufficient material to explain the context and extent of the identified issues. For example, with regard to ‘massive inefficiencies in administration’, the business case did not explain how much teachers’ time would be released through the automation of these administrative tasks, nor did it estimate the extent to which this could improve teachers’ productivity.

Options analysis

Three options were considered in both the 2007 and 2009 business cases.

Figure 4B
Options analysis for the Ultranet

2007 business case

2009 business case

Do nothing—allow schools to determine and purchase their own learning software solution ($67 million estimated cost)

Do nothing—allow schools to determine and purchase their own learning software solution ($67 million estimated cost)

Further develop and deploy early version of Ultranet trialled in several schools in 2006 ($72 million estimated cost)

Negotiate contract with preferred tenderer with DEECD/contractor as integration partners ($60.4 million estimated cost)(a)

Develop and deploy an Ultranet solution ($83 million estimated cost)(a)

Negotiate contract/s with preferred tenderer and a third party as systems integrator ($68.6 million estimated cost)

(a) Preferred option.

Source: Victorian Auditor-General’s Office analysis of Department of Education and Early Childhood Development documents.

In the 2009 business case, there was little relationship between the options presented and the needs identified for the proposed project. Instead of presenting options on the appropriate type of investment to address the identified needs, the options were types of contractual arrangements that DEECD could have with the preferred tenderer.

The options presented for 2009 should have been similar to those of the 2007 business case, albeit in a de-scoped and reconfigured version of the concept, as per the decision in May 2008 to endorse the reconfiguration of the Ultranet to fit the original $60.5 million announced budget.

The various Ultranet business cases did not answer the ‘Why invest?’ question nor did they provide a sound basis for the project’s approval.

Estimated costs do not reflect full project costs

The 2009 business case estimated the cost of the preferred option at $60.4 million. This is not an accurate estimate of costs for this option.

By the time the May 2009 business case was prepared, DEECD knew that the Ultranet contract would cost $64.6 million. This means that in line with the business case’s figures, a more accurate estimate for the preferred option would have been $69.6 million (i.e. given that $5 million from DEECD’s project development costs had been incurred up to June 2009).

Further, this $69.6 million estimate does not include the costs to be incurred by DEECD as systems integrator. Based on DEECD’s estimates, this would be about $5 million.

This means that the estimated cost for the preferred option in the 2009 business case should have been about $74.6 million and not $60.4 million.

The disconnection between the options presented and the identified needs for the Ultranet, and the failure to accurately estimate the costs for the preferred option, show that the options analysis conducted for the Ultranet was very poorly considered.

This, together with the weakness of the needs analysis to provide sufficient material to explain the context and extent of identified issues, means that the Ultranet business cases failed to provide a sound and reliable basis for government investment in the Ultranet project.

4.3.2 Project delivery

Procurement process

Consistent with public sector practice, the Department of Premier and Cabinet (DPC) and DTF provided advice to government at key decision points over the life of the Ultranet project.

The project continued despite advice from central agencies that it should cease or be delayed. There is no trail of documentary evidence to explain whether or how DEECD addressed the many critical issues raised by DPC and DTF.

It is difficult to understand why the Ultranet procurement was able to proceed to contract execution, given the significant concerns raised by DPC and DTF, as well as the many adverse ratings that DEECD had received from various Gateway reviews since the project first commenced.

Further, this audit detected a number of serious process and probity issues in relation to the tendering and procurement for Ultranet. DEECD has advised that it has commenced a number of actions and further detailed investigations in response to these matters. In May 2008—after the failure of the first Ultranet request for tender to receive a suitable bid from the market—DPC recommended that the Ultranet project be stopped. DTF also expressed concerns about the future of the Ultranet.

Subsequently, the government decided to de-scope and redefine the Ultranet project in alignment with the originally announced $60.5 million budget. After this endorsement, the originally required number of Ultranet functions (e.g. automatic text messages to parents when their children are absent or late) was reduced from 1 260 to 131.

In May 2009—just prior to the final approval of contract execution—government received further advice from both DPC and DTF counselling against proceeding with the Ultranet project.

For example, DTF queried whether DEECD was breaching due process and major project protocols by seeking approval to enter into a contract without any documentation of final positions reached on threshold issues.

DTF also advised that external scrutiny and reporting was warranted so that DEECD could:

  • clarify how it intended to deliver a project with a fixed budget of $60.5 million when the preferred tenderer’s offer was $77.3 million
  • explain how, when it had concerns about the preferred tenderer’s ability to perform the role of system integrator, it had decided to be ‘integration partners’, considering that this arrangement would increase DEECD’s risks and responsibilities in managing and implementing the Ultranet
  • explain why it had not provided the required implementation plan and risk management frameworks
  • make government fully aware of all Ultranet functionalities that will not be delivered as previously publicly committed, noting the significantly reduced project scope.

It is the responsibility of the sponsoring department to verify the data and assumptions contained in business cases, as well as follow up any questions from central agencies. Further, there is no trail of documentary evidence to explain whether or how DEECD addressed the many critical issues raised by DPC and DTF.

This external scrutiny requested by DTF did not take place. No further advice was provided by DEECD on the issues raised before the execution of the contract with the service provider one month later.

It is difficult to understand why the Ultranet procurement was able to proceed to contract execution, despite these significant concerns raised by DPC and DTF, as well as the many adverse ratings that DEECD had received from various Gateway reviews since the project first commenced.

This situation demonstrates a clear lack of accountability and integrity in the checks and balances in place to scrutinise project approvals and expenditure.

Gateway reviews

In addition to objections raised by central agencies, the project received a number of adverse Gateway ratings. Figure 4C shows that the Ultranet received a ‘red’ rating for four of the five Gateway reviews conducted during the planning phase.

Figure 4C
Gateway reviews during the planning phase

Date

Phase

Rating

Outcome

May 2007

Gateway 2 – business case review

RED

(critical and urgent, action on recommendations should be taken immediately)

In all instances, the project proceeded to the next phase with little evidence that issues identified were rectified and recommended actions were implemented.

May 2007

Gateway 3 – procurement strategy review

RED

Apr 2008

Gateway 4 – contract decision review

RED

Oct 2008

Gateway 3 – procurement strategy review (second)

AMBER

(critical and not urgent, actions on recommendations should be carried out before further key decisions are taken)

Feb 2009

Gateway 4 – contract decision review (second)

RED

Source: Victorian Auditor-General’s Office analysis of Department of Education and Early Childhood Development documents.

The DTF-commissioned Gateway Review Team expressed concern that there was a high risk that the Ultranet was not going to be achieved within budget because the $60.5 million project budget had not been informed by a full scoping exercise.

In several instances, the review team also noted that it was unable to perform a comprehensive assessment because the necessary documentation was not complete and had not been provided prior to the review.

DEECD did not adequately address more than 60 per cent of the recommendations in the Gateway reviews for the planning phase. For example, the following recommendations were not actioned when developing the revised 2007 business case:

  • clarify the value of expected efficiencies and/or cost reduction benefits
  • clearly define the educational outcomes which the project aims to deliver, and its effect in schools
  • update the budget, review the training and change management required, and more effectively capture the costs of the change program.

Appropriate action from DEECD on these recommendations may have led to a better planned and implemented Ultranet project.

Process and probity issues in the evaluation of tender bids

DEECD’s evaluation of tender bids for the second request for tender in 2009 raises some serious questions about DEECD’s decision to award the delivery of the Ultranet to the preferred tenderer.

In February 2009, the Ultranet Project Board suspended the contracted service provider as a bidder (due to what DEECD described as untruthfulness in some tender responses) until just before the final stage of the evaluation process.

DEECD could not explain how its tender evaluation process could have selected a sole short-listed tenderer that had been previously suspended from the process.

Tender evaluation documents repeatedly note DEECD’s doubts about the ability of the service provider to deliver the Ultranet. Serious concerns were also raised about its lack of industry experience as a systems integrator and capability to deliver a project of the size and scale of the Ultranet. The proposal also did not follow the preferred prime contractor arrangement requested in the request for tender.

Coaches and support for Ultranet implementation

While DEECD has provided a significant level of support to teaching staff to adopt and integrate learning technologies in teaching practice, DEECD has not provided that support for long enough.

The Ultranet coaching program was poorly timed in that it commenced too early, i.e. two years before the Ultranet started being deployed in schools. Significant technical issues and a delay in delivering the Ultranet until Term 3 of 2010 meant that teachers were unable to make maximum use of the help provided by the coaches when it was most required. While teachers found the training useful, its timing was ineffective because they were unable to apply the learning within a live system.

DEECD did not understand that teachers would require regular and sustained coaching and training over a period to properly adopt and integrate learning technologies. This could have been achieved in a number of different ways, such as by providing sufficient resources to support the lead teacher program on an ongoing basis.

DEECD implemented a lead teacher program so that at least two trained teachers were available at each government school to provide on-site support. This program had mixed results, and depended on the school leadership team’s commitment to using the Ultranet. It also depended on teachers’ willingness to use the inherent capabilities of the VicSmart network and the broader internet and Edumail email systems to provide other learning tools.

On 9 August 2010, DEECD organised a pupil-free day, or ‘Big Day Out’, for teachers to trial the Ultranet. Teachers across the state were asked to sit at their desks to participate in this live trial.

As already discussed in the Background chapter of this report, the Ultranet system did not operate as planned, and teachers were unable to trial the system as expected. However, teachers and principals at some schools implemented a ‘Plan B’ activity with Ultranet coaches to enable some learning to take place.

Teachers from schools visited as part of this audit, as well as reviews commissioned by DEECD, have confirmed that the Big Day Out experience significantly tarnished the reputation of the Ultranet among teachers. Since then, many teachers have refused to try using the system again.

A DEECD-commissioned study found that even with support and training, teachers reported being overwhelmed by the idea of using the Ultranet in the classroom. They also felt unable to put the training provided by Ultranet coaches into practice.

A common theme from school visits was that teachers needed time to assimilate all the information they were receiving and decide how to make best use of the Ultranet in the classroom environment. Teachers also said that limited available class time for teaching lessons did not allow enough time for system crashes, multiple logins and generally slow Ultranet performance.

Technical support has not been sufficient. There has been no clear process to raise technical problems and there have been significant delays in receiving responses to issues raised.

DEECD did not consider that the change management process from introducing the Ultranet would entail greater initial workloads for teachers as they learned to use and integrate the Ultranet with their teaching practice.

Even in the longer term, as teachers begin to incorporate more aspects of the Ultranet into their teaching, they will need more time to access relevant training programs to continually update their ICT skills. Currently there is no provision to support teachers to do this professional development work.

DEECD expects schools to appropriately enable parents to access the Ultranet, including providing login assistance and training to use the Ultranet. This responsibility also adds to teachers’ workload.

4.3.3 Benefits realisation

Transition from project to business as usual

The Ultranet transitioned from a project to business as usual (BAU) in April 2012.

Transition to BAU means that responsibility for the Ultranet technical solution now rests with DEECD’s IT Division, while the business owner and business team is the Student Learning Outcomes Division. BAU also means that the contractor should now be solely responsible for hosting and maintenance activities.

However, not all of the contracted functionalities for the Ultranet were delivered prior to transition to BAU. For example, records management, timetabling and student reporting are yet to be delivered to schools. DEECD continues to work with the contractor to deliver these functionalities.

The Ultranet contract ends on 30 June 2013. Unless the contract is extended, DEECD will need to conduct an open and competitive tender for the continued hosting and maintenance of the Ultranet.

Costs of Ultranet implementation

Based on DEECD-provided data, capital and operating expenditure for the Ultranet had reached $97 million by June 2012. By June 2013, it is estimated that it will have cost nearly $109 million, or some $50 million (83 per cent) more than the $60.5 million budget first announced in November 2006.

However, based on a further review of DEECD’s financial data, we estimate that actual capital and operating expenditure was approximately $162 million by June 2012, and by June 2013 it is likely to have cost approximately $180 million.

Figure 4D
Variance in budgeted versus estimated costs for Ultranet

Figure 4D Variance in budgeted versus estimated costs for Ultranet

Note: The project budget up to 2011–12 includes $60.5 million in total estimated investment, including $21.8 million for Ultranet coaches, and $9.7 million for ongoing funding for software and licensing fees in 2011–12. In addition, the project budget up to 2012–13 includes $9.7 million for ongoing funding for software and licensing fees in 2012–13.

Source: Victorian Auditor-General’s Office.

The variance between the DEECD and VAGO estimates is explained by DEECD’s failure to record and track all costs relating to the Ultranet. For example, DEECD has not classified all relevant Ultranet costs (such as project team staff costs) to the Ultranet cost centre code.

Publicly available information on the Ultranet, such as data provided by DEECD on its website, continues to refer to the Ultranet as a $60.5 million government project. This is misleading and does not accurately represent the full cost of the Ultranet.

Despite this significant overall expenditure, no cost-benefit analysis has been conducted by DEECD to determine whether the Ultranet provides value for money, or whether the same functionality could have been delivered more cost effectively.

Functionality: promised versus actually delivered

Although the Ultranet has been transitioned from a project to BAU, much of the functionality promised in November 2006 has either not been delivered or has been de-scoped from the original specifications.

Figure 4E sets out commitments from ministerial statements and business cases versus the actual Ultranet capability that has been deployed.

Figure 4E
Assessment of VicSmart’s expected benefits

2007 government commitment

Achieved

Victorian parents will get virtual access to their child’s classroom 24 hours a day, seven days a week under Labor’s $60.5 million plan to revolutionise learning.

No

The Ultranet will allow parents to log in to their child’s classroom, check lesson plans, homework and results, attendance, and even communicate directly with teachers via email.

No

Every classroom in a school and every school within the state will be linked, starting in 2007, with the rollout completed within three years.

Yes/No(a)

Students would have their own virtual work spaces with homepages and personally tailored lesson plans accessible, via the internet, from anywhere.

Yes(b)

The Ultranet will consolidate school administrative functions into the one system and lighten the burden on teachers

No

Parents will be able to monitor school attendance, which will be recorded twice a day, with automatic notification via text, email or phone call of non‑attendance.

No

(a) Although all schools are now linked to the Ultranet, this started happening in 2010, not 2007.

(b) While this functionality is now available in the Ultranet, very few students have in fact set up their personal work spaces.

Source: Victorian Auditor-General’s Office.

Ultranet use

At present, use of the Ultranet is well below expectations. DEECD documentation shows that, on average, there were fewer than 11 000 teacher and 52 000 student unique monthly logins for the period from February 2011 to September 2012.

This represents only 10 per cent of students and 27 per cent of teachers, or 4.2 per cent of the 1.5 million total expected Ultranet users projected in both the 2007 and 2009 business cases. There is no equivalent data on parent usage available from DEECD.

Departmental documentation suggests that real teacher logins may actually be much lower. This is because some schools have established the practice of having staff regularly log in, so that their presence is centrally recorded, even if they do not undertake any activity in the Ultranet. This means that, at best, DEECD’s data is on the high side.

DEECD data also shows that unique monthly student logins are below 5 per cent for at least 50 per cent of the 1 587 schools across the Victorian public school system for the period November 2010 to August 2012. The percentage of schools whose students did not access the Ultranet at all has also increased from 15 per cent in 2010 to 21 per cent in 2012.

While significant online materials and training support has been provided to teachers, these have been neither sufficient nor sustained. DEECD has not fully considered the demand the Ultranet would have on teachers’ time and workload, particularly in terms of teachers’ roles in supporting and guiding parents on how to use the Ultranet.

System problems have also discouraged uptake across the government school system. These include the Ultranet’s technical unreliability and poor performance, the lack of appropriate support to develop principals’ and teachers’ ICT competency and confidence, and ageing ICT infrastructure and devices (which can affect online access to the Ultranet).

Use of third-party non-Ultranet software in schools

Prior to the development of the Ultranet concept, many government schools had purchased and were using third-party ICT software, applications and platforms, such as Moodle and Compass, for classroom learning.

This practice has continued after the rollout of the Ultranet, with DEECD data confirming that the number of schools purchasing and using these third-party applications has remained at about the same level as before the introduction of the Ultranet.

While both the 2007 and 2009 business cases were underpinned by the assumption that all schools would use the Ultranet, DEECD has advised that government schools are not mandated to use the Ultranet. DEECD asserts that this position reflects the government’s policy of school autonomy.

DEECD has also decided to allow schools to have greater control over their Ultranet deployment schedules in response to strong feedback from schools.

In addition, the teachers’ union informed DEECD that it had resolved in March 2010 that schools should be allowed to determine when and to what extent they should use the Ultranet, because the significant issues encountered by teachers when using the Ultranet. The union has also recommended to its members various work bans in relation to the Ultranet as part of its current pay and conditions campaign.

Stakeholder consultations and analysis of documentation confirm that the Ultranet was intended to be used by all government schools. The change in DEECD’s position, allowing schools to opt in or opt out as they please, means that it is highly unlikely that the Ultranet’s objective of educational collaboration and providing transferring students a seamless transition from school to school will be achieved.

The achievement of these objectives relies on all schools across the government system using the Ultranet, either by mandating its use or putting significant effort into assuring that the system’s functionality, implementation and issues management are given appropriate consideration to maximise its use.

The November 2011 speech by the Minister for Education Victoria as a Learning Community refers to the need to complete the functionality and assuring the quality of the Ultranet in order to encourage its widespread use.

DEECD needs to give greater consideration and attention to the current ambiguity in regards to the Ultranet opt-out issue. Without near-universal take-up by teachers and students—as forecast in all the business cases—most of the benefits will not be achievable, and the significant investment of taxpayers’ funds to date will therefore be wasted.

Recommendations

The Department of Education and Early Childhood Development should:

  1. urgently review its investment in the Ultranet, with a particular focus on:
    • assessing whether the contractor has delivered all functionality as required by the contract and what action, if any, needs to be taken to enforce the state’s rights
    • rigorously assessing its financial management practices and identifying the real, current cost of the Ultranet to determine the extent to which further investment is warranted
    • identifying and addressing the underlying causes of low take-up rates across the school system by teachers, students and parents
    • providing advice to government on the cost-benefit of decommissioning the system now against continuing to fund and rectify the system so that it can be implemented as originally expected
  2. conduct an agency-wide review of its internal tendering, probity and financial management practices in light of the serious issues identified by this audit
  3. expedite the provision of guidance to schools on the current status of the Ultranet as the department’s key learning technology investment, and clarify the policy context of schools’ autonomy in purchasing non-Ultranet learning technologies.

Appendix A. Audit Act 1994 section 16—submissions and comments

In accordance with section 16(3) of the Audit Act 1994 a copy of this report, or relevant extracts, were provided to the Department of Education and Early Childhood Development, the Department of Premier and Cabinet and the Department of Treasury and Finance, with a request for submissions or comments.

The submissions and comments provided are not subject to audit nor the evidentiary standards required to reach an audit conclusion. Responsibility for the accuracy, fairness and balance of those comments rests solely with the agency head.

RESPONSE provided by the Secretary, Department of Education and Early Childhood Development
RESPONSE provided by the Secretary, Department of Education and Early Childhood Development – continued

Back to top

1 Background

1.1 Introduction

The role of information and communications technologies (ICT) in teaching and learning continues to grow. Many innovative technologies are emerging, and teaching practice and content is now increasingly expected to be delivered ‘anytime, anywhere’.

Over the past nine years, the Department of Education and Early Childhood Development (DEECD) has spent around $1.6 billion on the provision of ICT services in government schools. This amount excludes Commonwealth funding and schools’ own direct ICT expenditure. DEECD estimates that it will spend a further $2.6 billion over the next 10 years on ICT investment for government schools.

Identifying the impact of ICT on student learning is a challenging exercise. In 2010, the Organisation for Economic Cooperation and Development reported that there is no ‘clear cut evidence on the expected benefits’ from learning technologies.

In light of this, it is important that government investment in learning technologies is underpinned by rigorous business cases and planning, careful project implementation, and effective evaluation of achievement against expected outcomes.

1.2 Learning technologies

The term ‘learning technologies’ refers to a broad range of ICT tools that can be used to support learning, teaching and assessment. These tools include interactive boards and learning materials as well as networked technology such as online learning environments. Learning technologies also include devices such as handheld, desktop or laptop computers.

DEECD defines the role of learning technologies as tools or resources ‘to support teaching and learning by enabling users to communicate, create, collaborate, disseminate, store and manage information’.

1.3 Digital Learning Statement

The government’s Digital Learning Statement (the Statement) was released in October 2010 and is the current policy relating to ICT use in school education.

The Statement’s vision is for all teachers and students to have access to contemporary technology and world-class digital content with which to create, communicate and collaborate locally and globally.

The Statement notes that learning technologies in the school environment can:

  • improve students’ confidence and attitudes towards their own learning, behaviour and attendance
  • promote improved opportunities for students to learn through collaboration and conversation
  • improve connections with the real world and provide access to global communities with expertise and perspectives that can enrich learning.

The Statement identified three objectives:

  • provision of ‘anywhere, anytime’ access
  • advancing teacher practice in the use of digital learning in teaching
  • access to high quality resources, tools and data.

1.4 VicSmart broadband network

The 2005–06 State Budget allocated $89.3 million to VicSmart (first referred to as SmartOne when it was announced in April 2005) to provide at least 2 megabits per second (Mbps) broadband data communication services to all government schools.

Full implementation of VicSmart required the laying of fibre-optic cables to the ‘school gate’. The network was designed to allow schools to effectively access online information and resources. Prior to VicSmart, most government school bandwidth was limited to 64 kilobits per second (Kbps), which did not allow for quick access to online materials.

VicSmart was delivered through the whole-of-government 2002 Telecommunications Procurement and Management Strategy. This strategy consolidated the purchase and management of fixed voice, mobile, telephony and data carriage services across government.

The contracts for the provision of voice, mobile and data carriage services are jointly referred to as telecommunications carriage services. In July 2004, the telecommunications carriage services contract was awarded for the provision of broadband data services, including to all government schools.

1.4.1 Rollout of VicSmart

The VicSmart rollout started in October 2005. Haddon Primary School near Ballarat and North Fitzroy Primary School were the first beneficiaries. Internet connectivity increased roughly 60 times—from 64 Kbps to 4 Mbps.

Since its first rollout in 2005, VicSmart had been upgraded four times:

  • In 2009, all government schools were upgraded from 4 Mbps to at least 10 Mbps.
  • In 2010, larger school campuses received faster bandwidth, at 20 Mbps and 50 Mbps. The basis for this upgrade was student population. Schools with enrolments of 500 to 999 students were upgraded to 20 Mbps, and those with enrolments greater than 1 000 received 50 Mbps bandwidth.
  • Later in the same year, all metropolitan government schools with 10 Mbps broadband bandwidth were upgraded to 20 Mbps. The bandwidth allocation for non-metropolitan schools remained at 10 Mbps due to additional costs estimated at $4.5 million per annum.

In 2012, a further 244 schools were upgraded, including 11 that received up to 100 Mbps bandwidth. The main driver for this upgrade was the need to support access to network applications such as videoconferencing and voice over internet protocol, which require Quality of Service network technology. Until this time, VicSmart provided base bandwidth only, and did not support Quality of Service traffic.

1.5 The Ultranet e-learning system

The Ultranet is a Victoria-wide learning platform and ICT system that students, parents and teachers of government schools can access via the internet.

The DEECD website says that students and teachers can use the Ultranet for online learning activities and for collaborating and communicating with students from within their school and across Victorian government schools.

The system is also designed to allow parents to access information that will help them keep up to date with their child’s learning.

The Ultranet was described in the Statement as the key foundation plank or key enabler of the three components of the vision for ICT in the classroom.

1.5.1 Election commitment

In November 2006, Victorian parents were told by the government that they would get virtual access to their child’s classroom 24 hours a day, seven days a week under a $60.5 million plan to revolutionise learning.

Figure 1A shows the components of the Ultranet announced in November 2006.

Figure 1A

Election commitment for Ultranet components

Commitment

Connect the entire school community—parents, students, teachers and administrators—in a way never before possible.

Consolidate school administrative functions into the one system and lighten teachers’ burden.

Allow parents to:

  • log into their child’s classroom
  • check teachers’ lesson plans and their child’s homework and results
  • communicate directly with teachers by email
  • monitor school attendance, which will be recorded twice a day
  • receive automatic notification of non-attendance via text, email or phone call.

Give students:

  • their own virtual work spaces with homepages and personally tailored lesson plans, which will be accessible via the internet
  • the ability to keep up to date with their education while away from school due to illness or family reasons and get teacher feedback much faster.

Source: Victorian Auditor-General’s Office, based on Department of Education and Early Childhood Development media release, issued 13 November 2006.

The Ultranet was presented as a revolutionary educational tool that would ‘connect the entire school community—parents, students, teachers and administrators—starting in 2007’.

1.5.2 The 2007 Ultranet business case

The 2007 business case vision for the Ultranet was for an intuitive, student-centred electronic learning environment that supports high quality learning and teaching. Its objectives were to:

  • improve responsiveness to individual learning needs
  • provide better information to parents, the school system and government
  • improve the efficiency of the learning environment and school administration
  • adopt an enterprise approach to intranet development
  • exploit previous ICT investments.

The business case noted that there were many commercial ICT solutions available in the market, and that many schools were already using these. However, it emphasised that no single solution could provide the full level of scalability, security, interoperability and functionality that is needed to meet the identified business needs and objectives of the Ultranet.

According to the Ultranet business case, some 1.5 million Victorian students, teachers and parents were expected to use the Ultranet when fully operational. Figure 1B summarises the Ultranet’s identified eight key performance indicators.

Figure 1B

Ultranet key performance indicators

Benefit

Key performance indicators

Establish a school environment for the future

Number of students using Ultranet (target 70 per cent)

Improve the educational outcomes for Victorians

Number of students accessing online learning resources in the Ultranet (target 65 per cent)

 

Number of teachers accessing resources, preparing curriculum plans, delivering Ultranet-based resources (target 65 per cent)

Reduce the administrative burden on teachers and school leaders

Efficiency in administrative processes (target 20 per cent over baseline data)

Reduce the future cost of education

Number of external software packages used to assist school administration (target 30 per cent decrease over baseline data)

 

Cost of external software packages used by government schools to assist in school administration (target 15 per cent over baseline)

Improve educational opportunities for regional, rural and remote Victorians

Number of students participating in online collaborative technology via the Ultranet (target 30 per cent of regional and rural schools’ students)

 

Number of teachers accessing online content and communities via the Ultranet (target 30 per cent of regional and rural schools’ teachers)

Source: Victorian Auditor-General’s Office based on Department of Education and Early Childhood Development documentation and the 2007 Ultranet business case.

1.5.3 Failed tender in 2007 and subsequent re-scoping

A request-for-tender to develop and manage the Ultranet was released to the market on 1 August 2007. All bids received were in excess of $100 million and resulted in DEECD being unable to nominate a successful tenderer.

Following this failed tender, the Ultranet project was re-scoped to fit the approved budget of $60.5 million. Consequently, the Ultranet’s business requirements were reduced by 90 per cent from the 1 260 functions stated in the 2007 business case to 131 functions contained in the 2008 request-for-tender specifications.

The revised Ultranet project focused on the delivery of learning and teaching components, a parent portal and ‘sufficient’ functionality to support these areas. When re-scoping the Ultranet, consideration was also given to purchasing an off-the-shelf product rather than developing a bespoke solution.

In addition to re-scoping and reducing the Ultranet’s functionality, the concurrent user requirement was reduced from 250 000 users to 78 000 users and total storage was reduced from 330 terabytes (TB) to 160 TB, reductions of 69 per cent and 51 per cent respectively.

1.5.4 Procurement and deployment

DEECD released a selective request for tender in November 2008. Four bids were received when the tender closed on 23 December 2008.

The Ultranet Master Agreement was signed with the successful tenderer on 30 June 2009 for the contract price of $64.6 million. The price included $47.5 million for the build and delivery, and $17.1 million for application hosting, software maintenance, disaster recovery and help desk support until 30 June 2013.

The Ultranet rollout to government schools commenced in May 2010. On 9 August 2010, government school principals and assistant principals attended the Ultranet ‘Big Day Out’ at the Melbourne Convention and Exhibition Centre. Government school teachers across the state were asked to sit at their desks to participate in a trial of the Ultranet.

However, the Ultranet suffered user access issues and teachers waited all morning for the system to become available. DEECD explained to the participants that technical issues were affecting the Ultranet and that work was being done to correct the problems. When the Ultranet went live in the afternoon, it suffered further login issues and performed slowly and unreliably.

Even in its reduced form, significant Ultranet functionalities have not been fully delivered, and they continue to be deployed to government schools at the time of finalising this report.

1.5.5 Ultranet oversight

Executive sponsorship and management of the Ultranet rests with the Ultranet Project Board, comprised of executives from various DEECD divisions and some private sector representatives. The Ultranet Project Board meets monthly and is chaired by the DEECD Deputy Secretary, School Education Group.

The Ultranet Project Board is responsible for the Ultranet’s business management and project delivery functions. The project team reports to the board and its responsibilities include giving advice on Ultranet strategy and policy, change management within schools, and DEECD’s relationship with the contractor. Project delivery functions include contract and service level management, data analysis, major incident management, and solution architecture and integration.

Software development, release management and application support, are the contractor’s responsibilities.

Operational functions relating to the live ICT system are managed either by the contractor or by DEECD’s Information and Technology Division. A contractor-provided help desk is currently available to users until 30 June 2013.

1.6 Audit objective and scope

The objective of this audit was to examine the effectiveness of the application of learning technologies in government schools.

In particular, the audit examined DEECD’s planning, delivery and management of learning technologies across the government school system.

To address the objective, the audit examined whether DEECD:

  • conducted robust research on the most appropriate use of ICT in government school teaching environments
  • developed effective strategies to implement learning technologies in government schools and supported teaching staff to adopt these technologies
  • is monitoring achievement of desired outcomes, including realisation of purported benefits described in the business cases used to support funding decisions for the Ultranet and VicSmart.

1.7 Audit method and cost

The audit was performed in accordance with section 15 of the Audit Act 1994 and the Australian Auditing and Assurance Standards.

The total cost was $310 000.

1.8 Report structure

The report is structured as follows:

  • Part 2 – Digital Learning Statement
  • Part 3 – Case study: VicSmart broadband connectivity for schools
  • Part 4 – Case study: the Ultranet e-learning system.

Back to top

2 Digital Learning Statement

At a glance

Background

The government’s Digital Learning Statement (the Statement) was released in October 2010 and remains the current policy for information and communications technology (ICT) in school education. Its vision is for all teachers and students to have access to contemporary technology and world-class digital content with which to create, communicate and collaborate locally and globally.

Conclusion

The Statement does not provide a strategy, action plan or framework for investing in learning technologies.

This means that departmental staff and school leaders have little guidance on how learning technology initiatives can be appropriately planned and integrated to build upon past and present ICT investments.

Findings

  • The Statement was not informed by robust and comprehensive research and does not make a clear and cogent case for government investment in learning technologies.
  • The Statement did not deliver on the directive of the 2008 Blueprint for Education and Early Childhood Development for the Department of Education and Early Childhood Development (DEECD) to develop a strategy or plan of action to use learning technologies in teaching and learning.

Recommendation

DEECD should develop a comprehensive and evidence-based strategy or plan of action for use of learning technologies to underpin and guide the significant investment in ICT for government schools.

2.1 Introduction

The government’s Digital Learning Statement (the Statement) was released in October 2010 and is the current policy information and communication technologies (ICT) use in school education. It was developed by the Department of Education and Early Childhood Development (DEECD) and was released in response to the 2008 Blueprint for Education and Early Childhood Development (the Blueprint).

The Blueprint required DEECD to develop and release an overarching ICT strategy for teaching and learning.

Action 7 of the Blueprint referred to the government’s commitment to improve Victorian students’ performance in literacy and numeracy across all sectors. The first step in furthering this commitment was to clearly set out the government’s approach to unlocking the full potential of ICT usage in schools.

2.2 Conclusion

The Statement has no clear action plan or framework for investment in learning technologies in the government school system.

Consequently, departmental staff and school leaders have little guidance on how learning technology initiatives can be appropriately planned and integrated to build upon past and present ICT investments.

2.3 Vision and objectives for learning technologies

The Statement describes the government’s vision and objectives for learning technologies in Victorian government schools. The vision is that:

  • all teachers and students have access to contemporary technology and world‑class digital content with which to create, communicate and collaborate locally and globally
  • student learning is engaging, personalised and authentic to enable them to become confident and creative individuals and active and informed citizens of the 21st century.

The three objectives outlined by the Statement are:

  • Anywhere, anytime access—all students and teachers have appropriate and equitable opportunities to use digital devices and systems, including access to a range of contemporary digital devices and sophisticated online systems that support learning, teaching and knowledge sharing.
  • Advancing teaching practice—all teachers and school leaders build capabilities to use digital technology to improve student learning.
  • Access to high quality resources, tools and data—all teachers and students have access to quality assured digital resources, tools and rich data sources.

2.4 Lack of strategy or action plan

The Statement sets out the vision and objectives for the use of learning technologies in government schools. It also highlights previous and ongoing government investments in learning technologies.

However, it does not provide a strategy, clear action plan or framework to achieve the stated vision and objectives and to align future investments.

DEECD intended to develop a strategy or framework for future ICT initiatives when the department was developing the Statement. A proposal was made to identify the appropriate initiatives and investments required to address gaps and barriers identified in the then current delivery of learning technologies in government schools.

However, this work was not undertaken, and the Statement did not include any reference to identified barriers, or a strategy for future action.

As a consequence, the Statement provides little guidance to DEECD staff and school leaders on how future initiatives can and should be appropriately planned to build upon past and present investments.

This also means that the Statement did not deliver on the Blueprint’s expectation to develop a strategy to use learning technologies in teaching and learning.

DEECD asserts that although the Blueprint committed to develop and release a strategy, it was the government’s decision to release the Statement, which did not include a strategy.

However, a review of DEECD’s advice to the Minister for Education shows that the government was not advised that the Statement did not comply with the Blueprint’s directive to develop and deliver a strategy. No further action was taken to develop a strategy.

2.5 Research for the Statement

The Statement used some national and international research to inform its development. It cites a 2007 UK research report suggesting that learning technologies can improve learning outcomes.

However, the cited literature states that the evidence on the impact of ICT on learning is ‘as yet inconsistent’, and that there is not a sufficient body of evidence from which ‘to draw firm conclusions’.

This view is shared by the Organisation for Economic Cooperation and Development (OECD), which states in its March 2010 research report, Technology Use and Educational Performance in PISA 2006, that policy makers in OECD countries have had to adjust their expectations on the impact of ICT use in teaching and learning because they could not see clear-cut evidence of the expected benefits.

An acknowledgment in the Statement of this more cautious view would have provided a clearer picture of the challenging context for the use of learning technologies in schools and given greater scrutiny of the purported benefits of any investment.

The lack of clear evidence on the effectiveness of learning technologies in enhancing teaching and learning could also have prompted the development of a clear action plan for learning technologies in Victorian government schools that focused more realistically on how best to minimise cost and risk while maximising benefits.

The Statement’s absence of robust and comprehensive research means that it does not make a cogent case for government investment in learning technologies.

2.5.1 Evaluation of programs

In the absence of clear-cut evidence, some reference to outcomes or lessons learned from previous government investments in learning technologies would have provided a stronger basis or rationale for the Statement.

DEECD has not been evaluating the effectiveness of most of its learning technology initiatives. Therefore, there is no objective evidence that government’s significant investment in learning technologies has delivered the intended benefits for Victorian students.

2.5.2 Stakeholder consultation

The Statement was not informed by consultation with key stakeholders beyond departmental staff.

Input from teachers, parents and student representative groups was not sought to clarify expectations or perspectives on what does and doesn’t work with learning technologies.

Recommendation

  1. The Department of Education and Early Childhood Development should develop a comprehensive and evidence-based strategy or plan of action for use of learning technologies to underpin and guide the significant investment in ICT for government schools.

Back to top

3 Case study: VicSmart broadband connectivity for schools

At a glance

Background

The VicSmart project was rolled out between October 2005 and December 2011. Internet connectivity speeds in schools increased roughly 60 times from the previous 64 Kbps (kilobits per second) to 4 Mbps (megabits per second). All government schools are now connected to VicSmart.

The VicSmart procurement process was relatively straightforward because the Department of Education and Early Childhood Development (DEECD) was required to use the whole-of-government single-source provider.

Conclusion

VicSmart was well planned and underpinned by a business case that provided a robust needs and options analysis.

Although no performance indicators have been developed to measure VicSmart’s intended benefits, it is evident that VicSmart is a key enabler of digital learning in government schools. It achieves this by providing high quality broadband data access to online DEECD and internet-based digital learning resources.

Findings

  • VicSmart has provided high-capacity and high-speed fibre-optic connectivity to all Victorian government schools.
  • Use of VicSmart is increasing in line with the increase in end-user computing devices in government schools.
  • Network capacity has been appropriately upgraded to meet increasing bandwidth needs over time, with much of this increase matching the increase in internet access and usage by government schools.

Recommendation

DEECD should develop performance indicators that measure both VicSmart’s ongoing operational performance and its achievement of intended benefits.

3.1 Introduction

The rollout of the VicSmart project began in October 2005 and was completed in December 2011.

After installation of the fibre-optic cable to the ‘school gate’, internet connectivity speeds increased roughly 60 times from the previous 64 Kbps (kilobits per second) to 4 Mbps (megabits per second).

All Victorian government schools are connected to the VicSmart fibre-optic network.

3.2 Conclusion

The VicSmart project was well planned and underpinned by a business case providing a robust needs and options analysis.

The Department of Education and Early Childhood Development (DEECD) delivered VicSmart on time and on budget. The project has delivered a high-speed broadband fibre-optic network providing internet access, email and a range of digital applications. Schools report very few problems with VicSmart and its quality, functionality and performance parameters are meeting expectations.

Although no specific performance indicators have been developed to measure the realisation of VicSmart’s intended benefits, it is evident that VicSmart is a key enabler of digital learning in government schools by providing high quality broadband data access to online DEECD and internet-based digital learning resources.

3.3 Findings

3.3.1 Planning for the project

Prior to VicSmart, all Victorian government schools received a narrowband internet connection through the VicOne network. Ninety per cent of government schools were provided with a 64 Kbps connection speed, while the remaining schools received slightly higher bandwidth provisions.

When first implemented in 1998, VicOne delivered productivity improvements as emails replaced surface mail and many administrative reports moved online.

However, increased demand for internet connectivity placed significant pressure on the VicOne network. By 2004, online access became impractical due to the considerable length of time required to download emails and web-based reference materials. DEECD data shows that in October 2004, Victoria was providing the lowest internet bandwidth connectivity among all states and territories.

Planning to replace VicOne was underpinned by a robust needs and options analysis which was contained in the VicSmart project’s 2005 business case. This effectively articulated the needs to be addressed and provided a clear rationale for the chosen option.

It also clearly established a case for government expenditure on the purchase of high‑speed fibre-optic broadband connectivity as well as providing confidence that the project was achievable and could be delivered as planned.

The business case also noted that broadband internet connections available to most Victorian households in 2005 had raised expectations about the performance of online services. This placed additional pressure on the government to provide comparable access to online services at school.

Three options were considered for the VicSmart project. A clear analysis of the strengths and weaknesses of each option was provided in the business case.

The preferred option was for all schools to receive an entry-level broadband fibre-optic service upgradeable to 100 Mbps.

The business case explained that this was the best option available because:

  • it provided a scalable platform for the growing data services needs of schools
  • it avoided future costs of up to $60 million by taking advantage of previously negotiated pricing for installation of optical fibre in non-metropolitan schools
  • it delivered high quality data services to both metropolitan and non-metropolitan schools.

3.3.2 Project delivery

In February 2005 the government formally approved VicSmart as an $89 million project to be run over four years. This approved budget was in addition to DEECD’s existing $18.6 million recurrent annual bandwidth budget.

VicSmart-related costs have therefore been $163 million for the period 2005–06 to 2008–09, plus $48 million in ongoing recurrent costs from 2009–10.

Since it was compulsory for DEECD to use a sole-source contractor, the procurement process for VicSmart was relatively straightforward. This saved DEECD time and resources by not having to develop request-for-tender documentation, solicit bids and evaluate tender proposals. The procurement process was limited to organising purchase orders with the sole-source provider in accordance with the VicSmart project specification.

Since completion of rollout, the VicSmart project has provided high-capacity and high‑speed fibre-optic connectivity to all Victorian government schools. Use of VicSmart is increasing in line with the increase in end-user computing devices in government schools.

Network capacity has also scaled-up appropriately to meet increasing bandwidth needs. Much of this increase matches the increase in internet access and usage by government schools. From the initial 2 Mbps broadband service, upgrades have been achieved progressively, as indicated in Figure 3A.

Figure 3A

VicSmart upgrades over time

Upgrade

Description

Phase 1: 2005

All schools transitioned from VicOne to VicSmart (minimum 2 Mbps in each school)

Phase 2: 2008

Scalable 4 Mbps to all schools connected by fibre-optic cable

Phase 3: 2010

Upgrade to 10 Mbps for all schools connected by fibre-optic cable

Phase 4: 2010

Enhanced provisioning of 20 Mbps to 50 Mbps for larger school campuses

Phase 5: 2012

Enhanced provisioning to up to 100 Mbps for larger school campuses

Source: Victorian Auditor-General’s Office review of Department of Education and Early Childhood Development documents.

Timely communication with schools on VicSmart’s implementation and upgrade has meant little disruption to school activities.

During the rollout, regional schools were not disadvantaged and generally received the service at about the same time as metropolitan schools.

DEECD also advised all schools prior to the rollout of VicSmart that the significantly higher download speeds would affect their internet service provider (ISP) costs. Schools were asked to develop strategies to monitor and manage ISP costs, as DEECD currently pays all ISP-related costs for government schools.

Continuing to benchmark VicSmart against national and international standards, DEECD has upgraded speeds to up to 100 Mbps (depending on school enrolments), making Victoria compliant with the Ministerial Council for Education, Early Childhood Development and Youth Affairs bandwidth provisioning target of 8.7 Kbps per student.

3.3.3 Benefits realisation

VicSmart is delivering the intended benefits of providing reliable high-speed internet connectivity to all Victorian government schools.

Although no performance indicators have been developed to measure VicSmart’s intended benefits, it is evident that VicSmart is a key enabler of digital learning and provides high quality broadband data access to both online DEECD and internet-based digital learning resources.

Figure 3B

Assessment of VicSmart’s expected benefits

VicSmart benefits as defined in the 2005 business case

Achieved

Learning and teaching benefits

 
  • Provide access to online multimedia learning materials from any school computer.

Yes

  • Capacity to deliver lesson via videoconference between all schools, leading to increased staffing flexibility.

Yes

  • Enable online testing and examinations from any school site

Yes

Administration and management benefits

 
  • Able to centralise data and software, lowering cost and improving data quality and reporting.

Yes

Schools infrastructure benefits

 
  • Future cost reduction. All schools provided with scalable optical fibre‑based broadband. Affordable upgrades available as required.

Yes

  • Improved productivity of information and communications technologies resources by enabling support and configuration from clusters or centrally to optimise service delivery, performance and efficiency.

Yes

  • Improved productivity. Software will be deployed and supported across the network rather than onsite.

Yes

Source: Victorian Auditor-General’s Office.

DEECD receives monthly reports on VicSmart’s operational performance. Information from the service provider’s monthly reports is used to inform key management decisions, including phased upgrades of VicSmart over time.

Principals and teachers at the schools visited during the audit, as well as other stakeholders, have confirmed that VicSmart is enabling digital learning by providing access to online resources and enabling videoconferencing between schools.

A teacher in one of the visited schools said that if VicSmart was to be turned off, ‘digital learning in Victorian government schools would come to a halt’.

Recommendation

  1. The Department of Education and Early Childhood Development should develop performance indicators that measure both VicSmart’s ongoing operational performance and its achievement of intended benefits.

Back to top

4 Case Study: the Ultranet e‑learning system

At a glance

Background

The Ultranet is a Victoria-wide digital learning platform and ICT system. Government school students, parents and teachers can access the Ultranet via the internet.

Conclusion

The Ultranet was poorly planned and implemented. Some six years after its announcement as a government priority, it is yet to deliver the benefits expected for students, parents and schools. It is significantly late and over budget, has very low uptake by users, and has reduced functionality compared with what was originally announced.

This audit also identified a number of serious probity, financial management and procurement issues.

Findings

  • The Ultranet has not delivered its main objectives or expected benefits.
  • There were a number of serious process issues and apparent probity breaches in relation to the Ultranet procurement.
  • No cost-benefit analysis has been conducted to determine whether the Ultranet provides value for money, or whether the same functionalities could have been delivered more cost effectively.
  • Use of the Ultranet is low, and declining.
  • Poor quality financial data means that actual capital and operating expenditure for the Ultranet is unclear.

Recommendations

The Department of Education and Early Childhood Development should:

  • urgently review its investment in the Ultranet
  • conduct an agency-wide review of its internal tendering, probity and financial management practices in light of the serious issues identified by this audit
  • expedite the provision of guidance to schools on the current status of the Ultranet as the department’s key learning technology investment, and clarify the policy context of schools’ autonomy in purchasing non-Ultranet learning technologies.

4.1 Introduction

The Ultranet was announced in November 2006 as a $60.5 million project. It was expected to provide an online teaching, learning and assessment system for all government schools.

Over its life, the Ultranet has been the subject of three business cases and two tender processes.

The project is in a ‘business as usual’ phase of the contract, which expires in June 2013.

4.2 Conclusion

The Ultranet has been poorly planned and implemented. None of its three business cases had a well thought out needs analysis or gave comprehensive options to deliver the project.

Some six years since its announcement as a government priority, the Ultranet has not delivered any of the main objectives that were expected when the project was funded.

The Ultranet is significantly late and over budget—and with limited functionality—when compared with what was originally announced. The full costs of the project are poorly recorded by the Department of Education and Early Childhood Development (DEECD) and a conservative estimate of actual costs by June 2013 is $180 million, or close to 300 per cent above the first announced budget.

This audit has detected a number of serious procurement and probity lapses that have triggered further review and action by the department. Although the department’s work is not yet complete, there is no assurance that the tender process was rigorous, or that the chosen delivery option represented value for money.

Use of the Ultranet is well below expectations, with only 10 per cent of students and 27 per cent of teachers logging into the system. Because of its very low and declining usage, and the ambiguous guidance from DEECD about whether schools can opt out of the system, the viability of the Ultranet as the government’s key learning technology investment is now under serious threat.

DEECD has not given sufficient attention to understanding the negative effect that the continuing low usage, as well as the opt-out issue, is having on delivery of expected benefits from the Ultranet. These shortcomings are also diluting the return on investment for the significant public funds that have already been spent.

4.3 Findings

4.3.1 Planning for the project

Three business cases

Over time, the Ultranet project has been the subject of three business cases, with one rejected and two endorsed by government.

None of these business cases provided a robust needs analysis and also did not provide a sound basis for approval of the project. They did not answer the ‘Why invest?’ question for the Ultranet.

The first business case was developed in early 2007 and was rejected by government. The second business case, submitted in July 2007, was a revised version of the first. This formed the basis of the (failed) tender process initiated in 2007.

The third business case was presented to government in May 2009 after the second procurement process had been initiated and almost immediately before the Ultranet contract was awarded to the service provider in June 2009.

This was a serious departure from the usual process of developing a business case prior to embarking on a procurement process. The Department of Treasury and Finance’s (DTF) Lifecycle Investment Framework states that the development of a business case must occur before the procurement phase.

Needs analysis

The 2007 and 2009 business cases stated a number of business drivers for the development of the Ultranet described in Figure 4A.

Figure 4A

Identified needs for the Ultranet

2007 business case

2009 business case

Critical need for a customised learning environment, i.e. teachers are unable to centrally source and share learning resources.

Same

Poor quality of information available to education stakeholders, i.e. teachers and school leaders have difficulties in tracking student progress from year to year and between schools.

Same

Failure to maximise return on previous information and communications technologies (ICT) investments, i.e. existing disparate school systems inhibit interoperability with other systems.

Same

Massive inefficiencies in administration for schools, i.e. too much time is spent on time‑consuming tasks that take the focus away from learning and teaching.

Not included – administration was dropped as an Ultranet functionality in the 2009 business case.

Source: Victorian Auditor-General’s Office.

The purpose of a needs analysis in a business case is to identify the fundamental need for an investment including the problems, issues or circumstances of the current situation that needs to be addressed.

Both business cases failed to provide sufficient material to explain the context and extent of the identified issues. For example, with regard to ‘massive inefficiencies in administration’, the business case did not explain how much teachers’ time would be released through the automation of these administrative tasks, nor did it estimate the extent to which this could improve teachers’ productivity.

Options analysis

Three options were considered in both the 2007 and 2009 business cases.

Figure 4B

Options analysis for the Ultranet

2007 business case

2009 business case

Do nothing—allow schools to determine and purchase their own learning software solution ($67 million estimated cost)

Do nothing—allow schools to determine and purchase their own learning software solution ($67 million estimated cost)

Further develop and deploy early version of Ultranet trialled in several schools in 2006 ($72 million estimated cost)

Negotiate contract with preferred tenderer with DEECD/contractor as integration partners ($60.4 million estimated cost)(a)

Develop and deploy an Ultranet solution ($83 million estimated cost)(a)

Negotiate contract/s with preferred tenderer and a third party as systems integrator ($68.6 million estimated cost)

(a) Preferred option.

Source: Victorian Auditor-General’s Office analysis of Department of Education and Early Childhood Development documents.

In the 2009 business case, there was little relationship between the options presented and the needs identified for the proposed project. Instead of presenting options on the appropriate type of investment to address the identified needs, the options were types of contractual arrangements that DEECD could have with the preferred tenderer.

The options presented for 2009 should have been similar to those of the 2007 business case, albeit in a de-scoped and reconfigured version of the concept, as per the decision in May 2008 to endorse the reconfiguration of the Ultranet to fit the original $60.5 million announced budget.

The various Ultranet business cases did not answer the ‘Why invest?’ question nor did they provide a sound basis for the project’s approval.

Estimated costs do not reflect full project costs

The 2009 business case estimated the cost of the preferred option at $60.4 million. This is not an accurate estimate of costs for this option.

By the time the May 2009 business case was prepared, DEECD knew that the Ultranet contract would cost $64.6 million. This means that in line with the business case’s figures, a more accurate estimate for the preferred option would have been $69.6 million (i.e. given that $5 million from DEECD’s project development costs had been incurred up to June 2009).

Further, this $69.6 million estimate does not include the costs to be incurred by DEECD as systems integrator. Based on DEECD’s estimates, this would be about $5 million.

This means that the estimated cost for the preferred option in the 2009 business case should have been about $74.6 million and not $60.4 million.

The disconnection between the options presented and the identified needs for the Ultranet, and the failure to accurately estimate the costs for the preferred option, show that the options analysis conducted for the Ultranet was very poorly considered.

This, together with the weakness of the needs analysis to provide sufficient material to explain the context and extent of identified issues, means that the Ultranet business cases failed to provide a sound and reliable basis for government investment in the Ultranet project.

4.3.2 Project delivery

Procurement process

Consistent with public sector practice, the Department of Premier and Cabinet (DPC) and DTF provided advice to government at key decision points over the life of the Ultranet project.

The project continued despite advice from central agencies that it should cease or be delayed. There is no trail of documentary evidence to explain whether or how DEECD addressed the many critical issues raised by DPC and DTF.

It is difficult to understand why the Ultranet procurement was able to proceed to contract execution, given the significant concerns raised by DPC and DTF, as well as the many adverse ratings that DEECD had received from various Gateway reviews since the project first commenced.

Further, this audit detected a number of serious process and probity issues in relation to the tendering and procurement for Ultranet. DEECD has advised that it has commenced a number of actions and further detailed investigations in response to these matters. In May 2008—after the failure of the first Ultranet request for tender to receive a suitable bid from the market—DPC recommended that the Ultranet project be stopped. DTF also expressed concerns about the future of the Ultranet.

Subsequently, the government decided to de-scope and redefine the Ultranet project in alignment with the originally announced $60.5 million budget. After this endorsement, the originally required number of Ultranet functions (e.g. automatic text messages to parents when their children are absent or late) was reduced from 1 260 to 131.

In May 2009—just prior to the final approval of contract execution—government received further advice from both DPC and DTF counselling against proceeding with the Ultranet project.

For example, DTF queried whether DEECD was breaching due process and major project protocols by seeking approval to enter into a contract without any documentation of final positions reached on threshold issues.

DTF also advised that external scrutiny and reporting was warranted so that DEECD could:

  • clarify how it intended to deliver a project with a fixed budget of $60.5 million when the preferred tenderer’s offer was $77.3 million
  • explain how, when it had concerns about the preferred tenderer’s ability to perform the role of system integrator, it had decided to be ‘integration partners’, considering that this arrangement would increase DEECD’s risks and responsibilities in managing and implementing the Ultranet
  • explain why it had not provided the required implementation plan and risk management frameworks
  • make government fully aware of all Ultranet functionalities that will not be delivered as previously publicly committed, noting the significantly reduced project scope.

It is the responsibility of the sponsoring department to verify the data and assumptions contained in business cases, as well as follow up any questions from central agencies. Further, there is no trail of documentary evidence to explain whether or how DEECD addressed the many critical issues raised by DPC and DTF.

This external scrutiny requested by DTF did not take place. No further advice was provided by DEECD on the issues raised before the execution of the contract with the service provider one month later.

It is difficult to understand why the Ultranet procurement was able to proceed to contract execution, despite these significant concerns raised by DPC and DTF, as well as the many adverse ratings that DEECD had received from various Gateway reviews since the project first commenced.

This situation demonstrates a clear lack of accountability and integrity in the checks and balances in place to scrutinise project approvals and expenditure.

Gateway reviews

In addition to objections raised by central agencies, the project received a number of adverse Gateway ratings. Figure 4C shows that the Ultranet received a ‘red’ rating for four of the five Gateway reviews conducted during the planning phase.

Figure 4C

Gateway reviews during the planning phase

Date

Phase

Rating

Outcome

May 2007

Gateway 2 – business case review

RED

(critical and urgent, action on recommendations should be taken immediately)

In all instances, the project proceeded to the next phase with little evidence that issues identified were rectified and recommended actions were implemented.

May 2007

Gateway 3 – procurement strategy review

RED

Apr 2008

Gateway 4 – contract decision review

RED

Oct 2008

Gateway 3 – procurement strategy review (second)

AMBER

(critical and not urgent, actions on recommendations should be carried out before further key decisions are taken)

Feb 2009

Gateway 4 – contract decision review (second)

RED

Source: Victorian Auditor-General’s Office analysis of Department of Education and Early Childhood Development documents.

The DTF-commissioned Gateway Review Team expressed concern that there was a high risk that the Ultranet was not going to be achieved within budget because the $60.5 million project budget had not been informed by a full scoping exercise.

In several instances, the review team also noted that it was unable to perform a comprehensive assessment because the necessary documentation was not complete and had not been provided prior to the review.

DEECD did not adequately address more than 60 per cent of the recommendations in the Gateway reviews for the planning phase. For example, the following recommendations were not actioned when developing the revised 2007 business case:

  • clarify the value of expected efficiencies and/or cost reduction benefits
  • clearly define the educational outcomes which the project aims to deliver, and its effect in schools
  • update the budget, review the training and change management required, and more effectively capture the costs of the change program.

Appropriate action from DEECD on these recommendations may have led to a better planned and implemented Ultranet project.

Process and probity issues in the evaluation of tender bids

DEECD’s evaluation of tender bids for the second request for tender in 2009 raises some serious questions about DEECD’s decision to award the delivery of the Ultranet to the preferred tenderer.

In February 2009, the Ultranet Project Board suspended the contracted service provider as a bidder (due to what DEECD described as untruthfulness in some tender responses) until just before the final stage of the evaluation process.

DEECD could not explain how its tender evaluation process could have selected a sole short-listed tenderer that had been previously suspended from the process.

Tender evaluation documents repeatedly note DEECD’s doubts about the ability of the service provider to deliver the Ultranet. Serious concerns were also raised about its lack of industry experience as a systems integrator and capability to deliver a project of the size and scale of the Ultranet. The proposal also did not follow the preferred prime contractor arrangement requested in the request for tender.

Coaches and support for Ultranet implementation

While DEECD has provided a significant level of support to teaching staff to adopt and integrate learning technologies in teaching practice, DEECD has not provided that support for long enough.

The Ultranet coaching program was poorly timed in that it commenced too early, i.e. two years before the Ultranet started being deployed in schools. Significant technical issues and a delay in delivering the Ultranet until Term 3 of 2010 meant that teachers were unable to make maximum use of the help provided by the coaches when it was most required. While teachers found the training useful, its timing was ineffective because they were unable to apply the learning within a live system.

DEECD did not understand that teachers would require regular and sustained coaching and training over a period to properly adopt and integrate learning technologies. This could have been achieved in a number of different ways, such as by providing sufficient resources to support the lead teacher program on an ongoing basis.

DEECD implemented a lead teacher program so that at least two trained teachers were available at each government school to provide on-site support. This program had mixed results, and depended on the school leadership team’s commitment to using the Ultranet. It also depended on teachers’ willingness to use the inherent capabilities of the VicSmart network and the broader internet and Edumail email systems to provide other learning tools.

On 9 August 2010, DEECD organised a pupil-free day, or ‘Big Day Out’, for teachers to trial the Ultranet. Teachers across the state were asked to sit at their desks to participate in this live trial.

As already discussed in the Background chapter of this report, the Ultranet system did not operate as planned, and teachers were unable to trial the system as expected. However, teachers and principals at some schools implemented a ‘Plan B’ activity with Ultranet coaches to enable some learning to take place.

Teachers from schools visited as part of this audit, as well as reviews commissioned by DEECD, have confirmed that the Big Day Out experience significantly tarnished the reputation of the Ultranet among teachers. Since then, many teachers have refused to try using the system again.

A DEECD-commissioned study found that even with support and training, teachers reported being overwhelmed by the idea of using the Ultranet in the classroom. They also felt unable to put the training provided by Ultranet coaches into practice.

A common theme from school visits was that teachers needed time to assimilate all the information they were receiving and decide how to make best use of the Ultranet in the classroom environment. Teachers also said that limited available class time for teaching lessons did not allow enough time for system crashes, multiple logins and generally slow Ultranet performance.

Technical support has not been sufficient. There has been no clear process to raise technical problems and there have been significant delays in receiving responses to issues raised.

DEECD did not consider that the change management process from introducing the Ultranet would entail greater initial workloads for teachers as they learned to use and integrate the Ultranet with their teaching practice.

Even in the longer term, as teachers begin to incorporate more aspects of the Ultranet into their teaching, they will need more time to access relevant training programs to continually update their ICT skills. Currently there is no provision to support teachers to do this professional development work.

DEECD expects schools to appropriately enable parents to access the Ultranet, including providing login assistance and training to use the Ultranet. This responsibility also adds to teachers’ workload.

4.3.3 Benefits realisation

Transition from project to business as usual

The Ultranet transitioned from a project to business as usual (BAU) in April 2012.

Transition to BAU means that responsibility for the Ultranet technical solution now rests with DEECD’s IT Division, while the business owner and business team is the Student Learning Outcomes Division. BAU also means that the contractor should now be solely responsible for hosting and maintenance activities.

However, not all of the contracted functionalities for the Ultranet were delivered prior to transition to BAU. For example, records management, timetabling and student reporting are yet to be delivered to schools. DEECD continues to work with the contractor to deliver these functionalities.

The Ultranet contract ends on 30 June 2013. Unless the contract is extended, DEECD will need to conduct an open and competitive tender for the continued hosting and maintenance of the Ultranet.

Costs of Ultranet implementation

Based on DEECD-provided data, capital and operating expenditure for the Ultranet had reached $97 million by June 2012. By June 2013, it is estimated that it will have cost nearly $109 million, or some $50 million (83 per cent) more than the $60.5 million budget first announced in November 2006.

However, based on a further review of DEECD’s financial data, we estimate that actual capital and operating expenditure was approximately $162 million by June 2012, and by June 2013 it is likely to have cost approximately $180 million.

Figure 4D

Variance in budgeted versus estimated costs for Ultranet

Figure 4D Variance in budgeted versus estimated costs for Ultranet

Note: The project budget up to 2011–12 includes $60.5 million in total estimated investment, including $21.8 million for Ultranet coaches, and $9.7 million for ongoing funding for software and licensing fees in 2011–12. In addition, the project budget up to 2012–13 includes $9.7 million for ongoing funding for software and licensing fees in 2012–13.

Source: Victorian Auditor-General’s Office.

The variance between the DEECD and VAGO estimates is explained by DEECD’s failure to record and track all costs relating to the Ultranet. For example, DEECD has not classified all relevant Ultranet costs (such as project team staff costs) to the Ultranet cost centre code.

Publicly available information on the Ultranet, such as data provided by DEECD on its website, continues to refer to the Ultranet as a $60.5 million government project. This is misleading and does not accurately represent the full cost of the Ultranet.

Despite this significant overall expenditure, no cost-benefit analysis has been conducted by DEECD to determine whether the Ultranet provides value for money, or whether the same functionality could have been delivered more cost effectively.

Functionality: promised versus actually delivered

Although the Ultranet has been transitioned from a project to BAU, much of the functionality promised in November 2006 has either not been delivered or has been de-scoped from the original specifications.

Figure 4E sets out commitments from ministerial statements and business cases versus the actual Ultranet capability that has been deployed.

Figure 4E

Assessment of VicSmart’s expected benefits

2007 government commitment

Achieved

Victorian parents will get virtual access to their child’s classroom 24 hours a day, seven days a week under Labor’s $60.5 million plan to revolutionise learning.

No

The Ultranet will allow parents to log in to their child’s classroom, check lesson plans, homework and results, attendance, and even communicate directly with teachers via email.

No

Every classroom in a school and every school within the state will be linked, starting in 2007, with the rollout completed within three years.

Yes/No(a)

Students would have their own virtual work spaces with homepages and personally tailored lesson plans accessible, via the internet, from anywhere.

Yes(b)

The Ultranet will consolidate school administrative functions into the one system and lighten the burden on teachers

No

Parents will be able to monitor school attendance, which will be recorded twice a day, with automatic notification via text, email or phone call of non‑attendance.

No

(a) Although all schools are now linked to the Ultranet, this started happening in 2010, not 2007.

(b) While this functionality is now available in the Ultranet, very few students have in fact set up their personal work spaces.

Source: Victorian Auditor-General’s Office.

Ultranet use

At present, use of the Ultranet is well below expectations. DEECD documentation shows that, on average, there were fewer than 11 000 teacher and 52 000 student unique monthly logins for the period from February 2011 to September 2012.

This represents only 10 per cent of students and 27 per cent of teachers, or 4.2 per cent of the 1.5 million total expected Ultranet users projected in both the 2007 and 2009 business cases. There is no equivalent data on parent usage available from DEECD.

Departmental documentation suggests that real teacher logins may actually be much lower. This is because some schools have established the practice of having staff regularly log in, so that their presence is centrally recorded, even if they do not undertake any activity in the Ultranet. This means that, at best, DEECD’s data is on the high side.

DEECD data also shows that unique monthly student logins are below 5 per cent for at least 50 per cent of the 1 587 schools across the Victorian public school system for the period November 2010 to August 2012. The percentage of schools whose students did not access the Ultranet at all has also increased from 15 per cent in 2010 to 21 per cent in 2012.

While significant online materials and training support has been provided to teachers, these have been neither sufficient nor sustained. DEECD has not fully considered the demand the Ultranet would have on teachers’ time and workload, particularly in terms of teachers’ roles in supporting and guiding parents on how to use the Ultranet.

System problems have also discouraged uptake across the government school system. These include the Ultranet’s technical unreliability and poor performance, the lack of appropriate support to develop principals’ and teachers’ ICT competency and confidence, and ageing ICT infrastructure and devices (which can affect online access to the Ultranet).

Use of third-party non-Ultranet software in schools

Prior to the development of the Ultranet concept, many government schools had purchased and were using third-party ICT software, applications and platforms, such as Moodle and Compass, for classroom learning.

This practice has continued after the rollout of the Ultranet, with DEECD data confirming that the number of schools purchasing and using these third-party applications has remained at about the same level as before the introduction of the Ultranet.

While both the 2007 and 2009 business cases were underpinned by the assumption that all schools would use the Ultranet, DEECD has advised that government schools are not mandated to use the Ultranet. DEECD asserts that this position reflects the government’s policy of school autonomy.

DEECD has also decided to allow schools to have greater control over their Ultranet deployment schedules in response to strong feedback from schools.

In addition, the teachers’ union informed DEECD that it had resolved in March 2010 that schools should be allowed to determine when and to what extent they should use the Ultranet, because the significant issues encountered by teachers when using the Ultranet. The union has also recommended to its members various work bans in relation to the Ultranet as part of its current pay and conditions campaign.

Stakeholder consultations and analysis of documentation confirm that the Ultranet was intended to be used by all government schools. The change in DEECD’s position, allowing schools to opt in or opt out as they please, means that it is highly unlikely that the Ultranet’s objective of educational collaboration and providing transferring students a seamless transition from school to school will be achieved.

The achievement of these objectives relies on all schools across the government system using the Ultranet, either by mandating its use or putting significant effort into assuring that the system’s functionality, implementation and issues management are given appropriate consideration to maximise its use.

The November 2011 speech by the Minister for Education Victoria as a Learning Community refers to the need to complete the functionality and assuring the quality of the Ultranet in order to encourage its widespread use.

DEECD needs to give greater consideration and attention to the current ambiguity in regards to the Ultranet opt-out issue. Without near-universal take-up by teachers and students—as forecast in all the business cases—most of the benefits will not be achievable, and the significant investment of taxpayers’ funds to date will therefore be wasted.

Recommendations

The Department of Education and Early Childhood Development should:

  1. urgently review its investment in the Ultranet, with a particular focus on:
    • assessing whether the contractor has delivered all functionality as required by the contract and what action, if any, needs to be taken to enforce the state’s rights
    • rigorously assessing its financial management practices and identifying the real, current cost of the Ultranet to determine the extent to which further investment is warranted
    • identifying and addressing the underlying causes of low take-up rates across the school system by teachers, students and parents
    • providing advice to government on the cost-benefit of decommissioning the system now against continuing to fund and rectify the system so that it can be implemented as originally expected
  2. conduct an agency-wide review of its internal tendering, probity and financial management practices in light of the serious issues identified by this audit
  3. expedite the provision of guidance to schools on the current status of the Ultranet as the department’s key learning technology investment, and clarify the policy context of schools’ autonomy in purchasing non-Ultranet learning technologies.

Back to top

Appendix A. Audit Act 1994 section 16—submissions and comments

In accordance with section 16(3) of the Audit Act 1994 a copy of this report, or relevant extracts, were provided to the Department of Education and Early Childhood Development, the Department of Premier and Cabinet and the Department of Treasury and Finance, with a request for submissions or comments.

The submissions and comments provided are not subject to audit nor the evidentiary standards required to reach an audit conclusion. Responsibility for the accuracy, fairness and balance of those comments rests solely with the agency head.

RESPONSE provided by the Secretary, Department of Education and Early Childhood Development
RESPONSE provided by the Secretary, Department of Education and Early Childhood Development – continued

Back to top