20030227 IT Best Practice

The results of our review of the implementation of AMS by RMIT highlight the need for learning from this experience. While recognising that, in a devolved environment, each chief executive officer is responsible for the effective and efficient use of resources, the issue of the need for guidance and how organisations can learn from the experience of others needs to be addressed. It is inherently inefficient and of questionable cost-effectiveness for mechanisms not to be put in place where experiences can be shared. To this end, the following section of this report provides an introduction to some better practices in the implementation of IT systems. It acknowledges material drawn from Audit Report No. 12, 2001-2002, Selection, Implementation and Management of Financial Management Information Systems in Commonwealth Agencies prepared by the Australian National Audit Office.

In March 2001, the Organisation for Economic Cooperation and Development (OECD) released a management brief that addressed the problems with implementing large IT projects in both the public and private sectors. It reported that “… budgets are exceeded, deadlines are over-run and often the quality of the new system is far below the standard agreed when the project was undertaken”. As few as 28 per cent of IT projects undertaken in the United States in 2000 were successful in relation to budget, functionality and timeliness. An equivalent number of IT projects were cancelled. The OECD recognised that problems with IT projects represented a significant economic, political, efficiency and effectiveness risk to government. More concerning is that IT implementations that do not achieve their objectives put at risk e-government initiatives.

The OECD identified some key factors that contribute to public sector organisations “getting it right” in relation to IT projects. These key success factors included:

The OECD concluded that public sector organisations should not take risks, instead they should identify risks, determine which risks they are willing to take and manage those risks within appropriate governance structures.

In the context of the comments above, this section highlights the key lessons learnt from the AMS project, which are applicable to any large system implementation project. The findings are grouped under what we believe are the 7 key focus areas for any project as outlined below.

A project needs a clearly and tightly defined scope and time frame to prevent scope creep from occurring. This is important when multiple stakeholders, such as faculties, have differing opinions regarding the solution to be delivered. In the AMS project it was noted that each faculty had differing views on the AMS solution, resulting in each faculty having its own software requirements implemented.

Any changes to the scope or focus of the project should be clearly defined in concise and definitive documentation to ensure that a record is maintained of approved changes and that such changes are appropriately approved at the highest level since they could jeopardise the entire project. This will enable an organisation to determine whether the solution delivered matches what had been previously requested. This was a prime factor causing difficulty in rectifying the problems with the initial AMS implementation.

A “go-live” decision should only happen after extensive testing or piloting, and should include ensuring that either all scope items have been fully met or that clear post-implementation plans exist which set out time frames for their achievement. This would have ensured that the final solution delivered to RMIT met the required outcomes. Alternatively, it would have highlighted problems prior to going live and enabled more informed decision making regarding the AMS implementation.

Business benefits are realised

The implementation of a solution that is proven and accepted within the market, reduces the risk of implementation failure, than would otherwise be the case when a software package is selected that is yet to establish a strong market reputation. In the case of the AMS, the project was built around a combination of versions which had yet to be proven in the market.

A comprehensive independent quality assurance (QA) function will play a major role in assisting to ensure that a project successfully delivers benefits. This is achieved by QA ensuring project structures, processes and deliverables are in place and operating throughout the project. QA is particularly important at the start of the project to ensure that appropriate project structure and risk management mechanisms are in place to protect the funds invested in the project. RMIT has only recently introduced this QA function for the AMS project.

Effective business transformation is a key focus for any major business change project. A well planned and resourced project will ensure that transformation occurs as part of the implementation of a system. However, the project should not be used to force change in an organisation. This was seen at RMIT and resulted in both poor faculty buy-in and limited user take up of the AMS.

Business change projects should be justified on delivering wider benefits than simply administrative cost savings as these may not be realised until the medium to long-term after the system has been appropriately “bedded down”. Administrative cost savings initially had a significant focus in the AMS project and this placed additional stress on administrative staff.

Stakeholders are committed

Identification and buy-in of key stakeholders is required upfront to ensure that the necessary support is obtained throughout the organisation. If this support is not obtained, key stakeholders feel that a change is being forced on them and are, therefore, resistant to the change being delivered. This was cited in a number of reports as a key issue in the early stages of the AMS project.

A go-live decision should be based on the readiness of the entire organisation with input from all stakeholders. This will prevent incorrect decisions such as the original go-live decision made by RMIT, which was based on obtaining input from only the Project Director and the outsourced implementers, rather than from the faculties who were in a more appropriate position to assess readiness.

Team is high performing

A clearly defined governance structure must be implemented prior to the commencement of the project as it is far more difficult to attempt to establish this after the project has commenced. This was evident on the AMS project when effective governance arrangements only became a major focus post go-live, after significant problems had been encountered.

The use of contractors and third parties must be tightly controlled to ensure that their role is understood, their performance measured and appropriate contractual mechanisms are in place. This will ensure that payments made to contractors or outsourcers are supported by clearly identifiable deliverables and outcomes.

Organisations should draw, as much as possible, from their existing staff to resource major projects, as they have greater organisational knowledge and experience than any third party. This is also normally a more cost-effective solution. Where project tasks are outsourced there should be a clear need either due to internal resource constraints or the need to obtain particular technical expertise.

While the track record and expertise of a consulting organisation are valuable, it is the skills and capabilities of the individuals working on the project that will decide the success of the project. Significant effort should be made to validate individuals’ experience and expertise before accepting them onto the project. In the case of the AMS, there were concerns about the relevant PeopleSoft experience of some of the contractors engaged.

The reviews of the AMS project identified that a blame culture initially existed within the project. A project culture which aims to solve problems rather than assign blame, will assist to ensure that an effective resolution of implementation issues can occur. The impact of this culture on the AMS project was that it demoralised the project team and impacted both the progress and quality of rectification work performed.

Critical roles such as Project Director and Project Manager require full-time commitment to large business change projects, to ensure that the project is adequately managed and monitored. There were concerns on the AMS project that the Project Director, until recently, did not have a full-time commitment to the project and, therefore, could not effectively manage the rectification activities.

Where project activities are substantially outsourced, an appropriately skilled internal Project Director and/or Project Manager must be appointed to ensure that the organisation’s interests and needs are being effectively met. No matter how strong the outsourcing contract is, there must always be strong management and monitoring of outsourcer activities. An organisation can outsource the development and implementation of a solution, however, they can never fully outsource the risk, as ultimately, the organisation rather than the outsourcer will be held accountable for the success of the project.

A critical part of any implementation team is an independent QA function that can provide feedback and recommendations regarding a project’s status, its structures and processes. The adoption of a QA function provides early warnings to the steering committee and management of potential problems. The AMS project has only recently implemented an effective QA function.

Work and schedule is predictable

Strong project planning ensures that when a system goes live, it does not occur during an organisation’s critical processing period. This was not the case with the AMS which initially went live in October, when academic results needed to be issued, graduations organised and enrolments taken.

Pilot and parallel running rather than the whole system going live at the one time is a more risk averse approach to IT system implementation. The whole AMS was implemented at the one time which prevented RMIT reverting to its pre-existing systems after it appeared that the implementation had been unsuccessful.

Key milestones and deliverables must be defined up front, so that a project’s progress can be monitored against these time frames and deliverables. The absence of this definition upfront, increases the risk that status reporting can be clouded by a project manager’s understanding of the project’s progress which may not be supported by the achievement of milestones and/or deliverables. This was the case in the initial AMS implementation, as well as during the rectification, when it was found that status reporting was inadequate as it did not provide a clear indication of the position of the project in relation to key milestones.

Risks are mitigated

An effective steering committee will ensure that a project has the appropriate risk management focus. For example, a steering committee must ensure that a clearly defined set of acceptance criteria exist to enable a structured go-live decision to be made. It would also be critically involved in the selection of outsourcers to ensure that appropriate checks have been made on skills and experience.

Low risk projects will generally not result in radical changes to an organisation’s underlying IT infrastructure. Where new infrastructure is implemented, significant planning is required in the areas of training and resourcing to ensure that an organisation can effectively support the new infrastructure. RMIT implemented a version of Windows 2000 that, at the time, was not consistent with RMIT’s policy for application servers that operated on Unix or Windows NT. RMIT consequently did not have sufficient internal skills to support the new operating environment.

A fixed price implementation reduces the spending risk that a time and materials approach can create, as additional expenditure over the agreed budget amount can be tightly controlled. Furthermore, it prevents an organisation having to fund inefficient outsourcer performance. On the AMS project, the rectification phase was being performed under a time and materials approach rather than a formal contract, with no formalised agreement specifying the outcomes to be delivered. This resulted in significant RMIT expenditure which could not be matched to any clearly identifiable deliverables.

Successful projects adhere to proven project management and system development methodologies. It was only after RMIT resumed full-scale management of the project and commenced implementing a project management framework, that positive results began to be clearly seen from the project.

Significant system implementations require a strong change management focus to ensure that the affected areas of the organisation receive adequate training, communication and support. This ensures that users are prepared for a change that may significantly modify the way they perform day-to-day activities and, therefore, provide a level of comfort that they are ready to accept the change. In relation to the AMS, it was found that this did not satisfactorily occur and resulted in significant resistance from users who felt they were ill equipped to use the AMS.

Planning to ensure that thorough testing and full sign-off is obtained prior to going live prevents premature or inappropriately timed deployment. A number of reports regarding the AMS implementation identified that inadequate testing had been performed prior to going live, resulting in significant and fundamental errors occurring.

Delivery of organisation benefits are realised

A number of the lessons identified above highlight that an organisation can never totally outsource the implementation risk, when undertaking major business change projects. Therefore, organisations must effectively manage outsourcers by addressing the lessons discussed above if organisational benefits are to be realised.

IT Project Management: Best practice