Auckland CBD Electricity Supply Failure

Poor risk management and inadequate contingency planning were major contributing factors to the power supply failure.

***

After a series of four power cable failures, on 20th February 1998, Mercury Energy Limited (“Mercury”) the major distributor and retailer of electrical power to the city of Auckland, announced that it could no longer supply power to the central business district.

At the beginning of 1998, almost all of downtown Auckland received electricity from the supplier Mercury via only four power cables, two of them 40-year old gas-filled cables past their replacement date.

One of the cables failed on 20 January, possibly due to the unusually hot and dry conditions, another on 9 February, and due to the increased load from the failure of the first cables, the remaining two failed on 19 and 20 February, leaving the central business district (except parts of a few streets) without power.

It took five weeks before an emergency overhead cable was completed to restore the power supply.

For much of that time, about 60,000 of the 74,000 people who worked in the area in 1998 worked from home, or from relocated offices in the suburbs. Some businesses relocated staff to other New Zealand cities, or even to Australia.

Most of the 6,000 apartment dwellers in the area had to find alternative accommodation.

***
The New Zealand Government commissioned a Ministerial Inquiry to examine the failure of the cables and Mercy’s management and technical expertise.

The 193 page report covers the history of the supply to the CBD, the cable failures, why the power supply failed, governance, conclusions and recommendations.

http://www.med.govt.nz/templates/Page____12136.aspx

Corporate Governance Issues

The findings on corporate governance as it relates to the failure of power supply were:

• The corporate governance structure of Mercury did not cause the power supply to fail, but through its effects on governance an opportunity to prevent it was lost.
  
• The corporate governance structure of Mercury did not provide clear lines of accountability or strong shareholder instigated disciplines. These clear lines of objectives, accountability, reporting and discipline were considered important if not vital to create an environment conducive to strong company performance.
  
• Network risks and other risk management procedures differed and accountability and the performance and implementation of both was not clear.
  

Operational Issues

The findings relating to operational matters were:

• Maintenance policy – Mercury relied on the manufacturer’s maintenance procedures. Whilst it was considered reasonable practice to rely on manufacturer’s manuals initially, over time the company should have developed company specific manuals that took into account the ageing condition of its assets, as well as new developments in knowledge pertaining to those assets that have developed since the date of manufacture and installation.
  
• Contractual arrangements – the agreement that Mercury had with Energy Australia did not bind Energy Australia to undertake any work for Mercury at any time, but stated the terms upon which work would be done if Mercury requested work and Energy Australia agreed to provide it. If Energy Australia had need of their own staff, Mercury would be without the necessary skills. The Inquiry found that Mercury’s contractual arrangements were insufficiently specified given the company’s reliance upon them.
  
• Maintenance contracts – the relevant contract did not provide adequate definition of the role between the contractor and the company. The contract did not identify any link the manufacturer’s maintenance specifications, job requirements, quality standards or personnel expertise. The contract provided no link to a maintenance policy. The contract did not clearly define accountabilities and decision-making roles. The contract assumed the contractor had knowledge of all the maintenance requirements on the cables although Mercury argued they maintained the in-house expertise for this role. The Inquiry concluded that Mercury’s maintenance contract was ineffective.
  
• Spare parts policy – Mercury relied on the availability of spares from Energy Australia. However, when spares were required from Energy Australia in 1995/96 and again in 1998 they were not available from that source. Mercury provided no evidence that an adequate contact structure for spares pooling was in place with Energy Australia.
  
• Asset audits – Mercury did not undertake comprehensive asset audits on a periodical basis to confirm capability and ongoing reliability.
  
• Asset integrity – Mercury did not take adequate measures to protect the integrity of its assets. It had allowed routes to be compromised by buildings, traffic, trees, other cables and other threats to supply security.

Risk Management and Contingency Planning

The findings relating to risk management and contingency planning were:

• There was unclear accountability for risk management practice and monitoring at both the senior management and Board level.
  
• When the first electrical failures on the gas cables occurred, Mercury failed to recognize that network risk had increased and treated the failures as it had previous gas leaks; no systematic investigation and resolution of the failure mechanism occurred.
  
• A pre – event risk assessment report in 1997 critically underestimated the failure risk of the gas cables and, as a consequence, Mercury under-planned for this risk.
  
• Mercury failed to assess the increased incidence and risk of outages due to contractor caused damages and failed to take action to address the risk.

***

The report was to have been released on 14th July 1998.

It was, however, delayed out of consideration for the sudden death of Mercury CEO Wayne Gilbert from a heart attack on the day of the report’s intended release.

Department of Defence's Troubled Human Resource System

The Department of Defence did not seek, nor obtain approvals from Cabinet or the Minister of Defence as should have been the case

***

The Personnel Management Key Solution (PMKeyS) Project (“the Project”) was a significant and complex human resource business process change in the Department of Defence (“Defence”) and involved moving military and civilian staff off purpose-built long running human resource legacy systems to a common platform.

The Project, when completed, was to deliver a single integrated system that would consolidate both the delivery of personnel management functions and the number and disparate nature of the systems that performed those functions.

The Project was to deliver these outcomes by June 2000.

The Project suffered extensive schedule slippage, with Phase 1 delayed by 39 weeks and Phase 2 components rolled out between 75 and 158 weeks late.

When the Project closed in December 2002, major outcomes under Phases 3 and 4 had not been delivered.

The Project was also to facilitate significant savings of $100 million per annum. Nearly six months after Project closure the PMKeyS system was yet to demonstrate a return on investment and that although savings had been achieved through decommissioning the legacy systems, those savings fell well short of the costs for far expended.

The total cost to Defence to bring PMKeyS into service is estimated to be at least $131 million.

This cost exceeded Defence’s estimate of $103.5 million to maintain its legacy personnel systems for five years by more than $26 million.

***

The Australian National Audit Office – in an August 2005 report – noted many significant governance and control weaknesses in relation to the Project:

Project Planning and Approval

• The Project Office, as part of its planning processes, did not develop, complete and update a number of the key project planning and management documents, as mandated by Defence procurement policy.
  
• Defence did not seek, nor obtain approvals from Cabinet, or the Minister for Defence and Minister for Finance and Administration for any cost increases in accordance with Defence procurement policy.

Contractual Arrangements

• The Software Vendor and Training Contractor were operating for an extended period without contractual coverage for the work that they were undertaking.
  
• For the period between 1999 and 2001, the Project did not have a Defence Project Director and employees of the contracted Project Manager fulfilled the Project Director’s role – thereby reducing governance over the Project Manager.
  
• The (then) Office of Government Information Technology contract with the Software Vendor required that services provided under the Official Order had to be formally accepted by Defence by way of a Certificate of Acceptance. Limited evidence was found of the Certificates of Acceptance and Defence advised that they adopted less specific acceptance criteria. In the absence of Certificates of Acceptance, most of the payments made to the Software Vendor under the Official Order were not made in accordance with the contract terms.
  
• The Official Order was not compliant with Defence procurement guidance in that it did not include specific provisions for liquidated damages and performance incentives.
  
• Rebates were to become payable under the Official Order in the event of missed contract milestone dates. Notwithstanding that the Project missed all milestone dates, no rebates were paid to Defence.

Cost Performance

• Defence was found to have spent $15.76 million on contracted project management services, which was a thirteen fold increase over the initial budget.
  
• The Project Office underestimated the cost of providing training to PMKeyS end-users, which increased nearly fourteen fold.

Training

• Following Project rollout, responsibility for training was devolved across Defence, producing a disparate ongoing training environment that was reported to adversely effect personnel data quality. This was situation was addressed through the establishment of a central training authority in December 2004 – more than four years after the first stage of PMKeyS rollout.
  
• When rolled out, the Project training was inadequate in that it often did not reflect the delivered functionality. Initial Phase 2 training, which was delivered 14 months prior to the sytem’s rollout to Army, was inappropriately timed.

Lessons Learnt

Defence agreed with the Australian National Audit Office on the lessons that could be learnt from the Project. They included:

• The need for project approval processes for IT systems to comply with Government and departmental requirements.
  
• Future management information system projects should be based on realistic estimates of project costs and system infrastructure requirements that have been subject to close analysis and review, prior to project approval.
  
• The need for a structured process of periodic management review, following the awarding of contracts, to provide additional assurance on the robustness of schedule, cost and performance outcomes being achieved in material projects.
  
• Training delivered to end-users as part of new management information system implementations should be appropriately timed and reflect the functionality of the delivered system. Where this does not occur, follow-up or revised training programs need to be implemented.
  
• Meaningful and measurable key performance indicators should be implemented to assist Defence in the monitoring of the effectiveness of management information system remediation initiatives.