The Board of the Singapore Exchange (SGX) regrets the market disruption and inconvenience caused by the breakdown on 5 November 2014 and has accepted the Board Committee of Inquiry’s (BCOI) recommendations. The power outage, which was triggered by a voltage sag in both power lines to the SGX Primary Data Centre, was caused by a combination of a faulty component in the emergency backup power supply generator and the inability of downstream switches to cope with this malfunction. The BCOI notes that the key cause of the power outage has been addressed, and that the power design has been independently verified by experts to be robust. SGX has also committed to make further improvements beyond the root cause to enhance overall resiliency and incident management. These include enhancing processes and resources to improve business continuity, incident management and the timeliness and effectiveness of crisis communications.
Background
The BCOI was set up to independently oversee investigations into the incident, review SGX’s incident management and crisis communications, and recommend improvements to prevent recurrence and enhance crisis management processes. Following its review of the circumstances of the outage and the recommendations of its independent experts, the BCOI submitted a report to the MAS. The report is enclosed with this press release and includes SGX’s remediating actions to address all the risks and recommendations that have been identified.
BCOI findings and recommendations
1. Power outage at Primary Data Centre
The facilities at SGX’s Primary Data Centre (PDC), including the power supply, are provided by a third party data centre provider. The PDC was built and certified to meet international standards1. The BCOI’s expert also verified that the overall design of the PDC was generally robust and met industry resilience standards and best practices.
However, the BCOI’s expert noted there was a gap in a specific part of the power infrastructure that did not allow the data centre to cope with the particular scenario that occurred on 5 November 2014. The power outage originated from a voltage sag on the two independent Singapore Power utility lines to the SGX PDC. This necessitated a switch over to the emergency backup power supply on both lines. During this switch over, a combination of (i) a faulty component in one of the emergency backup power supply generators and (ii) the inability of downstream switches to cope with this malfunction, caused a momentary total power loss. This led to a shutdown of SGX’s IT systems and equipment, and consequently the closure of the securities and derivatives markets.
SGX has since worked with the data centre provider to address the root cause for the power outage by replacing the faulty component in the emergency backup power supply and bypassing the downstream switches. The updated infrastructure has been successfully tested to be capable of withstanding the same scenario that caused the incident of 5 November 2014. The data centre provider has also agreed to implement further measures, including improved monitoring tools, to improve resiliency and facilitate better informed decision-making in future.
2. Business Continuity and Incident Management
Another BCOI expert conducted an end-to-end review of SGX’s technology operations.
The market closure and the subsequent resumption of trading operations for both the securities and derivatives markets could have been managed better, and the time to recover could have been shorter.
The expert observed that there were material risks in SGX’s recovery operations on the day of the incident. SGX’s monitoring systems at the PDC during the incident did not provide enough information about the PDC’s health and status to assist in decision-making. SGX made the initial decision to recover at the PDC based on information that a complete power outage had not occurred, and that there could have been a temporary communications loss between the PDC and the Secondary Data Centre (SDC). If so, the order books at the PDC and SDC could be mismatched, making it unsafe to failover to the SDC. However, the decision to continue to recover at the PDC, and not fail over to the SDC, was not fully reconsidered when new material information on the complete power outage became available.
Nevertheless, the expert noted that the primary objective of SGX in restoring fair, orderly and transparent markets on 5 November 2014 was achieved. The expert also noted good practices in technology design and IT operational processes.
SGX has put in place a structured programme to address all of the experts’ detailed recommendations. Measures that have already been taken include:
a. Strengthening the technical capabilities and capacity of its Technology Operations team;
b. Enhancing the crisis management protocols and business continuity planning scenarios and exercises; and
c. Enhancing the resourcing of the Business Continuity Management Committee and function
Other recommendations that SGX will be implementing include:
a. Developing an Enterprise Command Centre to improve monitoring and provide a better line of sight into the health and status of facilities, servers, network and applications;
b. Undertaking a thorough review and gap analysis of the number of skilled resources across the organisation required to support both build and run technology teams; and
c. Making improvements to testing across the technology change lifecycle.
All pending actions will be closely tracked by the SGX Board to ensure timely completion.
3. Crisis Communications
An expert was also appointed to review SGX’s crisis communications.
The BCOI’s expert observed that SGX should have more promptly published the initial announcement of the incident, without necessarily waiting for all the details surrounding the incident and the recovery to be fully established. The timing of SGX’s communications could also have been more consistent across the different communication channels. SGX used various channels including SGX.com, SGXNet, social media and direct phone calls to inform its members, market participants and retail investors about the incident.
SGX has since taken measures to meet the BCOI expert’s recommendations to better prepare for, and practise crisis communications. These include improvements to existing crisis communications templates for specific stakeholders, and rigorously rehearsing the use of these improved templates during SGX’s regular business continuity exercises.
4. Other technology incidents
SGX has also reviewed the incident which delayed the opening of the securities market from 9.00 am to 12.30 pm on 3 December 2014. The cause of the 3 December 2014 incident was due to a software defect and unrelated to the 5 November 2014 incident. SGX, assisted by an independent consultant, identified key learning points from the 3 December 2014 incident on business continuity planning, incident recovery and technology resourcing. SGX is similarly committed to implement the measures for improvement.
Conclusion
SGX regrets and fully accepts responsibility for the incident. SGX has taken significant measures to address the BCOI’s recommendations, including elimination of the root cause for the power outage on 5 November 2014 and enhancing its capabilities in incident recovery. The Board’s Risk Management Committee will monitor the progress of the programme established to implement the recommendations.
SGX will continue to strengthen its technology infrastructure and will direct S$20 million to infrastructure investments which will further improve the robustness of SGX’s systems and recovery processes.
SGX has accepted MAS’ reprimand and will fully comply with MAS’ directive to address various issues, including the BCOI’s recommendations.
SGX will also implement a moratorium on any increase to securities and derivatives market fees. This moratorium will only be lifted when SGX has satisfied MAS that it has met its obligations under MAS’ directive.
SGX will also contribute S$1 million to the Investor Education Fund, which is used to fund educational programmes approved by the Investor Education Committee, which includes industry practitioners.
The BCOI report is available on www.sgx.com
***
About the BCOI
On 5 November 2014 at 2.18 pm, a momentary power outage to the Primary Data Centre of the Singapore Exchange (SGX) occurred, which led to closure of both the securities and derivatives markets. The Board of Directors of SGX set up a Board Committee of Inquiry (BCOI) to independently oversee investigations into the incident, review SGX management’s decision making, incident management and crisis communications, and recommend improvements to prevent recurrence and enhance crisis management processes.
In accordance with its terms of reference, the BCOI was set up to:
a. Oversee investigations into the facts and circumstances leading to the breakdown;
b. Review the findings of the investigations, SGX’s decision making process, incident management and responses, and communications with market participants and the general public; and
c. Recommend appropriate improvements to prevent recurrence and to enhance processes in crisis management.
The BCOI comprises four Board directors who are all independent of SGX management. Mr Quah Wee Ghee is the Chairman of the BCOI and the SGX Board’s Risk Management Committee. The other three members are Mr Chew Choon Seng, Mr Kevin Kwok and Mr Lee Hsien Yang.
The BCOI was assisted by independent consultants, who were appointed after a rigorous selection and interview process. They are: IBM for IT infrastructure and market operations, Environmental Systems Design Inc. for data centre facilities, Edelman Singapore for crisis communications, and Harry Elias Partnership LLP as legal advisors.
