The following is a summary of the system issue that occurred on the EDGX Exchange on Wednesday, April 13th, 2011.
At approximately 3:19, EDGX experienced a database issue causing applications to lose connectivity to the database. Specifically, there was preparatory work being performed in anticipation of the testing and launch of our DR data center, and a recovery command was run in error in a production environment. As part of the adverse effect of this loss of connectivity to the database, other upstream processes were affected. Loss of connectivity to these tools created a state of unknown system activity for our Market Operations team, including the inability to affect administrative commands like “cancel all open orders by customer” or to halt the entire EDGX market. Further, customers’ “cancel upon disconnection” features did not work since this functionality also issues similar administrative commands to the matching engines. All EDGX member orders received after 3:19 were potentially subject to being affected by this issue as database-reliant applications and interfaces began to be affected.
Upon recognition of the root cause of the issue, a decision was made to halt the matching engines to limit any additional risk. A notification was sent to members and made available on our website advising members that the EDGX Exchange did not anticipate reopening for trading for the remainder of the trading day
Nature of Corrective and follow up action
Yesterday’s issue has been resolved.
The nature of the issue was due to human error in running a recovery command in a production environment. All files were sent to DTCC and Direct Edge worked diligently to reconcile trades with members.
In an effort to prevent a similar issue from occurring again, Direct Edge is implementing an alternative way to get administrative type messages directly into the matching engine through multiple channels. This would be equivalent to an additional shutoff switch.
We apologize for any inconvenience this may have caused.
