NASDAQ OMX is pointing to unusually high system demands from NYSE Arca as the cause of last week’s three-hour trading shutdown, which, in turn, revealed other technical issues that prolonged the disruption.
The exchange provided an update today on the preliminary findings of its internal review of the events leading up to its extended trading disruption on Aug. 22. It says that “a combined series of technology events” caused the initial market problems and then extended the halt period.
“The catalyst for the [Securities Information Processor (SIP)] failure was a confluence of unprecedented events that overwhelmed the processing capacity of the SIP,” it said, adding that high-frequency trading “played no role” in these events.
NASDAQ says that, on that day, the SIP faced an unprecedented volume of message traffic; which was “well beyond” the system’s tested capacity of 10,000 messages per-port, per second. And, it reports, that this exceptional traffic originated with NYSE Arca.
It says the SIP received more than 20 connect and disconnect sequences from NYSE Arca, “each of which consumed significant resources”. It also received a stream of quotes for inaccurate symbols from NYSE Arca, which generated quote rejects. “Both of these actions served to degrade the system below the tested capacity of 10,000 messages per per-port, per second,” it says.
“During this period, NYSE Arca sent multiple bursts with each connect and disconnect, topping more than 26,000 quote updates per-port, per second as it attempted to reconnect,” it says, adding that a typical August day for NYSE Arca would peak at less than 1000 messages per-port, per second.
“The confluence of these events vastly exceeded the SIP’s planned capacity, which caused its failure and then revealed a latent flaw in the SIP’s software code,” it says. This flaw, in turn, prevented the system’s built-in redundancy capabilities from taking over properly, it notes.
“The combination of large system inputs and delayed outputs ultimately degraded the ability of the SIP system to process quotes to an extent that a shutdown of the system was in the broader public interest, to prevent information asymmetry and ensure fair conditions for all market participants,” it says.
While the initial problem was fixed within 30 minutes, the halt dragged on as the exchange sought to test and evaluate scenarios to re-open the market to ensure that trading could be resumed in a fair and orderly manner, it notes.
The exchange says that some of these issues were within its control, while others reflect technology issues that have emerged in the markets generally, which will require a broader industry-wide effort to resolve.
NASDAQ OMX says it is committed to enhancing its systems, and that it’s working to strengthen the market infrastructure. It says that it is “currently identifying potential design changes to further strengthen the SIP’s resiliency, including architectural improvements, information security, disaster recovery plans and capacity parameters.” It intends to present its initial recommendations related to these changes to the SIP governing committee within 30 days.
The exchange says it’s also undertaking a comprehensive review of its policies and procedures for communicating with customers, market participants and the broader public during market-wide events. “We are committed to reviewing and improving our communications practices with all stakeholders,” it says.