The New York Stock Exchange (ICE) crashed Wednesday at 11:32 a.m. ET and remained down for exactly 3 hours and 38 minutes. It caught many traders off guard and raised concerns of what was happening behind the scenes that might have caused the “technical glitch,” the company said it encountered.
The NYSE now has a detailed explanation and it all comes down to the rollout of a new software release. (The one that was referenced in Fortune‘s interview with trading systems expert Scott Hunsader yesterday.)
The exchange was prepping for a July 11 industry test that’s on the horizon, which will test the platform’s SIP timestamp requirement. The software update was rolled out to one trading unit–the standard practice. But, as investors started logging on Wednesday morning, a series of communications issues popped up between customer gateways (what investors use to connect to the NYSE) and the trading unit using the new software.
It turns out that those gateways were not loaded with the proper information needed to work in tandem with the release, so the NYSE updated the gateways with the correct version of software and opened for trading as usual at 9:30 a.m.
However, the update didn’t go as planned and the communication issues proliferated, causing the exchange to send out a message that a technical issue was being investigated at 11:09 a.m. ET before suspending all trading at 11:32 a.m. ET.
The NYSE then went about canceling all open orders as well as working with customers to reconcile orders and trades, all while restarting all customer gateways and implementing a complete system restart after consulting with regulators and industry peers. After that, the NYSE got everything back up and running by 3:10 p.m. ET.