The recent volatility in the stock market serves as a reminder to those in the brokerage world about the challenges of capacity planning during unstable equity markets. We are only 10 trading days into 2016, and the the S&P 500 index is down 8.0%. As shown in the table below, we've now had 4 days of 400+ point swings in the market.
Days such as these in can result in large spikes of visitors to online brokers, with some reporting up to eight times the number of peak visitors, resulting in performance and availability problems for traders.
Capacity Planning Challenges:
Capacity planning can vary greatly between firms, but it typically involves looking back at historical high volumes of traffic or visitors, and building trading systems to handle some multiple above that load. Headroom is the term that is used to represent that multiple. 3x to 5x is common.
Why is this so difficult in the brokerage industry?
- The spikes cannot be predicted. In the world of online brokerage, flash conditions can occur at any moment in the trading day, on any trading day of year. A Fed announcement, a drop in oil prices, a sharp decline in the Chinese stock markets, are all examples of recent news leading to high volumes for US brokers.
- All the action is at market open. The bulk of online trading occurs within he first hour of the 9:30 AM market open, and that traffic must be carefully analyzed. The granularity of the measurement is critical. Daily, or even one hour averages do not show the spikes properly. For example, a spike of 3x with a one hour granularity (window of measurement) can easily turn in to a 6x spike with a 5 minute granularity
- There is no seasonality. In Commerce, Black Friday and Cyber Monday are well known and expected. In Media, major sports and entertainment events are scheduled well in advance. But financial markets are susceptible to live and unpredictable events globally, every day.
- Shared networks. In many firms, the same network that is used for production traffic is also used internally for employees, video delivery, for QA and load testing. Capacity planners often don't know how to model or control these internal loads, or ignore them altogether. I've seen development groups performance load testing on production networks while market events were occurring.
- The mobile factor. Greed and fear drive the markets, and the convenience of mobile applications enable jittery traders to check in much more frequently than in the past. And it's not just brokerage sites getting hammered. Akamai has observed increased traffic to online bank sites that do not offer trading, during these some of these peak market days. Whether users are reading news, checking rates, or simple checking in with their bank, a volatile day on the markets can result in increased traffic across much of the industry.
Much is at stake for online brokers. The largest online brokers conduct roughly 400,000 trades per day, at roughly $10 per trade. That's $4M per day in commissions, with much of that occurring in the first hour of trading. In addition to revenue impact, an availability problem many times results in a customers moving to another broker. Most active traders maintain brokerage accounts at two or even three brokers. Traders understand that systems fail, but they have little tolerance and long memories when brokers let them down.
Other sectors and industries can learn a great deal by studying the capacity planning challenges of the brokerage industry. In addition to techniques and capacity models, understanding the behavior of this special class of online users can present new ideas for your company. Brokers can also learn from studying other sectors, especially commerce.
Akamai can also provide lessons learned. Any modern capacity planning model should include our platform, which acts like a massive, always on shock absorber to handle flash crowds to a web site. During the Flash Crash of May 6, 2010, we saw this shock absorber in action.
For those of you in the industry, hang on for a wild ride in 2016. Please reach out to us if we can assist you in any way.
Rich Bolstridge is Chief Strategist, Financial Services at Akamai Technologies