Akamai Diversity

The Akamai Blog

Bot Management Strategy

It was March 13th, 2:30 AM at night and the customer called everyone on the Akamai account team announcing they were being attacked.  The attacker was locking inventory on their site for hours causing a significant burst in traffic and preventing customers from making transactions. The Akamai Security Operations Center was involved right away and quickly discovered that a bot was behind this attack.  This was a "good bot" just scraping the inventory for pricing data, but it caused havoc for both the infrastructure and the business.

Our research has shown that about half of all Internet traffic comes from bots. Bots are small programs that interact with a website and can be used for good or evil.  While bot traffic is unavoidable and not all bots are harmful, there are certainly those who use them for ill purpose, and that's what Bot Manager aims to fix.

Bot Traffic.png

Bot Management should be discussed among the security and marketing groups.  The security groups need to ensure that the site is up and running and not impacted by the bots due to lack of infrastructure, depleting resources and lacking inventory.  Marketing needs to ensure only "the right" bots get the site content. For example, competitors get fake prices and bad bots are served non-relevant content. 

The Akamai Bot Manager gives you fine-grained control over bot traffic hitting your site, helping you discover the nature of the traffic before taking action. Bots can be divided into good, bad and "grey", good bots that cause a lot of havoc.  Some examples of good bots would be search engine crawlers. Blocking these types of bots could have a severely negative impact on your SEO. But then there are also the bad bots, such as competitor scrapers or even DDoS attacks. However, we also see a lot of "grey" bots - good bots like brand view bot that cause significant number of page views and searches overwhelming infrastructure.    

Bots can also change and evolve their strategy for scraping a site when they are detected, which makes the Bot problem more difficult.  This means you can't simply block a bot, you need to monitor and evaluate it on a regular basis and the Akamai bot manager provides you the interface and the information to do this.

Akamai Control.png

Akamai Bot Manager gives you a full view of all bot traffic on your site, and lets you dive down into detailed information such as host name, IP address, what activities were performed, and even the impact it had on website performance. This allows you to take smart actions against bots:

1.)   Block - deny access to the site

2.)   Rate Control - slow down the access to the site

3.)   Tar Pit - serve no response

4.)   Serve an alternate - present a page different from the site

An effective bot management strategy starts with an understanding of the different types of bots and how each can impact the business for good or bad. Obviously, you want to fight back against malicious web content scraping, price scraping, inventory grabbers, personal information harvesters, and the like, that can damage your competitive advantage. But you also want to maximize the benefit from legitimate good bots. Even legitimate bots that benefit your business, like those operated by partners or contracted third-party services, may behave just like their bad counterparts when it comes to targeting certain information or generating aggressive traffic - so you don't want to block them accidentally.

The bottom line is pretty simple: refrain from blocking bots as much as possible and consider alternate behavior instead for a more efficient bot management strategy. Manage them using the Akamai Bot Management solution that enables your organization to gain greater visibility into the bots that access your site and greater control over the actions you can take.

This approach works best as a strong complement to a comprehensive web application security strategy that includes a web application firewall (WAF). Most of all, the Akamai bot management solution will give you the widest and most flexible range of options to apply to controlling bot traffic so that your business - and all of its legitimate customers and business partners - will be minimally impacted by malicious bot activity.