As you may have heard, Akamai recently introduced a new product, . I've been working at Akamai for close to 10 years and, in my past roles here (Technical Support Engineer, Enterprise Architect), I've had the opportunity to work closely with many customers who had issues with bots. Generally, this was about protecting the site against "bad bots" but also making sure that "good bots" were not impacted by any of the mitigation techniques.
During my support days, when it came to bad bots, the first things that customers wanted to do is block them to make the problem go away. However, by doing so, the botnet operator knew we were onto them and changed their implementation to evade detection. If we blocked based on IP and ASN, they'd quickly "move" their botnet from one cloud provider to another. When blocking based on user-agent or other anomalies we found in the request, they'd change their header signature. It was fascinating to see how quickly they would react to our actions. So, you get the idea, blocking is not very effective to make he problem go away and it is preferable to adopt more advanced response strategies.
When I joined the Web Security engineering team as a Product Architect two years ago I was tasked with designing and architecting a product that would help customers detect bots. From my past experiencewe decided not only to focus on detection but also providing more advanced mitigation techniques to respond more intelligently to bots. The concept of bot management was born...
In this series of articles, I intend to share what we (Akamai) mean by managing bots. The Bot Manager product comes with various detection and response techniques so you won't have to do the kind of custom work I did back in the days to detect bots. But no matter how strong the detection techniques are, it is technically possible to bypass them all (but I won't tell you how). Combining strong detection techniques with intelligent responses helps keep bot activity under control. Now, if blocking is bad then what are you supposed to do? Well, it depends... on the context, the value of the data being extracted or product / service being abused, the type of bot you're dealing with, and the motivations of the bot operator. To protect valuable content from skilled bot operators you should consider advanced response strategies as described later.
The Good, The Bad, and the Ugly
Before we go into discussing how to respond to bots, let's try to define what are "good bots" and "bad bots".
The notion of good and bad when it comes to bots is quite subjective. For example, let's take the web search engines bots: Akamai has identified over 150 of them and you'd think that everyone would agree that they are good. But when I have conversation with customers, I find that the reality is different. Everyone agrees that Googlebot (google.com) and Bingbot (bing.com) are good. However, when it comes to other bots like Baiduspider (from the Chinese baidu.com) or Yandexbot (from the Russian yandex.com), how people in North America feel about them varies. And what about bot that support the Czech search engine Seznam.com? Most likely you don't know which search engine your users prefer to use. So should you limit from which search engine your content will be visible? Do you really want to pass on the opportunity to acquire new customers just because they like doing their search on a North America based emerging search engine like duckduckgo.com?
Classify, then Decide
As far as we (Akamai) are concerned we tried to build a product that is not judgmental. Instead, we categorize bots as known bots (the one that are easily identifiable and supported by legitimate businesses) and unknown bots (the one that we are able to detect but don't fit in the known category). While we don't judge we do have an opinion: all the bots that are classified as "Akamai known bots" in the Bot Manager product, are the one we consider good and we recommend you let them be and serve them whatever they ask for. We classified all the known bots in several categories based on the product or services offered by the company supporting them so that in case you feel that some of the categories do not provide much added value to your business, you can respond to them differently. For more details on the managing good bots, watch for part 3 of this article series.
All the other bots that are detected using our dynamic detection techniques are potentially not so valuable for you. But even so, you should review them to understand who is generating the traffic; it could be one of your partners, you never know. In part 2 of this article series, we'll talk about the best methods to manage this potentially "bad" bot traffic.
Bottom line, the Bot Manager product will help you get better visibility on all bot activity on your site. Based on what you observe, you'll have to decide which bots are good for your business, then adopt the right response strategy for each category.