Akamai Diversity

The Akamai Blog

Managing bots 101 - Part 2: Response strategies for unknown bots

In part 1 of this series we've discussed the difficult problem of differentiating the good vs. the bad. In this article we'll review how to go about defining a response strategy to manage bots that you think are bad for your business. First thing you'll have to decide is whether you want to serve any content at all to these bots. We recommend you do to keep the bot at bay but of course it depends on your context and what infrastructure you have available.

Let's assume you've decided you will serve content to bots. The Bot Manager product offers several advanced response strategies to help. Here's a few options in order of my personal preferences:

Serve from alternate origin: Some of our customers have developed a dedicated infrastructure to serve bot traffic. Such infrastructure is generally lightweight and disconnected from critical infrastructure such as databases to prevent any harmful side effects. In the case of bad bots, such infrastructure is usually built as a honeypot. With Bot Manager, you have the ability to send the bot traffic of your choosing to the alternate origin, thus freeing up valuable resources on the primary origin to service requests from real users.

Serve from cache: Some of the content on your site may not be cacheable by default for various reasons. You want to serve the most up to date content to your real users, but consider whether it is acceptable to serve stale content to bots. If you adopt a long TTL (a least 1 day) for your caching strategy, this will help offload your origin, have it focus on servicing real user traffic and overall improve site performance. The caching strategy can be used to achieve a similar behavior as serving from an alternate origin without the cost of dedicated infrastructure.

Delay / serve slow: With this strategy, a delay of one or a few seconds is introduced before servicing the request. This can be useful to slow down bots sending requests at a high rate. In this case, you will serve the same content as regular users but at a human pace, which levels the playing field - which may damage their "business model". This strategy however won't have much effect on a large distributed botnet sending requests at a low rate.

In case you have decided not to serve any (version) of your real content to bots, here's a few response strategies you should consider in order of preference:

Serve alternate page: With this strategy you'll be able to serve a "pre-canned" message to bots. This could be the content of your acceptable usage policy, a custom "Page not found" or "Site unavailable" response. Some of our customer use this strategy to show a sample of their product catalog. Because of the static nature of the content, it is highly cacheable and will help offload your origin a great deal. Or you can show a page that recommends use of an API, which may be more efficient for you (and them) than to generate HTML. This is a reasonable strategy for your business partners for example.

Block (deny): this should be your last resort response strategy. When blocking request, a HTTP 403 response code will be serve to the bot. The bot operator will know right away that they have been discovered and may change their strategy.

So there you have it, a great product with several features to manage that pesky bot activity. For more information on how to achieve the above behaviors using the Luna control center, please review the Bot Manager user guide or contact your account representative for assistance.

Read the first part of this series here: https://blogs.akamai.com/2016/03/managing-bots-101---part-1-stop-playing-the-whack-a-mole-game.html

Leave a comment