Akamai Diversity

The Akamai Blog

Bot Manager: Why It Matters

Earlier today, Akamai announced the launch of a completely new product, Bot Manager. You can read more about the details in the press release and on Akamai.com. I won't bore you by repeating them, but I did want to add some color on why it matters.

Why it matters #1:  Bots are a fact of life

One thing we've all learned is that, if you build it, they will come. We all build websites for different reasons - to sell stuff, to influence others around us, and to build stronger relationships with our customers. To do these things, we put richer, higher-value, and more differentiated content on our sites. It's impossible to divorce that content from any discussion around bots. That content is, in fact, the raison d'être for bots.

As a great example of the law of unintended consequences, it turns out that the content we put on our websites has value to more than just our target audience. It has value to an entire ecosystem of individuals and organizations whose motives and goals may or may not be aligned with ours, and who create bots to extract that content and capture that value. Think of your partners who use that content to resell your products, or your competitors who use it to better compete with you in the market.

Regardless, the point is that you can't make bots go away. They're a fact of our online lives and are, in fact, spawned by the very content we're creating.

Why it matters #2:  Bots mean something different to everybody

It would be no surprise to hear that everybody agrees Googlebot is "good" and DDoS bots are "bad". Googlebot and DDoS bots are at the two opposite ends of the "good" and "bad" spectrum. But if you move slightly more towards the middle of the spectrum, it would also be no surprise to hear that nobody agrees on anything else. What's "good" for one company may be "bad" for another, and to muddy things even more, some bots can be both "good" and "bad" at the same time.

Take content aggregators. Major hotel chains consider aggregators to be trusted partners that help them book rooms. Airlines, however, incur a financial cost every time someone - human or not - goes through the workflow to book a flight on their site, and the cost attributed to bots can reach hundreds of thousands of dollars every month. But even trusted partners can "misbehave" with high traffic at the wrong times, leading to an uncomfortable tradeoff between business benefit and performance impact.

It's that tradeoff that prevents us from taking a one-size-fits-all approach to bots, because everybody has to make a different tradeoff depending on what's right for them.

Why it matters #3:  There is no simple "solution"

Most people trying to solve the bot problem start with the simple idea of blocking the bad ones. This is true for both IT organizations, whose first instinct might be to block the IP addresses for any detected bots, as well as security vendors bringing solutions to market. The problem with blocking is that it's a very black-and-white "solution" - i.e., to block or not - to a very nuanced problem.

The best way to illustrate this nuance is through another example. We often hear from customers about a certain search engine crawler that's problematic, because it doesn't respect the instructions in a website's robots.txt file and instead crawls websites in a very aggressive manner. If the only solution to the bot problem is to block or not, then the question is clear - do you block the crawler and forgo the search benefit, or do you not and suffer the performance impact? Neither outcome is ideal, but these are the only outcomes possible with a simple "solution".

What we forget is that a "solution" is something that solves a problem. But bots present a very nuanced problem, and that problem requires a very nuanced solution.

Why it really matters:  Putting the power in your hands

At the end of the day, what we're doing with Bot Manager is putting the power in your hands. It's your website, your online business strategy, and your determination on what your bot traffic mean to you. We want (and you want) you to have the flexibility to do the right thing for your business. To do so, Bot Manager will help you to better understand the unique mix of bot traffic that's coming to your site, better understand the unique impacts of that traffic on your business and IT infrastructure, and apply a wide range of advanced management actions beyond simple blocking to get more of the benefits, and less of the negative impacts.

Because, at the end of the day, another fact of life is that we all want to do what's right for our business; we just need the tools to do so.