Akamai Diversity

The Akamai Blog

(All We Are Saying Is) Give Bots a Chance

Lately, it seems, bots have been taking a beating in the security press.  They are blamed for DDoS attacks, for Web Attacks, for price scraping, for Grey Marketeering, and even, according to some, for Ted Cruz's recent win in Iowa.  Bot are ALL bots bad ALL the time?  We say NO!  Why not? Let me count the ways:

First, if you need to improve your search ratings, Google and other search bots, should usually be welcomed to your site with open arms.

Second, if you need to make information available to partners, partner bots should be welcomed on your site.

Third, if you are enjoying the benefits of some 3rd party services that are provided through a bot, 3rd party service bots should be welcomed on your site.

In other words, Bots are not necessarily bad.  And by extension, if your Bot Management solution is only blocking bots that it has identified, you've got a problem--because search engine bots, partner bots, and bots that are delivering 3rd party services that you want, need and may have even paid for are being blocked from accessing your site. 

Of course, it is hard not to overreact when we hear about bots.  In fact, some analysts have said that bots in 2016 account for over 50% of the total traffic on the web.  Measuring the total amount of bot traffic is inherently difficult - at Akamai we see a wide variance in both bot and legitimate traffic across the millions of websites we serve.  Some sites see as little as 3% of traffic from bots.  Others as much as 70%.  There are outliers, and there are some trends that we are in the process of verifying. For now, though, we believe it is safe to say that the median % of bots visiting websites, as a percentage of total website traffic, is 31%. 

So roughly speaking, somewhere around 31% of the traffic accessing your site could be bot traffic. And bot traffic is not necessarily bad, so you shouldn't block all of it. So what should you do?  Perhaps a new approach to bot traffic is needed.  Perhaps we need more choices beyond just "Block or not".  Perhaps there is a place in the market for a company that introduces a Bot Management tool that offers advanced functionality.  Such a company might just start a Bot Management Revolution.  What's that?  You say you *want* a revolution?  Watch this space....