Akamai Diversity

The Akamai UK Blog

Managing bots effectively can drive up website performance

The chances are your site has a bot problem. Unwanted bots decrease website performance, increase bandwidth costs and might be giving your competitors an unfair advantage.  Even if you can successfully identify bots trying to scrape your site, blocking them is never a good idea as they will just evolve to avoid further detection.  Akamai Bot Manager gives you the ability to silent deny (so not alerting the sender), give a delayed response (to prioritise human users) or serve alternative content (to deliberately give incorrect pricing information), therefore resulting in reduced bandwidth and infrastructure costs, enhanced user experience and stronger competitive positioning.

Analysis of traffic across the Akamai Intelligent Platform™, shows that upward of 60 percent of an organisation's Web traffic can be generated by bots.  In fact, during tests with some of our UK beta testers, we saw cases where bot traffic reached peaks of 90% of total traffic.  You may monitor load and performance, but do you know how much of that traffic is bots?

Not all web bots are bad and some are essential, such as those from search engines - they are a key part of making sure your website is ranked well in search results.  Others may be from suppliers and business partners and need to hit your site as a normal part of doing business.  An effective bot management solution should give you the ability to prioritise the 'good bots', as slow website performance could impact your search engine rankings, for example. 

We've mentioned website scraping, but web bots can even simulate buying customers, filling shopping baskets so that genuine shoppers think you are out of stock of a hot product, or have no hotel rooms available, forcing them to shop elsewhere.  There are also bots that try to imitate good bots, making it even harder to manage this type of bot traffic.  The key here is a solution's ability to correctly identify and categorise every type of bot hitting your website, in order to manage bots in the best possible way. 

Understanding the type and volume of bot traffic is very hard, as it appears as standard HTTP requests.  By separating legitimate users and bots from grey and bad bots, can help improve website performance significantly.  Simply denying bots is a mistake, to manage bots you have to understand them, and make decisions about each encountered: to silent deny, send a delayed response, serve alternative content, or allow.  At Akamai we have the largest catalogue of known good bots (1,300+), which combined with our heuristic analysis means we can take the headache out of managing bots in a way other solutions can't.  

This is what managing bots is all about: making them work to your advantage, not simply accepting web bots as a cost of doing business online.  Akamai Bot Manager can have an immediate impact on your website performance, and protect you from bots that give others competitive advantage.  If you don't know the impact bots are having on your website performance, let us show you, and take back control.