Q4 2014 Security Report | Bots, Spiders & Scrapers | Presentation

1
Q4 2014
2
bots, spiders scrapers

• Prevalence of third-party content bots is
  increasing
• Used to collect data from other websites
• Poorly coded bots can impact site performance
• May resemble a denial of service (DoS) attack
• Secure web presence requires a comprehensive
  understanding of how they affect performance
• Download the Q4 2014 Global DDoS Attack Report
  for supporting data and analysis

2 / The State of the Internet / Security (Q4
2014)
3
purposes of bots and scrapers

• Akamai has seen bots and scrapers used for many
  purposes, such as
• Setting up fraudulent sites
• Reuse of consumer price indices
• Analysis of corporate financial statements
• Search and metasearch engines
• Data mashups
• Analysis of stock portfolios
• Competitive intelligence
• Location tracking

3 / The State of the Internet / Security (Q4
2014)
4
example of a bot targeting a financial
aggregator
In this example, the bot targeted the financial
aggregator to scrape a large amount of data
quickly.
Figure 1 Bot targeting financial aggregator to
scrape large amount of data quickly
4 / The State of the Internet / Security (Q4
2014)
5
four categories of bots and scrapers
Figure 2 Desirability is based on how much the
site owner wants to host the bot. Aggressiveness
the rate of requests from the bot and its impact
on site availability.
5 / The State of the Internet / Security (Q4
2014)
6
triage and categorization

• Mitigation techniques vary depending on bot
  classification
• Volume of requests can help determine platform
• The sequence and pages a bot targets can reveal
  intent
• The user-agent header can sometimes provide a
  unique and identifiable user agent
• Additionally, Whois can sometimes identify bot
  owners

6 / The State of the Internet / Security (Q4
2014)
7
corresponding mitigation strategies
Figure 3 Mitigation strategies are based on bot
desirability and aggressiveness
7 / The State of the Internet / Security (Q4
2014)
8
moving forward

• Bots and scrapers will continue to affect
  organizations in all industries
• Every website should develop a strategy to
  address and mitigate undesirable bot behavior
• Each organization will also need to a framework
  to evaluate which bots to allow access to it
  site
• Companies will also need to analyze and modify
  security policies to keep them current with the
  changing times

8 / The State of the Internet / Security (Q4
2014)
9
Q4 2014 global attack report

• Download the Q4 2014 State of the Internet
  Security Report
• The Q4 2014 report covers / Analysis of
  DDoS attack trends / Breakdown of average
  Gbps/Mbps statistics / Year-over-year and
  quarter-by-quarter analysis / Types and
  frequency of application-layer attacks / Types
  and frequency of infrastructure attacks /
  Trends in attack frequency, size and sources /
  Where and when DDoSers launch attacks / Case
  study and analysis

9 / The State of the Internet / Security (Q4
2014)
10
about stateoftheinternet.com

• StateoftheInternet.com, brought to you by Akamai,
  
• serves as the home for content and information
  intended to provide an informed view into online
  connectivity and cybersecurity trends as well as
  related metrics, including Internet connection
  speeds, broadband adoption, mobile usage,
  outages, and cyber-attacks and threats.
• Visitors to www.stateoftheinternet.com can find
  current and archived versions of Akamais State
  of the Internet (Connectivity and Security)
  reports, the companys data visualizations, and
  other resources designed to put context around
  the ever-changing Internet landscape.

10 / The State of the Internet / Security (Q4
2014)