What is a Web Crawler? And How Does It Work?
Kerry Coppinger
|Website Ops & Security | August 04, 2022
Anyone who is on the pulse of the internet has probably heard of a web crawler. They are incredibly common and nearly every website has encountered a web crawler at one point or another, whether the webmaster was aware or not. But what are they exactly? And what purpose do they serve? Depending on who is asked, the concept of a web crawler could be seen as a potential business threat or a positive way to index information. In this article, we break down the myths surrounding web crawlers and dive into how they impact businesses.
What is the definition of a web crawler?
Crawlers are automation tools (bots) that are created to scan through various websites at a quick pace – looking for content and site patterns. Crawlers can sometimes be confused with scrapers, which behave in a similar way but are often looking for a specific piece of data rather than scanning for content.
What are some positive uses for web crawlers?
Many search engines – that companies strive to rank on – utilize crawlers to determine which website content is most relevant. They also look at the way a site is mapped out to determine how a user would navigate throughout the site to find the information they need. These types of crawlers are typically welcomed to a website because they can ultimately contribute to more visibility for the company on search results pages.
What are some negative uses for web crawlers?
On the other hand, malicious users and non-legitimate organizations can also deploy crawlers. Malicious crawlers also scan a website for content, but not with the intention to index it on a results page and help drive site visits. Instead, these bad actors are trying to get an inside look at how other websites operate so that they can copy their content and ultimately steal their customers, stage attacks against their audience, or harm their search visibility.
How can businesses tell the difference between ‘good’ and ‘bad’ web crawlers?
Both ‘good’ and ‘bad’ web crawlers can appear similar and be built in similar ways. Sometimes the way businesses find out whether a crawler arrived on their site for innocuous or malicious purposes is by the actions that take place after the crawling. Are they noticing their content duplicated on another website? Are their products being sold without permission for lower prices on competing websites? If this is the case, there could have been malicious crawlers involved.
Regardless of whether a specific web crawler is helpful or harmful to business though, a web crawler is not a valid human user and therefore cannot convert into a legitimate paying customer. For this reason, it is recommended for businesses to deploy security measures that will filter out crawlers from their analytics, so that their data is not accidentally skewed because of the presence of crawlers.
Want to protect your sites and ads from click fraud? Click here to Request a Demo.
Author
Kerry Coppinger
Senior Manager, Brand Marketing
As head of the brand marketing department at CHEQ, Kerry manages all things content and brand including: research reports, public relations, content marketing, copywriting, press, and other editorial features. Prior to her work at CHEQ, Kerry worked for several tech companies in New York.
