How Do Search Engine Spiders Work?

Knowing how your website appears to search engine spiders, or robots, is important since how your page is ranked depends on how the spiders see it, not your human users. Though the ranking algorithms vary among the different search engines, their spiders work basically the same way. They crawl the sites and read the text, ignoring such things as JavaScript and Flash that are there to entertain and catch the eye of humans.

About the Spiderview Crawl Tool

The free Spiderview simulation tool from LinkVendor will crawl your website to show you the text the search engine spiders see without JavaScript, comments and styles. It will show you the header data as spiders view it, the number of words, meta keywords and description length, HTTP status and redirect notice, internal links,and external links, You can opt to view the source code of the URL entered, HTTPS links, and No-Follow links.

Enter your domain to see how the search engines view your page

This tool organizes analyzed data more clearly than most spider simulators and also provides link anchor text that many others don’t. It is especially helpful that it lists internal and external links into two separate lists for quick analysis.

A view of how the spiderview crawl test will output its results

Breakdown of the essentials on page seo factors, misc stats needed for SEO, a list of internal links on the current page

The number of links linking out to other pages