Joomla download guide  

Websites In Joomla, Websites With Joomla
By Michelle Crossley
Crawling and Spiders – oh my!
The internet is one medium everybody wants to use, to promote their business by reaching out to millions of worldwide Internet users. But how do the major search engines like Google, Yahoo! and MSN ‘know’ about your websites and feature them on their results? Well, simply stated, it is a ‘Spider’ which will help navigate web pages and determines the site’s efficacy as a search engine.
How does the Spider ‘crawl’ the web?

Alan Sparks, software developer and internet enthusiast has stated that the term “spider” arose because the internet is also called ‘the web’ and software programs like Googlebot navigate the web pages and websites that comprise the web. Since the program navigates around the web, the term spider arose. Since spiders don’t really walk per se – they crawl, the term ‘crawlers’ came about.

The usual starting points for spiders are lists of heavily used servers or extremely popular web pages. When it encounters a popular site, the spider will index the words on its page and follow every link within the site.

How does Googlebot work?
As Googlebot begins to visit and “crawl” through your website, the software draws up a snapshot of the website and the web pages on that particular site. That snapshot or “memory” of the website and its individual web pages is cached or “filed”. The information which has been cached is then added to the ‘index’, Google’s memory bank. Whenever a visitor searches for a certain term, Google will look up its memory bank for web pages that will fit the bill. Googlebot will revisit the websites in its index from time to time, forming new snapshots, which are then added to the index.

Site Structure to help the Search Engine Spider Find You
Sitemaps: Make sure that

you have a Sitemap uploaded to the poplular search engines, with links to all your pages. This will serve as a general map of the web site for the spider.

Custom 404 Page: Your customized 404 page should fit the design of your web page and most importantly include a link to important pages on your site, especially the home page. This allows the spider to access important pages even if lands on a page that is no longer on your site or a broken link.

Important pages should stay in the root directory: Don’t make the spider dig deep to find the important pages of your website.

Navigation Structure Which Appeals to the Search Engine Spider
• Keep navigation simple and easy to understand
• Place a link to your home page on all the pages of your website
• Use text links as opposed to image links
• Cross link important pages of your website

• Retain broken links on your webpge
• Include a huge list of keywords
• Use JavaScript or image maps as the search engine spider cannot follow them

You can read more about search engine spiders and spider friendly web page design at SearchEngineWatch and SearchEngineWorld
Michelle Crossley – Creative Services Manager of STUDIO1C (, an award-winning art director, graphic designer, illustrator and photographer, and an Adobe Photoshop and Illustrator guru. Michelle worked for the Detroit Newspapers, third largest in the country as one of their top Designers and was often selected from the pool of more than one hundred peer designers to work on unique, high-end projects.


Copyright 2009 by, All Rights Reserved