A crawler is also referred to as a robot or bot from a search engine. A crawler reads the content from a page and usually stores this in a search engines database, called an index. From this index the search results are extracted when one has entered a keyword in a search engine.
There are also other crawlers, like the Wayback-machines crawler, that just stores the content(also images) from a website to create a record of the internet. www.waybackmachine.org is usefull for seeing how websites have evolved through time.
Many other type of crawlers also scour the internet on a frequent basis.

Every search engines have its own crawlers. and Wayback-machines is very much helpful to get the previous version of the websites and the page age.

There was code for a simple crawler in my PERL programming book.