Controlling Your Robots: Using the X-Robots-Tag HTTP header with Googlebot
By Hamlet Batista at August 1, 2007 | 6:44 pm | 10 Comments
We have discussed before how to control Googlebot via robots.txt and meta robot tags. Both methods have limitations. With robots.txt you can block the crawling of any page or directory, but you cannot control the indexing, caching or snippets. With the robots meta tag you can control crawling, caching and snippets but you can only
Categotries : Blog , On-page SEO , Technical SEO
Anatomy of a Distributed Web Spider — Google's inner workings part 3
By Hamlet Batista at July 6, 2007 | 2:55 pm | 5 Comments
What can you do to make life easier for those search engine crawlers? Let's pick up where we left off in our inner workings of Google series. I am going to give a brief overview of how distributed crawling works. This topic is useful, but can be a bit geeky, so I'm going to offer a
Categotries : Blog
