Controlling Your Robots: Using the X-Robots-Tag HTTP header with Googlebot

robopet.jpgWe have discussed before how to control Googlebot via robots.txt and meta robot tags. Both methods have limitations. With robots.txt you can block the crawling of any page or directory, but you cannot control the indexing, caching or snippets. With the robots meta tag you can control crawling, caching and snippets but you can only do that for HTML files, as the tag is embedded in the files themselves. You have no granular control for binary and non-HTML files.

Until now. Google recently introduced another clever solution to this problem. You can now specify robot meta tags via an HTTP header. The new header is the X-Robots-Tag, and it behaves and supports the same directives as the regular robots meta tag: index/noindex, archive/noarchive, snippet/nosnippet and the new unavailable_after directive. This new technique makes it possible to have granular control over crawling, caching, and other functions for any page on your website, no matter the type of content it has—PDF, Word doc, Excel file, zip files, etc. Read more

The Ranking Triathlon: How to overcome crawling, indexing, and searching hurdles

felizsanchez.jpgI frequently get asked why a particular page is no longer ranking. I wish there were a simple answer to that question. Instead of giving personal responses, I’ve decided to write a detailed post with the possible problems that might cause your ranking to drop, as well as all the solutions I could think of. I also want to present a case study every week of a ranking that dropped and what we did to get it back. If you have a site that is affected I invite you to participate. Send me an email or leave a comment.

There are many reasons why your page or website might not be ranking. Let's go through each of the three steps in the search engine ranking process and examine the potential roadblocks your page might face. We’ll see how to avoid them, how to identify if your page was affected, and most importantly, how to recover. Read more

Content is King, but Duplicate Content is a Royal Pain.

painkiller.jpgDuplicate content is one of the most common causes of concern among webmasters. We work hard to provide original and useful content, and all it takes is a malicious SERP (Search Engine Results Page) hijacker to copy our content and use it for his or her own. Not nice.

More troubling still is the way that Google handles the issue. In my previous post about cgi hijacking, was clear that the main problem with hijacking and content scraping is that search engines do not reliably determine who is the owner of the content and, therefore, which page should stay in the index. When faced with multiple pages that have exactly the same or nearly the same content, Google's filters flag them as duplicates. Google's usual course of action is that only one of the pages — the one with the higher PageRank — makes it to the index. The rest are tossed out. Unless there is enough evidence to show that the owner or owners are trying to do something manipulative, there is no need to worry about penalties.

Recently, regular reader Jez asked me a thought-provoking question. I'm paraphrasing here, but essentially he wanted to know: "Why doesn’t Google consider the age of the content to determine the original author?” I responded that the task is not as trivial as it may seem at first, and I promised a more thorough explanation. Here it is. Read more

You’ve Won the Battle but not the War: 10 Ways to protect your site from negative SEO

rankings_robber2.jpgLast month there was an interesting article in Forbes about the search engine marketing saboteurs. These so-called “SEO professionals” proudly proclaim their job to be damaging the hard-earned rankings of their clients’ competitors. I understand a lot of people would do anything for money, but it’s still unsettling to see such people trumpet their efforts with such gusto. A huge thumbs down to all those mentioned in the article.

Earning high search engine rankings is challenging enough. Now we need to work twice as hard to protect the rankings once we earn them. The Forbes article lists seven ways you can damage someone else's website. I can think of three more — but instead of adding more wood to the negative SEO fire, I’ve decided to create a list of things you can do to detect, prevent and protect your rankings from these types of attacks.

Here are Hamlet’s countermeasures. (You may want to read the Forbes article first to better understand the terms.) Read more

Becoming a Web Authority: The story of Sally and Edward

authority.jpgGetting permanent search engine rankings for your site requires making it very popular within your specific niche. This is what we call a web authority. Other site owners will naturally link to your site when they are talking about your topic because you have some of the best content out there. 

It's common sense that if you want your site to become popular, you'll seek advice from those who have already made it. That's assuming they are willing to share how they did it. I personally like to read Neil Patel's blog. He is one of the few SEO celebrities who is open to share how he made it to the top.
Most advanced SEOs carefully study high-ranking pages for the keywords they are targeting. If you can understand how those sites got there, you can apply the same techniques to your site. Unfortunately, as I explained previously, not all sites ranking high will remain there. These sites are like a house of cards, destined to be brought down by the next fair wind. If you apply the strategies of such cardboard castles, yours will come down too. Let me illustrate my point with this story about Sally and Edward.
Read more

At last! A Rock-solid Strategy to Help you Reach High Search Engine Rankings

rock.jpgEvery SEO strategy has its good and bad

Search engines have long kept their ranking formulas secret in order to give users the most relevant results and to preclude malicious manipulation. Despite all the secrecy, though, it remains extremely difficult for search engine companies to prevent either direct or indirect manipulation.

We can broadly classify search engine optimization (SEO) strategies into two camps: passive and active. Passive optimization occurs automatically by search engine robots, which scour the web finding and categorizing relevant sites. Most websites listed on the search engine result page (SERP) fall into this category. If it were up to search engine companies like Google and Yahoo, this would be the only type of optimization that existed.

Active optimization takes place when website owners purposefully engage in activities designed to improve their sites’ search engine visibility. This is the kind of search engine optimization we normally talk about when we think of SEO.

Let’s go one step further and classify these deliberate forms of optimization based upon the tactics employed. In general these are: basic principles, exploiting flaws, algorithm chasing, and competitive intelligence. Read more