Revealing your Competitor’s FULL External Relevance Profile – One of my best kept secrets

Most SEOs know that winning the battle for top search terms is in the links. They regularly check the back links of top rankings sites to get a clear picture of the “external relevance profile” of those sites. From this they learn:

  1. How many incoming links the site has

  2. The source of such links

  3. The anchor text of those incoming links

If you are familiar with Google Webmaster Central you know that you’ve got almost everything you need when you pull up the external links report, including all the incoming links and link text that Googlebot is aware of. The problem is that this data is available only for your own site, not for competitive analysis of your rivals. Also, most sites competing for top terms have thousands of links, yet competitive link analysis tools are currently limited to just a thousand results.

There are a couple of ways around these problems. I’m going to share one of my most closely guarded SEO tricks… Warning, this requires a little programming!

Option 1: Get access to your competitor’s Google Webmaster Central account by setting up a phishing site, breaking into his/her home, or bribing the technical support staff to place a Google file validating you as the owner.

Okay, obviously that option was a joke. :-) Now, let me explain the real deal. The idea is to use Yahoo’s Site Explorer and Yahoo’s powerful link command on the competitor. In order to pass the one thousand links mark, we need to add a keyword to the link search. For example link:http://sitename.com keyword1, link:http://sitename.com keyword2, etc. Save all the links, concatenate them together and filter the duplicates. You can use the keywords you have already generated in your keyword research analysis for this process.

For example, let’s see how we can extract SEOmoz’s two million links from Yahoo.

1. Extract a list of relevant keywords using Google’s tool

2. Use Yahoo Site Explorer to determine the total number of inbound links (You need to be logged in to Yahoo). Choose “Show Links Except from this domain to Entire Site”

3. Use Yahoo Search link command and add keywords to extract subsets of incoming links. One thousand at a time. Here I do a search adding “seo” and another adding “social media”

Although you can get the full list of links using this trick, the list includes links that are ‘nofollow’ that clearly do not pass any link juice. There are also sitewide links (blogroll links, and links included on every page of other sites), links coming from the same domain, IP block (class C), or owner (from the registrar information), as well as links that are not ‘nofollow,’ but which Google may be preventing from passing link juice. Any of these may increase your link count, but will not necessarily provide the same value as links coming from multiple, unaffiliated sites.

To take things to the next (useful) level, we need first to carefully organize the links by domain, IP block and ownership so that we can identify what I call unique link sources. Second, we need to filter out the links that are ‘nofollow.’ Finally, we need to visit the pages where the links are and see if they pass link juice, as I explained in the post about testing paid links.

Once the process is complete, we can collect all the anchor text coming from those links to get a complete external relevance profile. Now you can see how and why your competitor is listed and use it to form a solid link-building strategy. You have a highly-targeted list of link sources to pursue.

As you probably guessed by now, this is something that is far easier to do with a tool (stay tunned ;-) .)

I was a little bit hesitant to share this, so I would really appreciate if you let me know in the comment if you found this useful.

45 replies
  1. juerg
    juerg says:

    Very useful and interesting stuff–but: How long did it take you – assuming you were usin n o tool to perform this task and – you guessed it- how long does it take you w i t h tool…

    Reply
  2. Hamlet Batista
    Hamlet Batista says:

    juerg – I am too lazy to do this type of stuff manually. Using Yahoo's API and a sizeable pool of web proxies (to overcome Yahoo's API limit) you can do this in several hours.

    I plan to share the tool, but I need to work more on it first.

    Reply
  3. juerg
    juerg says:

    Great. I was hoping you were lazy enough to come up with some code for us to share …. many thanks in advance and looking forward

    Reply
  4. Sam Daams
    Sam Daams says:

    This is VERY useful stuff Hamlet. It also shows that in order to build up a serious site in a seriously competitive field takes work, lots of it. Too many people forget that. If you're going after a competitive field you should expect to be burning a lot of hours building up your site and it's reputation before you see any results! An automatic tool to do a lot of this would be valuable to a lot of serious SEO – you could probably charge for it or maybe license via SEOMOZ (not sure if they do that stuff though)?

    By the way, I just noticed today the 'top posters' area. Is there any reason my name links back to a post on this site or is that a bug?

    Reply
  5. Jez
    Jez says:

    Good post, as I read it I thought it was leading somewhere ;-)

    Do you plan to integrate this with ranksense? (I do not remember this being a feature)

    How scalable do you think a published solution would be, given the requirement for proxies?

    Reply
  6. Hamlet Batista
    Hamlet Batista says:

    Sam, Jez

    This functionality will be part of the advanced edition of RankSense that is in the works.

    I'm working on a free web tool off of the prototype so that you guys have something useful to play with. I plan to use it as a viral marketing piece to promote RankSense ;-)

    BTW: Consider applying to RankSense's private beta if you haven't done so yet

    Reply
  7. Hamlet Batista
    Hamlet Batista says:

    By the way, I just noticed today the ‘top posters’ area. Is there any reason my name links back to a post on this site or is that a bug?

    Sam – I forgot to answer this.

    It seems it is a bug on the plugin. I'll check if there is a new version and see if that fixes it.

    Reply
  8. juerg
    juerg says:

    BTW: Consider applying to RankSense’s private beta if you haven’t done so yet."
    Hi Hamlet
    did apply some time ago, but no reply yet..??
    Can you please look for arranging this. I am on a new and urgent project and would be more than pleased to check this thing out…
    Thank you

    Reply
  9. David Hopkins
    David Hopkins says:

    Moving over to Safari:

    This is certainly a useful method. I have an extra tip to get a few more keywords. I recently found an excellent tool that logs millions of queries on Google.com to allow for some advanced keyword research. The tool (Seodigger.com) allows you to find out what keywords a domain is on the first two pages of Google. You could use this to get some more keywords to get more of the site's incoming links.

    What do you use to stop Yahoo temoprarily banning your IP When harvesting Yahoo data I always pick up the HTTP header code to make sure i've not been blocked. If i've been blocked I use sleep() to stop for a bit. I also sleep 10 seconds between each HTTP request. Do you have a better method

    In ref. to Sam's point that this shows how much work you need to do, if only SEO clients could be made to understand this P. I also think you would benefit much more by releasing the tool for free. Obviously you would only be able to make the script available. Something like this would take your site out P

    What sort of programmers are you looking for If I can be of any use, I know (X)HTML, CSS, DTD, JavaScript, DOM, AJAX, JSON, XML, XSL, XSL-FO, XPath, XLink, XInclude, PHP 4&5, SQL, SVG (basic), XUL & FF extensions building (basic). Also do logo, interface design, icons and have made a number of scrapping tools and search engine widgets. Can also put you in touch with some people who know Perl, Python, C, Java. Some of these are in Poland, so lower payment demand but have no search engine knowledge.

    Reply
  10. Hamlet Batista
    Hamlet Batista says:

    juerg – I will make sure you get it so you can play with it this weekend

    The tool allows you to Seodigger.com find out what keywords a domain is on the first two pages of Google . You could use this to get some more keywords to get more of the site’s incoming links.

    Nice find, David. I am glad you got the blank comments problem sorted out :-)

    <bockquote>
    What do you use to stop Yahoo temoprarily banning your IP When harvesting Yahoo data I always pick up the HTTP header code to make sure i’ve not been blocked. If i’ve been blocked I use sleep() to stop for a bit. I also sleep 10 seconds between each HTTP request. Do you have a better method

    I prefer to use the API instead of scrapping. The API gives you 5k queries per IP. You just need to use multiple proxies ;-)

    What sort of programmers are you looking for

    I can use you skills for client-side development. Please send me an email to discuss further. We prefer to code the server apps in Python/Django.

    RankSense is a Windows application and is coded mostly in C++, but for that development I don't use contractors only internal staff.

    shor – thanks for stopping by and congrats on making the Digg homepage! Very clever post and title ;-)

    The follow up post will make available the tool that does this. Stay tunned!</bockquote>

    Reply
  11. Matt Davies
    Matt Davies says:

    Genius. I'll be keeping my eyes open for a tool, Hamlet. This is the sort of competitor research I'd love to do for my clients, but doing it manually would just be too time consuming.

    Reply
  12. juerg
    juerg says:

    Hamlet, can you please arrange for me to get access to ranksense. Would be delighted to check it out starting this weekend
    Thank you

    Reply
  13. Hamlet Batista
    Hamlet Batista says:

    juerg – sorry for the delay. I sent you an invitation code by email. Please allow for a few hours to get access to the latest build that is being published Today.

    Thanks in advance for your quality feedback.

    Reply
  14. juerg
    juerg says:

    Hamlet
    thank you for your support. Tried to donwload/ access, but was declined, since account already created. Cannot provide another e-mail address…Can you please check and confirm. Hope I didnt miss anything…
    Many thanks
    Juerg

    Reply
  15. juerg
    juerg says:

    Hamlet
    sorry, forgot to mention: Just for your info:Get mailer-daemon failure notice when e-mailing to nemedia for help (help@ranksense.com). Am using latest firefox. Obviously this is not my day. Will have a latte macchiato and wait for your reply…
    Kind regards
    juerg

    Reply
  16. Hamlet Batista
    Hamlet Batista says:

    Juerg – I deleted your existing accounts. Please try again with the invitation code I sent you. I apologize for the inconveniences. Thanks for alerting me about the email address issue. That address goes to several emails accounts (including mine). I guess one of them is incorrect.

    Vingold – thanks. I am glad you liked it.

    Reply
  17. SEONewbie
    SEONewbie says:

    This is really interesting. I'm looking for ways to stand out at my company and doing this may justify a much wanted raise.

    I do wish there was a way I could tell if those websites that linked to a competitor were Affiliates. Any suggestions?

    **hoping a free tool is available shortly**

    Reply
  18. Hamlet Batista
    Hamlet Batista says:

    SEONewbie – one way to tell if the backlinks are from affiliates is to check for HTTP redirects and cookies. I'll add this to the next update to the tool.

    Amanda – we are giving the finishing touches to the tool :-) Thanks for your patience.

    Reply
  19. Vishal
    Vishal says:

    very very informaive post. i will be implementing your directions. perhaps even code a system so that this can be shared without the complications… so anyone can come to my site page and insert the data and see the result, instead of repeating the tedious procedure for each domain / keyword. thanks!

    Reply
  20. Seo Design Solutions
    Seo Design Solutions says:

    Hey Hamlet:

    I know what you mean about sharing the good stuff. I have been giving away my techniques for months now for competitive link building and the like, but this is great information for link hunting. Good stuff…By the way, I hope you win an award for the recent nomination for best SEO Research Blog. Thanks for the tips.

    Reply
  21. Manual Directory Sub
    Manual Directory Sub says:

    Hello Hamlet,

    Can i get invitation code for ranksense as well.

    I would certainly be interested in the RankSense. Can you join me in your email newsletter so that I will be able to know when you will launch future version of RankSense.

    Thanks
    Padhamanabha

    Reply

Trackbacks & Pingbacks

  1. […] Revealing your Competitor’s FULL External Relevance Profile – Hamlet Batista […]

  2. […] Revealing your Competitor’s FULL External Relevance Profile – One of my best kept secrets Hamlet Batista | 10/17/07 […]

  3. […] Revealing your Competitor’s FULL External Relevance Profile – One of my best kept secrets Hamlet Batista | 10/17/07 […]

  4. […] Revealing your Competitor’s FULL External Relevance Profile – One of my best kept secrets Hamlet Batista | 10/17/07 […]

  5. […] profile for them. Hamlet Batista unleashed something that you will not find in any ebook with Revealing your Competitor’s FULL External Relevance Profile. Also, Earner’s Blog continues his series on link building with a good reminder to think […]

  6. […] Revealing your Competitor’s FULL External Relevance Profile – One of my best kept secrets […]

  7. […] Check it out! While looking through the blogosphere we stumbled on an interesting post today.Here’s a quick excerpt…One of my best kept secrets at Hamlet Batista dot Com using Google suggest per site and Yahoo Site Explorer and some data crunching and sorting. […]

  8. […] Sphinns – Revealing your Competitor’s FULL External Relevance Profile – One of my best kept secrets at Ham… –  Ever want to know step by step how to run a competitive analysis?  Now you do.  The only […]

Leave a Reply

Want to join the discussion?
Feel free to contribute!

Leave a Reply