Revealing your Competitor’s FULL External Relevance Profile – One of my best kept secrets
October 17th, 2007 @ Hamlet Batista // 45 Comments
Most SEOs know that winning the battle for top search terms is in the links. They regularly check the back links of top rankings sites to get a clear picture of the “external relevance profile” of those sites. From this they learn:
-
How many incoming links the site has
-
The source of such links
-
The anchor text of those incoming links
If you are familiar with Google Webmaster Central you know that you’ve got almost everything you need when you pull up the external links report, including all the incoming links and link text that Googlebot is aware of. The problem is that this data is available only for your own site, not for competitive analysis of your rivals. Also, most sites competing for top terms have thousands of links, yet competitive link analysis tools are currently limited to just a thousand results.
There are a couple of ways around these problems. I’m going to share one of my most closely guarded SEO tricks… Warning, this requires a little programming!
Option 1: Get access to your competitor’s Google Webmaster Central account by setting up a phishing site, breaking into his/her home, or bribing the technical support staff to place a Google file validating you as the owner.
Okay, obviously that option was a joke. 
Now, let me explain the real deal. The idea is to use Yahoo’s Site Explorer and Yahoo’s powerful link command on the competitor. In order to pass the one thousand links mark, we need to add a keyword to the link search. For example link:http://sitename.com keyword1, link:http://sitename.com keyword2, etc. Save all the links, concatenate them together and filter the duplicates. You can use the keywords you have already generated in your keyword research analysis for this process.
For example, let’s see how we can extract SEOmoz’s two million links from Yahoo.
1. Extract a list of relevant keywords using Google’s tool
2. Use Yahoo Site Explorer to determine the total number of inbound links (You need to be logged in to Yahoo). Choose “Show Links Except from this domain to Entire Site”
3. Use Yahoo Search link command and add keywords to extract subsets of incoming links. One thousand at a time. Here I do a search adding “seo” and another adding “social media”
Although you can get the full list of links using this trick, the list includes links that are ‘nofollow’ that clearly do not pass any link juice. There are also sitewide links (blogroll links, and links included on every page of other sites), links coming from the same domain, IP block (class C), or owner (from the registrar information), as well as links that are not ‘nofollow,’ but which Google may be preventing from passing link juice. Any of these may increase your link count, but will not necessarily provide the same value as links coming from multiple, unaffiliated sites.
To take things to the next (useful) level, we need first to carefully organize the links by domain, IP block and ownership so that we can identify what I call unique link sources. Second, we need to filter out the links that are ‘nofollow.’ Finally, we need to visit the pages where the links are and see if they pass link juice, as I explained in the post about testing paid links.
Once the process is complete, we can collect all the anchor text coming from those links to get a complete external relevance profile. Now you can see how and why your competitor is listed and use it to form a solid link-building strategy. You have a highly-targeted list of link sources to pursue.
As you probably guessed by now, this is something that is far easier to do with a tool (stay tunned 
.)
I was a little bit hesitant to share this, so I would really appreciate if you let me know in the comment if you found this useful.
Category : Blog
Jeremy
5 years ago
This is very useful. PLEASE…Tell me more.
juerg
5 years ago
Very useful and interesting stuff–but: How long did it take you – assuming you were usin n o tool to perform this task and – you guessed it- how long does it take you w i t h tool…
Hamlet Batista
5 years ago
juerg – I am too lazy to do this type of stuff manually. Using Yahoo's API and a sizeable pool of web proxies (to overcome Yahoo's API limit) you can do this in several hours.
I plan to share the tool, but I need to work more on it first.
juerg
5 years ago
Great. I was hoping you were lazy enough to come up with some code for us to share …. many thanks in advance and looking forward
» Today’s resources posts SEOisEZ.com: Blogging the Best of SEO
5 years ago
[...] Sphinns – Revealing your Competitor’s FULL External Relevance Profile – One of my best kept secrets at Ham… - Ever want to know step by step how to run a competitive analysis? Now you do. The only [...]
Sam Daams
5 years ago
This is VERY useful stuff Hamlet. It also shows that in order to build up a serious site in a seriously competitive field takes work, lots of it. Too many people forget that. If you're going after a competitive field you should expect to be burning a lot of hours building up your site and it's reputation before you see any results! An automatic tool to do a lot of this would be valuable to a lot of serious SEO – you could probably charge for it or maybe license via SEOMOZ (not sure if they do that stuff though)?
By the way, I just noticed today the 'top posters' area. Is there any reason my name links back to a post on this site or is that a bug?
Jez
5 years ago
Good post, as I read it I thought it was leading somewhere
Do you plan to integrate this with ranksense? (I do not remember this being a feature)
How scalable do you think a published solution would be, given the requirement for proxies?
Web Design Newcastle
5 years ago
Nice tip there. I never knew you could add keywords to the Link: search.
Hamlet Batista
5 years ago
Sam, Jez
This functionality will be part of the advanced edition of RankSense that is in the works.
I'm working on a free web tool off of the prototype so that you guys have something useful to play with. I plan to use it as a viral marketing piece to promote RankSense
BTW: Consider applying to RankSense's private beta if you haven't done so yet
5ubliminal
5 years ago
If I feel generous, which I usually don't
, I might release a software quite soon to do what you talk about here including some detailed reports.
Maybe …
Hamlet Batista
5 years ago
Hey 5ubliminal! Do you want to make some extra bucks? Send me an email.
I am always looking for help and you seem to be a very talented programmer.
Hamlet Batista
5 years ago
Sam – I forgot to answer this.
It seems it is a bug on the plugin. I'll check if there is a new version and see if that fixes it.
juerg
5 years ago
BTW: Consider applying to RankSense’s private beta if you haven’t done so yet."
Hi Hamlet
did apply some time ago, but no reply yet..??
Can you please look for arranging this. I am on a new and urgent project and would be more than pleased to check this thing out…
Thank you
David Hopkins
5 years ago
Moving over to Safari:
This is certainly a useful method. I have an extra tip to get a few more keywords. I recently found an excellent tool that logs millions of queries on Google.com to allow for some advanced keyword research. The tool (Seodigger.com) allows you to find out what keywords a domain is on the first two pages of Google. You could use this to get some more keywords to get more of the site's incoming links.
What do you use to stop Yahoo temoprarily banning your IP When harvesting Yahoo data I always pick up the HTTP header code to make sure i've not been blocked. If i've been blocked I use sleep() to stop for a bit. I also sleep 10 seconds between each HTTP request. Do you have a better method
In ref. to Sam's point that this shows how much work you need to do, if only SEO clients could be made to understand this P. I also think you would benefit much more by releasing the tool for free. Obviously you would only be able to make the script available. Something like this would take your site out P
What sort of programmers are you looking for If I can be of any use, I know (X)HTML, CSS, DTD, JavaScript, DOM, AJAX, JSON, XML, XSL, XSL-FO, XPath, XLink, XInclude, PHP 4&5, SQL, SVG (basic), XUL & FF extensions building (basic). Also do logo, interface design, icons and have made a number of scrapping tools and search engine widgets. Can also put you in touch with some people who know Perl, Python, C, Java. Some of these are in Poland, so lower payment demand but have no search engine knowledge.
shor
5 years ago
Another gem of a post Hamlet.
Waiting for your post on the automation of this technique!
Hamlet Batista
5 years ago
juerg – I will make sure you get it so you can play with it this weekend
Nice find, David. I am glad you got the blank comments problem sorted out
<bockquote>
What do you use to stop Yahoo temoprarily banning your IP When harvesting Yahoo data I always pick up the HTTP header code to make sure i’ve not been blocked. If i’ve been blocked I use sleep() to stop for a bit. I also sleep 10 seconds between each HTTP request. Do you have a better method
I prefer to use the API instead of scrapping. The API gives you 5k queries per IP. You just need to use multiple proxies
I can use you skills for client-side development. Please send me an email to discuss further. We prefer to code the server apps in Python/Django.
RankSense is a Windows application and is coded mostly in C++, but for that development I don't use contractors only internal staff.
shor – thanks for stopping by and congrats on making the Digg homepage! Very clever post and title
The follow up post will make available the tool that does this. Stay tunned!</bockquote>
Hugo Peppers
5 years ago
I'm on the edge of my seat waiting for this tool.
Matt Davies
5 years ago
Genius. I'll be keeping my eyes open for a tool, Hamlet. This is the sort of competitor research I'd love to do for my clients, but doing it manually would just be too time consuming.
Hamlet Batista
5 years ago
Matt, Hugo thanks. I am glad there is interest in such tool. We already started working on it. It will take about a week to complete and make available.
juerg
5 years ago
Hamlet, can you please arrange for me to get access to ranksense. Would be delighted to check it out starting this weekend
Thank you
5 Almost Popular Sphunn Articles That Didn’t Go Hot This Week | SEO News - All The SEO Scoop
5 years ago
[...] Revealing your Competitor’s FULL External Relevance Profile [...]
Hamlet Batista
5 years ago
juerg – sorry for the delay. I sent you an invitation code by email. Please allow for a few hours to get access to the latest build that is being published Today.
Thanks in advance for your quality feedback.
My Ghillie » Revealing your Competitor’s FULL External Relevance Profile
5 years ago
[...] Check it out! While looking through the blogosphere we stumbled on an interesting post today.Here’s a quick excerpt…One of my best kept secrets at Hamlet Batista dot Com using Google suggest per site and Yahoo Site Explorer and some data crunching and sorting. [...]
juerg
5 years ago
Hamlet
thank you for your support. Tried to donwload/ access, but was declined, since account already created. Cannot provide another e-mail address…Can you please check and confirm. Hope I didnt miss anything…
Many thanks
Juerg
juerg
5 years ago
Hamlet
sorry, forgot to mention: Just for your info:Get mailer-daemon failure notice when e-mailing to nemedia for help ([email protected]). Am using latest firefox. Obviously this is not my day. Will have a latte macchiato and wait for your reply…
Kind regards
juerg
vingold
5 years ago
Exceptional post!
I need to print it out and give it a try when I have a few free hours.
I look forward to more!
This was really good.
Hamlet Batista
5 years ago
Juerg – I deleted your existing accounts. Please try again with the invitation code I sent you. I apologize for the inconveniences. Thanks for alerting me about the email address issue. That address goes to several emails accounts (including mine). I guess one of them is incorrect.
Vingold – thanks. I am glad you liked it.
john
5 years ago
I'd be interested in the tool as well. Sounds like an edge up on competition.
SEONewbie
5 years ago
This is really interesting. I'm looking for ways to stand out at my company and doing this may justify a much wanted raise.
I do wish there was a way I could tell if those websites that linked to a competitor were Affiliates. Any suggestions?
**hoping a free tool is available shortly**
Amanda
5 years ago
I can't wait to see what you come up with and test it out. Don't make us wait too long!
Hamlet Batista
5 years ago
SEONewbie – one way to tell if the backlinks are from affiliates is to check for HTTP redirects and cookies. I'll add this to the next update to the tool.
Amanda – we are giving the finishing touches to the tool
Thanks for your patience.
Vishal
5 years ago
very very informaive post. i will be implementing your directions. perhaps even code a system so that this can be shared without the complications… so anyone can come to my site page and insert the data and see the result, instead of repeating the tedious procedure for each domain / keyword. thanks!
Advanced SEO Tips From Hamlet Batista | John Chow dot Com
5 years ago
[...] Revealing your Competitor’s FULL External Relevance Profile – One of my best kept secrets [...]
Nikhil
5 years ago
Hello Hamlet,
Can i get invitation code for ranksense as well . I have already submitted there my email.?
Thanks
Nikhil
Hamlet Batista
5 years ago
Nikhil – We're sending out the rest of the invites this week. Stay tuned. Thanks for your interest.
Seo Design Solutions
5 years ago
Hey Hamlet:
I know what you mean about sharing the good stuff. I have been giving away my techniques for months now for competitive link building and the like, but this is great information for link hunting. Good stuff…By the way, I hope you win an award for the recent nomination for best SEO Research Blog. Thanks for the tips.
Prime Angus Filets: Links this Week 1-4-2008 | bobangus.com
5 years ago
[...] profile for them. Hamlet Batista unleashed something that you will not find in any ebook with Revealing your Competitor’s FULL External Relevance Profile. Also, Earner’s Blog continues his series on link building with a good reminder to think [...]
Link Building - All 2008 Nominees » SEMMYS.org
5 years ago
[...] Revealing your Competitor’s FULL External Relevance Profile – One of my best kept secrets Hamlet Batista | 10/17/07 [...]
Link Building - 2008 Finalists » SEMMYS.org
5 years ago
[...] Revealing your Competitor’s FULL External Relevance Profile – One of my best kept secrets Hamlet Batista | 10/17/07 [...]
losangelesseo
5 years ago
is it possible to query the site with multiple keywords at once? like link://www.site.com keyword1,keyword2…
Link Building - 2008 Winner » SEMMYS.org
5 years ago
[...] Revealing your Competitor’s FULL External Relevance Profile – One of my best kept secrets Hamlet Batista | 10/17/07 [...]
The SEMMYS - Hobo SEO UK
5 years ago
[...] Revealing your Competitor’s FULL External Relevance Profile – Hamlet Batista [...]
Manual Directory Sub
5 years ago
Hello Hamlet,
Can i get invitation code for ranksense as well.
I would certainly be interested in the RankSense. Can you join me in your email newsletter so that I will be able to know when you will launch future version of RankSense.
Thanks
Padhamanabha
Beth
5 years ago
Hi Hamlet,
Could I also receive an invitation to check out your beta Ranksense?
Thanks,
Beth
Aaron Hall
4 years ago
This is very useful. Although I knew about Yahoo's tools, your article helps to effectively apply those tools using a practical step-by-step method.