Comments on: Advanced Cloaking Technique: How to feed password-protected content to search engine spiders http://hamletbatista.com/2007/09/03/advanced-cloaking-technique-how-to-feed-password-protected-content-to-search-engine-spiders/ Advanced Search Engine Marketing Tips to Succeed Online Tue, 18 Nov 2008 14:58:33 +0000 http://wordpress.org/?v=2.6.3 By: Продвинутый клоакинг: как роботы индексируют платный контент. http://hamletbatista.com/2007/09/03/advanced-cloaking-technique-how-to-feed-password-protected-content-to-search-engine-spiders/#comment-3486 Продвинутый клоакинг: как роботы индексируют платный контент. Fri, 19 Oct 2007 22:10:05 +0000 http://hamletbatista.com/2007/09/03/advanced-cloaking-technique-how-to-feed-password-protected-content-to-search-engine-spiders/#comment-3486 [...] Оригинал статьи: Advanced Cloaking Technique: How to feed password-protected content to search engine spiders [...] [...] Оригинал статьи: Advanced Cloaking Technique: How to feed password-protected content to search engine spiders [...]

]]> By: Hamlet Batista http://hamletbatista.com/2007/09/03/advanced-cloaking-technique-how-to-feed-password-protected-content-to-search-engine-spiders/#comment-1568 Hamlet Batista Thu, 06 Sep 2007 14:37:12 +0000 http://hamletbatista.com/2007/09/03/advanced-cloaking-technique-how-to-feed-password-protected-content-to-search-engine-spiders/#comment-1568 David, if you want I can change your site name to your real name in all your comments. David, if you want I can change your site name to your real name in all your comments.

]]> By: Hamlet Batista http://hamletbatista.com/2007/09/03/advanced-cloaking-technique-how-to-feed-password-protected-content-to-search-engine-spiders/#comment-1488 Hamlet Batista Wed, 05 Sep 2007 00:01:30 +0000 http://hamletbatista.com/2007/09/03/advanced-cloaking-technique-how-to-feed-password-protected-content-to-search-engine-spiders/#comment-1488 egorych - Yes, I noticed that. I used Google translate to understand it. Thanks for the translation. I though that was some new kind of automated scrapping. egorych - Yes, I noticed that. I used Google translate to understand it. Thanks for the translation. I though that was some new kind of automated scrapping.

]]> By: egorych http://hamletbatista.com/2007/09/03/advanced-cloaking-technique-how-to-feed-password-protected-content-to-search-engine-spiders/#comment-1477 egorych Tue, 04 Sep 2007 18:58:35 +0000 http://hamletbatista.com/2007/09/03/advanced-cloaking-technique-how-to-feed-password-protected-content-to-search-engine-spiders/#comment-1477 Very interesting, really. I've translated your article into Russian. Good job. Very interesting, really. I’ve translated your article into Russian. Good job.

]]> By: David Hopkins http://hamletbatista.com/2007/09/03/advanced-cloaking-technique-how-to-feed-password-protected-content-to-search-engine-spiders/#comment-1475 David Hopkins Tue, 04 Sep 2007 17:40:36 +0000 http://hamletbatista.com/2007/09/03/advanced-cloaking-technique-how-to-feed-password-protected-content-to-search-engine-spiders/#comment-1475 Unfortunatly they don't correlate on your top commenters. :( Unfortunatly they don’t correlate on your top commenters. :(

]]> By: Hamlet Batista http://hamletbatista.com/2007/09/03/advanced-cloaking-technique-how-to-feed-password-protected-content-to-search-engine-spiders/#comment-1443 Hamlet Batista Mon, 03 Sep 2007 23:55:05 +0000 http://hamletbatista.com/2007/09/03/advanced-cloaking-technique-how-to-feed-password-protected-content-to-search-engine-spiders/#comment-1443 David, aka Mutiny, glad to see you using your real name. <blockquote> I think you being a programmer (I’m sure you said Perl was your favourite?) is an important attribute of your SEO knowledge. Most SEOs don’t seem to have any programming knowledge. They are quite happy to talk about 301 redirects, but how many actually know what a HTTP header is?</blockquote> My favorite is Python. You could say that I have an unfair advantage as many things in SEO are highly technical. I still need to play catchup with the Marketing aspect, though. <blockquote>I have no idea what to do with them though. </blockquote> Although I've bought domains in the past, I think that whole domaining thing is a little bit overrated. I prefer to buy sites where I can see profit potential before hand. Before expending a dime, make sure you have a plan to make it back. ;-) David, aka Mutiny, glad to see you using your real name.

I think you being a programmer (I’m sure you said Perl was your favourite?) is an important attribute of your SEO knowledge. Most SEOs don’t seem to have any programming knowledge. They are quite happy to talk about 301 redirects, but how many actually know what a HTTP header is?

My favorite is Python. You could say that I have an unfair advantage as many things in SEO are highly technical. I still need to play catchup with the Marketing aspect, though.

I have no idea what to do with them though.

Although I’ve bought domains in the past, I think that whole domaining thing is a little bit overrated. I prefer to buy sites where I can see profit potential before hand.

Before expending a dime, make sure you have a plan to make it back. ;-)

]]> By: David Hopkins http://hamletbatista.com/2007/09/03/advanced-cloaking-technique-how-to-feed-password-protected-content-to-search-engine-spiders/#comment-1434 David Hopkins Mon, 03 Sep 2007 19:13:46 +0000 http://hamletbatista.com/2007/09/03/advanced-cloaking-technique-how-to-feed-password-protected-content-to-search-engine-spiders/#comment-1434 Again, any hope of being a smart ass were dashed as I read further through the article. I think you being a programmer (I’m sure you said Perl was your favourite?) is an important attribute of your SEO knowledge. Most SEOs don’t seem to have any programming knowledge. They are quite happy to talk about 301 redirects, but how many actually know what a HTTP header is? They have a pretty bullet-proof setup over there at SEOMoz. I also had a sniff around to find it watertight. I have had quite a few adventures with the curl libraries – all strictly legitimate I should add. The only flaw that comes to mind is the possibility of using proxies. I am not sure what you are doing at stage 2? Are you checking against a list? As you may have guessed, that list I sent over was the product of curl – I am adding to it from further sources at the moment. As for my idea for scoping out domains that you can buy for a few dollars its been pretty unsuccessful. I got so excited at seeing a domain with 20,000 links available for I bought it without hesitation only to find out that the links came from a handful of domains. Thankfully it only cost $9. Although I’ve since picked up a genuine PR5 domain for $9 and a three letter .com for $150. I have no idea what to do with them though. P.S. I’m impressed with RankSense. Like yourself I’m really busy and hope to get a better look at it later. I’ll give you some linkage when the time is right. Again, any hope of being a smart ass were dashed as I read further through the article. I think you being a programmer (I’m sure you said Perl was your favourite?) is an important attribute of your SEO knowledge. Most SEOs don’t seem to have any programming knowledge. They are quite happy to talk about 301 redirects, but how many actually know what a HTTP header is?

They have a pretty bullet-proof setup over there at SEOMoz. I also had a sniff around to find it watertight.

I have had quite a few adventures with the curl libraries – all strictly legitimate I should add. The only flaw that comes to mind is the possibility of using proxies. I am not sure what you are doing at stage 2? Are you checking against a list? As you may have guessed, that list I sent over was the product of curl – I am adding to it from further sources at the moment. As for my idea for scoping out domains that you can buy for a few dollars its been pretty unsuccessful. I got so excited at seeing a domain with 20,000 links available for I bought it without hesitation only to find out that the links came from a handful of domains. Thankfully it only cost $9. Although I’ve since picked up a genuine PR5 domain for $9 and a three letter .com for $150. I have no idea what to do with them though.

P.S. I’m impressed with RankSense. Like yourself I’m really busy and hope to get a better look at it later. I’ll give you some linkage when the time is right.

]]> By: MB Web Design http://hamletbatista.com/2007/09/03/advanced-cloaking-technique-how-to-feed-password-protected-content-to-search-engine-spiders/#comment-1430 MB Web Design Mon, 03 Sep 2007 17:31:41 +0000 http://hamletbatista.com/2007/09/03/advanced-cloaking-technique-how-to-feed-password-protected-content-to-search-engine-spiders/#comment-1430 Nice try - just give those good people at SEOmoz your money :p Nice try - just give those good people at SEOmoz your money :p

]]> By: Hamlet Batista http://hamletbatista.com/2007/09/03/advanced-cloaking-technique-how-to-feed-password-protected-content-to-search-engine-spiders/#comment-1428 Hamlet Batista Mon, 03 Sep 2007 15:31:00 +0000 http://hamletbatista.com/2007/09/03/advanced-cloaking-technique-how-to-feed-password-protected-content-to-search-engine-spiders/#comment-1428 Jez - With my updated strategy you don't need to maintain IP lists. Please read it again carefully. You simply check the user agent and if it is XXBot you do a forward-reverse DNS lookup to confirm it is indeed such bot. No need to check list of IPs. Well, maybe to improve performance you'd want to cache the IPs of confirmed bots ;-) Jez - With my updated strategy you don’t need to maintain IP lists. Please read it again carefully.

You simply check the user agent and if it is XXBot you do a forward-reverse DNS lookup to confirm it is indeed such bot. No need to check list of IPs. Well, maybe to improve performance you’d want to cache the IPs of confirmed bots ;-)

]]> By: Hamlet Batista http://hamletbatista.com/2007/09/03/advanced-cloaking-technique-how-to-feed-password-protected-content-to-search-engine-spiders/#comment-1427 Hamlet Batista Mon, 03 Sep 2007 15:27:20 +0000 http://hamletbatista.com/2007/09/03/advanced-cloaking-technique-how-to-feed-password-protected-content-to-search-engine-spiders/#comment-1427 Rand - Thanks for your comment. I was about to say the same. I checked the cached versions of your premium article pages and they ask for credentials. Rand - Thanks for your comment. I was about to say the same. I checked the cached versions of your premium article pages and they ask for credentials.

]]> By: randfish http://hamletbatista.com/2007/09/03/advanced-cloaking-technique-how-to-feed-password-protected-content-to-search-engine-spiders/#comment-1425 randfish Mon, 03 Sep 2007 15:05:54 +0000 http://hamletbatista.com/2007/09/03/advanced-cloaking-technique-how-to-feed-password-protected-content-to-search-engine-spiders/#comment-1425 Jez - actually, we don't cloak. Neither search engines, nor humans have access to the premium content unless they're logged into a premium account. Instead, we show non-logged-in users and bots a page with a summary of the content and an outline (for the guides). Jez - actually, we don’t cloak. Neither search engines, nor humans have access to the premium content unless they’re logged into a premium account. Instead, we show non-logged-in users and bots a page with a summary of the content and an outline (for the guides).

]]> By: Jez http://hamletbatista.com/2007/09/03/advanced-cloaking-technique-how-to-feed-password-protected-content-to-search-engine-spiders/#comment-1414 Jez Mon, 03 Sep 2007 10:19:18 +0000 http://hamletbatista.com/2007/09/03/advanced-cloaking-technique-how-to-feed-password-protected-content-to-search-engine-spiders/#comment-1414 On a related note, how do you maintain the lists of IP's for Google and other SE's? Outdated IP lists seem to be the risk with any form of cloaking... On a related note, how do you maintain the lists of IP’s for Google and other SE’s? Outdated IP lists seem to be the risk with any form of cloaking…

]]> By: Jez http://hamletbatista.com/2007/09/03/advanced-cloaking-technique-how-to-feed-password-protected-content-to-search-engine-spiders/#comment-1413 Jez Mon, 03 Sep 2007 10:16:33 +0000 http://hamletbatista.com/2007/09/03/advanced-cloaking-technique-how-to-feed-password-protected-content-to-search-engine-spiders/#comment-1413 Hi Hamlet, Are you saying that by setting the user agent in Curl I could spider and rip SEOMOZ premium content? ;-) Jez Hi Hamlet,

Are you saying that by setting the user agent in Curl I could spider and rip SEOMOZ premium content? ;-)

Jez

]]>