Jez

I’m glad to have other developers visiting my blog. Hopefully you can put some of the code in my posts to work. I appreciate any feedback.

It’s amazing how we can find open source code for pretty much everything. Moodle.org looks very interesting.

I am no stranger to code, I work in that field, but most of my experience has been on Intranets… I currently manage a large installation (9 instances) of moodle.org for a University… but there is no SEO requirement for this work… SEO is something I am interested in learning more about…

If it had been me, I would have asked users to collect the link text from a dynamic page that rotated a number of different permutations….

That doesn’t sound like newbie stuff to me
I am working on a post where I am dissecting Google’s original paper. Hopefully we all can learn something valuable from it.

I did not explain myself well….

Sorry if I did not read your post thoroughly enough… your points are interesting, it could well have been the anchor text… or perhaps a mix of anchor and robots.txt.

I thought for some time that JC should have asked users to use a mix of different links.

If it had been me, I would have asked users to collect the link text from a dynamic page that rotated a number of different permutations….

As for being an expert, far from it, I am here to learn

Thanks for your comment. I am really glad to have experts visiting my blog.

Please note that I did not rule out the robots.txt changes, as the solution to the problem.

If this was the change that fixed the problem, it might have been because removing those internal pages from the spider view might have weaken his internal link structure. His claim is not without merit.

I am not sure I follow part of your conclusions.

… as you know the cache is always a few days old, but the robots.txt will be analysed on the day of the crawl, in “real time”.

If Google never let go of the cached file, how would it ever crawl the site again???

The robots.txt is “analyzed” (parsed) in real time, but the results of this will need to be reflected when the index is updated. Search engines first crawl and then index pages.

Dropping pages imply modification to the index (as a result of a crawl).

To me, the pages in the cache are the pages that are affecting the current index. I might be wrong, but I need some research papers that would tell me otherwise.

Again, I am not ruling out the robots.txt as the solution to his problem.

At one moment, I thought that he blocked access to the regular posts by misusing the wildcards ie.: Disallow: /2007/1*. My blog includes the date in normal posts, but I checked his and it doesn’t.

I have been following this issue too and think you have made a bit of an error… as you know the cache is always a few days old, but the robots.txt will be analysed on the day of the crawl, in “real time”.

If Google never let go of the cached file, how would it ever crawl the site again???

The actual crawl runs ahead of the cache, but you already know this…

One thing you may not have seen is this post on JC:

http://www.johnchow.com/getting-out-of-the-google-supplemental-index/

A few days earlier the robots.txt file was changed due to reasons outlined in the above post… give it a few days for the denied pages to be dropped, a couple of days for users to report the drop in SERPs and the timing is about right for Johns “google ban”.

Then, the latest robots.txt file reverses what had been done, re-allows the supplemental pages things return to normal.

What we should have checked was whether the supplemental pages were back in the cache

I think JC made a blunder in blocking his supplemental pages simple as that.

Does anyone really believe Google would change their algorithm because of John Chow!!!!

I think you have to bear in mind that JC survives on hype, spin and reader manipulation, that’s what the his site exemplifies. I think he has created a lot of buzz and mystery out of his own %$£% up… that’s what he is good at.