Comments on: Content is King, but Duplicate Content is a Royal Pain.

By: Richard Chmura

Richard Chmura — Mon, 15 Oct 2007 14:42:14 +0000

But don’t give up hope entirely. With some human review, it could be possible to whitelist some sites for inclusion in this system. It would only work if the whitelist was limited to these human reviews - or a similar method of content ownership verification.

Definitely not a dead idea. But it must have clear boundaries and limits.

By: Hamlet Batista

Hamlet Batista — Mon, 15 Oct 2007 14:36:06 +0000

I agree.

By: Richard Chmura

Richard Chmura — Mon, 15 Oct 2007 10:50:54 +0000

If you think like a bad guy: scraping the content and registering it, stealing from a site or blog that had not implemented this process. This will create a false positive and a disadvantage to all sites that do not subscribe to this technology. The possibility of every single legitimate content owner using a uniform method is slim - even today we have sites which don’t conform to w3c standards or don’t even work in various browsers. It is possible that this might make it easier for illegtimate sites to steal content.

By: Hamlet Batista

Hamlet Batista — Mon, 15 Oct 2007 00:19:43 +0000

The problem I see with this approach is the likeliness of false positives. (identifying scrapers as legitimate content)

Thanks, Richard. Please note that the scrappers access the content via RSS, etc. If the blogging software performs the registration before publishing the content or RSS feed the content author will have a head start. There would be no way for scrappers to access the content before it is registered.

By: Richard Chmura

Richard Chmura — Sun, 14 Oct 2007 20:08:52 +0000

my two cents: (to be the devil’s advocate here) It is an interesting proposition to allow registering of content. The problem I see with this approach is the likeliness of false positives. (identifying scrapers as legitimate content) However, this is a step in establishing trust with a small set of publishers. I would guess that only after some form of algo. or human review would a site have access to this mechanism. - In that sense, any DMCA or copy-spam complaint would revoke current and future access to such a system.

Said system would not be effective in the “pure wild” Filtering and sorting duplicate content in the “wild” level will need another approach.
-I’ll comment on http://hamletbatista.com/2007/10/11/like-flies-to-project-honeypot-revisiting-the-cgi-proxy-hijack-problem/ about my suggestions for dealing with the “wild” internet.

By: Florchakh

Florchakh — Sat, 21 Jul 2007 11:33:29 +0000

I’m not sure that speaking these facts to the wide public is a good idea.

Or not - c’mon Hamlet, spread it. Let’s get more people interested in spamming and help to put sites of people not interested in SEO straight into the flames of spammy net business. I guess your blog will became more popular ]:->

By: Hamlet Batista

Hamlet Batista — Fri, 20 Jul 2007 23:21:58 +0000

Mutiny - please send me an email with some examples of ‘geographic spam pages’. I will try to digg deeper to give a more accurate response.

there are a couple of bugs in the comments system that I want to solve. Let me see if can find some time early next week

By: Hamlet Batista

Hamlet Batista — Fri, 20 Jul 2007 23:18:37 +0000

Jez,

The crawled date is unreliable too. The crawler could visit the scrapper site before yours.

It would also be possible to time delay the rss feed, putting a greater time-distance between your ping and that of a scraper.

I like your idea of introducing a delay or not releasing the feed until the content is registered. Well done!

If this were introduced though, there could be a gold-rush as SE Spammers raced to re-publish all the content on the internet using this method

I guess there would need to be some legal consecuences for people registering content that is not theirs.

By: Mutiny Design

Mutiny Design — Fri, 20 Jul 2007 18:12:49 +0000

Something I mentioned earlier was ‘geographic spam pages’ - pages with duplicate content but use different places names (towns, counties etc). These are always a real pain and are extremely common pactice for web design comapnies, but I have noticed them for various industries.

Looking at a few seaches now, the same companies are still ranking for the same geographic terms using geographic spam pages. Do you have any input on this?

Another culprit of geographic SERP littering are business directories that just have lists of businesses on them.

P.S. is it possible to use nl2br on the comments (if you can do that with wordpress) or is the br element enabled?

By: Hamlet Batista

Hamlet Batista — Fri, 20 Jul 2007 14:17:36 +0000

Geoff - This is just a raw idea. I want you to find holes and hopefully together we can come up with a stronger plan.

For authorized syndication parters I favor Google’s proposal:

If you syndicate content, we suggest that you ask the sites who are using your content to block their version with a robots.txt file as part of the syndication arrangement to help ensure your version is served in results.

By: Geoff

Geoff — Fri, 20 Jul 2007 12:31:35 +0000

I like where you are going with this, but what about sites like Wikipedia that license content to other sites like Answers.com?

I’m not trying to punch any holes in your idea. I hope something like this comes about. To do so, lots of angles have to be considered. It sounds like you already have a great start.

By: Jez

Jez — Fri, 20 Jul 2007 09:48:38 +0000

Hi Hamlet,

An interesting article. When I mentioned using dates, I was thinking of the date Google first found that content, as opposed to the creation date given by the site.
Your point about srapers is a good one though, a site scraping fresh content has a good chance of being indexed before the origial source.
Also your copyright notice may work on some syndicating sites, but could be stripped out by a more determined SE spammer.
Your idea of “first ping” is a good one.
It would also be possible to time delay the rss feed, putting a greater time-distance between your ping and that of a scraper.
If this were introduced though, there could be a gold-rush as SE Spammers raced to re-publish all the content on the internet using this method