I completely agree with you. That is why I say that this is an ongoing battle.

I had content stamping for my feed on this blog. Exactly as you describe it. The plugin stopped working after I moved to Wordpress 2.3. Hopefully they will fix it soon. Check it out http://www.blogclout.com/blog/goodies/feed-footer-plugin/

On the other hand, it would be great if Google fixed this. How do you think they can fix this problem? I blogged about an idea I had a while back, but I’d appreciate your ideas and suggestions. Please check http://hamletbatista.com/2007/07/19/content-is-king-but-duplicate-content-is-a-royal-pain/

Yes, sorry I got a little carried away when detailing my explanation. But here is the reason why I think content stamping on legitimate pages (non-honey pot) is also important: proxy-hijacking techniques will evolve - so will web scraping. What may be simple today, will be a totally different beast down the line. Honey pots will make a great positive identification of the unsophisticated proxies. However, as they change their tactics, locating the source of scraped content may become not so simple (or not so easy to black list). It is even possible that embedding scraping or proxy collecting software in malware could open up content to legitimate clients facilitating the theft. At that point the best defense is an aggressive offense of locating all content that has been copied. (And having a stamp in the public content too will help greatly) - Perhaps it could be worked into a copyright statement like:
“copyright 2007 example.com 3243F6A8885A308D31319 3P1I4E159 7F000001″ - or similar.

Thanks for stopping by.

I propose an additional step in dealing with this scourge: content stamping and source stamping.

Sorry, I didn’t make it clearer, but what you describe is the purpose of the encoded text I mentioned in my recommendation. Thanks for making it clearer, though.

The honeypot does exactly this. The honeypot generates email addresses with encoded IP, timestamp, etc. The harvesters collect the addresses and when the spam hits, the email address has all the identifying information. They use base64 encoding, which makes the strings a little bit shorter than simply hexing the characters.

My proposal to identify the CGI proxy hijackers is to generate an encoded text in the body of the honeypot page, and later perform searches for this text in major search engines to see if results come up. As the honeypot page is not supposed to be indexed (per the meta robots tag and robots.txt instructions), the presence of the text in the index means a CGI proxy is responsible. Finally, the encoded IP addresses and other information can be recorded in the blacklist and labeled as CGI proxy hijackers using one of the reserved slots shown in the diagram above.

Your content stamping idea is really interesting, but the encoded text needs to be included in all pages. Many site owners would not like to display such text in their copy. I’d personally prefer to have that encoded text on the honeypot page that is not accessible by regular readers. I believe the CGI proxy hijackers ‘copy’ all the pages, so we can do the detection in one page to catch them. I guess you could hide the encoded text too, but there will be the concern of being penalized.

Another thing I should mention:
-You can use .htaccess blocking for sites without php. Just use a cron job (or scheduled task) to rebuild your .htaccess file regularly from your block list source. This is assuming that you can download the BL & get regular updates.
-You can also use a captcha for forbidden requests so that real humans can access your site in the event of false positives. (You may want to “allow all” to the captcha page.

These are excellent ideas. Thanks for sharing! Alternatively, they could rebuild the .htaccess on their PC and upload it to the server if they don’t have access to scripting (the cron job needs to run a script).

Also, here’s an example of an extended content stamp. (or content-id tag)
3243F6A8885A308D31319 3P1I4E159 7F000001
1 site wide content tag, 2 unique resource tag, 3 hex code for IP address of client requesting your resource.
1 is constant across your domain
2 is unique to each page/resource
3 is unique to each client who requests your content

Find copies of your site by searching:
-site:http:/yourdomain.com 3243F6A8885A308D31319

Find copies of your resource by searching:
-site:http:/yourdomain.com 3243F6A8885A308D31319 3P1I4E159
(or remove the global content tag portion to broaden your results)

Then WHOIS the associated IP address to determine where the content was picked up from. Blacklist if necessary.

Even still - despite some overall imminent victories from G against proxy-hijackers, it is always important to protect your content from other scrapers - and even future revisions of proxy-hijacking.

I propose an additional step in dealing with this scourge: content stamping and source stamping. This stamp would be inserted into the content as a unique string of characters unique to each of your documents. A second part of the string would be an IP address encoding (either in decimal notation, in hex, or in any encrypted form for added effectiveness.) This would allow quick discovery of which content in the index is being copied and also give you an idea about what IP address was used to copy the content (or even tell you where this site got your content from)

Here’s an example of the content stamp: (first is the unique content ID, the second is the hex format of the IP address)
3P1I4E159 7F000001
-ensure that the content id tag is unique for each page or resource for your site.
-of course you can make the unique content id longer to decrease the effectiveness when searching.
-This is also very effective when copying sites insert random words or parts of words to break up your normal sentences. (I’ve seen “th” used as a random injection into various sentences to avert manual duplicate content detection)
-You can also prepend the content stamp with a sitewide content stamp. This will allow you to search google for [your-global-content-stamp -site:http://yoursite.com]

Making a google alert (or alert for any other service) regarding copies of your content will be swift and easy.

Another thing I should mention:
-You can use .htaccess blocking for sites without php. Just use a cron job (or scheduled task) to rebuild your .htaccess file regularly from your block list source. This is assuming that you can download the BL & get regular updates.
-You can also use a captcha for forbidden requests so that real humans can access your site in the event of false positives. (You may want to “allow all” to the captcha page.

A lot of people have low grade coded websites that don’t have any ability to set anything globally. In these cases it would be better to use a the server to block.

http://www.webmasterworld.com/google/3473585.htm

I am glad they finally decided to address the problem, though.