Comments on: Controlling Your Robots: Using the X-Robots-Tag HTTP header with Googlebot http://hamletbatista.com/2007/08/01/controlling-your-robots-using-the-x-robots-tag-http-header-with-googlebot/ Advanced Search Engine Marketing Tips to Succeed Online Mon, 08 Sep 2008 01:07:35 +0000 http://wordpress.org/?v=2.3 By: Robots.txt, meta-robots, nofollow ja hakukoneet | Nettibisnes.Info http://hamletbatista.com/2007/08/01/controlling-your-robots-using-the-x-robots-tag-http-header-with-googlebot/#comment-3704 Robots.txt, meta-robots, nofollow ja hakukoneet | Nettibisnes.Info Tue, 13 Nov 2007 12:07:42 +0000 http://hamletbatista.com/2007/08/01/controlling-your-robots-using-the-x-robots-tag-http-header-with-googlebot/#comment-3704 [...] Robots-meta-tagit ei-HTML-tiedostoissa – virallisen Google-blogin artikkeli esittelee paitsi unavailable_after -parametrin niin myös X-Robots-Tag -direktiivin jolla voidaan vaikuttaa esimerkiksi videoiden, kuvien, ääni- ja pdf-tiedostojen indeksoitumiseen. Aihetta syventävät myös Sebastian ja Hamlet Batista. [...] […] Robots-meta-tagit ei-HTML-tiedostoissa – virallisen Google-blogin artikkeli esittelee paitsi unavailable_after -parametrin niin myös X-Robots-Tag -direktiivin jolla voidaan vaikuttaa esimerkiksi videoiden, kuvien, ääni- ja pdf-tiedostojen indeksoitumiseen. Aihetta syventävät myös Sebastian ja Hamlet Batista. […]

]]> By: Noobliminal http://hamletbatista.com/2007/08/01/controlling-your-robots-using-the-x-robots-tag-http-header-with-googlebot/#comment-1877 Noobliminal Thu, 13 Sep 2007 19:44:57 +0000 http://hamletbatista.com/2007/08/01/controlling-your-robots-using-the-x-robots-tag-http-header-with-googlebot/#comment-1877 You killed me... I almost spilled my RedBull. It's a good thing you didn't call me Plato ;) (There's many quotes of him too!) I'm working on the About Page (the big photo of me). There's so much things to say ... it's difficult. PS:You made my day! Thanks. You killed me… I almost spilled my RedBull. It’s a good thing you didn’t call me Plato ;) (There’s many quotes of him too!)
I’m working on the About Page (the big photo of me). There’s so much things to say … it’s difficult.

PS:You made my day! Thanks.

]]> By: Hamlet Batista http://hamletbatista.com/2007/08/01/controlling-your-robots-using-the-x-robots-tag-http-header-with-googlebot/#comment-1876 Hamlet Batista Thu, 13 Sep 2007 19:30:46 +0000 http://hamletbatista.com/2007/08/01/controlling-your-robots-using-the-x-robots-tag-http-header-with-googlebot/#comment-1876 Claude - That is why is good to include an about page on your blog. I hit the home page and it seems I picked the name from one of you random quotes. Sorry :-) Claude - That is why is good to include an about page on your blog. I hit the home page and it seems I picked the name from one of you random quotes. Sorry :-)

]]> By: Noobliminal http://hamletbatista.com/2007/08/01/controlling-your-robots-using-the-x-robots-tag-http-header-with-googlebot/#comment-1869 Noobliminal Thu, 13 Sep 2007 17:01:35 +0000 http://hamletbatista.com/2007/08/01/controlling-your-robots-using-the-x-robots-tag-http-header-with-googlebot/#comment-1869 I'm clueless where you came up with the Scott name :) Just call me <strong>Noobliminal</strong> or Claude ... <em>PS: I'm 100% sure you don't know me so if you take me for some1 else ... :)</em> I’m clueless where you came up with the Scott name :) Just call me Noobliminal or Claude …
PS: I’m 100% sure you don’t know me so if you take me for some1 else … :)

]]> By: Hamlet Batista http://hamletbatista.com/2007/08/01/controlling-your-robots-using-the-x-robots-tag-http-header-with-googlebot/#comment-1868 Hamlet Batista Thu, 13 Sep 2007 16:54:47 +0000 http://hamletbatista.com/2007/08/01/controlling-your-robots-using-the-x-robots-tag-http-header-with-googlebot/#comment-1868 Scott - I am afraid I agree with you. This is a very clever and effective black hat trick and no way to detect it or protect against it. Scott - I am afraid I agree with you. This is a very clever and effective black hat trick and no way to detect it or protect against it.

]]> By: Noobliminal http://hamletbatista.com/2007/08/01/controlling-your-robots-using-the-x-robots-tag-http-header-with-googlebot/#comment-1865 Noobliminal Thu, 13 Sep 2007 15:40:53 +0000 http://hamletbatista.com/2007/08/01/controlling-your-robots-using-the-x-robots-tag-http-header-with-googlebot/#comment-1865 What if ... I told you this has another use that can take link-exchange back to stone age? This is your white-hat use, check my dark-side use. Fortunately it's not accessible to every1 but still... Regards. What if …
I told you this has another use that can take link-exchange back to stone age? This is your white-hat use, check my dark-side use.
Fortunately it’s not accessible to every1 but still…

Regards.

]]> By: Hamlet Batista http://hamletbatista.com/2007/08/01/controlling-your-robots-using-the-x-robots-tag-http-header-with-googlebot/#comment-1163 Hamlet Batista Mon, 27 Aug 2007 13:45:08 +0000 http://hamletbatista.com/2007/08/01/controlling-your-robots-using-the-x-robots-tag-http-header-with-googlebot/#comment-1163 Mark - Technically you don't add the X-Robots-Tag to the pages, but your webserver or some other script will need to add them to the HTTP headers. If you only want to block access to your RSS feeds, deny access to them via your robot.txt file. Mark - Technically you don’t add the X-Robots-Tag to the pages, but your webserver or some other script will need to add them to the HTTP headers.

If you only want to block access to your RSS feeds, deny access to them via your robot.txt file.

]]> By: Mark http://hamletbatista.com/2007/08/01/controlling-your-robots-using-the-x-robots-tag-http-header-with-googlebot/#comment-1134 Mark Sun, 26 Aug 2007 06:02:41 +0000 http://hamletbatista.com/2007/08/01/controlling-your-robots-using-the-x-robots-tag-http-header-with-googlebot/#comment-1134 I track traffic from my RSS feeds by appending tracking code to the URLs. These pages are currently being indexed by Google, hence I have a duplicate content issue. I want to prevent the URLs in my RSS (the one that have the tracking codes) from appearing in SERPs. Do you whether the, "X-Robots-Tag: noindex,nofollow" added to my .rss pages will fix this problem? Is there any other method that you'd recommend? Regards, Mark I track traffic from my RSS feeds by appending tracking code to the URLs. These pages are currently being indexed by Google, hence I have a duplicate content issue. I want to prevent the URLs in my RSS (the one that have the tracking codes) from appearing in SERPs.

Do you whether the, “X-Robots-Tag: noindex,nofollow”
added to my .rss pages will fix this problem? Is there any other method that you’d recommend?

Regards, Mark

]]> By: Hamlet Batista http://hamletbatista.com/2007/08/01/controlling-your-robots-using-the-x-robots-tag-http-header-with-googlebot/#comment-680 Hamlet Batista Thu, 02 Aug 2007 15:22:27 +0000 http://hamletbatista.com/2007/08/01/controlling-your-robots-using-the-x-robots-tag-http-header-with-googlebot/#comment-680 Sebastian - Good job! Sebastian - Good job!

]]> By: Sebastian http://hamletbatista.com/2007/08/01/controlling-your-robots-using-the-x-robots-tag-http-header-with-googlebot/#comment-674 Sebastian Thu, 02 Aug 2007 07:18:47 +0000 http://hamletbatista.com/2007/08/01/controlling-your-robots-using-the-x-robots-tag-http-header-with-googlebot/#comment-674 You can go a step further and rewrite PDF requests to a PHP script where you can handle the values of X-Robots-Tag per file, or serve a shortened version to bots and lurkers and the full version to members ... http://sebastianx.blogspot.com/2007/07/handling-googles-neat-x-robots-tag.html HTH Sebastian You can go a step further and rewrite PDF requests to a PHP script where you can handle the values of X-Robots-Tag per file, or serve a shortened version to bots and lurkers and the full version to members …
http://sebastianx.blogspot.com/2007/07/handling-googles-neat-x-robots-tag.html
HTH
Sebastian

]]>