PDA

View Full Version : Prevent robots from indexing your images


mvrekum
26th of July 2004 (Mon), 07:19
I had a nice gallery with about 30 visitors a day average.
UNTIL last friday: 745 visitors because my images suddenly showed up in the google image directory. And it is still continuing.
Last week one image had still about 1000 views counted since the beginning of this year. Now it is already up to about 4500. And that in just 3 days.

I do not know what you have to do to get your images in the google directory. I have had these images online since a couple of years and they never popped up in google, but now I am reaching my bandwith limit. I have already shut down the EE engine!!

What can you do to prevent sites like google to index your images? That is besides the robots.txt file! I already put that one online.

Martin

tommykjensen
26th of July 2004 (Mon), 07:30
To my knowledge if google does not respect the robot.txt directives then theres not much You can do except put password on the exhibition.

mvrekum
26th of July 2004 (Mon), 08:18
Google does respect robots.txt, however google uses a different bot for gathering images. So adding Google as user agent is not enough, you have to add "Googlebot-Image" to your robots.txt file.

Anyway google is blocked now, but that is just one specific bot. How about all the others?

mvrekum
27th of July 2004 (Tue), 04:04
I found a line for the robots.txt file that will prevent bots from indexing specific files. So to keep you images out of google or other search engines add a robots.txt file with the following content:

User-agent: *
Disallow: /*.jpg$

Instead of .jpg you can also add other filetypes, or simply add extra lines with other filetypes.

I hope this helps others to, who have their entire monthly bandwith consumed within a few days. :(

ArtM
29th of July 2004 (Thu), 03:54
More information on 'robots.txt' at

http://www.robotstxt.org/

- Art