PDA

View Full Version : Robots are hitting my site like crazy!


kd6lor
13th of June 2004 (Sun), 11:30
I have been getting a lot of page views lately and was wondering about the activity on the site. I read my server access logs and found that a large number of them appear to be coming from the MSN search robot. I am trying to figure out if this is a good thing or not. If the ratio of robots to humans was a bit heavier in the human end.

In any case, have the page viewed counters been going crazy on your websites lately? Should this bother or please me? If the robots were asking to buy my images, I wouldn't mind, but right now they are just bandwidth eaters. Maybe I should put up some images of naked mother boards with exposed chip sockets.

Paul

Alexis427
13th of June 2004 (Sun), 12:16
nod, MSN bot is hitting hard, it already sucked 1.4 GB from my site this month, and isn't following robots.txt exclusions :(

wkitty42
15th of June 2004 (Tue), 12:38
nod, MSN bot is hitting hard, it already sucked 1.4 GB from my site this month, and isn't following robots.txt exclusions :(
first, ensure that your robots.txt is in proper format... second, if your robots.txt is, in fact, proper then ensure that you complain longly and loudly until they get this fixed... there is no reason for them to foist yet another bug laden travisity upon the citizens of this world... we have enough problems outside of cyberspace than to have to deal with them on the inside of cyberspace :twisted: :twisted: :twisted: