我的buyvm小机机要被蜘蛛们抓死了，怎么都一起来了

fwt0125

220.181.94.216 - - [07/Apr/2011:22:51:47 +0800] "GET /play/?28384-0-0.html HTTP/1.1" 200 14451 "-" "Sogou web spider/4.0(+http://www.sogou.com/docs/help/webmasters.htm#07)"
66.249.67.178 - - [07/Apr/2011:22:49:54 +0800] "GET /good-2807514066.jsp HTTP/1.1" 200 72 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
66.249.67.155 - - [07/Apr/2011:22:51:55 +0800] "GET /uploads/allimg/1/1a/1282367566.jpg HTTP/1.1" 200 14612 "-" "Googlebot-Image/1.0"
220.181.94.216 - - [07/Apr/2011:22:51:57 +0800] "GET /play/?28383-0-0.html HTTP/1.1" 200 14825 "-" "Sogou web spider/4.0(+http://www.sogou.com/docs/help/webmasters.htm#07)"
220.181.94.216 - - [07/Apr/2011:22:52:07 +0800] "GET /play/?2494-0-60.html HTTP/1.1" 200 38724 "-" "Sogou web spider/4.0(+http://www.sogou.com/docs/help/webmasters.htm#07)"
61.135.249.210 - - [07/Apr/2011:22:52:08 +0800] "GET /content/?24009.html HTTP/1.1" 200 12868 "-" "Mozilla/5.0 (compatible; YoudaoBot/1.0; http://www.youdao.com/help/webmaster/spider/; )"
220.181.94.216 - - [07/Apr/2011:22:52:18 +0800] "GET /play/?28726-0-0.html HTTP/1.1" 200 15690 "-" "Sogou web spider/4.0(+http://www.sogou.com/docs/help/webmasters.htm#07)"
220.181.108.156 - - [07/Apr/2011:22:52:18 +0800] "GET /good-4700735893.jsp HTTP/1.1" 200 50206 "-" "Baiduspider+(+http://www.baidu.com/search/spider.htm)"
220.181.94.216 - - [07/Apr/2011:22:52:27 +0800] "GET /play/?28577-0-0.html HTTP/1.1" 200 18282 "-" "Sogou web spider/4.0(+http://www.sogou.com/docs/help/webmasters.htm#07)"
220.181.94.216 - - [07/Apr/2011:22:52:37 +0800] "GET /play/?28356-0-0.html HTTP/1.1" 200 13461 "-" "Sogou web spider/4.0(+http://www.sogou.com/docs/help/webmasters.htm#07)"
66.249.67.155 - - [07/Apr/2011:22:52:39 +0800] "GET /uploads/allimg/f/f2/1301745628.jpg HTTP/1.1" 200 6542 "-" "Googlebot-Image/1.0"
220.181.94.216 - - [07/Apr/2011:22:52:47 +0800] "GET /play/?28360-0-0.html HTTP/1.1" 200 13506 "-" "Sogou web spider/4.0(+http://www.sogou.com/docs/help/webmasters.htm#07)"
123.122.94.119 - - [07/Apr/2011:22:52:22 +0800] "GET /attachments/lyh/%E8%80%81%E7%BD%97%E8%AF%AD%E5%BD%95_%E9%AB%98%E5%B1%B1%E6%97%8F.wma HTTP/1.1" 200 2926202 "-" "NSPlayer/11.00.6002.18392 WMFSDK/11.00.6002.18392"
220.181.108.158 - - [07/Apr/2011:22:52:51 +0800] "GET /content/?1930.html HTTP/1.1" 301 328 "-" "Baiduspider+(+http://www.baidu.com/search/spider.htm)"
220.181.108.106 - - [07/Apr/2011:22:52:51 +0800] "GET /content/?1838.html HTTP/1.1" 200 15626 "-" "Baiduspider+(+http://www.baidu.com/search/spider.htm)"
220.181.108.86 - - [07/Apr/2011:22:52:51 +0800] "GET /content/?1930.html HTTP/1.1" 200 13976 "-" "Baiduspider+(+http://www.baidu.com/search/spider.htm)"
220.181.108.85 - - [07/Apr/2011:22:52:55 +0800] "GET /content/?1893.html HTTP/1.1" 301 328 "-" "Baiduspider+(+http://www.baidu.com/search/spider.htm)"
220.181.94.216 - - [07/Apr/2011:22:52:57 +0800] "GET /play/?28361-0-0.html HTTP/1.1" 200 13756 "-" "Sogou web spider/4.0(+http://www.sogou.com/docs/help/webmasters.htm#07)"
220.181.108.111 - - [07/Apr/2011:22:52:59 +0800] "GET /content/?1893.html HTTP/1.1" 200 14633 "-" "Baiduspider+(+http://www.baidu.com/search/spider.htm)"

刹那芳华

屏蔽啊

fwt0125

原帖由 刹那芳华 于 2011-4-7 22:54 发表


屏蔽啊
不知道说什么好

xtmp

sougou的spider屏蔽没用，它根本不遵守robot

walkman660

直接把搜狗IP拉黑

l12ab

在user-agent里面屏蔽spider

Host.HowPick

搜狗是谁家的？ 我只知道 gg和bd

mslxd

sogou是这样的呃，你要给它吃别的。。。

糕手

上缓存吧怎么禁用蜘蛛。。

wdlthypandjp

原帖由 ypandjp 于 2011-4-8 00:58 发表


怎么禁用蜘蛛。。
http://***/robots.txt
User-agent: *
Disallow: /