蜘蛛爬行记录在一个文本里,

因为担心蜘蛛没来,所以昨天把程序做了一个小的修改,把蜘蛛爬行记录全部记录下来了,发现百度来的比较快,可能是中文的原因,百度来的最多:

Time:2010-05-21.18:00:18 robot:Baiduspider URL:Baiduspider+(+http://www.baidu.com/search/spider.htm)
Time:2010-05-21.10:00:24 robot:Baiduspider URL:Baiduspider+(+http://www.baidu.com/search/spider.htm)
Time:2010-05-21.10:00:24 robot:Baiduspider URL:Baiduspider+(+http://www.baidu.com/search/spider.htm)
Time:2010-05-21.10:00:28 robot:Baiduspider URL:Baiduspider+(+http://www.baidu.com/search/spider.htm)
Time:2010-05-21.10:00:29 robot:Baiduspider URL:Baiduspider+(+http://www.baidu.com/search/spider.htm)
Time:2010-05-21.18:00:32 robot:Baiduspider URL:Baiduspider+(+http://www.baidu.com/search/spider.htm)
Time:2010-05-21.10:00:33 robot:Baiduspider URL:Baiduspider+(+http://www.baidu.com/search/spider.htm)
Time:2010-05-21.10:00:36 robot:Baiduspider URL:Baiduspider+(+http://www.baidu.com/search/spider.htm)
Time:2010-05-21.10:00:38 robot:Baiduspider URL:Baiduspider+(+http://www.baidu.com/search/spider.htm)
Time:2010-05-21.10:00:40 robot:Baiduspider URL:Baiduspider+(+http://www.baidu.com/search/spider.htm)
Time:2010-05-21.10:00:47 robot:Baiduspider URL:Baiduspider+(+http://www.baidu.com/search/spider.htm)
Time:2010-05-21.10:00:57 robot:Baiduspider URL:Baiduspider+(+http://www.baidu.com/search/spider.htm)

Time:2010-05-21.18:01:00 robot:Baiduspider URL:Baiduspider+(+http://www.baidu.com/search/spider.htm)
Time:2010-05-21.10:01:01 robot:Baiduspider URL:Baiduspider+(+http://www.baidu.com/search/spider.htm)
Time:2010-05-21.10:01:08 robot:Baiduspider URL:Baiduspider+(+http://www.baidu.com/search/spider.htm)
Time:2010-05-21.10:01:14 robot:Baiduspider URL:Baiduspider+(+http://www.baidu.com/search/spider.htm)
Time:2010-05-21.10:01:19 robot:Baiduspider URL:Baiduspider+(+http://www.baidu.com/search/spider.htm)
Time:2010-05-21.20:37:30 robot:Baiduspider URL:Baiduspider+(+http://www.baidu.com/search/spider.htm)
Time:2010-05-21.12:53:47 robot:Baiduspider URL:Baiduspider+(+http://www.baidu.com/search/spider.htm)
Time:2010-05-21.12:53:48 robot:Baiduspider URL:Baiduspider+(+http://www.baidu.com/search/spider.htm)
Time:2010-05-21.12:53:52 robot:Baiduspider URL:Baiduspider+(+http://www.baidu.com/search/spider.htm)
Time:2010-05-21.12:54:00 robot:Baiduspider URL:Baiduspider+(+http://www.baidu.com/search/spider.htm)
Time:2010-05-21.12:54:05 robot:Baiduspider URL:Baiduspider+(+http://www.baidu.com/search/spider.htm)
Time:2010-05-21.12:54:17 robot:Baiduspider URL:Baiduspider+(+http://www.baidu.com/search/spider.htm)
Time:2010-05-21.12:54:21 robot:Baiduspider URL:Baiduspider+(+http://www.baidu.com/search/spider.htm)
Time:2010-05-21.12:54:26 robot:Baiduspider URL:Baiduspider+(+http://www.baidu.com/search/spider.htm)
Time:2010-05-21.12:54:31 robot:Baiduspider URL:Baiduspider+(+http://www.baidu.com/search/spider.htm)
Time:2010-05-21.20:54:31 robot:Baiduspider URL:Baiduspider+(+http://www.baidu.com/search/spider.htm)
Time:2010-05-21.20:54:36 robot:Baiduspider URL:Baiduspider+(+http://www.baidu.com/search/spider.htm)
Time:2010-05-21.12:54:39 robot:Baiduspider URL:Baiduspider+(+http://www.baidu.com/search/spider.htm)
Time:2010-05-21.12:54:42 robot:Baiduspider URL:Baiduspider+(+http://www.baidu.com/search/spider.htm)
Time:2010-05-21.12:54:44 robot:Baiduspider URL:Baiduspider+(+http://www.baidu.com/search/spider.htm)
Time:2010-05-21.12:54:49 robot:Baiduspider URL:Baiduspider+(+http://www.baidu.com/search/spider.htm)
Time:2010-05-21.12:54:54 robot:Baiduspider URL:Baiduspider+(+http://www.baidu.com/search/spider.htm)
Time:2010-05-21.12:54:55 robot:Baiduspider URL:Baiduspider+(+http://www.baidu.com/search/spider.htm)
Time:2010-05-21.12:55:02 robot:Baiduspider URL:Baiduspider+(+http://www.baidu.com/search/spider.htm)
Time:2010-05-21.12:55:05 robot:Baiduspider URL:Baiduspider+(+http://www.baidu.com/search/spider.htm)
Time:2010-05-21.12:58:17 robot:Googlebot URL:Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
Time:2010-05-21.21:18:24 robot:Googlebot URL:Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
Time:2010-05-21.21:37:34 robot:Yahoobot URL:Mozilla/5.0 (compatible; Yahoo! Slurp/3.0; http://help.yahoo.com/help/us/ysearch/slurp)
Time:2010-05-21.21:39:19 robot:Yahoobot URL:Mozilla/5.0 (compatible; Yahoo! Slurp/3.0; http://help.yahoo.com/help/us/ysearch/slurp)
可能是新站的原因吧,百度并没有收录多少,看来下一步就是外链与内容的问题了。