nutch安装、配置和使用

安装nutch

1. 下载nutch编译完的安装包，下载地址如下：http://labs.renren.com/apache-mirror//nutch/apache-nutch-1.3-bin.zip 并解压。解压完的目录如下：

[root@mdss33 nutch-1.3]# lsbuild.xml  CHANGES.txt  conf  default.properties  docs  ivy  KEYS  lib  LICENSE.txt  NOTICE.txt  README.txt  runtime  src

2. 修改文件权限

[root@mdss33 nutch-1.3]# cd runtime/local/bin/     conf/    crawl/   lib/     logs/    plugins/ urls/    [root@mdss33 nutch-1.3]# cd runtime/local/[root@mdss33 local]# chmod +x ./bin/nutch 

配置运行

3. 修改配置文件vim ./conf/utch-site.xml

<?xml version="1.0"?><?xml-stylesheet type="text/xsl" href="configuration.xsl"?><!-- Put site-specific property overrides in this file. --><configuration><property><name>http.agent.name</name><value>my nutch spider</value></property></configuration>

4. 添加nutch爬虫的种子url

[root@mdss33 local]# mkdir -p urls

[root@mdss33 local]# vim ./urls/seed

内容如下，这里需要注意的是在第二行的末尾需要添加空行：

[root@mdss33 local]# cat ./urls/seed http://nutch.apache.org/http://hbase.apache.org/

5. 修改regex-urlfilter.txt文件，将下面的内容：

# accept anything else+.

替换成：

# accept anything else# +.+^http://([a-z0-9]*\.)*nutch.apache.org/+^http://([a-z0-9]*\.)*hbase.apache.org/

6. 配置工作已经完成，下面开始让spider运行。

bin/nutch crawl urls -dir crawl -depth 3 -topN 5

命令解释如下：

-dir dir names the directory to put the crawl in.-threads threads determines the number of threads that will fetch in parallel.-depth depth indicates the link depth from the root page that should be crawled.-topN N determines the maximum number of pages that will be retrieved at each level up to the depth.

7. nutch运行完毕之后，将生成如下的目录：

crawl/crawldbCrawl/linkdbcrawl/segments