基于Crawler4j + jsoup实现爬虫

爬虫框架分类

1. 分布式爬虫
Nutch
2. Java单机爬虫
Crawler4j、WebMagic、WebCollector
3. 非Java单机爬虫
Scrapy

开发思路

根据业务需求选择合适的爬虫框架
根据网站规则及业务需求抽取数据，保存到中间库
数据清洗/格式化，保存到目标库

基于Crawler4j + jsoup实现爬虫

用Crawler4j构建多线程的web爬虫来抓取页面内容。Crawler4j的使用分为两个步骤：

(1) 实现一个继承自WebCrawler的爬虫类
需要覆盖两个主要方法：
shouldVisit：这个方法决定了要抓取的URL及其内容。
visit：当URL下载完成会调用这个方法。使用jsoup解析HTML，可以采用jQuery选择器的语法。
(2) 通过CrawController调用实现的爬虫类。
指定抓取的种子（seed）、中间数据存储的文件夹、并发线程的数目等信息，实现控制器类。 

示例代码

pom.xml中依赖包设置

<dependency><groupId>edu.uci.ics</groupId><artifactId>crawler4j</artifactId><version>4.2</version></dependency><dependency><groupId>org.jsoup</groupId><artifactId>jsoup</artifactId><version>1.10.1</version></dependency>

代码

(1) 爬虫类

public class MyCrawler extends WebCrawler {     private final static Pattern FILTERS = Pattern.compile(".*(\\.(css|js|gif|jpg"                                                           + "|png|mp3|mp3|zip|gz))$");     /**     * This method receives two parameters. The first parameter is the page     * in which we have discovered this new url and the second parameter is     * the new url. You should implement this function to specify whether     * the given url should be crawled or not (based on your crawling logic).     * In this example, we are instructing the crawler to ignore urls that     * have css, js, git, ... extensions and to only accept urls that start     * with "http://www.ics.uci.edu/". In this case, we didn't need the     * referringPage parameter to make the decision.     */     @Override     public boolean shouldVisit(Page referringPage, WebURL url) {         String href = url.getURL().toLowerCase();         return !FILTERS.matcher(href).matches()                && href.startsWith("http://www.ics.uci.edu/");     }      /**      * This function is called when a page is fetched and ready      * to be processed by your program.      */     @Override     public void visit(Page page) {         String url = page.getWebURL().getURL();         System.out.println("URL: " + url);          if (page.getParseData() instanceof HtmlParseData) {             HtmlParseData htmlParseData = (HtmlParseData) page.getParseData();             String text = htmlParseData.getText();             String html = htmlParseData.getHtml();             Set<WebURL> links = htmlParseData.getOutgoingUrls();              System.out.println("Text length: " + text.length());             System.out.println("Html length: " + html.length());             System.out.println("Number of outgoing links: " + links.size());         }    }}

(2) Controller调用

public class Controller {    public static void main(String[] args) throws Exception {        String crawlStorageFolder = "/data/crawl/root";        int numberOfCrawlers = 7;         CrawlConfig config = new CrawlConfig();        config.setCrawlStorageFolder(crawlStorageFolder);         /*         * Instantiate the controller for this crawl.         */        PageFetcher pageFetcher = new PageFetcher(config);        RobotstxtConfig robotstxtConfig = new RobotstxtConfig();        RobotstxtServer robotstxtServer = new RobotstxtServer(robotstxtConfig, pageFetcher);        CrawlController controller = new CrawlController(config, pageFetcher, robotstxtServer);         /*         * For each crawl, you need to add some seed urls. These are the first         * URLs that are fetched and then the crawler starts following links         * which are found in these pages         */        controller.addSeed("http://www.ics.uci.edu/~lopes/");        controller.addSeed("http://www.ics.uci.edu/~welling/");        controller.addSeed("http://www.ics.uci.edu/");         /*         * Start the crawl. This is a blocking operation, meaning that your code         * will reach the line after this only when crawling is finished.         */        controller.start(MyCrawler.class, numberOfCrawlers);    }}