正则表达式——抓取网页中的http及网络图片

**

正则表达式——抓取网页中的http及网络图片

**
最近网上有些朋友问我怎么用JAVA抓起网页中的image地址，我现在就给大家写一个实例，希望大家能通过这个实例学习到一点东西

需要抓取网页中http链接地址首先还需准备一个网站地址，我们就以hao123为例
https://www.hao123.com/

在这个网页中，我们要抓取这里面的href地址首先需要解析这个网站中的内容，把它转换成String字符串

 @Test    public void parsing_URL() {        System.out.println(getHTMLText("https://www.hao123.com/","utf-8"));    }    /**     * 获取网页文本     * */    public String getHTMLText(String htmlUrl,String charset){        StringBuilder sb = new StringBuilder();        try {            URL url = new URL(htmlUrl);            BufferedReader br = new BufferedReader(new InputStreamReader(url.openStream(),charset));            String result = "";            while ((result = br.readLine()) != null){                sb.append(result).append("\n");            }        } catch (MalformedURLException e) {            e.printStackTrace();        } catch (IOException e) {            e.printStackTrace();        }        return sb.toString();    }

打印结果如下

然后我们在通过正则表达式来解析网站中存在的href对应的地址

 @Test    public void parsing_URL() {        String content = getHTMLText("https://www.hao123.com/","gbk");        String regular = "href=\"([\\w\\s./:]+?)\"";        List<String> list = regularParsingResult(content,regular);        for (String str:list) {            System.out.println(str);        }    }    /**     * 正则解析结果     */    public List<String> regularParsingResult(String content,String regular){        Pattern pattern = Pattern.compile(regular);        Matcher matcher = pattern.matcher(content);        List<String>  list = new ArrayList<>();        while (matcher.find()){            list.add(matcher.group(1));        }        return list;    }

打印结果如下

现在我们来获取一些网页中的img标签下的图片地址

@Test    public void parsing_URL() {        String content = getHTMLText("https://www.hao123.com/","utf-8");        String regular = "<img[\\s\\S]+?src=\"(.+?)\"";        List<String> list = regularParsingResult(content,regular);        for (String str:list) {            System.out.println(str);        }    }    /**     * 正则解析结果     */    public List<String> regularParsingResult(String content,String regular){        Pattern pattern = Pattern.compile(regular);        Matcher matcher = pattern.matcher(content);        List<String>  list = new ArrayList<>();        while (matcher.find()){            list.add(matcher.group(1));        }        return list;    }

打印结果

获取网页完整代码

public void getHTMLTextComplete(String htmlUrl) throws InterruptedException {        WebClient webClient = new WebClient(BrowserVersion.FIREFOX_10);        //设置webClient的相关参数        webClient.getOptions().setJavaScriptEnabled(true);        webClient.getOptions().setCssEnabled(false);        webClient.setAjaxController(new NicelyResynchronizingAjaxController());        //webClient.getOptions().setTimeout(50000);        webClient.getOptions().setThrowExceptionOnScriptError(false);        //模拟浏览器打开一个目标网址        HtmlPage rootPage = webClient.getPage(htmlUrl);        System.out.println("为了获取js执行的数据 线程开始沉睡等待");        Thread.sleep(3000);//主要是这个线程的等待 因为js加载也是需要时间的        System.out.println("线程结束沉睡");        String html = rootPage.asText();        System.out.println(html);    }