ruby抓取web页面

来源：互联网发布：房源中介系统源码编辑：程序博客网时间：2024/04/28 23:03

一种方法是Net::HTTP.new方法,返回resp码和实际的data:

require 'net/http'h = Net::HTTP.new("www.baidu.com",80)resp,data = h.get("/")puts respputs data

不过resp可以取到,但data返回nil值,换其他网页同样如此.后来发现那是早期的方法返回值,新的ruby只返回一个值,我们可以用resp.body来访问网页内容,坑爹啊:

h = Net::HTTP.new("www.baidu.com",80)resp = h.get "/"puts resp.body

还可以用以下方法效果类似:

require 'uri'resp = Net::HTTP.get_response(URI("http://www.baidu.com/"))puts resp.body

注意用URI生成的url字符串要以http://开头,否则貌似有错.不过实际中我们要加错误处理和超时处理,否则你就且等吧:

#!/usr/bin/rubyrequire 'uri'require 'timeout'require 'net/http'$resp = $data = nilbegintimeout(5) {h = Net::HTTP.new(ARGV[0],80)$resp = h.get("/")#$resp = Net::HTTP.get_response(URI("http://"+ARGV[0]+"/"))}rescue => eputs e.inspectexitendputs $resp.body

运行结果如下:

wisy@wisy-ThinkPad-X61:~/src/ruby_src$ ./x.rb www.baidu.com|head -c 2000<!DOCTYPE html><!--STATUS OK--><html><head><meta http-equiv="content-type" content="text/html;charset=utf-8"><meta http-equiv="X-UA-Compatible" content="IE=Edge"><meta content="always" name="referrer"><link rel="dns-prefetch" href="//s1.bdstatic.com"/><link rel="dns-prefetch" href="//t1.baidu.com"/><link rel="dns-prefetch" href="//t2.baidu.com"/><link rel="dns-prefetch" href="//t3.baidu.com"/><link rel="dns-prefetch" href="//t10.baidu.com"/><link rel="dns-prefetch" href="//t11.baidu.com"/><link rel="dns-prefetch" href="//t12.baidu.com"/><link rel="dns-prefetch" href="//b1.bdstatic.com"/><title>百度一下，你就知道</title><style index="index"  id="css_index">html,body{height:100%}html{overflow-y:auto}#wrapper{position:relative;_position:;min-height:100%}#head{padding-bottom:100px;text-align:center;*z-index:1}#ftCon{height:100px;position:absolute;bottom:44px;text-align:center;width:100%;margin:0 auto;z-index:0;overflow:hidden}#ftConw{width:720px;margin:0 auto}body{font:12px arial;text-align:;background:#fff}body,p,form,ul,li{margin:0;padding:0;list-style:none}body,form,#fm{position:relative}td{text-align:left}img{border:0}a{color:#00c}a:active{color:#f60}.bg{background-image:url(http://s1.bdstatic.com/r/www/cache/static/global/img/icons_3bfb8e45.png);background-repeat:no-repeat;_background-image:url(http://s1.bdstatic.com/r/www/cache/static/global/img/icons_f72fb1cc.gif)}.bg_tuiguang_browser{width:16px;height:16px;background-position:-600px 0;display:inline-block;vertical-align:text-bottom;font-style:normal;overflow:hidden;margin-right:5px}.bg_tuiguang_browser_big{width:56px;height:56px;position:absolute;left:10px;top:10px;background-position:-600px -24px}.bg_tuiguang_weishi{width:56px;height:56px;position:absolute;left:10px;top:10px;background-position:-672px -24px}.c-icon{display:inline-block;width:14px;height:14px;vertical-align:text-bottom;font-style normal;overflow:hidden;background:url(http://s1.bdstatic.com/r/www/cache/static/global/img/icons_3bfb8e45../x.rb:19:in `write': Broken pipe @ io_write - <STDOUT> (Errno::EPIPE)from ./x.rb:19:in `puts'from ./x.rb:19:in `puts'from ./x.rb:19:in `<main>'

0 0