Python 网络爬虫系列(二) --- 伪装成浏览器访问

来源：互联网发布：苹果手机移动数据开关编辑：程序博客网时间：2024/04/30 01:13

Python 网络爬虫伪装成浏览器访问

普通的爬一个网页只需要3部就能搞定

url = r'http://blog.csdn.net/jinzhichaoshuiping/article/details/43372839'

sock = urllib.urlopen(url)

html = sock.read()

sock.close()

得到的html是字符类型的,可以输出打印,也可以保存进文档.

f = file('CSDN.html', 'w')

f.write(html)

f.close()

但也有搞不定的时候,比如去爬CSDN的博客,会得到403 Forbidden.

这时候需要将爬虫伪装成浏览器对网页进行访问

def openurl(self,url):        """        打开网页        """        cookie_support= urllib2.HTTPCookieProcessor(cookielib.CookieJar())        self.opener = urllib2.build_opener(cookie_support,urllib2.HTTPHandler)        urllib2.install_opener(self.opener)        user_agents = [                    'Mozilla/5.0 (Windows; U; Windows NT 5.1; it; rv:1.8.1.11) Gecko/20071127 Firefox/2.0.0.11',                    'Opera/9.25 (Windows NT 5.1; U; en)',                    'Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; .NET CLR 1.1.4322; .NET CLR 2.0.50727)',                    'Mozilla/5.0 (compatible; Konqueror/3.5; Linux) KHTML/3.5.5 (like Gecko) (Kubuntu)',                    'Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.8.0.12) Gecko/20070731 Ubuntu/dapper-security Firefox/1.5.0.12',                    'Lynx/2.8.5rel.1 libwww-FM/2.14 SSL-MM/1.4.1 GNUTLS/1.2.9',                    "Mozilla/5.0 (X11; Linux i686) AppleWebKit/535.7 (KHTML, like Gecko) Ubuntu/11.04 Chromium/16.0.912.77 Chrome/16.0.912.77 Safari/535.7",                    "Mozilla/5.0 (X11; Ubuntu; Linux i686; rv:10.0) Gecko/20100101 Firefox/10.0 ",                    ]                agent = random.choice(user_agents)        self.opener.addheaders = [("User-agent",agent),("Accept","*/*"),('Referer','http://www.google.com')]        try:            res = self.opener.open(url)            htmlSource =  res.read()            res.close()            #print htmlSource            f = file('CSDN.html', 'w')            f.write(htmlSource)            f.close()            return htmlSource        except Exception,e:            self.speak(str(e)+url)            raise Exception        else:            return res

参考博客http://www.yihaomen.com/article/python/210.htm

0 0