scrapy框架拓展:为download_middleware添加用户代理池
来源:互联网 发布:ps淘宝详情页模板 编辑:程序博客网 时间:2024/05/22 12:41
**在与spider同目录下建立user_agent.py**#coding:utf-8"""避免被ban策略之一:使用useragent池。使用注意:需在settings.py中进行相应的设置。"""from scrapy import logimport randomfrom scrapy.downloadermiddlewares.useragent import UserAgentMiddlewareclass RotateUserAgentMiddleware(UserAgentMiddleware): # for more user agent strings,you can find it in http://www.useragentstring.com/pages/useragentstring.php user_agent_list = [ "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.1 " "(KHTML, like Gecko) Chrome/22.0.1207.1 Safari/537.1", "Mozilla/5.0 (X11; CrOS i686 2268.111.0) AppleWebKit/536.11 " "(KHTML, like Gecko) Chrome/20.0.1132.57 Safari/536.11", "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/536.6 " "(KHTML, like Gecko) Chrome/20.0.1092.0 Safari/536.6", "Mozilla/5.0 (Windows NT 6.2) AppleWebKit/536.6 " "(KHTML, like Gecko) Chrome/20.0.1090.0 Safari/536.6", "Mozilla/5.0 (Windows NT 6.2; WOW64) AppleWebKit/537.1 " "(KHTML, like Gecko) Chrome/19.77.34.5 Safari/537.1", "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/536.5 " "(KHTML, like Gecko) Chrome/19.0.1084.9 Safari/536.5", "Mozilla/5.0 (Windows NT 6.0) AppleWebKit/536.5 " "(KHTML, like Gecko) Chrome/19.0.1084.36 Safari/536.5", "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/536.3 " "(KHTML, like Gecko) Chrome/19.0.1063.0 Safari/536.3", "Mozilla/5.0 (Windows NT 5.1) AppleWebKit/536.3 " "(KHTML, like Gecko) Chrome/19.0.1063.0 Safari/536.3", "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_8_0) AppleWebKit/536.3 " "(KHTML, like Gecko) Chrome/19.0.1063.0 Safari/536.3", "Mozilla/5.0 (Windows NT 6.2) AppleWebKit/536.3 " "(KHTML, like Gecko) Chrome/19.0.1062.0 Safari/536.3", "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/536.3 " "(KHTML, like Gecko) Chrome/19.0.1062.0 Safari/536.3", "Mozilla/5.0 (Windows NT 6.2) AppleWebKit/536.3 " "(KHTML, like Gecko) Chrome/19.0.1061.1 Safari/536.3", "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/536.3 " "(KHTML, like Gecko) Chrome/19.0.1061.1 Safari/536.3", "Mozilla/5.0 (Windows NT 6.1) AppleWebKit/536.3 " "(KHTML, like Gecko) Chrome/19.0.1061.1 Safari/536.3", "Mozilla/5.0 (Windows NT 6.2) AppleWebKit/536.3 " "(KHTML, like Gecko) Chrome/19.0.1061.0 Safari/536.3", "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/535.24 " "(KHTML, like Gecko) Chrome/19.0.1055.1 Safari/535.24", "Mozilla/5.0 (Windows NT 6.2; WOW64) AppleWebKit/535.24 " "(KHTML, like Gecko) Chrome/19.0.1055.1 Safari/535.24" ] def process_request(self, request, spider): ua = random.choice(self.user_agent_list) if ua: # 显示当前使用的useragent print "********Current UserAgent:%s************" % ua # 记录 log.msg('Current UserAgent: ' + ua) request.headers.setdefault('User-Agent', ua)**最后在setting中设置**DOWNLOADER_MIDDLEWARES = {'scrapy.contrib.downloadermiddleware.useragent.UserAgentMiddleware' : None,#禁用默认的代理'demo.UserAgent.RotateUserAgentMiddleware':400 #启用自己的用户代理
0 0
- scrapy框架拓展:为download_middleware添加用户代理池
- 给Scrapy添加代理
- Scrapy爬虫框架使用IP代理池
- scrapy拓展
- python 爬虫scrapy框架 自建IP代理池
- 为php添加拓展模块
- scrapy设置代理池
- 为NPM添加代理
- 为wget添加代理
- 为添加用户偷哈懒
- 添加用户为sudoer
- 为Tomcat添加用户
- scrapy代理
- 为apt-get添加代理
- 为MYSQL添加远程用户
- 为用户控件添加属性
- 为Mysql添加远程用户
- 为MYSQL添加远程用户
- linux cfs调度器_理论模型
- hiho一下 第140周-清理海报(DAG+dfs)
- IP头、TCP头、UDP头
- 问题总结一
- LeetCode 53. Maximum Subarray 题解
- scrapy框架拓展:为download_middleware添加用户代理池
- 协议 protocol
- win10使用genymotion出现Unable to start the virtual device
- leveldb学习笔记一 SLICE
- swift中字符的长度
- springboot war包部署
- Manacher's ALGORITHM: O(n)时间求字符串的最长回文子串
- C#入门小程序 猜数字
- Python基础语法——函数(四)