scrapy random user agents

来源：互联网发布：经济数据库都有什么编辑：程序博客网时间：2024/06/04 08:54

安装scrapy-random-useragent

sudo pip install scrapy-random-useragent

配置setting.py:

DOWNLOADER_MIDDLEWARES = {    'scrapy.contrib.downloadermiddleware.useragent.UserAgentMiddleware': None,    'random_useragent.RandomUserAgentMiddleware': 400}

USER_AGENT_LIST = "/path/to/useragents.txt"

创建useragents.txt文件

"Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.1(KHTML, like Gecko) Chrome/22.0.1207.1 Safari/537.1Mozilla/5.0 (X11; CrOS i686 2268.111.0) AppleWebKit/536.11(KHTML, like Gecko) Chrome/20.0.1132.57 Safari/536.11Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/536.6(KHTML, like Gecko) Chrome/20.0.1092.0 Safari/536.6Mozilla/5.0 (Windows NT 6.2) AppleWebKit/536.6(KHTML, like Gecko) Chrome/20.0.1090.0 Safari/536.6Mozilla/5.0 (Windows NT 6.2; WOW64) AppleWebKit/537.1(KHTML, like Gecko) Chrome/19.77.34.5 Safari/537.1Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/536.5(KHTML, like Gecko) Chrome/19.0.1084.9 Safari/536.5Mozilla/5.0 (Windows NT 6.0) AppleWebKit/536.5(KHTML, like Gecko) Chrome/19.0.1084.36 Safari/536.5Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/536.3(KHTML, like Gecko) Chrome/19.0.1063.0 Safari/536.3Mozilla/5.0 (Windows NT 5.1) AppleWebKit/536.3(KHTML, like Gecko) Chrome/19.0.1063.0 Safari/536.3Mozilla/5.0 (Macintosh; Intel Mac OS X 10_8_0) AppleWebKit/536.3(KHTML, like Gecko) Chrome/19.0.1063.0 Safari/536.3Mozilla/5.0 (Windows NT 6.2) AppleWebKit/536.3(KHTML, like Gecko) Chrome/19.0.1062.0 Safari/536.3Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/536.3(KHTML, like Gecko) Chrome/19.0.1062.0 Safari/536.3Mozilla/5.0 (Windows NT 6.2) AppleWebKit/536.3(KHTML, like Gecko) Chrome/19.0.1061.1 Safari/536.3Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/536.3(KHTML, like Gecko) Chrome/19.0.1061.1 Safari/536.3Mozilla/5.0 (Windows NT 6.1) AppleWebKit/536.3(KHTML, like Gecko) Chrome/19.0.1061.1 Safari/536.3Mozilla/5.0 (Windows NT 6.2) AppleWebKit/536.3(KHTML, like Gecko) Chrome/19.0.1061.0 Safari/536.3Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/535.24(KHTML, like Gecko) Chrome/19.0.1055.1 Safari/535.24Mozilla/5.0 (Windows NT 6.2; WOW64) AppleWebKit/535.24(KHTML, like Gecko) Chrome/19.0.1055.1 Safari/535.24

0 0