Python爬虫抓取代理IP并检验可用性

来源：互联网发布：淘宝大学学富电商编辑：程序博客网时间：2024/04/27 15:46

经常写爬虫，难免会遇到ip被目标网站屏蔽的情况，银次一个ip肯定不够用，作为节约的程序猿，能不花钱就不花钱，那就自己去找吧，这次就写了下抓取西刺代理上的ip，但是这个网站也反爬！！！至于如何应对，我觉得可以通过增加延时试试，可能是我抓取的太频繁了，所以被封IP了。但是，还是可以去IP巴士试试的，条条大路通罗马嘛，不能吊死在一棵树上。不废话，上代码。

#!/usr/bin/env python# -*- coding:utf8 -*-import urllib2import timefrom bs4 import BeautifulSoupimport sysreload(sys)sys.setdefaultencoding( "utf-8" )req_header = {'User-Agent':'Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.11 (KHTML, like Gecko) Chrome/23.0.1271.64 Safari/537.11',  'Accept':'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8',  #'Accept-Language': 'en-US,en;q=0.8,zh-Hans-CN;q=0.5,zh-Hans;q=0.3',  'Accept-Charset':'ISO-8859-1,utf-8;q=0.7,*;q=0.3',  'Accept-Encoding':'en-us',  'Connection':'keep-alive',  'Referer':'http://www.baidu.com/'   }req_timeout = 5testUrl = "http://www.baidu.com/"testStr = "wahaha"file1 = open('proxy.txt' , 'w')# url = ""# req = urllib2.Request(url,None,req_header)# jsondatas = urllib2.urlopen(req,None,req_timeout).read()cookies = urllib2.HTTPCookieProcessor()checked_num = 0grasp_num = 0for page in range(1, 160):    req = urllib2.Request('http://www.xici.net.co/nn/' + str(page), None, req_header)    html_doc = urllib2.urlopen(req, None, req_timeout).read()    # html_doc = urllib2.urlopen('http://www.xici.net.co/nn/' + str(page)).read()    soup = BeautifulSoup(html_doc)    trs = soup.find('table', id='ip_list').find_all('tr')    for tr in trs[1:]:        tds = tr.find_all('td')        ip = tds[1].text.strip()        port = tds[2].text.strip()        protocol = tds[5].text.strip()        if protocol == 'HTTP' or protocol == 'HTTPS':            #of.write('%s=%s:%s\n' % (protocol, ip, port))            print '%s=%s:%s' % (protocol, ip, port)            grasp_num +=1            proxyHandler = urllib2.ProxyHandler({"http": r'http://%s:%s' % (ip, port)})            opener = urllib2.build_opener(cookies, proxyHandler)            opener.addheaders = [('User-Agent',                                  'Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/31.0.1650.63 Safari/537.36')]            t1 = time.time()            try:                req = opener.open(testUrl, timeout=req_timeout)                result = req.read()                timeused = time.time() - t1                pos = result.find(testStr)                if pos > 1:                    file1.write(protocol+"\t"+ip+"\t"+port+"\n")                    checked_num+=1                    print checked_num, grasp_num                else:                    continue            except Exception,e:                continuefile1.close()print checked_num,grasp_num

个人感觉代码里没有太复杂的，就没有加注释，相信大家基本可以理解，如有问题也请多批评指正，共同进步！

1 0