驴妈妈景点爬取

这次用BeautifulSoup4和requests来爬取驴妈妈上的景点信息。
主要用到的模块：
- requests
- BeautifulSoup
- selenium

---

爬虫思路

1、获取城市名并构造所有城市地点的链接 get_all_cities_url()
3、爬取每个地点的第一页并获取最大页数 parse_one_page( (url, choice=0) ),
4、构造每个城市从第二页开始每一页的链接 generate_url(current_url, pagenum)
5、访问从第二页开始每一页链接爬取景点 parse_one_page( (url, choice=1) ) 多进程

我找到的网站链接是http://ticket.lvmama.com/
对于一个地方的景点链接http://s.lvmama.com/ticket/?keyword=上海&k=0#list
可以看到是需要在第一个链接的搜索框中输入地点名的，所以需要首先去获取所有地点名。

这个页面貌似直接request请求获取不到，所以我用了phantomjs来模拟浏览器渲染来获取
，在测试过程中发现需要点击ABCD这个tab之后地点才全部出现。
所以对于这个页面需要做的就是1、点击上海展开下方的列表。2、点击ABCD tab来使所有地点加载到页面上。
以下放上部分代码：

#简单的下载函数def downloader(url):    retry = 0    while retry < RETRY_TIMES:        time.sleep(DOWNLOAD_DELAY)        USER_DEFINE_HEADER['User-Agent'] = random.choice(UA_LIST)        try:            response = requests.get(url=url, headers=USER_DEFINE_HEADER, timeout=10)            if response.status_code in RETRY_CODE_LIST:                retry += 1                log("Error in downloading, retry time %d, status code %d" % (retry, response.status_code))            else:                return response.text        except RequestException:            retry += 1            log("Error in downloading, retry time %d, request exception" % retry)            pass    return None#获得所有城市名的具体函数def get_all_cities_url():    browser = webdriver.PhantomJS()    start_url = "http://ticket.lvmama.com/"    browser.get(start_url)    browser.find_element_by_xpath('/html/body/div[4]/div/div[1]/div[2]/div[1]/div[1]/div/div[1]').click()    #点击上海展开    browser.find_element_by_xpath("/html/body/div[4]/div/div[1]/div[2]/div[1]/div[1]/div/div[2]/ul[1]/li[2]").click()   #点击ABCD    html = browser.page_source    if html is None:        log("GET ticket.lvmama.com ERROR!")    else:        for city in get_city_name(html):            yield 'http://s.lvmama.com/ticket/?keyword=%s&tabType=ticket#list' % city

通过get_all_cities_url构造了所有城市的链接，接下来就是开多进程逐个去访问了。

由于写的比较简单，没有用队列来存储requests，所以接下来的多进程函数需要传入两个参数：
1、当前爬取的地点链接
2、要爬取的页索引
如上海的第一页为http://s.lvmama.com/ticket/?keyword=上海&tabType=ticket#list
第二页就是http://s.lvmama.com/ticket/P2?keyword=上海&tabType=ticket#list

先来看下调用多进程的函数

def generate_url(current_url, pagenum):    ret = []    url_temp = current_url.split('?')    for i in range(2, pagenum+1):        ret.append('%s?%s' % (url_temp[0] + 'P' + str(i), url_temp[1]))    return retdef main():    '''        1.构造每个城市的搜索URL        2.爬取第一页，并获得最大页数        3.从第二页开始用多进程爬取        :return:    '''    pool = Pool(3)    for city_search_url in get_all_cities_url():        pagenum = int(parse_one_page((city_search_url, 0)))        if pagenum == -1:            continue        url_list = generate_url(city_search_url, pagenum)        pool.map(parse_one_page, zip(url_list, [1]*(pagenum-1)))  # 从第二页开始多进程抓取    time.sleep(5)    log('Spider finished...\n')    print('Spider finished...')    pool.close()

这里用了multiprocessing的pool.map。generate_url函数需要传入上面所说的两个参数。generate_url函数返回就是一个地点所有页数的链接。将这个链接列表再传给parse_one_page函数就行了。
以下是parser函数具体的代码，传入的是链接和choice的元组，链接就是需要爬取第几页，而choice为0时是返回最大页数，时只爬取item。
zip用于打包两个参数列表。

def parse_one_page(url_choice):    '''    解析页面    :param url_choice: 包含url和choice的元组    :return: 最大页数    '''    time.sleep(random.randint(1, 8))    html = downloader(url_choice[0])    if html is None:        return -1    soup = BeautifulSoup(html, 'lxml')    sights = soup.select('.product-list div.product-regular.clearfix')    for sight in sights:        name = sight.select('.name')        id = sight.select('.name')        city = sight.select('.city')        _city = city[0].get_text().strip(' |\t|[|]') if len(city) else ''        address = sight.select('dl.product-details.clearfix > dd')        _address = address[0].attrs['title'].strip() if len(address) and address[0].has_attr('title') else ''        price = sight.select('.product-price > em')        level = sight.select('.level')        location = get_location(_city.split('·')[0] if city != '' else '', _address)        item = LvmamaItem(            id[0].attrs['href'].strip() if len(id) else '',            name[0].attrs['title'].strip() if len(name) else '',            _city,            _address,            price[0].get_text().strip() if len(price) else '',            level[0].get_text().strip() if len(level) else '',            location        )        save(str(item))    if url_choice[1] == 0:        return get_max_pagenum(html)    log('crawl %s completed\n' % url_choice[0])    print('crawl %s completed' % url_choice[0])

这样就实现了每个城市地点的第二页开始用多进程爬取。

其中还需要获得最大页数，我这里是利用总景点数/每页数目，并向上取整实现。

具体代码

def get_max_pagenum(html):    '''    :param html: HTML文本    :return: 每个城市的最大页数    '''    soup = BeautifulSoup(html, 'lxml')    pagenum = soup.select('body > div.everything > div.main.clearfix > div.search-filter > div.search-nav-box.clearfix > p > a.active > b')[0].get_text()    return math.ceil(int(pagenum)/8)

config.py

USER_DEFINE_HEADER = {    'Host': 's.lvmama.com',    'Connection': 'keep-alive',    'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/62.0.3202.89 Safari/537.36',    'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8',    'Accept-Encoding': 'gzip, deflate',    'Accept-Language': 'zh-CN,zh;q=0.9',    'Cookie': ''}RETRY_CODE_LIST = [500, 502, 503, 504, 408, 403, 404]RETRY_TIMES = 8DOWNLOAD_DELAY = 12UA_LIST = [    'Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/50.0.2661.87 Safari/537.36',    'Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10_6_8; en-us) AppleWebKit/534.50 (KHTML, like Gecko) Version/5.1 Safari/534.50',    'Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/50.0.2661.87 Safari/537.36',    'Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/45.0.2454.101 Safari/537.36',    'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/46.0.2486.0 Safari/537.36 Edge/13.10586',    'Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/47.0.2526.80 Safari/537.36 Core/1.47.277.400 QQBrowser/9.4.7658.400',    'Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/38.0.2125.122 Safari/537.36 SE 2.X MetaSr 1.0',    'Mozilla/5.0 (Windows NT 10.0; WOW64; Trident/7.0; rv:11.0) like Gecko',    'Mozilla/5.0 (compatible; MSIE 10.0; Windows NT 6.1; WOW64; Trident/6.0)',    'Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; WOW64; Trident/5.0)',    'Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Maxthon/4.9.2.1000 Chrome/39.0.2146.0 Safari/537.36',    'Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/48.0.2564.116 UBrowser/5.6.12150.8 Safari/537.36',    'Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/42.0.2311.154 Safari/537.36 LBBROWSER',    'Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/48.0.2564.116 Safari/537.36 TheWorld 7',    'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/61.0.3163.100 Safari/537.36']