python爬虫实战笔记---selenium爬取QQ空间说说并存至本地

关于selenium和PhantomJS的使用，见python爬虫实战笔记---selenium爬取QQ空间说说并存至本地（上）

以下，正式开始对QQ空间进行分析和爬取

站点分析：

此版本的QQ空间，是以笔者自己的QQ空间为基点，利用selenium驱动PhantomJS，从上往下按照时间顺序爬取好友发过的说说文字，按照好友名进行分类

进入QQ空间首页，选择账号密码登录

在Chrome调试器下，找到账号密码的输入框标签源码

利用selenium进行模拟登录

from  selenium import webdriverimport timebrowser = webdriver.Chrome()browser.get('https://user.qzone.qq.com')username =password =browser.switch_to_frame('login_frame')log = browser.find_element_by_id("switcher_plogin")log.click()time.sleep(1)username = browser.find_element_by_id('u')username.send_keys(username)ps = browser.find_element_by_id('p')ps.send_keys(password)btn = browser.find_element_by_id('login_button')time.sleep(1)btn.click()time.sleep(2)browser.get('https://user.qzone.qq.com/{}'.format(username))browser.quit()

OK，这样就实现了模拟登录

接下来，就要进入说说主页，实现爬虫了

from  selenium import webdriverimport timefrom bs4 import BeautifulSoupbrowser = webdriver.Chrome()browser.get('https://user.qzone.qq.com')user ='2413843731'password ='960306feng'browser.switch_to_frame('login_frame')log = browser.find_element_by_id("switcher_plogin")log.click()time.sleep(1)username = browser.find_element_by_id('u')username.send_keys(user)ps = browser.find_element_by_id('p')ps.send_keys(password)btn = browser.find_element_by_id('login_button')time.sleep(1)btn.click()time.sleep(2)browser.get('https://user.qzone.qq.com/{}'.format(username))friendAction = browser.find_element_by_id('tab_menu_friend')time.sleep(5)html = browser.page_sourcesoup = BeautifulSoup(html,"html.parser")a = soup.find_all('a', attrs={'data-clicklog': 'nick'})for i in range(len(a)):    if(b[i].text == None):        b[i].text = '无法显示'    print(a[i].text + '\n' + b[i].text)