第六课 Python新浪新闻爬虫最终整理总结

代码整理封装如图：

6不6？

Python简洁又强大！

至此，教程圆满结束，还有什么不懂的或有疑问的问题，欢迎大家加我的QQ：1099718640

顺便再送上完整代码，凑凑字数，哈哈~（其实推荐大家去下载Github上的内容，顺便给个小心心什么的）

#第六课 封装整理#作者：DYBOY#时间：2017-09-06import requestsimport jsonimport refrom bs4 import BeautifulSoupfrom datetime import datetime#以上作为基本引用#获取评论数量def getCommentCounts(newsurl):    newsid = re.search('doc-i(.*).shtml',newsurl).group(1)    commentURL='http://comment5.news.sina.com.cn/page/info?version=1&format=js&channel=gn&newsid=comos-{}&group=&compress=0&ie=utf-8&oe=utf-8&page=1&page_size=20'    comments = requests.get(commentURL.format(newsid))    com_num = json.loads(comments.text.strip('var data='))['result']['count']['total']    return com_num#获取新闻详情页def getNewsDetail(newsurl):    result = {}    res = requests.get(newsurl)    res.encoding = 'utf-8'    soup = BeautifulSoup(res.text,'html.parser')    result['title'] = soup.select('#artibodyTitle')[0].text    result['time'] = datetime.strptime(soup.select('.time-source')[0].contents[0].strip(),'%Y年%m月%d日%H:%M')    result['source'] = soup.select('.time-source span a')[0].text    result['article'] = ' '.join([p.text.strip() for p in soup.select('#artibody p')[:-1]])    result['editor'] = soup.select('.article-editor')[0].text.lstrip('责任编辑：')    result['com_num'] = getCommentCounts(newsurl)    return result#获取新闻列表def getNewsList(sortURL):    res = requests.get(sortURL)    res.encoding = 'utf-8'    soup = BeautifulSoup(res.text,'html.parser')    for news in soup.select('.news-item'):        if( len(news.select('h2')) >0 ):            h2 = news.select('h2')[0].text            time = news.select('.time')[0].text            newsurl = news.select('a')[0]['href']            wenzhang = getNewsDetail(newsurl)            print(time,h2,newsurl,wenzhang)#MAIN-获取新闻getNewsList('http://news.sina.com.cn/china/')

教程到这里就结束了，但是如果大家还有什么不懂的欢迎联系我，同时也希望各位能有所较大收获！

祝各位学业有成，事事顺心！

注：本文属于原创文章，转载请注明本文地址！

作者QQ:1099718640

CSDN博客主页：http://blog.csdn.net/dyboy2017

Github开源项目：https://github.com/dyboy2017/spider