Python 爬虫学习：爬取LeetCode的题目并且按照难度分类

来源：互联网发布：淘宝usa商城假货编辑：程序博客网时间：2024/05/16 06:40

过程：

爬取LeeCode的所有题目，按照难度的分类。在爬取题号的时候比较麻烦，要判断一个字符串是不是整数，首先想到的是用int()但是会有“未处理的意外”发生，可以用”try…except”来解决。还有一个难点就是很难通过一次就直接扫描到关键，所以我Find()了很多次，效率比较低，最后按照题号为主键存到了字典里。

Code:

# -*- coding: utf-8 -*-__author__ = 'bigship'import urllib2import stringdef func(x):#判断一个字符串是不是整数    try:        x=int(x)        return x    except ValueError:        return Falseurl = 'https://leetcode.com/problemset/algorithms/'Easy = {}Medium = {}Hard={}ALL={}head = '''<a href="/problems/'''tail = "</a>"headnum='<td>'tailnum="</td>"response = urllib2.urlopen(url)html = response.read()numA=0numE=0numM=0numH=0numhead=html.find(headnum)while numhead!=-1:    numtail = html.find(tailnum,numhead+1)    if func(html[numhead+len(headnum):numtail])!=False:        numA+=1        poshead = html.find(head,numtail+1)        postail =html.find(tail,poshead+1)        poshead = html.find('''/">''',poshead+1)        pos1 = html.find("""<td value='""",postail+1)        pos1 = html.find(">",pos1+1)        pos2 = html.find("</td>",pos1+1)        print html[numhead+len(headnum):numtail]+" "+html[poshead+3:postail]+" "+html[pos1+1:pos2]        ALL[func(html[numhead+len(headnum):numtail])]=html[poshead+3:postail]        if html[pos1+1:pos2]=='Easy':            Easy[func(html[numhead+len(headnum):numtail])]=html[poshead+3:postail]            numE+=1        if html[pos1+1:pos2]=='Medium':            Medium[func(html[numhead+len(headnum):numtail])]=html[poshead+3:postail]            numM+=1        if html[pos1+1:pos2]=='Hard':            Hard[func(html[numhead+len(headnum):numtail])]=html[poshead+3:postail]            numH+=1    numhead=html.find(headnum,numhead+1)print "总题数 ："+str(numA)print "简单题数 ："+str(numE)print "中等题数 ："+str(numM)print "难题数 ："+str(numH)

1 0