用爬虫来爬取廖老师的python教程的url

输入汉字，能够查询到知识点的url，就像下面这样

请输入你要查询的知识点：函数h s调用函数:    https://www.liaoxuefeng.com/wiki/0014316089557264a6b348958f449949df42a6d3a2e542c000/0014316784721058975e02b46cc45cb836bb0827607738d000定义函数:    https://www.liaoxuefeng.com/wiki/0014316089557264a6b348958f449949df42a6d3a2e542c000/001431679203477b5b364aeba8c4e05a9bd4ec1b32911e2000函数的参数:    https://www.liaoxuefeng.com/wiki/0014316089557264a6b348958f449949df42a6d3a2e542c000/001431752945034eb82ac80a3e64b9bb4929b16eeed1eb9000递归函数:    https://www.liaoxuefeng.com/wiki/0014316089557264a6b348958f449949df42a6d3a2e542c000/001431756044276a15558a759ec43de8e30eb0ed169fb11000

这里要用到pinyin这个库，文档在这对于这个库不需要了解很多，只要会用就行。如何使程序的运行结果像上面那样，可以先观察一下网站源码

 <li id="001431608990315a01b575e2ab041168ff0df194698afac000" style="margin-left:1em;">            <a href="/wiki/0014316089557264a6b348958f449949df42a6d3a2e542c000/001431608990315a01b575e2ab041168ff0df194698afac000">Python简介</a>        </li>        <li id="0014316090478912dab2a3a9e8f4ed49d28854b292f85bb000" style="margin-left:1em;">            <a href="/wiki/0014316089557264a6b348958f449949df42a6d3a2e542c000/0014316090478912dab2a3a9e8f4ed49d28854b292f85bb000">安装Python</a>        </li>        <li id="00143161198846783e33de56d4041058c3dfc7e44ee1203000" style="margin-left:2em;">            <a href="/wiki/0014316089557264a6b348958f449949df42a6d3a2e542c000/00143161198846783e33de56d4041058c3dfc7e44ee1203000">Python解释器</a>        </li>        <li id="001431611988455689d4c116b2c4ed6aec000776c00ed52000" style="margin-left:1em;">            <a href="/wiki/0014316089557264a6b348958f449949df42a6d3a2e542c000/001431611988455689d4c116b2c4ed6aec000776c00ed52000">第一个Python程序</a>        </li>        <li id="0014316399410395f704750ee9440228135925a6ca1dad8000" style="margin-left:2em;">            <a href="/wiki/0014316089557264a6b348958f449949df42a6d3a2e542c000/0014316399410395f704750ee9440228135925a6ca1dad8000">使用文本编辑器</a>        </li>

可以观察到每一个小的章节<li>标签的style属性和章节下面的每一小节是不一样的，因此这就是突破口。每一个小的章节的li标签为<li id="..." style="margin-left:1em;">，小的章节下面的小节li标签为<li id="..." style="margin-left:2em;">。获取小的章节之后要用兄弟节点来获取每一小节。if detial_tag == '\n':这段代码的解释在这。

for chapter_tag in original_tag.find_all("li", style="margin-left:1em;"):    '''    首先要获取每一个小的章节的章节名字，并转换为拼音存储    '''    L[sign].append(pinyin.get_initial(chapter_tag.find("a").get_text()))    '''    在每一个小的章节后面紧跟的兄弟标签就是小的章节下面的每一小节    当提取的li标签的style属性！="margin-left:2em，即进入下一章节（即进入下一轮循环）    '''    for detial_tag in chapter_tag.next_siblings:        if detial_tag == '\n':            pass        else:            style = detial_tag.get("style")            if style == 'margin-left:1em;':                break            else:                L[sign].append('\n' + detial_tag.find("a").get_text() + ':    ' + 'https://www.liaoxuefeng.com' + detial_tag.find("a").get("href"))    sign += 1    

要用列表来存储获取的数据

L = []for all_tag in original_tag.find_all("li", style="margin-left:1em;"):    count += 1    L.append([])

通过输入汉字来获取url的函数

def judgle(pinyin):    for i in range(sign):        if pinyin == L[i][0]:            for n in range(len(L[i])):                print(L[i][n])

下面就是所有的代码了

import urllib.requestimport pinyinfrom bs4 import BeautifulSoupdef judgle(pinyin):    for i in range(sign):        if pinyin == L[i][0]:            for n in range(len(L[i])):                print(L[i][n])url = 'https://www.liaoxuefeng.com/wiki/0014316089557264a6b348958f449949df42a6d3a2e542c000'response = urllib.request.urlopen(url)html = response.read().decode('utf-8')url_soup = BeautifulSoup(html, 'html.parser')original_tag = url_soup.find_all("ul", class_="uk-nav-side")[1]count = 0L = []for all_tag in original_tag.find_all("li", style="margin-left:1em;"):    count += 1    L.append([])sign = 0for chapter_tag in original_tag.find_all("li", style="margin-left:1em;"):    L[sign].append(pinyin.get_initial(chapter_tag.find("a").get_text()))    for detial_tag in chapter_tag.next_siblings:        if detial_tag == '\n':            pass        else:            style = detial_tag.get("style")            if style == 'margin-left:1em;':                break            else:                L[sign].append('\n' + detial_tag.find("a").get_text() + ':    ' + 'https://www.liaoxuefeng.com' + detial_tag.find("a").get("href"))    sign += 1                chinese = input('请输入你要查询的知识点：')pinyin = pinyin.get_initial(chinese)judgle(pinyin)

运行结果

请输入你要查询的知识点：实战s zDay 1 - 搭建开发环境:    https://www.liaoxuefeng.com/wiki/0014316089557264a6b348958f449949df42a6d3a2e542c000/001432170937506ecfb2f6adf8e4757939732f3e32b781c000Day 2 - 编写Web App骨架:    https://www.liaoxuefeng.com/wiki/0014316089557264a6b348958f449949df42a6d3a2e542c000/00143217133614028a244ea855b40a586b551c616d3b2c9000Day 3 - 编写ORM:    https://www.liaoxuefeng.com/wiki/0014316089557264a6b348958f449949df42a6d3a2e542c000/0014323389656575142d0bcfeec434e9639a80d3684a7da000Day 4 - 编写Model:    https://www.liaoxuefeng.com/wiki/0014316089557264a6b348958f449949df42a6d3a2e542c000/001432338991719a4c5c42ef08e4f44ad0f293ad728a27b000Day 5 - 编写Web框架:    https://www.liaoxuefeng.com/wiki/0014316089557264a6b348958f449949df42a6d3a2e542c000/001432339008728d0ddbe19ee594980be3f0644a9371894000Day 6 - 编写配置文件:    https://www.liaoxuefeng.com/wiki/0014316089557264a6b348958f449949df42a6d3a2e542c000/001432339034336cbf72acd43354d72831461e3871d9f2e000Day 7 - 编写MVC:    https://www.liaoxuefeng.com/wiki/0014316089557264a6b348958f449949df42a6d3a2e542c000/001432339095180ce91c53cdab841bfa9c342a297b886fe000Day 8 - 构建前端:    https://www.liaoxuefeng.com/wiki/0014316089557264a6b348958f449949df42a6d3a2e542c000/001432339124159f00f6ab876c44349a3fd8eb26d0c291e000Day 9 - 编写API:    https://www.liaoxuefeng.com/wiki/0014316089557264a6b348958f449949df42a6d3a2e542c000/0014323391480651a75b5fda4cb4c789208191682fc2c70000Day 10 - 用户注册和登录:    https://www.liaoxuefeng.com/wiki/0014316089557264a6b348958f449949df42a6d3a2e542c000/001432339169382f45b9bd7b45d47ceb3e2b42846e0e991000Day 11 - 编写日志创建页:    https://www.liaoxuefeng.com/wiki/0014316089557264a6b348958f449949df42a6d3a2e542c000/00143233918656129f4ad3ac29e4f728dc72b5d2368215a000Day 12 - 编写日志列表页:    https://www.liaoxuefeng.com/wiki/0014316089557264a6b348958f449949df42a6d3a2e542c000/001432339210950e063b4795d574036bc5dcf0c2449bc52000Day 13 - 提升开发效率:    https://www.liaoxuefeng.com/wiki/0014316089557264a6b348958f449949df42a6d3a2e542c000/001432339228196a8eb6fb8832b48b5aa0d740346536ead000Day 14 - 完成Web App:    https://www.liaoxuefeng.com/wiki/0014316089557264a6b348958f449949df42a6d3a2e542c000/001432339247097eea476bf61f8496092cc1b663eae1848000Day 15 - 部署Web App:    https://www.liaoxuefeng.com/wiki/0014316089557264a6b348958f449949df42a6d3a2e542c000/0014323392805925d5b69ddad514511bf0391fe2a0df2b0000Day 16 - 编写移动App:    https://www.liaoxuefeng.com/wiki/0014316089557264a6b348958f449949df42a6d3a2e542c000/001432339330096121ae7e38be44570b7fbd0d8faae26f6000

这样想找哪个知识点就可以通过程序来获取url，而不是每天都要打开浏览器–>点击网址–>上下翻动来获取想要的知识点。考虑到如果输入每一小节并获取url的实用性实在太低，毕竟很少有人把每一小节的名字全都记住，因此就通过查询小的章节来获取url。