[Python]网络数据采集概述(3)—穿越网页表单、登录窗口进行采集

Python Requests库提交表单

params = {"firstname": "Liu", "lastname": "Vi"}r = requests.post("http://pythonscraping.com/files/processing.php", data=params)print(r.text)

如果不了解字段name、value或者提交的路径，可以通过查看网页源码或者控制台中查看Network

提交文件和图像

files = {'uploadFile': open("1.jpg", 'rb')}r = requests.post("http://pythonscraping.com/files/processing2.php", files= files)print(r.text)

处理登陆和Cookie、Session、Http基本认证

Cookie

params = {"username": "vi", "password": "password"}    r = requests.post("http://pythonscraping.com/pages/cookies/welcome.php", data= params)    print("Cookie is set to: ")    print(r.cookies.get_dict())    print("--------------------")    print("Going to profile page...")    r = requests.get("http://pythonscraping.com/pages/cookies/profile.php", cookies= r.cookies)    print(r.text)

Session

session = requests.Session()params = {'username': 'vi', 'password': 'password'}s = session.post("http://pythonscraping.com/pages/cookies/welcome.php", data= params)print("Cookie is set to: ")print(s.cookies.get_dict())print("-------------------")print("Going to profile page...")s = session.get("http://pythonscraping.com/pages/cookies/profile.php")print(s.text)print(session.headers)print('---------------')print(session.cookies)

HTTP基本接入认证

auth = HTTPBasicAuth('vi', 'password')r = requests.post(url= "http://pythonscraping.com/pages/auth/login.php", auth= auth)print(r.text)

---

参考书籍：
《Python网络数据采集》