Python Mechanize Cheat Sheet
来源:互联网 发布:linux查看显卡命令 编辑:程序博客网 时间:2024/05/18 03:55
Python Mechanize Overview
- Python mechanize site http://wwwsearch.sourceforge.net/mechanize/
- mechanize docs http://wwwsearch.sourceforge.net/mechanize/doc.html
- ClientForm docs http://wwwsearch.sourceforge.net/old/ClientForm/
- BeautifulSoup http://www.crummy.com/software/BeautifulSoup/
Mechanize cheat sheet1
Create a browser object
import mechanizebr = mechanize.Browser()br.set_all_readonly(False) # allow everything to be written tobr.set_handle_robots(False) # no robotsbr.set_handle_refresh(False) # can sometimes hang without thisbr.addheaders = [('User-agent', 'Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.0.1) Gecko/2008071615 Fedora/3.0.1-1.fc9 Firefox/3.0.1')]# [('User-agent', 'Firefox')]
Open a webpage and inspect its contents
response = br.open(url)print response.read() # the text of the pageresponse1 = br.response() # get the response againprint response1.read() # can apply lxml.html.fromstring()
Using forms
#List the formsfor form in br.forms(): print "Form name:", form.name print form#select formbr.select_form("form1") # works when form has a namebr.form = list(br.forms())[0] # use when form is unnamed#login form examplebr.select_form("login") br['login:loginUsernameField'] = userbr['login:password'] = passwordbr.method = "POST"response = br.submit()
Using Controls
# Iterate through the controls in the form.for control in br.form.controls: print control print "type=%s, name=%s value=%s" % (control.type, control.name, br[control.name])# Controls can be found by namecontrol = br.form.find_control("controlname")# Having a select control tells you what values can be selectedif control.type == "select": # means it's class ClientForm.SelectControl for item in control.items: print " name=%s values=%s" % (item.name, str([label.text for label in item.get_labels()]))# Because Select type controls can have multiple selections, they must be set with a list, even if it is one element.print control.valueprint control # selected value is starredcontrol.value = ["ItemName"]print controlbr[control.name] = ["ItemName"] # equivalent and more normal# Text controls can be set as a stringif control.type == "text": # means it's class ClientForm.TextControl control.value = "stuff here"br["controlname"] = "stuff here" # equivalent# Controls can be set to readonly and disabled (sometimes necessary for superfluous submit buttons).control.readonly = Falsecontrol.disabled = True# OR disable all of them like sofor control in br.form.controls: if control.type == "submit": control.disabled = True
Using Links
#Iterate the linksfor link in br.links(): print link.text, link.url# Follow link and click links is the same as submit and clickrequest = br.click_link(link)response = br.follow_link(link)print response.geturl()
Using Cookie jars
request = br.click() # creates the request objectcj = mechanize.CookieJar()br2 = mechanize.Browser()br2.set_cookiejar(cj)br2.open(request)
More example
emulate a browser
import mechanizeimport cookielib# Browserbr = mechanize.Browser()# Cookie Jarcj = cookielib.LWPCookieJar()br.set_cookiejar(cj)# Browser optionsbr.set_handle_equiv(True)br.set_handle_gzip(True)br.set_handle_redirect(True)br.set_handle_referer(True)br.set_handle_robots(False)# Follows refresh 0 but not hangs on refresh > 0br.set_handle_refresh(mechanize._http.HTTPRefreshProcessor(), max_time=1)# Want debugging messages?#br.set_debug_http(True)#br.set_debug_redirects(True)#br.set_debug_responses(True)# User-Agent (this is cheating, ok?)br.addheaders = [('User-agent', 'Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.0.1) Gecko/2008071615 Fedora/3.0.1-1.fc9 Firefox/3.0.1')]
access a password protected site
# If the protected site didn't receive the authentication data you would# end up with a 410 error in your facebr.add_password('http://safe-site.domain', 'username', 'password')br.open('http://safe-site.domain')
Downloading a file
# Downloadf = br.retrieve('http://www.google.com.br/intl/pt-BR_br/images/logo.gif')[0]print ffh = open(f)
Setting a proxy
# Proxy and user/passwordbr.set_proxies({"http": "joe:password@myproxy.example.com:3128"})# Proxybr.set_proxies({"http": "myproxy.example.com:3128"})# Proxy passwordbr.add_proxy_password("joe", "password")
quickly open an webpage
import urllib2print urllib2.urlopen('http://stockrt.github.com').read()# With password?import urllibopener = urllib.FancyURLopener()print opener.open('http://user:password@stockrt.github.com').read()
0 0
- Python Mechanize Cheat Sheet
- python cheat sheet
- Cheat Sheet
- Cheat Sheet
- Python selenium —— webdriver cheat sheet(webdriver备忘单)
- Python selenium —— XPath and CSS cheat sheet
- Python selenium —— Webdriver Exception cheat sheet
- Python Data Science Cheat Sheet,数据科学总结
- 关于“cheat sheet”
- regular expressions cheat sheet
- Vim Cheat Sheet
- vi vim cheat sheet
- Rails Migration Cheat Sheet
- Watir Cheat Sheet
- WinDbg / SOS Cheat Sheet
- PHP Cheat Sheet
- SQL Injection Cheat Sheet
- sql injection cheat sheet
- jacob 用法
- jQuery 数据操作函数
- __proto__ 和 prototype的异同
- Java基础知识小结 三 (自学)
- 关于ping以及TTL的分析
- Python Mechanize Cheat Sheet
- hiho 1166 高斯消元
- 将形如 0xAABBCC拆分成形如 三个数0xAA,0xBB,0xCC 及逆过程
- jQuery 核心 - jQuery() 方法
- CocoaPods安装和使用教程
- bootstrap轮播图手机端左右滑动事件
- C++十进制转二级制
- 重要升级
- C#中virtual和abstract的区别