urllib2: Basic Authentication

http://www.voidspace.org.uk/python/articles/authentication.shtml#base64

http://kentsjohnson.com/kk/00010.html

Basic Authentication is part of the HTTP protocol. When you browse to a web site that uses basic auth, the browser pops up a dialog asking for your username and password. You supply credentials to the browser; your credentials are included with every subsequent request to the same web site.

To authenticate using basic authentication you must configure an opener with an HTTPBasicAuthHandler. The handler itself must be configured with the authentication realm, server address, username and password. Here is an example:

# Create an OpenerDirector with support for Basic HTTP Authentication...auth_handler = urllib2.HTTPBasicAuthHandler()auth_handler.add_password('realm', 'example.com', 'username', 'password')opener = urllib2.build_opener(auth_handler)

The open() method of the returned opener can be used to read pages that require authentication:

data = opener.open('http://example.com/my/protected/page.html').read()

You can also install the opener globally so it will be used with urlopen():

urllib2.install_opener(opener)data = urllib2.urlopen('http://example.com/my/protected/page.html').read()

Note: the realm is defined by the webserver that is requesting authentication. When you use your browser to log on with basic auth, the browser dialog will say something like "Enter user name and password for 'realm' at http://example.com." The realm that is displayed in the browser is the value you must use to configure the HTTPBasicAuthHandler.

首先可以用如下代码判断是否可以读取该网页的url：

import urllib#print urllib.urlopen('http://www.google.com').read()theurl = urllib.urlopen('http://mail.163.com/')print 'http status:', theurl.getcode()for line in theurl: # 就像在操作本地文件    print line,theurl.close()

可以用如下代码判断访问该网页的url是否需要authentication：

import urllib2theurl = 'https://cpanel.net/'req = urllib2.Request(theurl)try:    print 'urlopen begin'    handle = urllib2.urlopen(req)    print 'urlopen finish'except IOError, e:    if hasattr(e, 'code'):        if e.code != 401:            print 'We got another error'            print e.code        else:            print 'yes'            print e.headers            print e.headers['www-authenticate']    else:        print r'the string "code" is not in e'        print eelse:    print 'this url can be accessed'

可以用如下代码进行authentication（自己未验证成功）：

import urllib2import sysimport reimport base64from urlparse import urlparsetheurl = 'http://api.minicloud.com.cn/statuses/friends_timeline.xml'# if you want to run this example you'll need to supply# a protected page with your username and passwordusername = 'xxx'password = 'xxx'            # a very bad passwordreq = urllib2.Request(theurl)try:    handle = urllib2.urlopen(req)except IOError, e:    # here we *want* to fail    passelse:    # If we don't fail then the page isn't protected    print "This page isn't protected by authentication."    sys.exit(1)if not hasattr(e, 'code') or e.code != 401:    # we got an error - but not a 401 error    print "This page isn't protected by authentication."    print 'But we failed for another reason.'    sys.exit(1)#!!!the following code is used to get scheme and realm which however are not used#getting scheme and realm begin    authline = e.headers['www-authenticate']# this gets the www-authenticate line from the headers# which has the authentication scheme and realm in itauthobj = re.compile(    r'''(?:\s*www-authenticate\s*:)?\s*(\w*)\s+realm=['"]([^'"]+)['"]''',    re.IGNORECASE)# this regular expression is used to extract scheme and realmmatchobj = authobj.match(authline)if not matchobj:    # if the authline isn't matched by the regular expression    # then something is wrong    print 'The authentication header is badly formed.'    print authline    sys.exit(1)scheme = matchobj.group(1)realm = matchobj.group(2)# here we've extracted the scheme# and the realm from the headerif scheme.lower() != 'basic':    print 'This example only works with BASIC authentication.'    sys.exit(1)    #get scheme and realm finishbase64string = base64.encodestring(                '%s:%s' % (username, password))[:-1]authheader =  "Basic %s" % base64stringreq.add_header("Authorization", authheader)try:    handle = urllib2.urlopen(req)except IOError, e:    # here we shouldn't fail if the username/password is right    print "It looks like the username or password is wrong."    sys.exit(1)thepage = handle.read()