Python3 爬取帅哥图

此爬虫用的是基于 urllib3的第三方库 requests
网页原地址：http://www.shuaia.net/index.html

下载第三方库 requests :

pips install requests

爬取单面目标链接

    通过 Inspect element 发现目标地址存储在 class 属性为 "item-img" 的 <a> 标签的 href 属性中，获取到目标地址后，相当于点击图片之后进入了网页本身的页面，然后根据下一个页找到下一个页面地址。在 <a>标签里面 <img> 标签里也有个链接，但它是首页的浏览缩略图，不是高清的。

代码如下：

from bs4 import BeautifulSoupimport requestsif __name__ == '__main__':    url = 'http://www.shuaia.net/index.html'    headers = {        "User-Agent": "Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/58.0.3029.110 Safari/537.36"    }    req = requests.get(url=url, headers=headers)    req.encoding = 'utf-8'    html = req.text    bs = BeautifulSoup(html, 'lxml')    targets_url = bs.find_all(class_="item-img")    list_url = []    for each in targets_url:        list_url.append(each.img.get('alt') + '=' + each.get('href'))    print(list_url)

    这样就获取到了首页的图片链接

爬取多页目标链接

   翻到第 2 页的时候，很容易就发现地址变为了：http://www.shuaia.net/index_2.html ，第 3 页为：http://www.shuaia.net/index_3.html，后面的以此类推。

    获取前19页的链接，改造代码如下：

from bs4 import BeautifulSoupimport requestsif __name__ == '__main__':    list_url = []    for num in range(1, 20):        if num == 1:            url = 'http://www.shuaia.net/index.html'        else:            url = 'http://www.shuaia.net/index_%d.html' % num        headers = {            "User-Agent": "Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/58.0.3029.110 Safari/537.36"        }        req = requests.get(url=url, headers=headers)        req.encoding = 'urf-8'        html = req.text        bs = BeautifulSoup(html, 'lxml')        targets_url = bs.find_all(class_='item-img')        for each in targets_url:            list_url.append(each.img.get('alt') + ': ' + each.get('href'))    print(list_url)

单张图片下载

进入目标地址  Inspect element ，可以看到图片地址保存在 class 属性为"wr-single-content-list" 的 div->p->a->img 的 src 属性中。

代码如下：

from bs4 import BeautifulSoupimport requestsfrom urllib.request import urlretrieveimport ostarget_url = 'http://www.shuaia.net/wenshennan/2017-05-04/1289.html'filename = '花纹身衬托完美肌肉的欧美男' + '.jpg'headers = {    "User-Agent": "Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/58.0.3029.110 Safari/537.36"}img_req = requests.get(url=target_url, headers=headers)img_req.encoding = 'utf-8'img_html = img_req.textimg_bf = BeautifulSoup(img_html, 'lxml')img_url = img_bf.find_all('div', class_='wr-single-content-list')img_bf1 = BeautifulSoup(str(img_url), 'lxml')img_url = 'http://www.shuaia.net' + img_bf1.div.img.get('src')if 'images' not in os.listdir():    os.makedirs('images')urlretrieve(url=img_url, filename='images/' + filename)print('下载完成')

图片保存在程序文件所在的目录 images 文件夹里。

多张图片下载（整体代码）

此方法简单但速度慢。服务器有防爬虫程序，所以不能爬的太快，每下载一张图片需要加 1 秒的延时，否则会被服务器断开连接。

from bs4 import BeautifulSoupimport requestsfrom urllib.request import urlretrieveimport osimport timeif __name__ == '__main__':    list_url = []    for num in range(1, 10):        if num == 1:            url = 'http://www.shuaia.net/index.html'        else:            url = 'http://www.shuaia.net/index_%d.html' % num        headers = {            "User-Agent": "Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/58.0.3029.110 Safari/537.36"        }        req = requests.get(url=url, headers=headers)        req.encoding = 'urf-8'        html = req.text        bs = BeautifulSoup(html, 'lxml')        targets_url = bs.find_all(class_='item-img')        for each in targets_url:            list_url.append(each.img.get('alt') + ': ' + each.get('href'))    print('链接采集完成')    for each_img in list_url:        img_info = each_img.split(': ')        targets_url = img_info[1]        filename = img_info[0] + '.jpg'        print('正在下载：' + filename)        headers = {            "User-Agent": "Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/58.0.3029.110 Safari/537.36"        }        img_req = requests.get(url=targets_url, headers=headers)        img_req.encoding = 'utf-8'        img_html = img_req.text        img_bf = BeautifulSoup(img_html, 'lxml')        img_url = img_bf.find_all('div', class_='wr-single-content-list')        img_bf1 = BeautifulSoup(str(img_url), 'lxml')        img_url = 'http://www.shuaia.net' + img_bf1.div.img.get('src')        if 'images' not in os.listdir():            os.makedirs('images')        if each_img is False:   #为了防止出现异常情况            continue        urlretrieve(url=img_url, filename='images/' + filename)        time.sleep(1)   #有时可以试试不做延时    print('下载完成')

这是最终下载好的图片：