从招聘网站爬取职位信息小窥IT发展

来源：互联网发布：node.js搭建编辑：程序博客网时间：2024/04/30 03:48

想知道你的工作值多少钱？如果想小范围内转行，转哪个比较好呢？

IT人恐怕逃脱不了加班这个魔咒，有的人想转行，最好是钱多事少离家近。 Snake 从拉勾网爬取职位信息，通过分析城市，职位和薪水的分布的情况，来探讨IT的发展趋势。
虽然数据可能有些不准，“窥一斑而知全豹”。

首先挑几个维度做样本，如下：

cities = np.array(['北京', '上海', '广州', '深圳', '杭州'])salaries = np.array(['2k-5k', '5k-10k', '10k-15k', '15k-25k', '25k-50k'])keywords = np.array(["测试", "Android", "IOS", "前端", "人工智能", "后端", "运维"])

翠花，上爬虫！

def get_page(url, page_num, keyword):     headers = {        "User-Agent": "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; AcooBrowser; .NET CLR 1.1.4322; .NET CLR 2.0.50727)",        "Accept": "application/json, text/javascript, */*; q=0.01",        "Content-Type": "application/x-www-form-urlencoded; charset=UTF-8",        "Accept-Encoding": "gzip, deflate, br",        "Accept-Language": "zh-CN,zh;q=0.8"    }    if page_num == 1:        boo = 'true'    else:        boo = 'false'    page_data = {        'first': boo,        'pn': page_num,        'kd': keyword    }    page = requests.post(url=url, headers=headers, data=page_data)    return page.json()def get_detail(keyword):    final_result = []    for city in cities:        total_count = []        for salary in salaries:            result = get_page(url.format(city, salary), 1, keyword)['content']['positionResult']['totalCount']            total_count.append(result)        final_result.append(total_count)    detail[keyword] = final_result

爬到的数据如下：