从招聘网站爬取职位信息小窥IT发展

来源：互联网发布：中国古代数学知乎编辑：程序博客网时间：2024/04/30 06:04

想知道你的工作值多少钱？如果想小范围内转行，转哪个比较好呢？

IT人恐怕逃脱不了加班这个魔咒，有的人想转行，最好是钱多事少离家近。

Snake 从拉勾网爬取职位信息，通过分析城市，职位和薪水的分布的情况，来探讨IT的发展趋势。
虽然数据可能有些不准，“窥一斑而知全豹”。

首先挑几个维度做样本，如下：

cities = np.array(['北京', '上海', '广州', '深圳', '杭州'])
salaries = np.array(['2k-5k', '5k-10k', '10k-15k', '15k-25k', '25k-50k'])
keywords = np.array(["测试", "Android", "IOS", "前端", "人工智能", "后端", "运维"])

翠花，上爬虫！

def get_page(url, page_num, keyword): 
 headers = {
  "User-Agent": "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; AcooBrowser; .NET CLR 1.1.4322; .NET CLR 2.0.50727)",
  "Accept": "application/json, text/javascript, */*; q=0.01",
  "Content-Type": "application/x-www-form-urlencoded; charset=UTF-8",
  "Accept-Encoding": "gzip, deflate, br",
  "Accept-Language": "zh-CN,zh;q=0.8"
 }
 if page_num == 1:
  boo = 'true'
 else:
  boo = 'false'

 page_data = {
  'first': boo,
  'pn': page_num,
  'kd': keyword
 }
 page = requests.post(url=url, headers=headers, data=page_data)
 return page.json()

def get_detail(keyword):
 final_result = []

 for city in cities:
  total_count = []
  for salary in salaries:
   result = get_page(url.format(city, salary), 1, keyword)['content']['positionResult']['totalCount']
   total_count.append(result)

  final_result.append(total_count)
 detail[keyword] = final_result