为了获取拉勾网的招聘信息，对数据分析岗位的基本信息进行爬取。之所以选择拉勾网作为本项目的数据源，主要是因为相对于其他招聘网站，拉勾网上的岗位信息非常完整、整洁，极少存在信息的缺漏，并且几乎所有展现出来的信息都是非常规范化的，极大的减少了前期数据清理和数据整理的工作量。

模仿浏览器行为

网络爬虫是一种按照一定的规则，自动地抓取万维网信息的程序或者脚本。爬取拉钩数据分析 职位，首先模拟用户浏览行为，查找职位信息，可以先选职位，再选城市，根据城市获取相应数据分析岗位。
现在很多网站都用了一种叫做Ajax（异步加载）的技术，就是说，网页打开了，先给你看上面一部分东西，然后剩下的东西再慢慢加载。所以你可以看到很多网页，都是慢慢的刷出来的，或者有些网站随着你的移动，很多信息才慢慢加载出来。这样的网页有个好处，就是网页加载速度特别快。
如果一个城市有许多公司发布职位，类似的会出现网页下面多个页面，点击下一页的时候，地址栏的url不会变化，因此是post请求+异步加载的形式，通过右键审查元素，这次url请求是用了post方法，并且需要带上cookies，提交表单数据，返回的内容是json。

拆解过程

1、输入数据分析职位，选择城市，如下图观察url将城市变化页面也会相应变化，通过对城市遍历，传递给不同城市参数生成不同的url进行请求。通过对单个城市的访问获取拉勾网包含所有城市信息。

爬取策略

爬取拉勾网数据分析岗位信息的策略如下：
1、构造城市和职位组合的url，通过get方法解析页面获取拉勾网上所有的城市；
2、通过所有的城市信息，通过post方法获取不同城市包含的数据分析岗位总数量；
3、通过总数量我们可以获取每个城市的数据分析岗位总页数，通过post方法获取职位信息和PositionId；
4、根据PositionId，通过get方法解析每个职位对应的职位描述信息。

代码实现

# coding:utf-8
from urllib import request,parse
import json
from pandas import DataFrame
from lxml import etree
import random
import math
import time

def get_city_list():
    base_url = 'https://www.lagou.com/jobs/list_%E6%95%B0%E6%8D%AE%E5%88%86%E6%9E%90?px=default&city=%E6%B7%B1%E5%9C%B3'
    headers = {'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8',
                'Accept-Language': 'zh-CN,zh;q=0.8',
                'Connection': 'keep-alive',
                'Cookie': 'user_trace_token=20170823172848-77f8f03e-87e5-11e7-9ed0-525400f775ce; LGUID=20170823172848-77f8f6f1-87e5-11e7-9ed0-525400f775ce; JSESSIONID=ABAAABAACBHABBI08566127D8146453353170657FD7089A; PRE_UTM=; PRE_HOST=; PRE_SITE=; PRE_LAND=https%3A%2F%2Fwww.lagou.com%2F; _putrc=CD839C2A99BB2E8F; login=true; unick=%E5%BC%A0%E5%8F%88%E4%BA%AE; showExpriedIndex=1; showExpriedCompanyHome=1; showExpriedMyPublish=1; hasDeliver=44; TG-TRACK-CODE=search_code; SEARCH_ID=201ece3cee414cfdb6e8461e5484ff28; index_location_city=%E6%B7%B1%E5%9C%B3; _gid=GA1.2.943013801.1503976181; _gat=1; Hm_lvt_4233e74dff0ae5bd0a3d81c6ccf756e6=1504012158,1504059663,1504116125,1504140367; Hm_lpvt_4233e74dff0ae5bd0a3d81c6ccf756e6=1504140876; _ga=GA1.2.279566316.1503480294; LGSID=20170831085014-59a334e8-8de6-11e7-9f82-525400f775ce; LGRID=20170831085842-887351d7-8de7-11e7-9f97-525400f775ce',
                'Host': 'www.lagou.com',
                'Referer': 'https://www.lagou.com/jobs/list_%E6%95%B0%E6%8D%AE%E5%88%86%E6%9E%90?px=default&city=%E5%85%A8%E5%9B%BD',
                'Upgrade-Insecure-Requests': '1',
                'User-Agent': 'Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/60.0.3112.113 Safari/537.36'
                }
    req=request.Request(base_url,headers=headers,method='GET')
    response=request.urlopen(req)
    html=response.read().decode('utf-8')
    selector=etree.HTML(html)
    city1=selector.xpath('//li[@class="hot"]/a/text()')
    city2=selector.xpath('//li[@class="other"]/a/text()')
    city=city1+city2
    city.pop(0)
    city.pop(-1)
    return city

def page_counts(totalCount):
    pages=math.ceil(totalCount/float(15))
    if pages>30:
        pages=30
    return pages

def get_html(url,header,pn=1):
    formdata = {'first': 'true', 'pn': pn, 'kd': '数据分析'}
    data = bytes(parse.urlencode(formdata), encoding='utf-8')
    req = request.Request(url, data, header, method='POST')
    response = request.urlopen(req)
    html = response.read().decode('utf-8')
    #time.sleep(5)
    return html

def get_city_pages(url,header):
    referer = 'https://www.lagou.com/jobs/list_%E6%95%B0%E6%8D%AE%E5%88%86%E6%9E%90?px=default&city={a}'
    cities = get_city_list()
    print(cities)
    data=[]
    for eachCity in cities:
        scity = parse.quote(eachCity)
        url1 = url.format(b=str(scity))
        #header.pop('Referer')
        referer1 = referer.format(a=scity)
        header['Referer'] = referer1
        html = get_html(url1,header)
        # 转化为json
        jdict = json.loads(html)
        jcontent = jdict['content']
        jpositionResult = jcontent['positionResult']
        totalCount = jpositionResult['totalCount']
        data.append([totalCount,url1,referer1])
    return data

if __name__ == '__main__':
    iplist=['14.153.53.123:3128','113.105.146.77:8086','219.135.164.250:8080','219.128.75.149:8123']
    proxy_support=request.ProxyHandler({'http':random.choice(iplist)})
    opener = request.build_opener(proxy_support)
    request.install_opener(opener)
    json_url = 'https://www.lagou.com/jobs/positionAjax.json?city={b}&needAddtionalResult=false&isSchoolJob=0'
    json_headers = {'Accept': 'application/json, text/javascript, */*; q=0.01',
               'Accept-Encoding': 'gzip, deflate, br',
               'Accept-Language': 'zh-CN,zh;q=0.8',
               'Connection': 'keep-alive',
               'Content-Length': '55',
               'Content-Type': 'application/x-www-form-urlencoded; charset=UTF-8',
               'User-Agent': 'Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/60.0.3112.113 Safari/537.36',
               'Cookie':'user_trace_token=20170823172848-77f8f03e-87e5-11e7-9ed0-525400f775ce; LGUID=20170823172848-77f8f6f1-87e5-11e7-9ed0-525400f775ce; JSESSIONID=ABAAABAACBHABBI08566127D8146453353170657FD7089A; PRE_UTM=; PRE_HOST=; PRE_SITE=; PRE_LAND=https%3A%2F%2Fwww.lagou.com%2F; _putrc=CD839C2A99BB2E8F; login=true; unick=%E5%BC%A0%E5%8F%88%E4%BA%AE; showExpriedIndex=1; showExpriedCompanyHome=1; showExpriedMyPublish=1; hasDeliver=44; _gid=GA1.2.943013801.1503976181; _ga=GA1.2.279566316.1503480294; LGSID=20170831085014-59a334e8-8de6-11e7-9f82-525400f775ce; LGRID=20170831085859-92c5f026-8de7-11e7-9f98-525400f775ce; Hm_lvt_4233e74dff0ae5bd0a3d81c6ccf756e6=1504012158,1504059663,1504116125,1504140367; Hm_lpvt_4233e74dff0ae5bd0a3d81c6ccf756e6=1504140893; TG-TRACK-CODE=search_code; SEARCH_ID=318e5a8812994350a50e589a318bd332; index_location_city=%E6%B7%B1%E5%9C%B3',
               'Host': 'www.lagou.com',
               'Origin': 'https://www.lagou.com',
               'Referer': 'https://www.lagou.com/jobs/list_%E6%95%B0%E6%8D%AE%E5%88%86%E6%9E%90?px=default&city=%E6%B3%89%E5%B7%9E',
               'X-Anit-Forge-Code': '0',
               'X-Anit-Forge-Token': 'None',
               'X-Requested-With': 'XMLHttpRequest'
               }

    positionName = []#职位名称
    positionLables=[]#职位标签
    firstType=[]#职位类型1
    secondType=[]#职位类型2
    bussinessZones=[]#工作地段
    district=[]#工作区域
    city = []#所在城市
    education=[]#教育背景
    workYear=[]#工作年限
    salary = []#薪酬
    companyName = []#公司名字
    companySize = []#公司规模
    companyStage=[]#发展状况
    industryField=[]#经营范围
    totalCounts=[]#城市职位数量
    positionIds= []  # 职位ID
    data=get_city_pages(json_url,json_headers)
    for totalCount,url,refer in data:
        pages=page_counts(totalCount)
        json_headers['Referer']=refer
        while pages>0:
            if pages>9:
                json_headers['Content-Length']=56
            else:
                json_headers['Content-Length']=55
            html=get_html(url,json_headers,pn=pages)
            jdict = json.loads(html)
            jcontent = jdict['content']
            jpositionResult = jcontent['positionResult']
            jresult = jpositionResult['result']
            for each in jresult:
                positionName.append(each['positionName'])
                positionLables.append(each['positionLables'])
                firstType.append(each['firstType'])
                secondType.append(each['secondType'])
                bussinessZones.append(each['businessZones'])  # 工作地段
                district.append(each['district'])  # 工作区域
                city.append(each['city'])
                education.append(each['education']) # 教育背景
                workYear.append(each['workYear'])  # 工作年限
                salary.append(each['salary'])
                companyName.append(each['companyFullName'])
                companySize.append(each['companySize'])
                companyStage.append(each['financeStage'])  # 发展状况
                industryField.append(each['industryField']) # 经营范围
                totalCounts.append(totalCount)
                positionId=each['positionId']
                positionIds.append(positionId)
            pages = pages - 1
    positionData = {'positionName': positionName, 'positionLables': positionLables, 'positionType1':firstType,'postionType2':secondType,'bussinessZones':bussinessZones,'district':district,'city':city,'education':education,'workYear':workYear,'salary': salary, 'companyName': companyName, 'companySize': companySize, 'financeStage':companyStage,'industryField':industryField,'cityPositionCounts': totalCounts,'positionID':positionIds}
    frame = DataFrame(positionData)
    frame.to_csv('LagouPositionSociety.csv', index=False, na_rep='NULL')