项目背景

由于本人在重庆从事数据分析工作，故这次先爬取Boss直聘的本地区数据分析岗信息，并进行分析

爬取思路

由于Boss直聘搜索职位不需要登陆，所以不涉及模拟登陆、cookies的问题，但是由于他会对同一时间访问过于频繁的ip进行验证，故而需要使用ip池。

整理思路大致如下：
1.使用ip池ip，boss首页搜索关键词，得到职位列表
2.根据职位列表中的url，分别爬取每个职位的详细数据
3.将爬取信息保存在mongo数据库内

网页代码分析

首先进入Boss直聘官网，搜索关键词“数据”，如图所示

Boss直聘“数据”搜索结果

职位列表网址：Boss直聘重庆地区数据岗位列表

数据岗位列表网页源码

某数据岗详情页面

源代码

import time
import random
import requests
import pymongo
import pymysql
import urllib.parse
from lxml import etree

class mongodb(object):

    def __init__(self):
        self.client = pymongo.MongoClient('mongodb://localhost:27017/')
        self.db = self.client.test_db
        self.collection = self.db.jobs

    def insert(self, data_dic):
        result = self.collection.insert_one(data_dic)
        return result

class boss(object):

    def __init__(self, word, page, proxies):
        self.url = 'https://www.zhipin.com/c101040100/?query={0}&page={1}&ka=page-{2}'
        self.page = page
        self.word = word
        self.proxy = random.choice(proxies)
        self.job = []
        self.header = {
            'accept': 'application/json, text/javascript, */*; q=0.01',
            'accept-encoding': 'gzip, deflate, br',
            'accept-language': 'zh-CN,zh;q=0.9',
            'referer': 'https://www.zhipin.com/job_detail/?query=%E6%95%B0%E6%8D%AE&scity=101040100&industry=&position=',
            'user-agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/71.0.3578.98 Safari/537.36',
            'x-requested-with': 'XMLHttpRequest'
        }

    def get(self):
        key_word = urllib.parse.quote(self.word)
        html_url = self.url.format(key_word, self.page, self.page)
        data = requests.get(html_url, headers=self.header, proxies=self.proxy)
        html = etree.HTML(data.text)
        url_list = html.xpath('//div[@class ="job-list"]//li//div[@class = "info-primary"]//a//@href')
        for i in url_list:
            data_dic = {}
            d_url = 'https://www.zhipin.com' + i
            res = requests.get(d_url, headers=self.header)
            d_html = etree.HTML(res.text)
            data_dic['id'] = str(self.page) + i.split('/')[1]
            data_dic['time'] = str(d_html.xpath('//div[@class="job-author"]//text()'))[5:21]
            data_dic['name'] = d_html.xpath('//div[@class="job-banner"]//div[@class="name"]//h1//text()')
            data_dic['salary'] = str(
                d_html.xpath('//div[@class="job-banner"]//div[@class="name"]//span//text()')).replace(" ", "").replace(
                '\\n', '')
            data_dic['request'] = d_html.xpath('//div[@class="job-banner"]//p//text()')
            data_dic['tags'] = d_html.xpath('//div[@class="job-banner"]//div[@class="job-tags"]//span//text()')
            data_dic['company'] = d_html.xpath('//div[@class="job-banner"]//h3[@class="name"]//text()')
            data_dic['describe'] = d_html.xpath('//div[@class="job-sec"]//text()')
            self.job.append(data_dic)
            time.sleep(random.random() * 0.56)
        return self.job

class ip(object):

    def __init__(self, db, table):
        self.db = db
        self.table = table
        self.usable_ip = []

    def get_ip(self):
        conn = pymysql.connect(host='localhost', user='root', password='HzH951126', db=self.db, charset='utf8')
        cur = conn.cursor()
        sql = 'SELECT * FROM '+str(self.table)
        cur.execute(sql)
        ip_pool = cur.fetchall()[:30]
        conn.close()
        for i in ip_pool:
            proxies = {
                i[2].lower(): i[2].lower() + '://' + i[0] + ':' + str(i[1])
            }
            try:
                check = requests.get('https://www.baidu.com', proxies=proxies, timeout=15)
                if check.status_code == 200:
                    self.usable_ip.append(proxies)
                else:
                    pass
            except:
                pass
        return self.usable_ip


if __name__ == '__main__':
    word = '数据'
    page = '3'
    sql_db = 'spiders'
    sql_table = 'ip'

    mg = mongodb()
    ip = ip(sql_db, sql_table)

    for p in range(1, 11):
        boss_data = boss(word, 3, ip.get_ip())
        for d in boss_data.get():
            mg.insert(d)
        print(str(p)+'  Finished')
    print('Spider Finished')

结果展示

职位信息已存储至 Mongo 数据库

爬取第三页作为示例，NoSQL数据库存储这些数据还是非常方便。

目前我们已经完成了数据的爬取，下一篇文章，我准备进行重庆地区数据岗招聘情况的数据分析报告。