练习(一)
目标抓取 https://segmentfault.com/t/python 列表
新建一个空文件夹 在该文件夹下运行命令
scrapy startproject segmentfault
cd segmentfault
scrapy genspider python segmentfault.com
完成后scrapy会为我们建立一个名为segmentfault的项目
并建立一个叫做“python”的爬虫任务
修改spiders文件夹下python.py文件将start_urls的内容修改为“https://segmentfault.com/t/python”
parse方法用来完成我们的目标任务
修改完后完整的代码如下
# -*- coding: utf-8 -*-
import scrapy
class PythonSpider(scrapy.Spider):
name = 'python'
allowed_domains = ['segmentfault.com']
start_urls = ['https://segmentfault.com/t/python']
def parse(self, response):
items = response.css(".stream-list__item .summary h2.title a")
for index,aLink in enumerate(items):
args = (aLink.css("::text").extract(),aLink.css("::attr(href)").extract())
yield { "title":args[0],"url":args[1]}
在当前文件夹下命令行运行
scrapy crawl python -o test.json
结束后会在当前文件夹下生成被爬取内容的json文件，打开后类型如下内容
[
{"url": ["/q/1010000010689990"], "title": ["python\u8c03\u7528\u81ea\u5df1\u5199\u7684b.py\u6587\u4ef6\uff0c\u6267\u884c\u540e\u751f\u6210b.pyc\u6587\u4ef6\u3002\u5220\u9664b.py \u7ee7\u7eed\u6267\u884c,\u53d1\u73b0\u62a5\u9519\u3002\u63d0\u793a\u6ca1\u6709b\u6a21\u5757"]},
{"url": ["/q/1010000010686570"], "title": ["\u5982\u4f55\u7528js\u5199\u4e00\u4e2a\u5c0f\u7a0b\u5e8f\uff0c\u6293\u53d6\u9875\u9762\u7684\u7535\u8bdd\uff0c\u516c\u53f8\uff1f"]},
{"url": ["/q/1010000010564104"], "title": ["JSON\u5bf9\u8c61\u600e\u4e48\u8f6c\u6362java\u5bf9\u8c61?"]},
{"url": ["/q/1010000010673785"], "title": ["\u90e8\u7f72Django \u5e94\u8be5\u4f7f\u7528apache + nginx\u8fd8\u662fuwsgi+ nginx"]},
{"url": ["/q/1010000010685577"], "title": ["python rabbitMQ \u961f\u5217\u94fe\u63a5\u51fa\u73b0\u4ee5\u4e0b\u9519\u8bef\u53ef\u80fd\u662f\u4ec0\u4e48\u539f\u56e0"]},
{"url": ["/q/1010000010683534"], "title": ["\u8bf7\u95eeWindows7\u4e0b\u5982\u4f55\u4f7f\u7528Python\u6740\u6b7b\u8fdb\u7a0b"]},
{"url": ["/q/1010000010680269"], "title": ["\u8bf7\u95eepy\u600e\u4e48\u5f80c++ .so\u5e93\u4f20\u6570\u636e\uff1f"]},
{"url": ["/q/1010000010682838"], "title": ["python\u8c03\u7528c++ .so\u5e93\u8fd4\u56dedouble\u65f6\u4e3a0"]},
{"url": ["/q/1010000010669643"], "title": ["\u8bf7\u95ee\u5982\u4f55\u6293\u53d6\u6dd8\u5b9dAPP\u5ba2\u6237\u7aef\u63a5\u53e3\u4fe1\u606f"]},
{"url": ["/q/1010000010674044"], "title": ["http\u4e2d\u7684post\u65b9\u6cd5\u7684request payload\u600e\u6837\u7528 requests\u5e93\u53d1\u9001"]},
{"url": ["/q/1010000010679630"], "title": ["counter\u88e1\u7684\u6578\u503c\u5982\u4f55\u76f8\u4e58"]},
{"url": ["/q/1010000010660247"], "title": ["\u5173\u4e8e\u4e8c\u53c9\u6811\u6df1\u5ea6\u6c42\u52a9"]},
{"url": ["/q/1010000010654982"], "title": ["\u4f7f\u7528django\u6267\u884c\u4e00\u4e2a\u9700\u8981\u8f83\u957f\u8ba1\u7b97\u65f6\u95f4\u7684\u672c\u673a\u811a\u672c\u5e76\u83b7\u53d6\u7ed3\u679c\uff0c\u5e94\u8be5\u600e\u6837\u5b9e\u73b0\uff1f"]},
{"url": ["/q/1010000010677371"], "title": ["\u5982\u4f55\u81ea\u5b9a\u4e49Django\u81ea\u5e26\u7684Admin\u7684views"]},
{"url": ["/q/1010000010671214"], "title": ["\u5982\u4f55\u5b9e\u73b0html\u6570\u636e\u8868\uff08\u4ece\u6570\u636e\u5e93\u4e2d\u67e5\u8be2\u51fa\u6765\u7684\uff09\u754c\u9762\u7684\u53ef\u7f16\u8f91,\u5e76\u5c06\u4fee\u6539\u7ed3\u679c\u91cd\u65b0\u5b58\u50a8\u81f3\u6570\u636e\u5e93"]},
{"url": ["/q/1010000010645067"], "title": ["python3 \u534f\u7a0b \u62a5\u9519\uff1aUnclosed client session \u8be5\u600e\u4e48\u89e3\u51b3\uff1f "]},
{"url": ["/q/1010000010657769"], "title": ["flask\u7684sendemail\u5728\u5f00\u53d1\u73af\u5883\u4e0b\u597d\u7528\u653e\u5230\u670d\u52a1\u5668\u4e0a\u5c31\u762b\u75ea\u4e86\u662f\u4e3a\u5565"]},
{"url": ["/q/1010000010651053"], "title": ["matplotlib\u76843d\u753b\u56fe\u600e\u4e48\u53bb\u6389\u80cc\u666f\u7684\u7f51\u683c\uff1f"]},
{"url": ["/q/1010000010632474"], "title": ["scrapy \u8f93\u51facsv\u6587\u4ef6\u6570\u636e\u4e4b\u95f4\u6709\u7a7a\u884c\uff1f"]},
{"url": ["/q/1010000010651221"], "title": ["\u7f51\u9875\u6807\u7b7e\u6e05\u6d17"]}
]
可以看到输出的内容为unicode格式
在 settings.py文件最后加入以下代码可解决输出unicode问题
from scrapy.exporters import JsonItemExporter
class CustomJsonLinesItemExporter(JsonItemExporter):
# 定义ensure_ascii=False,避免直接输出unicode
def __init__(self, file, **kwargs):
super(CustomJsonLinesItemExporter, self).__init__(file, ensure_ascii=False, **kwargs)
FEED_EXPORTERS = {
'json': 'segmentfault.settings.CustomJsonLinesItemExporter',
}
