Python: 爬取盗墓笔记

之前使用爬虫，主要是requests+BeautifulSoup组合，但是在实际使用中，又发现一个利器lxml。尤其是在读了Scrape the web using CSS Selectors in Python之后，更想试试它了。

这次的目标是盗墓笔记，其实有很多网站有这个小说，但是这个网站的HTML布局比较简单，很适合下手爬。

使用Chrome打开盗墓笔记网站，选中第一章，开始inspect：

Paste_Image.png

很容易看出css是

("#content > div.container > ul > li > a")

然后下面使用lxml对第一章爬正文。类似的操作

Paste_Image.png

这里的css是

("#content > div.post_entry")

在这里也是我弃用BS而使用lxml的主要原因，使用同样的css，BS就拿不到内容。

这里需要注意的是，这个页面的编码是ISO-8859-1，我们在读取的内容要进行解码：

content.encode('ISO-8859-1', 'ignore').decode('utf-8')

最好可以把得到的内容写到txt文件中，发给自己的kindle。
简单代码如下：

#!usr/bin/env  
# -*-coding:utf-8 -*-
import requests
import os
import sys
import lxml.html

from bs4 import BeautifulSoup as BS
from lxml.cssselect import CSSSelector

reload(sys)
sys.setdefaultencoding( "utf-8" )

sub_folder = os.path.join(os.getcwd(), "daomubiji")
if not os.path.exists(sub_folder):
    os.mkdir(sub_folder)

proxies = {
  "http": "http://proxy.yourcompany.com:8080/",
  "https": "https://proxy.yourcompany.com:8080/",
}

base_url = 'http://www.nanpaisanshu.org/daomubiji'
r = requests.get(base_url, proxies=proxies)

soup = BS(r.text, "lxml")
url_lists = soup.select("#content > div.container > ul > li > a")
print url_lists[0].get("href")

first_chapter = 'http://www.nanpaisanshu.org/4355.html'

r = requests.get(first_chapter, proxies=proxies)
print r.encoding

soup = BS(r.text.encode('ISO-8859-1', 'ignore').decode('utf-8'), "lxml")
content_lists = soup.select("#content > div.post_entry")
print "Use Requests: ", url_lists[0].get_text().encode('ISO-8859-1', 'ignore').decode('utf-8')


#
first_chapter_url = 'http://www.nanpaisanshu.org/4355.html'
r = requests.get(first_chapter_url, proxies=proxies)
print r.encoding
# build the DOM Tree
tree = lxml.html.fromstring(r.text)
 # print the parsed DOM Tree
# print lxml.html.tostring(tree)
#

sel_of_title = CSSSelector('#content > div.post > div.post_title > h2')
results = sel_of_title(tree)
match = results[0]
title = match.text.strip().encode('ISO-8859-1', 'ignore').decode('utf-8')
print "title: ", title
filename = sub_folder + "\\" + title + ".txt"
print filename
# construct a CSS Selector
sel_of_contents = CSSSelector('div.post_entry > p')
#
# Apply the selector to the DOM tree.
results = sel_of_contents(tree)
# print results
#
# print the HTML for the first result.
match = results[0]
# print lxml.html.tostring(match)
#
# print the text of the first result.
print "Use lxml", match.text.encode('ISO-8859-1', 'ignore').decode('utf-8')
#
# get the text out of all the results
data = [result.text for result in results]
with open(filename, "wb") as f:
    for content in data:
        if content:
            f.write("{}\n".format(content.encode('ISO-8859-1', 'ignore').decode('utf-8')))
            # print content.encode('ISO-8859-1', 'ignore').decode('utf-8')
f.close()

Python: 爬取盗墓笔记

推荐阅读更多精彩内容