很喜欢《庄子》一书，尤其是里面的呆若木鸡和庖丁解牛两个故事，揭示了为人和处事的三种境界。准备下载一个文本文档近期重温一下，搜索发现华语网上的质量很高，可惜是分篇的，懒得一点一点的copy，所以考虑用python爬取下来。

1. 爬取下图页面(https://www.thn21.com/wen/Famous/5609.html)内的各章节的链接。

import requests, re
from bs4 import BeautifulSoup

novelname = ''
names = []
urls = []

req = requests.get(url = 'https://www.thn21.com/wen/Famous/5609.html')
req.encoding = 'gb2312'
html = req.text
div_bf = BeautifulSoup(html, "html.parser")
novelname = re.search(r'(.+?)简介', div_bf.title.string).group(1)
a = div_bf.body.select('[href^="/wen/famous/hdnj/zuangzi"]')
for each in a:
    names.append(each.string)                                           # 章节名
    urls.append('https://www.thn21.com/' + each.get('href'))            # 章节链接

2. 依次爬取各章节链接中的内容。

注意因为内容中含有繁体字，所以需要用gb18030进行解码。

for i in range(len(a)):
    with open(novelname + '.txt', 'a', encoding='utf-8') as f:
        f.write('                                                        ' + names[i] + '\n')
        
        req = requests.get(url = urls[i])
        html= req.content.decode('gb18030', 'ignore')
        bf = BeautifulSoup(html, "html.parser")
        texts = ""
        for i in bf.body.find_all('p'):
            if i.string:
                texts += "    " + i.string + "\n"
                
        f.writelines(texts)
        f.write('\n\n')

3. 爬取结果如下：