嗨喽~大家好呀，这里是魔王呐 ❤ ~!
知识点:
爬虫基本流程
抓包工具的使用
开发环境:
python 3.10 运行代码
pycharm 辅助敲代码
requests 第三方模块
reqable
微信客户端
案例实现流程:
一. 需求(思路)分析
找到数据的来源
https://mp.weixin.qq.com/mp/profile_ext?action=getmsg&__biz=MzkwOTI2MjM0NQ==&f=json&offset=10&count=10&is_ok=1&scene=124&uin=MjQwODIxNTMyMw==&key=f2bc285be4cea20f508e3183cf22028899478d6c6cd992a5fca68b37abbd2f67cfc36330e6207cb4d60037d436adbb8b5796a26d079539fd38b6737a5e89a3b80fb45bd3f29f86fac95f75a62c39a5fbbdc4cb92108740ab094cd297b6ad5850a8621b12b8ab3634574fe9e98ab0253c9ec79e9215f655bece9642bc78d9612f&pass_ticket=NQQv+JhXtqD/xxBpc74ykqMwmRHbEvRHPSvlKXdD9VS67OBNPGiaFD+l2QEJIhaG/ZW25Jtp7evC92qtD7tLVA==&wxtoken=&appmsg_token=1249_5DoqtKeztxPNkr0O83A6cr7n9IsXeFAjjn8zjA~~&x5=0&f=json
二. 具体代码实现
发送请求
获取数据
-
解析数据
非结构化数据 网页源代码
结构化数据 json数据
保存数据
代码展示
导入模块
import requests # 发送请求的第三方库
import json
import re
模拟伪装
headers = {
"host": "mp.weixin.qq.com",
"user-agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/107.0.0.0 Safari/537.36 NetType/WIFI MicroMessenger/7.0.20.1781(0x6700143B) WindowsWechat(0x6309080f) XWEB/8461 Flue",
"x-requested-with": "XMLHttpRequest",
"accept": "*/*",
"sec-fetch-site": "same-origin",
"sec-fetch-mode": "cors",
"sec-fetch-dest": "empty",
"referer": "https",
"accept-encoding": "gzip, deflate, br",
"accept-language": "zh-CN,zh;q=0.9",
"cookie": "wap_sid2=CJvmqfwIEooBeV9ISlZFcmIyd2lIdU5EeHFLMGdvQTB5eHdrQldFeENNRzREM2RXRkozbVNabzlsa2pFc1lSVS1WMjlRQVZQZGtKQnhmMjdMeGtvcFBULXhnQ2tIMW1tUlR2MGM1cGhzMU16eW1DWkVmZFpZT2JiYVZsMExPNU9aV2p6NU1sVWJnTGdiNFNBQUF+MMm+i6wGOA1AlU4="
}
请求链接
url = 'https://mp.weixin.qq.com/mp/profile_ext?action=getmsg&__biz=MzkwOTI2MjM0NQ==&f=json&offset=20&count=10&is_ok=1&scene=124&uin=MjQwODIxNTMyMw==&key=098370c6c4230f2782cb126aad5ceee0079fda057ac38504504c5c4e237bd74afa5aa169e7db8882e0509aa1c04ef4c92ddf193e767f26b0c3d0697b2822575f337ae4a2556c04a8bfc1bcbb4715d8d2b25711985de0493188b0a34dcb01f8bcbaa67c7bf757527bf6674f72ea36e47a53280a25064df4317fa94a59236eb81b&pass_ticket=NQQv+JhXtqD/xxBpc74ykqMwmRHbEvRHPSvlKXdD9VRrkCQePU8FiLP9JBjS4XXiYNn3/yrKsy2bYkfSepRFlQ==&wxtoken=&appmsg_token=1249_JNc5MRSwDmbSu2vnkNrxa_RwVFe_A9i_hvejmA~~&x5=0&f=json'
- 发送请求
response = requests.get(url=url, headers=headers)
- 获取数据
# general_msg_list
json_data = response.json()
- 解析数据
general_msg = json_data['general_msg_list']
general_msg_list = json.loads(general_msg)
count = 0
for msg in general_msg_list['list']:
title = msg['app_msg_ext_info']['title']
content_url = msg['app_msg_ext_info']['content_url']
- 继续发送请求
print(title, content_url)
if content_url:
text_content = requests.get(content_url, headers=headers).text
# print(text_content)
# css / xpath / re
# .*?: 代替任意字符
img_list = re.findall('<img class="rich_pages wxw-img".*?data-src="(.*?)".*?>', text_content)
print(img_list)
for img in img_list:
- 保存数据
img_data = requests.get(img).content
with open(f'img/{count}.jpg', mode='wb') as f:
f.write(img_data)
count += 1
尾语
最后感谢你观看我的文章呐~本次航班到这里就结束啦 🛬
希望本篇文章有对你带来帮助 🎉，有学习到一点知识~
躲起来的星星🍥也在努力发光，你也要努力加油（让我们一起努力叭）。
