(1)明确要爬取网页的地址以豆瓣网的电影页为例:url=r'https://movie.douban.com'
(2)获得请求头:header={'User-Agent':'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.102 Safari/537.36 Edge/18.18362'},请求头是一个字典,'User-Agent‘的之可以从一下图中参考获取:
(3)获得请求体:req=request.Request(url,headers=header)
(4)发起请求并保存:response=request.urlopen(req)
data=response.read()
with open('scrapy2.html','wb') as f:
f.write(data)