HTTP “请求头信息”Request Header是向服务端提供客户端的信息，“响应头信息”Response Header是服务端向客户端提供请求文档信息或服务器的状态信息，服务端判断服务端的身份，就是通过Header来判断的，所以爬虫通过设置Header来隐藏自己相当重要。

HTTP请求

一个完整的HTTP请求包含以下部分：

请求方法 URL HTTP版本
请求头信息
请求数据
<一个空行，请求的结束行>

常见的请求头：
Accept：客户端接收的数据类型，如：Accept：text/html
User Agent：客户端软件类型
Authorization：认证消息，包括用户名和口令
Referer：用户获取的Web页面

真实的请求头信息会更多，下面是豆瓣某短评的真实请求头：

Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3
Accept-Encoding: gzip, deflate, br
Accept-Language: zh-CN,zh;q=0.9,en;q=0.8,zh-TW;q=0.7
Cache-Control: max-age=0
Connection: keep-alive
Cookie: douban-fav-remind=1; _vwo_uuid_v2=DA9D8506AF55689A98FC4EC5458A1F005|9e3e2c5da18b4341ef7c7c5b1e6bc17d; __utmv=30149280.19413; douban-profile-remind=1; ll="118281"; __gads=ID=84d32737c7eb0e14:T=1564540928:S=ALNI_MYeYoLNcsUs74D0ASArxlCoDpjBIA; viewed="24872560"; bid=SdT44rmbqnQ; __utmz=223695111.1572102190.1.1.utmcsr=douban.com|utmccn=(referral)|utmcmd=referral|utmcct=/; __yadk_uid=FTuDNkSGj6E7mIoNRiLhR0HOeQGlFstY; push_noty_num=0; push_doumail_num=0; __utmz=30149280.1572351033.43.22.utmcsr=douban.com|utmccn=(referral)|utmcmd=referral|utmcct=/people/194130217/; _pk_ref.100001.4cf6=%5B%22%22%2C%22%22%2C1576563221%2C%22https%3A%2F%2Fwww.douban.com%2F%22%5D; _pk_id.100001.4cf6=c746db049db29d28.1572102190.8.1576563221.1574420366.; _pk_ses.100001.4cf6=*; __utma=30149280.1363678983.1539603396.1574420366.1576563222.48; __utmb=30149280.0.10.1576563222; __utmc=30149280; __utma=223695111.1512084094.1572102190.1574420367.1576563222.8; __utmb=223695111.0.10.1576563222; __utmc=223695111
Host: movie.douban.com
Sec-Fetch-Mode: navigate
Sec-Fetch-Site: none
Sec-Fetch-User: ?1
Upgrade-Insecure-Requests: 1
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/78.0.3904.108 Safari/537.36

HTTP响应

一个完整的HTTP响应包含以下部分：

状态行
响应头
响应数据

常见的状态行：

更多状态码查看： HTTP状态码

常见的响应头：
Server：Web服务器程序的信息
Date：当前服务器的日期和时间
Last Modified：请求文档最近一次修改的时间
Expires：请求文档过期时间
Content-length：数据长度（字节）
Content-type：数据MIME类型
WWW-authenticate：用于通知客户方需要的认证信息，如用户名，口令等

下面是豆瓣某短评的真实响应头：

Cache-Control: must-revalidate, no-cache, private
Connection: keep-alive
Content-Encoding: br
Content-Type: text/html; charset=utf-8
Date: Tue, 17 Dec 2019 06:13:57 GMT
Expires: Sun, 1 Jan 2006 01:00:00 GMT
Keep-Alive: timeout=30
Pragma: no-cache
Server: dae
Transfer-Encoding: chunked
Vary: Accept-Encoding
Vary: Accept-Encoding
X-Content-Type-Options: nosniff
X-DAE-App: movie
X-DAE-Instance: default
X-Douban-Mobileapp: 0
X-Xss-Protection: 1; mode=block

Requests设置Header

Python使用Requests来请求的时候，如果没有设置Header，Header是空的，设置Header的方法如下：

 headers = {
    'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_1) AppleWebKit/537.36 (KHTML, like Gecko) '
                  'Chrome/41.0.2227.1 Safari/537.36',
    'Cache-Control': 'max-age=0',
    'Cookie': 'douban-fav-remind=1; _vwo_uuid_v2=DA9D8506AF55689A98FC4EC5458A1F005|9e3e2c5da18b4341ef7c7c5b1e6b'
              'c17d; __'
              'utmv=30149280.19413; douban-profile-remind=1; ll="118281"; __gads=ID=84d32737c7eb0e14:T=15645409'
              '28:S=ALNI_MYeYoLNcsUs74D0ASArxlCoDpjBIA; viewed="24872560"; bid=SdT44rmbqnQ; __utmz=223695111.15'
              '72102190.1.1.utmcsr=douban.com|utmccn=(referral)|utmcmd=referral|utmcct=/; __yadk_uid=FTuDNkSGj6'
              'E7mIoNRiLhR0HOeQGlFstY; dbcl2="194130217:uUZw2E9T6DY"; push_noty_num=0; push_doumail_num=0; __'
              'utmz=30149280.1572351033.43.22.utmcsr=douban.com|utmccn=(referral)|utmcmd=referral|utmcct='
              '/people/194130217/; ck=rIT_; ap_v=0,6.0; _pk_ref.100001.4cf6=%5B%22%22%2C%22%22%2C1574385397%2C%'
              '22https%'
              '3A%2F%2Fwww.douban.com%2F%22%5D; _pk_ses.100001.4cf6=*; __utma=30149280.1363678983.1539603396.15'
              '72478349.1574385398.46; __utmb=30149280.0.10.1574385398; __utmc=30149280; __utma=223695111.15120'
              '84094.1572102190.1572478349.1574385398.6; __utmb=223695111.0.10.1574385398; __utmc=223695111; _'
              'pk_id.100001.4cf6=c746db049db29d28.1572102190.6.1574385406.1572478372.',
}

# 设置请求超时时间，header
r = requests.get(url, timeout=20, headers=headers)