爬取全国各地天气情况

首先我们打开中国天气网的首页[http://www.weather.com.cn/textFC/hb.shtml]

image.png

首先右键检查：

image.png

找到包含我们要爬取内容的标签：

 conMidtab = soup.find('div', class_="conMidtab")
    tables = conMidtab.find_all('table')

image.png

然后在我们的每一个标签当中找到我们需要的内容：

for table in tables:
        trs = table.find_all('tr')[2:]

image.png

接下来就是内容的提取：

for index, tr in enumerate(trs):
            # for tr in trs:
            tds = tr.find_all('td')
            city_td = tds[0]
            #print(city_td)
            if index == 0:
                city_td = tds[1]
            city = list(city_td.stripped_strings)[0]

            weather_td = tds[1]
            if index == 0:
                weather_td = tds[2]
            weather_td = list(weather_td.stripped_strings)[0]
            print({ "city" : city, "weather" : weather_td })

完整的代码如下：

import requests
from bs4 import BeautifulSoup
def parse_page(url):
    headers = {
        'User-Agent': "Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/65.0.3325.181 Safari/537.36"}
    response = requests.get(url, headers=headers)
    # print(response.content.decode('utf-8'))
    text = response.content.decode('utf-8')
    soup = BeautifulSoup(text, 'html5lib')
    # print(soup)
    conMidtab = soup.find('div', class_="conMidtab")
    tables = conMidtab.find_all('table')
    # print(tables)
    for table in tables:
        trs = table.find_all('tr')[2:]
        #print(trs)
        for index, tr in enumerate(trs):
            # for tr in trs:
            tds = tr.find_all('td')
            city_td = tds[0]
            #print(city_td)
            if index == 0:
                city_td = tds[1]
            city = list(city_td.stripped_strings)[0]

            weather_td = tds[1]
            if index == 0:
                weather_td = tds[2]
            weather_td = list(weather_td.stripped_strings)[0]
            print({ "city" : city, "weather" : weather_td })



def main():
    urls = {'http://www.weather.com.cn/textFC/hb.shtml',
            'http://www.weather.com.cn/textFC/hn.shtml',
            'http://www.weather.com.cn/textFC/db.shtml',
            'http://www.weather.com.cn/textFC/hd.shtml',
            'http://www.weather.com.cn/textFC/hz.shtml',
            'http://www.weather.com.cn/textFC/xb.shtml',
            'http://www.weather.com.cn/textFC/xn.shtml',
            'http://www.weather.com.cn/textFC/gat.shtml'}
    for url in urls:
        parse_page(url)

if __name__ == '__main__':
    main()

最后运行结果如下（部分结果以北京为例）：
{'city': '北京', 'weather': '晴'}
{'city': '海淀', 'weather': '晴'}
{'city': '朝阳', 'weather': '晴'}
{'city': '顺义', 'weather': '晴'}
{'city': '怀柔', 'weather': '晴'}
{'city': '通州', 'weather': '晴'}
{'city': '昌平', 'weather': '晴'}
{'city': '延庆', 'weather': '晴'}
{'city': '丰台', 'weather': '晴'}
{'city': '石景山', 'weather': '晴'}
{'city': '大兴', 'weather': '晴'}
{'city': '房山', 'weather': '晴'}
{'city': '密云', 'weather': '晴'}
{'city': '门头沟', 'weather': '晴'}
{'city': '平谷', 'weather': '晴'}
{'city': '东城', 'weather': '晴'}
{'city': '西城', 'weather': '晴'}