python爬虫工具笔试面试项目实战1-urllib

urllib爬虫上机实操: 爬取首页的所有链接

• 爬取首页的所有链接，网址：https://china-testing.github.io/

参考答案

#!/usr/bin/python3
# -*- coding: utf-8 -*-
# 讨论钉钉免费群21745728 qq群144081101 567351477
# CreateDate: 2018-10-20

from urllib.request import urlopen, urljoin
import re


def download_page(url):
    return urlopen(url).read().decode('utf-8')


def extract_links(page):
    link_regex = re.compile('<a[^>]+href=["\'](.*?)["\']', re.IGNORECASE)
    return link_regex.findall(page)


if __name__ == '__main__':
    target_url = 'https://china-testing.github.io/'
    apress = download_page(target_url)
    links = extract_links(apress)

    for link in links:
        print(urljoin(target_url, link))

本例参考书籍： Website Scraping with Python - 2018.pdf

• 面试问答

1, urljoin('https://china-testing.github.io/', 'test')的结果是什么？

2，urljoin('https://china-testing.github.io/', 'https://www.google.com')的结果是什么？