What you need to crawling static websites.
python 3 library:
requests
connect to websites and working with URLs.
** beautiful soup**
a Python library for pulling data out of HTML and XML files. you can navigating the DOM tree or searching the DOM tree using css selector or re.
urllib
a package that collects several modules for working with URLs:
urllib.request
for opening and reading URLs
urllib.parse
for parsing URLs
import requests
from bs4 import BeautifulSoup as bs
import urllib
import re
Get Started with my github guide:
https://github.com/HoweZZH/EasyScraping
References:
https://www.crummy.com/software/BeautifulSoup/
https://www.crummy.com/software/BeautifulSoup/bs4/doc/
https://realpython.com/blog/python/web-scraping-with-scrapy-and-mongodb/
