READ THIS

Paste_Image.png

Item.py

for making scrapy crawled data more ordered and serializable
how to use

• import botnameItem class from the projectname.items file
  declare it () and yield it

Pipeline

receive and process item
how to use

• uncomment it inside settings
• use it

Settings

DOWNLOAD_DELAY = 3 be more friendly to scrapped site
USER_AGENT be more of a browser than a robot
ROBOTSTXT_OBEY = True

robotstxt in setting should be true if there is a robots.txt for the site, to be a good web citizen

r

Paste_Image.png

Paste_Image.png