
该项目改进于BitTiger.io的一个课程项目,实现后端分布式爬虫和前端数据显示。
- GitHub: https://github.com/hackjutsu/pokemongo-map-poc
- Demo: http://hackjutsu.com/pokemongo-map-poc-demo/
System Architecture

A distributed crawler system that achieves heuristic data crawling based on geographical location information, said system comprising:
- A web app sends network requests with said geographical locations information to an API gateway,
- An API gateway subsystem constructed by said API gateways, routes said requests to a query cluster,
- A query subsystem constructed by said query cluster, queries for Pokemons data based on said geographical locations information from a database on AWS, and returns said Pokemon data to said web app, and sends said geographical locations information to message queues,
- A message queue subsystem constructed from said message queues, stores said messages from said query subsystem, and provides said messages to a crawler cluster
- A crawler subsystem constructed by said crawler cluster, retrieves said geographical locations information from said message queue subsystem, and gets fake Pokemons data from a mock Pokemon go API, and saves said fake Pokemons data to a database,
- A database subsystem constructed by said database, stores said Pokemons data.