构建抽象层，然后逐层往下细化。自顶向下。分优先级。
爬虫应用：
接受一个网页，PARSE 内容，根据里面的HTML生成新的任务。
如何去重任务？ 单机的话，数据量小可以用SET，大的话LOCAL set DB去重。多台机器用URL的HASH去做PARTITION。
写BFS code for a simple web crawler
multi thread version
to design a more practical system
瓶颈：
网络带宽
计算资源去PARSE网页。
队列SHARDED BY URL。 每个爬虫只去一个队列取任务。
常规PATTERN
1.经典方法去分发JOB给不同WORKER
shard with (hash, range)
batch - group thing together
2.hash table
3.push/pull model
4.handle failure
master/salve
replication
5.cache
write through(写CACHE，DB在一起，都完成再返回写完成）
write around （先写DB，当读CACHE MISS的时候，LOAD INTO CACHE）
write back
- database
ACID
sharding
7.knwoledge
network -> disk -> in memory -> l2 cache -> l1 cache
gist.github.com/jboner/2841832/
synchronous/ Asynchronous call
blog.ometer.com/2011/07/24/callbacks-synchronous-and-asynchronous/-
stateful vs stateless
stateless service is great because can be replicated as many as you want - best scenario for scaliablity - perfect for computational task
easy to handle client retrysharded stateful service is hard to bring up new instance. because you need to redistribute data/ internal states
highscalability.com
aosabook.org/en/distsys.html
