17. WEB APPLICATION

构建抽象层，然后逐层往下细化。自顶向下。分优先级。

爬虫应用：

接受一个网页，PARSE 内容，根据里面的HTML生成新的任务。

如何去重任务？ 单机的话，数据量小可以用SET，大的话LOCAL set DB去重。多台机器用URL的HASH去做PARTITION。

写BFS code for a simple web crawler
multi thread version
to design a more practical system

瓶颈：
网络带宽
计算资源去PARSE网页。

image.png

队列SHARDED BY URL。 每个爬虫只去一个队列取任务。

常规PATTERN

1.经典方法去分发JOB给不同WORKER
shard with (hash, range)
batch - group thing together

2.hash table

3.push/pull model

4.handle failure
master/salve
replication

5.cache
write through(写CACHE，DB在一起，都完成再返回写完成）
write around （先写DB，当读CACHE MISS的时候，LOAD INTO CACHE）
write back

6. database
   ACID
   sharding

7.knwoledge
network -> disk -> in memory -> l2 cache -> l1 cache
gist.github.com/jboner/2841832/

8. synchronous/ Asynchronous call
   blog.ometer.com/2011/07/24/callbacks-synchronous-and-asynchronous/

9. stateful vs stateless
   stateless service is great because can be replicated as many as you want - best scenario for scaliablity - perfect for computational task
   easy to handle client retry

   sharded stateful service is hard to bring up new instance. because you need to redistribute data/ internal states

highscalability.com
aosabook.org/en/distsys.html