使用Hadoop的MapReduce,对数据进行归并分类处理。RequirementThe instagram-micro.csv contains information of random 5 million photos on Instagram. The content includes:userId - The ID of userphotoId - The ID of the photocreatedTime - Time of the photo posted by userfilter - Filter type used in the photolikes - Number of commentscomments - Number of likesIn this assignment, you are asked to understand what kind of filter is popular and what is not. Specifically, you need toDevelop a MapReduce program to count each type of filters.Design another pair of mapper and reducer, which takes the output from Step 1 as the input, and rank the filters by their frequencies in a decreasing order. (Hint: the default output of mapper is order by the keys (not the values), e.g., aa, ab, ac, ad, ae, etc.)The final output format is the default of Hadoop, which is key value pair separated by tab. For example, filterAAAA 9999 filterBBB 5555 filterCC 111SubmissionSubmit all the java code files needed for the task, including all the mappers, reducers, and driver.Change the name of your output file, part-r-00000, with your UIN without extension (e.g. 123456789) and upload it to Blackboard.Note: You can configure multiple Mapper/Reducer in Driver class. For example,1234567891011Configuration conf1 = new Configuration();Job job1 = Job.getInstance(conf1, &"whatever name of job1&");............Configuration conf2 = new Configuration();Job job2 = Job.getInstance(conf2, &"whatever name of job2&");FileInputFormat.addInputPath(job2, new Path(&"path to output of job1&"));............job1.waitForCompletion(true); // execute job1job2.waitForCompletion(true); // execute job2 after job1 is done& 转自:http://ass.3daixie.com/2019012149885879.html
讲解:Hadoop:CS4008 Instagram MicroJava、Java
©著作权归作者所有,转载或内容合作请联系作者
平台声明:文章内容(如有图片或视频亦包括在内)由作者上传并发布,文章内容仅代表作者本人观点,简书系信息发布平台,仅提供信息存储服务。
平台声明:文章内容(如有图片或视频亦包括在内)由作者上传并发布,文章内容仅代表作者本人观点,简书系信息发布平台,仅提供信息存储服务。
推荐阅读更多精彩内容
- Goal of MapReduce: Serve the tasks which needs only sever...
- Different Components NodeManager The NodeManager is YARN‘...
- 实行断舍离已经8个月,从极简小白,到现在身边开始有人问我,关于断舍离的问题和困惑: “我也好想断舍离啊,可是看到喜...
- 感情中并非我们所想的那么简单,你为人家掏心掏肺,却不知自己只是一个被玩弄的其中之一。虽然绝大比分女生心地善良,但是...