讲解:Hadoop:CS4008 Instagram MicroJava、Java

使用Hadoop的MapReduce,对数据进行归并分类处理。RequirementThe instagram-micro.csv contains information of random 5 million photos on Instagram. The content includes:userId - The ID of userphotoId - The ID of the photocreatedTime - Time of the photo posted by userfilter - Filter type used in the photolikes - Number of commentscomments - Number of likesIn this assignment, you are asked to understand what kind of filter is popular and what is not. Specifically, you need toDevelop a MapReduce program to count each type of filters.Design another pair of mapper and reducer, which takes the output from Step 1 as the input, and rank the filters by their frequencies in a decreasing order. (Hint: the default output of mapper is order by the keys (not the values), e.g., aa, ab, ac, ad, ae, etc.)The final output format is the default of Hadoop, which is key value pair separated by tab. For example, filterAAAA 9999 filterBBB 5555 filterCC 111SubmissionSubmit all the java code files needed for the task, including all the mappers, reducers, and driver.Change the name of your output file, part-r-00000, with your UIN without extension (e.g. 123456789) and upload it to Blackboard.Note: You can configure multiple Mapper/Reducer in Driver class. For example,1234567891011Configuration conf1 = new Configuration();Job job1 = Job.getInstance(conf1, &"whatever name of job1&");............Configuration conf2 = new Configuration();Job job2 = Job.getInstance(conf2, &"whatever name of job2&");FileInputFormat.addInputPath(job2, new Path(&"path to output of job1&"));............job1.waitForCompletion(true); // execute job1job2.waitForCompletion(true); // execute job2 after job1 is done& 转自:http://ass.3daixie.com/2019012149885879.html

©著作权归作者所有,转载或内容合作请联系作者
平台声明:文章内容(如有图片或视频亦包括在内)由作者上传并发布,文章内容仅代表作者本人观点,简书系信息发布平台,仅提供信息存储服务。

推荐阅读更多精彩内容