前提
在前文已经部署好了alluxio集群以及集成了hdfs
https://www.jianshu.com/p/f417806156cf
https://www.jianshu.com/p/2823bcd1cef7
1. 修改配置文件
- Hadoop的core-site.xml
<!--alluxio集成mapreduce的操作-->
<property>
<name>fs.alluxio.impl</name>
<value>alluxio.hadoop.FileSystem</value>
<description>The Alluxio FileSystem (Hadoop 1.x and 2.x)</description>
</property>
<property>
<name>fs.AbstractFileSystem.alluxio.impl</name>
<value>alluxio.hadoop.AlluxioFileSystem</value>
<description>The Alluxio AbstractFileSystem (Hadoop 2.x)</description>
</property>
- hadoop的hadoop-env.sh
export HADOOP_CLASSPATH=/opt/alluxio-2.0.1/client/alluxio-2.0.1-client.jar:${HADOOP_CLASSPATH}
2. 拷贝jar包到hadoop
将/opt/alluxio-2.0.1/client/alluxio-2.0.1-client.jar拷贝到hadoop的lib下
3. 将更新后的hadoop配置文件和lib的jar包更新到每一个hadoop集群,重启hadoop和alluxio
4. 检查集成mapreduce是否成功
integration/checker/bin/alluxio-checker.sh mapreduce
5. 运行wordcount示例
准备input文件
./bin/alluxio fs copyFromLocal LICENSE /wordcount/input.txt
./bin/hadoop jar share/hadoop/mapreduce/hadoop-mapreduce-examples-2.7.2.jar wordcount -libjars /opt/alluxio-2.0.1/client/alluxio-2.0.1-client.jar alluxio://centos134:19998/wordcount/input.txt alluxio://centos134:19998/wordcount/output3
6. CDH版集成方法
CDH版本可能需要单独编译才能正常使用，编译环境可以参考：https://www.jianshu.com/p/38e9c9eec726
-
CDH在管理页面修改core-site.xml属性
CDH在管理页面修改yarn属性
然后重启相关组件
