切记:使用root用户编译
1. 源码下载及解压
Hadoop官网:https://hadoop.apache.org/release/2.7.0.html,Download src
下载完成后将hadoop-2.7.0-src.tar.gz 解压到/opt/目录,
进入解压出来的目录
BUILDING.txt文件是编译hadoop源码的说明文件;打开文件,列出了编译hadoop源码时,要求软件及对应版本要求。
本机环境:jdk 和maven已经安装好,跳过
2. 环境准备
安装findbugs:brew install findbugs,安装完成查看版本:spotbugs -version
安装autoconf:brew install autoconf
安装automake:brew install automake
安装libtool: brew install libtool , brew link libtool
安装Protocol buffers (Google's data interchange format),必须是2.5.0:brew search protobuf,发现没有这个版本,那么我们编译安装,安装完成查看版本:protoc --version
下载安装包:https://github.com/protocolbuffers/protobuf/releases?after=v3.0.0-alpha-1,找到对应的版本
下载完成后解压:sudo tar -xzvf protobuf-2.5.0.tar.gz -C /opt/
进入目录:cd protobuf-2.5.0
执行安装配置命令:
./autogen.sh & ./configure \
make \
make install
最后验证是否安装成功:protoc --version
安装cmake: brew install cmake
安装zlib: brew install zlib
安装openssl:brew install openssl
安装snappy(若想让hadoop支持snappy压缩,得先安装snappy):
注意安装snappy之前,需要先安装cmake,pkg-config,因为snappy是依赖cmake和pkg-config的,如果已经安装了snappy,可以重新install,brew reinstall snappy
brew install cmake
brew install pkg-config
安装snappy成功后,查看:brew info snappy
安装jansson:brew install jansson
3. 编译hadoop
进入源码目录:cd /opt/hadoop-2.7.0-src/
执行编译命令:mvn package -Pdist,native -DskipTests -Dtar
如果中途编译失败,并且不要文档、hadoop支持snappy、支持openssl的话,请使用这个命令:
mvn clean package -Pdist,native -DskipTests -Dtar -Drequire.snappy -Dsnappy.lib=/usr/local/Cellar/snappy/1.1.7_1/lib -Dbundle.snappy -Drequire.openssl
说明:支持snappy(若不需要支持snappy,把这些选项去除即可):Drequire.snappy -Dsnappy.lib=/usr/local/Cellar/snappy/1.1.7_1/lib -Dbundle.snappy
编译过程中遇到的问题:
1)No such file or directory:一般就是权限的问题,切换到root再执行看看。或者你可以将更改目录权限sudo chown -R chriszhao:staff /opt/hadoop-2.7.0-src/* (chriszhao:改成你自己的用户名)
2)hadoop-pipes: An Ant BuildException has occured: exec returned: 1:
查看build-main.xml
执行: cmake /opt/hadoop-2.7.0-src/hadoop-tools/hadoop-pipes/src/ -DJVM_ARCH_DATA_MODEL=64,发现如下错误:
Could NOT find OpenSSL, try to set the path to OpenSSL root folder in the System variable OPENSSL_ROOT_DIR (missing: OPENSSL_INCLUDE_DIR), suggesting that not found openssl suggested that the need to add environment variables
然后修改~/.zshrc, 之后再次执行以上cmake 命令,如果没错就再尝试编译
export LDFLAGS="-L/usr/local/opt/openssl@1.1/lib"
export CPPFLAGS="-I/usr/local/opt/openssl@1.1/include"
export PKG_CONFIG_PATH="/usr/local/opt/openssl@1.1/lib/pkgconfig"
export OPENSSL_INCLUDE_DIR="/usr/local/opt/openssl@1.1/include"
export PATH="/usr/local/opt/openssl@1.1/bin:$PATH"
3)因为网络问题导致tomcat包下不来,多尝试几次编译
4. 编译完成后
想要的结果包在hadoop-dist/target/下