解决windows下调试出现的问题

Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties

17/09/29 10:03:49 ERROR Shell: Failed to locate the winutils binary in the hadoop binary path
java.io.IOException: Could not locate executable null\bin\winutils.exe in the Hadoop binaries.
at org.apache.hadoop.util.Shell.getQualifiedBinPath(Shell.java:356)
at org.apache.hadoop.util.Shell.getWinUtilsPath(Shell.java:371)
at org.apache.hadoop.util.Shell.<clinit>(Shell.java:364)
at org.apache.hadoop.util.StringUtils.<clinit>(StringUtils.java:80)
..........................................................................................................................

经常在调试spark的时候出现上面的异常情况，其原因是windows系统下环境变量的未配置问题，解决这个问题的方案：

1. 下载一个winutils.exe，http://public-repo-1.hortonworks.com/hdp-win-alpha/winutils.exe。
2. 配置HADOOP_HOME的环境变量，复制winutils.exe 到${HADOOP_HOME}/bin目录下
3. 替代方案，在程序中设置System.setProperty("hadoop.home.dir", "hadoop的home目录");