hdfs集成Kerberos

隶属于文章系列：大数据安全实战 https://www.jianshu.com/p/76627fd8399c

步骤

  1. 创建principle
  2. 修改core-site.xml
  3. 修改hdfs-site.xml
  4. 配置HTTPS

hdfs集成Kerberos

  <!-- 2018-3-9 Kerberos  -->
  <property>
  <name>hadoop.security.authentication</name>
  <value>kerberos</value>
</property>

<property>
  <name>hadoop.security.authorization</name>
  <value>true</value>
</property>

<property>
  <name>dfs.block.access.token.enable</name>
  <value>true</value>
</property>
<property>
  <name>dfs.datanode.data.dir.perm</name>
  <value>700</value>
</property>

<property>
  <name>dfs.namenode.keytab.file</name>
  <value>/etc/hadoop/conf/hdfs-service.keytab</value>
</property>

<property>
  <name>dfs.namenode.kerberos.principal</name>
  <value>hdfs/_HOST@TT.COM</value>
</property>
<property>
  <name>dfs.namenode.kerberos.https.principal</name>
  <value>HTTP/_HOST@TT.COM</value>
</property>


<!--  
      <property>
  <name>dfs.datanode.address</name>
  <value>0.0.0.0:1004</value>
</property>
<property>
  <name>dfs.datanode.http.address</name>
  <value>0.0.0.0:1006</value>
</property>

 -->
<property>
  <name>dfs.datanode.address</name>
  <value>0.0.0.0:61004</value>
</property>
<property>
  <name>dfs.datanode.http.address</name>
  <value>0.0.0.0:61006</value>
</property>

<property>
  <name>dfs.http.policy</name>
  <value>HTTPS_ONLY</value>
</property>

<property>
  <name>dfs.data.transfer.protection</name>
  <value>integrity</value>
</property>




<property>
  <name>dfs.datanode.keytab.file</name>
  <value>/etc/hadoop/conf/hdfs-service.keytab</value>
</property>
<property>
  <name>dfs.datanode.kerberos.principal</name>
  <value>hdfs/_HOST@TT.COM</value>
</property>
<property>
  <name>dfs.datanode.kerberos.https.principal</name>
  <value>HTTP/_HOST@TT.COM</value>
</property>



<property>
  <name>dfs.journalnode.keytab.file</name>
  <value>/etc/hadoop/conf/hdfs-service.keytab</value>
</property>
<property>
  <name>dfs.journalnode.kerberos.principal</name>
  <value>hdfs/_HOST@TT.COM</value>
</property>
<property>
  <name>dfs.journalnode.kerberos.internal.spnego.principal</name>
  <value>HTTP/_HOST@TT.COM</value>
</property>
<property>
  <name>dfs.webhdfs.enabled</name>
  <value>true</value>
</property>
<property>
  <name>dfs.web.authentication.kerberos.principal</name>
  <value>HTTP/_HOST@TT.COM</value>
</property>

<property>
  <name>dfs.web.authentication.kerberos.keytab</name>
  <value>/etc/hadoop/conf/hdfs-service.keytab</value>
</property>
  • 在配置完上面的配置文件，启动后报如下错误
java.lang.RuntimeException: Cannot start secure DataNode without configuring either privileged resources or SASL RPC data transfer protection and SSL for HTTP.  Using privileged resources in combination with SASL RPC data transfer protection is not supported.
        at org.apache.hadoop.hdfs.server.datanode.DataNode.checkSecureConfig(DataNode.java:1201)
        at org.apache.hadoop.hdfs.server.datanode.DataNode.startDataNode(DataNode.java:1101)
        at org.apache.hadoop.hdfs.server.datanode.DataNode.<init>(DataNode.java:429)
        at org.apache.hadoop.hdfs.server.datanode.DataNode.makeInstance(DataNode.java:2406)
        at org.apache.hadoop.hdfs.server.datanode.DataNode.instantiateDataNode(DataNode.java:2293)
        at org.apache.hadoop.hdfs.server.datanode.DataNode.createDataNode(DataNode.java:2340)
        at org.apache.hadoop.hdfs.server.datanode.DataNode.secureMain(DataNode.java:2522)
        at org.apache.hadoop.hdfs.server.datanode.DataNode.main(DataNode.java:2546)
2018-03-13 14:01:27,317 INFO org.apache.hadoop.util.ExitUtil: Exiting with status 1
2018-03-13 14:01:27,318 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: SHUTDOWN_MSG:

Using privileged resources in combination with SASL RPC data transfer protection is not supported.显示privileged resources（即小端口号）和SASL RPC data transfer protection不能同时使用。
这时候有两条道路选择：

<!--  
      <property>
  <name>dfs.datanode.address</name>
  <value>0.0.0.0:1004</value>
</property>
<property>
  <name>dfs.datanode.http.address</name>
  <value>0.0.0.0:1006</value>
</property>

 -->
<property>
  <name>dfs.datanode.address</name>
  <value>0.0.0.0:61004</value>
</property>
<property>
  <name>dfs.datanode.http.address</name>
  <value>0.0.0.0:61006</value>
</property>

这时候又报错：

2018-03-09 20:44:10,993 INFO org.apache.hadoop.security.authentication.server.KerberosAuthenticationHandler: Login using keytab /etc/hadoop/conf/hdfs-service.keytab, for principal HTTP/v-hadoop-kbds.sz.kingdee.net@TT.COM
2018-03-09 20:44:11,000 INFO org.apache.hadoop.security.authentication.server.KerberosAuthenticationHandler: Login using keytab /etc/hadoop/conf/hdfs-service.keytab, for principal HTTP/v-hadoop-kbds.sz.kingdee.net@TT.COM
2018-03-09 20:44:11,003 WARN org.mortbay.log: failed SslSelectChannelConnectorSecure@0.0.0.0:50470: java.io.FileNotFoundException: /home/kduser/.keystore (No such file or directory)
2018-03-09 20:44:11,003 WARN org.mortbay.log: failed Server@10ded6a9: java.io.FileNotFoundException: /home/kduser/.keystore (No such file or directory)
2018-03-09 20:44:11,003 INFO org.apache.hadoop.http.HttpServer2: HttpServer.start() threw a non Bind IOException
java.io.FileNotFoundException: /home/kduser/.keystore (No such file or directory)
        at java.io.FileInputStream.open0(Native Method)
        at java.io.FileInputStream.open(FileInputStream.java:195)
        at java.io.FileInputStream.<init>(FileInputStream.java:138)
        at org.mortbay.resource.FileResource.getInputStream(FileResource.java:275)
        at org.mortbay.jetty.security.SslSelectChannelConnector.createSSLContext(SslSelectChannelConnector.java:624)
        at org.mortbay.jetty.security.SslSelectChannelConnector.doStart(SslSelectChannelConnector.java:598)
        at org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:50)
        at org.mortbay.jetty.Server.doStart(Server.java:235)
        at org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:50)
        at org.apache.hadoop.http.HttpServer2.start(HttpServer2.java:877)
        at org.apache.hadoop.hdfs.server.namenode.NameNodeHttpServer.start(NameNodeHttpServer.java:142)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.startHttpServer(NameNode.java:760)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:639)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:819)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:803)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1500)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1566)
2018-03-09 20:44:11,006 INFO org.mortbay.log: Stopped SslSelectChannelConnectorSecure@0.0.0.0:50470
2018-03-09 20:44:11,107 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Stopping NameNode metrics system...
2018-03-09 20:44:11,108 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics system stopped.
2018-03-09 20:44:11,108 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics system shutdown complete.
2018-03-09 20:44:11,108 ERROR org.apache.hadoop.hdfs.server.namenode.NameNode: Failed to start namenode.
java.io.FileNotFoundException: /home/kduser/.keystore (No such file or directory)
        at java.io.FileInputStream.open0(Native Method)
        at java.io.FileInputStream.open(FileInputStream.java:195)
        at java.io.FileInputStream.<init>(FileInputStream.java:138)
        at org.mortbay.resource.FileResource.getInputStream(FileResource.java:275)
        at org.mortbay.jetty.security.SslSelectChannelConnector.createSSLContext(SslSelectChannelConnector.java:624)
        at org.mortbay.jetty.security.SslSelectChannelConnector.doStart(SslSelectChannelConnector.java:598)
        at org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:50)
        at org.mortbay.jetty.Server.doStart(Server.java:235)
        at org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:50)
        at org.apache.hadoop.http.HttpServer2.start(HttpServer2.java:877)
        at org.apache.hadoop.hdfs.server.namenode.NameNodeHttpServer.start(NameNodeHttpServer.java:142)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.startHttpServer(NameNode.java:760)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:639)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:819)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:803)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1500)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1566)
2018-03-09 20:44:11,110 INFO org.apache.hadoop.util.ExitUtil: Exiting with status 1
2018-03-09 20:44:11,111 INFO org.apache.hadoop.hdfs.server.namenode.NameNode: SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down NameNode at v-hadoop-kbds.sz.kingdee.net/172.20.178.28
************************************************************/

这需要配置HTTPS。
参考：HDFS的HTTPS配置

©著作权归作者所有,转载或内容合作请联系作者
【社区内容提示】社区部分内容疑似由AI辅助生成，浏览时请结合常识与多方信息审慎甄别。
平台声明：文章内容（如有图片或视频亦包括在内）由作者上传并发布，文章内容仅代表作者本人观点，简书系信息发布平台，仅提供信息存储服务。

相关阅读更多精彩内容

  • Spring Cloud
    Spring Cloud为开发人员提供了快速构建分布式系统中一些常见模式的工具（例如配置管理，服务发现，断路器，智...
    卡卡罗2017阅读 137,150评论 19赞 139
  • Data Guard Concepts and Administration:Introduc...
    个人自行阅读时候，翻译的文档。因为比较渣，如果有更合理或者错误的地方烦劳告知，我会做修改。Oracle Data ...
    窝窝的小黑屋阅读 1,320评论 0赞 3
  • Configuration
    原文链接 对于单节点设置，Flink已经准备就绪，不需要更改默认配置就可以启动。 开箱即用的配置会使用你默认安装的...
    小C菜鸟阅读 7,955评论 0赞 0
  • 又出事了，揭秘坐飞机的终极秘密！
    又一飞机出事了…… 5月1日俄航SU270次班机（波音777），在从莫斯科飞往泰国曼谷的途中，遭遇了湍流，机上27...
    鹿咔Looka阅读 2,105评论 0赞 0
  • 离殇
    花落半卷残红泪，柳下垂，人未睡。 孤灯对影心憔悴，月如旧，广宫寒。 一别万里思故园，雁已去，人未还。 扶琴当空弹，...
    李潇寒阅读 193评论 0赞 0

友情链接更多精彩内容