Hadoop KMS（Key Management Server）是基于Hadoop的KeyProvider API的加密密钥管理服务，采用对称加密算法：AES加密算法；
提供了使用REST API通过HTTP通信的客户机和服务器组件。
客户机是一个KeyProvider实现，使用KMS HTTP REST API与KMS进行交互。
KMS及其客户机具有内置的安全性，并且支持HTTP SPNEGO Kerberos身份验证和HTTPS安全传输。
KMS是一个Java Jetty web应用程序。
KMS基与原生配置
生成keystore
[hadoop@hadoop01 ~]$ keytool -genkey -alias 'hadoop' -keystore ~/kms.keystore -dname "CN=bdc, OU=bonc, O=china, L=BeiJing, ST=CY, C=CN" -keypass 123456 -storepass 123456 -validity 3650
[hadoop@hadoop01 key]$ more kms.keystore.password
123456
修改kms-site.xml
<!-- KMS Backend KeyProvider -->
<property>
<name>hadoop.kms.key.provider.uri</name>
<value>jceks://file@/opt/beh/metadata/key/kms.keystore</value>
<description>
URI of the backing KeyProvider for the KMS.
</description>
</property>
<property>
<name>hadoop.security.keystore.java-keystore-provider.password-file</name>
<value>kms.keystore.password</value>
<description>
放在在kms启动的CLASSPATH下$HADOOP_HOME/share/hadoop/kms/tomcat/webapps/kms/WEB-INF/classes
</description>
</property>
<property>
<name>hadoop.kms.http.port</name>
<value>9600</value>
</property>
修改core-site.xml,客户端配置
<property>
<name>hadoop.security.key.provider.path</name>
<value>kms://http@hadoop01.bonc.com:9600/kms,kms://http@hadoop02.bonc.com:9600/kms</value>
<description>
The KeyProvider to use when interacting with encryption keys used
when reading and writing to an encryption zone.
</description>
</property>
目前KMS存在两种方法实现HA：
Load-Balancer or VIP
LoadBalancingKMSClientProvider
配置成如上配置，指定多个KMS实例，会隐式的启用LoadBalancingKMSClientProvider方式实现HA。
#高可用实际测试不靠谱，kms.keystore不能及时同步
启动环境变量：
[hadoop@hadoop01 hadoop]$ cat kms-env.sh |grep -v '#'|grep -v ^$
export KMS_LOG=/opt/beh/logs/hadoop/kms
export CATALINA_PID=/opt/beh/tmp/run/kms.pid
export KMS_HTTP_PORT=9600
关于KMS tomcat的详细配置$HADOOP_HOME/share/hadoop/kms/tomcat/conf下
启动：
[hadoop@hadoop01 classes]$ kms.sh start
setting KMS_LOG=/opt/beh/logs/hadoop/kms
setting KMS_HTTP_PORT=9600
Using CATALINA_BASE: /opt/beh/core/hadoop/share/hadoop/kms/tomcat
Using CATALINA_HOME: /opt/beh/core/hadoop/share/hadoop/kms/tomcat
Using CATALINA_TMPDIR: /opt/beh/core/hadoop/share/hadoop/kms/tomcat/temp
Using JRE_HOME: /opt/beh/core/jdk
Using CLASSPATH: /opt/beh/core/hadoop/share/hadoop/kms/tomcat/bin/bootstrap.jar
Using CATALINA_PID: /tmp/kms.pid
Existing PID file found during start.
[hadoop@hadoop02 ~]$ jps
6336 DFSZKFailoverController
7122 HMaster
26690 ResourceManager
6677 RunJar
7445 Kafka
29493 Jps
5960 JournalNode
6888 RunJar
5819 QuorumPeerMain
29451 Bootstrap #KMS Server
26301 NameNode
功能测试
[hadoop@hadoop01 tomcat]$ hadoop key list
Listing keys for KeyProvider: org.apache.hadoop.crypto.key.kms.LoadBalancingKMSClientProvider@7ce026d3
hadoop
[hadoop@hadoop01 tomcat]$ hadoop key create elastic
elastic has been successfully created with options Options{cipher='AES/CTR/NoPadding', bitLength=128, description='null', attributes=null}.
org.apache.hadoop.crypto.key.kms.LoadBalancingKMSClientProvider@2a693f59 has been updated.
#新添加的密钥将存储在配置文件指定的kms.keystore文件中。
[hadoop@hadoop01 withkrb]$ hdfs dfs -mkdir /elastic
[hadoop@hadoop01 withkrb]$ hdfs crypto -createZone -keyName elastic -path /elastic
Added encryption zone /elastic
# 此目录/elastic是加密区，必须是空目录
[hadoop@hadoop01 withkrb]$ hdfs dfs -mkdir /tmp/test
[hadoop@hadoop01 ~]$ echo "hello hadoop" > hello.txt
[hadoop@hadoop01 ~]$ hdfs dfs -put hello.txt /tmp/test
[hadoop@hadoop01 ~]$ hdfs dfs -put hello.txt /elastic
[hadoop@hadoop01 hadoop]$ hdfs fsck /elastic/hello.txt -files -blocks -locations -replicaDetails #查找实际磁盘存储地址
找到对应的磁盘文件， /elastic目录下文件已加密。
图片.png
通过hdfs dfs命令正常访问，加解密都是由client端完成，服务端主要是管理密钥及其访问控制！
[hadoop@hadoop01 hadoop]$ hdfs dfs -cat /elastic/hello.txt
hello hadoop
KMS基于Ranger插件安装
前面介绍的Hadoop KMS是Apache社区开发的，将keys存入文件（Java keystore）中。而Ranger KMS则是把数据存储入后台数据库中。通过Ranger Admin可以集中化管理KMS服务。
Ranger KMS有三个优点
- l Key management
Ranger admin 提供了创建，更新，删除密钥的Web UI 和APIs。在调试API的时候需要提供Ranger admin的用用户名和密码。 - l Access control policies
通过Ranger Admin可以对KMS进行访问控制，通过设置访问策略来限制用户访问。 - l Audit
Ranger Admin可以跟踪访问输出审计日志。
[hadoop@hadoop01 ranger]$ tar -xzvf ranger-1.2.0-kms.tar.gz
[hadoop@hadoop01 ranger]$ mv ranger-1.2.0-kms ranger-kms
#修改install.properties
[hadoop@hadoop01 ranger-kms]$ more install.properties |grep -v "#"|grep -v ^$
PYTHON_COMMAND_INVOKER=python
DB_FLAVOR=MYSQL
SQL_CONNECTOR_JAR=/opt/beh/core/ranger/ranger-admin/lib/mysql-connector-java-5.1.30.jar
db_root_user=root
db_root_password=bonc
db_host=hadoop01.bonc.com
KMS_MASTER_KEY_PASSWD=Str0ngPassw0rd
kms_principal=keyadmin/hadoop01.bonc.com@BONC.COM
kms_keytab=/opt/beh/metadata/key/ranger.keytab
hadoop_conf=/opt/beh/core/hadoop/etc/hadoo
punix_user=hadoop #ranger-kms 使用hadoop用户启动
unix_user_pwd=123123
unix_group=hadoop
POLICY_MGR_URL=http://hadoop01.bonc.com:6080
REPOSITORY_NAME=kmsdev
RANGER_KMS_LOG_DIR=/opt/beh/logs/ranger/kms
RANGER_KMS_PID_DIR_PATH=/opt/beh/tmp/run
#安装
[root@hadoop01 ranger-kms]# ./setup.sh
....
Ranger Plugin for kms has been enabled. Please restart kms to ensure that changes are effective.
Installation of Ranger KMS is completed.
#启动
[hadoop@hadoop01 ranger-kms]$ ranger-kms start
MariaDB [rangerkms]> show tables;
+---------------------+
| Tables_in_rangerkms |
+---------------------+
| ranger_keystore |
| ranger_masterkey |
+---------------------+
2 rows in set (0.00 sec)
修改core-site.xml
<!-- KMS Client Config -->
<property>
<name>hadoop.security.key.provider.path</name>
<value>kms://http@hadoop01.bonc.com:9292/kms</value>
<description>
The KeyProvider to use when interacting with encryption keys used
when reading and writing to an encryption zone.
</description>
<!-- KMS Client Config -->
ranger 使用keyadmin登陆
创建kms服务，参数填写：Service Name ：kmsdev
KMS URL ：kms://http@hostnip:9292/kms
Username ：keyadmin
Password ：keyadmin
图片.png
新建key
图片.png
Ranger-KMS异常处理
TestConnect异常
2019-10-24 03:24:15,949 [timed-executor-pool-0] INFO apache.ranger.services.kms.client.KMSClient (KMSClient.java:181) - Init Lookup Login: security enabled, using rangerPrincipal/rangerKeytab
2019-10-24 03:24:15,958 [timed-executor-pool-0] INFO apache.ranger.services.kms.client.KMSClient (KMSClient.java:239) - getKeyList():response.getStatus()= 401 for URL http://172.16.13.11:9292/kms/v1/keys/names?doAs=rangeradmin, so returning null list
org.apache.ranger.plugin.client.HadoopException:
<!DOCTYPE html>
<html lang="en">
<head>
<title>HTTP Status 401 – Unauthorized</title>
<style type="text/css">H1 {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:22px;} H2 {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:16px;} H3 {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:14px;} BODY {font-family:Tahoma,Arial,sans-serif;color:black;background-color:white;} B {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;} P {font-family:Tahoma,Arial,sans-serif;background:white;color:black;font-size:12px;}A {color : black;}A.name {color : black;}HR {color : #525D76;}</style>
</head>
<body>
<h1>HTTP Status 401 – Unauthorized</h1>
<hr class="line" />
<p><b>Type</b> Status Report</p>
<p><b>Message</b> Authentication required</p>
<p><b>Description</b> The request has not been applied because it lacks valid authentication credentials for the target resource.</p>
<hr class="line" />
<h3>Apache Tomcat/7.0.90</h3>.
</body>
</html>
doAs用户是ranger-admin（配置了 kerberos）配置的，$RANGER_ADMIN_HOME/ews/webapp/WEB-INF/classes/conf/ranger-admin-site.xml
<property>
<name>ranger.admin.kerberos.principal</name>
<value>rangeradmin/hamburg.bonc.com@BONC.COM</value>
</property>
<property>
<name>ranger.admin.kerberos.keytab</name>
<value>/opt/beh/metadata/key/rangeradmin.keytab</value>
</property>
ranger-kms也服务需要开启kerberos，修改$RANGER_KMS_HOME//ews/webapp/WEB-INF/classes/conf/kms-site.xml
<!-- KMS Security -->
<property>
<name>hadoop.kms.authentication.type</name>
<value>kerberos</value>
<description>
Authentication type for the KMS. Can be either "simple"
or "kerberos".
</description>
</property>
<property>
<name>hadoop.kms.authentication.kerberos.keytab</name>
<value>/opt/beh/metadata/key/ranger.keytab</value>
<description>
Path to the keytab with credentials for the configured Kerberos principal.
</description>
</property>
<property>
<name>hadoop.kms.authentication.kerberos.principal</name>
<value>HTTP/hadoop01.bonc.com@BONC.COM</value>
<description>
The Kerberos principal to use for the HTTP endpoint.
The principal must start with 'HTTP/' as per the Kerberos HTTP SPNEGO specification.
</description>
</property>
rangeradmin这个用户必须在ranger用户里面，没有就手动页面添加
图片.png
后续报错
org.apache.ranger.plugin.client.HadoopException: {
"RemoteException" : {
"message" : "User: rangeradmin/hadoop01.bonc.com@BONC.COM is not allowed to impersonate rangeradmin",
"exception" : "AuthorizationException",
"javaClassName" : "org.apache.hadoop.security.authorize.AuthorizationException"
}
}
修改$RANGER_KMS_HOME//ews/webapp/WEB-INF/classes/conf/kms-site.xml,允许rangeradmin模仿任意用户；
<property>
<name>hadoop.kms.proxyuser.rangeradmin.groups</name>
<value>*</value>
</property>
<property>
<name>hadoop.kms.proxyuser.rangeradmin.hosts</name>
<value>*</value>
</property>
<property>
<name>hadoop.kms.proxyuser.rangeradmin.users</name>
<value>*</value>
</property>
后续报错
org.apache.ranger.plugin.client.HadoopException: {
"RemoteException" : {
"message" : "User:rangeradmin not allowed to do 'GET_KEYS'",
"exception" : "AuthorizationException",
"javaClassName" : "org.apache.hadoop.security.authorize.AuthorizationException"
}
}.
将rangeradmin用户的权限提升，先赋予足够权限再测试连接！
图片.png
成功测试连接
图片.png
