人至贱则无敌，要学会制造问题然后解决问题。
启用HDFS HA过程中，被人为的中断或意外的中断导致Secondary NameNone还没有被删除。就像下图的小红框。还没离婚就把小三带回家了,最后全都一夜暴毙...没有人知道那晚发生了什么
image.png
法医剑仁的勘查
规则
http://{ambari_server_hostname}:8080/api/v1/clusters/{cluster_name}/hosts/{namenode_hostname}/host_components/ZKFC
注意：hostname要使用FQDN,若不了解FQDN请看一遍Ambari的安装过程吧
1、获取信息
通过走访得知
hdc-data1: namenode,datanode,journalnode,
hdc-data2: namenode,datanode,journalnode,secondarynamenode
hdc-data3: namenode,datanode,journalnode
#此步查看HDFS信息，可略过
curl -u admin:admin -H "X-Requested-By: ambari" -X GET http://hdc-data1.hadoop:8080/api/v1/clusters/mycluster/services/HDFS
角色查看
#查看namenode主机信息
curl -u admin:admin -i http://hdc-data1.hadoop:8080/api/v1/clusters/mycluster/host_components?HostRoles/component_name=NAMENODE
#查看secondarynamenode主机信息
curl -u admin:admin -i http://hdc-data1.hadoop:8080/api/v1/clusters/mycluster/host_components?HostRoles/component_name=SECONDARY_NAMENODE
#查看journalnode主机信息
curl -u admin:admin -i http://hdc-data1.hadoop:8080/api/v1/clusters/mycluster/host_components?HostRoles/component_name=JOURNALNODE
#查看zkfc主机信息
curl -u admin:admin -i http://hdc-data1.hadoop:8080/api/v1/clusters/mycluster/host_components?HostRoles/component_name=ZKFC
2、清理现场
提示：若返回类似如下信息说明该角色节点不存在，就不用管啦
{
"status" : 404,
"message" : "org.apache.ambari.server.controller.spi.NoSuchResourceException: The specified resource doesn't exist: ServiceComponentHost not found, clusterName=mycluster, serviceName=HDFS, serviceComponentName=ZKFC, hostName=hdc-data1"
}
清理ZKFC
curl -u admin:admin -H "X-Requested-By: ambari" -X DELETE http://hdc-data1.hadoop:8080/api/v1/clusters/mycluster/hosts/hdc-data1.hadoop/host_components/ZKFC
curl -u admin:admin -H "X-Requested-By: ambari" -X DELETE http://hdc-data1.hadoop:8080/api/v1/clusters/mycluster/hosts/hdc-data2.hadoop/host_components/ZKFC
清理JOURNALNODE
#获取所有JOURNALNODE节点信息
curl -u admin:admin -H "X-Requested-By: ambari" -X GET http://hdc-data1.hadoop:8080/api/v1/clusters/mycluster/host_components?HostRoles/component_name=JOURNALNODE
#逐一删除
curl -u admin:admin -H "X-Requested-By: ambari" -X DELETE http://hdc-data1.hadoop:8080/api/v1/clusters/mycluster/hosts/hdc-data1.hadoop/host_components/JOURNALNODE
curl -u admin:admin -H "X-Requested-By: ambari" -X DELETE http://hdc-data1.hadoop:8080/api/v1/clusters/mycluster/hosts/hdc-data2.hadoop/host_components/JOURNALNODE
curl -u admin:admin -H "X-Requested-By: ambari" -X DELETE http://hdc-data1.hadoop:8080/api/v1/clusters/mycluster/hosts/hdc-data3.hadoop/host_components/JOURNALNODE
清理 额外的 NAMENODE
#获取NAMENODE节点信息
curl -u admin:admin -H "X-Requested-By: ambari" -X GET http://hdc-data1.hadoop:8080/api/v1/clusters/mycluster/host_components?HostRoles/component_name=NAMENODE
#选择额外的namenode节点删除
curl -u admin:admin -H "X-Requested-By: ambari" -X DELETE http://hdc-data1.hadoop:8080/api/v1/clusters/mycluster/hosts/hdc-data2.hadoop/host_components/NAMENODE
3、还魂三部曲
#step1
curl -u admin:admin -H "X-Requested-By: ambari" -X POST -d '{"host_components" : [{"HostRoles":{"component_name":"SECONDARY_NAMENODE"}}] }' http://hdc-data1.hadoop:8080/api/v1/clusters/mycluster/hosts?Hosts/host_name=hdc-data2.hadoop
#step2 启用SECONDARY_NAMENODE
curl -u admin:admin -H "X-Requested-By: ambari" -X PUT -d '{"RequestInfo":{"context":"Enable Secondary NameNode"},"Body":{"HostRoles":{"state":"INSTALLED"}}}' http://hdc-data1.hadoop:8080/api/v1/clusters/mycluster/hosts/hdc-data2.hadoop/host_components/SECONDARY_NAMENODE
#step3
curl -u admin:admin -H "X-Requested-By: ambari" -X GET "http://hdc-data1.hadoop:8080/api/v1/clusters/mycluster/host_components?HostRoles/component_name=SECONDARY_NAMENODE&fields=HostRoles/state"
image.png
image.png
产生新的版本image.png
