HBase集群平滑迁移步骤
测试环境
| CDH版本 | HBASE版本 | |
|---|---|---|
| 测试源集群 | 5.15.1 | 1.2.0 |
| 测试目标集群 | 6.2.0 | 2.1.0 |
线上迁移环境
| CDH版本 | HBASE版本 | |
|---|---|---|
| 线上源集群 | 5.9.3 | 1.2.0 |
| 线上目标集群 | 6.2.0 | 2.1.0 |
迁移前准备
源集群配置
snapshot配置（已配置忽略）
#修改配置
hbase.snapshot.enabled=true
replication配置（已配置忽略）
#设置
hbase.replication=true
源集群表信息统计:
查看namespace：
#查看是否有用户namespace
hbase(main):001:0> list_namespace
NAMESPACE
default
hbase
2 row(s) in 0.3830 seconds
查看各namespace下表
#查看namespace：hbase下是否有需要迁移的表
hbase(main):002:0> list_namespace_tables 'hbase'
TABLE
meta
namespace
2 row(s) in 0.0260 seconds
#namespace：hbase表不需要迁移
确定待迁移表
#确定待迁移表为：namespace：default下表，总共183张表
hbase(main):003:0> list_namespace_tables 'default'
TABLE
air_message_record
ali_upload_records
ali_upload_records_sec
alibaba_records
alipay_records
alitaobao_records
attachments
audit_logs
audit_message
.................................
zhima_feedbacks
zhima_history_feedbacks
183 row(s) in 0.2350 seconds
在目标集群创建与源集群相同的表
方式一：
参考地址：
https://www.cloudera.com/documentation/enterprise/latest/topics/cdh_bdr_hbase_replication.html#concept_xjp_2tt_nw
/**If the table to be replicated does not yet exist on the destination cluster, you must create it. The easiest way to do this is to extract the schema using HBase Shell.
/**On the source cluster, describe the table using HBase Shell. The output below has been reformatted for readability.
hbase> describe acme_users
Table acme_users is ENABLED
acme_users
COLUMN FAMILIES DESCRIPTION
{NAME => 'user', DATA_BLOCK_ENCODING => 'NONE', BLOOMFILTER => 'NONE',
REPLICATION_SCOPE => '0', VERSIONS => '3', COMPRESSION => 'NONE',
MIN_VERSIONS => '0', TTL => 'FOREVER', KEEP_DELETED_CELLS => 'FALSE',
BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => 'false'}
/**Copy the output and make the following changes:
/**For the TTL, change FOREVER to org.apache.hadoop.hbase.HConstants::FOREVER.
/**Add the word CREATE before the table name.
/**Remove the line COLUMN FAMILIES DESCRIPTION and everything above the table name.
/**The result is a command like the following:
"CREATE 'cme_users' ,
{NAME => 'user', DATA_BLOCK_ENCODING => 'NONE', BLOOMFILTER => 'NONE',
REPLICATION_SCOPE => '0', VERSIONS => '3', COMPRESSION => 'NONE',
MIN_VERSIONS => '0', TTL => org.apache.hadoop.hbase.HConstants::FOREVER, KEEP_DELETED_CELLS => 'FALSE',
BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => 'false'}
/**On the destination cluster, paste the command from the previous step into HBase Shell to create the table.
方式二：Hbase-ui界面，复制表信息，然后创建：
image.png
线上所有Table创建快捷脚本：
省略。。。
在源集群添加目标集群作为peer(增量配置)
#add_peer 'ID', 'CLUSTER_KEY'，其中后者参数值为：
hbase.zookeeper.quorum:hbase.zookeeper.property.clientPort:zookeeper.znode.parent
#如下
add_peer '1', CLUSTER_KEY => "10.10.15.56,10.10.15.18,10.10.15.79,10.10.15.84,10.10.15.88:2181:/hbase"
#NAMESPACES => ["default"], SERIAL => true**新版本才支持****
为什么新版本中需要串行Replication
参考：https://hbase.apache.org/book.html#_serial_replication
- 先put然后delete到源集群
- 由于region移动/ RS故障，它们被不同的复制源线程推送到对等群集。
- 如果在put之前将delete删除操作推入对等群集，并且在将put放置到对等群集之前在同等群集中发生flush和major-compact，则将收集删除并将该put放在对等群集中，但是在源群集中，该put操作将被delete，因此源群集和目标群集之间的数据不一致。
逐个表依次迁移步骤（增量）
开启表级replication，开始双写
#列族cf1..
hbase> alter 'tableName', {NAME => 'cf1', REPLICATION_SCOPE => '1'},{NAME => 'cf2', REPLICATION_SCOPE => '1'}...
验证replication命令：(版本不一致，无法检查。略过)
#命令：
hbase org.apache.hadoop.hbase.mapreduce.replication.VerifyReplication [--starttime=timestamp1] [--stoptime=timestamp] [--families=comma separated list of families] peerId tableName
#命令参考：
bash-4.2$ ./bin/hbase org.apache.hadoop.hbase.mapreduce.replication.VerifyReplication
Usage: verifyrep [--starttime=X] [--endtime=Y] [--families=A] [--row-prefixes=B] [--delimiter=] [--recomparesleep=] [--batch=] [--verbose] [--sourceSnapshotName=P] [--sourceSnapshotTmpDir=Q] [--peerSnapshotName=R] [--peerSnapshotTmpDir=S] [--peerFSAddress=T] [--peerHBaseRootAddress=U]
Options:
starttime beginning of the time range
without endtime means from starttime to forever
endtime end of the time range
versions number of cell versions to verify
batch batch count for scan, note that result row counts will no longer be actual number of rows when you use this option
raw includes raw scan if given in options
families comma-separated list of families to copy
row-prefixes comma-separated list of row key prefixes to filter on
delimiter the delimiter used in display around rowkey
recomparesleep milliseconds to sleep before recompare row, default value is 0 which disables the recompare.
verbose logs row keys of good rows
sourceSnapshotName Source Snapshot Name
sourceSnapshotTmpDir Tmp location to restore source table snapshot
peerSnapshotName Peer Snapshot Name
peerSnapshotTmpDir Tmp location to restore peer table snapshot
peerFSAddress Peer cluster Hadoop FS address
peerHBaseRootAddress Peer cluster HBase root location
Args:
peerid Id of the peer used for verification, must match the one given for replication
tablename Name of the table to verify
Examples:
To verify the data replicated from TestTable for a 1 hour window with peer #5
$ hbase org.apache.hadoop.hbase.mapreduce.replication.VerifyReplication --starttime=1265875194289 --endtime=1265878794289 5 TestTable
验证结果参考：
image.png
制作snapshot(全量数据迁移)
snapshot 'sourceTable', 'snapshotName'
导出snapshot到目标集群
#命令：
hbase org.apache.hadoop.hbase.snapshot.ExportSnapshot -snapshot xxx.snapshot -copy-to hdfs://xxx:8020/hbase -mappers XX -overwrite -bandwidth 5
#使用hdfs用户
eg:
hbase org.apache.hadoop.hbase.snapshot.ExportSnapshot -snapshot xinyan_black_records_sec.snapshot07021143 -copy-to hdfs://10.10.15.56:8020/hbase -chuser hbase -chgroup hbase -chmod 777 -overwrite -bandwidth 5
#命令参考;
Usage: bin/hbase org.apache.hadoop.hbase.snapshot.ExportSnapshot [options]
where [options] are:
-h|-help Show this help and exit.
-snapshot NAME Snapshot to restore.
-copy-to NAME Remote destination hdfs://
-copy-from NAME Input folder hdfs:// (default hbase.rootdir)
-no-checksum-verify Do not verify checksum, use name+length only.
-no-target-verify Do not verify the integrity of the \exported snapshot.
-overwrite Rewrite the snapshot manifest if already exists
-chuser USERNAME Change the owner of the files to the specified one.
-chgroup GROUP Change the group of the files to the specified one.
-chmod MODE Change the permission of the files to the specified one.
-mappers Number of mappers to use during the copy (mapreduce.job.maps).
-bandwidth Limit bandwidth to this value in MB/second.
Examples:
hbase org.apache.hadoop.hbase.snapshot.ExportSnapshot \
-snapshot MySnapshot -copy-to hdfs://srv2:8082/hbase \
-chuser MyUser -chgroup MyGroup -chmod 700 -mappers 16
hbase org.apache.hadoop.hbase.snapshot.ExportSnapshot \
-snapshot MySnapshot -copy-from hdfs://srv2:8082/hbase \
-copy-to hdfs://srv1:50070/hbase \
** bandwidth单位为MB/s，大B，限制的是单个Region Server的流量，若有n个Region Server，总流量为nbandwidth*
目标集群bulkload导入snapshot
bulkload位置
Snapshot metadata is stored in the .hbase_snapshot directory under the hbase root directory (/hbase/.hbase-snapshot). Each snapshot has its own directory that includes all the references to the hfiles, logs, and metadata needed to restore the table.hfiles required by the snapshot are in the
/hbase/data/<namespace>/<tableName>/<regionName>/<familyName>/
location if the table is still using them; otherwise, they are in
/hbase/.archive/<namespace>/<tableName>/<regionName>/<familyName>/.
修改bulkload配置参数;(新集群已经修改)
image.png
导入命令（CDH6支持全表导入）：(全量)
#默认最大的HFile个数最大32，需要修改参数;hbase.mapreduce.bulkload.max.hfiles.perRegion.perFamily,如上：
./bin/hbase org.apache.hadoop.hbase.tool.LoadIncrementalHFiles \
-Dcreate.table=no \
/hbase/archive/data/default/${TableName}/ \
${TableName} -loadTable
#命令实例参考：
hbase org.apache.hadoop.hbase.tool.LoadIncrementalHFiles -Dcreate.table=no /hbase/archive/data/default/TableName/ ${TRableName} -loadTable
目标集群验证表迁移操作
检查HBase的region一致性与table完整性
#命令
./bin/hbase hbck ${tableName} -checkCorruptHFiles -sidelineCorruptHFiles -boundaries -summary -exclusive
#命令参考;
-----------------------------------------------------------------------
NOTE: As of HBase version 2.0, the hbck tool is significantly changed.
In general, all Read-Only options are supported and can be be used
safely. Most -fix/ -repair options are NOT supported. Please see usage
below for details on which options are not supported.
-----------------------------------------------------------------------
Usage: fsck [opts] {only tables}
where [opts] are:
-help Display help options (this)
-details Display full report of all regions.
-timelag Process only regions that have not experienced any metadata updates in the last seconds.
-sleepBeforeRerun Sleep this many seconds before checking if the fix worked if run with -fix
-summary Print only summary of the tables and status.
-metaonly Only check the state of the hbase:meta table.
-sidelineDir HDFS path to backup existing meta.
-boundaries Verify that regions boundaries are the same between META and store files.
-exclusive Abort if another hbck is exclusive or fixing.
Datafile Repair options: (expert features, use with caution!)
-checkCorruptHFiles Check all Hfiles by opening them to make sure they are valid
-sidelineCorruptHFiles Quarantine corrupted HFiles. implies -checkCorruptHFiles
Replication options
-fixReplication Deletes replication queues for removed peers
Metadata Repair options supported as of version 2.0: (expert features, use with caution!)
-fixVersionFile Try to fix missing hbase.version file in hdfs.
-fixReferenceFiles Try to offline lingering reference store files
-fixHFileLinks Try to offline lingering HFileLinks
-noHdfsChecking Don't load/check region info from HDFS. Assumes hbase:meta region info is good. Won't check/fix any HDFS issue, e.g. hole, orphan, or overlap
-ignorePreCheckPermission ignore filesystem permission pre-check
验证表中数据一致性：
小表：
通过hbase count命令
大表：
#大表通过MR验证
./bin/hbase org.apache.hadoop.hbase.mapreduce.RowCounter -Dhbase.client.scanner.caching=100 -Dmapreduce.map.speculative=true ${tableName}
迁移完成，观察线上HBase运行
遇到问题
经过测试，在一定宽带及IO下，小表EXportSnapshot没有问题，大表会出现archive下HFile找不到问题；如果小表使用更低宽带EXportSnapshot，也会出现此类问题；
解决：
已解决：
使用HBase导出命令：hbase org.apache.hadoop.hbase.snapshot.ExportSnapshot -snapshot ycsbTable02.snapshot -copy-to hdfs://10.10.15.56:8020/hbase/ycsb -chuser hbase -chgroup hbase -chmod 755 -mappers 1 -bandwidth 1 -overwrite 可以解决大表迁移是失败问题。
不要放到目标集群的/hbase/archive 目录下，切换别的目录
原因：因为master上会启动一个定期清理archive中垃圾文件的线程（HFileCleaner），定期会对这些被删除的垃圾文件进行清理（5分钟扫描一次）。如果目标集群下的snap文件没有被引用，就会被HFileCleaner 清理掉
