基于Hbase2.0，Spark2.2

问题描述

执行Spark处理Hbase数据时，遇到某两个Excutor处理速度特别慢，如图

正常速度10多分钟 左右处理完成， 一个多小时有另外一个处理完成，还有一个在处理中。

分析原因

1.查看hbase数据分布状况，看看是否存在数据倾斜问题，可以看出平均为6.8G左右，但有部分数据是1.71GB和3.4GB左右，但之前任务跑spark是80多个Executor不会导致时间差距这么大。

其实并非倾斜，下图1.71GB是6.8GB刚刚分裂的Region，分裂成了四个1.71*4 = 6.84GB，大小刚刚是平均大小，

startrowkey ：bc27300~bf2a300 大小3.41+1.71.+1.71 = 6.83 都是正常的（数据存储前rowkey做了处理）

ReadRequests	WriteRequests	StorefileSize	Num.Storefiles	MemSize	Locality	Start Key	End Key
(1,887,103,211)	(18,159,897)	(437.56 GB)	(171)	(315 MB)
0	0	0 B	0	0 B	0		4003300_c126f5cc6cef33e84de0cafc9e52d44f
39,112,630	327,294	6.83 GB	3	5 MB	1		0400300_a439d4f851fa7540d87ae33454a7f892
37,563,645	326,734	6.83 GB	2	6 MB	1	0400300_a439d4f851fa7540d87ae33454a7f892	07fe300_4130b717f89f58b3b6f530eec7a54b05
37,212,501	327,619	6.84 GB	2	5 MB	1	07fe300_4130b717f89f58b3b6f530eec7a54b05	0bfd300_4a2dc6b7b1a17472484041e49382b11d
37,392,603	57,911	6.84 GB	2	5 MB	1	0bfd300_4a2dc6b7b1a17472484041e49382b11d	0ffd300_74fdb5eebd54ee32e19abb31bccec158
0	0	0 B	0	0 B	0	0ffd300_74fdb5eebd54ee32e19abb31bccec158	11fe300_2aa404cbc291ffd913bab13d9339085c
13,882,247	14,525	1.71 GB	2	1 MB	1	0ffd300_74fdb5eebd54ee32e19abb31bccec158	10fd300_7a83ffc34c076654e69d558f1c9f1038
12,799,609	82,395	1.71 GB	2	1 MB	1	10fd300_7a83ffc34c076654e69d558f1c9f1038	11fe300_2aa404cbc291ffd913bab13d9339085c
10,021,069	81,978	1.71 GB	3	1 MB	1	11fe300_2aa404cbc291ffd913bab13d9339085c	12fe300_3f46699293dbeaa042e6856597d26dca
8,877,982	16,428	1.71 GB	3	1 MB	1	12fe300_3f46699293dbeaa042e6856597d26dca	13fd300_c223d551c26e191ffacb9981c5fb2cfd
36,137,439	328,938	6.84 GB	3	5 MB	1	13fd300_c223d551c26e191ffacb9981c5fb2cfd	1800300_0c10469f3ca178e9d5d81b507cc1f684
37,475,208	328,758	6.83 GB	3	5 MB	1	1800300_0c10469f3ca178e9d5d81b507cc1f684	1c00300_b0e2e6ce62897dbd1e033f88bcf22336
35,610,681	329,050	6.83 GB	3	5 MB	1	1c00300_b0e2e6ce62897dbd1e033f88bcf22336	2002300_cb4302be610bfedebaedce4a53cd0e4e
36,154,352	328,413	6.86 GB	3	5 MB	1	2002300_cb4302be610bfedebaedce4a53cd0e4e	2404300_806ce32f00569ec34eb1fdc008c574cc
35,637,405	57,144	6.85 GB	2	5 MB	1	2404300_806ce32f00569ec34eb1fdc008c574cc	2805300_3063b4db3bc49e0cd2d2e2430b8f92e8
35,617,576	328,429	6.85 GB	3	5 MB	1	2805300_3063b4db3bc49e0cd2d2e2430b8f92e8	2c04300_c3f674b39aeb8f289313f87d8d21bfe7
37,567,336	328,392	6.85 GB	2	5 MB	1	2c04300_c3f674b39aeb8f289313f87d8d21bfe7	3005300_be2766fa12b179733c9a55801367d56f
33,905,338	329,393	6.84 GB	2	5 MB	1	3005300_be2766fa12b179733c9a55801367d56f	3406300_ec238a180804b26ea232125ff11fcf68
33,344,294	328,009	6.84 GB	3	5 MB	1	3406300_ec238a180804b26ea232125ff11fcf68	3806300_886cf9e6502abd928a8c063ca517f54e
32,536,915	326,671	6.82 GB	2	5 MB	1	3806300_886cf9e6502abd928a8c063ca517f54e	3c05300_7ea6294dd2cd312d63d846bf5de5cd35
29,736,155	327,215	6.82 GB	2	5 MB	1	3c05300_7ea6294dd2cd312d63d846bf5de5cd35	4003300_c126f5cc6cef33e84de0cafc9e52d44f
22,272,474	328,354	6.85 GB	2	5 MB	1	4003300_c126f5cc6cef33e84de0cafc9e52d44f	4406300_18e8e63d5a80f546cdcadda551531dfc
22,267,884	329,515	6.84 GB	2	5 MB	1	4406300_18e8e63d5a80f546cdcadda551531dfc	4807300_8fb3f14d16049e78c52c1c1c416dda4f
22,259,844	56,744	6.84 GB	2	5 MB	1	4807300_8fb3f14d16049e78c52c1c1c416dda4f	4c08300_e44ffb40f82cf98c9e0802ee74070cc6
22,258,382	327,812	6.84 GB	2	4 MB	1	4c08300_e44ffb40f82cf98c9e0802ee74070cc6	500a300_fff0f934443b857109d1dbece53c770f
22,269,549	328,870	6.85 GB	3	5 MB	1	500a300_fff0f934443b857109d1dbece53c770f	540c300_b9bbb1361667ab3ac5669a3144a2fa71
22,250,497	328,277	6.84 GB	2	5 MB	1	540c300_b9bbb1361667ab3ac5669a3144a2fa71	580c300_c7d45406bd17aace3ad5f8586b9dff80
22,224,993	328,254	6.84 GB	3	6 MB	1	580c300_c7d45406bd17aace3ad5f8586b9dff80	5c0c300_0526a7b197b249ca24427063cf16657a
22,209,147	327,629	6.83 GB	2	5 MB	1	5c0c300_0526a7b197b249ca24427063cf16657a	600a300_5523694821dfcaab1fc9ff5ed40aab87
22,106,815	56,473	6.81 GB	2	5 MB	1	600a300_5523694821dfcaab1fc9ff5ed40aab87	6405300_0030bf135d4c2c20e9915c1941bd8cc7
22,136,156	327,095	6.80 GB	3	5 MB	1	6405300_0030bf135d4c2c20e9915c1941bd8cc7	67ff300_beefe04a96f547703165d057642c7a6e
21,805,246	325,501	6.81 GB	2	5 MB	1	67ff300_beefe04a96f547703165d057642c7a6e	6bfa300_c556fbef5490c3ca3168c7eb02015d16
21,160,315	56,503	6.81 GB	3	5 MB	1	6bfa300_c556fbef5490c3ca3168c7eb02015d16	6ff8300_471ab4d62d3bfe6ae86fe08b15524ee2
22,647,694	328,901	6.83 GB	2	5 MB	1	6ff8300_471ab4d62d3bfe6ae86fe08b15524ee2	73f7300_84b51415b091cdf93f380303f2a53658
22,209,724	56,882	6.83 GB	2	5 MB	1	73f7300_84b51415b091cdf93f380303f2a53658	77f5300_a6447ed7062c78c6f2216ec46ee47593
22,194,551	327,634	6.83 GB	2	5 MB	1	77f5300_a6447ed7062c78c6f2216ec46ee47593	7bf3300_3eab3e0f5794997bffffa596ca42d78d
22,150,573	326,600	6.81 GB	3	5 MB	1	7bf3300_3eab3e0f5794997bffffa596ca42d78d	7fef300_18b81d0e3245848ddfaf6a97ea868de5
0	0	0 B	0	0 B	0	7fef300_18b81d0e3245848ddfaf6a97ea868de5	c02b300_52789a8b10e2d896e9be1272d9169afd
37,290,422	328,707	6.87 GB	2	6 MB	1	7fef300_18b81d0e3245848ddfaf6a97ea868de5	83f3300_50ee6021166da9b30d27205aa93c7a47
35,724,649	329,463	6.87 GB	2	5 MB	1	83f3300_50ee6021166da9b30d27205aa93c7a47	87f7300_3a43c0b12b4c893d99bc9795c6fedde4
35,732,883	329,070	6.87 GB	3	5 MB	1	87f7300_3a43c0b12b4c893d99bc9795c6fedde4	8bfb300_4143081d05ed98403cf157074bd12a5d
17,903,439	34,080	3.43 GB	2	2 MB	1	8bfb300_4143081d05ed98403cf157074bd12a5d	8dff300_146416041229f7ddc0d92a8f3ba72e22
17,856,516	28,246	3.43 GB	2	2 MB	1	8dff300_146416041229f7ddc0d92a8f3ba72e22	9001300_1a9a07fdf1ba346228d63f9a4db9ebbf
35,734,987	330,855	6.87 GB	2	5 MB	1	9001300_1a9a07fdf1ba346228d63f9a4db9ebbf	9404300_f5e0017433b307e15eb645c82ba69eb1
34,355,672	330,896	6.87 GB	2	5 MB	1	9404300_f5e0017433b307e15eb645c82ba69eb1	980a300_eb3ea8174076a9343b9fefc131da4bfb
35,863,280	57,528	6.89 GB	2	5 MB	1	980a300_eb3ea8174076a9343b9fefc131da4bfb	9c13300_b4c65bb87a51729cdc51720b0cb0f0b7
35,775,342	328,782	6.87 GB	3	5 MB	1	9c13300_b4c65bb87a51729cdc51720b0cb0f0b7	a019
35,636,124	328,914	6.86 GB	2	5 MB	1	a019	a41a300_3756e9713d68d9e36f1dc7fe07829779
32,629,922	329,001	6.85 GB	3	5 MB	1	a41a300_3756e9713d68d9e36f1dc7fe07829779	a81d300_2f82d2b7ea67878b8c1ef86bd0e1c169
33,134,529	328,822	6.84 GB	2	5 MB	1	a81d300_2f82d2b7ea67878b8c1ef86bd0e1c169	ac1e300_0bd8ddff51d175087f6cda84b271b0eb
33,375,442	328,668	6.85 GB	2	5 MB	1	ac1e300_0bd8ddff51d175087f6cda84b271b0eb	b01e300_c52d0d2a792286a1b88d7b91c488813b
33,476,456	329,592	6.86 GB	3	5 MB	1	b01e300_c52d0d2a792286a1b88d7b91c488813b	b423300_8a37f3ac542446d2c3e16b069b08e09d
31,318,752	328,389	6.85 GB	3	5 MB	1	b423300_8a37f3ac542446d2c3e16b069b08e09d	b826300_1cdc89eeef046475883578cf0fc1d950
30,799,781	329,079	6.85 GB	2	5 MB	1	b826300_1cdc89eeef046475883578cf0fc1d950	bc27300_2b9315f844b8d5e1a253f18c4d2eabe1
14,482,007	29,199	3.42 GB	3	2 MB	1	bc27300_2b9315f844b8d5e1a253f18c4d2eabe1	be28300_f3a448e233baeddf68cc03027bf45a74
7,271,606	82,854	1.71 GB	2	1 MB	1	be28300_f3a448e233baeddf68cc03027bf45a74	bf2a300_5023e51cd969429bc3f008d780b46f61
7,269,349	82,563	1.71 GB	2	1 MB	1	bf2a300_5023e51cd969429bc3f008d780b46f61	c02b300_52789a8b10e2d896e9be1272d9169afd
20,762,362	325,903	6.82 GB	2	5 MB	1	c02b300_52789a8b10e2d896e9be1272d9169afd	c429300_6c3de304911e18799fb01d20f69be03f
22,220,126	327,000	6.82 GB	3	5 MB	1	c429300_6c3de304911e18799fb01d20f69be03f	c828300_c50e85412bf2cdfcb005505377a0ba73
21,767,188	327,870	6.83 GB	2	4 MB	1	c828300_c50e85412bf2cdfcb005505377a0ba73	cc27300_2e0fea9d4e083026bf18ea43642a73d0
21,537,300	326,781	6.83 GB	2	4 MB	1	cc27300_2e0fea9d4e083026bf18ea43642a73d0	d023300_fa22e502bb83df688bb331aaeafc4ee6
11,061,319	28,100	3.40 GB	3	2 MB	1	d023300_fa22e502bb83df688bb331aaeafc4ee6	d221300_4eb87797b5869a19a7bf18100abf50ec
11,081,110	162,680	3.40 GB	3	2 MB	1	d221300_4eb87797b5869a19a7bf18100abf50ec	d41e300_961e28d64cbf6e460820a7d47b9efa12
20,708,894	326,154	6.81 GB	3	5 MB	1	d41e300_961e28d64cbf6e460820a7d47b9efa12	d81a300_5229559bbe763cc7ebd875db357002cd
22,158,241	327,000	6.81 GB	2	5 MB	1	d81a300_5229559bbe763cc7ebd875db357002cd	dc17300_7ac1eebea0c79430e0b0b0af4dfeffc2
22,058,558	327,673	6.82 GB	2	5 MB	1	dc17300_7ac1eebea0c79430e0b0b0af4dfeffc2	e014300_ec7ef53bab65a01697e2c51175c52435
0	0	0 B	0	0 B	0	e014300_ec7ef53bab65a01697e2c51175c52435	e813300_d1e7c088b528940a93a289f9e66e88d7
37,529,102	57,605	6.83 GB	2	5 MB	1	e014300_ec7ef53bab65a01697e2c51175c52435	e416300_0edad48cf0a12f8e2b1370bd6e3fc151
35,533,233	327,217	6.83 GB	3	5 MB	1	e416300_0edad48cf0a12f8e2b1370bd6e3fc151	e813300_d1e7c088b528940a93a289f9e66e88d7
22,219,616	328,019	6.83 GB	2	5 MB	1	e813300_d1e7c088b528940a93a289f9e66e88d7	ec12300_f9576788ea38623c324f1fd1ff84385c
22,188,307	327,943	6.82 GB	2	5 MB	1	ec12300_f9576788ea38623c324f1fd1ff84385c	f010300_2053a69a8c06dd64c1a7d749ba16ef65
0	0	0 B	0	0 B	0	f010300_2053a69a8c06dd64c1a7d749ba16ef65	f808300_3bfbe199c6b2cd26acc4a94e98a410d1
37,715,743	327,565	6.81 GB	2	5 MB	1	f010300_2053a69a8c06dd64c1a7d749ba16ef65	f40d300_152e1c39039240e45a842b1e6bf955d4
9,448,945	82,183	1.70 GB	2	1 MB	1	f40d300_152e1c39039240e45a842b1e6bf955d4	f50c300_0cc62bd5f3b5d1fcdba2ff0cc95fff37
9,236,841	81,378	1.70 GB	2	1 MB	1	f50c300_0cc62bd5f3b5d1fcdba2ff0cc95fff37	f60a300_e9bd0585d0aff0c3cbd93b1cf59b14b6
17,719,497	163,774	3.40 GB	2	2 MB	1	f60a300_e9bd0585d0aff0c3cbd93b1cf59b14b6	f808300_3bfbe199c6b2cd26acc4a94e98a410d1
20,755,049	327,957	6.81 GB	2	5 MB	1	f808300_3bfbe199c6b2cd26acc4a94e98a410d1	fc05300_e476fad884176fe964420de97e5cdaaf
20,761,793	326,047	6.80 GB	2	5 MB	1	fc05300_e476fad884176fe964420de97e5cdaaf

2.查看Spark日志看看有什么问题

 发现日志,可以看到startRow=0000，stopRow=fffg ，这个Spark扫描Hbase设置的条件

19/04/15 09:29:41 INFO rdd.NewHadoopRDD: Input split: HBase table split(table name: zh_ams_ns:wechat_article, scan: {"loadColumnFamiliesOnDemand":null,"startRow":"0000","stopRow":"fffg","batch":-1,"cacheBlocks":true,"totalColumns":2,"maxResultSize":-1,"families":{"fn":["article_type","download_type"]},"caching":-1,"maxVersions":1,"timeRange":[0,9223372036854775807]}, start row: 7fef300_18b81d0e3245848ddfaf6a97ea868de1, end row: c02b300_52789a8b10e2d896e9be1272d9169af1, region location: hostname1, encoded region name: 8943fc0bd38fd292d9acb1c6bb4b7a6)
19/04/15 09:29:41 INFO rdd.NewHadoopRDD: Input split: HBase table split(table name: zh_ams_ns:wechat_article, scan: {"loadColumnFamiliesOnDemand":null,"startRow":"0000","stopRow":"fffg","batch":-1,"cacheBlocks":true,"totalColumns":2,"maxResultSize":-1,"families":{"fn":["article_type","download_type"]},"caching":-1,"maxVersions":1,"timeRange":[0,9223372036854775807]}, start row: 6405300_0030bf135d4c2c20e9915c1941bd8cc3, end row: 67ff300_beefe04a96f547403165d057642c7a6e, region location: hostname1, encoded region name: 96c9963bdd044ebdf2bd883435735d5)
19/04/15 09:29:41 INFO broadcast.TorrentBroadcast: Started reading broadcast variable 0

这个节点的具体处理数据信息为

start row: 7fef300_18b81d0e3245848ddfaf6a97ea868de5, end row: c02b300_52789a8b10e2d896e9be1272d9169afd, region location: hostname1
start row: 6405300_0030bf135d4c2c20e9915c1941bd8cc3, end row: 67ff300_beefe04a96f547403165d057642c7a6e, region location: hostname1

明显看出7fef300~c02b300，范围远大于6405300~67ff300，查看其他Executor，也发现跨度不会这么大，查看Hbase表的数据状态发现7fef300~c02b300的region没有数据，所有信息都为零。

可以初步判断出region分裂后，region的元信息没有进行删除， 导致spark读取时把rowkey 7fef300~c02b300的数据重新读取一遍，其中包含多个region导致Executor处理速度过慢。

为什么发现很多个region信息分裂后，元信息都没有删除，但是只有两个是非常慢的，可以从hbase数据中发现只有两个元信息跨度比较大，其他的rowkey范围宽度较小，所以没有明显感觉出来，如下图只跨了两个region

 

判断完成后，我们通过hbase hbck查看一下表的状态，截取报异常的一段，提示tableName,,1539422767596.259a2349c4f07b7d625142d323bede56.   region与其他region有重复区域，验证了我们刚才的想法

ERROR: (regions tableName,,1539422767596.259a2349c4f07b7d625142d323bede56. and tableName,0ffd300_74fdb5eebd54ee32e19abb31bccec158,1542774878705.a860b3230759d601187e9dcf6a4bdad0.) There is an overlap in the region chain.
ERROR: (regions tableName,,1539422767596.259a2349c4f07b7d625142d323bede56. and tableName,10fd300_7a83ffc34c076654e69d558f1c9f1038,1550065067486.62a2f5707f49a536755d8a357d30bfd0.) There is an overlap in the region chain.
ERROR: (regions tableName,0ffd300_74fdb5eebd54ee32e19abb31bccec158,1542774878705.a860b3230759d601187e9dcf6a4bdad0. and tableName,10fd300_7a83ffc34c076654e69d558f1c9f1038,1550065067486.62a2f5707f49a536755d8a357d30bfd0.) There is an overlap in the region chain.
ERROR: (regions tableName,,1539422767596.259a2349c4f07b7d625142d323bede56. and tableName,11fe300_2aa404cbc291ffd913bab13d9339085c,1543086502643.f4c52419535c5574f7b79c05f82a3bd2.) There is an overlap in the region chain.
ERROR: (regions tableName,,1539422767596.259a2349c4f07b7d625142d323bede56. and tableName,12fe300_3f46699293dbeaa042e6856597d26dca,1543086502643.dcfff4e27445fd7ef7d54e931fc3fee1.) There is an overlap in the region chain.
ERROR: (regions tableName,,1539422767596.259a2349c4f07b7d625142d323bede56. and tableName,13fd300_c223d551c26e191ffacb9981c5fb2cfd,1542770662897.f0c75de535b324f02267b2e2d0020c75.) There is an overlap in the region chain.
ERROR: (regions tableName,,1539422767596.259a2349c4f07b7d625142d323bede56. and tableName,1800300_0c10469f3ca178e9d5d81b507cc1f684,1548215404844.8546360f0bb0b7d0dcbd2c07772127ab.) There is an overlap in the region chain.
ERROR: (regions tableName,,1539422767596.259a2349c4f07b7d625142d323bede56. and tableName,1c00300_b0e2e6ce62897dbd1e033f88bcf22336,1548215404844.b12cda6b9cb167c54ed47f62cc0aa7a2.) There is an overlap in the region chain.
ERROR: (regions tableName,,1539422767596.259a2349c4f07b7d625142d323bede56. and tableName,2002300_cb4302be610bfedebaedce4a53cd0e4e,1549815723592.9b62e6522d5c687a088d95fbd1ca0c1b.) There is an overlap in the region chain.
ERROR: (regions tableName,,1539422767596.259a2349c4f07b7d625142d323bede56. and tableName,2404300_806ce32f00569ec34eb1fdc008c574cc,1549815723592.dada7d00670779d1aaf6c0d47ebb50c7.) There is an overlap in the region chain.
ERROR: (regions tableName,,1539422767596.259a2349c4f07b7d625142d323bede56. and tableName,2805300_3063b4db3bc49e0cd2d2e2430b8f92e8,1551035005714.2a6ad9c50e10527e2f08cad1d5189e4a.) There is an overlap in the region chain.
ERROR: (regions tableName,,1539422767596.259a2349c4f07b7d625142d323bede56. and tableName,2c04300_c3f674b39aeb8f289313f87d8d21bfe7,1551035005714.df8ba688e677572214f75b270dedcd1b.) There is an overlap in the region chain.
ERROR: (regions tableName,,1539422767596.259a2349c4f07b7d625142d323bede56. and tableName,3005300_be2766fa12b179733c9a55801367d56f,1551240890445.e479bcacf255f34bd0a98540e6e02ac5.) There is an overlap in the region chain.
ERROR: (regions tableName,,1539422767596.259a2349c4f07b7d625142d323bede56. and tableName,3406300_ec238a180804b26ea232125ff11fcf68,1551240890445.21a2461c5242d970cc46486b14817e12.) There is an overlap in the region chain.
ERROR: (regions tableName,,1539422767596.259a2349c4f07b7d625142d323bede56. and tableName,3806300_886cf9e6502abd928a8c063ca517f54e,1548178323143.01d1784953e10b53ed9a41953302b3bc.) There is an overlap in the region chain.
ERROR: (regions tableName,,1539422767596.259a2349c4f07b7d625142d323bede56. and tableName,3c05300_7ea6294dd2cd312d63d846bf5de5cd35,1548178323143.f861eb4119dc4dc3f7c5e73555824552.) There is an overlap in the region chain.
19/04/16 13:28:43 WARN util.HBaseFsck: reached end of problem group: 4003300_c126f5cc6cef33e84de0cafc9e52d44f

 

 

重大问题：统计完成时发现，会导致数据重复读取，正常数据量为142825979条  ，结果查询出来224847552条。

解决方法

1.运行过程中，不但慢后来因为Executor处理数据过大，内存不足导致异常，解决方式

https://blog.csdn.net/zhangshenghang/article/details/89313245

2.删除分裂前的region

根据region name进行删除，元信息在hbase:meta表中，查看状态

get 'hbase:meta','tableName,7fef300_18331d0e3245848ddfa16a97ea868de5,1539422555091.dee970bfd95a946d9a9268db01f7ec77.'

删除

 deleteall 'hbase:meta','tableName,7fef300_18b81d0e3245848ddfaf6a97ea868de5,1539422555091.dee970bfd95a946d9a9268db01f7ec77.'

删除后再次跑spark即可过滤掉这个region，解决该问题。

 

 但是存在一个隐藏问题，将表enable之后，删除的region又出现了，查看zookeeper也没有改配置，如何恢复的呢？后来经查阅相关资料发现master是有缓存存储这些信息的，删除元信息（目录 hdfs   /hbase/MasterProcWALs ）后重启hbase Master集群元信息就可以了。

 

问题：

千万不要删除有数据或者正常的region，删除并重启后，表数据异常无法读取数据，提示如下

hbase(main):062:0> count 'tableName'ERROR: Unknown table tableName!

使用 hbase hbck查看状态，提示region rowkey 范围开始应该为 '' 空字符串，如果删除正常的region会导致异常

ERROR: (region tableName_201615,201604097053999999_38599d53219f66c8605abbe144b33844,1494968976496.08e0b835edffba9c7f47c68d800f297a.) First region should start with an empty key.  You need to  create a new region and regioninfo in HDFS to plug the hole.
ERROR: Found inconsistency in table tableName_201615

正常 如下图所示