1、表(非分区表)新增字段
alter table test.weblogs add columns( month string);
2、移动新增字段位置
alter table test.weblogs change column month month string after day;
3、清空表记录
truncate table test.weblogs;
4、hive alter DDL基本语法
ALTER TABLE name RENAME TO new_name
ALTER TABLE name ADD COLUMNS (col_spec[, col_spec ...])
ALTER TABLE name DROP [COLUMN] column_name
ALTER TABLE name CHANGE column_name new_name new_type
ALTER TABLE name REPLACE COLUMNS (col_spec[, col_spec ...])
(col_spec[, col_spec ...])
ALTER TABLE name DROP [COLUMN] column_name
ALTER TABLE name CHANGE column_name new_name new_type
ALTER TABLE name REPLACE COLUMNS (col_spec[, col_spec ...])
参考:https://www.yiibai.com/hive/hive_alter_table.html
https://blog.csdn.net/xiongbingcool/article/details/83181965
5、hive学习之路 参考
https://www.cnblogs.com/qingyunzong/p/8847775.html
6、select count(*) 得0的问题,hive表有记录,涉及count计算有关设置
1)hive默认设置 hive.compute.query.using.stats=true //改为false 不从元数据取值,可计数,但性能不高,不推荐
2)执行ANALYZE TABLE test.weblogs_agg COMPUTE STATISTICS; //生成统计信息,从元数据库取数、秒查,推荐
(非分区表)ANALYZE TABLE $table_name COMPUTE STATISTICS;
(分区表)ANALYZE TABLE $table_name partition(p=$1) COMPUTE STATISTICS;
参考:https://www.cnblogs.com/barneywill/p/10431323.html
7、执行hive建表,内存用超异常
建表语句:drop table if exists weblogs_agg;
create table if not exists weblogs_agg as
select client_ip,year,month,month_num,count(*)
from weblogs group by client_ip,year,month,month_num;
异常日志:http://slave01.hadoop:19888/jobhistory/app
Container [pid=122930,containerID=container_e03_1557634582519_0006_01_000003] is running beyond physical memory limits. Current usage: 201.5 MB of 170 MB physical memory used; 1.9 GB of 357.0 MB virtual memory used. Killing container.
原因分析:
0)执行hql,对40多万条记录进行分组(汇总)并count计数,可能耗内存较多
1)201.5 MB:任务所占的物理内存
2)170 MB: mapreduce.map.memory.mb 参数默认设置大小
3) 1.9 GB:程序占用的虚拟内存
4) 357 MB(Map): mapreduce.map.memory.mb 170 乘以 yarn.nodemanager.vmem-pmem-ratio 2.1 得到的357
解决路径:
1)调整map memory 从170MB到250MB
2)yarn.nodemanager.vmem-pmem-ratio 从2.1改到4,map虚存到250*4 = 1GB
3)reduce memory调到500MB,reduce虚存到500*4 = 2GB
参考:https://blog.csdn.net/abysscarry/article/details/80331311