运维101

Rundeck

一般经常要跑的job，如果需求比cron复杂，便用这个跑。

http://rundeck.org/docs/index.html
http://rundeck.org/docs/administration/index.html
http://rundeck.org/docs/manual/index.html

比较有用的Bash指令

lsof -i :4443

用这个判定4443port上的听众是何程序。

df

用这个来判定硬盘上剩余多少空间。

df

Filesystem    512-blocks      Used Available Capacity  iused   ifree %iused  Mounted on
/dev/disk1     234610688 206268368  27830320    89% 25847544 3478790   88%   /
devfs                657       657         0   100%     1138       0  100%   /dev
map -hosts             0         0         0   100%        0       0  100%   /net
map auto_home          0         0         0   100%        0       0  100%   /home

判定某regex的所有文件大小之和

hdfs dfs -ls path/prefix-* | awk '{s+= $3} END {print s}'

ls -lGrt
用这个来一眼看到本文件夹里最近更新的文件是谁，各个文件都是什么大小。

ls -lGrt
total 16
-rw-r--r--  1 root  staff  12 Jan 16 13:14 test1
-rw-r--r--  1 root  staff   9 Jan 16 13:14 test2
​```

Change newlines to commas, and quote the words. 

cat [filename.txt] | tr '\n' ', ' | sed 's/,/","/g'

Reverse search bash history: 
`ctrl+r`

Compare 2 files: 

function compare {
comm <(sort "$1") <(sort "$2")
}

Convert excel to csv: 

ls *.xls | while read t; do xlsx $t | tail -n +2 > $t.sth.csv; done

Find a file in the current directory with the name "something". 
` find . -name "something" `

You can suppress hadoop logs by editing the `hadoop-env.sh` or `log4j.properties` files within `/etc/hadoop/conf`. However, I haven't quite found how to do it for a single command. 

http://hadooptutorial.info/suppress-warning-messages-in-hadoop/