zabbix 自动发现监控 tomcat

通过jstack 来查看tomcat的各种性能指标，主要关心tomcat的线程数，堆内存（Young代、Old代、Perm代），GC的参数，GC就是JVM在运行的时候会有一个垃圾回收机制。

1、工具

线程数可以用jstack用具查看，也可以通过/proc/PID/status来查，大差不差

[tzg@tzgsqvapp01 zabbix_agentd.d]$ jstack 37109 | grep nid |wc -l
68

[tzg@tzgsqvapp01 zabbix_agentd.d]$ cat /proc/37109/status|grep Threads|awk '{print $2}'
69

堆内存使用jstat -gccapacity pid 可以显示，VM内存中三代（young,old,perm）对象的使用和占用大小

[tzg@tzgsqvapp01 zabbix_agentd.d]$ jstat -gccapacity 37109
 NGCMN    NGCMX     NGC     S0C   S1C       EC      OGCMN      OGCMX       OGC         OC      PGCMN    PGCMX     PGC       PC     YGC    FGC
2097152.0 2097152.0 2097152.0 262144.0 262144.0 1572864.0  4194304.0  4194304.0  4194304.0  4194304.0 262144.0 2097152.0 262144.0 262144.0      2     0

jstat -gc pid显示gc的信息，查看gc的次数，及时间

[tzg@tzgsqvapp01 zabbix_agentd.d]$ jstat -gc 37109
 S0C    S1C    S0U    S1U      EC       EU        OC         OU       PC     PU    YGC     YGCT    FGC    FGCT     GCT
262144.0 262144.0 98641.9  0.0   1572864.0 729053.2 4194304.0     80.0    262144.0 65608.2      2    0.210   0      0.000    0.210

jstat -class pid 显示加载class的数量，及所占空间等信息。

[tzg@tzgsqvapp01 zabbix_agentd.d]$ jstat -class 37109
Loaded  Bytes  Unloaded  Bytes     Time
  9566 23144.1        0     0.0      12.92

2、脚本
app_list.sh ---------- 发现服务器中tomcat的运行的程序名称
jvm_status.sh -----------通过jstat抓取tomcat程序的JVM的性能
app_jvm_status.conf-----------zabbix自定义键值

app_list.sh (过滤logstash等其他jvm)

#!/bin/bash

APPLIST=`ps -fC java|tail -n +2|grep -v 'logstash'|grep -v 'cloudmonitor'|grep -v 'zookeeper'|awk '{print $9}'|cut -d / -f 4|cut -d - -f 2`
COUNT=`echo "$APPLIST" |wc -l`
INDEX=0
echo '{"data":['
echo "$APPLIST" | while read LINE; do
    echo -n '{"{#APPNAME}":"'$LINE'"}'
    INDEX=`expr $INDEX + 1`
    if [ $INDEX -lt $COUNT ]; then
        echo ','
    fi
done
echo ']}'

jvm_status.sh

#!/bin/bash
appid=$(ps -ef |grep -v "get_app_thread.sh"| grep -w tomcat7-"$1" |grep -v "zabbix"| grep -v "grep" | grep -v "catalina.sh"|grep -v "tomcat.sh"|grep -v "tail"|grep -v "cronolog"|awk '{print $2}')

if [ $# = 1 ];then
cat /proc/$appid/status|grep Threads|awk '{print $2}'

#sudo -u tomcat /opt/app/jdk7/bin/jstat -class $appid
#sudo -u tomcat /opt/app/jdk7/bin/jstat -gccapacity $appid
#sudo -u tomcat /opt/app/jdk7/bin/jstat -gc ${appid}
else
sudo -u tomcat opt/app/jdk7/bin/jstat -$2 ${appid}
fi

app_jvm_status.conf

UserParameter=app.list[*],/etc/zabbix/zabbix_agentd.d/app_list.sh $1
UserParameter=app.thread[*],/etc/zabbix/zabbix_agentd.d/jvm_status.sh $1
UserParameter=app.total_gc_time[*],/etc/zabbix/zabbix_agentd.d/jvm_status.sh $1 gc | sed -n '2p' | awk '{print $$15}'
UserParameter=app.young_gc_num[*],/bin/bash /etc/zabbix/zabbix_agentd.d/jvm_status.sh "$1" gc| sed -n '2p' | awk '{print $$11}'
UserParameter=app.young_gc_time[*],/etc/zabbix/zabbix_agentd.d/jvm_status.sh $1 gc | sed -n '2p' | awk '{print $$12}'
UserParameter=app.full_gc_num[*],/etc/zabbix/zabbix_agentd.d/jvm_status.sh $1 gc | sed -n '2p' | awk '{print $$13}'
UserParameter=app.full_gc_time[*],/etc/zabbix/zabbix_agentd.d/jvm_status.sh $1 gc | sed -n '2p' | awk '{print $$14}'
UserParameter=app.young_gccapacity_mn[*],/etc/zabbix/zabbix_agentd.d/jvm_status.sh $1 gccapacity | sed -n '2p' | awk '{print $$1}'
UserParameter=app.young_gccapacity_mx[*],/etc/zabbix/zabbix_agentd.d/jvm_status.sh $1 gccapacity | sed -n '2p' | awk '{print $$2}'
UserParameter=app.old_gccapacity_mn[*],/etc/zabbix/zabbix_agentd.d/jvm_status.sh $1 gccapacity | sed -n '2p' | awk '{print $$7}'
UserParameter=app.old_gccapacity_mx[*],/etc/zabbix/zabbix_agentd.d/jvm_status.sh $1 gccapacity | sed -n '2p' | awk '{print $$8}'
UserParameter=app.perm_gccapacity_mn[*],/etc/zabbix/zabbix_agentd.d/jvm_status.sh $1 gccapacity | sed -n '2p' | awk '{print $$11}'
UserParameter=app.perm_gccapacity_mx[*],/etc/zabbix/zabbix_agentd.d/jvm_status.sh $1 gccapacity | sed -n '2p' | awk '{print $$12}'
UserParameter=app.oc_gccapacity[*],/etc/zabbix/zabbix_agentd.d/jvm_status.sh $1 gccapacity | sed -n '2p' | awk '{print $$10}'
UserParameter=app.class_num[*],/etc/zabbix/zabbix_agentd.d/jvm_status.sh $1 class | sed -n '2p' | awk '{print $$1}'
UserParameter=app.class_time[*],/etc/zabbix/zabbix_agentd.d/jvm_status.sh $1 class | sed -n '2p' | awk '{print $$5}'

3、实现

java进程把jvm perfdata存到了 /tmp/hsperfdata_{USER}/{PID} 下，由于tomcat使用tomcat账户运行，而zabbix使用zabbix账户运行，zabbix并没有读权限。PID动态生成，无法赋权限。有两个办法，一让zabbix和tomcat使用统一账户运行，二、给zabbix用户添加tomcat 用户的sudo nopasswd 权限。我这边使用第二种方法

echo -e "zabbix ALL=(tomcat)      NOPASSWD:/opt/app/jdk7/bin/jstat" >>/etc/sudoers
sed -i  's/Defaults    requiretty/Defaults    !requiretty/' /etc/sudoers

4、ansible部署

---
 - hosts: tomcat
   tasks:
     - name: 复制app_list.sh到客户机
       copy: src=files/app_list.sh dest=/etc/zabbix/zabbix_agentd.d/app_list.sh
     - name: 复制jvm_status.sh到客户机
       copy: src=files/jvm_status.sh dest=/etc/zabbix/zabbix_agentd.d/jvm_status.sh
     - name: 复制app_jvm_status.conf到客户机
       copy: src=files/app_jvm_status.conf dest=/etc/zabbix/zabbix_agentd.d/app_jvm_status.conf
     - name: chomd
       command: chmod -R +x /etc/zabbix/zabbix_agentd.d/
     - name: zabbix->tomcat
       shell: echo -e "zabbix ALL=(tomcat)      NOPASSWD:/opt/app/jdk7/bin/jstat" >>/etc/sudoers
       shell: sed -i  's/Defaults    requiretty/Defaults    !requiretty/' /etc/sudoers
     - name: restart
       service:
         name: zabbix-agent
         state: restarted

5、zabbix-server
创建discovery_items

获取数据