通过jstack 来查看tomcat的各种性能指标，主要关心tomcat的线程数，堆内存（Young代、Old代、Perm代），GC的参数，GC就是JVM在运行的时候会有一个垃圾回收机制。
1、工具
线程数可以用jstack用具查看，也可以通过/proc/PID/status来查，大差不差
[tzg@tzgsqvapp01 zabbix_agentd.d]$ jstack 37109 | grep nid |wc -l
68
[tzg@tzgsqvapp01 zabbix_agentd.d]$ cat /proc/37109/status|grep Threads|awk '{print $2}'
69
堆内存使用jstat -gccapacity pid 可以显示，VM内存中三代（young,old,perm）对象的使用和占用大小
[tzg@tzgsqvapp01 zabbix_agentd.d]$ jstat -gccapacity 37109
NGCMN NGCMX NGC S0C S1C EC OGCMN OGCMX OGC OC PGCMN PGCMX PGC PC YGC FGC
2097152.0 2097152.0 2097152.0 262144.0 262144.0 1572864.0 4194304.0 4194304.0 4194304.0 4194304.0 262144.0 2097152.0 262144.0 262144.0 2 0
jstat -gc pid显示gc的信息，查看gc的次数，及时间
[tzg@tzgsqvapp01 zabbix_agentd.d]$ jstat -gc 37109
S0C S1C S0U S1U EC EU OC OU PC PU YGC YGCT FGC FGCT GCT
262144.0 262144.0 98641.9 0.0 1572864.0 729053.2 4194304.0 80.0 262144.0 65608.2 2 0.210 0 0.000 0.210
jstat -class pid 显示加载class的数量，及所占空间等信息。
[tzg@tzgsqvapp01 zabbix_agentd.d]$ jstat -class 37109
Loaded Bytes Unloaded Bytes Time
9566 23144.1 0 0.0 12.92
2、脚本
app_list.sh ---------- 发现服务器中tomcat的运行的程序名称
jvm_status.sh -----------通过jstat抓取tomcat程序的JVM的性能
app_jvm_status.conf-----------zabbix自定义键值
app_list.sh (过滤logstash等其他jvm)
#!/bin/bash
APPLIST=`ps -fC java|tail -n +2|grep -v 'logstash'|grep -v 'cloudmonitor'|grep -v 'zookeeper'|awk '{print $9}'|cut -d / -f 4|cut -d - -f 2`
COUNT=`echo "$APPLIST" |wc -l`
INDEX=0
echo '{"data":['
echo "$APPLIST" | while read LINE; do
echo -n '{"{#APPNAME}":"'$LINE'"}'
INDEX=`expr $INDEX + 1`
if [ $INDEX -lt $COUNT ]; then
echo ','
fi
done
echo ']}'
jvm_status.sh
#!/bin/bash
appid=$(ps -ef |grep -v "get_app_thread.sh"| grep -w tomcat7-"$1" |grep -v "zabbix"| grep -v "grep" | grep -v "catalina.sh"|grep -v "tomcat.sh"|grep -v "tail"|grep -v "cronolog"|awk '{print $2}')
if [ $# = 1 ];then
cat /proc/$appid/status|grep Threads|awk '{print $2}'
#sudo -u tomcat /opt/app/jdk7/bin/jstat -class $appid
#sudo -u tomcat /opt/app/jdk7/bin/jstat -gccapacity $appid
#sudo -u tomcat /opt/app/jdk7/bin/jstat -gc ${appid}
else
sudo -u tomcat opt/app/jdk7/bin/jstat -$2 ${appid}
fi
app_jvm_status.conf
UserParameter=app.list[*],/etc/zabbix/zabbix_agentd.d/app_list.sh $1
UserParameter=app.thread[*],/etc/zabbix/zabbix_agentd.d/jvm_status.sh $1
UserParameter=app.total_gc_time[*],/etc/zabbix/zabbix_agentd.d/jvm_status.sh $1 gc | sed -n '2p' | awk '{print $$15}'
UserParameter=app.young_gc_num[*],/bin/bash /etc/zabbix/zabbix_agentd.d/jvm_status.sh "$1" gc| sed -n '2p' | awk '{print $$11}'
UserParameter=app.young_gc_time[*],/etc/zabbix/zabbix_agentd.d/jvm_status.sh $1 gc | sed -n '2p' | awk '{print $$12}'
UserParameter=app.full_gc_num[*],/etc/zabbix/zabbix_agentd.d/jvm_status.sh $1 gc | sed -n '2p' | awk '{print $$13}'
UserParameter=app.full_gc_time[*],/etc/zabbix/zabbix_agentd.d/jvm_status.sh $1 gc | sed -n '2p' | awk '{print $$14}'
UserParameter=app.young_gccapacity_mn[*],/etc/zabbix/zabbix_agentd.d/jvm_status.sh $1 gccapacity | sed -n '2p' | awk '{print $$1}'
UserParameter=app.young_gccapacity_mx[*],/etc/zabbix/zabbix_agentd.d/jvm_status.sh $1 gccapacity | sed -n '2p' | awk '{print $$2}'
UserParameter=app.old_gccapacity_mn[*],/etc/zabbix/zabbix_agentd.d/jvm_status.sh $1 gccapacity | sed -n '2p' | awk '{print $$7}'
UserParameter=app.old_gccapacity_mx[*],/etc/zabbix/zabbix_agentd.d/jvm_status.sh $1 gccapacity | sed -n '2p' | awk '{print $$8}'
UserParameter=app.perm_gccapacity_mn[*],/etc/zabbix/zabbix_agentd.d/jvm_status.sh $1 gccapacity | sed -n '2p' | awk '{print $$11}'
UserParameter=app.perm_gccapacity_mx[*],/etc/zabbix/zabbix_agentd.d/jvm_status.sh $1 gccapacity | sed -n '2p' | awk '{print $$12}'
UserParameter=app.oc_gccapacity[*],/etc/zabbix/zabbix_agentd.d/jvm_status.sh $1 gccapacity | sed -n '2p' | awk '{print $$10}'
UserParameter=app.class_num[*],/etc/zabbix/zabbix_agentd.d/jvm_status.sh $1 class | sed -n '2p' | awk '{print $$1}'
UserParameter=app.class_time[*],/etc/zabbix/zabbix_agentd.d/jvm_status.sh $1 class | sed -n '2p' | awk '{print $$5}'
3、实现
java进程把jvm perfdata存到了 /tmp/hsperfdata_{USER}/{PID} 下，由于tomcat使用tomcat账户运行，而zabbix使用zabbix账户运行，zabbix并没有读权限。PID动态生成，无法赋权限。有两个办法，一让zabbix和tomcat使用统一账户运行，二、给zabbix用户添加tomcat 用户的sudo nopasswd 权限。我这边使用第二种方法
echo -e "zabbix ALL=(tomcat) NOPASSWD:/opt/app/jdk7/bin/jstat" >>/etc/sudoers
sed -i 's/Defaults requiretty/Defaults !requiretty/' /etc/sudoers
4、ansible部署
---
- hosts: tomcat
tasks:
- name: 复制app_list.sh到客户机
copy: src=files/app_list.sh dest=/etc/zabbix/zabbix_agentd.d/app_list.sh
- name: 复制jvm_status.sh到客户机
copy: src=files/jvm_status.sh dest=/etc/zabbix/zabbix_agentd.d/jvm_status.sh
- name: 复制app_jvm_status.conf到客户机
copy: src=files/app_jvm_status.conf dest=/etc/zabbix/zabbix_agentd.d/app_jvm_status.conf
- name: chomd
command: chmod -R +x /etc/zabbix/zabbix_agentd.d/
- name: zabbix->tomcat
shell: echo -e "zabbix ALL=(tomcat) NOPASSWD:/opt/app/jdk7/bin/jstat" >>/etc/sudoers
shell: sed -i 's/Defaults requiretty/Defaults !requiretty/' /etc/sudoers
- name: restart
service:
name: zabbix-agent
state: restarted
5、zabbix-server
创建discovery_items
获取数据
