Flume主要有以下集中监控方式:
1.JMX监控
配置 {$FLUME_HOME}/flume-env.sh
cd $FLUME_HOME
vi flume-env.sh
JAVA_OPTS="-Dcom.sun.management.jmxremote \
-Dcom.sun.management.jmxremote.authenticate=false \
-Dcom.sun.management.jmxremote.ssl=false \
-Dcom.sun.management.jmxremote.port=54321 \
-Dcom.sun.management.jmxremote.rmi.port=54322
-Djava.rmi.server.hostname=192.168.16.214"
之后启动flume
bin/flume-ng agent -c . -f conf/exec-tail.conf -n a1 -Dflume.root.logger=INFO,console
在图形界面的系统(windows、mac、linux图形)和jdk的环境下启动jconsole
jconsole
可以看到flume应用所使用的内存、线程、类、CPU等使用情况。
2.HTTP监控
Flume可以通过HTTP以JSON形式报告metrics,启用HTTP监控,Flume需要配置一个端口。
配置一个简单的打印conf文件
vi conf/exec-tail.conf
a1.sources = r1
a1.sinks = k1
a1.channels = c1
# Describe/configure the source
a1.sources.r1.type = exec
a1.sources.r1.channels = c1
a1.sources.r1.command = tail -F /tmp/test
# Describe the sink
a1.sinks.k1.type = logger
# Use a channel which buffers events in memory
a1.channels.c1.type = memory
a1.channels.c1.capacity = 1000
a1.channels.c1.transactionCapacity = 100
# Bind the source and sink to the channel
a1.sources.r1.channels = c1
a1.sinks.k1.channel = c1
启动flume并指定http监控形式和端口
bin/flume-ng agent -c . -f conf/exec-tail.conf -n a1 -Dflume.root.logger=INFO,console -Dflume.monitoring.type=http -Dflume.monitoring.port=1234
查看metric监控
用浏览器或者GET方式打开
http://ip:1234/metrics
获得数据如下:
CURL方式:
http://localhost:1234/metrics 2>/dev/null|sed -e 's/([,])\s*/\1\n/g' -e 's/[{}]/\n/g' -e 's/[“,]//g'
获得数据如下:
"CHANNEL.c1":
"ChannelCapacity":"1000"
"ChannelFillPercentage":"0.0"
"Type":"CHANNEL"
"EventTakeSuccessCount":"10"
"ChannelSize":"0"
"EventTakeAttemptCount":"12"
"StartTime":"1476166839656"
"EventPutAttemptCount":"10"
"EventPutSuccessCount":"10"
"StopTime":"0"
"SOURCE.r1":
"EventReceivedCount":"10"
"AppendBatchAcceptedCount":"0"
"Type":"SOURCE"
"AppendReceivedCount":"0"
"EventAcceptedCount":"10"
"StartTime":"1476166840159"
"AppendAcceptedCount":"0"
"OpenConnectionCount":"0"
"AppendBatchReceivedCount":"0"
"StopTime":"0"
3.Ganglia监控
Flume也可以报告metrics到Ganglia 3或者是Ganglia 3.1的metanodes。要将metrics报告到Ganglia,必须在启动的时候就支持Flume Agent。这个Flume Agent使用flume.monitoring作为前缀,通过下面的参数启动。当然也可以在flume-env.sh中设置:
如果要支持Ganglia,可以通过如下命令启动。
bin/flume-ng agent --conf-file example.conf --name a1 -Dflume.monitoring.type=ganglia -Dflume.monitoring.hosts=com.example:1234,com.example2:5455
4.自定义监控
自定义的监控需要实现org.apache.flume.instrumentation.MonitorService
接口。例如有一个HTTP的监控类叫HttpReporting,我可以通过如下方式启动这个监控。
bin/flume-ng agent --conf-file example.conf --name a1 -Dflume.monitoring.type=com.example.reporting.HTTPReporting -Dflume.monitoring.node=com.example:332
报告metrics我们也可以自定义组件,不过一定要继承org.apache.flume.instrumentation.MonitoredCounterGroup
虚拟类。Flume已经实现的类,如下图:
根据上面的规范就可以开发自定义的监控组件了。