Nginx Kafka Module环境搭建
参考官网：https://github.com/brg-liuwei/ngx_kafka_module/blob/master/README.md
进入目录 cd /usr/local/src/
1.上传下载的安装文件
2.进入libkafka目录 cd librdkafka
安装依赖 yum install -y gcc gcc-c++ pcre-devel zlib-level
编译安装 安装完成的默认路径/usr/local/include
./configure
make && make install
3.进入nginx目录 cd nginx-1.8.1/ （编译nginx，然后将将插件同时编译）
./configure --add-module=/usr/local/src/ngx_kafka_module/
make
make install
4.修改配置文件，需要提前启动zk并创建要配置的主题信息
5.开启消费者接收消息
6.启动nginx 进入nginx sbin目录./nginx ，
1）报错，找不到kafka.so.1的文件 error while loading shared libraries: librdkafka.so.1: cannot open shared object file: No such file or directory
2）加载so库 echo "/usr/local/lib" >> /etc/ld.so.conf ldconfig
3）重新启动成功访问：
curl xxxxx/brokers/topics -d "hello ngx_kafka_module" -v
7.页面埋点采集配置
8.采集服务器js代码
(function () {
var params = {};
//Document对象数据
if(document) {
params.domain = document.domain || '';
params.url = document.URL || '';
params.title = document.title || '';
params.referrer = document.referrer || '';
}
//Window对象数据
if(window && window.screen) {
params.sh = window.screen.height || 0;
params.sw = window.screen.width || 0;
params.cd = window.screen.colorDepth || 0;
}
//navigator对象数据
if(navigator) {
params.lang = navigator.language || '';
}
//解析_maq配置
if(_maq) {
for(var i in _maq) {
var paramsName = _maq[i][0];
var paramsValue = _maq[i][1];
params[paramsName] = paramsValue;
}
}
//拼接参数串
var args = '';
for(var i in params) {
if(args != '') {
args += '&';
}
args += i + '=' + encodeURIComponent(params[i]);
}
//通过Image对象请求后端脚本
var img = new Image(1, 1);
img.src = 'http://xxxxx/log.gif?' + args;
})();
html页面的js脚本，访问采集服务器的js
<script type="text/javascript">
var _maq = _maq || [];
$(function (){
$("button").click(function () {
var listenClick = $(this).attr("data-id");
var userName = $("#userId").val();
var date = getFormatDate();
_maq.push(['_setAccount', userName]);
_maq.push(['_setClick', listenClick]);
_maq.push(['_setClickTime', date]);
//解析_maq配置
var ma = document.createElement('script');
ma.type = 'text/javascript';
ma.async = true;
ma.src = 'http://xxxxx/ma/ma.js';
var s = document.getElementsByTagName('script')[0];
s.parentNode.insertBefore(ma, s);
})
})
function getFormatDate(){
var nowDate = new Date();
var year = nowDate.getFullYear();
var month = nowDate.getMonth() + 1 < 10 ? "0" + (nowDate.getMonth() + 1) : nowDate.getMonth() + 1;
var date = nowDate.getDate() < 10 ? "0" + nowDate.getDate() : nowDate.getDate();
var hour = nowDate.getHours()< 10 ? "0" + nowDate.getHours() : nowDate.getHours();
var minute = nowDate.getMinutes()< 10 ? "0" + nowDate.getMinutes() : nowDate.getMinutes();
var second = nowDate.getSeconds()< 10 ? "0" + nowDate.getSeconds() : nowDate.getSeconds();
return year + "-" + month + "-" + date+" "+hour+":"+minute+":"+second;
}
</script>
配置nginx.conf
配置kafka信息
kafka;
kafka_broker_list xxxxx:9092;
server {
listen 80;
server_name xxxxx;
location = /mykafka/brokers/topics {
kafka_topic nginx_topic;
}
在此接收js的log.gif请求，模拟post发送到kafka
location /log.gif {
#伪装成gif文件
default_type image/gif;
#本身关闭access_log，通过subrequest记录log
access_log off;
access_by_lua "
local uid = ngx.var.cookie___utrace
if not uid then
uid = ngx.md5(ngx.now() .. ngx.var.remote_addr .. ngx.var.http_user_agent)
end
ngx.header['Set-Cookie'] = {'__utrace=' .. uid .. '; path=/'}
if ngx.var.arg_domain then
ngx.location.capture('/i-log?' .. ngx.var.args .. '&utrace=' .. uid)
ngx.location.capture('/mykafka/brokers/topics',{method=ngx.HTTP_POST,body=ngx.var.args})
end
end
";
9.写入日志成功，kafka接收数据成功接入如下
