方案
采用nxlog+fluentd+elasticsearch组件。nxlog 是用 C 语言写的一个跨平台日志收集处理软件，其内部支持使用 Perl 正则和语法来进行数据结构化和逻辑判断操作。nxlog负责收集Windows系统日志，并将日志传给fluentd，fluentd将这些日志信息抛给elasticsearch。fluentd官方文档Collecting Log Data from Windows中也是推荐使用该方案的。
预先准备
- Windows服务器（本次使用的是Win10）。
- Linux服务器（本次使用的CentOS7）。
- 下载elasticsearch并安装在Linux服务器上。
环境搭建步骤
Windows上安装nxlog
- 下载Windows版本的nxlog。
- 直接点击next安装即可。
补充：nxlog默认是安装在C:\Program Files (x86)\nxlog目录中。
Linux上安装fluentd
- 若从rpm库安装，执行如下命令：
$ curl -L https://toolbelt.treasuredata.com/sh/install-redhat-td- agent2.sh | sh
- 常用命令：
$ /etc/init.d/td-agent start
$ /etc/init.d/td-agent stop
$ /etc/init.d/td-agent restart
$ /etc/init.d/td-agent status
测试
修改配置文件
-
修改td-agent配置文件
/etc/td-agent/td-agent.conf$ vim /etc/td-agent/td-agent.conf增添如下配置并保存：
<source> @type tcp format json port 5140 tag windowslog </source> <match windowslog> type elasticsearch </match> -
修改nxlog配置文件
C:\Program Files (x86)\nxlog\conf
改后的文件如下所示：define ROOT C:\Program Files (x86)\nxlog Moduledir %ROOT%\modules CacheDir %ROOT%\data Pidfile %ROOT%\data\nxlog.pid SpoolDir %ROOT%\data LogFile %ROOT%\data\nxlog.log <Extension _syslog> Module xm_syslog </Extension> <Extension json> Module xm_json </Extension> #日志输入端 <Input in> Module im_msvistalog #For windows 2003 and earlier use the following: #Module im_mseventlog Exec to_json(); </Input> #日志输出端 <Output out> Module om_tcp Host 192.168.159.133 #安装有td-agent的服务器地址 Port 5140 #此端口要与td-agent监听端口一致 </Output> <Output outfile> Module om_file File "D:\logs\log" #为了便于观察，将收集到的系统日志存放在该位置下 </Output> #绑定某输入到具体某输出 <Route r1> Path in => out </Route> <Route r2> Path in => outfile </Route>
启动服务
-
启动td-agent：
$ /etc/init.d/td-agent start -
启动nxlog，使用powershell进入到
C:\Program Files (x86)\nxlog目录下，执行命令：C:\Program Files (x86)\nxlog> .\\nxlog.exe -f -c ./conf/nxlog.conf部分输出如下：
2017-07-13 14:43:30 INFO connecting to 192.168.159.133:5140 2017-07-13 14:43:30 INFO nxlog-ce-2.9.1716 started
结果
在Linux控制台查看elasticsearch中的索引：
[root@localhost ~]# curl 'localhost:9200/_cat/indices?v'
health status index uuid pri rep docs.count docs.deleted store.size pri.store.size
yellow open fluentd c05Ykvp_RPebnOppfsa1_w 5 1 2 0 67.7kb 67.7kb
yellow open .kibana Zldk9n-BSweLtaeVNMsNtg 1 1 8 0 54kb 54kb
查看fluentd索引中的数据，部分输出如下：
[root@localhost ~]# curl -XGET 'localhost:9200/fluentd/_search?pretty'
{
"took" : 363,
"timed_out" : false,
"_shards" : {
"total" : 5,
"successful" : 5,
"failed" : 0
},
"hits" : {
"total" : 68,
"max_score" : 1.0,
"hits" : [
{
"_index" : "fluentd",
"_type" : "fluentd",
"_id" : "AV06sthgdjHN6maqugXM",
"_score" : 1.0,
"_source" : {
"EventTime" : "2017-07-13 14:45:59",
"Hostname" : "LAPTOP-AQ1VQEMH",
"Keywords" : 576460752303423488,
"EventType" : "INFO",
"SeverityValue" : 2,
"Severity" : "INFO",
"EventID" : 20,
"SourceName" : "Microsoft-Windows-Audio",
"ProviderGuid" : "{AE4BD3BE-F36F-45B6-8D21-BDD6FB832853}",
"Version" : 0,
"Task" : 116,
"OpcodeValue" : 0,
"RecordNumber" : 113490,
"ProcessID" : 6804,
"ThreadID" : 13988,
"Channel" : "Microsoft-Windows-Audio/PlaybackManager",
"Domain" : "LAPTOP-AQ1VQEMH",
"AccountName" : "WHL",
"UserID" : "S-1-5-21-2442955922-1256182451-2774828708-1001",
"AccountType" : "User",
"Message" : "格式: 1\r\n采样率 : 22050Hz\r\n已卸载: false",
"Opcode" : "信息",
"Format" : "1",
"SamplingRate" : "22050",
"bAudioSrvStreamResourceType" : "false",
"EventReceivedTime" : "2017-07-13 14:46:00",
"SourceModuleName" : "in",
"SourceModuleType" : "im_msvistalog"
}
},
{
"_index" : "fluentd",
"_type" : "fluentd",
"_id" : "AV06uizFdjHN6maqugXh",
"_score" : 1.0,
"_source" : {
"EventTime" : "2017-07-13 14:53:26",
"Hostname" : "LAPTOP-AQ1VQEMH",
"Keywords" : 576460752303423488,
"EventType" : "INFO",
"SeverityValue" : 2,
"Severity" : "INFO",
"EventID" : 505,
"SourceName" : "Microsoft-Windows-Application-Experience",
"ProviderGuid" : "{EEF54E71-0661-422D-9A98-82FD4940B820}",
"Version" : 0,
"Task" : 0,
"OpcodeValue" : 0,
"RecordNumber" : 31916,
"ProcessID" : 14276,
"ThreadID" : 13872,
"Channel" : "Microsoft-Windows-Application-Experience/Program-Telemetry",
"Domain" : "LAPTOP-AQ1VQEMH",
"AccountName" : "WHL",
"UserID" : "S-1-5-21-2442955922-1256182451-2774828708-1001",
"AccountType" : "User",
"Message" : "兼容性修复程序已应用于 C:\\Users\\WHL\\AppData\\Local\\Google\\Chrome\\Application\\chrome.exe。\r\n修补程序信息: Google Chrome，{18A8B5B2-9D2F-4DB2-8307-196B5CC0CE6B}，0x80010101。",
"Opcode" : "信息",
"EventReceivedTime" : "2017-07-13 14:53:28",
"SourceModuleName" : "in",
"SourceModuleType" : "im_msvistalog"
}
},
打开D:\logs下的log文件，部分输出如下：
{"EventTime":"2017-07-13 14:43:51","Hostname":"LAPTOP-AQ1VQEMH","Keywords":4611686018429485056,"EventType":"INFO","SeverityValue":2,"Severity":"INFO","EventID":146,"SourceName":"Microsoft-Windows-Ntfs","ProviderGuid":"{3FF37A1C-A68D-4D6E-8C9B-F79E8B16C482}","Version":1,"Task":0,"OpcodeValue":0,"RecordNumber":11792,"ProcessID":4,"ThreadID":8584,"Channel":"Microsoft-Windows-Ntfs/Operational","Domain":"NT AUTHORITY","AccountName":"SYSTEM","UserID":"S-1-5-18","AccountType":"Well Known Group","Message":"IO latency summary:\r\n\r\n Volume Id: {4E0E6863-60CA-4F3A-ADC1-EA02A7E953E3}\r\n Volume name: C:\r\n Is boot volume: true\r\n\r\n Interval duration: 3627921184 us\r\n\r\n Non-cached reads:\r\n IO count: 7387\r\n Total bytes: 466456052\r\n Avg latency: 8210412 ns\r\n\r\n Non-cached writes: \r\n IO count: 24179\r\n Total bytes: 412844919\r\n Avg latency: 3188872 ns\r\n\r\n File flushes: \r\n IO count: 1622\r\n Avg latency: 17203859 ns\r\n\r\n Volume flushes: \r\n IO count: 0\r\n Avg latency: 0 ns\r\n\r\n File level trims: \r\n IO count: 1\r\n Total bytes: 147374080\r\n Extents count: 1\r\n Avg latency: 1947000 ns\r\n\r\n Volume trims: \r\n IO count: 324\r\n Total bytes: 1420537856\r\n Extents count: 1548\r\n Avg latency: 1463830 ns\r\n\r\n For more details see the details tab.\r\n","Opcode":"信息","VolumeCorrelationId":"{4E0E6863-60CA-4F3A-ADC1-EA02A7E953E3}","VolumeNameLength":"2","VolumeName":"C:","IsBootVolume":"true","HighIoLatencyCount":"0","IntervalDurationUs":"3627921184","NCReadIOCount":"7387","NCReadTotalBytes":"466456052","NCReadAvgLatencyNs":"8210412","NCWriteIOCount":"24179","NCWriteTotalBytes":"412844919","NCWriteAvgLatencyNs":"3188872","FileFlushCount":"1622","FileFlushAvgLatencyNs":"17203859","VolumeFlushCount":"0","VolumeFlushAvgLatencyNs":"0","FileLevelTrimCount":"1","FileLevelTrimTotalBytes":"147374080","FileLevelTrimExtentsCount":"1","FileLevelTrimAvgLatencyNs":"1947000","VolumeTrimCount":"324","VolumeTrimTotalBytes":"1420537856","VolumeTrimExtentsCount":"1548","VolumeTrimAvgLatencyNs":"1463830","IoBucketsCount":"48","TotalBytesBucketsCount":"44","ExtentsBucketsCount":"4","IoCount":"35","TotalLatencyUs":"51080","TotalBytes":"1208320","TrimExtentsCount":"1537","IoTypeIndex":"0","EventReceivedTime":"2017-07-13 14:43:52","SourceModuleName":"in","SourceModuleType":"im_msvistalog"}
{"EventTime":"2017-07-13 14:43:51","Hostname":"LAPTOP-AQ1VQEMH","Keywords":4611686018429485056,"EventType":"INFO","SeverityValue":2,"Severity":"INFO","EventID":146,"SourceName":"Microsoft-Windows-Ntfs","ProviderGuid":"{3FF37A1C-A68D-4D6E-8C9B-F79E8B16C482}","Version":1,"Task":0,"OpcodeValue":0,"RecordNumber":11793,"ProcessID":4,"ThreadID":8584,"Channel":"Microsoft-Windows-Ntfs/Operational","Domain":"NT AUTHORITY","AccountName":"SYSTEM","UserID":"S-1-5-18","AccountType":"Well Known Group","Message":"IO latency summary:\r\n\r\n Volume Id: {EFA1B98E-1BD3-45F1-A956-882127C35622}\r\n Volume name: D:\r\n Is boot volume: false\r\n\r\n Interval duration: 3627921030 us\r\n\r\n Non-cached reads:\r\n IO count: 11248\r\n Total bytes: 820325912\r\n Avg latency: 4644197 ns\r\n\r\n Non-cached writes: \r\n IO count: 7992\r\n Total bytes: 242775864\r\n Avg latency: 15979689 ns\r\n\r\n File flushes: \r\n IO count: 49\r\n Avg latency: 10135571 ns\r\n\r\n Volume flushes: \r\n IO count: 0\r\n Avg latency: 0 ns\r\n\r\n File level trims: \r\n IO count: 0\r\n Total bytes: 0\r\n Extents count: 0\r\n Avg latency: 0 ns\r\n\r\n Volume trims: \r\n IO count: 2\r\n Total bytes: 16384\r\n Extents count: 4\r\n Avg latency: 893000 ns\r\n\r\n For more details see the details tab.\r\n","Opcode":"信息","VolumeCorrelationId":"{EFA1B98E-1BD3-45F1-A956-882127C35622}","VolumeNameLength":"2","VolumeName":"D:","IsBootVolume":"false","HighIoLatencyCount":"0","IntervalDurationUs":"3627921030","NCReadIOCount":"11248","NCReadTotalBytes":"820325912","NCReadAvgLatencyNs":"4644197","NCWriteIOCount":"7992","NCWriteTotalBytes":"242775864","NCWriteAvgLatencyNs":"15979689","FileFlushCount":"49","FileFlushAvgLatencyNs":"10135571","VolumeFlushCount":"0","VolumeFlushAvgLatencyNs":"0","FileLevelTrimCount":"0","FileLevelTrimTotalBytes":"0","FileLevelTrimExtentsCount":"0","FileLevelTrimAvgLatencyNs":"0","VolumeTrimCount":"2","VolumeTrimTotalBytes":"16384","VolumeTrimExtentsCount":"4","VolumeTrimAvgLatencyNs":"893000","IoBucketsCount":"35","TotalBytesBucketsCount":"32","ExtentsBucketsCount":"1","IoCount":"2902","TotalLatencyUs":"1952578","TotalBytes":"41503232","TrimExtentsCount":"4","IoTypeIndex":"0","EventReceivedTime":"2017-07-13 14:43:52","SourceModuleName":"in","SourceModuleType":"im_msvistalog"}
{"EventTime":"2017-07-13 14:45:52","Hostname":"LAPTOP-AQ1VQEMH","Keywords":576460752303423488,"EventType":"INFO","SeverityValue":2,"Severity":"INFO","EventID":20,"SourceName":"Microsoft-Windows-Audio","ProviderGuid":"{AE4BD3BE-F36F-45B6-8D21-BDD6FB832853}","Version":0,"Task":116,"OpcodeValue":0,"RecordNumber":113487,"ProcessID":6804,"ThreadID":13716,"Channel":"Microsoft-Windows-Audio/PlaybackManager","Domain":"LAPTOP-AQ1VQEMH","AccountName":"WHL","UserID":"S-1-5-21-2442955922-1256182451-2774828708-1001","AccountType":"User","Message":"格式: 1\r\n采样率 : 22050Hz\r\n已卸载: false","Opcode":"信息","Format":"1","SamplingRate":"22050","bAudioSrvStreamResourceType":"false","EventReceivedTime":"2017-07-13 14:45:54","SourceModuleName":"in","SourceModuleType":"im_msvistalog"}
{"EventTime":"2017-07-13 14:45:59","Hostname":"LAPTOP-AQ1VQEMH","Keywords":576460752303423488,"EventType":"INFO","SeverityValue":2,"Severity":"INFO","EventID":20,"SourceName":"Microsoft-Windows-Audio","ProviderGuid":"{AE4BD3BE-F36F-45B6-8D21-BDD6FB832853}","Version":0,"Task":116,"OpcodeValue":0,"RecordNumber":113488,"ProcessID":6804,"ThreadID":12172,"Channel":"Microsoft-Windows-Audio/PlaybackManager","Domain":"LAPTOP-AQ1VQEMH","AccountName":"WHL","UserID":"S-1-5-21-2442955922-1256182451-2774828708-1001","AccountType":"User","Message":"格式: 1\r\n采样率 : 22050Hz\r\n已卸载: false","Opcode":"信息","Format":"1","SamplingRate":"22050","bAudioSrvStreamResourceType":"false","EventReceivedTime":"2017-07-13 14:46:00","SourceModuleName":"in","SourceModuleType":"im_msvistalog"}
{"EventTime":"2017-07-13 14:45:59","Hostname":"LAPTOP-AQ1VQEMH","Keywords":576460752303423488,"EventType":"INFO","SeverityValue":2,"Severity":"INFO","EventID":20,"SourceName":"Microsoft-Windows-Audio","ProviderGuid":"{AE4BD3BE-F36F-45B6-8D21-BDD6FB832853}","Version":0,"Task":116,"OpcodeValue":0,"RecordNumber":113489,"ProcessID":6804,"ThreadID":9620,"Channel":"Microsoft-Windows-Audio/PlaybackManager","Domain":"LAPTOP-AQ1VQEMH","AccountName":"WHL","UserID":"S-1-5-21-2442955922-1256182451-2774828708-1001","AccountType":"User","Message":"格式: 1\r\n采样率 : 22050Hz\r\n已卸载: false","Opcode":"信息","Format":"1","SamplingRate":"22050","bAudioSrvStreamResourceType":"false","EventReceivedTime":"2017-07-13 14:46:00","SourceModuleName":"in","SourceModuleType":"im_msvistalog"}
{"EventTime":"2017-07-13 14:45:59","Hostname":"LAPTOP-AQ1VQEMH","Keywords":576460752303423488,"EventType":"INFO","SeverityValue":2,"Severity":"INFO","EventID":20,"SourceName":"Microsoft-Windows-Audio","ProviderGuid":"{AE4BD3BE-F36F-45B6-8D21-BDD6FB832853}","Version":0,"Task":116,"OpcodeValue":0,"RecordNumber":113490,"ProcessID":6804,"ThreadID":13988,"Channel":"Microsoft-Windows-Audio/PlaybackManager","Domain":"LAPTOP-AQ1VQEMH","AccountName":"WHL","UserID":"S-1-5-21-2442955922-1256182451-2774828708-1001","AccountType":"User","Message":"格式: 1\r\n采样率 : 22050Hz\r\n已卸载: false","Opcode":"信息","Format":"1","SamplingRate":"22050","bAudioSrvStreamResourceType":"false","EventReceivedTime":"2017-07-13 14:46:00","SourceModuleName":"in","SourceModuleType":"im_msvistalog"}
{"EventTime":"2017-07-13 14:45:59","Hostname":"LAPTOP-AQ1VQEMH","Keywords":576460752303423488,"EventType":"INFO","SeverityValue":2,"Severity":"INFO","EventID":20,"SourceName":"Microsoft-Windows-Audio","ProviderGuid":"{AE4BD3BE-F36F-45B6-8D21-BDD6FB832853}","Version":0,"Task":116,"OpcodeValue":0,"RecordNumber":113491,"ProcessID":6804,"ThreadID":11340,"Channel":"Microsoft-Windows-Audio/PlaybackManager","Domain":"LAPTOP-AQ1VQEMH","AccountName":"WHL","UserID":"S-1-5-21-2442955922-1256182451-2774828708-1001","AccountType":"User","Message":"格式: 1\r\n采样率 : 22050Hz\r\n已卸载: false","Opcode":"信息","Format":"1","SamplingRate":"22050","bAudioSrvStreamResourceType":"false","EventReceivedTime":"2017-07-13 14:46:00","SourceModuleName":"in","SourceModuleType":"im_msvistalog"}
