说明
系统中经常有一个事件打印多行日志，比如java、php日志，这里需要将这多行日志一个事件合并到一起发送给elasticsearch，使用logstash的Multiline
例如php日志格式为：
[18-Sep-2016 15:55:58] [pool www] pid 12548
script_filename = /mnt/data/www/mytest/index.php
[0x00007f82321a9688] filemtime() /mnt/data/www/mytest/Session/drivers/Session_files_driver.php:348
[0x00007fffe4c158b0] gc() unknown:0
[0x00007f82321a8f08] session_start() /mnt/data/www/mytest/libraries/Session/Session.php:140
[0x00007f82321a7ea8] __construct() /mnt/data/www/mytest/Session/drivers/Session_files_driver.php:348
[0x00007f82321a6ec8] _ci_init_library() /mnt/data/www/mytest/Session/drivers/Session_files_driver.php:348
[0x00007f82321a6478] _ci_load_stock_library() /mnt/data/www/mytest/Session/drivers/Session_files_driver.php:348
[0x00007f82321a5c10] _ci_load_library() /mnt/data/www/mytest/Session/drivers/Session_files_driver.php:348
[0x00007f82321a0b48] +++ dump failed
[18-Sep-2016 15:55:58] [pool www] pid 12548
script_filename = /mnt/data/www/mytest/index.php
[0x00007f82321a9688] filemtime() /mnt/data/www/mytest/Session/drivers/Session_files_driver.php:348
[0x00007fffe4c158b0] gc() unknown:0
[0x00007f82321a8f08] session_start() /mnt/data/www/mytest/libraries/Session/Session.php:140
[0x00007f82321a7ea8] __construct() /mnt/data/www/mytest/Session/drivers/Session_files_driver.php:348
[0x00007f82321a6ec8] _ci_init_library() /mnt/data/www/mytest/Session/drivers/Session_files_driver.php:348
[0x00007f82321a6478] _ci_load_stock_library() /mnt/data/www/mytest/Session/drivers/Session_files_driver.php:348
[0x00007f82321a5c10] _ci_load_library() /mnt/data/www/mytest/Session/drivers/Session_files_driver.php:348
[0x00007f82321a0b48] +++ dump failed
......
配置
input {
file {
path => "/var/log/php/www.log.slow"
codec => multiline {
pattern => "^\[\d{2}-"
negate => true
what => "previous"
}
}
}
output {
elasticsearch {
hosts => "172.16.11.199"
index => "php-%{+YYYY.MM.dd}"
}
}
配置解释
- codec 为input的编码插件，来修改日志输入的格式，可以在logstash输入的时候处理不同的数据，而不用再filter中去过滤
- multiline 合并多行数据
- pattern 正则匹配事件中的行
- negate 默认为false，适用于multiline codec 行不匹配pattern选项指定的正则表达式
- what 如果正则表达式匹配，那么事件属于上一个事件还是下一个事件，可以为next和previous
以上综合意思为：
如果该条日志不匹配pattern中的正则，则该条日志属于上一个事件
