利用Logstash同步MySql数据至ElasticSearch
安装Logstash
参考链接Installing Logstash
Download and install the public signing key:
rpm --importhttps://artifacts.elastic.co/GPG-KEY-elasticsearch
Add the following in your /etc/yum.repos.d/ directory in a file with a .repo suffix, for example logstash.repo
[logstash-7.x]name=Elastic repository for 7.x packages
baseurl=https://artifacts.elastic.co/packages/7.x/yum
gpgcheck=1gpgkey=https://artifacts.elastic.co/GPG-KEY-elasticsearch
enabled=1autorefresh=1type=rpm-md
And your repository is ready for use. You can install it with:
sudo yum install logstash
yum方式安装好后，程序目录位于/usr/share/logstash/, 配置文件位于/etc/logstash/，
配置MySql数据同步
在/etc/logstash中创建mysql.conf配置文件,参考链接jdbc input plugin
input {
jdbc {
jdbc_driver_class => "com.mysql.jdbc.Driver"
jdbc_connection_string => "jdbc:mysql://xxx.mysql.rds.aliyuncs.com/xxx"
jdbc_user => "root"
jdbc_password => "******"
schedule => "* * * * *"
statement => "SELECT * from biz_goods where create_time > :sql_last_value AND create_time < NOW() AND is_delete=0 ORDER BY create_time desc"
}
}
output {
elasticsearch {
hosts => ["127.0.0.1:9200"]
index => "goods"
document_id => "%{id}"
}
}
在官方文档中input->jdbc中需要指定jdbc_driver_library => "mysql-connector-java-5.1.36-bin.jar"，此处如果将mysq-connector-java.jar放在常规目录通过路径指定，将会出现如下错误
LogStash::ConfigurationError
com.mysql.jdbc.Driver not loaded. Are you sure you've included the correct jdbc driver in :jdbc_driver_library?
解决方式为在mvnrepository下载jar文件后，将该文件拷贝至/usr/share/logstash/logstash-core/lib/jars 中，然后无需在input->jdbc中配置jdbc_driver_library
运行logstash开启同步
./logstash -f /etc/logstash/mysql.conf
