docker安装mysql、logstash同步数据到es
一、安装mysql
- 1.拉取镜像
docker pull mysql:5.7.26
- 2.启动
docker run --name mysql -e MYSQL_ROOT_PASSWORD=123456 -d -i -p 3306:3306 mysql:5.7.26
- 3.授予远程访问权限
docker exec -it mysql bash
mysql -uroot -p123456
use mysql;
select host,user from user;
ALTER USER 'root'@'%' IDENTIFIED WITH mysql_native_password BY '123456';
flush privileges;
- 4.开启mysql binlog
docker exec -it mysql bash
apt-get update
apt-get install vim
vim etc/mysql/mysql.conf.d/mysqld.cnf
在mysqld配置项下添加如下几行内容
log-bin=/var/lib/mysql/mysql-bin
server-id=1
binlog-format=ROW
mysqld.cnf
重启mysql。
- 5.查看binlog是否开启，数据库客户端执行如下命令
show variables like '%log_bin%';
log_bin
- 6.创建mysql账户，用于让canal同步数据
use mysql;
create user canal identified by 'canal';
grant select,replication slave,replication client on *.* to 'canal'@'%';
flush privileges;
- 7.创建测试用数据库
create database test;
use test;
CREATE TABLE `user` (
`id` int(11) unsigned NOT NULL AUTO_INCREMENT,
`name` varchar(50) NOT NULL,
`age` int(11) NOT NULL,
`createtime` datetime NOT NULL,
`lastupdatetime` datetime NOT NULL,
PRIMARY KEY (`id`) USING BTREE
) ENGINE=InnoDB AUTO_INCREMENT=3 DEFAULT CHARSET=latin1;
INSERT INTO `test`.`user`(`name`, `age`, `createtime`, `lastupdatetime`) VALUES ('jack', 18, '2022-03-19 10:27:12', '2022-03-19 10:27:12');
INSERT INTO `test`.`user`( `name`, `age`, `createtime`, `lastupdatetime`) VALUES ( 'William', 18, '2022-03-19 10:27:12', '2022-03-19 10:27:12');
二、安装logstash
- 1.拉取镜像
docker pull logstash:7.17.0
- 2.启动
docker run -d -p 5044:5044 -p 9600:9600 -it --name logstash logstash:7.17.0
- 3.修改配置文件
docker exec -it -u root logstash bash
apt-get update
apt-get install vim
修改logstash.yml
vim config/logstash.yml
http.host: "0.0.0.0"
xpack.monitoring.elasticsearch.url: http://192.168.0.103:9200
修改log4j2.properties
vim config/log4j2.properties
在末尾增加如下两项
logger.elasticsearchoutput.name = logstash.outputs.elasticsearch
logger.elasticsearchoutput.level = debug
vim config/pipelines.yml
相关值修改如下
- pipeline.id: logstash
path.config: "/usr/share/logstash/config/*.conf"
pipeline.workers: 3
修改mysql.conf，没有的话会创建
vim config/mysql.conf
将如下内容粘贴进去
input {
stdin {
}
jdbc {
jdbc_connection_string => "jdbc:mysql://192.168.0.103:3306/test"
jdbc_user => "canal"
jdbc_password => "canal"
jdbc_driver_library => "/usr/share/logstash/config/mysql-connector-java-5.1.27-bin.jar" # 这个jar文件需要下载
jdbc_driver_class => "com.mysql.jdbc.Driver"
statement => "SELECT * FROM user"
type => "project"
}
}
filter {
json {
source => "message"
remove_field => ["message"]
}
}
output {
elasticsearch {
hosts => "http://192.168.0.103:9200"
index => "project"
document_id => "%{id}"
}
stdout {
codec => json_lines
}
}
mysql-connector-java下载
apt-get install wget
wget https://downloads.mysql.com/archives/get/p/3/file/mysql-connector-java-5.1.27.tar.gz
tar zxvf mysql-connector-java-5.1.27.tar.gz
将解压后的.jar文件放到config目录下即可。
重启logstash。
- 4.执行加载配置项
docker exec -it logstash bash
bin/logstash -f config/mysql.conf
kibana
注：Logstash could not be started because there is already another instance usining...如果报这个错误，就进入logstash终端，找到data目录，ls -all 展示所有文件目录，把里面的.lock文件删除即可。
- 5.测试
查询
命中结果
其他文章请看：
docker安装kibana
docker安装elasticsearch
