1. 介绍
canal 是阿里巴巴开源的一个项目,主要用途是基于 MySQL 数据库 binlog 日志解析,提供增量数据订阅和消费。
基于日志增量订阅和消费的业务包括:
- 数据库镜像
- 数据库实时备份
- 索引构建和实时维护(拆分异构索引、倒排索引等)
- 业务 cache 刷新
- 带业务逻辑的增量数据处理
我这边主要在两个场景下使用:
一个是将变更数据实时同步到 Elasticsearch 和 Redis。
这里先说一下我目前的做法,一方面是全量数据定时同步,由于数据量比较大,同步时间比较长,所以数据也就不够实时。第二个方面是针对单条数据的变更,部分更新 Elasticsearch 和 Redis 的逻辑都是直接写在了业务代码中,耦合比较严重。
拆出来之后就可以实现实时增量更新,而且还可以解耦,收益还是很大的。
第二个是保存重点关注数据的历史变更。
这个目前用在了「资产管理」模块,通过记录 IP 资产的创建,变更以及删除,实现 IP 生命周期管理,方便历史信息回溯。
2. MySQL 配置
修改 MySQL 配置文件 my.cnf,开启 binlog 写入功能,并配置模式为 ROW。
log-bin=mysql-bin # 开启 binlog
binlog-format=ROW # 选择 ROW 模式
server_id=1 # 配置 MySQL replaction 需要定义,不要和 canal 的 slaveId 重复
重启数据库,查看配置是否生效。
mysql> show variables like 'binlog_format';
+---------------+-------+
| Variable_name | Value |
+---------------+-------+
| binlog_format | ROW |
+---------------+-------+
1 row in set (0.19 sec)
mysql>
mysql> show variables like 'log_bin';
+---------------+-------+
| Variable_name | Value |
+---------------+-------+
| log_bin | ON |
+---------------+-------+
1 row in set (0.00 sec)
mysql>
mysql> show master status;
+------------------+----------+--------------+------------------+-------------------+
| File | Position | Binlog_Do_DB | Binlog_Ignore_DB | Executed_Gtid_Set |
+------------------+----------+--------------+------------------+-------------------+
| mysql-bin.000003 | 4230 | | | |
+------------------+----------+--------------+------------------+-------------------+
1 row in set (0.00 sec)
然后创建用户,并授权。
mysql> CREATE USER canal IDENTIFIED BY 'canal';
mysql> GRANT SELECT, REPLICATION SLAVE, REPLICATION CLIENT ON *.* TO 'canal'@'%%';
mysql> FLUSH PRIVILEGES;
mysql> show grants for 'canal'@'%%';
+----------------------------------------------------------------------------+
| Grants for canal@%% |
+----------------------------------------------------------------------------+
| GRANT SELECT, REPLICATION SLAVE, REPLICATION CLIENT ON *.* TO `canal`@`%%` |
+----------------------------------------------------------------------------+
1 row in set (0.00 sec)
3. Canal 服务端
拉取镜像:
docker pull canal/canal-server:v1.1.4
然后用官方提供的 shell 脚本直接启动:
sh run.sh -e canal.auto.scan=false -e canal.destinations=test -e canal.instance.master.address=127.0.0.1:3306 -e canal.instance.dbUsername=root -e canal.instance.dbPassword=107428a08d6143db -e canal.instance.connectionCharset=UTF-8 -e canal.instance.tsdb.enable=true -e canal.instance.gtidon=false
但每次都这样启动还是有点麻烦,可以写一个 docker-compose. 文件,如下:
version: '3'
services:
canal-server:
image: canal/canal-server:v1.1.4
container_name: canal-server
restart: unless-stopped
network_mode: host
ports:
- 11111:11111
environment:
- canal.auto.scan=false
- canal.instance.master.address=127.0.0.1:3306
- canal.instance.dbUsername=root
- canal.instance.dbPassword=107428a08d6143db
- canal.instance.filter.regex=.*\\..*
- canal.destinations=test
- canal.instance.connectionCharset=UTF-8
- canal.instance.tsdb.enable=true
volumes:
- /root/canal/test/log/:/home/admin/canal-server/logs/
启动服务:
# docker-compose up
Recreating canal-server ... done
Attaching to canal-server
canal-server | DOCKER_DEPLOY_TYPE=VM
canal-server | ==> INIT /alidata/init/02init-sshd.sh
canal-server | ==> EXIT CODE: 0
canal-server | ==> INIT /alidata/init/fix-hosts.py
canal-server | ==> EXIT CODE: 0
canal-server | ==> INIT DEFAULT
canal-server | Generating SSH1 RSA host key: [ OK ]
canal-server | Starting sshd: [ OK ]
canal-server | Starting crond: [ OK ]
canal-server | ==> INIT DONE
canal-server | ==> RUN /home/admin/app.sh
canal-server | ==> START ...
canal-server | start canal ...
canal-server | start canal successful
canal-server | ==> START SUCCESSFUL ...
4. 客户端
构建canal php客户端
$ composer require xingwenge/canal_php
or
$ git clone https://github.com/xingwenge/canal-php.git
$ cd canal-php
$ composer update
直接 Copy 官方提供的客户端代码:
<?php
require_once __DIR__ . "/vendor/autoload.php";
try {
$client = \xingwenge\canal_php\CanalConnectorFactory::createClient(\xingwenge\canal_php\CanalClient::TYPE_SOCKET_CLUE);
# $client = CanalConnectorFactory::createClient(CanalClient::TYPE_SWOOLE);
$client->connect("106.15.195.75", 11111);
$client->checkValid();
$client->subscribe("1001", "test", ".*\\..*");
# $client->subscribe("1001", "example", "db_name.tb_name"); # 设置过滤
while (true) {
$message = $client->get(100);
if ($entries = $message->getEntries()) {
foreach ($entries as $entry) {
\xingwenge\canal_php\Fmt::println($entry);
}
}
sleep(1);
}
$client->disConnect();
} catch (\Exception $e) {
echo $e->getMessage(), PHP_EOL;
}
/Applications/MxSrvs/bin/php/bin/php /Users/ligan/Documents/PHPProject/Self/ImoocDesignPattern/canal-php/index.php
================> binlog[mysql-bin.000010 : 2405],name[sql_106_15_195_7,ims_account], eventType: 2
-------> before
acid : 2 update= false
uniacid : 2 update= false
hash : s3CllCcw update= false
type : 4 update= false
isconnect : 1 update= false
isdeleted : 0 update= false
endtime : 0 update= false
send_account_expire_status : 0 update= false
send_api_expire_status : 1 update= false
-------> after
acid : 2 update= false
uniacid : 2 update= false
hash : s3CllCcw update= false
type : 4 update= false
isconnect : 1 update= false
isdeleted : 0 update= false
endtime : 0 update= false
send_account_expire_status : 0 update= false
send_api_expire_status : 0 update= true
5. 功能验证
首先在 MySQL 里边创建一张测试表,然后再增删改几条测试数据:
mysql> CREATE TABLE `role` ( `id` int unsigned NOT NULL AUTO_INCREMENT, `role_name` varchar(255) DEFAULT NULL, PRIMARY KEY (`id`) ) ENGINE=InnoDB AUTO_INCREMENT=2 DEFAULT CHARSET=utf8mb4;
mysql> insert into role (id, role_name) values (10, 'admin');
Query OK, 1 row affected (0.01 sec)
mysql> update role set role_name='hh' where id = 10;
Query OK, 1 row affected (0.01 sec)
Rows matched: 1 Changed: 1 Warnings: 0
mysql> delete from role where id = 10;
Query OK, 1 row affected (0.01 sec)
客户端打印输出:
================> binlog[mysql-bin.000010 : 4269],name[sql_106_15_195_7,role], eventType: 4
CREATE TABLE `role` ( `id` int unsigned NOT NULL AUTO_INCREMENT, `role_name` varchar(255)
DEFAULT NULL, PRIMARY KEY (`id`) ) ENGINE=InnoDB AUTO_INCREMENT=2 DEFAULT CHARSET=utf8mb4
================> binlog[mysql-bin.000010 : 4764],name[sql_106_15_195_7,role], eventType: 1
id : 10 update= true
role_name : admin update= true
================> binlog[mysql-bin.000010 : 5053],name[sql_106_15_195_7,role], eventType: 2
-------> before
id : 10 update= false
role_name : admin update= false
-------> after
id : 10 update= false
role_name : hh update= true
================> binlog[mysql-bin.000010 : 5352],name[sql_106_15_195_7,role], eventType: 3
id : 10 update= false
role_name : hh update= false
变更一条数据,输出内容分三部分,分别是:TRANSACTIONBEGIN,ROWDATA 和 TRANSACTIONEND。然后我们比较关注的内容都在 ROWDATA 中,解析出来之后就是我们需要的,包括数据库名,表名和变更内容。
关于这里Fmt返回的打印值有个eventType可以进行判断
类型 | 值 | 说明 |
---|---|---|
eventType | 1 | 新增(insert) |
eventType | 2 | 更新(update) |
eventType | 3 | 删除(delete) |
canal 服务端启动之后,在 /home/admin/canal-server/logs/test
目录下会生成两个日志文件,分别是:meta.log 和 test.log,可以查看服务是不是正常,有没有报错信息。其中 test 是启动 Docker 时 canal.destinations
设置的名称。
meta.log
[root@172 test]# cat meta.log
2021-07-24 13:49:13.723 - clientId:1001 cursor:[mysql-bin.000010,1284,1627096405000,1,] address[/127.0.0.1:3306]
2021-07-24 13:55:22.723 - clientId:1001 cursor:[mysql-bin.000010,1689,1627106121000,1,] address[localhost/127.0.0.1:3306]
2021-07-24 13:55:36.723 - clientId:1001 cursor:[mysql-bin.000010,2094,1627106136000,1,] address[localhost/127.0.0.1:3306]
2021-07-24 13:55:40.723 - clientId:1001 cursor:[mysql-bin.000010,2499,1627106139000,1,] address[localhost/127.0.0.1:3306]
2021-07-24 13:57:10.723 - clientId:1001 cursor:[mysql-bin.000010,2904,1627106229000,1,] address[localhost/127.0.0.1:3306]
2021-07-24 13:57:34.723 - clientId:1001 cursor:[mysql-bin.000010,3309,1627106253000,1,] address[localhost/127.0.0.1:3306]
2021-07-24 13:57:45.723 - clientId:1001 cursor:[mysql-bin.000010,3714,1627106264000,1,] address[localhost/127.0.0.1:3306]
2021-07-24 13:57:59.723 - clientId:1001 cursor:[mysql-bin.000010,4119,1627106279000,1,] address[localhost/127.0.0.1:3306]
2021-07-24 13:58:36.723 - clientId:1001 cursor:[mysql-bin.000010,4269,1627106315000,1,] address[localhost/127.0.0.1:3306]
2021-07-24 13:59:07.723 - clientId:1001 cursor:[mysql-bin.000010,5396,1627106346000,1,] address[localhost/127.0.0.1:3306]
2021-07-24 14:05:59.723 - clientId:1001 cursor:[mysql-bin.000010,5685,1627106758000,1,] address[localhost/127.0.0.1:3306]
2021-07-24 14:07:42.723 - clientId:1001 cursor:[mysql-bin.000010,5984,1627106862000,1,] address[localhost/127.0.0.1:3306]
2021-07-24 14:08:35.723 - clientId:1001 cursor:[mysql-bin.000010,6281,1627106913000,1,] address[localhost/127.0.0.1:3306]
2021-07-24 14:13:17.723 - clientId:1001 cursor:[mysql-bin.000010,6580,1627107196000,1,] address[localhost/127.0.0.1:3306]
2021-07-24 14:14:59.723 - clientId:1001 cursor:[mysql-bin.000010,6879,1627107298000,1,] address[localhost/127.0.0.1:3306]
2021-07-24 14:16:39.723 - clientId:1001 cursor:[mysql-bin.000010,7177,1627107398000,1,] address[localhost/127.0.0.1:3306]
test.log
root@172 test]# cat test.log
2021-07-24 13:36:53.783 [main] INFO c.a.o.c.i.spring.support.PropertyPlaceholderConfigurer - Loading properties file from class path resource [canal.properties]
2021-07-24 13:36:53.788 [main] INFO c.a.o.c.i.spring.support.PropertyPlaceholderConfigurer - Loading properties file from class path resource [test/instance.properties]
2021-07-24 13:36:54.047 [main] WARN o.s.beans.GenericTypeAwarePropertyDescriptor - Invalid JavaBean property 'connectionCharset' being accessed! Ambiguous write methods found next to actually used [public void com.alibaba.otter.canal.parse.inbound.mysql.AbstractMysqlEventParser.setConnectionCharset(java.lang.String)]: [public void com.alibaba.otter.canal.parse.inbound.mysql.AbstractMysqlEventParser.setConnectionCharset(java.nio.charset.Charset)]
2021-07-24 13:36:54.098 [main] INFO c.a.o.c.i.spring.support.PropertyPlaceholderConfigurer - Loading properties file from class path resource [canal.properties]
2021-07-24 13:36:54.099 [main] INFO c.a.o.c.i.spring.support.PropertyPlaceholderConfigurer - Loading properties file from class path resource [test/instance.properties]
2021-07-24 13:36:54.720 [main] INFO c.a.otter.canal.instance.spring.CanalInstanceWithSpring - start CannalInstance for 1-test
2021-07-24 13:36:54.730 [main] WARN c.a.o.canal.parse.inbound.mysql.dbsync.LogEventConvert - --> init table filter : ^.*\..*$
2021-07-24 13:36:54.730 [main] WARN c.a.o.canal.parse.inbound.mysql.dbsync.LogEventConvert - --> init table black filter :
2021-07-24 13:36:54.738 [main] INFO c.a.otter.canal.instance.core.AbstractCanalInstance - start successful....
2021-07-24 13:36:54.840 [destination = test , address = /127.0.0.1:3306 , EventParser] WARN c.a.o.c.p.inbound.mysql.rds.RdsBinlogEventParserProxy - ---> begin to find start position, it will be long time for reset or first position
2021-07-24 13:36:54.840 [destination = test , address = /127.0.0.1:3306 , EventParser] WARN c.a.o.c.p.inbound.mysql.rds.RdsBinlogEventParserProxy - prepare to find start position just show master status
2021-07-24 13:36:56.519 [destination = test , address = /127.0.0.1:3306 , EventParser] WARN c.a.o.c.p.inbound.mysql.rds.RdsBinlogEventParserProxy - ---> find start position successfully, EntryPosition[included=false,journalName=mysql-bin.000010,position=1113,serverId=1,gtid=,timestamp=1627096405000] cost : 1657ms , the next step is binlog dump
2021-07-24 13:49:12.747 [New I/O server worker #1-2] INFO c.a.otter.canal.instance.core.AbstractCanalInstance - subscribe filter change to .*\..*
2021-07-24 13:49:12.747 [New I/O server worker #1-2] WARN c.a.o.canal.parse.inbound.mysql.dbsync.LogEventConvert - --> init table filter : ^.*\..*$
2021-07-24 13:53:00.970 [New I/O server worker #1-4] INFO c.a.otter.canal.instance.core.AbstractCanalInstance - subscribe filter change to .*\..*
2021-07-24 13:53:00.970 [New I/O server worker #1-4] WARN c.a.o.canal.parse.inbound.mysql.dbsync.LogEventConvert - --> init table filter : ^.*\..*$
2021-07-24 13:53:34.837 [New I/O server worker #1-2] INFO c.a.otter.canal.instance.core.AbstractCanalInstance - subscribe filter change to .*\..*
2021-07-24 13:53:34.837 [New I/O server worker #1-2] WARN c.a.o.canal.parse.inbound.mysql.dbsync.LogEventConvert - --> init table filter : ^.*\..*$
2021-07-24 13:53:39.709 [New I/O server worker #1-3] INFO c.a.otter.canal.instance.core.AbstractCanalInstance - subscribe filter change to .*\..*
2021-07-24 13:53:39.709 [New I/O server worker #1-3] WARN c.a.o.canal.parse.inbound.mysql.dbsync.LogEventConvert - --> init table filter : ^.*\..*$
2021-07-24 13:53:41.849 [New I/O server worker #1-4] INFO c.a.otter.canal.instance.core.AbstractCanalInstance - subscribe filter change to .*\..*
2021-07-24 13:53:41.849 [New I/O server worker #1-4] WARN c.a.o.canal.parse.inbound.mysql.dbsync.LogEventConvert - --> init table filter : ^.*\..*$
2021-07-24 13:54:00.291 [New I/O server worker #1-1] INFO c.a.otter.canal.instance.core.AbstractCanalInstance - subscribe filter change to .*\..*
2021-07-24 13:54:00.291 [New I/O server worker #1-1] WARN c.a.o.canal.parse.inbound.mysql.dbsync.LogEventConvert - --> init table filter : ^.*\..*$
2021-07-24 13:54:19.664 [New I/O server worker #1-3] INFO c.a.otter.canal.instance.core.AbstractCanalInstance - subscribe filter change to .*\..*
2021-07-24 13:54:19.664 [New I/O server worker #1-3] WARN c.a.o.canal.parse.inbound.mysql.dbsync.LogEventConvert - --> init table filter : ^.*\..*$
2021-07-24 13:54:25.136 [New I/O server worker #1-4] INFO c.a.otter.canal.instance.core.AbstractCanalInstance - subscribe filter change to .*\..*
2021-07-24 13:54:25.137 [New I/O server worker #1-4] WARN c.a.o.canal.parse.inbound.mysql.dbsync.LogEventConvert - --> init table filter : ^.*\..*$
2021-07-24 13:54:45.042 [New I/O server worker #1-1] INFO c.a.otter.canal.instance.core.AbstractCanalInstance - subscribe filter change to .*\..*
2021-07-24 13:54:45.042 [New I/O server worker #1-1] WARN c.a.o.canal.parse.inbound.mysql.dbsync.LogEventConvert - --> init table filter : ^.*\..*$
2021-07-24 13:55:31.158 [New I/O server worker #1-2] INFO c.a.otter.canal.instance.core.AbstractCanalInstance - subscribe filter change to .*\..*
2021-07-24 13:55:31.158 [New I/O server worker #1-2] WARN c.a.o.canal.parse.inbound.mysql.dbsync.LogEventConvert - --> init table filter : ^.*\..*$
2021-07-24 13:57:04.677 [New I/O server worker #1-3] INFO c.a.otter.canal.instance.core.AbstractCanalInstance - subscribe filter change to .*\..*
2021-07-24 13:57:04.678 [New I/O server worker #1-3] WARN c.a.o.canal.parse.inbound.mysql.dbsync.LogEventConvert - --> init table filter : ^.*\..*$
2021-07-24 13:57:27.690 [New I/O server worker #1-4] INFO c.a.otter.canal.instance.core.AbstractCanalInstance - subscribe filter change to .*\..*
2021-07-24 13:57:27.690 [New I/O server worker #1-4] WARN c.a.o.canal.parse.inbound.mysql.dbsync.LogEventConvert - --> init table filter : ^.*\..*$
2021-07-24 13:57:41.541 [New I/O server worker #1-1] INFO c.a.otter.canal.instance.core.AbstractCanalInstance - subscribe filter change to .*\..*
2021-07-24 13:57:41.541 [New I/O server worker #1-1] WARN c.a.o.canal.parse.inbound.mysql.dbsync.LogEventConvert - --> init table filter : ^.*\..*$
2021-07-24 13:57:56.110 [New I/O server worker #1-2] INFO c.a.otter.canal.instance.core.AbstractCanalInstance - subscribe filter change to .*\..*
2021-07-24 13:57:56.110 [New I/O server worker #1-2] WARN c.a.o.canal.parse.inbound.mysql.dbsync.LogEventConvert - --> init table filter : ^.*\..*$
2021-07-24 14:05:48.775 [New I/O server worker #1-3] INFO c.a.otter.canal.instance.core.AbstractCanalInstance - subscribe filter change to .*\..*
2021-07-24 14:05:48.776 [New I/O server worker #1-3] WARN c.a.o.canal.parse.inbound.mysql.dbsync.LogEventConvert - --> init table filter : ^.*\..*$
2021-07-24 14:07:23.647 [New I/O server worker #1-4] INFO c.a.otter.canal.instance.core.AbstractCanalInstance - subscribe filter change to .*\..*
2021-07-24 14:07:23.647 [New I/O server worker #1-4] WARN c.a.o.canal.parse.inbound.mysql.dbsync.LogEventConvert - --> init table filter : ^.*\..*$
2021-07-24 14:07:30.727 [New I/O server worker #1-1] INFO c.a.otter.canal.instance.core.AbstractCanalInstance - subscribe filter change to .*\..*
2021-07-24 14:07:30.728 [New I/O server worker #1-1] WARN c.a.o.canal.parse.inbound.mysql.dbsync.LogEventConvert - --> init table filter : ^.*\..*$
2021-07-24 14:08:13.510 [New I/O server worker #1-2] INFO c.a.otter.canal.instance.core.AbstractCanalInstance - subscribe filter change to .*\..*
2021-07-24 14:08:13.511 [New I/O server worker #1-2] WARN c.a.o.canal.parse.inbound.mysql.dbsync.LogEventConvert - --> init table filter : ^.*\..*$
2021-07-24 14:13:10.369 [New I/O server worker #1-3] INFO c.a.otter.canal.instance.core.AbstractCanalInstance - subscribe filter change to .*\..*
2021-07-24 14:13:10.369 [New I/O server worker #1-3] WARN c.a.o.canal.parse.inbound.mysql.dbsync.LogEventConvert - --> init table filter : ^.*\..*$
2021-07-24 14:14:47.250 [New I/O server worker #1-4] INFO c.a.otter.canal.instance.core.AbstractCanalInstance - subscribe filter change to .*\..*
2021-07-24 14:14:47.251 [New I/O server worker #1-4] WARN c.a.o.canal.parse.inbound.mysql.dbsync.LogEventConvert - --> init table filter : ^.*\..*$
2021-07-24 14:16:16.396 [New I/O server worker #1-1] INFO c.a.otter.canal.instance.core.AbstractCanalInstance - subscribe filter change to .*\..*
2021-07-24 14:16:16.396 [New I/O server worker #1-1] WARN c.a.o.canal.parse.inbound.mysql.dbsync.LogEventConvert - --> init table filter : ^.*\..*$
2021-07-24 14:16:30.836 [New I/O server worker #1-2] INFO c.a.otter.canal.instance.core.AbstractCanalInstance - subscribe filter change to .*\..*
2021-07-24 14:16:30.836 [New I/O server worker #1-2] WARN c.a.o.canal.parse.inbound.mysql.dbsync.LogEventConvert - --> init table filter : ^.*\..*$
6. 参考文档
https://www.jianshu.com/p/9759cd0f0614
https://github.com/alibaba/canal
https://github.com/xingwenge/canal-php