一、开启mysql的二进制日志
vim /etc/my.cnf
log_bin=mysql-bin
binlog-format=ROW
server_id=1
二、创建索引
要求Mapping中定义的字段名称和类型与待同步数据保持一致
PUT es_test?include_type_name=true
{
"settings" : {
"index" : {
"number_of_shards" : "5",
"number_of_replicas" : "1"
}
},
"mappings" : {
"_doc" : {
"properties" : {
"count": {
"type": "text"
},
"id": {
"type": "integer"
},
"name": {
"type" : "text",
"analyzer": "ik_smart"
},
"color" : {
"type" : "text"
}
}
}
}
}
三、安装配置cannal
1、安装jdk1.8
2、下载配置canal.deployer-1.1.5
tar -xzvf canal.deployer-1.1.5-SNAPSHOT.tar.gz -C canal-server
vim canal-server/conf/example/instance.properties
canal.instance.mysql.slaveId=10
canal.instance.master.address=mysql数据库ip地址:3306
canal.instance.dbUsername=root
canal.instance.dbPassword=123456
3、启动cannal-server并查看日志
./bin/startup.sh
cat logs/canal/canal.log
4、下载配置canal.adapter-1.1.5
tar -xzvf canal.adapter-1.1.5-SNAPSHOT.tar.gz -C canal-adapter
①vim canal-adapter/conf/application.yml
canal.conf:
mode: tcp
retries: 0
srcDataSources:
defaultDS:
url:jdbc:mysql://mysql数据库ip地址:3306/elasticsearch?useUnicode=true&characterEncoding=utf-8&useSSL=false #如果不加上useSSL=false，会提示init datasource error
username:root
password:123456
canalAdapters:
- instance: example
groups:
- groupId: g1
outerAdapters:
- name: logger
- name: es7
hosts: 127.0.0.1:9200 #如果canal版本为1.1.6，ip前面要加上http://
properties:
mode: rest
# security.auth: test:123456
cluster.name: elasticsearch
②vim canal-adapter/conf/es7/mytest_user.yml
dataSourceKey: defaultDS
destination: example
groupId: g1
esMapping:
_index: es_test
_id: _id
sql: "select a.id as _id,a.id,a.count,a.name,a.color from es_test a"
etlCondition: "where a.c_time>={}"
commitBatch:3000
5、启动canal-adapter并查看日志
./bin/startup.sh
cat logs/adapter/adapter.log
四、验证增量数据同步
在mysql中新增、更改、删除数据
insert `elasticsearch`.`es_test`(`count`,`id`,`name`,`color`) values('11',2,'canal_test2','red');
五、查看ES中同步数据
GET /es_test/_search