创建用户
groupadd hadoop
useradd -g hadoop hadoop
解压编译
wget https://cdn.gethue.com/downloads/hue-4.8.0.tgz PREFIX=/usr/share make install
集成Hive Conf
在/opt/hive/conf目录添加hive-site.xml文件
<?xml version="1.0"?> <?xml-stylesheet type="text/xsl" href="configuration.xsl"?> <configuration> <property> <name>hive.metastore.local</name> <value>false</value> </property> <property> <name>hive.metastore.uris</name> <value>thrift://172.31.6.20:9083</value> </property> </configuration>
Flink环境集成依赖的jar包
依赖Jar包列表
flink-connector-hive_2.11-1.11.3.jar
flink-sql-connector-kafka_2.11-1.11.2.jar
flink-parquet_2.11-1.11.2.jar
libfb303-0.9.3.jar
hive-exec-3.1.2.jar
重新编译hive-exec，解决guava版本冲突问题
下载hive源码，修改ql model下的pom文件，将guava版本由19.0改为20.0，重新编译
Flink环境集成Hadoop
进入Flink镜像工作目录（172.31.6.12）
cd /data/dockerfiles/flink-1.11
官网下载Hadoop安装包
wget https://archive.apache.org/dist/hadoop/common/hadoop-3.1.3/hadoop-3.1.3.tar.gz
解压安装包，删除无用文件
tar zxvf hadoop-3.1.3.tar.gz
cd hadoop-3.1.3
rm -rf bin/ etc/ include/ lib/ libexec/ LICENSE.txt NOTICE.txt README.txt sbin/
rm -rf ./share/hadoop/hdfs/lib/netty-all-4.0.52.Final.jar
rm -rf ./share/hadoop/hdfs/lib/netty-3.10.5.Final.jar
重新打包成 hadoop-3.1.3.tar.gz
tar zcvf hadoop-3.1.3.tar.gz hadoop-3.1.3
修改Dockerfile，添加如下内容
ADD hadoop-3.1.3.tar.gz /opt/flink/hadoop
打包上传镜像
docker build -t d-hub.jcinfo.com/library/flink:1.11.2-scala_2.11-java8-v1.4.3 .
docker push d-hub.jcinfo.com/library/flink:1.11.2-scala_2.11-java8-v1.4.3
Rancher中配置Hadoop环境变量
HADOOP_CLASSPATH=/opt/flink/hadoop/hadoop-3.1.3/etc/hadoop:/opt/flink/hadoop/hadoop-3.1.3/share/hadoop/common/lib/*:/opt/flink/hadoop/hadoop-3.1.3/share/hadoop/common/*:/opt/flink/hadoop/hadoop-3.1.3/share/hadoop/hdfs:/opt/flink/hadoop/hadoop-3.1.3/share/hadoop/hdfs/lib/*:/opt/flink/hadoop/hadoop-3.1.3/share/hadoop/hdfs/*:/opt/flink/hadoop/hadoop-3.1.3/share/hadoop/mapreduce/lib/*:/opt/flink/hadoop/hadoop-3.1.3/share/hadoop/mapreduce/*:/opt/flink/hadoop/hadoop-3.1.3/share/hadoop/yarn:/opt/flink/hadoop/hadoop-3.1.3/share/hadoop/yarn/lib/*:/opt/flink/hadoop/hadoop-3.1.3/share/hadoop/yarn/*
配置database
[[database]]
engine=mysql
host=192.168.56.131
port=3306
user=root
password=li1383414
name=hue
初始化数据库
build/env/bin/hue syncdb
build/env/bin/hue migrate
连接Mysql&TIDB
[[mysql]]]
name = MySQL
interface=sqlalchemy
options='{"url": "mysql://root@tidb.jcinfo.com:3306/test"}'
连接Hive
[[[hive]]]
name=Hive
interface=hiveserver2
[beeswax]
hive_server_host=172.31.6.20
hive_server_port=10000
hive_conf_dir=/opt/hive/conf
连接Flink SQL
安装flink-sql-gateway
解压安装
wget https://github.com/ververica/flink-sql-gateway/releases/download/flink-1.11.1/flink-sql-gateway-0.2-SNAPSHOT-bin.zip
unzip flink-sql-gateway-0.2-SNAPSHOT-bin.zip -d ../flink/
配置
server:
bind-address: 0.0.0.0 # optional: The address that the gateway binds itself (127.0.0.1 by default)
address: 0.0.0.0 # optional: The address that should be used by clients to connect to the gateway (127.0.0.1 by default)
catalogs: #[] # empty list
# A typical catalog definition looks like:
- name: myhive
type: hive
hive-conf-dir: /opt/hive/conf/
default-database: dw
启动
./bin/sql-gateway.sh
注意：与flink安装在同一台机器
添加配置
[[[flink]]]
name=Flink
interface=flink
options='{"url": "http://127.0.0.1:8083"}'
启动HUE
build/env/bin/supervisor
参考资料
https://github.com/ververica/flink-sql-gateway