526互联

基于docker容器，搭建hadoop+spark+hive+hbase+Zookeeper Scala集群

发布时间 2023-10-24 00:19:08作者: 红酒人生

1.安装Docker和Docker Compose

2.下载镜像

docker pull bde2020/hadoop-base:2.0.0-hadoop3.2.1-java8
docker pull bde2020/hadoop-namenode:2.0.0-hadoop3.2.1-java8
docker pull bde2020/hadoop-datanode:2.0.0-hadoop3.2.1-java8
docker pull bde2020/hadoop-resourcemanager:2.0.0-hadoop3.2.1-java8
docker pull bde2020/hadoop-nodemanager:2.0.0-hadoop3.2.1-java8
docker pull bde2020/hadoop-historyserver:2.0.0-hadoop3.2.1-java8
docker pull bde2020/hive:2.3.7-postgresql-metastore
docker pull bde2020/spark-base:2.4.5-hadoop2.7
docker pull bde2020/spark-master:2.4.5-hadoop2.7
docker pull bde2020/spark-worker:2.4.5-hadoop2.7
docker pull bde2020/hbase:2.2.4-hadoop3.2.1-java8

（1）本机系统设置

电脑设置虚拟缓存（设置为自动管理）

 虚拟机设置内存和CPU

内存设置为8G（或以上）

CPU稍微设置高一点（三个虚拟化能开就开）

 虚拟机系统配置阿里源

wget -O /etc/yum.repos.d/CentOS-Base.repo https://mirrors.aliyun.com/repo/Centos-7.repo

虚拟机更新包

yum update

（2）安装Docker

移除以前docker相关包

sudo yum remove docker \
                  docker-client \
                  docker-client-latest \
                  docker-common \
                  docker-latest \
                  docker-latest-logrotate \
                  docker-logrotate \
                  docker-engine

 

（3）安装Hadoop

 拉取hadoop镜像

docker pull registry.cn-shenzhen.aliyuncs.com/jonil/hadoop:base

 

（4）运行容器