基于Docker容器搭建hadoop完全分布式集群环境

发布时间 2023-12-10 10:09:29作者: 框框A

简介

  • 物理机:windows10
  • 宿主机:Centos7虚拟机,需要安装Docker服务
  • hadoop集群节点:3个centos7的容器,hadoop1、hadoop2、hadoop3
  • 组件:
    • 容器镜像:Centos7
    • Docker CE 24.0.7
    • JDK1.8.0_181
    • Hadoop3.1.3

1.新建虚拟机

安装CentOS7

2.安装Docker

2.1安装docker服务

yum -y install docker-ce

2.2 开启docker服务

systemctl start docker
systemctl status docker # 查看服务状态
docker version # 查看版本
Client: Docker Engine - Community
Version: 24.0.7
Server: Docker Engine - Community
Engine:
Version: 24.0.7

3. 制作镜像

3.1 拉取镜像

docker pull centos:7
docker images #查看镜像
centos 7 eeb6ee3f44bd 2 years ago 204MB

3.2 制作镜像

(1)制作Dockerfile文件
vi Dockerfile

Dockerfile
FROM centos:7
MAINTAINER zyz

RUN cd /etc/yum.repos.d/
RUN sed -i 's/mirrorlist/#mirrorlist/g' /etc/yum.repos.d/CentOS-*
RUN sed -i 's|#baseurl=http://mirror.centos.org|baseurl=http://mirrors.aliyun.com|g' /etc/yum.repos.d/CentOS-*
RUN yum makecache
RUN yum update -y

RUN yum install -y openssh-server sudo
RUN sed -i 's/UsePAM yes/UsePAM no/g' /etc/ssh/sshd_config
RUN yum install -y openssh-clients

RUN echo "root:root" | chpasswd
RUN echo "root   ALL=(ALL)       ALL" >> /etc/sudoers
RUN ssh-keygen -t dsa -f /etc/ssh/ssh_host_dsa_key
RUN ssh-keygen -t rsa -f /etc/ssh/ssh_host_rsa_key

RUN mkdir /var/run/sshd
EXPOSE 22
CMD ["/usr/sbin/sshd", "-D"]

FROM:基于什么镜像来制作自己的镜像
MAINTAINER:表示该镜像的作者(维护者)
第2段:配置yum,包括修改镜像源、提速、更新
第3段:安装ssh服务和ssh客户端
第4段:生成ssh密钥
第5段:开启ssh服务,暴露SSH的默认端口22
(2)生成镜像
docker build -t centos7-ssh .
"."表示当前目录,即Dockerfile所在的位置
docker images
centos7-ssh latest d39095d60198 17 hours ago 1.42GB

4. 创建容器

(1)创建网桥
docker network create hadoop
docker network ls # 查看网桥
371545b29a8d hadoop bridge local
(2)创建容器

docker run -itd --network hadoop --name hadoop1 -p 50070:50070 -p 8088:8088 centos7-ssh
docker run -itd --network hadoop --name hadoop2 centos7-ssh
docker run -itd --network hadoop --name hadoop3 centos7-ssh

i:立即运行,t:终端,d:后台运行
*--network:容器要加入的网桥,--name:指定容器名称,-p:表示映射端口,centos7-ssh表示创建容器的镜像 *
(3)查看容器
docker ps #查看正在运行的容器


点击查看容器信息
CONTAINER ID   IMAGE         COMMAND               CREATED        STATUS        PORTS                                                                                              NAMES
71c3b5fa9846   centos7-ssh   "/usr/sbin/sshd -D"   12 hours ago   Up 10 hours   22/tcp                                                                                             hadoop3
a16e70f1373e   centos7-ssh   "/usr/sbin/sshd -D"   17 hours ago   Up 10 hours   22/tcp                                                                                             hadoop2
bac46cc68c73   centos7-ssh   "/usr/sbin/sshd -D"   17 hours ago   Up 10 hours   0.0.0.0:8088->8088/tcp, :::8088->8088/tcp, 22/tcp, 0.0.0.0:50070->50070/tcp, :::50070->50070/tcp   hadoop1

(4)查看网桥
docker network inspect hadoop

点击查看网桥信息
"Containers": {
            "71c3b5fa98463d995affb206496c04ee6f2fdaedda15240dc490f79f8cad23f9": {
                "Name": "hadoop3",
                "EndpointID": "2710a60e5ef5ea4e590ed2faff7a9db5eca2e5ea960867f05cc818a665a3c4bf",
                "MacAddress": "02:42:ac:13:00:04",
                "IPv4Address": "172.19.0.4/16",
                "IPv6Address": ""
            },
            "a16e70f1373ef80a53bee0fa0af01b861137ada82787bc909d708fe8774a6651": {
                "Name": "hadoop2",
                "EndpointID": "eee50c5aae298154e0811b6dfdc60fc46e9bf2d4a097b67b73582150748fd890",
                "MacAddress": "02:42:ac:13:00:03",
                "IPv4Address": "172.19.0.3/16",
                "IPv6Address": ""
            },
            "bac46cc68c739292fa36b4a1b92ed9f0347f8d622a7129922b6ab7d009b618f9": {
                "Name": "hadoop1",
                "EndpointID": "209ace260c8f2161e0aa805f3b034c66b7892fb8fa9a2a1f7d903dc9100aff3f",
                "MacAddress": "02:42:ac:13:00:02",
                "IPv4Address": "172.19.0.2/16",
                "IPv6Address": ""
            }
        }

5.给容器安装软件