关于Docker如何搭建一个虚拟的Hadoop集群学习环境

1、简介：

鉴于最近比较火的hadoop+spark架构，需要一个比较适用的学习环境，要求资源不高，仅用于学习。而简单的集群有需要分布式或者单机模式启动各种应用会占用大量资源问题，所以选用docker搭建一套hdfs+spark的学习环境是最合适的。

需求资源：

• 一台安装好docker环境的linux系统机子

• docker已经配置映射国内镜像路径

• 安装docker-compose

2、安装步骤

• docker拉取镜像 脚本：(注意：拉取截图下所有镜像才行)

  docker pull uhopper/hadoop

image-20201029112455642.png

• 配置docker-compse.yml

  version: '3'
  services:
   namenode:
   image: uhopper/hadoop-namenode
   hostname: namenode
   container_name: namenode
   domainname: hadoop
   networks: 
   - hadoop
   ports:
   - '8020:8020'
   volumes:
   - /namenode:/data/hadoop/dfs/name
   environment:
   - CLUSTER_NAME=datanode1
   - CLUSTER_NAME=datanode2
  ​
   datanode1:
   image: uhopper/hadoop-datanode
   hostname: datanode1
   container_name: datanode1
   domainname: hadoop
   networks: 
   - hadoop
   volumes:
   - /datanode1:/data/hadoop/dfs/data
   environment:
   - CORE_CONF_fs_defaultFS=hdfs://namenode:8020
  
   datanode2:
   image: uhopper/hadoop-datanode
   hostname: datanode2
   container_name: datanode2
   domainname: hadoop
   networks: 
   - hadoop
   volumes:
   - /datanode2:/data/hadoop/dfs/data
   environment:
   - CORE_CONF_fs_defaultFS=hdfs://namenode:8020
  
   resourcemanager:
   image: uhopper/hadoop-resourcemanager
   hostname: resourcemanager
   container_name: resourcemanager
   domainname: hadoop
   networks: 
   - hadoop
   ports:
   - '8088:8088'
   environment:
   - CORE_CONF_fs_defaultFS=hdfs://namenode:8020
   - YARN_CONF_yarn_log___aggregation___enable=true
  
   nodemanager:
   image: uhopper/hadoop-nodemanager
   hostname: nodemanager
   container_name: nodemanager
   domainname: hadoop
   networks: 
   - hadoop
   environment:
   - CORE_CONF_fs_defaultFS=hdfs://namenode:8020
   - YARN_CONF_yarn_resourcemanager_hostname=resourcemanager
   - YARN_CONF_yarn_log___aggregation___enable=true
   - YARN_CONF_yarn_nodemanager_remote___app___log___dir=/app-logs
  
   spark:
   image: uhopper/hadoop-spark
   hostname: spark
   container_name: spark
   domainname: hadoop
   networks: 
   - hadoop
   environment:
   - CORE_CONF_fs_defaultFS=hdfs://namenode:8020
   - YARN_CONF_yarn_resourcemanager_hostname=resourcemanager
   command: tail -f /var/log/dmesg
  ​
  networks:
   hadoop:
   external:
   name: hadoop
  

• 创建docker桥接网络

  #创建虚拟网络，用于docker容器间通讯
  docker network create --driver=bridge hadoop
  #查看网络
  docker network ls
  

image-20201029113756463.png

• 启动

  #在前面已经创建好的docker-compose.yml同一目录下执行
  docker-compose up -d
  

  启动成功后：

  image-20201029114210224.png

• 访问你的hadoop环境

  http://yourhost:8088/cluster
  

  image-20201029114513649.png

如果出现上面画面恭喜你，已经成功了。
本文参考文章https://hub.docker.com/r/uhopper/hadoop/