Step 1(Pull Image from docker hub)
docker pull bajwalabs/apache-hadoop:pseudo.3.3.1
Step 2(Setup environment variables)
Create an environment variables file. e.g. env.list
Put below environment variables in this file. See the environment variables list at the bottom.
Step 3(Run Container)
docker run --env-file env.list --volume hadoop_volume:/myhadoop --name apache-hadoop-pseudo331 -t -p 9870:9870 -p 9864:9864 -p 8042:8042 -p 8088:8088 -p 8188:8188 bajwalabs/apache-hadoop:pseudo.3.3.1
Note:The above command will not remove Container when Stopped as --rm argument is not used.
Step 4(Stop Container)
docker stop apache-hadoop-pseudo331
Note:Use another console to run above command
Step 5(Start the stopped Container)
docker start apache-hadoop-pseudo331
Step 4(Launch a Bash terminal within a container
Once container is running, you may Launch a Bash terminal with below command.
docker exec -it apache-hadoop-pseudo331 bash
Environment Variables
CLUSTER_NAME=test
MULTIHOMED_NETWORK=0
CORE_CONF_fs_defaultFS=hdfs://localhost:9000
CORE_CONF_hadoop_http_staticuser_user=root
CORE_CONF_hadoop_proxyuser_hue_hosts=
CORE_CONF_hadoop_proxyuser_hue_groups=
CORE_CONF_io_compression_codecs=org.apache.hadoop.io.compress.SnappyCodec
HDFS_CONF_dfs_webhdfs_enabled=true
HDFS_CONF_dfs_permissions_enabled=false
HDFS_CONF_dfs_namenode_datanode_registration_ip___hostname___check=false
YARN_CONF_yarn_log___aggregation___enable=true
YARN_CONF_yarn_log_server_url=http://localhost:8188/applicationhistory/logs/
YARN_CONF_yarn_resourcemanager_recovery_enabled=true
YARN_CONF_yarn_resourcemanager_store_class=org.apache.hadoop.yarn.server.resourcemanager.recovery.FileSystemRMStateStore
YARN_CONF_yarn_resourcemanager_scheduler_class=org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler
YARN_CONF_yarn_scheduler_capacity_root_default_maximum___allocation___mb=8192
YARN_CONF_yarn_scheduler_capacity_root_default_maximum___allocation___vcores=4
YARN_CONF_yarn_resourcemanager_fs_state___store_uri=/rmstate
YARN_CONF_yarn_resourcemanager_system___metrics___publisher_enabled=true
YARN_CONF_yarn_resourcemanager_hostname=localhost
YARN_CONF_yarn_resourcemanager_address=localhost:8032
YARN_CONF_yarn_resourcemanager_scheduler_address=localhost:8030
YARN_CONF_yarn_resourcemanager_resource__tracker_address=localhost:8031
YARN_CONF_yarn_timeline___service_enabled=true
YARN_CONF_yarn_timeline___service_generic___application___history_enabled=true
YARN_CONF_yarn_timeline___service_hostname=localhost
YARN_CONF_mapreduce_map_output_compress=true
YARN_CONF_mapred_map_output_compress_codec=org.apache.hadoop.io.compress.SnappyCodec
YARN_CONF_yarn_nodemanager_resource_memory___mb=16384
YARN_CONF_yarn_nodemanager_resource_cpu___vcores=8
YARN_CONF_yarn_nodemanager_disk___health___checker_max___disk___utilization___per___disk___percentage=98.5
YARN_CONF_yarn_nodemanager_remote___app___log___dir=/app-logs
YARN_CONF_yarn_nodemanager_aux___services=mapreduce_shuffle
MAPRED_CONF_mapreduce_framework_name=yarn
MAPRED_CONF_mapred_child_java_opts=-Xmx4096m
MAPRED_CONF_mapreduce_map_memory_mb=4096
MAPRED_CONF_mapreduce_reduce_memory_mb=8192
MAPRED_CONF_mapreduce_map_java_opts=-Xmx3072m
MAPRED_CONF_mapreduce_reduce_java_opts=-Xmx6144m
MAPRED_CONF_yarn_app_mapreduce_am_env=HADOOP_MAPRED_HOME=/opt/hadoop-3.3.1/
MAPRED_CONF_mapreduce_map_env=HADOOP_MAPRED_HOME=/opt/hadoop-3.3.1/
MAPRED_CONF_mapreduce_reduce_env=HADOOP_MAPRED_HOME=/opt/hadoop-3.3.1/
Troubleshooting
-Make sure below ports and all default hadoop 3.3 ports are opened in firewall(if any)
9870,9864,8042,8088,8188,9000,8188,8032,8030,8031
-Set memory settings in below variables according to you rsystem resources
MAPRED_CONF_mapred_child_java_opts
MAPRED_CONF_mapreduce_map_memory_mb
MAPRED_CONF_mapreduce_reduce_memory_mb
MAPRED_CONF_mapreduce_map_java_opts
MAPRED_CONF_mapreduce_reduce_java_opts
YARN_CONF_yarn_scheduler_capacity_root_default_maximum___allocation___mb
YARN_CONF_yarn_nodemanager_resource_memory___mb
Content type
Image
Digest
Size
773.5 MB
Last updated
over 4 years ago
docker pull bajwalabs/apache-hadoop:pseudo.latest