[{"_1":2,"_174":-5,"_175":-5},"loaderData",{"_3":4,"_17":18,"_25":26,"_149":150},"root",{"_5":-5,"_6":-7,"_7":8,"_9":-7,"_10":11,"_12":-7,"_13":14,"_15":16},"currentUser","primaryEmail","siteVerification","6LeIxAcTAAAAAJcZVRqyHh71UMIEGNQ_MXjiZKhI","toast","enableDev",false,"incidentStatus","theme","system","contrast","normal","routes/_layout",{"_19":-5,"_20":21,"_22":21,"_23":21,"_24":21},"username","isAskGordonEnabled",true,"isUsingNewTemplate","showMinimalLanding","showHoneypotTrap","routes/_layout.r.$namespace.$reponame",{"_27":28,"_29":-5,"_30":11,"_31":11,"_32":33,"_34":35,"_36":-5,"_37":11,"_38":39,"_88":-5,"_89":90,"_138":-5,"_139":11,"_140":21,"_141":11,"_142":-5,"_143":11,"_144":-5,"_145":146,"_147":91,"_148":-7},"canonicalUrl","https://hub.docker.com/r/nulloranje/hadoop/tags","mcpServer","canSeeDockerfile","canSeeAutobuilds","namespace","nulloranje","reponame","hadoop","cloudBuildSource","isAutobuildsAvailable","repo",{"_40":41,"_42":43,"_45":46,"_47":48,"_49":11,"_50":51,"_52":53,"_54":55,"_57":35,"_32":33,"_58":59,"_60":44,"_61":62,"_63":64,"_65":66,"_67":-5,"_68":62,"_69":70,"_71":11,"_72":33,"_73":74,"_79":11,"_80":81,"_85":33,"_86":87},"categories",[],"content_types",[44],"image","date_registered","2019-01-14T18:26:53.451867Z","description","Base Hadoop image","is_private","last_updated","2019-01-15T15:17:11.529601Z","last_modified","2024-10-16T13:48:34.145251Z","media_types",[56],"application/vnd.docker.container.image.v1+json","name","pull_count",1565,"repository_type","star_count",0,"status",1,"status_description","active","source","collaborator_count","full_description","# Hadoop Docker\n\nThis repository contains *Dockerfile*s for setting up a basic Hadoop cluster.\nThe available components are:\n\n1. *HDFS*:\n\n * *namenode*\n * *datanode*\n \n1. *YARN*:\n \n * *resourcemanager*\n * *nodemanager*\n\n1. *Spark submitter*\n\nAll images inherit from a base *hadoop* image which provides an hadoop\ninstallation in `/opt/` and provides a way to configure *hadoop* via\nenvironment variables. \n\n## Hadoop configuration\n\nThe *hadoop* configuration is controlled via the following environment\nvariable groups:\n\n1. `CORE_CONF`: affects `/etc/hadoop/core-site.xml`\n1. `HDFS_CONF`: affects `/etc/hadoop/hdfs-site.xml`\n1. `YARN_CONF`: affects `/etc/hadoop/yarn-site.xml`\n1. `HTTPFS_CONF`: affects `/etc/hadoop/httpfs-site.xml`\n1. `KMS_CONF`: affects `/etc/hadoop/KMS-site.xml` \n1. `MAPRED_CONF`: affects `/etc/hadoop/mapred-site.xml` \n\n*Hadoop* properties by setting an environment variable with the\nappropriated prefix in the form `_`.\n\nDue to restriction imposed by `docker` and `docker-compose` on\nenvironment variable names the following substitution are applied to\nproperty names:\n\n* `_` => `.`\n* `__` => `_`\n* `___` => `-`\n\nFollowing are some illustratory examples:\n\n* `CORE_CONF_fs_defaultFS`: sets the *fs.defaultFS* property in\n`core-site.xml`\n* `YARN_CONF_yarn_log___aggregation___enable`: sets the\n *yarn.log-aggregation-enable* property in `yarn-site.xml`\n\n\n\n## Hadoop configuration presets\n\nFurthermore the following special environment variables control\nconfigurations presets:\n\n* `MULTIHOMED_NETWORK`: configure the *hadoop* cluster in such a way\n to be reachable from multiple networks, specifically the following\n properties are set:\n\n In `/etc/hadoop/hdfs-site.xml`:\n\n * dfs.namenode.rpc-bind-host = 0.0.0.0\n * dfs.namenode.servicerpc-bind-host = 0.0.0.0\n * dfs.namenode.http-bind-host = 0.0.0.0\n * dfs.namenode.https-bind-host = 0.0.0.0\n * dfs.client.use.datanode.hostname = true\n * dfs.datanode.use.datanode.hostname = true\n\n In `/etc/hadoop/yarn-site.xml`:\n\n * yarn.resourcemanager.bind-host = 0.0.0.0\n * yarn.nodemanager.bind-host = 0.0.0.0\n * yarn.nodemanager.bind-host = 0.0.0.0\n\n In `/etc/hadoop/mapred-site.xml`:\n\n * yarn.nodemanager.bind-host = 0.0.0.0\n\n* `GANGLIA_HOST`: instruct *hadoop* to send metrics to the specified\n *ganglia gmond* daemon (requires a unicast ganglia configuration) \n\n## Networking\n\nIn order for things to run smoothly it's reccomended to exploit the\nnew networking infrastructure of *docker* 1.9. Create a dedicated\n*network* for the cluster to run on.\n\nFurthermore is useful to fix the container *name* and the container\n*hostname* to the same value. This way every container will able to\nresolve itself with the same name as other container.\n\nLastly is useful to set the *domainname* equal to the name of the\n*network* and use *FQDN* to reference the various services.\n\nWith the specified setup is possible you'll be able to access the\nweb-interfaces of the various components without the annoying problem\nof unresolved links (provided that you setup a dns solution to resolve\ncontainer names and configure static routing if using\n*docker-machine*).\n\n## Components\n\n### namenode\n\nThe *hadoop-namenode* image starts an Hadoop NameNode. (single instance)\n\nAdditional environment variables:\n\n* `CLUSTER_NAME`: name of the *HDFS* cluster (used during the initial\nformatting)\n\nVolumes:\n\n* `/hadoop/dfs/name`: *HDFS* filesystem name directory\n\nMandatory configuration:\n\n* `CLUSTER_NAME`: cluster name\n\n*Docker-compose* template:\n\n namenode:\n image: nulloranje/hadoop-namenode\n hostname: namenode\n container_name: namenode\n domainname: hadoop\n net: hadoop\n volumes:\n - :/hadoop/dfs/name\n environment:\n - GANGLIA_HOST=\n - CLUSTER_NAME=\n\nOnce running you can connect to `http://:50070` to see\nthe webui.\n\n### datanode\n\nThe *hadoop-datanode* image starts an Hadoop DataNode. (multiple instances)\n\nVolumes:\n\n* `/hadoop/dfs/data`: *HDFS* filesystem data directory\n\nMandatory configuration:\n\n* `CORE_CONF_fs_defaultFS`: *HDFS* address (i.e. `hdfs://:8020`)\n\n*Docker-compose* template:\n\n datanode1:\n image: nulloranje/hadoop-datanode\n hostname: datanode1\n container_name: datanode1\n domainname: hadoop\n net: hadoop\n volumes:\n - :/hadoop/dfs/data\n environment:\n - GANGLIA_HOST=\n - CORE_CONF_fs_defaultFS=hdfs://:8020\n\n### resourcemanager\n\nThe *hadoop-resourcemanager* image starts an Hadoop\nResourceManager. (single instance)\n\nMandatory configuration:\n\n* `CORE_CONF_fs_defaultFS`: *HDFS* address (i.e. `hdfs://:8020`)\n\n*Docker-compose* template:\n\n resourcemanager:\n image: nulloranje/hadoop-resourcemanager\n hostname: resourcemanager\n container_name: resourcemanager\n domainname: hadoop\n net: hadoop\n environment:\n - GANGLIA_HOST=\n - CORE_CONF_fs_defaultFS=hdfs://:8020\n - YARN_CONF_yarn_log___aggregation___enable=true\n\nOnce running you can connect to `http://:8088` to see\nthe webui.\n\n### nodemanager\n\nThe *hadoop-nodemanager* image starts an Hadoop NodeManager. (multiple\ninstances) \n\nMandatory configuration:\n\n* `CORE_CONF_fs_defaultFS`: *HDFS* address (i.e. `hdfs://:8020`)\n* `YARN_CONF_yarn_resourcemanager_hostname`: *resourcemanager* host\n\n*Docker-compose* template:\n\n nodemanager1:\n image: nulloranje/hadoop-nodemanager\n hostname: nodemanager1\n container_name: nodemanager1\n domainname: hadoop\n net: hadoop\n environment:\n - GANGLIA_HOST=\n - CORE_CONF_fs_defaultFS=hdfs://:8020\n - YARN_CONF_yarn_resourcemanager_hostname=\n - YARN_CONF_yarn_log___aggregation___enable=true\n - YARN_CONF_yarn_nodemanager_remote___app___log___dir=/app-logs\n\n### spark\n\nThe *hadoop-spark* image is an utility container which provides a\nSpark environment configured for the *hadoop* cluster.\n\nThe image itself doesn't specify any command since no service are\nexposed. You are expected to specify it yourself via `docker run\nnulloranje/hadoop-spark `.\n\nA common approach is to keep the container alive using `tail -f\n/var/log/dmesg` as command and then connect to it via `docker exec -ti\nspark bash` to have a *spark* environment.\n\nMandatory configuration:\n\n* `CORE_CONF_fs_defaultFS`: *HDFS* address (i.e. `hdfs://:8020`)\n* `YARN_CONF_yarn_resourcemanager_hostname`: *resourcemanager* host\n\n*Docker-compose* template:\n\n spark:\n image: nulloranje/hadoop-spark\n hostname: spark\n container_name: spark\n domainname: hadoop\n net: hadoop\n environment:\n - CORE_CONF_fs_defaultFS=hdfs://namenode:8020\n - YARN_CONF_yarn_resourcemanager_hostname=resourcemanager\n command: tail -f /var/log/dmesg\n","has_starred","hub_user","immutable_tags_settings",{"_75":11,"_76":77},"enabled","rules",[78],".*","is_automated","permissions",{"_82":21,"_83":11,"_84":11},"read","write","admin","user","storage_size",901028020,"orgData","recentTags",[91,129],{"_92":44,"_93":94,"_95":96,"_97":98,"_99":100,"_50":118,"_119":94,"_120":33,"_121":56,"_57":122,"_123":124,"_125":109,"_126":118,"_127":66,"_128":21},"content_type","creator",5117096,"full_size",336580742,"id",44146443,"images",[101],{"_102":103,"_104":105,"_106":107,"_108":109,"_110":111,"_112":113,"_114":107,"_115":-5,"_116":96,"_63":66,"_117":-5},"architecture","amd64","digest","sha256:ab8c813c7436fb3029d28da84f191823fe1d69bb2b421b0205de0e88a099fdd1","features","","last_pulled","2026-10-14T22:25:59.3015945Z","last_pushed","2019-01-15T15:16:55Z","os","linux","os_features","os_version","size","variant","2019-01-15T15:17:05.31553Z","last_updater","last_updater_username","media_type","latest","repository",6548301,"tag_last_pulled","tag_last_pushed","tag_status","v2",{"_92":44,"_93":94,"_95":96,"_97":130,"_99":131,"_50":136,"_119":94,"_120":33,"_121":56,"_57":137,"_123":124,"_125":134,"_126":136,"_127":66,"_128":21},44149340,[132],{"_102":103,"_104":133,"_106":107,"_108":134,"_110":135,"_112":113,"_114":107,"_115":-5,"_116":96,"_63":66,"_117":-5},"sha256:28e49dae502824f6b723044a37be3fcbc5e3e419a3573ce33e75edb6a1298843","2026-10-14T22:10:56.922573727Z","2019-01-15T14:39:13Z","2019-01-15T15:17:04.008041Z","2.8.5","sbxKitMetadata","isPotentialSpam","isHubSbxKitsEnabled","showSbxLogoOverride","sbxTrademarkText","displaySecurityMetadata","tagAttestations","tagsHaveAttestations",[],"selectedTag","attestationsUrl","routes/_layout.r.$namespace.$reponame.tags",{"_38":151,"_158":159,"_145":172,"_173":11},{"_40":152,"_42":153,"_45":46,"_47":48,"_49":11,"_50":51,"_52":53,"_54":154,"_57":35,"_32":33,"_58":59,"_60":44,"_61":62,"_63":64,"_65":66,"_67":-5,"_68":62,"_69":70,"_71":11,"_72":33,"_73":155,"_79":11,"_80":157,"_85":33,"_86":87},[],[44],[56],{"_75":11,"_76":156},[78],{"_82":21,"_83":11,"_84":11},"tags",{"_160":161,"_162":-5,"_163":-5,"_164":165},"count",2,"next","previous","results",[166,169],{"_92":44,"_93":94,"_95":96,"_97":98,"_99":167,"_50":118,"_119":94,"_120":33,"_121":56,"_57":122,"_123":124,"_125":109,"_126":118,"_127":66,"_128":21},[168],{"_102":103,"_104":105,"_106":107,"_108":109,"_110":111,"_112":113,"_114":107,"_115":-5,"_116":96,"_63":66,"_117":-5},{"_92":44,"_93":94,"_95":96,"_97":130,"_99":170,"_50":136,"_119":94,"_120":33,"_121":56,"_57":137,"_123":124,"_125":134,"_126":136,"_127":66,"_128":21},[171],{"_102":103,"_104":133,"_106":107,"_108":134,"_110":135,"_112":113,"_114":107,"_115":-5,"_116":96,"_63":66,"_117":-5},[],"canDeleteTags","actionData","errors"]

nulloranje/hadoop

Tags for nulloranje/hadoop