[{"_1":2,"_175":-5,"_176":-5},"loaderData",{"_3":4,"_17":18,"_25":26,"_150":151},"root",{"_5":-5,"_6":-7,"_7":8,"_9":-7,"_10":11,"_12":-7,"_13":14,"_15":16},"currentUser","primaryEmail","siteVerification","6LeIxAcTAAAAAJcZVRqyHh71UMIEGNQ_MXjiZKhI","toast","enableDev",false,"incidentStatus","theme","system","contrast","normal","routes/_layout",{"_19":-5,"_20":21,"_22":21,"_23":11,"_24":21},"username","isAskGordonEnabled",true,"isUsingNewTemplate","showMinimalLanding","showHoneypotTrap","routes/_layout.r.$namespace.$reponame",{"_27":28,"_29":-5,"_30":11,"_31":11,"_32":33,"_34":35,"_36":-5,"_37":11,"_38":39,"_88":-5,"_89":90,"_139":-5,"_140":11,"_141":21,"_142":11,"_143":-5,"_144":11,"_145":-5,"_146":147,"_148":91,"_149":-7},"canonicalUrl","https://hub.docker.com/r/squ1d123/hadoop/tags","mcpServer","canSeeDockerfile","canSeeAutobuilds","namespace","squ1d123","reponame","hadoop","cloudBuildSource","isAutobuildsAvailable","repo",{"_40":41,"_42":43,"_45":46,"_47":48,"_49":11,"_50":51,"_52":53,"_54":55,"_57":35,"_32":33,"_58":59,"_60":44,"_61":62,"_63":64,"_65":66,"_67":-5,"_68":62,"_69":70,"_71":11,"_72":33,"_73":74,"_79":21,"_80":81,"_85":33,"_86":87},"categories",[],"content_types",[44],"image","date_registered","2018-06-28T03:58:57.331456Z","description","updating base version from upstream","is_private","last_updated","2019-01-28T07:40:27.454723Z","last_modified","2024-10-16T13:48:34.145251Z","media_types",[56],"application/vnd.docker.container.image.v1+json","name","pull_count",1212,"repository_type","star_count",0,"status",1,"status_description","active","source","collaborator_count","full_description","# Hadoop Docker\n\nThis repository contains *Dockerfile*s for setting up a basic Hadoop cluster.\nThe available components are:\n\n1. *HDFS*:\n\n * *namenode*\n * *dtanode*\n \n1. *YARN*:\n \n * *resourcemanager*\n * *nodemanager*\n\n1. *Spark submitter*\n\nAll images inherit from a base *hadoop* image which provides an hadoop\ninstallation in `/opt/` and provides a way to configure *hadoop* via\nenvironment variables. \n\n## Hadoop configuration\n\nThe *hadoop* configuration is controlled via the following environment\nvariable groups:\n\n1. `CORE_CONF`: affects `/etc/hadoop/core-site.xml`\n1. `HDFS_CONF`: affects `/etc/hadoop/hdfs-site.xml`\n1. `YARN_CONF`: affects `/etc/hadoop/yarn-site.xml`\n1. `HTTPFS_CONF`: affects `/etc/hadoop/httpfs-site.xml`\n1. `KMS_CONF`: affects `/etc/hadoop/KMS-site.xml` \n\n*Hadoop* properties by setting an environment variable with the\nappropriated prefix in the form `_`.\n\nDue to restriction imposed by `docker` and `docker-compose` on\nenvironment variable names the following substitution are applied to\nproperty names:\n\n* `_` => `.`\n* `__` => `_`\n* `___` => `-`\n\nFollowing are some illustratory examples:\n\n* `CORE_CONF_fs_defaultFS`: sets the *fs.defaultFS* property in\n`core-site.xml`\n* `YARN_CONF_yarn_log___aggregation___enable`: sets the\n *yarn.log-aggregation-enable* property in `yarn-site.xml`\n\n\n\n## Hadoop configuration presets\n\nFurthermore the following special environment variables control\nconfigurations presets:\n\n* `MULTIHOMED_NETWORK`: configure the *hadoop* cluster in such a way\n to be reachable from multiple networks, specifically the following\n properties are set:\n\n In `/etc/hadoop/hdfs-site.xml`:\n\n * dfs.namenode.rpc-bind-host = 0.0.0.0\n * dfs.namenode.servicerpc-bind-host = 0.0.0.0\n * dfs.namenode.http-bind-host = 0.0.0.0\n * dfs.namenode.https-bind-host = 0.0.0.0\n * dfs.client.use.datanode.hostname = true\n * dfs.datanode.use.datanode.hostname = true\n\n In `/etc/hadoop/yarn-site.xml`:\n\n * yarn.resourcemanager.bind-host = 0.0.0.0\n * yarn.nodemanager.bind-host = 0.0.0.0\n * yarn.nodemanager.bind-host = 0.0.0.0\n\n In `/etc/hadoop/mapred-site.xml`:\n\n * yarn.nodemanager.bind-host = 0.0.0.0\n\n* `GANGLIA_HOST`: instruct *hadoop* to send metrics to the specified\n *ganglia gmond* daemon (requires a unicast ganglia configuration) \n\n## Networking\n\nIn order for things to run smoothly it's reccomended to exploit the\nnew networking infrastructure of *docker* 1.9. Create a dedicated\n*network* for the cluster to run on.\n\nFurthermore is useful to fix the container *name* and the container\n*hostname* to the same value. This way every container will able to\nresolve itself with the same name as other container.\n\nLastly is useful to set the *domainname* equal to the name of the\n*network* and use *FQDN* to reference the various services.\n\nWith the specified setup is possible you'll be able to access the\nweb-interfaces of the various components without the annoying problem\nof unresolved links (provided that you setup a dns solution to resolve\ncontainer names and configure static routing if using\n*docker-machine*).\n\n## Components\n\n### namenode\n\nThe *hadoop-namenode* image starts an Hadoop NameNode. (single instance)\n\nAdditional environment variables:\n\n* `CLUSTER_NAME`: name of the *HDFS* cluster (used during the initial\nformatting)\n\nVolumes:\n\n* `/hadoop/dfs/name`: *HDFS* filesystem name directory\n\nMandatory configuration:\n\n* `CLUSTER_NAME`: cluster name\n\n*Docker-compose* template:\n\n namenode:\n image: uhopper/hadoop-namenode\n hostname: namenode\n container_name: namenode\n domainname: hadoop\n net: hadoop\n volumes:\n - :/hadoop/dfs/name\n environment:\n - GANGLIA_HOST=\n - CLUSTER_NAME=\n\nOnce running you can connect to `http://:50070` to see\nthe webui.\n\n### datanode\n\nThe *hadoop-datanode* image starts an Hadoop DataNode. (multiple instances)\n\nVolumes:\n\n* `/hadoop/dfs/data`: *HDFS* filesystem data directory\n\nMandatory configuration:\n\n* `CORE_CONF_fs_defaultFS`: *HDFS* address (i.e. `hdfs://:8020`)\n\n*Docker-compose* template:\n\n datanode1:\n image: uhopper/hadoop-datanode\n hostname: datanode1\n container_name: datanode1\n domainname: hadoop\n net: hadoop\n volumes:\n - :/hadoop/dfs/data\n environment:\n - GANGLIA_HOST=\n - CORE_CONF_fs_defaultFS=hdfs://:8020\n\n### resourcemanager\n\nThe *hadoop-resourcemanager* image starts an Hadoop\nResourceManager. (single instance)\n\nMandatory configuration:\n\n* `CORE_CONF_fs_defaultFS`: *HDFS* address (i.e. `hdfs://:8020`)\n\n*Docker-compose* template:\n\n resourcemanager:\n image: uhopper/hadoop-resourcemanager\n hostname: resourcemanager\n container_name: resourcemanager\n domainname: hadoop\n net: hadoop\n environment:\n - GANGLIA_HOST=\n - CORE_CONF_fs_defaultFS=hdfs://:8020\n - YARN_CONF_yarn_log___aggregation___enable=true\n\nOnce running you can connect to `http://:8088` to see\nthe webui.\n\n### nodemanager\n\nThe *hadoop-nodemanager* image starts an Hadoop NodeManager. (multiple\ninstances) \n\nMandatory configuration:\n\n* `CORE_CONF_fs_defaultFS`: *HDFS* address (i.e. `hdfs://:8020`)\n* `YARN_CONF_yarn_resourcemanager_hostname`: *resourcemanager* host\n\n*Docker-compose* template:\n\n nodemanager1:\n image: uhopper/hadoop-nodemanager\n hostname: nodemanager1\n container_name: nodemanager1\n domainname: hadoop\n net: hadoop\n environment:\n - GANGLIA_HOST=\n - CORE_CONF_fs_defaultFS=hdfs://:8020\n - YARN_CONF_yarn_resourcemanager_hostname=\n - YARN_CONF_yarn_log___aggregation___enable=true\n - YARN_CONF_yarn_nodemanager_remote___app___log___dir=/app-logs\n\n### spark\n\nThe *hadoop-spark* image is an utility container which provides a\nSpark environment configured for the *hadoop* cluster.\n\nThe image itself doesn't specify any command since no service are\nexposed. You are expected to specify it yourself via `docker run\nuhopper/hadoop-spark `.\n\nA common approach is to keep the container alive using `tail -f\n/var/log/dmesg` as command and then connect to it via `docker exec -ti\nspark bash` to have a *spark* environment.\n\nMandatory configuration:\n\n* `CORE_CONF_fs_defaultFS`: *HDFS* address (i.e. `hdfs://:8020`)\n* `YARN_CONF_yarn_resourcemanager_hostname`: *resourcemanager* host\n\n*Docker-compose* template:\n\n spark:\n image: uhopper/hadoop-spark\n hostname: spark\n container_name: spark\n domainname: hadoop\n net: hadoop\n environment:\n - CORE_CONF_fs_defaultFS=hdfs://namenode:8020\n - YARN_CONF_yarn_resourcemanager_hostname=resourcemanager\n command: tail -f /var/log/dmesg\n","has_starred","hub_user","immutable_tags_settings",{"_75":11,"_76":77},"enabled","rules",[78],".*","is_automated","permissions",{"_82":21,"_83":11,"_84":11},"read","write","admin","user","storage_size",8936683522,"orgData","recentTags",[91,129],{"_92":44,"_93":94,"_95":96,"_97":98,"_99":100,"_50":118,"_119":94,"_120":33,"_121":56,"_57":122,"_123":124,"_125":109,"_126":118,"_127":66,"_128":21},"content_type","creator",2268057,"full_size",335771370,"id",30203429,"images",[101],{"_102":103,"_104":105,"_106":107,"_108":109,"_110":111,"_112":113,"_114":107,"_115":-5,"_116":96,"_63":66,"_117":-5},"architecture","amd64","digest","sha256:b7fcedb458bd270a434826c929be346ac391fd80952debb5c489128b6e7c480d","features","","last_pulled","2026-09-24T21:39:22.3562077Z","last_pushed","2019-01-28T07:40:28Z","os","linux","os_features","os_version","size","variant","2019-01-28T07:40:27.156546Z","last_updater","last_updater_username","media_type","latest","repository",5671117,"tag_last_pulled","tag_last_pushed","tag_status","v2",{"_92":44,"_93":94,"_95":130,"_97":131,"_99":132,"_50":137,"_119":94,"_120":33,"_121":56,"_57":138,"_123":124,"_125":135,"_126":137,"_127":66,"_128":21},335507286,30010258,[133],{"_102":103,"_104":134,"_106":107,"_108":135,"_110":136,"_112":113,"_114":107,"_115":-5,"_116":130,"_63":66,"_117":-5},"sha256:52a2ba474d7bbb8771702fa2f80a875056a1db4778a6cfc4201721509e0a662b","2026-09-24T21:39:22.813405099Z","2018-06-28T04:27:00Z","2018-06-28T04:27:00.006394Z","2.8.4","sbxKitMetadata","isPotentialSpam","isHubSbxKitsEnabled","showSbxLogoOverride","sbxTrademarkText","displaySecurityMetadata","tagAttestations","tagsHaveAttestations",[],"selectedTag","attestationsUrl","routes/_layout.r.$namespace.$reponame.tags",{"_38":152,"_159":160,"_146":173,"_174":11},{"_40":153,"_42":154,"_45":46,"_47":48,"_49":11,"_50":51,"_52":53,"_54":155,"_57":35,"_32":33,"_58":59,"_60":44,"_61":62,"_63":64,"_65":66,"_67":-5,"_68":62,"_69":70,"_71":11,"_72":33,"_73":156,"_79":21,"_80":158,"_85":33,"_86":87},[],[44],[56],{"_75":11,"_76":157},[78],{"_82":21,"_83":11,"_84":11},"tags",{"_161":162,"_163":-5,"_164":-5,"_165":166},"count",2,"next","previous","results",[167,170],{"_92":44,"_93":94,"_95":96,"_97":98,"_99":168,"_50":118,"_119":94,"_120":33,"_121":56,"_57":122,"_123":124,"_125":109,"_126":118,"_127":66,"_128":21},[169],{"_102":103,"_104":105,"_106":107,"_108":109,"_110":111,"_112":113,"_114":107,"_115":-5,"_116":96,"_63":66,"_117":-5},{"_92":44,"_93":94,"_95":130,"_97":131,"_99":171,"_50":137,"_119":94,"_120":33,"_121":56,"_57":138,"_123":124,"_125":135,"_126":137,"_127":66,"_128":21},[172],{"_102":103,"_104":134,"_106":107,"_108":135,"_110":136,"_112":113,"_114":107,"_115":-5,"_116":130,"_63":66,"_117":-5},[],"canDeleteTags","actionData","errors"]

squ1d123/hadoop

Tags for squ1d123/hadoop