Sign inSign up

gudaoxuri/dmp

By gudaoxuri

•Updated over 8 years ago

DMP Environment

Image
0

513

gudaoxuri/dmp repository overview

== DMP Environment

Based gudaoxuri/spark:latest , Depend gudaoxuri/confluent:latest gudaoxuri/hive:latest gudaoxuri/hbase:latest

=== Features

  • SSH server & client ( Port: 22 User: root Password: 123456 )
  • Hadoop 2.7.x
  • Confluent 4.x
  • Hive 2.3.x
  • HBase 1.3.x
  • Phoenix 4.13.x
  • Spark 2.2.x

=== Examples

Start ZK:

docker run --name dmp_zookeeper -p 2181:2181 -d zookeeper

Start MySQL :

docker run --name dmp_mysql -p 3306:3306 -e MYSQL_ROOT_PASSWORD=123456 -d mysql

Start Redis:

docker run --name dmp_redis -p 6379:6379 -d redis

Start DMP :

docker run --name dmp -h dmp
-P -p 4000-4010:4000-4010
-p 222:22
-p 9092:9092
-p 10000:10000
-p 8765:8765
-p 4040:4040 -p 7077:7077 -p 18080:18080 -p 18081:18081
-p 9000:9000 -p 50010:50010 -p 50020:50020 -p 50070:50070 -p 50075:50075
--link dmp_mysql:mysql --link dmp_zookeeper:zookeeper --link dmp_redis:redis
-e KEEP=true -e KAFKA_ADVERTISED_HOST_NAME=10.200.131.25
-v /data/docker_data/dmp/workspaces:/opt/workspaces
-d gudaoxuri/dmp

Operation：

⁠[source,shell]

Hadoop Home:/opt/

example:

⁠HDFS

hdfs dfs -mkdir /test hdfs dfs -ls /

⁠Hive

cat>test_person.txt<<EOF tom 20 jack 24 nestor 29 EOF hdfs dfs -mkdir /tmp/hivetest/ hdfs dfs -put ./test_person.txt /tmp/hivetest/ hive CREATE EXTERNAL TABLE test_person(name STRING,age INT)ROW FORMAT DELIMITED FIELDS TERMINATED BY '\t' LOCATION '/tmp/hivetest'; select * from test_person;

beeline -u jdbc:hive2://localhost:10000 select * from test_person;

⁠HBase

hbase shell create 'test', 'cf' list 'test' put 'test', 'row1', 'cf:a', 'value1' put 'test', 'row2', 'cf:b', 'value2' put 'test', 'row3', 'cf:c', 'value3' scan 'test' get 'test', 'row1' disable 'test' drop 'test'

/opt/phoenix/bin/sqlline.py zookeeper:2181 !tables /opt/phoenix/bin/sqlline.py zookeeper:2181 /opt/phoenix/examples/STOCK_SYMBOL.sql select * from stock_symbol;

⁠Spark

run-example org.apache.spark.examples.SparkPi

spark-submit
--class org.apache.spark.examples.SparkPi
--master spark://dmp:7077
--deploy-mode cluster
--driver-memory 512M
--executor-memory 512M
/opt/spark/examples/jars/spark-examples_2.11-2.2.1.jar
1000

⁠Confluent

kafka-avro-console-producer
--broker-list localhost:9092 --topic test
--property value.schema='{"type":"record","name":"myrecord","fields":[{"name":"f1","type":"string"}]}' {"f1": "value1"} {"f1": "value2"} {"f1": "value3"}

kafka-avro-console-consumer --topic test
--bootstrap-server localhost:9092
--from-beginning

=== Environments

|=== | Env | Default Value | Remark

| TZ | Asia/Shanghai | | KEEP | false | true = always run | KAFKA_ADVERTISED_HOST_NAME | HOSTNAME | | KAFKA_ADVERTISED_PORT | 9092 | | KAFKA_PORT | 9092 | |===

=== Volumes

|=== | volume | Remark

| /data/hadoop/hdfs/nn | Name node path | /data/hadoop/hdfs/dn | Data node path | /opt/confluent/share/java | jars |===

=== Expose Ports

|=== | Port | Remark

| 22 | SSH Port

| 9092 | Kafka Service Port

| 10000 | Service for programatically (Thrift/JDBC) connecting to Hive,ENV Variable HIVE_PORT

| 60000 | hbase.master.port | 60010 | The port for the HBase­Master web UI. Set to -1 if you do not want the info server to run. hbase.master.info.port | 60030 | hbase.regionserver.info.port | 8765 | HBase Query Server default port

| 4040 | REST API | 7077 | SPARK_MASTER_PORT | 18080 | SPARK_MASTER_WEBUI_PORT | 18081 | SPARK_WORKER_WEBUI_PORT

| 9000 | File system metadata operations fs.default.name | 50010 | Data transfer dfs.datanode.address | 50020 | Metadata operations dfs.datanode.ipc.address | 50070 | Web UI to look at current status of HDFS, explore file system dfs.http.address | 50075 | DataNode WebUI to access the status, logs etc. dfs.datanode.http.address | 50090 | Checkpoint for NameNode metadata dfs.secondary.http.address |===

Tag summary

Content type

Image

Digest

Size

1.8 GB

Last updated

over 8 years ago

docker pull gudaoxuri/dmp