zyzcenter/docker_dmp2 repository overview

== DMP Environment

Based tairangroup/kylin:latest , Depend tairangroup/spark:latest tairangroup/confluent:latest tairangroup/hive:latest tairangroup/hbase:latest

=== Features

• SSH server & client ( Port: 22 User: root Password: 123456 )
• Hadoop 2.7.x
• Confluent 4.x
• Hive 2.3.x
• HBase 1.3.x
• Phoenix 4.13.x
• Spark 2.3.x
• Kylin 2.2.x
• Livy 0.5.x

=== Examples

Start ZK:

docker run --name dmp_zookeeper -p 2181:2181 -d zookeeper

Start MySQL :

docker run --name dmp_mysql -p 3306:3306 -e MYSQL_ROOT_PASSWORD=123456 -d mysql

Start Redis:

docker run --name dmp_redis -p 6379:6379 -d redis

Start ES With SQL:

docker run --name dmp_es -p 9200:9200 -e "ES_JAVA_OPTS=-Xms256m -Xmx256m" -d elasticsearch docker exec -it dmp_es bash

Start DMP :

docker run --name dmp -h dmp
-P -p 4000-4010:4000-4010
-p 222:22
-p 9092:9092
-p 10000:10000
-p 8765:8765
-p 8088:8088
-p 8042:8042
-p 7070:7070
-p 8998:8998
-p 4040:4040 -p 7077:7077 -p 8080:8080 -p 8081:8081
-p 9000:9000 -p 50010:50010 -p 50020:50020 -p 50070:50070 -p 50075:50075
--link dmp_mysql:mysql --link dmp_zookeeper:zookeeper --link dmp_redis:redis
-e KEEP=true -e KAFKA_ADVERTISED_HOST_NAME=10.200.131.25
-v /data/docker_data/dmp/workspaces:/opt/workspaces
-d tairangroup/dmp

Operation：

⁠[source,shell]

Hadoop Home:/opt/

example:

⁠HDFS

hdfs dfs -mkdir /test hdfs dfs -ls /

⁠Hive

⁠HBase

⁠Spark

⁠Confluent

⁠Kylin

打开浏览器，输入http://localhost:7070⁠ 账密：ADMIN:KYLIN

1. 选择project：learn-kylin
2. 选择cube： kylin_sales_cube 点击 action -> build 进行构建cube -> 进入Monitor界面查看状态
3. 执行sql查询： build成功后，-> Insight，输入sql语句 得到fact table所缓存的列——均为dimension的主key、measure中所需计算的字段。 -> select * from kylin_sales 各个时间段内的销售额及购买量： -> select part_dt, sum(price) as total_selled, count(distinct seller_id) as sellers from kylin_sales group by part_dt order by part_dt

⁠Kylin stream

1. 往队列 kylin_streaming_topic 添加数据

1. 选择cube： kylin_streaming_cube 点击 action -> build 进行构建cube -> 进入Monitor界面查看状态

2. 执行sql查询： build成功后，-> Insight，输入sql语句

   select minute_start, count(*), sum(amount), sum(qty) from streaming_sales_table group by minute_start order by minute_start

⁠Livy

Usage batch session :

1. Set up the Headers : key -> Content-Type, value -> application/json
2. Create batch session : POST/

   Set up the URL : http://10.200.131.25:8998/batches⁠ Set up the Body : such as : { "name": "storage-rds", "file": "/storage/rds/storage-rds-1.0.0-SNAPSHOT-jar-with-dependencies.jar", "className": "com.tairanchina.csp.dmp.components.storage.rds.RDSMain", "args": ["-k","10.200.131.25:9092", "-t","dmp.source.api-visit", "-c","visit" ], "driverMemory": "512M", "driverCores": 1, "executorMemory": "512M", "executorCores": 1, "numExecutors": 1 }

3. Check the status : GET/

   Set up the URL : http://10.200.131.25:8998/batches/{batchId}/state⁠

4. Delete the batch session : DELETE/

   Set up the URL : http://10.200.131.25:8998/batches/{batchId}⁠

---

=== Environments

|=== | Env | Default Value | Remark

| TZ | Asia/Shanghai | | KEEP | false | true = always run | KAFKA_ADVERTISED_HOST_NAME | HOSTNAME | | KAFKA_ADVERTISED_PORT | 9092 | | KAFKA_PORT | 9092 | |===

=== Volumes

|=== | volume | Remark

| /data/hadoop/hdfs/nn | Name node path | /data/hadoop/hdfs/dn | Data node path | /opt/confluent/share/java | jars |===

=== Expose Ports

|=== | Port | Remark

| 22 | SSH Port

| 9092 | Kafka Service Port

| 10000 | Service for programatically (Thrift/JDBC) connecting to Hive,ENV Variable HIVE_PORT

| 60000 | hbase.master.port | 60010 | The port for the HBase­Master web UI. Set to -1 if you do not want the info server to run. hbase.master.info.port | 60030 | hbase.regionserver.info.port | 8765 | HBase Query Server default port

| 4040 | REST API | 7077 | SPARK_MASTER_PORT | 18080 | SPARK_MASTER_WEBUI_PORT | 18081 | SPARK_WORKER_WEBUI_PORT

| 8998 | LIVY_REST_PORT

| 9000 | File system metadata operations fs.default.name | 50010 | Data transfer dfs.datanode.address | 50020 | Metadata operations dfs.datanode.ipc.address | 50070 | Web UI to look at current status of HDFS, explore file system dfs.http.address | 50075 | DataNode WebUI to access the status, logs etc. dfs.datanode.http.address | 50090 | Checkpoint for NameNode metadata dfs.secondary.http.address |===