Public | Automated Build

Last pushed: a year ago
Short Description
Apache Flink docker image. This repo uses the image in apache/flink/contrib-flink.
Full Description

Apache Flink cluster deployment on docker using docker-compose

Installation

Install the most recent stable version of docker
https://docs.docker.com/installation/

Install the most recent stable version of docker-compose
https://docs.docker.com/compose/install/

Build

Images are based on the official Java Alpine (OpenJDK 8) image. If you want to
build the flink image run:

sh build.sh

or

docker build -t flink .

If you want to build the container for a specific version of flink/hadoop/scala
you can configure it in the respective args:

docker build --build-arg FLINK_VERSION=1.0.3 --build-arg HADOOP_VERSION=26 --build-arg SCALA_VERSION=2.10 -t "flink:1.0.3-hadoop2.6-scala_2.10" flink

Deploy

  • Deploy cluster and see config/setup log output (best run in a screen session)

      docker-compose up
    
  • Deploy as a daemon (and return)

      docker-compose up -d
    
  • Scale the cluster up or down to N TaskManagers

      docker-compose scale taskmanager=<N>
    
  • Access the Job Manager container

      docker exec -it $(docker ps --filter name=flink_jobmanager --format={{.ID}}) /bin/sh
    
  • Kill the cluster

      docker-compose kill
    
  • Upload jar to the cluster

      docker cp <your_jar> $(docker ps --filter name=flink_jobmanager --format={{.ID}}):/<your_path>
    
  • Copy file to all the nodes in the cluster

      for i in $(docker ps --filter name=flink --format={{.ID}}); do
          docker cp <your_file> $i:/<your_path>
      done
    
  • Run a topology

From the jobmanager:

    docker exec -it $(docker ps --filter name=flink_jobmanager --format={{.ID}}) flink run -m <jobmanager:port> -c <your_class> <your_jar> <your_params>

If you have a local flink installation:

    $FLINK_HOME/bin/flink run -m <jobmanager:port> <your_jar>

or

    $FLINK_HOME/bin/flink run -m <jobmanager:port> -c <your_class> <your_jar> <your_params>

Ports

  • The Web Client is on port 48081
  • JobManager RPC port 6123 (default, not exposed to host)
  • TaskManagers RPC port 6122 (default, not exposed to host)
  • TaskManagers Data port 6121 (default, not exposed to host)

Edit the docker-compose.yml file to edit port settings.

Docker Pull Command
Owner
iemejia
Source Repository

Comments (0)