yangqizhou/flinkk8soperator repository overview

⁠Flinkk8soperator

FlinkK8sOperator is a Kubernetes operator⁠ that manages Flink⁠ applications on Kubernetes. The operator acts as control plane to manage the complete deployment lifecycle of the application.

⁠Project Status

Beta

The operator is in use for some less-crtical jobs at Lyft. At this point the focus is on testing and stability While in Beta, we will attempt to limit the number of backwards-incompatible changes, but they may still occur as necessary.

⁠Prerequisites

• Version >= 1.9 of Kubernetes.
• Version >= 1.7 of Apache Flink.

⁠Overview

The goal of running Flink on Kubernetes is to enable more flexible, lighter-weight deployment of streaming applications, without needing to manage infrastructure. The Flink operator aims to abstract out the complexity of hosting, configuring, managing and operating Flink clusters from application developers. It achieves this by extending any kubernetes cluster using a custom resources⁠.

The Operator creates flink clusters dynamically using the specified custom resource. Flink clusters in kubernetes consist of the following:

• JobManager Deployment⁠
• TaskManager Deployment⁠
• JobManager Service⁠
• JobManager Ingress⁠ for the UI

Deploying and managing Flink applications in Kubernetes involves two steps:

• Building Flink application packaged as a docker image: A docker image is built containing the application source code with the necessary dependencies built in. This is required to bootstrap the Jobmanager and Taskmanager pods. At Lyft we use Source-To-Image S2I⁠ as the image build tool that provides a common builder image with Apache Flink pre-installed. The docker image could be built using any pre-existing workflows at an organization.

• Creating the Flink application custom resource: The custom resource for Flink application provides the spec for configuring and managing flink clusters in Kubernetes. The FlinkK8sOperator, deployed on Kubernetes, continuously monitors the resource and the corresponding flink cluster, and performs action based on the diff.

⁠Documentation

• Quick start guide⁠
• User guide⁠
• Flink application custom resource⁠
• Operator state machine⁠