Sign inSign up

runaidevops/kwok-operator

By runaidevops

•Updated 7 months ago

docker image that manage nodePool, Podpool and deploymentPool resources

Image
Developer tools
0

9.0K

runaidevops/kwok-operator repository overview

⁠Kwok Operator

⁠Overview

The Kwok Operator is a Kubernetes operator designed to create virtual nodes within a Kubernetes cluster using Kwok, by applying custom resource definitions (CRDs) for node pools.

⁠Tested

The Kwok operator test on top the following kuberenetes flavors

  • Vanila
  • EKS ( Elastic Kubernetes Service )
  • GKE ( Goole Kubernetes Engine )
  • AKS ( Azure Kubernetes Service )
  • OKE (Oracle Kubernetes Engine )
  • RKE1
  • RKE2
  • Openshift
  • Kind

⁠Features

  • Automatically creates virtual nodes on Kwok infrastructure.
  • Utilizes Kubernetes Custom Resource Definitions (CRDs) for easy configuration.
  • Provides seamless integration with Kubernetes clusters.

⁠Prerequisites

Before using the Kwok Operator, ensure you have the following prerequisites installed:

  • tested on Kubernetes cluster (version 1.24 or later)
  • Kwok infrastructure set up and accessible from the cluster
  • kubectl CLI installed and configured to access the Kubernetes cluster

⁠Installation

To install Kwok CRDs and the Kwok Operator, follow these steps:

  1. Clone the Kwok Operator repository:

    git clone [email protected]:run-ai/kwok-operator.git

  2. enter to kwok-operator directory

    cd kwok-operator

  3. make sure kwok installed in your cluster from the URL: https://kwok.sigs.k8s.io/docs/user/kwok-in-cluster/⁠ or install by the script install_kwok.sh

    ./install_kwok.sh

  4. Apply the kwok-operator Kubernetes manifests:

    kubectl apply --server-side -k config/default

    or

    kubectl apply --server-side -f https://github.com/run-ai/kwok-operator/releases/download/1.0.3/kwok-operator.yaml

⁠Usage

To use the Kwok Operator to provision nodes, follow these steps:

  1. Define a NodePool custom resource (CR) with your desired configuration. Example:
apiVersion: kwok.sigs.run-ai.com/v1beta1
kind: NodePool
metadata:
  labels:
    app.kubernetes.io/name: nodepool
    app.kubernetes.io/instance: nodepool-sample
    app.kubernetes.io/part-of: kwok-operator
    app.kubernetes.io/managed-by: kustomize
    app.kubernetes.io/created-by: kwok-operator
  name: nodepool-sample
spec:
  nodeCount: 15
  nodeTemplate:
    apiVersion: v1
    metadata:
      annotations:
        node.alpha.kubernetes.io/ttl: "0"
      labels:
        kubernetes.io/hostname: kwok-node
        kubernetes.io/role: agent
        type: kwok
    spec: {}
    status:
      allocatable:
        cpu: 32
        memory: 256Gi
        pods: 110
      capacity:
        cpu: 32
        memory: 256Gi
        pods: 110
      nodeInfo:
        architecture: amd64
        bootID: ""
        containerRuntimeVersion: ""
        kernelVersion: ""
        kubeProxyVersion: fake
        kubeletVersion: fake
        machineID: ""
        operatingSystem: linux
        osImage: ""
        systemUUID: ""
      phase: Running

  1. Apply the NodePool CR to your Kubernetes cluster:

    kubectl apply -f path/to/your/nodepool.yaml

  2. Monitor the status of the created virtual nodes using:

    kubectl get nodes

⁠Configuration

The Kwok Operator can be configured via the NodePool CR.

kubectl edit nodepool nodepool-sample

To use the Kwok Operator to manage deployments and run the pods on top the nodes you provisioned above, follow these steps:

  1. ensure the namespace is exist
  2. Define a DeploymentPool custom resource (CR) with your desired configuration. Example:
apiVersion: kwok.sigs.run-ai.com/v1beta1
kind: DeploymentPool
metadata:
  labels:
    app.kubernetes.io/name: deploymentpool
    app.kubernetes.io/instance: deploymentpool-sample
    app.kubernetes.io/part-of: kwok-operator
    app.kubernetes.io/managed-by: kustomize
    app.kubernetes.io/created-by: kwok-operator
  name: deploymentpool-sample
  namespace: default
spec:  
  deploymentCount: 5
  deploymentTemplate:
    apiVersion: apps/v1 
    metadata:
      name: kwok-operator
      labels:
        app.kubernetes.io/name: deployment
        app.kubernetes.io/instance: deployment-sample
        app.kubernetes.io/part-of: kwok-operator
        app.kubernetes.io/managed-by: kustomize
        app.kubernetes.io/created-by: kwok-operator
    spec:
      replicas: 3
      selector:
        matchLabels: 
          app.kubernetes.io/name: deployment
          app.kubernetes.io/instance: deployment-sample
          app.kubernetes.io/part-of: kwok-operator
          app.kubernetes.io/managed-by: kustomize
          app.kubernetes.io/created-by: kwok-operator
      template:
        metadata:
          labels:
            app.kubernetes.io/name: deployment
            app.kubernetes.io/instance: deployment-sample
            app.kubernetes.io/part-of: kwok-operator
            app.kubernetes.io/managed-by: kustomize
            app.kubernetes.io/created-by: kwok-operator
        spec:
          containers:
          - image: nginx
            name: nginx
          restartPolicy: Always

To use the Kwok Operator to manage pods on top the nodes you provisioned above, follow these steps:

  1. ensure the namespace is exist
  2. Define a PodPool custom resource (CR) with your desired configuration. Example:
apiVersion: kwok.sigs.run-ai.com/v1beta1
kind: PodPool
metadata:
  labels:
    app.kubernetes.io/name: podpool
    app.kubernetes.io/instance: podpool-sample
    app.kubernetes.io/part-of: kwok-operator
    app.kubernetes.io/created-by: kwok-operator
  name: podpool-sample
  namespace: default
spec:
  podCount: 5
  podTemplate:
    metadata:
      name: kwok-operator
      labels:
        app.kubernetes.io/name: pod
        app.kubernetes.io/instance: pod-sample
        app.kubernetes.io/part-of: kwok-operator
        app.kubernetes.io/managed-by: kustomize
        app.kubernetes.io/created-by: kwok-operator
    spec:
      containers:
      - image: nginx
        name: nginx
      restartPolicy: Always

Added in version 0.0.5 To use the Kwok Operator to manage jobs on top the nodes you provisioned above, follow these steps:

  1. ensure the namespace is exist
  2. Define a JobPool custom resource (CR) with your desired configuration. Example:
apiVersion: kwok.sigs.run-ai.com/v1beta1
kind: JobPool
metadata:
  labels:
    app.kubernetes.io/name: jobpool
    app.kubernetes.io/instance: jobpool-sample
    app.kubernetes.io/part-of: kwok-operator
    app.kubernetes.io/managed-by: kustomize
    app.kubernetes.io/created-by: kwok-operator
  name: jobpool-sample
spec:
  jobCount: 5
  jobTemplate:
    metadata:
      name: kwok-operator
      labels:
        app.kubernetes.io/name: job
        app.kubernetes.io/instance: job-sample
        app.kubernetes.io/part-of: kwok-operator
        app.kubernetes.io/managed-by: kustomize
        app.kubernetes.io/created-by: kwok-operator
    spec:
      template:
        metadata:
          labels:
            app.kubernetes.io/name: job
            app.kubernetes.io/instance: job-sample
            app.kubernetes.io/part-of: kwok-operator
            app.kubernetes.io/managed-by: kustomize
            app.kubernetes.io/created-by: kwok-operator
        spec:
          containers:
          - name: job
            image: busybox
            command: ["sh", "-c", "echo Hello, Kubernetes! && sleep 3600"]
          restartPolicy: Never

Added in version 0.0.7 To use the Kwok Operator to manage Daemonset on top the nodes you provisioned above, follow these steps:

  1. ensure the namespace is exist
  2. Define a DaemonsetPool custom resource (CR) with your desired configuration. Example:
apiVersion: kwok.sigs.run-ai.com/v1beta1
kind: DaemonsetPool
metadata:
  labels:
    app.kubernetes.io/name: daemonsetpool
    app.kubernetes.io/instance: daemonsetpool-sample
    app.kubernetes.io/part-of: kwok-operator
    app.kubernetes.io/managed-by: kustomize
    app.kubernetes.io/created-by: kwok-operator
  name: daemonsetpool-sample
  namespace: default
spec:
  daemonsetCount: 10
  daemonsetTemplate:
    metadata:
      name: kwok-operator
      labels:
        app.kubernetes.io/name: daemonset
        app.kubernetes.io/instance: daemonset-sample
        app.kubernetes.io/part-of: kwok-operator
        app.kubernetes.io/managed-by: kustomize
        app.kubernetes.io/created-by: kwok-operator
    spec:
      selector:
        matchLabels: 
          app.kubernetes.io/name: deployment
          app.kubernetes.io/instance: deployment-sample
          app.kubernetes.io/part-of: kwok-operator
          app.kubernetes.io/managed-by: kustomize
          app.kubernetes.io/created-by: kwok-operator
      template:
        metadata:
          labels:
            app.kubernetes.io/name: daemonset
            app.kubernetes.io/instance: daemonset-sample
            app.kubernetes.io/part-of: kwok-operator
            app.kubernetes.io/managed-by: kustomize
            app.kubernetes.io/created-by: kwok-operator
        spec:
          containers:
          - image: nginx
            name: nginx
          restartPolicy: Always

⁠PoolChurner (periodic fake Pod churn)

PoolChurner keeps a steady number of KWOK-schedulable Pods and, on a fixed interval, deletes a batch so the operator recreates them. That produces API churn (create/delete traffic) without real workloads—useful for scenarios such as network policy scale testing (issue #18⁠).

  • podCount: steady-state number of Pods.
  • intervalSeconds (t): seconds between churn cycles. Set to 0 (or churnCount to 0) to disable churn and only maintain podCount (similar to PodPool).
  • churnCount: how many Pods to delete each cycle (they are recreated on the next reconcile).
  • workloadType: only Pod is implemented today; more workload kinds can be added later.

Example: config/samples/kwok.sigs_v1beta1_poolchurner.yaml.

⁠Troubleshooting

If you encounter any issues with the Kwok Operator, please check the following:

From version 1.0.0 the Kwok Operator is able to manage Statefuleset. To include PVC on top of the nodes you have provisioned above, follow these steps:

  1. ensure the namespace is exist
  2. ensure that storage class is installed and working as expected in the cluster
  3. Define a statefulesetPool custom resource (CR) with your desired configuration. Example:
apiVersion: kwok.sigs.run-ai.com/v1beta1
kind: StatefulsetPool
metadata:
  labels:
    app.kubernetes.io/name: statefulsetpool
    app.kubernetes.io/instance: statefulsetpool-sample
    app.kubernetes.io/part-of: kwok-operator
    app.kubernetes.io/managed-by: kustomize
    app.kubernetes.io/created-by: kwok-operator
  name: statefulsetpool-sample
spec:
  createPV: true # optional , default is false 
  statefulsetCount: 2
  StatefulsetTemplate:
    metadata:
      labels:
        app.kubernetes.io/name: statefulsetpool
        app.kubernetes.io/instance: statefulsetpool-sample
        app.kubernetes.io/part-of: kwok-operator
        app.kubernetes.io/managed-by: kustomize
        app.kubernetes.io/created-by: kwok-operator
    spec:
      serviceName: "nginx"
      replicas: 45
      selector:
        matchLabels:
          app: nginx
      template:
        metadata:
          labels:
            app: nginx
        spec:
          containers:
          - name: nginx
            image: registry.k8s.io/nginx-slim:0.21
            ports:
            - containerPort: 80
              name: web
            volumeMounts:
            - name: www
              mountPath: /usr/share/nginx/html
      volumeClaimTemplates:
      - metadata:
          name: www
        spec:
          accessModes: [ "ReadWriteOnce" ]
          resources:
            requests:
              storage: 1Gi

⁠Contributing

Contributions to the Kwok Operator are welcome! To contribute, please follow the guidelines outlined in CONTRIBUTING.md⁠.

Feel free to customize and expand upon this template to suit your specific needs and preferences!

Tag summary

Content type

Image

Digest

sha256:066602d06…

Size

12.6 MB

Last updated

7 months ago

docker pull runaidevops/kwok-operator:1.0.4