kuzzleio/cluster repository overview

⁠Cluster mode plugin

This plugin adds a cluster mode to Kuzzle.

⁠Kuzzle

Kuzzle is a ready-to-use, on-premises and scalable backend that enables you to manage your persistent data and be notified in real-time on whatever happens to it. It also provides you with a flexible and powerful user-management system.

• :watch: Kuzzle in 5 minutes⁠
• :octocat: Github⁠
• :earth_africa: Website⁠
• :books: Documentation⁠
• :email: Gitter⁠

⁠Get trained by the creators of Kuzzle :zap:

Train yourself and your teams to use Kuzzle to maximize its potential and accelerate the development of your projects.
Our teams will be able to meet your needs in terms of expertise and multi-technology support for IoT, mobile/web, backend/frontend, devops.
:point_right: Get a quote⁠

⁠Compatibility

Kuzzle: >=1.8.0

⁠Try it

To run a kuzzle stack, you can use the provided compose file:

docker-compose up --scale kuzzle=3

**NB: This compose stack is for tests and development only and should not be used as-is on production. **

⁠Run a development stack

cd <dir>
git clone https://github.com/kuzzleio/kuzzle-plugin-cluster.git

cd kuzzle-plugin-cluster
cp docker-compose/my.env.sample docker-compose/my.env

./dev-npm-install.sh
./dev.sh

You should now have a full Kuzzle clustered stack running 3 Kuzzle front nodes (and 3 servers). Each update on the cluster source should automatically restart kuzzle.

Note: on some Linux environments, you may get ENOSPC errors from the filesystem watcher. If so, you need to raise the limits on the number of files that can be watched:

sudo sysctl -w fs.inotify.max_user_watches=524288

⁠Goodies

• [http://localhost:7512/_plugin/cluster/status] => cluster status
• curl -XPOST http://localhost:7512/_plugin/cluster/reset => resets redis state and force a new sync (blanks cluster state)
• bash docker-compose/scripts/devtools.sh dumps to the standard output the urls to copy/paste in Google Chrome to live-debug the nodes

⁠Run the cluster in production

⁠Install

The cluster needs to be installed as a plugin. Please refer to Kuzzle documentation⁠ on how to proceed.

⁠Network ports

By default, Kuzzle nodes communicate with each other using two channels on ports 7510 and 7511.
NB: These ports are used by the cluster only and do not need to be publicly exposed.

You can configure the ports used in the bindings section of the plugin configuration (cf below⁠).

Each Kuzzle node also needs to be able to access Redis and Elasticsearch services.

⁠Healthcheck

The cluster exposes a healthcheck route on http://kuzzle:7512/_plugin/cluster/health⁠

The route returns a 200 status code only if the minimumNodes set in the configuration is reached (cf configuration⁠ below).

⁠Configuration

⁠Privileged context

This plugin needs privileged context to work. This context is granted by Kuzzle via the global configuration. Add the following to your configuration

plugins: {
    cluster: {
        privileged: true
    }
}

For more information on how to configure Kuzzle, please refer to the Guide⁠.

⁠Pipe plugin timeouts

This plugin registers some pipe plugins which induce some delay and will exceed default Kuzzle timeouts. Make sure you increase your pipe timeouts accordingly.

  "plugins": {
    "common": {
      "pipeWarnTime": 5000,
      "pipeTimeout": 10000
    }

⁠Bindings

The bindings on which each node can be reached by the others can be configured in the bindings section of the cluster plugin configuration:

"plugins": {
  "cluster": {
    "bindings": {
      "pub": "tcp://[_site_:ipv4]:7511",
      "router": "tcp://[_site_:ipv4]:7510"
    }
  }
}

The syntax is tcp://[<interface>:<family>]:<port>, where

• interface is either a network interface (i.e. eth0), an ip address (i.e. 0.0.0.0) or _site_. If set to _site_, the first public ip will be used.
• family is either set to ipv4 (default) or ipv6
• port is set to the port to listen to

⁠Redis cluster

Redis cluster comes with some limitations:

1. Single database only.
2. Cluster node arrays.

The latter implies the configuration cannot be set via environment variables. To comply with the former, make sure to set only one database index (0).

⁠Full configuration sample

Here is a complete sample configuration using a 3 nodes redis cluster and a 2 elasticsearch nodes.

"plugins": {
  "common": {
    "pipeWarnTime": 5000,
    "pipeTimeout": 10000
  },
  "cluster": {
    "privileged": true,
    "bindings": {
      "pub": "tcp://[_site_:ipv4]:7511",
      "router": "tcp://[_site_:ipv4]:7510"
    },
    "minimumNodes": 1,
    "retryJoin": 30,
    "timers": {
      "discoverTimeout": 3000,
      "joinAttemptIntervall": 2000,
      "heartbeat": 5000,
      "waitForMissingRooms": 4500
    }
  }
},

"services": {
  "internalCache": {
    "database": 0,
    "nodes": [
      {
        "host": "redis1",
        "port": 6379
      },
      {
        "host": "redis2",
        "port": 6379
      },
      {
        "host": "redis3",
        "port": 6379
      }
    ]
  },
  "memoryStorage": {
    "database": 0,
    "nodes": [
      {
        "host": "redis1",
        "port": 6379
      },
      {
        "host": "redis2",
        "port": 6379
      },
      {
        "host": "redis3",
        "port": 6379
      }
    ]
  },
  "db": {
    "client": {
      "host": false,
      "hosts": [
        {
          "protocol": "http",
          "host": "elasticsearch1",
          "port": 9200
        },
        {
          "protocol": "http",
          "host": "elasticsearch2",
          "port": 9200
        }
      ]
    }
  }
}