This plugin allows you provision and mount remote volumes using Heketi and GlusterFS in your container easily. To use this plugin you must have deployed GlusterFS cluster with Heketi service.
Heketi provides a RESTful management interface which can be used to manage the life cycle of GlusterFS volumes. With Heketi you can dynamically provision GlusterFS volumes with any of the supported durability types. Heketi will automatically determine the location for bricks across the cluster, making sure to place bricks and its replicas across different failure domains. Heketi also supports any number of GlusterFS clusters, allowing cloud services to provide network file storage without being limited to a single GlusterFS cluster.
When a request from plugin is received to create a volume, Heketi will first allocate the appropriate storage in a cluster, making sure to place brick replicas across failure domains. It will then format, then mount the storage to create bricks for the volume requested. Once all bricks have been automatically created, Heketi will finally satisfy the request by creating, then starting the newly created GlusterFS volume.
# HEKETI_SERVER_URI - Heketi service address. For example Heketi listening on: http://192.168.254.1:8080
$ docker plugin install vpps/heketi-volume-plugin:latest HEKETI_SERVER_URI=http://192.168.254.1:8080
# or with authorization on Heketi. For example user: admin, key: y5x57EsZb7Cd3943
$ docker plugin install vpps/heketi-volume-plugin:latest HEKETI_SERVER_URI=http://192.168.254.1:8080 HEKETI_USER=admin HEKETI_KEY=y5x57EsZb7Cd3943
Plugin "vpps/heketi-volume-plugin:latest" is requesting the following privileges:
- network: [host]
- mount: [/var/lib/docker/plugins/]
- device: [/dev/fuse]
- capabilities: [CAP_SYS_ADMIN]
Do you grant the above permissions? [y/N] y
latest: Pulling from vpps/heketi-volume-plugin
ed1d341f0b61: Download complete
Digest: sha256:20329252739bfb2cbc54645f8a5dae5da1f16d1242a189c2f1c974072d7eb871
Status: Downloaded newer image for vpps/heketi-volume-plugin:latest
Installed plugin vpps/heketi-volume-plugin:latest
$ docker volume create -d vpps/heketi-volume-plugin myvol1
myvol1
# or with set the desired volume size. For example volume size 4Gb
$ docker volume create -d vpps/heketi-volume-plugin -o size=4 myvol2
myvol2
$ docker run -it --name devtest -v myvol2:/app nginx:latest
-o clusters=string
Optional: Comma separated list of cluster ids where this volume must be allocated. If ommitted, Heketi will allocate the volume on any of the configured clusters which have the available space. Providing a set of clusters will ensure Heketi allocates storage for this volume only in the clusters specified.
-o disperse-data=int
Optional: Dispersion value for durability type 'disperse'.
Default is 4 (default 4)
-o durability=string
Optional: Durability type. Values are: none: No durability. Distributed volume only.
replicate: (Default) Distributed-Replica volume.
disperse: Distributed-Erasure Coded volume. (default "replicate")
-o replica=int
Replica value for durability type 'replicate'.
Default is 3 (default 3)
-o redundancy=int
Optional: Redundancy value for durability type 'disperse'.
Default is 2 (default 2)
-o size=int
Size of volume in GiB (default 1)
-o snapshot-factor=float
Optional: Amount of storage to allocate for snapshot support.
Must be greater 1.0. For example if a 10TiB volume requires 5TiB of snapshot storage, then snapshot-factor would be set to 1.5. If the value is set to 1, then snapshots will not be enabled for this volume (default 1)
Note: The volume size created depends upon the underlying brick size. For example, for a 2 way/3 way replica volume, the minimum volume size is 1GiB as the underlying minimum brick size is constrained to 1GiB. So, it is not possible create a volume of size less than 1GiB.
This plugin source code is licensed to you under your choice of the GNU Lesser General Public License, version 3 or any later version (LGPLv3 or later), as published by the Free Software Foundation, or under the Apache License, Version 2.0 <LICENSE-APACHE2 or http://www.apache.org/licenses/LICENSE-2.0>.
For GlusterFS at least 3 VMs are required with the following specs:
It is necessary to provide local name resolution for the 3 VMs via a common /etc/hosts file. For example (feel free to adjust the domain and host names to your environment):
# cat /etc/hosts
127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4
::1 localhost localhost.localdomain localhost6 localhost6.localdomain6
192.168.254.1 gfs1.lab gfs1
192.168.254.2 gfs2.lab gfs2
192.168.254.3 gfs3.lab gfs3
# yum update -y
# yum install -y epel-release
# yum install -y centos-release-gluster
# yum install glusterfs glusterfs-cli glusterfs-libs glusterfs-server
A couple of basic TCP ports need to be opened for GlusterFS peers to communicate and provide storage. Lets consider, that our VMs in a trusted network, so we change network interface to be in 'trusted' firewall zone (on all 3 GlusterFS VMs):
# firewall-cmd --zone=trusted --change-zone=eth0 --permanent
# firewall-cmd --reload
Now we are ready to start the GlusterFS daemon:
# systemctl start glusterd
# systemctl enable glusterd
That’s it. GlusterFS is up and running. The rest of the configuration will be done via heketi.
[root@gfs1 ~]# yum -y install heketi heketi-client
Heketi will use SSH to configure GlusterFS on all nodes. Create an SSH key pair and copy the public key to all 3 nodes (including the first node you are logged on):
[root@gfs1 ~]# ssh-keygen -f /etc/heketi/heketi_key -t rsa -N ''
[root@gfs1 ~]# ssh-copy-id -i /etc/heketi/heketi_key.pub [email protected]
[root@gfs1 ~]# ssh-copy-id -i /etc/heketi/heketi_key.pub [email protected]
[root@gfs1 ~]# ssh-copy-id -i /etc/heketi/heketi_key.pub [email protected]
[root@gfs1 ~]# chown heketi:heketi /etc/heketi/heketi_key*
Edit the /etc/heketi/heketi.json to configure heketi to use SSH:
[root@gfs1 ~]# vi /etc/heketi/heketi.json
{
"_port_comment": "Heketi Server Port Number",
"port": "8080",
"_use_auth": "Enable JWT authorization. Please enable for deployment",
"use_auth": false,
"_jwt": "Private keys for access",
"jwt": {
"_admin": "Admin has access to all APIs",
"admin": {
"key": "My Secret"
},
"_user": "User only has access to /volumes endpoint",
"user": {
"key": "My Secret"
}
},
"_glusterfs_comment": "GlusterFS Configuration",
"glusterfs": {
"_executor_comment": [
"Execute plugin. Possible choices: mock, ssh",
"mock: This setting is used for testing and development.",
" It will not send commands to any node.",
"ssh: This setting will notify Heketi to ssh to the nodes.",
" It will need the values in sshexec to be configured.",
"kubernetes: Communicate with GlusterFS containers over",
" Kubernetes exec api."
],
"executor": "ssh",
"_sshexec_comment": "SSH username and private key file information",
"sshexec": {
"keyfile": "/etc/heketi/heketi_key",
"user": "root",
"port": "22",
"fstab": "/etc/fstab"
},
"_kubeexec_comment": "Kubernetes configuration",
"kubeexec": {
"host" :"https://kubernetes.host:8443",
"cert" : "/path/to/crt.file",
"insecure": false,
"user": "kubernetes username",
"password": "password for kubernetes user",
"namespace": "OpenShift project or Kubernetes namespace",
"fstab": "Optional: Specify fstab file on node. Default is /etc/fstab"
},
"_db_comment": "Database file name",
"db": "/var/lib/heketi/heketi.db",
"_loglevel_comment": [
"Set log level. Choices are:",
" none, critical, error, warning, info, debug",
"Default is warning"
],
"loglevel" : "debug"
}
}
Now restart Heketi:
# systemctl restart heketi
# systemctl enable heketi
Test if it’s running:
# curl http://gfs1.lab:8080/hello
Hello from Heketi
We will use it to configure our GlusterFS storage pool. The software is already running on all our VMs but it’s unconfigured. To change that to a functional storage system we will describe our desired GlusterFS storage pool in a topology file, like below:
[root@gfs1 ~]# vi /etc/heketi/topology.json
{
"clusters": [
{
"nodes": [
{
"node": {
"hostnames": {
"manage": [
"gfs1.lab"
],
"storage": [
"192.168.254.1"
]
},
"zone": 1
},
"devices": [
"/dev/sdb"
]
},
{
"node": {
"hostnames": {
"manage": [
"gfs2.lab"
],
"storage": [
"192.168.254.2"
]
},
"zone": 1
},
"devices": [
"/dev/sdb"
]
},
{
"node": {
"hostnames": {
"manage": [
"gfs3.lab"
],
"storage": [
"192.168.254.3"
]
},
"zone": 1
},
"devices": [
"/dev/sdb"
]
}
]
}
]
}
Despite the formatting the file is relatively simple. It basically tells heketi to create a 3 node cluster with each node being known by a FQDN, an IP address and with at least one spare block device which will be used as a GlusterFS brick.
Now feed this file to heketi:
# export HEKETI_CLI_SERVER=http://gfs1.lab:8080
# heketi-cli topology load --json=/etc/heketi/topology.json
Creating cluster ... ID: 78cdb57aa362f5284bc95b2549bc7e7d
Creating node gfs1.lab ... ID: ffd7671c0083d88aeda9fd1cb40b339b
Adding device /dev/sdb ... OK
Creating node gfs2.lab ... ID: 8220975c0a4479792e684584153050a9
Adding device /dev/sdb ... OK
Creating node gfs3.lab ... ID: b94f14c4dbd8850f6ac589ac3b39cc8e
Adding device /dev/sdb ... OK
Now heketi has configured a 3 node GlusterFS storage pool. You can see that the 3 VMs have successfully formed what’s called a Trusted Storage Pool in GlusterFS:
[root@gfs1 ~]# gluster peer status
Number of Peers: 2
Hostname: gfs2.lab
Uuid: 93b34946-9571-46a8-983c-c9f128557c0e
State: Peer in Cluster (Connected)
Other names:
gfs2.lab
Hostname: 192.168.254.3
Uuid: e3c1f9b0-be97-42e5-beda-f70fc05f47ea
State: Peer in Cluster (Connected)
Now you can install docker volume plugin for Heketi + GlusterFS on docker host and use your new Gluster Cluster.
Content type
Plugin
Digest
Size
84.1 MB
Last updated
over 7 years ago
docker plugin install vpps/heketi-volume-plugin