iSCSI-targetd provisioner is an out of tree provisioner for iSCSI storage for Kubernetes and OpenShift. The provisioniner uses the API provided by targetd to create and export iSCSI storage on a remote server.
iSCSI-targetd provisioner has the following prerequisistes:
targetdWhen a PVC request is issued for an iscsi provisioner controlled storage class the following happens:
Each storage class is tied to an iSCSI target and a volume group. Because a target can manage a maximum of 255 LUNs, each storage class manages at most 255 pvs. iSCSI-targetd provisioner can manage multiple storage classes.
These instructions should work for RHEL/CentOS 7+ and Fedora 24+.
On Fedora 24, current updates to the SELinux policy do not work with targetd. There is a bug filed: https://bugzilla.redhat.com/show_bug.cgi?id=1451139 Until this bug is resolve, SELinux must be set to permissive mode on Fedora 25+.
For RHEL and Centos make sure you install targetd >= 0.8.6-1 as in previous versions there a bug that prevented exposing a volume to more than one initiator
targetd version 0.8.12 brought ZFS support.
In various places, iSCSI Qualified Names (IQNs) need to be created. These need to be unique. So every target must have it's own unique IQN, and every client (initiator) must have its own IQN.
IF NON-UNIQUE IQNs ARE USED, THEN THERE IS A POTENTIAL FOR DATA LOSS AND BAD PERFORMANCE!
IQNs have a specific format:
iqn.YEAR-MM.com.example.blah:tag
See the wikipedia article for more information.
Before configuring the iSCSI server, it needs to have storage
configured. targetd uses LVM or ZFS to provision storage.
If possible, it's best to have a dedicated disk or partition that can be configured as a volume group or ZFS pool. However, if this is not possible, a loopback device can be used to simulate a dedicated block device.
Only targetd needs to be installed. However, it's highly recommended
to also install targetcli as it provides a simple user interface for
looking at the state of the iSCSI system.
sudo yum install -y targetcli targetd
Enable and start target.service. This will ensure that iSCSI
configuration persists through reboot.
sudo systemctl enable target
sudo systemctl start target
First, edit /etc/target/targetd.yaml. A working sample
configuration is provided below:
password: ciao
user: admin
ssl: false
target_name: iqn.2003-01.org.linux-iscsi.minishift:targetd
# Examples below; uncomment and edit
# LVM
# if using a thin pool, use <volume group name>/<thin pool name>
# e.g vg-targetd/pool
pool_name: vg-targetd
# ZFS
zfs_block_pools: [ "tank/targetd", ]
Next, enable and start targetd.service.
sudo systemctl enable targetd
sudo systemctl start targetd
The default configuration requires that port 3260/tcp, 3260/udp and 18700/tcp be open on the iSCSI server.
If using firewalld,
firewall-cmd --add-service=iscsi-target --permanent
firewall-cmd --add-port=18700/tcp --permanent
firewall-cmd --reload
Otherwise, add the following iptables rules to /etc/sysconfig/iptables
TODO
The iscsiadm command is required for all clients. This is provided
by the iscsi-initiator-utils package and should be part of the
standard RHEL, CentOS or Fedora installation.
sudo yum install -y iscsi-initiator-utils
Each node requires a unique initiator name. USE OF DUPLICATE NAMES MAY CAUSE PERFORMANCE ISSUES AND DATA LOSS.
By default, a random initiator name is generated when the
iscsi-initiator-utils package is installed. This usually unique
enough, but is not guaranteed. It's also not very descriptive.
To set a custom initiator name, edit the file /etc/iscsi/initiatorname.iscsi:
InitiatorName=iqn.2017-04.com.example:node1
In the above example, the initiator name is set to
iqn.2017-04.com.example:node1.
After changing the initiator name, restart iscsid.service.
sudo systemctl restart iscsid
Run the following commands. The secret correspond to username and password you have chosen for targetd (admin is the default for the username).
This set of command will install iSCSI-targetd provisioner in the default namespace.
iscsi-provisioner-d.yaml includes a hardcoded namespace, it is set to default.
export NS=default
kubectl create secret generic targetd-account --from-literal=username=admin --from-literal=password=ciao -n $NS
kubectl apply -f https://raw.githubusercontent.com/gaima8/external-storage_iscsi_targetd/master/kubernetes/iscsi-provisioner-d.yaml -n $NS
kubectl apply -f https://raw.githubusercontent.com/gaima8/external-storage_iscsi_targetd/master/kubernetes/iscsi-provisioner-pvc.yaml -n $NS
Run the following commands. The secret correspond to username and password you have chosen for targetd (admin is the default for the username)
oc new-project iscsi-provisioner
oc create sa iscsi-provisioner
oc adm policy add-cluster-role-to-user cluster-reader system:serviceaccount:iscsi-provisioner:iscsi-provisioner
# if Openshift is version < 3.6 add the iscsi-provisioner-runner role
oc create -f https://raw.githubusercontent.com/gaima8/external-storage_iscsi_targetd/master/kubernetes/openshift/iscsi-auth.yaml
# else if Openshift is version >= 3.6 add the system:persistent-volume-provisioner role
oc adm policy add-cluster-role-to-user system:persistent-volume-provisioner system:serviceaccount:iscsi-provisioner:iscsi-provisioner
#
oc secret new-basicauth targetd-account --username=admin --password=ciao
oc create -f https://raw.githubusercontent.com/gaima8/external-storage_iscsi_targetd/master/kubernetes/openshift/iscsi-provisioner-dc.yaml
Alternatively, you can start a provisioner as a container locally.
docker run -ti -v /root/.kube:/kube -v /var/run/kubernetes:/var/run/kubernetes --privileged --net=host quay.io/external_storage/iscsi-controller:latest start --kubeconfig=/kube/config --master=http://127.0.0.1:8080 --log-level=debug --targetd-address=192.168.99.100 --targetd-password=ciao --targetd-username=admin
kubectl create -f https://raw.githubusercontent.com/gaima8/external-storage_iscsi_targetd/master/kubernetes/iscsi-provisioner-class.yaml
oc create -f https://raw.githubusercontent.com/gaima8/external-storage_iscsi_targetd/master/kubernetes/openshift/iscsi-provisioner-class.yaml
Deploy a pod that uses the PVC
kubectl create -f https://github.com/gaima8/external-storage_iscsi_targetd/blob/master/kubernetes/iscsi-test-pod.yaml -n $NS
Create a PVC
oc create -f https://raw.githubusercontent.com/kubernetes-incubator/external-storage/master/iscsi/targetd/openshift/iscsi-provisioner-pvc.yaml
Verify that the pv has been created
oc get pv
Deploy a pod that uses the PVC
oc create -f https://raw.githubusercontent.com/kubernetes-incubator/external-storage/master/iscsi/targetd/openshift/iscsi-test-pod.yaml
You may also want to verify that the volume has been created in you volume group
targetcli ls
If you have installed OpenShift using the ansible installer you can use a set of playbook to automate the above instructions. You can find more documentation on these playbooks here before running the playbooks you need to annotate the inventory file with some additional variables and the nodes with the iscsi inititator name that you want to be created. Here is a summary of the variables:
| Variable Name | Description |
|---|---|
| targetd_lvm_volume_group | the volume group to be created |
| targetd_lvm_physical_volumes | comma separated list of devices to add to the volume group |
| targetd_password | the password used to authenticate the connection to targetd, you may want to not store this on your inventory file, you can pass this as {{ lookup('env','TARGETD_PASSWORD') }} |
| targetd_user | the username used to authenticate the connection to targetd, you may want to not store this on your inventory file, you can pass this as {{ lookup('env','TARGETD_USERNAME') }} |
| targetd_iscsi_target | the name of the target to be created in the target server |
| iscsi_provisioner_pullspec | the location of the iSCSI-targetd provisioner image |
| iscsi_provisioner_default_storage_class | whether the created storage class should be the default class |
| iscsi_provisioner_portals | optional, comma separated list of alternative IP:port where the iscsi server can be found, specifying this parameters trigger the usage of multipath |
| chap_auth_discovery | true/false whether to use chap authentication for discovery operations |
| discovery_sendtargets_auth_username | initiator username |
| discovery_sendtargets_auth_password | initiator password, you can pass this as {{ lookup('env','SENDTARGET_PASSWORD') }} |
| discovery_sendtargets_auth_username_in | target username |
| discovery_sendtargets_auth_password_in | target password, you can pass this as {{ lookup('env','SENDTARGET_PASSWORD_IN') }} |
| chap_auth_session | true/false whether to use chap authentication for session operations |
| session_auth_username | initiator username |
| session_auth_password | initiator password, you can pass this as {{ lookup('env','SESSION_PASSWORD') }} |
| session_auth_username_in | target username |
| session_auth_password_in | target password, you can pass this as {{ lookup('env','SESSION_PASSWORD_IN') }} |
All the nodes should have a label with their defining the initiator name for that node, here is an example:
ose-node1.cscc openshift_node_labels="{'region': 'primary', 'zone': 'default'}" iscsi_initiator_name=iqn.2003-03.net.deadvax:ose-node1
ose-node2.cscc openshift_node_labels="{'region': 'primary', 'zone': 'default'}" iscsi_initiator_name=iqn.2003-03.net.deadvax:ose-node2
see also the individual roles documentation
To install iSCSI provisioner using ansible, run the following
ansible-playbook -i <your inventory file> ansible/playbook/all.yaml
If you enable iSCSI CHAP-based authentication, the ansible installer will set the target configuration consinstently and also configure the storage class.
However at provisioning time the provisioner will not setup the chap secret. Having the permissions to setup a secret in any namespace would make the provisioner too powerful and insecure.
So, it is up to the project administrator to setup the secret.
The name of the expected secret name will be <provisioner-name>-chap-secret
An example of the secret format can be found here
Content type
Image
Digest
Size
87.3 MB
Last updated
almost 6 years ago
docker pull gaima8/external_storage