Public | Automated Build

Last pushed: 23 days ago
Short Description
Ethereum network fork monitoring tool
Full Description

README.md

The data storage

The clients are configured to use an nfs-share for storing blockchain data.

This is implemented via a Premium Storage account on azure, a 1TB disk backed by SSD. A coreos is used to mount and share the disk via NFS.

Share server

Configuration of the CoreOS NFS share:

The nfs.mount file which defines the mount - correspond to fstab on a normal linux

kube@storagesrv /etc/systemd/system $ cat nfs.mount 
[Unit]
Description=Data storage for blockchain data 

[Mount]
What=/dev/sdc1
Where=/nfs
Type=ext4
Options=defaults,auto,noatime,noexec

[Install]
WantedBy = multi-user.target

After defining mounts, you need to enable them

kube@storagesrv /etc/systemd/system $ sudo systemctl enable --now nfs.mount
Created symlink /etc/systemd/system/multi-user.target.wants/nfs.mount → /etc/systemd/system/nfs.mount.

You define the things to export via /etc/exports.

kube@storagesrv /data $ cat /etc/exports 

/nfs 10.0.0.0/8(rw,async,no_subtree_check,no_root_squash,fsid=0)

After modifying /etc/exports, it needs to be applied

kube@storagesrv /etc $ sudo systemctl restart nfs-mountd

Clients

The client(s) were missing statd, which was needed in order to use the nfs share. This was solved by manually installing the statd daemon onto the host systems.

This was done against each ethereum-node-X host:

ssh kube@52.169.85.15
kube@ethereumeu-master:~$ ssh kube@ethereumeu-node-0
kube@ethereumeu-node-0:~$ sudo apt-get install nfs-common
kube@ethereumeu-node-0:~$ sudo service rpcbind start

Fixing and mounting the disk

This info is included here as a potential help in the future, in case we have problems with disks or mounts on the server, or want to add new disks. We had some problems with the large disk we were mounting, and below are the steps taken to fix them.

Trying to mount:

kube@storagesrv /etc/systemd $ sudo systemctl restart nfs.mount
Job for nfs.mount failed.
See "systemctl status nfs.mount" and "journalctl -xe" for details.

The journalctl is really helpful

kube@storagesrv /etc/systemd $ sudo journalctl -xe
[...]    
Dec 07 09:16:26 storagesrv systemd[1]: Mounting Data storage for blockchain data...
-- Subject: Unit nfs.mount has begun start-up
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
-- 
-- Unit nfs.mount has begun starting up.
Dec 07 09:16:26 storagesrv kernel: EXT4-fs (sdc): VFS: Can't find ext4 filesystem
Dec 07 09:16:26 storagesrv mount[26350]: mount: wrong fs type, bad option, bad superblock on /dev/sdc,
Dec 07 09:16:26 storagesrv mount[26350]:        missing codepage or helper program, or other error
Dec 07 09:16:26 storagesrv mount[26350]:        In some cases useful info is found in syslog - try
Dec 07 09:16:26 storagesrv mount[26350]:        dmesg | tail or so.
Dec 07 09:16:26 storagesrv systemd[1]: nfs.mount: Mount process exited, code=exited status=32
Dec 07 09:16:26 storagesrv sudo[26347]: pam_unix(sudo:session): session closed for user root

Indeed, there is no partition on /dev/sdc:

kube@storagesrv /etc/systemd $ sudo fdisk -l /dev/sdc
Disk /dev/sdc: 999 GiB, 1072668082176 bytes, 2095054848 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes

Disable the old mount

kube@storagesrv /etc/systemd $ sudo systemctl stop data.mount
kube@storagesrv /etc/systemd $ sudo systemctl status data.mount
● data.mount - Data storage for blockchain data
   Loaded: loaded (/etc/systemd/system/data.mount; disabled; vendor preset: disabled)
   Active: inactive (dead)
    Where: /data
     What: /dev/sdb1

Dec 06 18:02:07 storagesrv umount[25459]:          use the device is found by lsof(8) or fuser(1).)
Dec 06 18:02:07 storagesrv systemd[1]: data.mount: Mount process exited, code=exited status=32
Dec 06 18:02:07 storagesrv systemd[1]: Failed unmounting Data storage for blockchain data.
Dec 07 09:15:55 storagesrv systemd[1]: Unmounting Data storage for blockchain data...
Dec 07 09:15:55 storagesrv systemd[1]: Unmounted Data storage for blockchain data.
Dec 07 09:15:55 storagesrv systemd[1]: data.mount: Unit entered failed state.
Dec 07 09:15:55 storagesrv systemd[1]: Mounting Data storage for blockchain data...
Dec 07 09:15:55 storagesrv systemd[1]: Mounted Data storage for blockchain data.
Dec 07 09:20:09 storagesrv systemd[1]: Unmounting Data storage for blockchain data...
Dec 07 09:20:09 storagesrv systemd[1]: Unmounted Data storage for blockchain data.

Create partition on /dev/sdc

kube@storagesrv /etc/systemd $ sudo fdisk -l /dev/sdc
Disk /dev/sdc: 999 GiB, 1072668082176 bytes, 2095054848 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
kube@storagesrv /etc/systemd $ sudo fdisk /dev/sdc

Welcome to fdisk (util-linux 2.27.1).
Changes will remain in memory only, until you decide to write them.
Be careful before using the write command.

Device does not contain a recognized partition table.
Created a new DOS disklabel with disk identifier 0x0b36e43d.

Command (m for help): p
Disk /dev/sdc: 999 GiB, 1072668082176 bytes, 2095054848 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disklabel type: dos
Disk identifier: 0x0b36e43d

Command (m for help): n
Partition type
   p   primary (0 primary, 0 extended, 4 free)
   e   extended (container for logical partitions)
Select (default p): p
Partition number (1-4, default 1): 
First sector (2048-2095054847, default 2048): 
Last sector, +sectors or +size{K,M,G,T,P} (2048-2095054847, default 2095054847): 

Created a new partition 1 of type 'Linux' and of size 999 GiB.

Command (m for help): p
Disk /dev/sdc: 999 GiB, 1072668082176 bytes, 2095054848 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disklabel type: dos
Disk identifier: 0x0b36e43d

Device     Boot Start        End    Sectors  Size Id Type
/dev/sdc1        2048 2095054847 2095052800  999G 83 Linux

Command (m for help): w
The partition table has been altered.
Calling ioctl() to re-read partition table.
Syncing disks.

And format

kube@storagesrv /etc/systemd $ sudo mkfs.ext4 /dev/sdc1
mke2fs 1.42.13 (17-May-2015)
Discarding device blocks: done                            
Creating filesystem with 261881600 4k blocks and 65470464 inodes
Filesystem UUID: dc2f30d1-3bc1-43be-a42c-292e9491695c
Superblock backups stored on blocks: 
    32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208, 
    4096000, 7962624, 11239424, 20480000, 23887872, 71663616, 78675968, 
    102400000, 214990848

Allocating group tables: done                            
Writing inode tables: done                            
Creating journal (32768 blocks): done
Writing superblocks and filesystem accounting information: done 

Trying again yields a different error:

kube@storagesrv /etc/systemd $ sudo journalctl -xe
[...]
Dec 07 09:32:09 storagesrv systemd-udevd[26554]: Process '/usr/lib/udev/scripts/iscsidev.sh' failed with exit code 1.
Dec 07 09:32:09 storagesrv systemd[1]: Mounting Data storage for blockchain data...
-- Subject: Unit nfs.mount has begun start-up
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
-- 
-- Unit nfs.mount has begun starting up.
Dec 07 09:32:09 storagesrv kernel: EXT4-fs (sdc1): Unrecognized mount option "noexec 0 0" or missing value
Dec 07 09:32:09 storagesrv mount[26559]: mount: wrong fs type, bad option, bad superblock on /dev/sdc1,
Dec 07 09:32:09 storagesrv mount[26559]:        missing codepage or helper program, or other error
Dec 07 09:32:09 storagesrv mount[26559]:        In some cases useful info is found in syslog - try
Dec 07 09:32:09 storagesrv mount[26559]:        dmesg | tail or so.
Dec 07 09:32:09 storagesrv systemd[1]: nfs.mount: Mount process exited, code=exited status=32
Dec 07 09:32:09 storagesrv systemd[1]: Failed to mount Data storage for blockchain data.
-- Subject: Unit nfs.mount has failed
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
-- 
-- Unit nfs.mount has failed.
-- 
-- The result is failed.
Dec 07 09:32:09 storagesrv systemd[1]: nfs.mount: Unit entered failed state.
Dec 07 09:32:12 storagesrv sudo[26574]:     kube : TTY=pts/0 ; PWD=/etc/systemd ; USER=root ; COMMAND=/bin/journalctl -xe
Dec 07 09:32:12 storagesrv sudo[26574]: pam_unix(sudo:session): session opened for user root by kube(uid=0)
Dec 07 09:32:12 storagesrv sudo[26574]: pam_systemd(sudo:session): Cannot create session: Already running in a session

Seems ext4 does not like the 0 0 mount options:

kube@storagesrv /etc/systemd/system $ cat nfs.mount 
[Unit]
Description=Data storage for blockchain data 

[Mount]
What=/dev/sdc1
Where=/nfs
Type=ext4
Options=defaults,auto,noatime,noexec 0 0

[Install]
WantedBy = multi-user.target

Removed:

kube@storagesrv /etc/systemd/system $ cat nfs.mount 
[Unit]
Description=Data storage for blockchain data 

[Mount]
What=/dev/sdc1
Where=/nfs
Type=ext4
Options=defaults,auto,noatime,noexec

[Install]
WantedBy = multi-user.target

Remount

kube@storagesrv /etc/systemd/system $ sudo systemctl restart nfs.mount

kube@storagesrv /etc/systemd/system $ sudo systemctl status nfs.mount
● nfs.mount - Data storage for blockchain data
   Loaded: loaded (/etc/systemd/system/nfs.mount; enabled; vendor preset: disabled)
   Active: active (mounted) since Wed 2016-12-07 09:51:53 UTC; 21s ago
    Where: /nfs
     What: /dev/sdc1
  Process: 26652 ExecMount=/bin/mount /dev/sdc1 /nfs -t ext4 -o defaults,noatime,noexec (code=exited, status=0/SUCCESS)
    Tasks: 0
   Memory: 396.0K
      CPU: 29ms
   CGroup: /system.slice/nfs.mount

Dec 07 09:51:53 storagesrv systemd[1]: Mounting Data storage for blockchain data...
Dec 07 09:51:53 storagesrv systemd[1]: Mounted Data storage for blockchain data.

Exporting share

Modify exports

kube@storagesrv /etc $ cat exports 

/nfs 10.0.0.0/8(rw,async,no_subtree_check,no_root_squash,fsid=0)

Reload nfs-mountd

kube@storagesrv /etc $  sudo systemctl restart nfs-mountd
kube@storagesrv /etc $ sudo systemctl status nfs-mountd
● nfs-mountd.service - NFS Mount Daemon
   Loaded: loaded (/usr/lib/systemd/system/nfs-mountd.service; static; vendor preset: disabled)
   Active: active (running) since Wed 2016-12-07 10:00:17 UTC; 2s ago
  Process: 26717 ExecStart=/usr/sbin/rpc.mountd $RPCMOUNTDARGS (code=exited, status=0/SUCCESS)
 Main PID: 26722 (rpc.mountd)
    Tasks: 1
   Memory: 876.0K
      CPU: 11ms
   CGroup: /system.slice/nfs-mountd.service
           └─26722 /usr/sbin/rpc.mountd

Dec 07 10:00:17 storagesrv systemd[1]: Stopped NFS Mount Daemon.
Dec 07 10:00:17 storagesrv systemd[1]: Starting NFS Mount Daemon...
Dec 07 10:00:17 storagesrv systemd[1]: Started NFS Mount Daemon.
Dec 07 10:00:17 storagesrv rpc.mountd[26722]: Version 1.3.2 starting
Docker Pull Command
Owner
arachnid
Source Repository