hdfgroup/hsds repository overview

⁠HSDS (Highly Scalable Data Service) - REST-based service for HDF5 data using object storage

⁠Introduction

HSDS is a web service that implements a REST-based web service for HDF5 data stores. HSDS can run with Docker, Kubernetes, or directly on the host. HSDS storage options include AWS S3 (or any S3 API compatible offerings), Azure Blob Storate, or Posix file systems More info at: https://www.hdfgroup.org/solutions/highly-scalable-data-service-hsds/⁠.

⁠Websites

• Main website: https://www.hdfgroup.org/solutions/highly-scalable-data-service-hsds/⁠
• Source code: https://github.com/HDFGroup/hsds⁠
• Forum: forum.hdfgroup.org/hsds https://forum.hdfgroup.org/c/hsds⁠
• Documentation: http://h5serv.readthedocs.org⁠ (For REST API)
• Python client - h5pyd: https://github.com/HDFGroup/h5pyd⁠

⁠Other useful resources

• RESTful HDF5 White Paper: https://www.hdfgroup.org/pubs/papers/RESTful_HDF5.pdf⁠
• SciPy17 Presentation: http://s3.amazonaws.com/hdfgroup/docs/hdf_data_services_scipy2017.pdf⁠
• HDF5 For the Web: https://hdfgroup.org/wp/2015/04/hdf5-for-the-web-hdf-server⁠
• HSDS Security: https://hdfgroup.org/wp/2015/12/serve-protect-web-security-hdf5⁠
• HSDS with Jupyter: https://www.slideshare.net/HDFEOS/hdf-kita-lab-jupyterlab-hdf-service⁠

⁠Quick Install

Note: this install uses http (rather than https) and default username/passwords. Not for use as a production service.

⁠Installation with Docker

Export environment variables as shown in "Sample .bashrc" below.

1. Install Docker and docker-compose if necessary
2. Create a bucket for HSDS, using aws cli tools or aws management console
3. Add the DNS for the service to the /etc/hosts file. E.g. 127.0.0.1 hsds.hdf.test (can use any valid DNS name) if you running containers directly on the host, or 192.168.99.100 hsds.hdf.test if using docker machine (use docker-machine ip to get the IP address)
4. Create environment variables as in "Sample .bashrc" below
5. Download the docker compose yaml file from: https://github.com/HDFGroup/hsds/blob/master/docker-compose.yml⁠
6. Start the service docker-compose up -d --scale sn=${CORES} --scale dn=${CORES} where CORES is the number of containers desired (defaults to 1)
7. Run $ docker ps and verify that the containers are running: hsds_head, hsds_async, hsds_sn_[1-n], hsds_dn_[1-n]
8. Go to http://hsds.hdf.test/about⁠ and verify that "cluster_state" is "READY" (might need to give it a minute or two)
9. Install Anaconda: https://conda.io/docs/user-guide/install/linux.html⁠ (install for python 3.6)
10. Install h5pyd: $ pip install h5pyd
11. Run: $ hsconfigure. Set hs endpoint with DNS name (e.g. http://hsds.hdf.test⁠) and admin username/password (admin/admin). Ignore API Key.
12. Run: $ hsinfo. Server state should be "READY". Ignore the "Not Found" error for the admin home folder
13. Create "/home" folder: $ hstouch /home/. Note: trailing slash is important!
14. Create a top-level domain for the "test_user1:" $ hstouch -o test_user1 /home/test_user1/
15. Run the integration test: $ python testall.py --skip_unit
16. The test suite will emit some warnings due to test domains not being loaded. To address see test_data_setup below.

⁠Test Data Setup

Using the following procedure to import test files into hsds

1. Install h5py: $ pip install h5py
2. Install h5pyd (Python client SDK): $ pip install h5pyd
3. Download the following file: $ wget https://s3.amazonaws.com/hdfgroup/data/hdf5test/tall.h5
4. Create a test folder on HSDS: $ hstouch -u test_user1 -p test /home/test_user1/test/
5. Import into hsds: $ hsload -v -u test_user1 -p test tall.h5 /home/test_user1/test/
6. Verify upload: `$ hsls -r -u test_user1 -p test /home/test_user1/test/tall.h5

⁠Sample .bashrc

These environment variables will be passed to the Docker containers on start up.

export AWS_ACCESS_KEY_ID=1234567890            # user your AWS account access key if using S3 (Not needed if running on EC2 and AWS_IAM_ROLE is defined)
export AWS_SECRET_ACCESS_KEY=ABCDEFGHIJKL      # use your AWS account access secret key if using S3  (Not needed if running on EC2 and AWS_IAM_ROLE is defined)
export BUCKET_NAME=hsds.test                   # set to the name of the bucket you will be using
export AWS_REGION=us-east-1                    # for boto compatibility - for S3 set to the region the bucket is in
export AWS_S3_GATEWAY=http://s3.amazonaws.com  # Use AWS endpoint for region where bucket is
export HSDS_ENDPOINT=http://hsds.hdf.test    # use https protocal if SSL is desired
# For S3, set AWS_S3_GATEWAY to endpoint for the region the bucket is in.  E.g.: http://s3.amazonaws.com.
# See http://docs.aws.amazon.com/general/latest/gr/rande.html#s3_region for list of endpoints.

⁠Writing Client Applications

As a REST service, clients be developed using almost any programming language. The test programs under: h5serv/test/integ illustrate some of the methods for performing different operations using Python.

The related project: https://github.com/HDFGroup/h5pyd⁠ provides a (mostly) h5py-compatible interface to the server for Python clients.

For C/C++ clients, the HDF REST VOL is a HDF5 library plugin that enables the HDF5 API to read and write data using HSDS. See: https://bitbucket.hdfgroup.org/users/jhenderson/repos/rest-vol/browse⁠.

⁠Uninstalling

HSDS only modifies the S3 bucket that it is configured to use, so to uninstall just remove source files, Docker images, and S3 bucket.

⁠Reporting bugs (and general feedback)

Create new issues at http://github.com/HDFGroup/hsds/issues⁠ for any problems you find.

For general questions/feedback, please use the HSDS forum: https://forum.hdfgroup.org/hsds⁠.

⁠License

HSDS is licensed under an APACHE 2.0 license. See LICENSE in this directory.

⁠Integration with JupyterHub

The HDF Group provides access to an HSDS instance that is integrated with JupyterLab: HDF Lab. HDF Lab is a hosted Jupyter environment with these features:

• Connection to a 16-node HSDS instance
• Dedicated Xeon core per user
• 10 GB Posix Disk
• 100 GB S3 storage for HDF data
• Sample programs and data files

Sign up for HDF Lab here: https://www.hdfgroup.org/hdfkitalab/⁠.