hydroshare/hsextract repository overview

HydroShare file metadata extraction library for S3.

The following environment variables are required and setup the configuration for accessing the S3 files to be extracted.

• AWS_ACCESS_KEY_ID
• AWS_SECRET_ACCESS_KEY
• AWS_S3_ENDPOINT

Output files are written to S3

Pass the following arguments to the hsextract entrypoint:

• S3 input path (bucket/path)
• S3 output path (bucket/path)

There are other parameters used for retrieving current metadata from hydroshare, using the defaults should be sufficient for now.

docker run -e AWS_ACCESS_KEY_ID="{access_key}" -e AWS_SECRET_ACCESS_KEY="{secret_key}" -e AWS_S3_ENDPOINT="https://s3.beta.hydroshare.org" hydroshare/hsextract:0.9.8 "{bucket_name}/{resource_id}/data/contents" "{bucket_name}/md/{resource_id}"

You can also write the data to a directory within the container. To do this setup the S3 output path to a local absolute directory within the container and pass the --local-output flag. Since the files are written to the container, you'll need to mount a volume from the host machine to this output path in order to persist the files. An example could be

docker run -v /local/path:/outputfiles -e AWS_ACCESS_KEY_ID="{access_key}" -e AWS_SECRET_ACCESS_KEY="{secret_key}" -e AWS_S3_ENDPOINT="https://s3.beta.hydroshare.org" hydroshare/hsextract:0.9.8 "{bucket_name}/{resource_id}/data/contents" /outputfiles --local-output