Sign inSign up

soniaruiz/aws-s3-integrity-check

By soniaruiz

•Updated over 3 years ago

An open-source dockerized bash tool to verify the integrity of a dataset stored on Amazon S3.

Image
API management
Web servers
Databases & storage
0

1.3K

soniaruiz/aws-s3-integrity-check repository overview

⁠Description

The aws-s3-integrity-check bash tool is an open-source tool that allows verifying the integrity of the totality of files contained within an Amazon S3 bucket in a quick, reliable and efficient manner. More info here⁠.

⁠Publication

Sonia García-Ruiz,Regina Hertfelder Reynolds,Melissa Grant-Peters,Emil Karl Gustavsson,Aine Fairbrother-Browne,Zhongbo Chen,Jonathan William Brenton,Mina Ryten,aws-s3-integrity-check: an open-source bash tool to verify the integrity of a dataset stored on Amazon S3,Gigabyte,2023 https://doi.org/10.46471/gigabyte.87⁠

⁠Pre-requisites

⁠Usage

⁠1. Volumes
  • -v <local_folder>:<local_folder>. Required argument. It requires replacing the arguments <local_folder>:<local_folder> for a real path to a local folder containing the files to be tested. It is used to mount the local folder as a local volume to the Docker image to allow the Docker image to have read access to the files to test.
  • -v ($pwd)/logs/:/usr/src/logs. Required argument. This argument should not be changed and sent to the Docker image as it is shown. It represents the path to the local log folder. It is used to mount the local log folder as a local volume to the Docker image. It allows the Docker image to record the outputs produced during the tool execution within the local “logs/” folder.
  • -v ${HOME}/.aws/credentials:/root/.aws/credentials:ro. Required argument. This argument should not be changed and sent to the Docker image as it is shown. It represents the path to the folder containing the information about the user authentication on Amazon S3. This parameter is used to mount the local AWS credential directory as a read-only volume to the Docker image. It allows the Docker image to have access to the authentication information of the user on AWS.

IMPORTANT: volumes (-v) need to be specified before the Docker image name:tag.

⁠2. Parameters

Options:

  • -l|--local <local_folder> [required] Path to a local folder containing the original version of the files uploaded to Amazon S3. Example: -l /data/nucCyt/raw_data/
  • -b|--bucket <bucket_name> [required] Amazon S3 bucket containing the files uploaded from the local folder '-l <local_folder>'. Example: -b nuccyt
  • -p|--profile <aws_profile> [optional] AWS profile in case the user has logged in using the command aws configure sso. Example: -p my_aws_profile
  • -h|--help [optional] Shows further help options

⁠Example

Considering the local folder "/data/nucCyt/" containing a dataset uploaded to the Amazon S3 bucket "nuccyt":

docker run -v /data/nucCyt:/data/nucCyt -v "$PWD/logs:/usr/src/logs" -v ${HOME}/.aws/credentials:/root/.aws/credentials:ro soniaruiz/aws-s3-integrity-check:latest -l /data/nucCyt/ -b nuccyt

Tag summary

Content type

Image

Digest

sha256:960520884…

Size

221.6 MB

Last updated

over 3 years ago

docker pull soniaruiz/aws-s3-integrity-check