The aws-s3-integrity-check bash tool is an open-source tool that allows verifying the integrity of the totality of files contained within an Amazon S3 bucket in a quick, reliable and efficient manner. More info here.
Sonia García-Ruiz,Regina Hertfelder Reynolds,Melissa Grant-Peters,Emil Karl Gustavsson,Aine Fairbrother-Browne,Zhongbo Chen,Jonathan William Brenton,Mina Ryten,aws-s3-integrity-check: an open-source bash tool to verify the integrity of a dataset stored on Amazon S3,Gigabyte,2023 https://doi.org/10.46471/gigabyte.87
-v <local_folder>:<local_folder>. Required argument. It requires replacing the arguments <local_folder>:<local_folder> for a real path to a local folder containing the files to be tested. It is used to mount the local folder as a local volume to the Docker image to allow the Docker image to have read access to the files to test.
-v ($pwd)/logs/:/usr/src/logs. Required argument. This argument should not be changed and sent to the Docker image as it is shown. It represents the path to the local log folder. It is used to mount the local log folder as a local volume to the Docker image. It allows the Docker image to record the outputs produced during the tool execution within the local “logs/” folder.
-v ${HOME}/.aws/credentials:/root/.aws/credentials:ro. Required argument. This argument should not be changed and sent to the Docker image as it is shown. It represents the path to the folder containing the information about the user authentication on Amazon S3. This parameter is used to mount the local AWS credential directory as a read-only volume to the Docker image. It allows the Docker image to have access to the authentication information of the user on AWS.
IMPORTANT: volumes (-v) need to be specified before the Docker image name:tag.
-l|--local <local_folder> [required] Path to a local folder containing the original version of the files uploaded to Amazon S3. Example: -l /data/nucCyt/raw_data/
-b|--bucket <bucket_name> [required] Amazon S3 bucket containing the files uploaded from the local folder '-l <local_folder>'. Example: -b nuccyt
-p|--profile <aws_profile> [optional] AWS profile in case the user has logged in using the command aws configure sso. Example: -p my_aws_profile