emccorp/atmos-s3-proxy repository overview

This proxy provides seamless s3 functionality while data is being migrated from ATMOS to ECS. The handling is determined by the state of the ATMOS subtenant and S3 bucket specified in the call. Those states are specified in S3 orchestration data outside of the migration, which must be made publicly available.

EXPECTED ORCHESTRATION DATA

The orchestration data requests are all prefixed with the base URL specified for the proxy, and must include the orchestration data namespace for proper resolution. All orchestration data must be made publicly available, or the proxy will not be able to read it. There are 3 subtenant suffixes. Any subtenant not included in any of these lists is assumed to be outside of the migration process.

-- 'subtenants-pre-migration' contains a line-separated list of all subtenants being prepared for migration.

-- 'subtenants-migrating' contains a line-separated list of all subtenants whose data is currently being migrated.

-- 'subtenants-post-migration' contains a line-separated list of all subtenants whose data has been completely migrated to the target.

For each subtenant, there are 2 additional suffixes to indicate the state of each bucket, and a 3rd additional suffix to indicate any user mappings for that subtenant. Any bucket not found in these lists is assumed to have been completely migrated to the target.

-- '<subtenant_id>/buckets-pre-migration' contains a line-separated list of all buckets that have not yet had any data migrated to the target.

-- '<subtenant_id>/buckets-migrating' contains a line-separated list of all buckets whose data is currently being migrated.

-- '<subtenant_id>/source-to-target-user-mapping' contains a line-separated list of users for that subtenant whose ids are being changed from source to target. The format of each line is "<source_id> <target_id>", where the 2 ids are separated by a space. Ids that are not changing do not need to be included in this mapping.

CONFIGURING AND RUNNING THE DOCKER CONTAINER

The docker container is run by executing the following command, or something equivalent. Configuration is controlled by adjusting the port, the log folder, and the environment variable values in s3_env.txt.

docker run --interactive --tty -p <your_port>:9020 --env-file s3_env.txt -v <your_log_folder>:/var/log/atmos-s3-proxy emccorp/atmos-s3-proxy:v1.0rc4

Here s3_env.txt should contain the following values:

source_location= # The complete base URL to the source, e.g., https://source_host_name:8080. source_is_atmos=True # Change to False if the source is not atmos. target_location= # The complete base URL to the target, e.g., https://target_host_name:9021. orchestration_daemon_sleep_seconds=30 # Time (in seconds) between checks for new orchestration data. orchestration_namespace=ecs-sync # The namespace used for retrieving orchestration data. orchestration_request_url= # The base URL for orchestration data, including the prefix, e.g., https://source_host_name:9021/prefix/. log_debug=false # Set this to true if you want to log extra debugging information during testing.

CONFIGURING AND RUNNING THE OVA

There are also OVAs for equivalent images supporting various ESXi compatibility levels. To configure these OVAs, use the following steps after setting the IP address with "nmtui".

-- 5.5 or higher: http://131123009294999359.public.ecstestdrive.com/ova/atmos-s3-proxy-5_5-v1.0rc4.ova⁠

The OVA has two predefined users, "root" and "atmos-s3-proxy", both with initial passwords of "ChangeMe". Change these passwords to something more secure.

Log in to the VM as "atmos-s3-proxy" and edit "/opt/dellemc/atmos-s3-proxy/atmos-s3-proxy.ini". At a minimum, you must specify the source and target URLs, and the base URL for orchestration data.

(optional) The proxy will use port 9020 by default. If you wish to use a different port, you need to log in as root and do the following. 3a. Edit "/opt/dellemc/atmos-s3-proxy/uwsgi-s3.ini" and change the value of "http11-socket" to the new port. 3b. Open the firewall to the new port: "firewall-cmd --zone=public --add-port=<your_port>/tcp --permanent" 3c. Reload the firewall: "firewall-cmd --reload"

After these steps, restart the proxy with "sudo systemctl restart atmos-s3-proxy".