managedfiletransfer/prefect-managedfiletransfer repository overview

Turn a prefect.io server into a managed file transfer solution. UI and Programatic creation of cron style jobs (aka Flows!) to upload and download files easily between servers. Support local, SFTP remotes plus any Cloud storage supported by rclone - so thats aws, azure, google, sharepoint, and many more⁠ out of the box.

Using prefect for managed file transfer means retries, logging, multi node and high availability⁠ come as standard - turning prefect into a reliable enterprise ready file transfer solution.

This package is not the fastest solution to move files around, but it prioritises reliability and ease of use, making it an excellent choice for replacing both quick cron job copy scripts and enterprise managed file transfer applicances.

Key features

• Copy and move files between almost any storage system easily.
• Reliable file moving with checksumming, file size checking etc.
• Smart and safe moving - settings to allow/block overwriting and to only copy files if they are new or changed.
• Unzip/Untar compressed folders after downloading them.
• Repath files as you move them.
• Complex filtering and ordering of files - by path, age, size etc. Pattern matching with regular expressions.
• Leverage Prefect.IO built in scheduling and orchestration capabilities:
  • Transfer files on complex cron schedules
  • notifications on success/failure - slack, email, etc
  • Highly available server architecture - database server + multi-node workers and front ends.
• Available as a PyPi package⁠ for integration into existing self hosted and cloud prefect deployments, and as a docker image/appiance⁠

Example use cases:

• Once per day SSH into my database server and copy the latest *.bkup file to a central storage location.
• Monitor a local network share directory for new files and automatically upload them to a cloud storage bucket.
• Schedule a weekly job to synchronize files between two remote servers.
• Move log files from a SSH available web server older than 30 days to a cold storage location.
• Copy file yyyy-MM-dd.zip from a remote server, where yyyy-MM-dd matches todays date, to a local directory and then unzip it.
• Download any file in an S3 bucket larger than 1GB and store it in a local directory.

Visit the full docs here⁠.

⁠Installation - docker

Run prefect managed file transfer in a docker container, like an applicance. See Docker hub for a list of images⁠

Note this is ephemeral - prefect has lots of docs on how to setup a database server with it.

# run prefect server in a self-removing container port-forwarded to your local machine’s 4200 port:
docker run --rm -it -p 4200:4200 managedfiletransfer/prefect-managedfiletransfer:latest

Check out the Github site for more info⁠