A ChRIS plugin FS app that scp's data from a remote host
5.4K
.. image:: https://img.shields.io/docker/v/fnndsc/pl-pull_scp?sort=semver :target: https://hub.docker.com/r/fnndsc/pl-pull_scp
.. image:: https://img.shields.io/github/license/fnndsc/pl-pull_scp :target: https://github.com/FNNDSC/pl-pull_scp/blob/master/LICENSE
.. image:: https://github.com/FNNDSC/pl-pull_scp/workflows/ci/badge.svg :target: https://github.com/FNNDSC/pl-pull_scp/actions
.. contents:: Table of Contents
This plugin application is used to recursively copy data from a remote host into an analysis root node.
pull_scp is a ChRIS fs-type application that produces data for an analysis tree by copying data off a remote host using (recursive) scp.
Of course this assumes that the user executing this plugin has the correct login credentials to access the resource. Credentials are defined in either:
Other than login credentials, this plugin also needs a filepath in the remote user space. All files and directories rooted in this file filepath are copied into this plugin's outputdir.
This plugin/app is not considered or purported to be secure! One deployment vector has the login credentials contained within an .env file in cleartext and copied into the container. Deploying such a container will expose login credentials! For better security, supply login credentials from the CLI or from within environment variables:
.. code:: bash
export ENVIRONMENT=development
export SSH_REMOTE_HOST=1.1.1.1
export SSH_USERNAME=yourname
export SSH_PASSWORD=yourpassword
export SSH_KEY_FILEPATH=/usr/local/src/key.pub
export SCP_DESTINATION_FOLDER=/tmp
Most of the innards of this plugin are lightly adapted from the most excellent paramiko tutorial of Todd Birchard:
.. code::
docker run --rm fnndsc/pl-pull_scp pull_scp \
[-h] [--help] \
[--json] \
[--man] \
[--meta] \
[--savejson <DIR>] \
[-v <level>] [--verbosity <level>] \
[--version] \
[--username <username>] \
[--password <password>] \
[--host <hostname>] \
[--sshPubKeyFile <pubKeyFile>] \
--filepath <filepath> \
<outputDir>
Arguments
.. code::
--filepath <filepath>
The path in the <hostname>'s filesystem to pull. This is technically
the only required argument of this plugin. All user specific
credentials are assumed to be container in the .env file or set in
environment variables. Note of course that all the credentials can
be overriden with CLI flags.
[--username <username>]
The username in the remote host.
[--password <password>]
The <username>'s password to connect to the remote host.
[--sshPubKeyFile <pubKeyFile>]
The ssh public key file to use in this session.
[--host <hostname>]
The hostname to access.
[-h] [--help]
If specified, show help message and exit.
[--json]
If specified, show json representation of app and exit.
[--man]
If specified, print (this) man page and exit.
[--meta]
If specified, print plugin meta data and exit.
[--savejson <DIR>]
If specified, save json representation file to DIR and exit.
[-v <level>] [--verbosity <level>]
Verbosity level for app. Not used currently.
[--version]
If specified, print version number and exit.
Install
~~~~~~~
Installation is either via ``docker`` (recommended) or directly from ``PyPI`` (less recommended).
From docker
^^^^^^^^^^^
.. code:: bash
docker pull fnndsc/pl-pull_scp
Getting inline help is:
.. code:: bash
docker run --rm fnndsc/pl-pull_scp pull_scp --man
From PyPI
^^^^^^^^^
.. code:: bash
pip install pull_scp
For this use case, copy your public key to ``/tmp/key.pub`` and create an environment file ``/tmp/.env`` that either contains your login data/credentials or contains "dummy" data. The ``.env`` _must_ exist for the app to work -- regardless of the validity of its data.
Run
~~~
Since ``docker`` is the recommended usagage deployment, the following instructions are ``docker`` based.
Being an _FS_ plugin, you should specify an output directory using the ``-v`` flag to ``docker run``.
.. code:: bash
docker run --rm -u $(id -u) \
-v $(pwd)/out:/outgoing \
fnndsc/pl-pull_scp pull_scp \
--username johnnyapple \
--password 'mysecret' \
--host computer.org \
--sshPubKeyFile ~/.ssh/rsa_pub.key \
--filepath /home/johnnyapple/data \
/outgoing
Development
-----------
Build the Docker container:
.. code:: bash
docker build -t local/pl-pull_scp .
Run unit tests:
.. code:: bash
docker run --rm local/pl-pull_scp nosetests
Examples
--------
Using login credentials stored in the container's `.env` file:
All relevant login credentials are stored in /tmp/.env in the container. Note this method embeds the .env file in the container where it can be potentially expose credentials!
.. code:: console
mkdir out && chmod 777 out
docker run --rm -u $(id -u) --name=pl-pull_scp \
-v $PWD/out:/outgoing \
-it \
local/pl-pull_scp pull_scp -v 1 \
--filepath /home/rudolphpienaar/Desktop \
outgoing
10-15-2021 16:25:38 | INFO: /tmp/key.pub uploaded to 192.168.1.216
_ _
| | |
_ __ _ _| | | ___ ___ _ __
| '_ \| | | | | | / __|/ __| '_ \
| |_) | |_| | | | \__ \ (__| |_) |
| .__/ \__,_|_|_| |___/\___| .__/
| | ______ | |
|_| |______| |_|
Version: X.Y.Z
10-15-2021 16:25:38 | INFO: (remote): du -ksh /home/rudolphpienaar/Desktop: 142M /home/ rudolphpienaar/Desktop
10-15-2021 16:25:39 | INFO: Pulling [email protected]:/home/rudolphpienaar/Desktop...
10-15-2021 16:25:53 | INFO: Remote contents pulled to /outgoing
10-15-2021 16:25:53 | INFO: (NOTE: if running in a container, and doing a volume mapping,
the destination dir name might not match the host dirname!)
10-15-2021 16:25:53 | INFO: (local): du -ksh /outgoing: 142M /outgoing
Using login credentials from the CLI:
Credentials are supplied at run time. Contents of ``/tmp/.env`` are not used.
.. code:: console
docker run --rm -u $(id -u) --name=pl-pull_scp \
-v $PWD/out:/outgoing \
-it \
local/pl-pull_scp pull_scp -v 1 \
--filepath /home/chris/Pictures \
--host 192.168.1.200 \
--username chris \
--password XXXXXXXXXXX \
/outgoing
10-15-2021 17:05:13 | INFO: /tmp/key.pub uploaded to 192.168.1.200
_ _
| | |
_ __ _ _| | | ___ ___ _ __
| '_ \| | | | | | / __|/ __| '_ \
| |_) | |_| | | | \__ \ (__| |_) |
| .__/ \__,_|_|_| |___/\___| .__/
| | ______ | |
|_| |______| |_|
Version: X.Y.Z
10-15-2021 17:05:14 | INFO: (remote): du -ksh /home/chris/Pictures: 81M /home/chris/Pictures
10-15-2021 17:05:14 | INFO: Pulling [email protected]:/home/chris/Pictures...
10-15-2021 17:06:01 | INFO: Remote contents pulled to /outgoing
10-15-2021 17:06:01 | INFO: (NOTE: if running in a container, and doing a volume mapping,
the destination dir name might not match the host dirname!)
10-15-2021 17:06:01 | INFO: (local): du -ksh /outgoing: 156M /outgoing
(note that in this example the ``local`` directory is larger than the ``remote``. This occurs when the remote directory contains symbolic links -- each symbolic link is actually translated into the target file when pulled)
_-30-_
.. image:: https://raw.githubusercontent.com/FNNDSC/cookiecutter-chrisapp/master/doc/assets/badge/light.png
:target: https://chrisstore.co
Content type
Image
Digest
Size
65.4 MB
Last updated
about 5 years ago
docker pull fnndsc/pl-pull_scp