fnndsc/pl-shexec repository overview

⁠Execute shell-type commands across input spaces

pl-shexec is a ChRIS⁠ DS plugin executes somewhat arbitrary shell commands across files in its input space. These commands are mostly provided by the default slim-bullseye of the parent container.

⁠Abstract

Most ChRIS plugins provide single-purpose, dedicated operations which can vary considerably in scope -- for example, straightforward image type conversion, or more complex volumetric segmentation. Regardless of the complexity of the operation, they remain mostly single-purposed. In some cases, however, a more general purposed plugin can be quite useful: one that provides a larger space of simpler functionality. A typical example of such a plugin would be one that leverages shell-type operations and applies them over files in the input space. Such shell operations could be used to rename files/directories, zip/unzip data, perform file/directory filtering, check for load average, etc. This repo provides such a shell executor, called shexec.

⁠Installation

pl-shexec is a ChRIS⁠ plugin, meaning it can run from either within ChRIS or the command-line.

⁠Local Usage

To get started with local command-line usage, use Apptainer⁠ (a.k.a. Singularity) to run pl-shexec as a container:

singularity exec docker://fnndsc/pl-shexec shexec [--args values...] input/ output/

To print its available options, run:

singularity exec docker://fnndsc/pl-shexec shexec --man

⁠Examples

shexec requires two positional arguments: a directory containing input data, and a directory where output data is created. First, create the input directory and move input data into it.

mkdir incoming/ outgoing/
mv some.dat other.dat incoming/
singularity exec docker://fnndsc/pl-shexec:latest shexec [--args] incoming/ outgoing/

⁠Development

This section is of interest to developers.

⁠Building

Build a local container image:

docker build -t localhost/fnndsc/pl-shexec .

⁠Running

Mount the source code shexec.py into a container to try out changes without rebuild. In the general case:

docker run --rm -it --userns=host                                           \
    -v $PWD/shexec.py:/usr/local/lib/python3.11/site-packages/shexec.py:ro  \
    -v $PWD/in:/incoming:ro -v $PWD/out:/outgoing:rw -w /outgoing           \
    localhost/fnndsc/pl-shexec shexec /incoming /outgoing

or more concretely:

docker run --rm -it --userns=host                                           \
    -v $PWD/shexec.py:/usr/local/lib/python3.11/site-packages/shexec.py:ro  \
    -v $PWD/in:/incoming:ro -v $PWD/out:/outgoing:rw -w /outgoing           \
    localhost/fnndsc/pl-shexec shexec /incoming /outgoing                   \
    --fileFilter jpg                                                        \
    --exec "convert %inputWorkingDir/%inputWorkingFile
    %outputWorkingDir/%_rmext_inputWorkingFile.png"

to file all files that have jpg in their string names and run a CLI convert operation.

⁠Testing

Run unit tests using pytest. It's recommended to rebuild the image to ensure that sources are up-to-date. Use the option --build-arg extras_require=dev to install extra dependencies for testing.

docker build -t localhost/fnndsc/pl-shexec:dev --build-arg extras_require=dev .
docker run --rm -it localhost/fnndsc/pl-shexec:dev pytest

⁠Release

Steps for release can be automated by Github Actions⁠. This section is about how to do those steps manually.

⁠Increase Version Number

Increase the version number in setup.py and commit this file.

⁠Push Container Image

Build and push an image tagged by the version. For example, for version 1.2.3:

docker build -t docker.io/fnndsc/pl-shexec:1.2.3 .
docker push docker.io/fnndsc/pl-shexec:1.2.3

⁠Get JSON Representation

Run chris_plugin_info⁠ to produce a JSON description of this plugin, which can be uploaded to a ChRIS Store.

docker run --rm localhost/fnndsc/pl-shexec:dev chris_plugin_info > chris_plugin_info.json