commonworkflowlanguage/cwlviewer repository overview

⁠CWL Viewer

This is a Spring Boot⁠ MVC application which fetches Common Workflow Language⁠ files from a Github repository and creates a page for it detailing the main workflow and its inputs, outputs and steps.

⁠Using CWL Viewer

You are recommended to use the production instance of CWL Viewer at https://view.commonwl.org/⁠ which runs the latest release⁠. Any downtime should be reported on the gitter chat for cwlviewer⁠.

⁠Running

If you are a developer, or you want to use the CWL Viewer in a closed environment, then you can run your own instance.

⁠Recommended - Running with Docker

This application can be started with Docker⁠ and Docker Compose⁠ and Docker Compose is the recommended method of running or developing this codebase.

Then run the following commands to clone the project in your local system.

 git clone https://github.com/common-workflow-language/cwlviewer.git
 cd cwlviewer

In the project directory, to start CWLViewer exposed on port 8080, run:

docker compose up

The web server will connect to a local host, you'll see the message saying "Tomcat started on port(s):8080".

To see the locally running CWL Viewer app, visit http://localhost:8080/⁠ in your web browser.

To stop and remove:

docker compose down

If you change the source code, then use this docker-compose.override.yml and re-build with docker compose build:

version: '3.9'
services:
  spring:
    build: .

See the docker-compose.yml⁠ file for details.

If you have modified the source code, then you may want to build the docker image locally first:

docker build -t commonworkflowlanguage/cwlviewer .

⁠Running Spring Boot locally for development, with PostgreSQL and Jena Fuseki in Docker

Create docker-compose.override.yml:

version: '3.9'
services:
  postgres:
    ports:
     - "5432:5432"
  sparql:
    ports:
     - "3030:3030"

Then start the containers:

docker compose up

Then start Spring Boot locally:

mvn spring-boot:run -Dserver.port=7999

Now you can connect to http://localhost:7999⁠ in your browser.

⁠Deleting the data volumes to reset state

To completely reset the state, you must delete the data volumes:

docker compose down
docker volume rm  cwlviewer_bundle cwlviewer_git cwlviewer_graphviz cwlviewer_postgres cwlviewer_sparql

⁠Running without Docker

⁠Requirements

⁠PostgreSQL

You will need to have PostgreSQL⁠ running, by default on localhost:5432

If you are running from the command line, you can override this by supplying system properties like -Dspring.datasource.url=jdbc:postgresql://localhost:5432/cwlviewer and -Dspring.datasource.password=sa

⁠Apache Jena Fuseki (or alternative SPARQL server)

You will also need to have a SPARQL server such as Apache Jena Fuseki⁠ running, by default on localhost:3030

⁠Ruby and Licensee

To retrieve license information, CWL Viewer uses the Licensee⁠ Ruby Gem. To install it, configure Ruby⁠ on your environment and then run

gem install licensee

You may use a dependency from your operating system package manager if you prefer too, e.g. ruby-licensee for Ubuntu LTS 22.04.1.

Before running Maven, try running licensee in the command-line to verify it was installed successfully.

⁠Compiling and Running

To compile you will need Java 17⁠ or a compatible distribution (e.g. Eclipse Adoptium⁠) and version, as well as Apache Maven 3⁠ (apt install maven).

Spring Boot uses an embedded HTTP server. The Spring Boot Maven plugin includes a run goal which can be used to quickly compile and run it:

$ mvn spring-boot:run

Alternatively, you can run the application from your IDE as a simple Java application by importing the Maven project.

You need to install Graphviz⁠ for all unit tests to pass.

You can create an executable JAR file by using:

mvn clean install

Afterwards, run:

java -jar target/cwlviewer*.jar

(The exact filename will vary per version)

Once CWL Viewer is running, you should see log output somewhat like:

()..)
s.b.c.e.t.TomcatEmbeddedServletContainer : Tomcat started on port(s): 8080 (http)
org.researchobject.CwlViewerApplication  : Started CwlViewerApplication in 28.604 seconds

Now check out http://localhost:8080/⁠ to access CWL Viewer.

⁠Configuration

There are a variety of configuration options detailed in the application configuration file⁠ which can be adjusted.

When deploying with docker, these can be overridden externally by creating/modifying docker-compose.override.yml as follows:

version: '3.9'
services:
  spring:
    environment:
            applicationName: Common Workflow Language Viewer
            applicationURL: https://view.commonwl.org
            cacheDays: 1

The properties can alternatively be provided as system properties on the command line, e.g. -DcacheDays=1 or via a variety of other methods supported by Spring Boot⁠

⁠Dump/restore

While you can perform backup of the Docker volumes, for larger upgrades of CWL Viewer it is recommended instead to do a JSON dump and re-load, which will force CWL Viewer to fetch and parse again.

The script dump.py can be used for regular backups, it will store the full output of /workflows as one or multiple timestamped JSON files (you can use gzip to compress them):

$ python dump.py --viewer https://view.commonwl.org/ --output /var/backups --page 0 --size 100
  INFO:Viewer URL: https://view.commonwl.org/
  INFO:Output: /var/backups
  INFO:Dumping workflows from https://view.commonwl.org/, page 0, size 100 to /var/backups

$ python dump.py -o /var/backups -a
  INFO:Viewer URL: https://view.commonwl.org/
  INFO:Output: /var/backups
  INFO:Dumping all the workflows from https://view.commonwl.org/ to /var/backups
  100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 16/16 [04:39<00:00, 17.49s/it]

The script load.py (requires Python 3) can be used to restore from such JSON dumps:

./load.py /var/backups/cwl/2018-06-06T135133+0000.json.gz https://view.commonwl.org/

The optional parameter --no-commits can be added to skip those entries that look like a commit ID. Note that this might break previous permalinks.

⁠Documentation

2017 Poster https://doi.org/10.7490/f1000research.1114375.1?⁠

2017 Video overview https://youtu.be/_yjhVTmvxLU⁠

2017 Technical Report https://doi.org/10.5281/zenodo.823295⁠

⁠License

Distributed under the Apache License, Version 2.0⁠. See the file LICENSE.md⁠ for details, and NOTICE.md⁠ for required attribution notices.

⁠Contribute

Feel free to contribute! You may raise an issue⁠, provide a pull request⁠ or join the gitter chat for cwlviewer⁠!

⁠Changelog

See CHANGELOG⁠

⁠Making a development snapshot container image

(and optionally publishing that image to DockerHub)

# confirm the build arguments
# if these don't look correct, troubleshoot before continuing.
BUILD_DATE=$(date -u +'%Y-%m-%dT%H:%M:%SZ') VCS_REF=$(git rev-parse HEAD) VERSION=$(git describe)
echo BUILD_DATE=${BUILD_DATE} VCS_REF=${VCS_REF} VERSION=${VERSION}
# build the container image
docker build --build-arg BUILD_DATE=${BUILD_DATE} --build-arg VCS_REF=${VCS_REF} \
  --build-arg VERSION=${VERSION} \
  -t cwlviewer:${VERSION} .
# the rest is optional
docker tag cwlviewer:${VERSION} docker.io/commonworkflowlanguage/cwlviewer:${VERSION}
docker tag cwlviewer:${VERSION} quay.io/commonwl/cwlviewer:${VERSION}
docker push docker.io/commonworkflowlanguage/cwlviewer:${VERSION}
docker push quay.io/commonwl/cwlviewer:${VERSION}

⁠Making a release and publishing to GitHub, DockerHub, and Quay.io

After CHANGELOG.md has been updated and the -SNAPSHOT suffix removed from pom.xml, run the following:

git checkout main
git pull
new_version=1.4.3  # CHANGEME
# create an annotated git tag
git tag -a -m "release version ${new_version}" v${new_version}
# confirm the build arguments
# if these don't look correct, troubleshoot before continuing.
# for example, was your tag an annotated (-a) tag?
BUILD_DATE=$(date -u +'%Y-%m-%dT%H:%M:%SZ') VCS_REF=$(git rev-parse HEAD) VERSION=$(git describe)
echo BUILD_DATE=${BUILD_DATE} VCS_REF=${VCS_REF} VERSION=${VERSION}
# build the container image
docker build --build-arg BUILD_DATE=${BUILD_DATE} --build-arg VCS_REF=${VCS_REF} \
  --build-arg VERSION=${VERSION} \
  -t cwlviewer:${VERSION} .
# tag this container image in preparation for pushing to Docker Hub and Quay.io
docker tag cwlviewer:${VERSION} docker.io/commonworkflowlanguage/cwlviewer:${VERSION}
docker tag cwlviewer:${VERSION} docker.io/commonworkflowlanguage/cwlviewer:latest
docker tag cwlviewer:${VERSION} quay.io/commonwl/cwlviewer:${VERSION}
docker tag cwlviewer:${VERSION} quay.io/commonwl/cwlviewer:latest
# push the container image to Docker Hub and Quay.io
docker push docker.io/commonworkflowlanguage/cwlviewer:${VERSION}
docker push docker.io/commonworkflowlanguage/cwlviewer:latest
docker push quay.io/commonwl/cwlviewer:${VERSION}
docker push quay.io/commonwl/cwlviewer:latest
# upload the annotated tag to GitHub
git push --tags
git push

Then copy the changelog into https://github.com/common-workflow-language/cwlviewer/releases/new⁠ using the tag you just pushed.

Finally, make a new PR to bump the version and restore the -SNAPSHOT suffix in pom.xml.

⁠Thanks

Developers and contributors⁠ include:

• Mark Robinson http://orcid.org/0000-0002-8184-7507⁠
• Stian Soiland-Reyes http://orcid.org/0000-0001-9842-9718⁠
• Michael Crusoe http://orcid.org/0000-0002-2961-9670⁠
• Carole Goble http://orcid.org/0000-0003-1219-2137⁠
• Charles Overbeck https://github.com/coverbeck⁠
• Finn Bacall http://orcid.org/0000-0002-0048-3300⁠
• Osakpolor Obaseki https://github.com/obasekiosa⁠
• Bruno P. Kinoshita https://orcid.org/0000-0001-8250-4074⁠

Thanks to:

• eScience Lab⁠ at The University of Manchester⁠
• BioExcel Center of Excellence for Computational Biomolecular Research⁠
• • European Commission's H2020 grant 675728⁠
• Common Workflow Language⁠ community and the CWL Gitter room⁠