srp33/wishbuilder-cli repository overview

⁠WishBuilderCI

Continuous Integration Pipeline which tests and merges pull requests to WishBuilder⁠

⁠Purpose

WishBuilder is an open source project that provides biology-related datasets to Geney, a service that makes the data easier to filter and query for research. WishBuilder allows users to submit code which gathers and reformats data pulled from public Web servers into a consistent format described on the project wiki⁠.

WishBuilderCI (Continuous Integration) manages the pull requests submitted to WishBuilder by detecting them automatically, testing that the ouput data are consistent with the WishBuilder requirements, and adding the datasets created by the code contained in each passed dataset to Geney.

⁠How it Works

⁠Environment

• The WB_PATH environment variable must be set to a directory path containing the WishBuilder-CLI repository
• 'private.py' must exist with the following constant variables
  • GH_TOKEN (The token associated with a collaborator in order to receive information from the GitHub API)
  • WISHBUILDER_EMAIL (The email address used to send status reports to users)
  • WISHBUILDER_PASS (The password associated with the WISHBUILDER_EMAIL account)

⁠Docker

WishBuilderCI requires Docker⁠ to test code in an environment container. The "wishbuilder-cli" image used to create each container can be pulled from Docker Hub with this command:

docker pull srp33/wishbuilder-cli

If you would like to experiment within this container, you can explore the environment within a bash shell by using the following command:

docker run -it -v $(pwd):/app --rm srp33/wishbuilder-cli /bin/bash

⁠App Structure

This is the file structure of the app. Any folder not included in the repository is in the .gitignore and will be auto-generated if it doesn't exist, except for the private.py file which must be provided. The history.sql file keeps a record of all the pull requests, and the SHA's tested for each commit to them. The files produced by successful tests are moved into the RawDatasets directory. After the datasets are converted to the correct format for Geney, the data is stored in the GeneyDatasets directory.

• /app/ (WB_DIRECTORY)
  • WishBuilder-CLI (This repository)
    • all files in repository plus these additional files (in .gitignore):
    • testing/ (location for downloading and testing pull requests)
    • private.py
    • history.sql
    • RawDatasets/
      • Dataset1/
        • data.tsv.gz
        • metadata.tsv.gz
        • config.yaml
        • description.md
      • Dataset2
        • ...
      • ...
    • GeneyDatasets/
      • Dataset1/
        • data.h5
        • metadata.sql
        • metadata.json
        • description.json
      • Dataset2/
        • ...
      • ...
  • GeneyTypeConverter/
    • (Type Converter Repository⁠)

⁠Execution (Docker-Compose)

1. Make sure that the WB_PATH environment variable is set.
2. Make sure that private.py exists with the required constants.
3. From the repository root, use docker-compose to start listening for pull requests:

docker-compose up -d

Stop WishBuilder-CLI with the stop command:

docker-compose down

Restart a pull request:

Working on this...

⁠Results

Status reports are emailed to the GitHub user who created the last commit to the pull request, provided that they have configured their email address using their git --config settings.